Information processing device, information processing method, and program

Abstract

To allow auto-focusing in an imaging apparatus using an event-based sensor, which is the problem to be solved by the present invention.SOLUTION: An information processing apparatus according to the present invention that solves the problem has: control means that controls the position of a focus lens of an imaging apparatus; acquisition means that acquires an address event signal indicating the positions of pixels where a change in luminance occurs and the time of the change in a predetermined period; and determination means that acquires an evaluation value based on the address event signal for every position of the focus lens to determine a focus position of the focus lens.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to autofocus for event-based sensors. [Background technology]

[0002] An event-based sensor is known that outputs a change in luminance of each pixel as an address event signal in real time (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-134271 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to enable autofocus in an imaging device using an event-based sensor. [Means for solving the problem]

[0005] The information processing device according to the present invention that solves the above problems comprises: Equipped with a photoelectric conversion element a control means for controlling the position of a focus lens of the imaging device; The detected Brightness change Indicates the direction of an acquisition means for acquiring an address event signal, for each position of the focus lens; Based on the number of pixels that detect the change in brightness, Get the rating value Evaluation tools and , Based on the evaluation value a determining means for determining a focus position of the focus lens; The evaluation means acquires the evaluation value based on the number of pixels whose luminance changes in a positive direction or the number of pixels whose luminance changes in a negative direction based on the address event signal, and the determination means determines the in-focus position of the focus lens based on the position of the focus lens when the evaluation value becomes equal to or less than a predetermined reference value. It is characterized by: [Effects of the Invention]

[0006] According to the present invention, autofocusing becomes possible in an imaging device that uses an event-based sensor. [Brief explanation of the drawings]

[0007] [Figure 1] A block diagram showing an example of the hardware configuration of an information processing device. [Figure 2] A diagram showing an example of the configuration of an event-based sensor [Figure 3] A block diagram showing an example of the functional configuration of an information processing device. [Figure 4] A diagram showing an example of a subject [Figure 5] A diagram showing an example of a focus method [Figure 6] 10 is a flowchart illustrating a process executed by an information processing device. [Figure 7] 10 is a flowchart illustrating a process executed by an information processing device. [Figure 8] A diagram showing an example of a focus method [Figure 9] A diagram showing an example of a focus method [Figure 10] A diagram showing an example of a focus method [Figure 11] A diagram showing an example of a focus method [Figure 12] A diagram showing an example of a display DETAILED DESCRIPTION OF THE INVENTION

[0008] There are high expectations for the application of event-based sensors to cameras for security applications. Security cameras sometimes shoot in low-light environments, such as during nighttime surveillance. In such scenes, the aperture value is often set to a small value to increase the amount of light received by the sensor. Setting a small aperture value results in a shallow depth of field, so it is often necessary to adjust the focus on the subject. For this reason, an autofocus (AF) function that allows users to easily adjust the focus is desirable.

[0009] Contrast AF is a well-known autofocus technology, but it is difficult to apply to cameras that use event-based sensors for the following reasons. Contrast AF determines the in-focus position based on a contrast evaluation value that represents the contrast level within a specific area within the image (hereafter referred to as the AF evaluation frame). In other words, the in-focus position is determined to be when the contrast evaluation value reaches its peak. Typical methods for calculating the contrast evaluation value include using the ratio of the maximum and minimum brightness within the AF evaluation frame, or using the difference between the maximum and minimum brightness.

[0010] However, with event-based sensors, changes in pixel brightness can only be expressed as three values (for example, +1 for a positive change in brightness, -1 for a negative change in brightness, and 0 for no change in brightness). With event-based sensors, both positive and negative changes in brightness often occur, except when the light incident on the sensor changes uniformly, such as when the brightness of the lighting or the diameter of the aperture is changed. For this reason, even if a known formula for calculating contrast evaluation values is applied, the evaluation value will be the same regardless of whether the image is in focus, making it impossible to determine the in-focus position. Therefore, the problem that the present invention aims to solve is to enable autofocus in an imaging device that uses an event-based sensor.

[0011] Hereinafter, an imaging device according to an embodiment of the present invention will be described with reference to the drawings. In this regard, components having the same functions in all the drawings will be assigned the same numbers, and repeated description will be omitted.

[0012] <Embodiment 1> <Hardware configuration of the imaging device 100: Figure 1> FIG. 1 is a schematic diagram showing an example of the hardware configuration of an imaging device (information processing device) 100. While the imaging device 100 is specifically an imaging device having an event-based sensor, the image processing function and the image analysis function (motion detection) may be performed by separate devices. The imaging device 100 includes an imaging unit 101 consisting of an imaging optical system 1010 and a photoelectric conversion element 1011, a CPU 102, a memory 103, a display unit 104, and an operation unit 105. The photoelectric conversion element 1011 is an event-based sensor that outputs an address event signal in response to received incident light. The event-based sensor detects a change in luminance for each pixel as an event, and the address event signal indicates the position and time of the pixel where the luminance change occurred. The imaging optical system 1010 is specifically a light-receiving lens that receives incident light and forms an image on the photoelectric conversion element 1011. The CPU 102 reads and executes the OS and other programs stored in the memory 103, controls each connected component, and performs calculations and logical judgments for various processes. The processes executed by the CPU 102 include the information processing according to this embodiment. The CPU 102 also controls the focus and aperture of the imaging optical system 1010, the photoelectric conversion element 1011, and other functions. The memory 103 is, for example, a hard disk drive or an external storage device, and stores programs and various data related to the information processing according to this embodiment. The display unit 104 is, for example, a display device that displays the results of calculations performed by the information processing device 100 in accordance with instructions from the CPU 102. The display device may be any type, such as a liquid crystal display device, a projector, or an LED indicator. The operation unit 105 is, for example, a touch panel, keyboard, mouse, or robot controller, and serves as a user interface that accepts input instructions from a user. The information processing device 100 may also have mechanisms other than the hardware configurations listed here.

[0013] <Photoelectric conversion element: Figure 2> An example of an event-based sensor according to this embodiment will be described. The event-based sensor counts the number of incident photons and determines the timing when the counted number of photons exceeds a predetermined threshold. The event-based sensor also measures the time (clock count) required for the number of photons to reach or exceed a first threshold, and detects a change in luminance by comparing the required times. Specifically, when the previously measured required time is T0 and the latest required time is T, if the difference T-T0 is greater than or equal to a second threshold, a negative change in luminance is detected. If the difference T0-T is greater than or equal to the second threshold, a positive change in luminance is detected. If the difference between T and T0 is less than the second threshold, no change in luminance is detected. The second threshold is a value greater than or equal to zero, and is set in accordance with a preset value or other parameters.

[0014] The detailed configuration will be described below. Fig. 2(a) is a diagram showing an example configuration of the photoelectric conversion element 103. The photoelectric conversion element 103 is composed of a pixel unit 110 and a peripheral circuit 120. The peripheral circuit 120 includes a vertical arbitration circuit 121 and a horizontal readout circuit 122.

[0015] FIG. 2(b) illustrates an example of the configuration of each pixel unit constituting an event-based sensor. The pixel unit 110 includes a photoelectric conversion unit 111, a pixel counter 112, a time counter 113, a first decision circuit 114, a memory 115, a comparator 116, a second decision circuit 117, a response circuit 118, and a selection circuit 119. The photoelectric conversion unit 111 includes an avalanche photodiode (SPAD) operating in Geiger mode, and the pixel counter 112 counts the number of photons incident on the photoelectric conversion unit 111. The time counter 113 counts the time during which a photon is incident on the photoelectric conversion unit 111. Using a SPAD to configure an event-based sensor enables detection of luminance changes at the level of a single photon. By detecting luminance changes at the level of a single photon, an address event signal can be acquired even in night vision conditions, such as at night.

[0016] When the number of photons counted by pixel counter 112 reaches a first threshold, first decision circuit 114 stops counting time by time counter 113. Past count values of time counter 113 are stored in memory 115, and comparator 116 is used to determine the difference between the current count value of time counter 113 and the past count value of time counter 113.

[0017] If the difference count value is equal to or greater than the second threshold, the second determination circuit 117 sends a request signal to the vertical arbitration circuit 121 via the response circuit 118. The response circuit 118 receives a response from the vertical arbitration circuit 121 indicating whether or not the output of address event data is permitted. If the difference count value is less than the second threshold, the response circuit 118 does not send a request signal.

[0018] When the response circuit 118 receives a response indicating permission to output, the selection circuit 119 outputs the count value of the time counter circuit 113 to the horizontal output circuit 122. The horizontal output circuit 122 outputs the received count value as an output signal from the photoelectric conversion element 103 to the detection unit 103.

[0019] Since the differential count value calculated by the comparator 116 corresponds to the reciprocal of the incident frequency of photons, the photoelectric conversion element 103 according to this embodiment has the function of measuring "changes in the incident frequency of photons," i.e., changes in luminance. Furthermore, using the second determination circuit 117, an address event is output only when the difference in the intervals at which the number of incident photons reaches the first threshold is equal to or greater than the second threshold. That is, the photoelectric conversion element outputs the incident frequency when the difference in the incident frequency is equal to or greater than the second threshold, and does not output the incident frequency when the difference is less than the threshold. This configuration realizes an asynchronous photoelectric conversion element that detects changes in luminance as address events in real time for each pixel address.

[0020] <Variations of photoelectric conversion elements> The above describes a case where a photoelectric conversion element is used that uses a SPAD as the photoelectric conversion unit and measures the time at which a photon is incident to detect changes in the frequency of photon incidence. However, the configuration shown in Figure 2 is not necessary as long as the photoelectric conversion element detects changes in luminance as address events in real time. For example, as described in Patent Document 1, a photoelectric conversion element that detects changes in luminance as voltage changes may be used.

[0021] <Image capture device: Figure 3> FIG. 3 shows an example of the functional configuration of an imaging device (information processing device) 100 according to this embodiment. First, an overall overview will be described, followed by details of each function. In FIG. 3, the imaging device 100 has an imaging unit 301, an evaluation unit 302, a control unit 303, an output unit 304, and an operation reception unit 305. The imaging unit 301 has an imaging optical system 10110 equipped with a focus control mechanism 1012 capable of changing the focus position, and a photoelectric conversion element 1011 which is an event-based sensor. The imaging unit 301 corresponds to the imaging unit 101 in FIG. 1. Components with the same reference numerals will be described as having the same functions.

[0022] Specifically, the imaging optical system 1011 is composed of a light-receiving lens, and the focus control mechanism 1012 is composed of a focus lens and an actuator for driving the lens. In this embodiment, it is composed of a stepping motor that can control the position of the focus lens using drive pulses. The imaging optical system 10111 and focus control mechanism 1012 form a subject image on the image plane of the photoelectric conversion element 1011. The photoelectric conversion element 1011 outputs an address event signal corresponding to the incident light it receives. More specifically, it outputs an address event signal indicating the address and time of a pixel where a luminance change occurred. The evaluation unit 302 acquires the address event signal output from the photoelectric conversion element 1011, counts the number of pixels where a luminance change occurred, compares the count with a predetermined threshold, and sends the result to the control unit 303. The control unit 303 controls the focus lens 1012 to drive it to the in-focus position in accordance with the output result of the evaluation unit 302. The output unit 304 displays the address of the pixel where a luminance change occurred, the direction of the luminance change, the AF evaluation frame, etc. The operation reception unit 305 receives, through user operations, settings of an AF evaluation frame, input of a threshold value for an address event (to be described later), etc. The output unit 304 and the operation reception unit 305 may be realized by devices external to the imaging device.

[0023] Each function will be explained in detail below.

[0024] <Evaluation unit 302> The evaluation unit 302 calculates the sum of the total number of pixels in which a luminance change occurs in a specific time range from the output signal (address event signal) of the photoelectric conversion element 1011 and a predetermined threshold value (TH AF ) is calculated as an evaluation value, and this evaluation value is transmitted to the control unit 303. AF ) is a value equal to or greater than 0, and is set based on a preset value or other parameters. The specific time range may be the minimum time width determined by the time resolution of the vertical arbitration circuit 121 of the photoelectric conversion element 1011, or may be integrated over a longer time range. A narrower time width improves the focusing speed, but a wider time width reduces the influence of errors due to random noise and the like, improving focusing accuracy.

[0025] The target area (AF evaluation frame) in which the evaluation unit 302 counts the total number of pixels in which a luminance change has occurred based on the address event signal may be the entire angle of view of the imaging device, or it may be one or more pre-set partial areas. It may also be a range specified by the user via the operation reception unit 305. Weighting may also be performed within the AF evaluation frame. For example, the number of pixels detected in peripheral areas of the AF evaluation frame may be multiplied by a certain coefficient (a value between 0 and 1). Setting it in this manner makes it possible to prioritize the degree of focus on the subject in the center of the AF evaluation frame over the peripheral areas.

[0026] <Control unit 303> The control unit 303 controls the position of the focus lens. Specifically, when the evaluation value transmitted from the evaluation unit 302 is 0 or more (i.e., the total number of pixels in which a luminance change has occurred is equal to or greater than a predetermined threshold value TH AF ), the focus lens 1012 continues to be driven in a constant direction at a predetermined speed. The following explanation will be made using diagrams. FIG. 4 is a diagram illustrating the subject and the background. In the figure, the round object 400 is the subject, and its brightness is higher than that of the background. FIG. 5 shows the image (incident light) formed on the image plane of the event-based sensor 1011 when the subject 400 in FIG. 4 is photographed by the imaging device 100, and the images photographed by the event-based sensor, arranged for each position of the focus lens. In the image photographed by the event-based sensor, a brightness change in the positive direction is represented by white, a brightness change in the negative direction by black, and no brightness change by gray. Position A on the left side of the figure represents the near side, and position F on the right side represents the far side. Note that the address event signal includes the direction of brightness change, and assigns a pixel value of +1 when the brightness of a pixel increases, or -1 when the brightness of a pixel decreases. In other words, in the image, a pixel value of +1 is represented by white, and a pixel value of -1 is represented by black. Here, pixels with no change in brightness have a pixel value of 0 and are displayed in gray in the image.

[0027] As can be seen from FIG. 5, when the focus lens 1012 is positioned far from the in-focus position, as in position A, the subject image is blurred, and the size of the blur decreases as the focus lens moves. The change in the size of the blur is detected as a change in brightness by the event-based sensor, so the number of pixels counted by the evaluation unit 302 is determined based on the threshold value (TH AF ), and the evaluation value is a value greater than or equal to 0. When the focus lens is further driven and the subject enters the depth of field (position C), the blur disappears and the subject image remains unchanged until position E. When the focus lens moves further and reaches position F, the subject blur begins to increase. Since there is less change in the subject image between positions C and E, the number of brightness changes detected by the event-based sensor also decreases. Therefore, the total number of pixels counted by evaluation unit 302 is less than the threshold value (TH AF ), and the evaluation value becomes negative. From the above, it can be determined that the range to which the focus lens is driven during the period when the evaluation value is negative (positions C to E) is the range in which the subject is in focus (hereinafter referred to as the in-focus range).

[0028] Furthermore, the control unit 303 controls the position of the focus lens based on the evaluation value. The control unit 303 counts the time during which the evaluation value is a negative value, or counts the number of drive pulses applied to the stepping motor of the focus lens during the period during which the evaluation value is a negative value. In other words, the control unit 303 counts the time during which the evaluation value is a negative value, or counts the number of drive pulses applied to the stepping motor of the focus lens during the period during which the evaluation value is a negative value. AFThe control unit 303 acquires the amount of movement of the focus lens while the evaluation value is negative (or less). When the evaluation value changes to a value greater than or equal to 0 again, the control unit 303 stops the focus lens 1012 and drives the focus lens in the reverse direction at the same speed for half the time during which the evaluation value was negative. Alternatively, the control unit 303 applies half the number of drive pulses applied to the stepping motor of the focus lens in the reverse direction as those applied during the period when the evaluation value was negative. In other words, when the evaluation value falls outside a predetermined range, the control unit 303 moves the position of the focus lens by a predetermined amount to align the focus lens with the in-focus position. By doing this, the focus lens is positioned at the center of the in-focus range (position D), thereby further improving focusing accuracy. A method for achieving even higher accuracy will be described later.

[0029] In addition, the direction in which to first move the focus lens (near or far direction) can be determined by first slightly moving the lens in one direction and observing the change in the total number of pixels where a luminance change occurred. For example, if the total number of pixels where a luminance change occurred increases, it is best to drive in the opposite direction, and if it decreases, it is best to drive in the same direction.

[0030] <Output unit 304> The output unit 304 is specifically composed of a liquid crystal display device, projector, LED indicator, or the like, and receives the output signal from the event-based sensor 103 to display the address of the pixel where a luminance change occurred and the direction of the luminance change. The image captured by the event-based sensor shown in FIG. 5 is an image displayed by the output unit 304. At the coordinates of the pixel where a luminance change occurred, white is displayed if the luminance change of the pixel was in the positive direction, black if the luminance change was in the negative direction, and gray if there was no luminance change. Furthermore, an AF evaluation frame or the like may be superimposed on the captured image to improve user convenience.

[0031] <Operation Reception Unit 305> The operation reception unit 305 is a part where the user controls the image capturing apparatus 100. Specifically, the operation reception unit 305 receives instructions to start autofocus, sets the AF evaluation frame, and sets the second threshold and the threshold THAF The operation reception unit 305 is configured with, for example, a touch panel, a keyboard, a mouse, a cross key, an operation dial, and the like.

[0032] The output unit 304 and operation reception unit 305 may be provided in an external device connected to the image capture device 100 via a network cable or wireless transmission.

[0033] <Flowchart> The flow of processing when performing the above-described autofocus will now be described. FIG. 6 is a flowchart illustrating processing executed by the imaging device 100. The processing shown in the flowchart in FIG. 6 is executed by the CPU 101 of FIG. 1, which is a computer, in accordance with a computer program stored in memory 103. In the following description, each process (step) will be denoted by adding an S to the beginning, and the process (step) will not be described in detail. However, the imaging device 100 does not necessarily have to perform all of the steps described in this flowchart.

[0034] In S601, the operation unit 107 sets an AF evaluation frame (position and size) based on a user input. If there is no setting from the user, a preset AF evaluation frame is set.

[0035] In S602, the control unit 303 controls the position of the focus lens. Here, the focus lens is moved from the near side as a starting point to the far side at a constant speed.

[0036] In S603, the evaluation unit 302 counts the number of pixels in which a luminance change occurs within the AF evaluation frame during a specific time range based on the address event signal acquired from the image capture unit 301, and calculates a threshold value TH AF The difference between the evaluation value and the actual value is sent as an evaluation value to the control unit 303. Counting is repeated until the evaluation value becomes a negative value. When the evaluation value becomes a negative value, the process proceeds to S604.

[0037] In S604, in response to the evaluation value becoming negative, the control unit 303 starts measuring time. Alternatively, instead of measuring time, the control unit 303 starts counting the number of drive pulses applied to the stepping motor of the focus lens. In other words, the control unit 303 starts measuring the amount of movement of the focus lens.

[0038] In S605, the evaluation unit 302 repeats counting the number of pixels if the evaluation value is a negative number based on the address event signal, and proceeds to S606 when the evaluation value becomes a value of 0 or greater.

[0039] In S606, the control unit 303 stops measuring the time or the number of drive pulses, and calculates the elapsed time or the number of drive pulses from S604. AF The amount by which the focus lens has moved while the focus lens was still below the predetermined value is obtained.

[0040] In S607, the control unit 303 moves the focus lens to the in-focus position based on the measured movement amount of the focus lens. For example, the control unit 303 drives the focus lens in the reverse direction at the same speed as in S602 for half the time during which the evaluation value determined in S606 was negative. Alternatively, the control unit 303 moves the focus lens by applying half the number of drive pulses determined in S606 in the reverse direction. This makes it possible to bring the subject to the center of the depth of field (focus range).

[0041] <Variations in lens drive methods> Although it is easier to calculate and control the focus position if the speed at which the focus lens is driven is constant, it is also possible to improve the focus speed by making the speed variable. For example, the speed can be changed according to the number of pixels in which a luminance change has occurred measured by the evaluation unit 302. Specifically, since it can be determined that there is a distance to the focus position during a period in which a large number of pixels in which a luminance change has occurred are large, the drive speed of the focus lens 1012 is increased and the number of pixels in which a luminance change has occurred reaches the threshold TH AFIn the range close to , the driving speed is reduced to increase the focusing accuracy. By controlling in this way, it is possible to shorten the total time required for focusing.

[0042] <Focus position variation 1> The aforementioned focus range is the combined range of the front and rear depth of field, with the rear depth of field being wider, so the focal plane that provides the best focus on the subject is located on the near side of the center of the focus range.

[0043] The hyperfocal distance can be calculated from three parameters: the focal length and F-number of the imaging optical system 1011, and the allowable circle of confusion that can be determined from the display magnification of the photoelectric conversion element 1011 and the output unit 304. If the focal length is f, the allowable circle of confusion is δ, and the F-number is F, the hyperfocal distance Y can be calculated using the following equation (1). Y=f 2 / (δ×F) (1)

[0044] From this hyperfocal distance and the subject distance estimated for the use case of the image capturing device 100, estimated values of the front depth of field and the rear depth of field can be calculated.

[0045] If the front depth of field is A, the rear depth of field is B, and the estimated subject distance is X, A and B are calculated using the following equations (2) and (3). A = XY / (Y + X) (2) B=XY / (YX) (3)

[0046] Since the true focus plane exists a distance equal to the forward depth of field from the near end of the focus range, the following steps can be taken to bring the best focus plane to the subject.

[0047] As the focus lens moves, the evaluation value becomes negative once, and then the focus lens is stopped when it returns to a value greater than or equal to 0. If the lens has been moved from Near to Far before stopping, the lens is driven in the opposite direction from the stopped position by a distance equivalent to the rear depth of field.

[0048] If the lens is moved from Far to Near, the lens should be driven in the opposite direction by a distance equivalent to the front depth of field. The following explains the case where the lens is driven from Far to Near as an example. First, the ratio of the front depth of field to the rear depth of field (hereinafter referred to as the front-rear depth ratio) derived from equations (2) and (3) is calculated. Next, the time (ΔT) during lens drive during which the evaluation value was negative is allocated based on the front-rear depth ratio. For example, if the front-rear depth ratio is 2:3 (front:rear), 2 / 5 of the negative time ΔT corresponds to the front depth of field. This time represents the time required for the focus lens, driven at a constant speed, to move the focal plane the same distance as the front depth of field. Therefore, if the focus lens is driven in the opposite direction at the same speed for the same amount of time, the subject will be brought into focus. Alternatively, the same result can be obtained by allocating the number of drive pulses applied to the stepping motor of the focus lens in the period during which the evaluation value is negative during lens drive in proportion to the front and rear depths of field, and applying the number of pulses corresponding to the front depth of field in the reverse direction. This method increases the amount of calculation, but enables more accurate autofocus.

[0049] <Focus position variation 2> Another method will now be described. This method shortens the time required for focusing without moving the focus lens from the near end to the far end of the focusing range. The following explanation assumes that the focus lens is driven from the near to far direction. The focus lens is driven to focus from the point at which the evaluation value changes from a value greater than or equal to 0 to a negative value as the focus lens moves. The focus lens is then driven to move the focal plane a distance equal to the forward depth of field, achieving focus. The driving distance of the focus lens in this case is calculated from the forward depth of field calculated using equation (2) and the amount of focal plane movement relative to the amount of focus lens movement (number of drive pulses) (hereinafter referred to as sensitivity). Dividing the forward depth of field by the sensitivity yields the distance the focus lens must be driven, i.e., the number of drive pulses. The sensitivity is a value specific to the imaging optical system 1011 being used and must be obtained in advance, for example by measurement. Although this method requires obtaining sensitivity in advance, it can reduce the time required for focusing because it does not drive the focus lens to the opposite end after reaching one end of the focusing range.

[0050] <Evaluation value variations> A method for calculating an evaluation value that is less susceptible to random noise will now be described. In the above explanation, the total number of pixels in which a luminance change occurred within a specific time span and the threshold value TH AF We have explained a method of using the difference between the two values as the evaluation value. This evaluation value requires less calculation, but since the number of pixels where a luminance change has occurred is accumulated regardless of the direction of the luminance change, luminance changes due to random noise are also included in the count. Here, we will explain a method of using the time until a luminance change in the opposite direction occurs for each pixel within the AF evaluation frame as the evaluation value, as a method of suppressing miscounts due to random noise. The time until a luminance change in the opposite direction occurs is the time difference between when a positive luminance change occurs and when a negative luminance change occurs, and the time difference between when a negative luminance change occurs and when a positive luminance change occurs. This time difference can be calculated by taking the difference in the time information (timestamp) included in the address event. The average value of this time difference within the AF evaluation frame is calculated and used to set the threshold TH TIf it is less than the threshold TH, it is considered to be random noise and the evaluation value is set to 0. T If it is equal to or greater than this, the time difference itself is used as the evaluation value.

[0051] As can be seen from the images captured by the event-based sensor in Figure 6, when the blur of a bright subject shrinks as the focus lens moves, as in positions A and B, the outline of the blur shows a negative change in brightness because the dark background is captured. On the other hand, when the blur expands as the lens moves, as in position E, the outline of the blur shows a positive change in brightness. In the in-focus range from positions C to E, there is no change in the subject image, so no change in brightness is detected. If we focus on the same pixel on the outline of the blur in the images on both sides of the in-focus range (positions B and F), we can see that it changes from black to gray to white. In other words, the time difference from black (negative brightness change) to white (positive brightness change) corresponds to the in-focus range.

[0052] The procedure for performing autofocus using this evaluation value will be described using the flowchart in Fig. 7. S701 and S702 are the same as those in the flowchart in Fig. 4. In S703, the evaluation unit 302 determines whether this evaluation value is 0. As described above, brightness changes in opposite directions immediately before the focus lens enters the in-focus range and immediately after it leaves the in-focus range, so this time difference becomes the evaluation value.

[0053] In other words, when the evaluation value in S703 is a value other than 0, the lens is at the opposite end of the focusing range. T is compared with the threshold TH T If it is less than the threshold value TH, repeat S703. TIf the value is equal to or greater than this, the lens is stopped and the process proceeds to S704. In S704, the control unit 303 moves the focus lens in the reverse direction at the same speed by half the evaluation value. The evaluation value is the time it takes for the focus lens to pass through the focus range, so if the focus lens is driven in the reverse direction for half the time of the evaluation value, the subject can be placed in the center of the focus range. Note that, as with the method described above, the focus position may be adjusted so that the best focus plane is located on the subject, taking into account the front depth of field and the rear depth of field.

[0054] With evaluation values that simply count the total number of pixels where brightness changes have occurred within the AF evaluation frame, it is essentially difficult to distinguish between brightness changes due to random noise and brightness changes in the subject, making it difficult to count them separately. However, with this method, brightness changes in the same direction due to random noise can be removed, making it possible to reduce miscounting due to random noise.

[0055] As described above, the method of this embodiment enables autofocus in an image capture device that uses an event-based sensor, thereby improving user convenience. The control method of this embodiment is particularly effective when quick focus setting is desired.

[0056] <Embodiment 2: Use of Optical Flow> The second embodiment differs in that an optical flow is generated by an evaluation unit 302 that receives the output of a photoelectric conversion element 1011. A commonly known method such as a gradient method or block matching method may be used to generate the optical flow. This makes it possible to suppress the influence of elements other than the subject, enabling autofocus that is more resistant to disturbances. The hardware configuration and functional configuration of the imaging device 100 are the same as those shown in FIGS. 1 and 3 of the first embodiment.

[0057] Figure 8 shows the image formed on the sensor image plane and the images captured by the event-based sensor for each focus lens position. The subject is the same as in Figure 5. Looking at the images captured by the event-based sensor in Figure 8, we can see that in the image just before entering the in-focus range (position A), optical flow is generated along the blurred contour of the subject image, indicating a contraction of the blur. We can also see that in the image captured after the optical flow disappears in the in-focus range and the subject has passed through the in-focus range (position E), optical flow is generated along the blurred contour of the subject image, indicating an expansion of the blur. In other words, by using the time from when the optical flow disappears until it reverses, it is possible to control the focus lens to the in-focus range. Specifically, this can be done as follows.

[0058] First, the evaluation unit 302 finds a group of vectors indicative of the contraction of an object in the generated optical flow. As the focus lens advances and enters the in-focus range, the group of vectors indicating the contraction disappears, and the evaluation unit 302 starts counting the time or the number of drive pulses applied to the focus lens. When the focus lens advances further and leaves the in-focus range, and a group of vectors indicating the expansion of the object appears, the evaluation unit 302 stops counting the time or the number of pulses, and stops the focus lens. The evaluation unit 302 then drives the focus lens in the reverse direction at the same speed for half the time measured by the evaluation unit 302. Alternatively, the evaluation unit 302 applies half the number of pulses measured in the reverse direction to the stepping motor. This makes it possible to position the subject in the center of the in-focus range.

[0059] The advantages of this embodiment will be described with reference to Fig. 9. Fig. 9 shows an image in which, in addition to the subject to be focused on, several distracting subjects are moving, or there is a moving object in the background.

[0060] In the figure, subject 500 in the image formed on the sensor image plane is the subject to be focused on, while subjects 501 and 502 are moving subjects that are not to be focused on. In particular, subject 502 is at a different distance from image capture device 100 than subjects 500 and 501. For this reason, the optical flows of subjects 500 and 501 disappear when they enter the in-focus range because there is no change in the subject blur, but for subject 502, which is at a different distance, subject blur remains at this focus position, so the optical flow also remains.

[0061] The evaluation unit 302 uses only optical flows that represent contraction or expansion of the contour of a certain object to determine focus, and ignores optical flows that represent the contour of a certain object moving in one direction, such as the optical flows of objects 501 and 502 in Figure 9. This makes it possible to suppress the influence of the object, which corresponds to a disturbance component. By controlling the focus lens in the same way as the method described above for only the optical flows that represent contraction / expansion, it becomes possible to place the object in the center of the focus range.

[0062] <GUIのバリエーション1> With an event-based sensor, the subject image disappears when the focus lens enters the in-focus range, which can be confusing for users accustomed to regular imaging devices as they are unable to intuitively determine whether the subject is in focus. In particular, when there are many still subjects, the subject image disappears as the lens stops after the autofocus operation is completed, making it difficult to determine whether the subject is in focus.

[0063] FIG. 10 shows an example of the display on the display unit 104 during autofocus control. Normally, the subject image would disappear during period C, which is the in-focus range, but here, the subject's outline is displayed in a peaking manner, making it easier to recognize that the subject is in focus. The peaking display can be achieved by tracing the subject's outline (change in brightness) during period A, and continuing the peaking display at that position when the brightness change disappears (entering the in-focus range). The peaking display may be set to occur only during position C, or it may be set to occur constantly from positions A to E. The former is desirable because when the peaking display is canceled, the user knows that the subject has gone out of focus, while the latter is desirable because it makes it easier for the user to recognize the change in blur of the subject image and the progress until the subject is in focus.

[0064] <GUIのバリエーション2> FIG. 11 shows an example of a display on the output unit 304 that allows the user to easily recognize the results of autofocus. The image captured by the event-based sensor in the figure is the display image of the output unit 304, and a score (referred to as the AF score) representing the degree of focus as a number from 1 to 100 is superimposed on the screen. The AF score may be the evaluation value calculated by the evaluation unit 302 itself, or it may be calculated using any formula as long as the score peaks at the best focus position and decreases as the distance from the best focus position increases. For example, the score may be highlighted when it exceeds a certain value (e.g., 80). Conversely, the score may be displayed only when the score is below a certain value (e.g., 40), prompting the user to perform a focusing operation. Such a superimposed display is desirable because it allows the user to easily grasp the focusing results. However, since the evaluation value to be displayed can only be calculated after the focus range has been measured, the evaluation value is superimposed during the second focusing operation, which is performed after one of the above-mentioned focusing methods has been performed once. This actually takes the time of two focussing operations, which slows down the focus speed, but the score is superimposed, making it easier for users to recognize the degree of focus.

[0065] <Focusing on multiple subjects> A case where there are multiple subjects to be focused on will be described using Fig. 12. In Fig. 12, multiple AF evaluation frames are set, with a subject close to the image capture device in AF evaluation frame 1 and a distant subject in AF evaluation frame 2. Here, the description will be given assuming that the image capture optical system 1011 of the image capture device 200 is provided with aperture control means 201 for changing the aperture.

[0066] In such cases, first use one of the focus methods described above to find the focus ranges for each of AF evaluation frame 1 and AF evaluation frame 2. If there is overlap between the focus ranges of AF evaluation frame 1 and 2, the focus lens is positioned at the center of the overlapping focus range. This makes it possible to focus on all subjects. If there is no overlap between the focus ranges of AF evaluation frame 1 and 2, use equations (1) to (3) to calculate the F-number required to make the focus ranges of AF evaluation frame 1 and 2 overlap, and change to that F-number. Then, the focus lens is moved to the center of the overlapping focus range. Control in this manner makes it possible to properly focus on multiple subjects at different distances.

[0067] As described above, by using the method of this embodiment, autofocus becomes possible in an imaging device that uses an event-based sensor, and user convenience can be improved.

[0068] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable storage medium. [Explanation of symbols]

[0069] 100 Imaging device 301 Imaging unit 1010 Imaging optical system 1012 focus lens 1011 Photoelectric conversion element 302 Evaluation Department 303 Control Unit 304 Output section 305 Operation reception section

Claims

1. a control means for controlling the position of a focus lens of an imaging device having a photoelectric conversion element; an acquisition means for acquiring an address event signal indicating a direction of change in luminance detected in each pixel of the photoelectric conversion element; an evaluation means for acquiring an evaluation value for the position of the focus lens based on the number of pixels for which the change in luminance is detected for each position of the focus lens; a determination means for determining a focus position of the focus lens based on the evaluation value, the evaluation means acquires the evaluation value based on the number of pixels whose luminance changes in a positive direction or a negative direction, based on the address event signal; The information processing apparatus is characterized in that the determining means determines the in-focus position of the focus lens based on the position of the focus lens when the evaluation value becomes equal to or smaller than a predetermined reference value.

2. 2. The information processing apparatus according to claim 1, wherein the determining means determines, as the in-focus position, a position obtained by moving the focus lens by a predetermined amount from the position at which the evaluation value becomes equal to or less than the predetermined reference value.

3. the control means measures the position of the focus lens when the evaluation value is within a specific range including the predetermined reference value; 3. The information processing apparatus according to claim 1, wherein the determining unit determines the in-focus position based on the measured position of the focus lens when the evaluation value is no longer included in the specific range.

4. 4. The information processing apparatus according to claim 1, wherein the control means changes a speed at which the focus lens is moved based on a difference between the evaluation value and the predetermined reference value.

5. 5. The information processing apparatus according to claim 1, wherein the control means controls the focus lens so as to move the focus lens from a predetermined starting point at a constant speed.

6. 6. The information processing apparatus according to claim 1, wherein the determining unit determines the focus range including the focus position further based on a depth of field of the imaging device.

7. 2. The information processing apparatus according to claim 1, wherein the determining means determines the focus position based on an optical flow generated based on the address event signal.

8. 8. The information processing apparatus according to claim 1, wherein the determining unit determines the evaluation value based on the address event signal in a partial area included in the angle of view of the imaging device.

9. 9. The information processing apparatus according to claim 8, wherein the determining means determines the evaluation value for each of a plurality of regions included in the angle of view of the imaging device.

10. the control means further controls the aperture diameter of the imaging device; The information processing device according to claim 9, characterized in that the determination means determines the aperture diameter and the position of the focus lens at which at least one or more partial areas are simultaneously in focus based on the evaluation value for each of at least one or more partial areas in the shooting range of the imaging device.

11. 11. The information processing apparatus according to claim 1, further comprising a display means for displaying an image indicating a position of a pixel where a change in luminance has occurred based on the address event signal, and the evaluation value.

12. 12. The information processing device according to claim 1, wherein the address event signal is output by a photoelectric conversion element having a pixel that outputs a signal in response to an incidence of a photon.

13. a control means for controlling the position of a focus lens of an imaging device having a photoelectric conversion element; an acquisition means for acquiring an address event signal indicating a direction of change in luminance detected in each pixel of the photoelectric conversion element; an evaluation unit that acquires an evaluation value for each position of the focus lens based on the address event signal; a determination means for determining a focus position of the focus lens based on the evaluation value, the acquiring means acquires, based on the address event signal, a positive pixel value when the direction of the change in luminance is a positive direction, and a negative pixel value when the direction of the change in luminance is a negative direction, as a pixel value of each pixel of the photoelectric conversion element; the evaluation means acquires, as the evaluation value, a time taken for a pixel value of a pixel of interest among the pixels of the photoelectric conversion element to change from positive to negative or from negative to positive; The information processing device is characterized in that the determination means determines the focus position of the focus lens based on the position of the focus lens when the pixel value of the target pixel changes from positive to negative or from negative to positive and the evaluation value.

14. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 13.

15. a control step of controlling the position of a focus lens of an imaging device including a photoelectric conversion element; an acquisition step of acquiring an address event signal indicating a direction of change in luminance detected in each pixel of the photoelectric conversion element; an evaluation step of acquiring an evaluation value for the position of the focus lens based on the number of pixels in which the change in luminance is detected for each position of the focus lens; a determining step of determining a focus position of the focus lens based on the evaluation value, In the evaluation step, the evaluation value is obtained based on the number of pixels whose luminance changes in a positive direction or a negative direction, based on the address event signal; and a step of determining the in-focus position of the focus lens based on the position of the focus lens when the evaluation value becomes equal to or less than a predetermined reference value, in the determination step.

16. The information processing method according to claim 15, characterized in that the determination step determines, as the in-focus position, a position moved a predetermined width from the position of the focus lens when the evaluation value becomes equal to or less than a predetermined reference value.

17. a control step of controlling the position of a focus lens of an imaging device including a photoelectric conversion element; an acquisition step of acquiring an address event signal indicating a direction of change in luminance detected in each pixel of the photoelectric conversion element; an evaluation step of acquiring an evaluation value for each position of the focus lens based on the address event signal; a determining step of determining a focus position of the focus lens based on the evaluation value, In the acquiring step, a positive pixel value is acquired as a pixel value of each pixel of the photoelectric conversion element when the direction of the change in luminance is a positive direction, and a negative pixel value is acquired when the direction of the change in luminance is a negative direction, based on the address event signal; In the evaluation step, a time required for a pixel value of a pixel of interest among the pixels of the photoelectric conversion element to change from positive to negative or from negative to positive is acquired as the evaluation value; The information processing device is characterized in that, in the determination step, a focus position of the focus lens is determined based on the position of the focus lens when the pixel value of the target pixel changes from positive to negative or from negative to positive and the evaluation value.

Images