Image processing apparatus and image processing method, and imaging system

Abstract

To provide an image processing device and image processing method capable of improving the accuracy of subject tracking processing. [Solution] The image processing device acquires frame images of a video and event data including the position and time at which a change in brightness was detected within the shooting range during the frame image accumulation period. The image processing device corrects the pixel values ​​of the frame images corresponding to the positions included in the event data with pixel values ​​obtained based on the pixel values ​​and the event data. Then, the image processing device performs subject tracking processing using the corrected frame images.

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus and an image processing method, as well as an imaging system, and more particularly to a subject tracking technology. [Background technology]

[0002] Conventionally, there are known technologies that use video data shot for live view display or recording to track a specific subject area. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2019-134438 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The storage period per frame of video data is limited by the frame rate, which can result in underexposure if the shooting scene is dark. Furthermore, if the subject moves during the storage period, motion blur can occur. Underexposure and motion blur can particularly reduce the accuracy of subject tracking processing, which is based on the degree of agreement or correlation between frame images.

[0005] In one embodiment, the present invention provides an image processing apparatus and an image processing method capable of improving the accuracy of subject tracking processing. [Means for solving the problem]

[0006] In one aspect of the present invention, there is provided an image processing apparatus including: acquisition means for acquiring a frame image of a moving image and event data including a position and time at which a luminance change is detected in a shooting range during an accumulation period of the frame image; correction means for correcting a pixel value of the frame image corresponding to the position included in the event data with a pixel value obtained based on the pixel value and the event data; and tracking means for performing a subject tracking process using the frame image corrected by the correction means.

Effects of the Invention

[0007] According to one aspect of the present invention, it is possible to provide an image processing apparatus and an image processing method capable of improving the accuracy of subject tracking processing.

Brief Description of the Drawings

[0008] [Figure 1] Block diagram showing a functional configuration example of an imaging system according to an embodiment [Figure 2] Diagram for explaining the operation of EVS11 [Figure 3] Diagram for explaining a framed event image in an embodiment [Figure 4] Diagram for explaining a framed event image in an embodiment [Figure 5] Diagram for explaining a frame image and event data related to a shooting scene including a moving subject [Figure 6] Diagram for explaining the generation operation of a tracking image by the data processing unit 15 in an embodiment [Figure 7] Schematic diagram of a tracking image obtained by correcting a frame image [Figure 8] Diagram for explaining the alignment process of the data processing unit 15 in the third embodiment [Figure 9] Diagram for explaining the alignment process of the data processing unit 15 in the third embodiment

Embodiments of the Invention

[0009] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0010] In the following description, the present invention will be implemented in an imaging system as an example of an image processing device. However, imaging functionality is not essential to the present invention, and it can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include imaging devices, computer equipment (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smartwatches, game consoles, robots, drones, and dashcams. These are examples, and the present invention can be implemented in other electronic devices as well.

[0011] Figure 1 is a block diagram showing an example of the functional configuration of an imaging system 10 according to an embodiment of the present invention. Note that among the configurations represented as functional blocks in Figure 1, those described as control units and processing units can be realized by hardware circuits, or by a combination of memory and a processor that executes a program stored in memory. Furthermore, one block may be realized by multiple integrated circuit packages, or multiple blocks may be realized by one integrated circuit package. Also, the same block may be implemented in different configurations depending on the operating environment and required capabilities.

[0012] The imaging system 10 includes an event-driven vision sensor or event-based vision sensor (EVS) 11 and a frame-type sensor 12. The event-driven vision sensor or event-based vision sensor (EVS) 11 detects brightness changes within the imaging range and outputs event data (also simply called an event) that includes information about the type of event detected, the location and time of the brightness change detection. There is no regularity in the timing of brightness change detection or the timing of event data output; both are irregular. Therefore, the concept of frame rate does not exist in the output of the EVS 11. In addition, the event data may include the following information. • Magnitude of brightness change: A numerical representation of the amount of brightness change. • Confidence Score: A score indicating the accuracy and confidence of event detection. • Sensor ID: An ID used to identify which sensor the data comes from.

[0013] On the other hand, the frame-type sensor 12 is a general single-chip color image sensor, such as a CMOS image sensor with a primary color Bayer array color filter, and outputs still image or video data on a frame-by-frame basis. An image in one frame is composed of signals read from each pixel in the pixel array of the image sensor, each having a value corresponding to the amount of light received during the exposure period. Here, it is assumed that the frame-type sensor 12 outputs video data (frame video data) having a specific frame rate.

[0014] Figure 2 is a schematic diagram illustrating the operation of EVS11. Figure 2 shows an example of the change in brightness over time for one pixel among multiple pixels arranged in two dimensions in the pixel array of EVS11, along with the output of the corresponding event data.

[0015] Vertical axis V P V is the output voltage of the pixel and has a value corresponding to the logarithm of the intensity of the incident light. EVS11 is V P An event is detected when the change exceeds the threshold Θ. First, the output voltage V at time t0 P With (t0) as the reference voltage, V P (t1)>VP Since it becomes (t0)+Θ, EVS11 detects an event in which the increase in the output voltage exceeds the threshold value Θ. Then, EVS11 outputs the corresponding event data and sets V P (t1) as the new reference voltage. After that, V P (t2)>V P (t1)+Θ, so EVS11 detects the event and outputs the corresponding event data. Also, EVS11 sets V P (t2) as the new reference voltage.

[0016] After that, EVS11 similarly outputs event data in response to the detection of the event. At time t6, since V P (t6)<V P (t5)-Θ, EVS11 detects the - event and outputs the corresponding event data. Thereafter, EVS11 repeats the output of event data and the update of the reference voltage every time the change in the output voltage exceeds the threshold value Θ.

[0017] In this way, EVS11 outputs event data for each pixel and at independent timings. The time included in the event data may be the time based on the internal clock (event detection device time) of EVS11, or the time based on the clock of the imaging system 10. Also, the event data may include other information such as information indicating the direction (±) of the luminance change or the actual change amount in addition to the position (e.g., pixel coordinates) and time at which the event was detected.

[0018] In this specification, outputting event data at independent timings for each pixel may be described as outputting event data asynchronously. On the other hand, the frame-type sensor 12 outputs image data (frame data) for one frame in synchronization with a vertical synchronization signal corresponding to a set fixed frame rate. Therefore, the frame video data output by the frame-type sensor 12 consists of frame data output at a period corresponding to the frame rate. The frame data includes imaging information (one or more of the following: storage period, storage start and end times, aperture value, AF frame position, focus distance, etc.) as supplementary information.

[0019] Returning to Figure 1, the imaging optical system 20 forms an optical image of the subject on the imaging surfaces of the EVS 11 and the frame-type sensor 12. Here, the optical image generated by one optical system is distributed to the respective imaging surfaces of the EVS 11 and the frame-type sensor 12 by a half-mirror or the like. However, the EVS 11 and the frame-type sensor 12 may each have their own independent imaging optical systems.

[0020] The imaging optical system 20 has a focus lens. The system control unit 13 controls the focus distance of the imaging optical system 20 by driving the focus lens based on known evaluation values.

[0021] The system control unit 13 is, for example, a program-executable processor (CPU, MPU, microprocessor, etc.). The system control unit 13 controls the operation of each part of the imaging system 10 and realizes the functions of the imaging system 10 by reading the program stored in the ROM 18 into the RAM 17 and executing it.

[0022] ROM18 is a rewritable non-volatile memory that stores programs executed by the system control unit 13, various settings for the imaging system 10, GUI data, and the like. RAM 17 is used as the main memory of the system control unit 13, a buffer for temporarily storing frame video data and event data, and work memory for temporarily storing data being processed by the data processing unit 15. In addition, a portion of RAM 17 may be used as video memory for storing image data to be displayed on a display unit (not shown).

[0023] The operation unit 19 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to the imaging system 10. Operations on the operation unit 19 are monitored by the system control unit 13. When the system control unit 13 detects an operation on the operation unit 19, it executes an action corresponding to the detected operation. The input devices that make up the operation unit 19 have names according to the function they are assigned. For example, the operation unit 19 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and an OK key.

[0024] The release switch is a switch for recording still images, and the system control unit 13 recognizes a half-pressed state of the release switch as a shooting preparation instruction and a fully pressed state as a shooting start instruction. The system control unit 13 also recognizes the video recording switch as a video recording start instruction when pressed in shooting standby mode, and as a recording stop instruction when pressed during video recording. The functions assigned to the same input device may be variable. The input device may also be a software button or key using a touch display. Furthermore, the operation unit 19 may include input devices that support non-contact input methods such as voice input or eye-tracking input.

[0025] The system control unit 13 sets the detection conditions calculated by the EVS control unit 14 to the EVS 11. The detection conditions include the threshold value Θ mentioned above. The system control unit 13 also sets imaging conditions for the frame-type sensor 12. The imaging conditions include the storage period (shutter speed), the shooting sensitivity, and so on.

[0026] The EVS11 and frame sensor 12 operate using a common timing signal or time information. The timing signal or time information may be supplied by the system control unit 13 or by other components. For example, either the EVS11 or the frame sensor 12 may supply it to the other. By operating the EVS11 and frame sensor 12 using a common timing signal or time information, the data processing unit 15 can process event data and frame video data corresponding to the same period.

[0027] The EVS control unit 14 calculates the conditions under which the EVS 11 detects an event based on the information obtained from the data processing unit 15. The EVS control unit 14 then notifies the system control unit 13 of the calculated conditions.

[0028] The data processing unit 15 receives event signals from the EVS 11 and frame video data (moving video data) from the frame sensor 12. The data processing unit 15 continuously generates image data suitable for subject tracking processing using the received event signals and frame video data. The data processing unit 15 sequentially supplies the generated image data to the tracking control unit 16. The data processing unit also receives subject tracking results from the tracking control unit 16.

[0029] The data processing unit 15 can generate image data for one frame (hereinafter referred to as a framed event image) from the event signal, which is suitable for subject tracking processing. The generation of a framed event image is not mandatory.

[0030] Figure 3 shows an example of a frame image 30 of frame video data output by the frame-type sensor 12 and a framed event image 31 that can be generated by the data processing unit 15 based on the event signal output by the EVS 11.

[0031] The frame image 30 is composed of a group of signals read from each pixel included in the readout area of ​​the frame sensor 12. In other words, the frame sensor 12 outputs the value of each pixel regardless of whether the pixel value has changed or not.

[0032] On the other hand, as described above, the EVS11 outputs event data only for pixels that detect a brightness change above a threshold. Therefore, the data processing unit 15 generates a framed event image for one frame using the event data received from the EVS11 during the same period as the accumulation period of the frame-type sensor 12.

[0033] Specifically, the data processing unit 15 processes the event data received from the EVS 11 during the storage period in the frame sensor 12, from the start to the end of the storage period. + The detection location of the event is set to a white pixel (255 if the brightness value is 8 bits). - The location where the event was detected is a black pixel (0). The location where no event was detected is shown in gray pixels (127). This generates a framed event image.

[0034] Assuming that the person in the frame image 30 is moving to the left, the EVS 11 detects events in the pixels surrounding the outline of the person, but not in the background. Therefore, the framed event image 31 will have the outline of the person in white or black, and the rest of the image in gray. Note that the pixel values ​​corresponding to the type of detected event and the pixel values ​​at positions where no events are detected are merely examples and may be set to other values. Also, the values ​​may be changed depending on the number of times an event has been detected at the same position.

[0035] Because the time resolution of event data from EVS11 is high, ranging from a few nanoseconds to a few microseconds, subject blur does not occur in framed event images for subjects moving at typical speeds. Furthermore, since the number of effective pixels (pixels in which events are detected) in framed event images is small, it has the advantage of efficiently detecting and tracking areas that change over time within the shooting range.

[0036] Figure 4 shows the frame image 40 and the corresponding framed event image 41 in the case of a dark shooting scene, similar to Figure 3. Since the upper limit of the accumulation period of the frame-type sensor 12 depends on the frame rate, in dark scenes the frame image 40 may be underexposed, as shown in Figure 4. In this case, it may be difficult to detect the subject from the frame image 40.

[0037] On the other hand, the pixels of the EVS11 logarithmically convert the brightness of the incident light into voltage, resulting in high sensitivity in low-brightness areas and low sensitivity in high-brightness areas. Therefore, even when the shooting scene is dark, the outline and direction of movement of moving subjects can be easily recognized in the framed event image 41.

[0038] The data processing unit 15 utilizes the characteristics of the event data output by the EVS 11 and corrects the frame image output by the frame sensor 12 to generate an image suitable for tracking. The details of how the data processing unit 15 generates the tracking image will be explained below with reference to Figures 5 and 6.

[0039] Figure 5(a) is a schematic diagram of a shooting scene in which a human subject moves from left to right. Both the EVS11 and the frame-type sensor 12 are assumed to capture the shooting range 50.

[0040] Specifically, Figure 5(a) shows that a person subject, who was at position 51 at time T0, moved to position 52 at time T1 (T1 = T0 + T). Hereinafter, T is the length of the accumulation period of the frame-type sensor 12. Furthermore, the shooting range 50 does not change between time T0 and T1, and the area of ​​the person subject is assumed to have higher brightness than other areas (background area).

[0041] Figure 5(b) shows the types and timing of event signals output by EVS11 between time T0 and T1 for points A to E in Figure 5(a).

[0042] At point A, a human subject is present at time T0, but not at time T1. Therefore, as the human subject moves away from point A, the brightness decreases, and multiple negative events are detected. Multiple detections occur because the time resolution of the event detection is fast relative to the subject's movement speed.

[0043] Similar to point A, point B also has a human subject at time T0, but the human subject is no longer present at time T1. However, because the timing of the human subject disappearing is later than at point A, the detection timing of the negative event is also delayed compared to point A.

[0044] At point C, no event was detected because a human subject remained present from time T0 to T1.

[0045] At point D, there is no human subject at time T0, but there is a human subject at time T1. Therefore, when the human subject moves to point D, the brightness increases, and multiple + events are detected.

[0046] Similar to point D, point E does not have a human subject at time T0, but it does have one at time T1. However, because the timing of the human subject's movement is later than at point D, the detection timing of the + event is also delayed.

[0047] In Figure 5, the event detection timings at points A and D, and points B and E are almost the same. This has no particular technical significance and is due to the relationship between the spacing between the points and the size of the subject.

[0048] Figure 5(c) schematically represents a frame image obtained by capturing the scene in Figure 5(a) with the frame-type sensor 12. In the frame image, the image 53 of the person subject moving during the accumulation period (from time T0 to T1) is blurred.

[0049] As mentioned above, among points A to E, the central point C has a human subject present throughout the entire accumulation period, so the brightness of image 53 is highest near point C. The brightness of image 53 decreases as you move away from point C to the left or right.

[0050] Using images with blurred subject areas, as shown in Figure 5(c), in subject tracking processes that utilize the degree of similarity or correlation between frame images, such as template matching, can reduce the accuracy of subject tracking.

[0051] Figure 5(d) schematically shows a framed event image generated by the data processing unit 15 using event data output by EVS 11 from time T0 to T1 for the scene shown in Figure 5(a).

[0052] As described above, the data processing unit 15 generates a framed event image in which the locations (pixels) where negative events are detected are black pixels, the locations where positive events are detected are white pixels, and the locations where no events are detected are gray pixels. In the framed event image, the black pixel region 54 is the region where a person subject exists at time T0 but does not exist at time T1. The white pixel region 55 is the region where a person subject does not exist at time T0 but exists at time T1. The background region and the region 56 where a person subject existed between time T0 and T1 are gray pixels because no events were detected in those areas.

[0053] Next, we will explain a method for generating tracking images with improved frame image blur based on event data obtained during the storage period.

[0054] Figure 6(a) shows the brightness change of point A in the scene shown in Figure 5(a), and Figure 6(b) shows the brightness change of point E in the scene shown in Figure 5(a). In Figure 6, the brightness change is shown as the change in output voltage 61 and 62 of the pixel corresponding to point A in EVS11. The relationship between the amount of incident light and the output voltage is expressed as a logarithmic function in which the sensitivity is high in the region of low incident light and low in the region of high incident light. This is an input / output characteristic similar to the so-called gamma curve with γ < 1.0.

[0055] Furthermore, V1 to V6 have intervals corresponding to the event detection threshold Θ(%). In Figure 6(a), V n+1 =V n The relationship ×0.9 is shown in Figure 6(b) as V n-1 =V n Let's assume a relationship of ×1.1. Here, the threshold Θ is set to a 10% change, but other values ​​may also be used.

[0056] Furthermore, the accumulation period (the period from time T0 to T1) is divided into intervals t1 to t6 based on the time the event was detected. a and V b This is the output voltage at time T1 at points A and B.

[0057] On the other hand, in the frame image output by the frame-type sensor 12, the pixel values ​​corresponding to points A and E correspond to the integral value of the incident light amount during the period from time T0 to T1. In other words, the pixel value corresponding to point A roughly corresponds to the area of ​​the shaded region in Figure 6(a), and the pixel value corresponding to point E roughly corresponds to the area of ​​the shaded region in Figure 6(b). It is not possible to know the change in the incident light amount during the storage period from the pixel values ​​of the frame image.

[0058] The data processing unit 15 generates a subject tracking image with suppressed blur by correcting the pixel values ​​of the frame image at the positions where events were detected by the EVS 11 during the storage period. Specifically, the data processing unit 15 replaces the pixel values ​​at the positions where events were detected with the pixel values ​​that would occur if the output voltage at time T1, when the storage period ends, continued throughout the storage period (times T0 to T1).

[0059] The pixel values ​​used for replacement are not limited to the output voltage at time T1, but can also be calculated using the output voltage at any time during the storage period (times T0 to T1). The output voltage at any time within the storage period is, for example, the output voltage V when the most recent event was detected. n It can be done this way.

[0060] Here, we assume that the pixel values ​​that make up the frame image take 8-bit values ​​(0 to 255), and that the pixel values ​​corresponding to points A and E are both 120.

[0061] As described above, the pixel value of point A in the frame image, 120, corresponds to the area of ​​the shaded region in Figure 6(a). Therefore, the output voltage V at time T1 is a Close to V a Pixel values ​​corresponding to larger V6 values ​​can be calculated from the following equations (1) and (2). t1V1+t2V2+t3V3+t4V4+t5V5+t6V6=120 (1) V n+1 =V n ×0.9 (n=1~5) (2)

[0062] Equation (1) shows that the sum of the cumulative values ​​of the length of each interval obtained by dividing the accumulation period at the time an event was detected, and the detection threshold for brightness change corresponding to that interval, corresponds to the pixel value at the corresponding position in the frame image. Equation (2) shows the relationship between the detection thresholds for brightness change. For example, if the ratios of t1 to t6 are 2:1:1:1:1:4 respectively, then from equations (1) and (2), the pixel value corresponding to V6 is approximately 97.

[0063] Similarly, for point E, the pixel value 120 corresponds to the area of ​​the shaded region in Figure 6(b). Therefore, the output voltage V at time T1 is b Close to V bThe pixel value corresponding to a larger V1 can be determined from the following equations (3) and (4). In this embodiment, V1 to V6 is V a V is a larger value a This was an approximation. However, V a Approximate with a smaller value, V a You can also approximate it with the value that has the smallest difference from V. b The same applies to this matter.

[0064] t1V6+t2V5+t3V4+t4V3+t5V2+t6V1=120 (3) V n =V n-1 / 1.1 (n=2~6) (4) For example, if the ratios of t1 to t6 are 4:1:1:1:1:2, then V1 can be calculated as approximately 157. Alternatively, instead of V1, the pixel value corresponding to V2, where the most recent event was detected, can be found.

[0065] By replacing the pixel values ​​of points A and E with the pixel values ​​obtained in this way, point A will have a brightness value close to the background (or a value with more emphasis on the background component), and point E will have a brightness value close to point C (or a value with more emphasis on the subject component). Similarly, by replacing the pixel values ​​at other locations where events were detected during the accumulation period, the frame image is corrected to match the output voltage at time T1 if it had continued for the accumulation period.

[0066] Figure 7 is a schematic diagram of the corrected frame image obtained by correcting the frame image shown in Figure 5(c) based on event data detected during the storage period. The contour of the subject area in the corrected frame image is closer to the contour of the subject at time T1, so the blurring of the subject area is suppressed compared to the frame image before correction. The data processing unit 15 generates the corrected frame image obtained in this way as an image for subject tracking and outputs it to the tracking control unit 16.

[0067] The data processing unit 15 continuously generates tracking images from the frame video data obtained during the period in which the tracking control unit 16 performs subject tracking processing, and supplies them to the tracking control unit 16. The generation of tracking images may be performed for all frames of the frame video, or for frame images at a specific period.

[0068] The tracking control unit 16 uses the tracking image provided by the data processing unit 15 to track the subject area using a method based on the correlation between frame images, such as template matching. It then outputs the results of the tracking process (position, size, and detection confidence of the subject area after movement, etc.) to the system control unit 13.

[0069] The system control unit 13 can use the results of the tracking process for purposes such as setting the focus detection area or automatic exposure control. Since any known method can be used for subject tracking processing and the use of tracking results in the tracking control unit 16, further details will be omitted.

[0070] If the number of pixels in the EVS 11 and the frame-type sensor 12 are different, the data processing unit 15 will process the data according to the lower pixel count. For example, if the frame image has a large number of pixels, the data processing unit 15 will generate a reduced image from the frame image that matches the pixel count of the EVS 11 and use it for the above processing. The reduced image can be generated using any known method.

[0071] As described above, according to this embodiment, by correcting the frame image based on event data detected by EVS during the accumulation period when the frame image is captured, blur of moving subjects in the frame image can be suppressed. By using an image with suppressed blur of moving subjects in the tracking process, it becomes possible to improve the accuracy of the tracking process, especially when the shooting scene is dark or when the moving subject is moving at a high speed.

[0072] ●<Second Embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment, the pixel values ​​of all pixels in the frame image corresponding to all positions where an event was detected during the storage period were corrected. However, in order to reduce the load required to calculate the pixel values ​​used for correction, only pixels corresponding to positions where the event data satisfies predetermined conditions may be subject to correction.

[0073] For example, only pixels corresponding to locations where events were detected more than a threshold number of times during the accumulation period can be targeted for correction. This eliminates the need to calculate correction values ​​for locations where events were detected only a very small number of times in areas other than moving subjects, thereby reducing the load required for correction (generation of tracking images) processing.

[0074] Furthermore, the conditions for correction may be dynamically changed depending on the current processing load and the frame rate and number of pixels (resolution) of the video output by the frame-type sensor 12. For example, if the frame rate or resolution exceeds a threshold, the threshold for the number of event detections can be increased to reduce the number of pixels to be corrected compared to when the threshold is not exceeded.

[0075] According to this embodiment, it is possible to reduce the processing load while improving the accuracy of the tracking process.

[0076] ●<Third Embodiment> Next, a third embodiment of the present invention will be described. In the first embodiment, the EVS 11 and the frame-type sensor 12 used a common imaging optical system 20 to match the shooting range of the EVS 11 and the frame-type sensor 12.

[0077] However, if the EVS 11 and the frame-type sensor 12 each have separate imaging optical systems, their shooting ranges will differ. In this case, the data processing unit 15 aligns the event detection position with the position within the frame image before generating the tracking image.

[0078] Specifically, the data processing unit 15 calculates the difference image of two consecutive frames from the frame video data output by the frame-type sensor 12.

[0079] Figure 9(a) shows the nth frame image of the frame video data, and Figure 9(b) shows the (n+1)th frame image output after the nth frame image. The moving subject continues to move to the right, and the position of the subject region 92 in the (n+1)th frame image is to the right of the position of the subject region 91 in the nth frame image.

[0080] In Figure 9(a), an image is recorded showing blurring of the subject area 91 from point A to point E, whereas in Figure 9(b), an image is recorded showing blurring of the subject area 92 from point C to point E, and even beyond point E, in the same way as the subject area 91.

[0081] The data processing unit 15 generates a framed event image using event data received from the EVS 11 from the start of the accumulation period for the nth frame image to the end of the accumulation period for the (n+1)th frame image. Except for the difference in the reception period of the event data used, the data processing unit 15 can generate a framed event image in the same manner as in the first embodiment.

[0082] Figure 8(a) schematically shows the position 81 of the human subject at the start time T0 of the accumulation period for the nth frame image, and the position 82 of the human subject at the end time T1 of the accumulation period for the (n+1)th frame image. Here, it is assumed that the shooting range does not move. Note that the meaning of time T1 is different from that of the first embodiment.

[0083] Figure 8(b) shows examples of pixel values ​​for points A through G in the (n+1)th frame image, the nth frame image, and the difference image obtained by subtracting the nth frame image from the (n+1)th frame image.

[0084] The dotted line 83 represents the pixel values ​​on the line from point A to point G in the (n+1)th frame image. The value (absolute value) is large because the brightness value at the position of the person's head is high, and the (n+1)th frame image is a "subtracted image," so it has a positive value. The dashed line 84 represents the pixel values ​​on the line from point A to point G in the nth frame image. The value (absolute value) is large because the brightness value at the head of the person is high, and the nth frame image is a "subtracted image," so it has a negative value.

[0085] The solid line 85 represents the pixel values ​​on the line from point A to point G in the difference image obtained by subtracting the nth frame image from the (n+1)th frame image. The sign of the pixel values ​​reverses around point D. However, since pixel values ​​do not inherently take negative values, negative pixel values ​​are corrected so that they become pixel values ​​of 0 or greater.

[0086] Figure 8(c) shows the cumulative number of events output from EVS11 from the start time T0 of the accumulation period for the nth frame image to the end time T1 of the accumulation period for the (n+1)th frame image. The cumulative number of events is the cumulative value obtained by adding 1 for a positive event and subtracting 1 for a negative event. If a positive event and a negative event are detected at the same location, the cumulative number of events is the difference in the number of detections.

[0087] Figure 8(c) shows the cumulative number of events detected while the human subject moves from position 81 to position 82. Therefore, it is considered that approximately the same number of negative events occurred at points A to C, where the human subject changes from being present to not being present. On the other hand, approximately the same number of positive events occurred at points D to G, where the human subject changes from not being present to being present. Therefore, the sign of the cumulative number of events reverses at point D.

[0088] From the shapes of pixel value 85 shown in Figure 8(b) and cumulative event count 86 shown in Figure 8(c), it can be seen that the pixel values ​​of the difference image and the cumulative event count become nearly similar in shape when trinarized (1, 0, -1) with a threshold of 0. Note that the threshold used for trinarization is not limited to 0.

[0089] The data processing unit 15 generates a trinarized difference image by trinarizing the pixel values ​​of the difference image. The data processing unit 15 also trinarizes the distribution of the number of accumulated events. Then, the data processing unit 15 searches for the relative position (shift amount) between the two trinarized images with the highest degree of agreement. This search can be performed using, for example, pattern matching.

[0090] Once the amount of displacement has been determined, the data processing unit 15 corrects the position included in the event data received from, for example, the EVS 11 using the amount of displacement. After the position correction of the event data is completed, a tracking image can be generated in the same manner as in the first or second embodiment. Note that, since the amount of displacement of the shooting range is basically fixed, if the amount of displacement has been determined in the past, it is not necessary to determine the amount of displacement again.

[0091] (Other embodiments) The above-described embodiment can also be implemented using recorded frame video data and event data. Therefore, the above-described embodiment can be implemented in any electronic device that can acquire recorded frame video data and event data and perform processing similar to that of the data processing unit 15.

[0092] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0093] This embodiment includes the following image processing apparatus, image processing method, and imaging system. (Item 1) An acquisition means for acquiring video frame images and event data including the position and time at which a change in brightness was detected within the shooting range during the accumulation period of the frame images, Correction means for correcting the pixel value of the frame image corresponding to the position included in the event data with a pixel value obtained based on the pixel value and the event data, An image processing apparatus characterized by having a tracking means that performs subject tracking processing using the frame image corrected by the correction means. (Item 2) The image processing apparatus according to item 1, characterized in that the event data is obtained by an event-based vision sensor. (Item 3) The image processing apparatus according to item 1 or 2, characterized in that the event data is generated each time the amount of change in brightness exceeds a threshold. (Item 4) The image processing apparatus according to any one of items 1 to 3, characterized in that the correction means corrects the frame image by replacing the pixel values ​​of the frame image with pixel values ​​obtained based on the pixel values ​​and the event data. (Item 5) The correction means is The accumulation period is divided into sections at the time when the brightness change is detected at the same position. For each section, the sum of the cumulative values ​​of the length of the section and the detection threshold for the brightness change corresponding to the section corresponds to the pixel value corresponding to the same position in the frame image. The pixel value corresponding to one of the detection thresholds is then calculated as a correction value. The image processing apparatus according to any one of items 1 to 3, characterized in that it corrects the frame image by replacing the pixel value corresponding to the same position in the frame image with the correction value. (Item 6) The image processing apparatus according to any one of items 1 to 5, characterized in that the tracking means tracks a subject area based on the degree of agreement or correlation between frame images. (Item 7) The image processing apparatus according to any one of items 1 to 6, characterized in that the correction means does not correct the pixel values ​​of the frame image corresponding to positions where the brightness change has not been detected more than a threshold number of times during the storage period. (Item 8) The image processing apparatus according to any one of items 1 to 7, wherein the correction means is characterized in that, if the positions of the event data and the frame image do not match, it performs alignment and then corrects the frame image. (Item 9) An image processing device described in any one of items 1 to 8, The image sensor that generates the frame image, An event-based vision sensor that generates the aforementioned event data, An imaging system characterized by having the following features. (Item 10) An image processing method performed by an image processing device, To obtain frame images from a video, The process involves acquiring event data that includes the location and time at which a change in brightness was detected within the shooting range during the accumulation period of the frame image, The pixel values ​​of the frame image corresponding to the positions included in the event data are corrected with pixel values ​​obtained based on the said pixel values ​​and the event data. An image processing method characterized by performing subject tracking processing using the corrected frame image. (Item 11) A program for causing a computer to function as one of the means of an image processing device described in any one of items 1 through 8.

[0094] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0095] 10…Imaging system, 11…EVS, 12…Frame-type sensor, 13…System control unit, 14…EVS control unit, 15…Data processing unit, 16…Tracking control unit

Claims

1. An acquisition means for acquiring video frame images and event data including the position and time at which a change in brightness was detected within the shooting range during the accumulation period of the frame images, Correction means for correcting the pixel value of the frame image corresponding to the position included in the event data with a pixel value obtained based on the pixel value and the event data, An image processing apparatus characterized by having a tracking means that performs subject tracking processing using the frame image corrected by the correction means.

2. The image processing apparatus according to claim 1, characterized in that the event data is obtained by an event-based vision sensor.

3. The image processing apparatus according to claim 1, characterized in that the event data is generated each time the amount of change in brightness exceeds a threshold.

4. The image processing apparatus according to claim 1, characterized in that the correction means corrects the frame image by replacing the pixel values ​​of the frame image with pixel values ​​obtained based on the pixel values ​​and the event data.

5. The correction means is The accumulation period is divided into sections at the time when the brightness change is detected at the same position. For each section, the sum of the cumulative values ​​of the length of the section and the detection threshold for the brightness change corresponding to the section corresponds to the pixel value corresponding to the same position in the frame image. The pixel value corresponding to one of the detection thresholds is then calculated as a correction value. The image processing apparatus according to claim 1, characterized in that it corrects the frame image by replacing the pixel value corresponding to the same position in the frame image with the correction value.

6. The image processing apparatus according to claim 1, characterized in that the tracking means tracks a subject area based on the degree of agreement or correlation between frame images.

7. The image processing apparatus according to claim 1, characterized in that the correction means does not correct the pixel values ​​of the frame image corresponding to positions where the brightness change has not been detected more than a threshold number of times during the storage period.

8. The image processing apparatus according to claim 1, wherein the correction means corrects the frame image after aligning the event data if the positions of the event data and the frame image do not match.

9. An image processing apparatus according to any one of claims 1 to 8, The image sensor that generates the frame image, An event-based vision sensor that generates the aforementioned event data, An imaging system characterized by having the following features.

10. An image processing method performed by an image processing device, To obtain frame images from a video, The process involves acquiring event data that includes the location and time at which a change in brightness was detected within the shooting range during the accumulation period of the frame image, The pixel values ​​of the frame image corresponding to the positions included in the event data are corrected with pixel values ​​obtained based on the said pixel values ​​and the event data. An image processing method characterized by performing subject tracking processing using the corrected frame image.

11. A program for causing a computer to function as each of the means of the image processing apparatus described in any one of claims 1 to 8.

Images