Image processing device and image processing method

Abstract

An image processing device is provided, comprising: a region recognition unit which, based on an event signal generated according to a change in light intensity in one or more pixels of an image captured by synchronous scanning, recognizes a first region in which an object in an image undergoes a first motion and a second region in which the object undergoes a second motion different from the first motion, or the object does not substantially move; and an image processing unit which performs image processing on the first region and the second region in the image.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and a program. Background Art

[0002] An event-driven visual sensor is known in which pixels detect intensity changes of incident light to generate signals asynchronously in time. Compared with a frame-based visual sensor that scans all pixels in a predetermined cycle, especially compared with an image sensor such as a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor), this event-driven visual sensor has an advantage in that it can operate at low power consumption and high speed. For example, technologies related to this event-driven visual sensor are listed in PTL 1 and PTL 2.

[0003] Patent Literature

[0004] [PTL 1]

[0005] JP 2014-535098T

[0006] [PTL2]

[0007] JP 2018-085725A Summary of the invention

[0008] [Technical issues]

[0009] However, regarding the event-driven vision sensor, although the above-mentioned advantages are known, it is difficult to say that a method of using the event-driven vision sensor in combination with another device (eg, a frame-based vision sensor) has been sufficiently proposed.

[0010] In view of the above, an object of the present invention is to provide an image processing device, an image processing method, and a program capable of obtaining advantageous effects by using an event-driven visual sensor and a frame-based visual sensor in combination.

[0011] [Solution to the problem]

[0012] According to one aspect of the present invention, there is provided an image processing device, comprising: a region identification part, which identifies, in an image, a first region in which an object undergoes a first motion and a second region in which the object undergoes a second motion different from the first motion, or the object does not substantially move, based on an event signal generated in response to a change in light intensity in one or more pixels of an image captured by synchronous scanning; and an image processing part, which performs different image processing between the first region and the second region in the image.

[0013] According to another aspect of the present invention, there is provided an image processing method, comprising: a step of identifying, in an image, a first region where an object undergoes a first motion and a second region where the object undergoes a second motion different from the first motion, or where the object does not substantially move, based on an event signal generated in response to a change in light intensity in one or more pixels of an image captured by synchronous scanning; and a step of performing different image processing between the first region and the second region in the image.

[0014] According to another aspect of the present invention, a program is provided that enables a computer to implement: a function of identifying a first region in an image where an object undergoes a first motion and a second region in which the object undergoes a second motion different from the first motion, or where the object does not substantially move, based on an event signal generated in response to a change in light intensity in one or more pixels of an image captured by synchronous scanning; and a function of performing different image processing between the first region and the second region in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram showing a schematic configuration of a system according to an embodiment of the present invention.

[0016] Figure 2 is a diagram schematically explaining calibration between a camera and a sensor according to an embodiment of the present invention.

[0017] Figure 3 is a diagram for explaining matching between an image and an event signal according to an embodiment of the present invention.

[0018] Figure 4 is a diagram for explaining an example of region recognition and image processing according to an embodiment of the present invention.

[0019] Figure 5 is a diagram for explaining an example of region recognition and image processing according to an embodiment of the present invention.

[0020] Figure 6 is a flowchart indicating an example of an image processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Hereinafter, some preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that in this specification and the accompanying drawings, the same reference numerals are given to constituent elements having substantially the same functional configuration, and description will not be repeated.

[0022] Figure 1 1 is a block diagram showing a schematic configuration of a system according to an embodiment of the present invention. Figure 1As shown, the system 10 includes an RGB (Red-Green-Blue) camera 100 , an EDS (Event Driven Sensor) 200 , and an image processing device 300 .

[0023] The RGB camera 100 is an example of a first visual sensor that captures images by synchronous scanning, and includes an image sensor 110, and a processing circuit 120 connected to the image sensor 110. For example, depending on a user operation, the image sensor 110 captures an RGB image 101 by synchronously scanning all pixels at a predetermined cycle or predetermined timing. In the present embodiment, the RGB image 101 includes a plurality of frames of images captured sequentially at a predetermined frame rate, and the plurality of frames of images constitute a motion image. For example, the processing circuit 120 converts the RGB image 101 into a format suitable for storage and transmission. In addition, the processing circuit 120 provides a timestamp 102 to the RGB image 101.

[0024] EDS200 is an example of a second visual sensor that generates an event signal when the sensor detects a change in light intensity, and includes sensors 210 constituting a sensor array and a processing circuit 220 connected to the sensors 210. Each sensor 210 includes a light receiving element and generates an event signal 201 when a change in the intensity of light to be incident is detected, more specifically, when a change in brightness is detected. Since the sensor 210 that does not detect a change in the intensity of light to be incident does not generate the event signal 201, the event signal 201 is generated asynchronously in time in EDS200. The event signal 201 output by the processing circuit 220 includes identification information about each sensor 210 (e.g., the position of a pixel), the polarity of the brightness change (higher or lower), and a timestamp 202.

[0025] Here, in the present embodiment, the time stamp 102 provided to the RGB image 101 and the time stamp 202 provided to the event signal 201 are synchronized with each other. More specifically, for example, by providing time information for generating the time stamp 202 in the EDS 200 to the RGB camera 100, the time stamp 102 can be synchronized with the time stamp 202. Alternatively, in a case where the time information for generating the time stamps 102 and 202 are independent of each other in the RGB camera 100 and the EDS 200, respectively, the time when a specific event (e.g., a change in an object in the entire image) has occurred is set as a reference to calculate an offset of the time stamp, so that the time stamp 102 and the time stamp 202 can be synchronized with each other from then on.

[0026] Furthermore, in the present embodiment, the sensor 210 of the EDS 200 corresponds to one or more pixels of the RGB image 101 through the calibration process of the RGB camera 100 and the EDS 200 performed in advance, and the event signal 201 is generated in response to the change of the light intensity in the one or more pixels. More specifically, as Figure 2As shown, for example, the RGB camera 100 and the EDS 200 capture a common calibration pattern 21 with each other (in the case of the EDS 200, for example, the calibration pattern can be captured by flashing the entire area of ​​the calibration pattern 21 using the light source 22), and the corresponding parameters between the camera and the sensor are calculated by the internal parameters and external parameters of the RGB camera 100 and the EDS 200 respectively, so that the sensor 210 can correspond to one or more pixels of the RGB image 101.

[0027] Figure 3 1 is a diagram for explaining the matching between an image and an event signal according to an embodiment of the present invention. In the example shown in the figure, an RGB image 101 captured by an RGB camera 100 and an image in which an event signal 201 output from an EDS 200 at a time corresponding to a scan of the RGB image 101 is arranged at a position of a corresponding pixel are presented. Figure 2 The corresponding parameters between the camera and the sensor are calculated in advance, so the coordinate system of the event signal 201 is converted into Figure 3 The coordinate system of the RGB image 101 shown enables the event signal 201 to be superimposed on the RGB image 101 .

[0028] Refer again Figure 1 The image processing device 300 is implemented by, for example, a computer including a communication interface, a processor and a memory, and includes the functions of a region identification part 310 and an image processing part 320, which are implemented by the processor operating according to a program stored in the memory or received through the communication interface. The functions of each part will be described in more detail below.

[0029] The region identification part 310 identifies a first region where the object undergoes a first motion and a second region where the object undergoes a second motion different from the first motion or the object does not substantially move in the RGB image 101 based on the event signal 201. In the case where motion occurs in the object in the RGB image 101, for example, a change in light intensity, specifically, a brightness change at the edge of the object is detected by the event signal 201, so that even if the entire object is not recognized in the RGB image 101, the region where the object has moved can be identified based on the event signal 201. In addition, for example, the region identification part 310 can also identify the direction of motion of the object in each region from the position change of the brightness change in the time series indicated by the event signal 201. As described above, in the present embodiment, the RGB image 101 includes a plurality of frames constituting a motion image, and thus the region identification part 310 can identify the first region and the second region in the plurality of frames.

[0030] The image processing section 320 performs different image processing on each of the first region and the second region in the RGB image 101 to obtain an output image 321. At this time, the image processing section 320 can associate the first region and the second region identified by the region identification section 310 with the RGB image 101 based on the time stamp 102 of the RGB image 101 and the time stamp 202 of the event signal 201. Although the content of the image processing performed by the image processing section 320 is not particularly limited, for example, the image processing section 320 can mark only the first region in the RGB image 101, or mark the first region and the second region in the RGB image 101 where the object moves differently in a different manner from each other. Alternatively, the image processing section 320 can cover the second region (for example, the background region) in the RGB image 101 where the object does not move substantially. Alternatively, in the case where the second region where the object does not move substantially is identified, the image processing section 320 covers the first region in each of the multiple frame images included in the RGB image 101 with the second region identified by the previous and next frame images, thereby hiding the moving object in the first region. Performing image processing that is different between the first region and the second region by the image processing portion 320 on the first region and the second region includes performing image processing on any one of the first region or the second region and outputting the other region as it is.

[0031] Figure 4 and Figure 5 1 is a diagram for explaining an example of region recognition and image processing according to an embodiment of the present invention. Figure 4 In the example shown, the region identification section 310 identifies the region R1 where the object moves and the region R2 where the object does not substantially move in time sequence from the event signal 201. For example, the region identification section 310 extracts the region R1 identified by the event signal 201 in time sequence, the magnitude of the optical flow F of which is equal to or greater than a predetermined threshold, and sets the remaining region as the region R2.

[0032] In another example, the region identification part 310 can divide the extracted region R1 according to the direction and magnitude of the motion indicated by the optical flow F, so as to identify a plurality of regions in which the object moves differently in each region. The optical flow F is an example of information indicating the direction of motion of the object. By identifying the direction of motion of the object, for example, in a plurality of frames included in the RGB image 101, the object corresponding to the region R1 can be continuously tracked (tracking is performed).

[0033] At the same time, if Figure 5 In the example shown, the image processing section 320 may overlay the image 101 with the image obtained by the region identification section 310 as described above. Figure 3The matching process between the images explained and the regions R1 and R2 identified by the event signal. In the illustrated example, the image processing section 320 masks the region R2 (background region) where the object does not substantially move in the RGB image 101, thereby obtaining an output image 321 in which the region R1 where the object movement occurs is cut out.

[0034] In another example, the image processing part 320 may mark the region R1 instead of masking the region R2 or together with masking the region R2. In the case where multiple regions where different motions of the object occur are identified, the image processing part 320 may mark the multiple regions in different ways from each other. For example, in the above example, where the motion direction of the object in each region is identified by the optical flow F and the object corresponding to each region is continuously tracked in the multiple frames of images included in the RGB image 101, the image processing part 320 may continuously perform masking processing on the above regions in the multiple frames of images or perform marking processing on the above regions.

[0035] In the embodiment of the present invention as described above, the region identification section 310 of the image processing device 300 identifies the region in which the object moves in the RGB image 101 based on the event signal 201. The event signal 201 is generated only in the case where a change in light intensity is detected in one or more pixels in the RGB image 101, and therefore, for example, the processing speed can be improved more than in the case where pixels in each of a plurality of RGB images 101 that are continuous in time are compared to identify the region in which the object moves. In addition, in general, since the EDS 200 can generate the event signal 201 at a cycle much shorter than the frame rate in the case where the RGB camera 100 captures a moving image, as in the present embodiment, for example, by using the event signal 201, the image processing can follow the movement of the object with higher accuracy than in the case where the movement of the object is identified by using only the RGB image 101 in the moving image.

[0036] Figure 6 1 is a flowchart indicating an example of an image processing method of an embodiment of the present invention. In the illustrated example, the RGB camera 100 captures the RGB image 101 (step S101), and at the same time, the EDS 200 generates the event signal 201 (step S102). Note that the step S102 of generating the event signal 201 is only performed when the sensor 210 corresponding to one or more pixels of the RGB image 101 detects a change in light intensity. The RGB image 101 is provided with a timestamp 102 (step S103), and the event signal is provided with a timestamp 202 (step S104).

[0037] Next, the processing in the image processing device 300 is performed. First, the region identification part 310 determines the first region and the second region in the RGB image 101 as described above based on the event signal 201 (step S105). Then, the image processing part 320 performs different image processing on the first and second regions identified in the RGB image 101 (step S106). At this time, the RGB image 101 on which the image processing is performed, and the first and second regions identified based on the event signal 201 are associated with each other by using the timestamps 102 and 202. For example, according to the above-mentioned processing, an output image 321 in which a moving object is cut and a background is masked can be obtained and the output image 321 is a moving image in which objects with different movements are tracked by marking in different ways.

[0038] Although some embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. Obviously, a person skilled in the art of the present invention can make various changes or modifications within the scope of the technical ideas described in the claims, and it should be understood that these changes and modifications obviously belong to the technical scope of the present invention.

[0039] [Reference Symbol List]

[0040] 10: System

[0041] 100: RGB Camera

[0042] 101: RGB images

[0043] 102: Timestamp

[0044] 110: Image sensor

[0045] 120: Processing circuit

[0046] 201: Event signal

[0047] 202: Timestamp

[0048] 210: Sensor

[0049] 220: Processing circuit

[0050] 300: Image processing equipment

[0051] 310: Region Identification Section

[0052] 320: Image processing part

[0053] 321: Output image

Claims

1. An image processing device, comprising: an area recognition section that recognizes, in the image, a first area in which a first motion of an object occurs and a second area in which the object does not substantially move, based on an event signal generated by a second event-driven vision sensor in response to a change in light intensity in one or more pixels of an image captured by synchronous scanning using the first vision sensor; as well as an image processing section that performs different image processing on the first region and the second region in the image, wherein: The image includes a plurality of frames constituting a motion image, The region identifying section identifies the first region and the second region in each of the plurality of frames of images, and The image processing section covers the first area with the second area identified by the preceding and succeeding frame images in each of the plurality of frame images.

2. The image processing apparatus according to claim 1, wherein: The image includes a plurality of frames constituting a motion image, The region recognition section recognizes the first region and the second region, and a moving direction of the object in at least the first region in each frame of the plurality of frames of images, and The image processing section continuously performs image processing on the first area and the second area in the plurality of frame images based on the motion direction.

3. The image processing device according to claim 1 or 2, wherein: The image processing section marks at least the first area in the image.

4. The image processing apparatus according to claim 3, wherein: The region identifying section identifies the second region in which the object undergoes a second motion different from the first motion, and The image processing section marks the first region and the second region in the image in different patterns.

5. The image processing apparatus according to claim 1, wherein: The image processing part masks the second area in the image.

6. An image processing method, comprising: The step of identifying, based on an event signal generated by a second event-driven vision sensor in response to a change in light intensity in one or more pixels of an image captured by synchronous scanning using the first vision sensor, a first region in which a first motion of an object occurs and a second region in which the object does not substantially move in the image; as well as A step of performing different image processing between the first region and the second region in the image, wherein: The image includes a plurality of frames constituting a motion image, The step of identifying includes identifying the first region and the second region in each frame of the plurality of frames of images, and The step of performing the image processing includes covering the first area with the second area identified by the previous and next frame images in each frame of the plurality of frame images.

7. A non-transitory computer-readable storage medium having stored thereon instructions which, when executed by a computer, cause the computer to implement: A function of identifying, in an image captured by synchronous scanning using the first vision sensor, a first region in which a first motion of an object occurs and a second region in which the object does not substantially move, based on an event signal generated by a second event-driven vision sensor in response to a change in light intensity in one or more pixels of the image captured using the first vision sensor by synchronous scanning; and A function of performing different image processing between the first region and the second region in the image, wherein: The image includes a plurality of frames constituting a motion image, The recognition function includes recognizing the first region and the second region in each frame of the plurality of frames of images, and The function of performing the image processing includes covering the first area with the second area identified by the previous and next frame images in each frame of the plurality of frame images.

Images