Image processing device and image processing method

Abstract

A distance image acquisition unit 22 acquires distance images from a distance measurement sensor unit 12 at a first frame rate. A luminance change image acquisition unit 23 acquires luminance change images from an EVS unit 13 at a second frame rate higher than the first frame rate. An interpolated image generation unit 25 generates an interpolated distance image between two temporally adjacent distance images on the basis of motion information obtained from a luminance change image in the same time slot.

Description

Image processing device and image processing method

[0001] The present invention relates to an image processing apparatus and an image processing method for generating an interpolated distance image between two distance images that are temporally adjacent.

[0002] In recent years, RGB cameras equipped with ToF (Time of Flight) sensors have become widespread. Cameras equipped with TOF sensors can generate 3D models by acquiring a distance image (depth map) from the TOF sensor and a color image from the RGB camera, and combining the two. The frame rate of distance images acquired from TOF sensors is generally around 30 fps. The frame rate of color images acquired from RGB cameras is typically 60 fps or higher.

[0003] When generating 3D data by combining a color image and a distance image, it is necessary to match the frame rate to the slower one, and the frame rate of the distance image has been a bottleneck in generating 3D data with smooth movement. In response to this, a method has been considered in which an interpolated image of the distance image is generated using a color image captured at a time between frames of the distance image (see, for example, Patent Document 1).

[0004] JP 2023-127450 A

[0005] When a moving subject is included in the field of view, it is necessary to pinpoint the position of the moving subject in the color image with high accuracy. Continuing to search for a moving subject in a color image increases the load, especially for high-resolution images. Furthermore, noise can cause stationary objects to be mistaken for moving objects.

[0006] The present embodiment has been made in view of the above circumstances, and its purpose is to provide a technique for generating an interpolated image of a distance image with high accuracy while suppressing the processing load.

[0007] In order to solve the above problem, an image processing device of one aspect of this embodiment comprises a first acquisition unit that acquires distance images from a ranging sensor unit at a first frame rate, a second acquisition unit that acquires brightness change images from an event vision sensor unit at a second frame rate that is faster than the first frame rate, and an interpolated image generation unit that generates an interpolated distance image between two temporally adjacent distance images based on movement information obtained from the brightness change images from the same time period.

[0008] Another aspect of this embodiment is an image processing method that includes the steps of acquiring distance images from a distance measurement sensor unit at a first frame rate, acquiring brightness transition images from an event vision sensor unit at a second frame rate that is faster than the first frame rate, and generating an interpolated distance image between two temporally adjacent distance images based on motion information obtained from the brightness transition images taken in the same time period.

[0009] Any combination of the above components, and conversion of the expression of this embodiment into a method, device, system, recording medium, computer program, etc. are also valid aspects of this embodiment.

[0010] According to this embodiment, an interpolated image of a distance image can be generated with high accuracy while suppressing the processing load.

[0011] FIG. 1 is a functional block diagram showing an example configuration of an image processing device according to an embodiment, and a sensor unit that supplies signals to the image processing device. FIG. 2 is a diagram showing an example of the progression of a color image and a distance image acquired from the sensor unit according to a comparative example. FIG. 3 is a diagram showing an example of the progression of a color image, a distance image, and a brightness transition image acquired from the sensor unit according to an embodiment. FIG. 4 is a diagram for explaining basic processing of generating an interpolated distance image using a brightness transition image. FIG. 5 is a diagram for explaining a first method of generating an interpolated distance image using a brightness transition image. FIG. 6 is a diagram for explaining a second method of generating an interpolated distance image using a brightness transition image. FIG. 7 is a diagram for explaining a third method of generating an interpolated distance image using a brightness transition image.

[0012] 1 is a functional block diagram showing an example configuration of an image processing device 20 according to an embodiment and a sensor unit 10 that supplies signals to the image processing device 20. The sensor unit 10 is a group of sensors mounted on a visible light camera with a ranging sensor. Examples of visible light cameras with ranging sensors include video cameras, surveillance cameras, in-vehicle cameras, and cameras mounted on multicopters (drones).

[0013] The image processing device 20 may be an image processing engine mounted on the camera, or may be a device separate from the camera that acquires and processes image data captured by the camera via a network or a recording medium. In the latter case, the image processing device 20 may be a general-purpose information processing device having an image processing function (e.g., a PC, a server, a tablet, or a smartphone), or may be a dedicated processing device having the image processing function according to this embodiment.

[0014] The sensor unit 10 includes a visible light imaging unit 11, a distance measurement sensor unit 12, and an EVS (Event-based Vision Sensor) unit 13. The visible light imaging unit 11 includes a lens, a color filter of three primary colors (RGB), and a solid-state imaging element. The solid-state imaging element may be, for example, a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor. The solid-state imaging element converts light collected by the lens and transmitted through the color filter into an electrical color image signal and outputs it to the image processing device 20.

[0015] The distance measurement sensor unit 12 acquires distance information to the subject from the distance measurement sensor unit 12 and outputs the information to the image processing device 20. In this embodiment, the distance measurement sensor unit 12 acquires distance information using a ToF method. The distance measurement sensor unit 12 includes a light source (e.g., an LD) and a light receiving sensor (e.g., a photodiode), and measures the distance to the subject based on the time difference between the emission timing of an infrared laser containing a predetermined pulse irradiated from the light source and the reception timing of light reflected from the subject by the light receiving sensor.

[0016] More specifically, the distance measurement sensor unit 12 irradiates the subject with an infrared laser whose intensity is modulated according to a predetermined irradiation pattern, and receives the infrared light reflected by the subject with an infrared light-receiving element. The distance measurement sensor unit 12 detects the time difference between irradiation and reception for each pixel according to the irradiation pattern, and calculates a depth value for each pixel. The distance measurement sensor unit 12 arranges the calculated depth values ​​for each pixel in a bitmap format to generate a distance image (depth map).

[0017] It is also possible to provide IR pixels in the solid-state imaging element of the visible light imaging unit 11 and use the IR pixels as infrared light receiving sensors. The ranging sensor unit 12 acquires distance information within the angle of view of the visible light imaging unit 11 by optically or mechanically changing the irradiation angle of the laser light or by diffusing the irradiation range of the laser light. It is also possible to use LiDAR (Laser Imaging Detection and Ranging) as the ranging sensor unit 12.

[0018] The EVS unit 13 compares the luminance change of the incident light with a predetermined threshold for each pixel, determines that a luminance change exceeding the predetermined threshold has occurred as an event, and outputs the coordinates of the pixel where the event occurred, the time, and the direction of the change in brightness. Each pixel operates independently and immediately outputs data upon detecting a luminance change. The EVS unit 13 accumulates the luminance change of the pixel for a period corresponding to the reciprocal of a predetermined frame rate, arranges the luminance change information of each pixel in a bitmap format, and outputs it as a luminance change image. Pixels where no luminance change has occurred during the period corresponding to the reciprocal of the predetermined frame rate are set to zero, resulting in an image in which only the contours of moving subjects (hereinafter referred to as moving objects) are extracted.

[0019] The light receiving sections of the visible light imaging section 11, the distance measurement sensor section 12, and the EVS section 13 are arranged on substantially the same optical axis. The frame rates of the visible light imaging section 11 and the EVS section 13 are set to the same value. The frame rates of the visible light imaging section 11 and the EVS section 13 are set to an integer multiple of the frame rate of the distance measurement sensor section 12. For example, the frame rate of the distance measurement sensor section 12 may be set to 30 fps, and the frame rate of the visible light imaging section 11 and the EVS section 13 may be set to 60 fps.

[0020] The frame rates of the visible light imaging unit 11 and the EVS unit 13 can also be set to 120 fps or 240 fps. The frame rates of the visible light imaging unit 11 and the EVS unit 13 can also be adaptively switched between 60 fps, 120 fps, and 240 fps. The frame rate of the EVS unit 13 can be changed arbitrarily by changing the accumulation time.

[0021] The frame rate of the EVS unit 13 may be designed to automatically synchronize with the frame rate of the visible light imaging unit 11. For example, when a user of a camera equipped with the sensor unit 10 changes the frame rate of video shooting, the frame rate of the EVS unit 13 may also be designed to automatically change to the frame rate changed by the user.

[0022] The image processing device 20 includes a color image acquisition unit 21, a distance image acquisition unit 22, a brightness variation image acquisition unit 23, a detection frame setting unit 24, and an interpolated image generation unit 25. The image processing device 20 is realized by a combination of hardware and software resources, or by hardware resources alone. Examples of hardware resources that can be used include a CPU, ROM, RAM, GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), and other LSIs. Examples of software resources that can be used include programs such as firmware.

[0023] The color image acquisition unit 21 acquires color images from the visible light imaging unit 11 at a default frame rate or a set frame rate changed by the user (e.g., 60 fps). The distance image acquisition unit 22 acquires distance images from the distance measurement sensor unit 12 at a frame rate slower than the set frame rate (e.g., 30 fps). The brightness variation image acquisition unit 23 acquires brightness variation images from the EVS unit 13 at the set frame rate.

[0024] The detection frame setting unit 24 sets a detection frame in a moving object region within the brightness transition image acquired by the brightness transition image acquisition unit 23. The detection frame setting unit 24 sets a rectangular detection frame that includes a cluster of brightness transition pixels of a certain number or more within the brightness transition image. The interpolated image generation unit 25 generates an interpolated distance image between two temporally adjacent distance images acquired from the distance measurement sensor unit 12, based on motion information obtained from the brightness transition images of the same time period.

[0025] 2 is a diagram showing an example of the progression of a color image and a distance image acquired from the sensor unit 10 according to a comparative example. The comparative example is an example in which the sensor unit 10 is not provided with an EVS unit 13. In the example shown in FIG. 2, the frame rate of the distance image is half that of the color image. To generate 3D data by combining the color image and the distance image, it is necessary to match the frame rate of the color image to the frame rate of the distance image, and in the comparative example, the color image Fcor2 was discarded.

[0026] 3 is a diagram showing an example of the progression of a color image, a distance image, and a brightness transition image acquired from the sensor unit 10 according to the embodiment. In the embodiment, an interpolated distance image between the first distance image Ftof1 and the second distance image Ftof2 is generated using information obtained from the brightness transition image Fevs2, thereby matching the frame rate of the distance image to the frame rate of the color image.

[0027] 4 is a diagram illustrating the basic process of generating an interpolated distance image using a brightness transition image. The interpolated image generation unit 25 searches for a cluster of brightness transition pixels of a certain number or more in the first brightness transition image Fevs1, and if a cluster of brightness transition pixels is detected, determines the detected cluster of brightness transition pixels as a moving object (a car in the example shown in FIG. 4 ). The interpolated image generation unit 25 sets the region of the detected cluster of brightness transition pixels as a moving object region in the first distance image Ftof1.

[0028] The interpolated image generation unit 25 searches for clusters of brightness-changing pixels of a certain number or more in the second brightness-changing image Fevs2, and if a cluster of brightness-changing pixels is detected, determines that the detected cluster of brightness-changing pixels is a moving object. The interpolated image generation unit 25 compares the cluster of brightness-changing pixels detected in the first brightness-changing image Fevs1 with the cluster of brightness-changing pixels detected in the second brightness-changing image Fevs2 to determine whether they are the same moving object. The interpolated image generation unit 25 determines whether they are the same moving object based on, for example, the similarity in size or shape of the cluster of brightness-changing pixels.

[0029] If the interpolated image generation unit 25 determines that the cluster of brightness-changing pixels detected in the first brightness-changing image Fevs1 and the cluster of brightness-changing pixels detected in the second brightness-changing image Fevs2 are the same moving object, it calculates a motion vector MV1 from the cluster of brightness-changing pixels detected in the first brightness-changing image Fevs1 to the cluster of brightness-changing pixels detected in the second brightness-changing image Fevs2.

[0030] The interpolated image generating unit 25 copies the first distance image Ftof1 to generate a first interpolated distance image Ftof1'. The interpolated image generating unit 25 overwrites the depth values ​​of the moving object region set in the first distance image Ftof1 by moving them according to the motion vector MV1 in the first interpolated distance image Ftof1'. The interpolated image generating unit 25 interpolates the depth values ​​of the region missing due to the movement of the moving object region from the depth values ​​of the surrounding background portion. For example, linear interpolation may be performed in the horizontal or vertical direction. Alternatively, the interpolated image generating unit 25 may fill in the missing region with the depth values ​​of the corresponding region in a distance image of another adjacent frame (for example, the second distance image Ftof2).

[0031] In the above description, the interpolated image generation unit 25 calculated the motion vector MV1 between the cluster of luminance-change pixels detected in the first luminance-change image Fevs1 and the cluster of luminance-change pixels detected in the second luminance-change image Fevs2. Alternatively, the interpolated image generation unit 25 may calculate the motion vector MV2 between the cluster of luminance-change pixels detected in the second luminance-change image Fevs2 and the cluster of luminance-change pixels detected in the third luminance-change image Fevs3. In this case, the interpolated image generation unit 25 sets the region of the cluster of luminance-change pixels detected in the third luminance-change image Fevs3 as the moving object region in the second distance image Ftof2. The interpolated image generation unit 25 generates the first interpolated distance image Ftof1' by copying the second distance image Ftof2. The interpolated image generating unit 25 overwrites the depth values ​​of the moving object region set in the second distance image Ftof2 in the first interpolated distance image Ftof1' by moving them in accordance with the motion vector MV2.

[0032] In the basic process for generating an interpolated distance image described above, if the orientation, size, or shape of a moving object changes between brightness-varying images, it is not possible to reflect this change in the interpolated distance image. Below, we will explain the process for generating an interpolated distance image that can reflect changes in the orientation of a moving object, etc.

[0033] 5 is a diagram illustrating a first method for generating an interpolated distance image using a brightness transition image. The interpolated image generation unit 25 generates an interpolated distance image Ftof1' based on the correspondence between multiple feature points in color image Fcor1, which is the same time as distance image Ftof1 of the two distance images Ftof1 and Ftof2, and multiple feature points in color image Fcor2, which is the same time as interpolated distance image Ftof1', as well as the one distance image Ftof1. This method will be described in detail below.

[0034] Similar to the basic processing described above, the interpolated image generation unit 25 detects clusters of brightness-varying pixels from the first brightness-varying image Fevs1, and sets the regions of the detected clusters of brightness-varying pixels as moving object regions in the first distance image Ftof1 and the first color image Fcor1.

[0035] The interpolated image generation unit 25 detects clusters of brightness-changing pixels from the second brightness-changing image Fevs2, similar to the basic processing described above. The interpolated image generation unit 25 sets the region of the detected cluster of brightness-changing pixels as a moving object region in the second color image Fcor2.

[0036] The interpolated image generation unit 25 extracts feature points from the moving object region in the first color image Fcor1 and extracts feature points from the moving object region in the second color image Fcor2. The interpolated image generation unit 25 tracks the movement of the moving object using the optical flow of the feature points. For example, corners detected by the Harris corner detection algorithm can be used as the feature points. The optical flow is a motion vector that represents the movement of the extracted feature points and can be calculated using, for example, the gradient method or the Lucas-Kanade method.

[0037] The interpolated image generation unit 25 copies the first distance image Ftof1 to generate a first interpolated distance image Ftof1'. The interpolated image generation unit 25 identifies, in the first distance image Ftof1, pixels of multiple feature points extracted from the moving object region in the first color image Fcor1. The interpolated image generation unit 25 identifies, in the first interpolated distance image Ftof1', pixels of multiple feature points extracted from the moving object region in the second color image Fcor2.

[0038] The interpolated image generation unit 25 moves the depth values ​​of the plurality of feature points identified in the first distance image Ftof1 to the pixels of the corresponding plurality of feature points identified in the first interpolated distance image Ftof1' and overwrites them. The interpolated image generation unit 25 overwrites the depth values ​​of pixels other than the feature points included in the moving object region in the first interpolated distance image Ftof1' with the average depth value of the moving object region in the first distance image Ftof1. Furthermore, the interpolated image generation unit 25 may linearly interpolate the depth values ​​of the moving object region in the first interpolated distance image Ftof1' using the depth values ​​of the feature points at the left and right ends of the moving object region, thereby adding a gradient to the depth values ​​within the moving object region. The interpolated image generation unit 25 omits the moving object region before movement in the first interpolated distance image Ftof1' and interpolates the depth values ​​of the omitted region from the depth values ​​of the surrounding background portion.

[0039] 6 is a diagram illustrating a second method for generating an interpolated distance image using a brightness transition image. Interpolated image generation unit 25 determines the motion vector of the detection frame and at least one of the rate of change of the size or the rate of change of the angle of the detection frame based on detection frame A set in brightness transition image Fevs1, which is the same time as distance image Ftof1 of the two distance images Ftof1, Ftof2, and detection frame B set in brightness transition image Fevs2, which is the same time as interpolated distance image Ftof1'.

[0040] The interpolated image generation unit 25 generates depth values ​​for the region in the interpolated distance image Ftof1′ that corresponds to the detection frame of the moving object, based on the motion vector of the detection frame, and at least one of the rate of change in size or the rate of change in angle of the detection frame, and the depth values ​​for the region in the one distance image Ftof1 that corresponds to the detection frame of the moving object. This will be explained in detail below.

[0041] Similar to the basic processing described above, the interpolated image generation unit 25 detects clusters of luminance-change pixels from the first luminance-change image Fevs1. The detection frame setting unit 24 sets a rectangular first detection frame A that contains the detected clusters of luminance-change pixels. Similar to the basic processing described above, the interpolated image generation unit 25 detects clusters of luminance-change pixels from the second luminance-change image Fevs2. The detection frame setting unit 24 sets a rectangular second detection frame B that contains the detected clusters of luminance-change pixels.

[0042] When the orientations of the first detection frame A and the second detection frame B do not match, the interpolated image generation unit 25 calculates the rate of change in the angle between the first detection frame A and the second detection frame B (i.e., the rotation angle). When the sizes of the first detection frame A and the second detection frame B do not match, the interpolated image generation unit 25 calculates the rate of change in size between the first detection frame A and the second detection frame B (enlargement rate or reduction rate).

[0043] The interpolated image generation unit 25 copies the first distance image Ftof1 to generate a first interpolated distance image Ftof1'. The interpolated image generation unit 25 cuts out depth values ​​of a region corresponding to first detection frame A from the first distance image Ftof1. The interpolated image generation unit 25 changes the size of the cut-out region by pixel interpolation or pixel thinning for the depth values ​​of the cut-out region according to the enlargement or reduction ratio of the sizes of first detection frame A and second detection frame B. The interpolated image generation unit 25 rotates the resized cut-out region according to the rotation angle from first detection frame A to second detection frame B. The interpolated image generation unit 25 overwrites the rotated cut-out region on the region of the first interpolated distance image Ftof1' that corresponds to second detection frame B. The interpolated image generation unit 25 omits the region corresponding to first detection frame A from the first interpolated distance image Ftof1' and interpolates the depth value of the missing region from the depth values ​​of the surrounding background portion.

[0044] 7 is a diagram illustrating a third method for generating an interpolated distance image using a brightness transition image. The third method for generating an interpolated distance image will be described below with reference to FIGS. 6 and 7. Interpolated image generation unit 25 determines the motion vector, the rate of change in size, and the rate of change in angle of the detection frame based on detection frames A and C set in two brightness transition images Febs1 and Febs3 that are generated at the same time as distance images Ftof1 and Ftof2, respectively, and detection frame B set in brightness transition image Febs2 that is generated at the same time as interpolated distance image Ftof1'.

[0045] The interpolated image generation unit 25 matches the sizes and orientations of the regions corresponding to detection frames A and C in the two distance images Ftof1 and Ftof2 based on the rate of change of the size and angle of the detection frames. The interpolated image generation unit 25 averages the depth values ​​of the regions corresponding to the detection frames of the moving object in the two distance images Ftof1 and Ftof2, whose sizes and orientations have been matched. The interpolated image generation unit 25 corrects the depth values ​​of the averaged regions corresponding to the detection frames of the moving object to match the size and orientation of detection frame B set in the brightness transition image Febs2 of the same time as the interpolated distance image Ftof1'. The interpolated image generation unit 25 generates depth values ​​of the regions corresponding to the detection frames of the moving object in the interpolated distance image based on the depth values ​​of the corrected regions corresponding to the detection frames of the moving object and the motion vectors of the detection frames. This is described in detail below.

[0046] Similar to the basic processing described above, the interpolated image generation unit 25 detects clusters of luminance-change pixels from the first luminance-change image Fevs1. The detection frame setting unit 24 sets a rectangular first detection frame A that contains the detected clusters of luminance-change pixels. Similar to the basic processing described above, the interpolated image generation unit 25 detects clusters of luminance-change pixels from the second luminance-change image Fevs2. The detection frame setting unit 24 sets a rectangular second detection frame B that contains the detected clusters of luminance-change pixels. Similar to the basic processing described above, the interpolated image generation unit 25 detects clusters of luminance-change pixels from the third luminance-change image Fevs3. The detection frame setting unit 24 sets a rectangular third detection frame C that contains the detected clusters of luminance-change pixels.

[0047] When the orientations of the first detection frame A and the third detection frame C do not match, the interpolated image generation unit 25 calculates the rate of change in the angle between the first detection frame A and the third detection frame C. When the sizes of the first detection frame A and the third detection frame C do not match, the interpolated image generation unit 25 calculates the rate of change in size (enlargement rate or reduction rate) between the first detection frame A and the third detection frame C.

[0048] The interpolated image generation unit 25 extracts depth values ​​of a region corresponding to the first detection frame A from the first distance image Ftof1. The interpolated image generation unit 25 extracts depth values ​​of a region corresponding to the third detection frame C from the second distance image Ftof2.

[0049] The interpolated image generation unit 25 rotates the clipped region Ac in the first distance image Ftof1 or the clipped region Cc in the second distance image Ftof2 in accordance with the rate of change of the angle between the first detection frame A and the third detection frame C. In the example shown in Fig. 7 , the clipped region Cc in the second distance image Ftof2 is rotated so that the orientations of the clipped region Ac in the first distance image Ftof1 and the clipped region Cc in the second distance image Ftof2 are aligned.

[0050] Interpolated image generation unit 25 interpolates or thins out pixels of the depth values ​​of clipped region Ac in first distance image Ftof1 or clipped region C'c in second distance image Ftof2 in accordance with the enlargement or reduction rate of the sizes of first detection frame A and third detection frame C, so that the sizes of clipped region Ac in first distance image Ftof1 or clipped region C'c in second distance image Ftof2 match. In the example shown in FIG. 7 , clipped region Ac in first distance image Ftof1 is enlarged so that the sizes of clipped region A'c in first distance image Ftof1 and clipped region C'c in second distance image Ftof2 match.

[0051] The interpolated image generation unit 25 averages the depth values ​​of the clipped region A'c in the first distance image Ftof1 and the depth values ​​of the clipped region C'c in the second distance image Ftof2 to generate the depth values ​​of the clipped region Bc for interpolation. The interpolated image generation unit 25 corrects the depth values ​​of the clipped region Bc for interpolation in accordance with the size and orientation of the detection frame B. In the example shown in FIG. 7 , the size of the clipped region Bc for interpolation is reduced and rotated counterclockwise.

[0052] Interpolated image generation unit 25 copies first distance image Ftof1 to generate first interpolated distance image Ftof1'. Interpolated image generation unit 25 overwrites clipped region B'c of the corrected interpolation clipped region Bc onto the region of first interpolated distance image Ftof1' that corresponds to second detection frame B. Interpolated image generation unit 25 omits the region that corresponds to first detection frame A from first interpolated distance image Ftof1', and interpolates the depth value of the omitted region from the depth value of the surrounding background portion.

[0053] The above-described process for generating an interpolated distance image allows the frame rate of the distance image to be matched to the frame rate of the color image. A 3D image data generator (not shown) can generate 3D image data by combining the color image and the distance image. The 3D image data is generated by generating point cloud data that expresses depth values ​​(Z) as points, generating mesh data composed of planes connecting the points from the point cloud data, and pasting a color image corresponding to the mesh plane. The 3D image based on the generated 3D image data can be displayed, for example, on a VR device or an AR device.

[0054] As described above, according to this embodiment, by using the brightness transition image output from the EVS unit 13, an interpolated image of a distance image can be generated with high accuracy while reducing the processing load. Detecting a moving object from a color image or a distance image requires complex calculations, which increases the processing load. In contrast, the brightness transition image contains only moving objects when it is output from the EVS unit 13, eliminating the need for processing to extract moving objects from the image through calculations, and enabling high-speed detection of moving objects.

[0055] Furthermore, since the EVS unit 13 is a sensor whose main purpose is to detect moving objects, it has high accuracy in detecting moving objects. When detecting moving objects by calculation from a color image or a distance image, a moving object may be mistakenly recognized as a stationary object, or a stationary object may be mistakenly recognized as a moving object, resulting in lower detection accuracy than when detecting moving objects from a brightness change image.

[0056] The first interpolated distance image generation method described above can reduce the processing load by identifying the position of a moving object in the interpolated distance image from a brightness transition image. It can also handle cases where the shape of a moving object changes irregularly. The second interpolated distance image generation method can generate an interpolated distance image with the lightest load. The third interpolated distance image generation method can improve the interpolation accuracy of moving objects compared to the second generation method.

[0057] As described above, in this embodiment, a 3D model of a subject moving at high speed can be generated with high accuracy.

[0058] The present invention has been described above based on the embodiments. These embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the respective components and treatment processes, and that such modifications are also within the scope of the present invention.

[0059] In the second interpolated distance image generation method described above, the interpolated image generation unit 25 determines the motion vector of the detection frame and at least one of the rate of change of the size or the rate of change of the angle of the detection frame based on the first detection frame A detected in the first brightness transition image Febs1 and the second detection frame B detected in the second brightness transition image Febs2. In contrast, the interpolated image generation unit 25 may determine the motion vector of the detection frame and at least one of the rate of change of the size or the rate of change of the angle of the detection frame based on the second detection frame B detected in the second brightness transition image Febs2 and the third detection frame C detected in the third brightness transition image Febs3.

[0060] In this case, the interpolated image generation unit 25 generates a depth value of the area corresponding to the detection frame of the moving object in the interpolated distance image Ftof1' based on the motion vector of the detection frame, and at least one of the rate of change of the size of the detection frame or the rate of change of the angle of the detection frame, and the depth value of the area corresponding to the detection frame of the moving object in the second distance image Ftof2.

[0061] In addition, the interpolated image generation unit 25 may compare the similarity between the first detection frame A and the second detection frame B with the similarity between the second detection frame B and the third detection frame C, and generate a depth value for the area corresponding to the detection frame of the moving object in the interpolated distance image Ftof1' based on the combination with the higher similarity.

[0062] In the third interpolated distance image generation method described above, the interpolated image generation unit 25 generates the first interpolated distance image Ftof1' by copying the first distance image Ftof1. However, the interpolated image generation unit 25 may also generate the first interpolated distance image Ftof1' by copying the second distance image Ftof2. In this case, the interpolated image generation unit 25 removes the area corresponding to the third detection frame C from the first interpolated distance image Ftof1', and interpolates the depth value of the removed area from the depth value of the surrounding background portion.

[0063] Alternatively, interpolated image generation unit 25 may generate first interpolated distance image Ftof1' by averaging the first distance image Ftof1 and the second distance image Ftof2. In this case, interpolated image generation unit 25 removes the regions corresponding to first detection frame A and third detection frame C from first interpolated distance image Ftof1', and interpolates the depth values ​​of the removed regions from the depth values ​​of the surrounding background.

[0064] Furthermore, if the resolutions of the sensors are different, one pixel of a sensor with a low resolution may be associated with a plurality of pixels of a sensor with a high resolution.

[0065] The present invention can be used in a visible light camera equipped with a distance measurement sensor.

[0066] REFERENCE SIGNS LIST 10 Sensor unit, 11 Visible light imaging unit, 12 Distance measurement sensor unit, 13 EVS unit, 20 Image processing device, 21 Color image acquisition unit, 22 Distance image acquisition unit, 23 Brightness change image acquisition unit, 24 Detection frame setting unit, 25 Interpolated image generation unit.

Claims

1. An image processing device comprising: a first acquisition unit that acquires distance images from a distance measurement sensor unit at a first frame rate; a second acquisition unit that acquires brightness change images from an event vision sensor unit at a second frame rate that is faster than the first frame rate; and an interpolated image generation unit that generates an interpolated distance image between two distance images that are adjacent in time based on motion information obtained from the brightness change images in the same time period.

2. The image processing device of claim 1, further comprising a detection frame setting unit that sets a detection frame on a moving object in the brightness change image, wherein the interpolated image generation unit determines a motion vector of the detection frame and at least one of a rate of change of size or a rate of change of angle of the detection frame based on a detection frame set in the brightness change image at the same time as one of the distance images of the two distance images and a detection frame set in the brightness change image at the same time as the interpolated distance image, and generates a depth value of a region corresponding to the detection frame of the moving object in the interpolated distance image based on the motion vector of the detection frame and at least one of the rate of change of size or the rate of change of angle of the detection frame and a depth value of a region corresponding to the detection frame of the moving object in one of the distance images.

3. The image processing device of claim 1, further comprising a detection frame setting unit that sets a detection frame on a moving object in the brightness change image, wherein the interpolated image generation unit: determines the motion vector of the detection frame, the rate of change in size of the detection frame, and the rate of change in angle of the detection frame based on the detection frame set in the two brightness change images at the same time as each of the two distance images and the detection frame set in the brightness change image at the same time as the interpolated distance image; matches the size and orientation of the area corresponding to the detection frame in the two distance images based on the rate of change in size of the detection frame and the rate of change in angle of the detection frame; averages the depth values ​​of the area corresponding to the detection frame of the moving object in the two distance images whose sizes and orientations have been matched; corrects the depth value of the averaged area corresponding to the detection frame of the moving object to match the size and orientation of the detection frame set in the brightness change image at the same time as the interpolated distance image; and generates a depth value of the area corresponding to the detection frame of the moving object in the interpolated distance image based on the depth value of the area corresponding to the detection frame of the moving object after correcting the detection frame and the motion vector of the detection frame.

4. An image processing device as described in claim 1, further comprising a third acquisition unit that acquires color images from a visible light imaging unit at the second frame rate, wherein the interpolated image generation unit generates the interpolated distance image based on a correspondence between a plurality of feature points in the color image at the same time as one of the distance images of the two distance images and a plurality of feature points in the color image at the same time as the interpolated distance image, and the one distance image.

5. An image processing method comprising the steps of: acquiring a distance image from a distance measurement sensor unit at a first frame rate; acquiring a brightness change image from an event vision sensor unit at a second frame rate faster than the first frame rate; and generating an interpolated distance image between two temporally adjacent distance images based on motion information obtained from the brightness change images in the same time period.

Images