Image processing device and image processing method

Abstract

To generate an interpolated distance image with high accuracy while suppressing processing load.SOLUTION: A color image acquisition unit 21 acquires distance images from a distance sensor unit 12 at a first frame rate. A distance image acquisition unit 22 acquires luminance change images from an event-based vision sensor (EVS) unit 13 at a second frame rate higher than the first frame rate. An interpolated image generation unit 25 generates an interpolated distance image between two temporally adjacent distance images on the basis of motion information obtained from a luminance change image in the same time slot.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method for generating an interpolated distance image between two temporally adjacent distance images.

Background Art

[0002] In recent years, RGB cameras with ToF (Time of Flight) sensors have become widespread. In a camera with a ToF sensor, a distance image (depth map) can be acquired from the ToF sensor, a color image can be acquired from the RGB camera, and a 3D model can be generated by combining the two. The frame rate of the distance image acquired from the ToF sensor is generally about 30 fps. The frames of the color image acquired from the RGB camera mainly have a frame rate of 60 fps or more.

[0003] When generating 3D data by combining a color image and a distance image, it is necessary to match the slower frame rate, and the frame rate of the distance image has been a bottleneck in generating smooth motion 3D data. In contrast, a method of generating an interpolated image of the distance image using the color image captured at the time between the frames of the distance image can be considered (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When the captured subject moves in the angle of view, it is necessary to accurately identify the position of the moving subject in the color image. Continuing the search process for the moving subject in the color image increases the load. This is especially true for high-resolution images. In addition, due to noise, a stationary object may be misjudged as a moving object.

[0006] This embodiment has been made in view of such a situation, and its object is to provide a technique for generating an interpolated image of a distance image with high accuracy while suppressing the processing load.

Means for Solving the Problem

[0007] In order to solve the above problems, an image processing apparatus according to an aspect of the present embodiment includes a first acquisition unit that acquires a distance image from a distance measurement sensor unit at a first frame rate, and an event vision sensor unit that acquires a luminance change image from the event vision sensor unit at a second frame rate higher than the first frame rate, and an interpolation image generation unit that generates an interpolated distance image between two temporally adjacent distance images based on motion information obtained from the luminance change image in the same time period.

[0008] Another aspect of the present invention is an image processing method. This method includes a step of acquiring a distance image from a distance measurement sensor unit at a first frame rate, a step of acquiring a luminance change image from an event vision sensor unit at a second frame rate higher than the first frame rate, and a step of generating an interpolated distance image between two temporally adjacent distance images based on motion information obtained from the luminance change image in the same time period.

[0009] Note that any combination of the above components, and those obtained by converting the expressions of this embodiment among a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as aspects of this embodiment.

Advantages of the Invention

[0010] According to this embodiment, an interpolated image of a distance image can be generated with high accuracy while suppressing the processing load.

Brief Description of Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0012] FIG. 1 is a functional block diagram showing a configuration example of an image processing apparatus 20 according to an embodiment and a sensor unit 10 that supplies signals to the image processing apparatus 20. The sensor unit 10 is a group of sensors mounted on a visible light camera with a distance measuring sensor. Examples of the visible light camera with a distance measuring sensor include a video camera, a surveillance camera, an in-vehicle camera, and a camera mounted on a multicopter (drone).

[0013] The image processing device 20 may be an image processing engine mounted on the camera, or may be a device separate from the camera that acquires and processes image data captured by the camera via a network or a recording medium. In the latter case, the image processing device 20 may be a general-purpose information processing device having an image processing function (e.g., a PC, a server, a tablet, a smartphone), or may be a dedicated processing device having the image processing function according to the present embodiment.

[0014] The sensor unit 10 includes a visible light imaging unit 11, a distance measurement sensor unit 12, and an EVS (Event-based Vision Sensor) unit 13. The visible light imaging unit 11 includes a lens, a color filter of three primary colors (RGB), and a solid-state imaging device. For the solid-state imaging device, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor can be used. The solid-state imaging device condenses light with a lens, converts the light transmitted through the color filter into an electrical color image signal, and outputs it to the image processing device 20.

[0015] The distance measurement sensor unit 12 acquires distance information from the distance measurement sensor unit 12 to the subject and outputs it to the image processing device 20. In the present embodiment, the distance measurement sensor unit 12 acquires distance information by the ToF method. The distance measurement sensor unit 12 includes a light source (e.g., an LD) and a light receiving sensor (e.g., a photodiode), and measures the distance to the subject based on the time difference between the emission timing of an infrared laser including a predetermined pulse irradiated from the light source and the reception timing of the reflected light from the subject by the light receiving sensor.

[0016] More specifically, the distance measurement sensor unit 12 irradiates a subject with an infrared laser intensity-modulated by a predetermined irradiation pattern, and receives the infrared light reflected by the subject with a light receiving element for infrared light. The distance measurement sensor unit 12 detects the time difference from irradiation to light reception for each pixel according to the irradiation pattern, and calculates the depth value of each pixel. The distance measurement sensor unit 12 arranges the calculated depth values of each pixel in a bitmap pattern to generate a distance image (depth map).

[0017] In addition, IR pixels may be provided in the solid-state imaging device of the visible light imaging unit 11, and the IR pixels may be used as light receiving sensors for infrared light. The distance measurement sensor unit 12 acquires distance information within the angle of view range of the visible light imaging unit 11 by optically or mechanically changing the irradiation angle of the laser light or diffusing the irradiation range of the laser light. Note that LiDAR (Laser Imaging Detection and Ranging) may be used as the distance measurement sensor unit 12.

[0018] The EVS unit 13 compares the luminance change of the incident light with a predetermined threshold value for each pixel, determines that a luminance change exceeding the predetermined threshold value is an event occurrence, and outputs the coordinates, time, and light and dark change direction of the pixel where the event has occurred. Each pixel operates independently and outputs data immediately when a luminance change is detected. The EVS unit 13 accumulates the luminance changes of the pixels for a period corresponding to the reciprocal of a predetermined frame rate, arranges the luminance change information of each pixel in a bitmap pattern, and outputs it as a luminance change image. For pixels where no luminance change has occurred during a period corresponding to the reciprocal of the predetermined frame rate, the value becomes zero, resulting in an image in which only the contour portion of a moving subject (hereinafter referred to as a moving object) is extracted.

[0019] The light receiving parts of the visible light imaging unit 11, the distance measurement sensor unit 12, and the EVS unit 13 are arranged on substantially the same optical axis. The frame rates of the visible light imaging unit 11 and the EVS unit 13 are set to the same value. The frame rates of the visible light imaging unit 11 and the EVS unit 13 are set to an integer multiple of the frame rate of the distance measurement sensor unit 12. For example, the frame rate of the distance measurement sensor unit 12 may be set to 30 fps, and the frame rates of the visible light imaging unit 11 and the EVS unit 13 may be set to 60 fps.

[0020] Note that the frame rates of the visible light imaging unit 11 and the EVS unit 13 can also be set to 120 fps or 240 fps. Further, the frame rates of the visible light imaging unit 11 and the EVS unit 13 may be adaptively switched among 60 fps, 120 fps, and 240 fps. The frame rate of the EVS unit 13 can be arbitrarily changed by changing the above-described accumulation time.

[0021] Note that the frame rate of the EVS unit 13 may be designed to be automatically synchronized with the frame rate of the visible light imaging unit 11. For example, when a user of a camera equipped with the sensor unit 10 changes the frame rate of video shooting, the frame rate of the EVS unit 13 may also be designed to be automatically changed to the frame rate changed by the user.

[0022] The image processing apparatus 20 includes a color image acquisition unit 21, a distance image acquisition unit 22, a luminance change image acquisition unit 23, a detection frame setting unit 24, and an interpolation image generation unit 25. The image processing apparatus 20 is realized by the cooperation of hardware resources and software resources, or by hardware resources only. As the hardware resources, a CPU, a ROM, a RAM, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and other LSIs can be used. As the software resources, programs such as firmware can be used.

[0023] The color image acquisition unit 21 acquires a color image from the visible light imaging unit 11 at a set frame rate (for example, 60 fps) that is default or set and changed by the user. The distance image acquisition unit 22 acquires a distance image from the distance measurement sensor unit 12 at a frame rate lower (for example, 30 fps) than the set frame rate. The luminance change image acquisition unit 23 acquires a luminance change image from the EVS unit 13 at the set frame rate.

[0024] The detection frame setting unit 24 sets a detection frame for a moving object region in the luminance change image acquired by the luminance change image acquisition unit 23. The detection frame setting unit 24 sets a rectangular detection frame that encloses a block of a certain number or more of luminance change pixels in the luminance change image. The interpolation image generation unit 25 generates an interpolated distance image between two temporally adjacent distance images acquired from the distance measurement sensor unit 12 based on the motion information obtained from the luminance change image in the same time zone.

[0025] FIG. 2 is a diagram showing an example of the transition of the color image and the distance image acquired from the sensor unit 10 according to the comparative example. The comparative example is an example in which the EVS unit 13 is not provided in the sensor unit 10. In the example shown in FIG. 2, the frame rate of the distance image is 1 / 2 of the frame rate of the color image. In order to generate 3D data by combining the color image and the distance image, it is necessary to match the frame rate of the color image to the frame rate of the distance image, and in the comparative example, the color image Fcor2 was discarded.

[0026] FIG. 3 is a diagram showing an example of the transition of the color image, the distance image, and the luminance change image acquired from the sensor unit 10 according to the embodiment. In the embodiment, by generating an interpolated distance image between the first distance image Ftof1 and the second distance image Ftof2 using the information obtained from the luminance change image Fevs2, the frame rate of the distance image is matched to the frame rate of the color image.

[0027] FIG. 4 is a diagram for explaining the basic process of generating an interpolated distance image using the luminance change image. The interpolation image generation unit 25 searches for a block of a certain number or more of luminance change pixels from the first luminance change image Fevs1, and when a block of luminance change pixels is detected, determines the detected block of luminance change pixels as a moving object (an automobile in the example shown in FIG. 4). The interpolation image generation unit 25 sets the region of the detected block of luminance change pixels as the moving object region in the first distance image Ftof1.

[0028] The interpolation image generation unit 25 searches for a block of a certain number or more of luminance change pixels from the second luminance change image Fevs2, and when detecting a block of luminance change pixels, determines the detected block of luminance change pixels as a moving object. The interpolation image generation unit 25 compares the block of luminance change pixels detected in the first luminance change image Fevs1 with the block of luminance change pixels detected in the second luminance change image Fevs2 to determine whether they are the same moving object. For example, the interpolation image generation unit 25 determines whether they are the same moving object based on the similarity of the size or shape of the block of luminance change pixels.

[0029] When the interpolation image generation unit 25 determines that the block of luminance change pixels detected in the first luminance change image Fevs1 and the block of luminance change pixels detected in the second luminance change image Fevs2 are the same moving object, the interpolation image generation unit 25 calculates a motion vector MV1 from the block of luminance change pixels detected in the first luminance change image Fevs1 to the block of luminance change pixels detected in the second luminance change image Fevs2.

[0030] The interpolation image generation unit 25 copies the first distance image Ftof1 to generate a first interpolated distance image Ftof1'. In the first interpolated distance image Ftof1', the depth value of the moving object region set in the first distance image Ftof1 is moved and overwritten according to the motion vector MV1. The interpolation image generation unit 25 interpolates the depth value of the region missing due to the movement of the moving object region from the depth values of the surrounding background portions. For example, linear interpolation may be performed in the horizontal or vertical direction. Also, the interpolation image generation unit 25 may fill the missing region with the depth value of the corresponding region of the distance image of another adjacent frame (for example, the second distance image Ftof2).

[0031] In the above description, the interpolation image generation unit 25 calculated the motion vector MV1 between the block of luminance change pixels detected in the first luminance change image Fevs1 and the block of luminance change pixels detected in the second luminance change image Fevs2. In this regard, the interpolation image generation unit 25 may calculate the motion vector MV2 between the block of luminance change pixels detected in the second luminance change image Fevs2 and the block of luminance change pixels detected in the third luminance change image Fevs3. In that case, the interpolation image generation unit 25 sets the region of the block of luminance change pixels detected in the third luminance change image Fevs3 as the moving object region in the second distance image Ftof2. The interpolation image generation unit 25 copies the second distance image Ftof2 to generate the first interpolated distance image Ftof1'. In the first interpolated distance image Ftof1', the interpolation image generation unit 25 moves and overwrites the depth value of the moving object region set in the second distance image Ftof2 according to the motion vector MV2.

[0032] In the basic process of generating the interpolated distance image described above, when the orientation, size, or shape of the moving object changes between the luminance change images, the change cannot be reflected in the interpolated distance image. Hereinafter, the generation process of the interpolated distance image that can reflect changes such as the orientation of the moving object will be described.

[0033] FIG. 5 is a diagram for explaining a first method of generating an interpolated distance image using a luminance change image. The interpolation image generation unit 25 generates the interpolated distance image Ftof1' based on the correspondence relationship between a plurality of feature points in one of the two distance images Ftof1, Ftof2, i.e., the distance image Ftof1, and a plurality of feature points in the color image Fcor1 at the same time, and the correspondence relationship between a plurality of feature points in the interpolated distance image Ftof1' and the color image Fcor2 at the same time, and the above-mentioned one distance image Ftof1. Hereinafter, it will be specifically described.

[0034] Similar to the basic process described above, the interpolation image generation unit 25 detects a block of luminance change pixels from the first luminance change image Fevs1. The interpolation image generation unit 25 sets the region of the detected block of luminance change pixels as the moving object region in the first distance image Ftof1 and the first color image Fcor1.

[0035] Similar to the above-described basic process, the interpolation image generation unit 25 detects a block of luminance change pixels from the second luminance change image Fevs2. The interpolation image generation unit 25 sets the region of the detected block of luminance change pixels as the moving object region in the second color image Fcor2.

[0036] The interpolation image generation unit 25 extracts feature points from the moving object region in the first color image Fcor1 and extracts feature points from the moving object region in the second color image Fcor2. The interpolation image generation unit 25 follows the movement of the moving object using the optical flow of the feature points. As the feature points, for example, corners detected by the Harris corner detection algorithm can be used. The optical flow is a motion vector representing the movement of the extracted feature points and can be calculated using, for example, the gradient method or the Lucas-Kanade method.

[0037] The interpolation image generation unit 25 copies the first distance image Ftof1 to generate a first interpolated distance image Ftof1'. The interpolation image generation unit 25 specifies the pixels of a plurality of feature points extracted from the moving object region in the first color image Fcor1 in the first distance image Ftof1. The interpolation image generation unit 25 specifies the pixels of a plurality of feature points extracted from the moving object region in the second color image Fcor2 in the first interpolated distance image Ftof1'.

[0038] The interpolation image generation unit 25 moves and overwrites the depth values of each of the plurality of feature points identified in the first distance image Ftof1 to the pixels of the corresponding plurality of feature points identified in the first interpolated distance image Ftof1'. The interpolation image generation unit 25 overwrites the depth values of the pixels other than the feature points included in the moving object region in the first interpolated distance image Ftof1' with the average value of the depth values of the moving object region in the first distance image Ftof1. Further, the interpolation image generation unit 25 may apply a slope to the depth values within the moving object region by linearly interpolating the depth values of the moving object region in the first interpolated distance image Ftof1' using the depth values of the feature points at the left end and the right end of the moving object region. The interpolation image generation unit 25 deletes the moving object region before movement in the first interpolated distance image Ftof1' and interpolates the depth values of the deleted region from the depth values of the surrounding background portion.

[0039] FIG. 6 is a diagram for explaining a second method of generating an interpolated distance image using a luminance change image. The interpolation image generation unit 25 is based on the detection frame A set in the luminance change image Fevs1 at the same time as one of the two distance images Ftof1 and Ftof2, i.e., the distance image Ftof1, and the detection frame B set in the luminance change image Fevs2 at the same time as the interpolated distance image Ftof1', and identifies at least one of the motion vector of the detection frame, the change rate of the size of the detection frame, or the change rate of the angle of the detection frame.

[0040] The interpolation image generation unit 25 generates the depth values of the region corresponding to the detection frame of the moving object in the interpolated distance image Ftof1' based on at least one of the motion vector of the detection frame, the change rate of the size of the detection frame, or the change rate of the angle of the detection frame, and the depth values of the region corresponding to the detection frame of the moving object in the above-mentioned one distance image Ftof1. The following is a specific explanation.

[0041] Similar to the above-described basic process, the interpolation image generation unit 25 detects a block of luminance change pixels from the first luminance change image Fevs1. The detection frame setting unit 24 sets a first detection frame A, which is a rectangle enclosing the detected block of luminance change pixels. Similar to the above-described basic process, the interpolation image generation unit 25 detects a block of luminance change pixels from the second luminance change image Fevs2. The detection frame setting unit 24 sets a second detection frame B, which is a rectangle enclosing the detected block of luminance change pixels.

[0042] When the orientations of the first detection frame A and the second detection frame B do not match, the interpolation image generation unit 25 calculates the rate of change of the angle (i.e., the rotation angle) between the first detection frame A and the second detection frame B. When the sizes of the first detection frame A and the second detection frame B do not match, the interpolation image generation unit 25 calculates the rate of change of the sizes (expansion rate or reduction rate) between the first detection frame A and the second detection frame B.

[0043] The interpolation image generation unit 25 copies the first distance image Ftof1 to generate a first interpolated distance image Ftof1'. The interpolation image generation unit 25 cuts out the depth value of the region corresponding to the first detection frame A from the first distance image Ftof1. The interpolation image generation unit 25 performs pixel interpolation or pixel decimation on the depth value of the cut-out region according to the expansion rate or reduction rate of the sizes between the first detection frame A and the second detection frame B to change the size of the cut-out region. The interpolation image generation unit 25 rotates the cut-out region with the changed size according to the rotation angle from the first detection frame A to the second detection frame B. The interpolation image generation unit 25 overwrites the rotated cut-out region on the region corresponding to the second detection frame B of the first interpolated distance image Ftof1'. The interpolation image generation unit 25 deletes the region corresponding to the first detection frame A in the first interpolated distance image Ftof1' and interpolates the depth value of the deleted region from the depth values of the surrounding background portions.

[0044] FIG. 7 is a diagram for explaining a third method of generating an interpolation distance image using a luminance change image. Hereinafter, a third method of generating an interpolation distance image will be described with reference to FIGS. 6 and 7. The interpolation image generation unit 25 determines a motion vector of the detection frame, a change rate of the size of the detection frame, and a change rate of the angle of the detection frame based on the detection frames A and C set in the two distance images Ftof1 and Ftof2 and the two luminance change images Fevs1 and Fevs3 at the same time, and the detection frame B set in the luminance change image Fevs2 at the same time as the interpolation distance image Ftof1'.

[0045] The interpolation image generation unit 25 makes the sizes and orientations of the regions corresponding to the detection frames A and C in the two distance images Ftof1 and Ftof2 correspond to each other based on the change rate of the size of the detection frame and the change rate of the angle of the detection frame. The interpolation image generation unit 25 averages the depth values of the regions corresponding to the moving object detection frames in the two distance images Ftof1 and Ftof2 with the sizes and orientations corresponding to each other. The interpolation image generation unit 25 corrects the depth value of the region corresponding to the averaged moving object detection frame according to the size and orientation of the detection frame B set in the luminance change image Fevs2 at the same time as the interpolation distance image Ftof1'. The interpolation image generation unit 25 generates the depth value of the region corresponding to the moving object detection frame in the interpolation distance image based on the corrected depth value of the region corresponding to the moving object detection frame and the motion vector of the detection frame. Hereinafter, a specific description will be given.

[0046] The interpolation image generation unit 25 detects a block of luminance change pixels from the first luminance change image Fevs1 in the same manner as the above-described basic process. The detection frame setting unit 24 sets a first detection frame A, which is a rectangle enclosing the detected block of luminance change pixels. The interpolation image generation unit 25 detects a block of luminance change pixels from the second luminance change image Fevs2 in the same manner as the above-described basic process. The detection frame setting unit 24 sets a second detection frame B, which is a rectangle enclosing the detected block of luminance change pixels. The interpolation image generation unit 25 detects a block of luminance change pixels from the third luminance change image Fevs3 in the same manner as the above-described basic process. The detection frame setting unit 24 sets a third detection frame C, which is a rectangle enclosing the detected block of luminance change pixels.

[0047] When the orientations of the first detection frame A and the third detection frame C do not match, the interpolation image generation unit 25 calculates the rate of change of the angle between the first detection frame A and the third detection frame C. When the sizes of the first detection frame A and the third detection frame C do not match, the interpolation image generation unit 25 calculates the rate of change of the sizes (expansion rate or reduction rate) between the first detection frame A and the third detection frame C.

[0048] The interpolation image generation unit 25 extracts the depth value of the region corresponding to the first detection frame A from the first distance image Ftof1. The interpolation image generation unit 25 extracts the depth value of the region corresponding to the third detection frame C from the second distance image Ftof2.

[0049] The interpolation image generation unit 25 rotates the cutout region Ac of the first distance image Ftof1 or the cutout region Cc of the second distance image Ftof2 according to the rate of change of the angle between the first detection frame A and the third detection frame C. In the example shown in FIG. 7, the cutout region Cc of the second distance image Ftof2 is rotated to align the orientations of the cutout region Ac of the first distance image Ftof1 and the cutout region Cc of the second distance image Ftof2.

[0050] The interpolation image generation unit 25 performs pixel interpolation or pixel decimation on the depth value of the cutout region Ac of the first distance image Ftof1 or the cutout region C'c of the second distance image Ftof2 according to the expansion rate or reduction rate of the sizes between the first detection frame A and the third detection frame C to make the sizes of the cutout region Ac of the first distance image Ftof1 or the cutout region C'c of the second distance image Ftof2 match. In the example shown in FIG. 7, the cutout region Ac of the first distance image Ftof1 is enlarged to make the sizes of the cutout region A'c of the first distance image Ftof1 and the cutout region C'c of the second distance image Ftof2 match.

[0051] The interpolation image generation unit 25 averages the depth value of the cutout region A'c of the first distance image Ftof1 and the depth value of the cutout region C'c of the second distance image Ftof2 to generate the depth value of the cutout region Bc for interpolation. The interpolation image generation unit 25 corrects the depth value of the cutout region Bc for interpolation according to the size and orientation of the detection frame B. In the example shown in FIG. 7, the size of the cutout region Bc for interpolation is reduced and rotated counterclockwise.

[0052] The interpolation image generation unit 25 copies the first distance image Ftof1 to generate a first interpolated distance image Ftof1'. The interpolation image generation unit 25 overwrites the cutout area B'c of the corrected cutout area Bc for interpolation with the area corresponding to the second detection frame B of the first interpolated distance image Ftof1'. The interpolation image generation unit 25 deletes the area corresponding to the first detection frame A in the first interpolated distance image Ftof1', and interpolates the depth value of the deleted area from the depth values of the surrounding background portions.

[0053] By the generation process of the interpolated distance image described above, the frame rate of the distance image can be adjusted to match the frame rate of the color image. A three-dimensional image data generation unit (not shown) can generate three-dimensional image data by combining the color image and the distance image. The three-dimensional image data generates point cloud data representing the depth value (Z) as points, generates mesh data composed of planes connecting the points from the point cloud data, and generates a color image corresponding to the plane of the mesh and pastes it. The three-dimensional image based on the generated three-dimensional image data can be displayed on, for example, a VR device or an AR device.

[0054] As described above, according to the present embodiment, by using the luminance change image output from the EVS unit 13, an interpolated image of the distance image can be generated with high accuracy while suppressing the processing load. Complicated calculations are required to detect a moving object from a color image or a distance image, and the processing load increases. In this regard, the luminance change image is an image that only includes a moving object when it is output from the EVS unit 13, and the process of extracting the moving object from the image by calculation is unnecessary, and the moving object can be detected at high speed.

[0055] Also, since the EVS unit 13 is a sensor mainly for detecting a moving object, the detection accuracy of the moving object is high. When detecting a moving object by calculation from a color image or a distance image, the moving object may be misrecognized as a stationary object, or a stationary object may be misrecognized as a moving object, and the detection accuracy is lower than when detecting a moving object from the luminance change image.

[0056] In the first method for generating the interpolation distance image described above, the processing load can be reduced by specifying the position of the moving object in the interpolation distance image from the luminance change image. Also, it can handle the case where the shape of the moving object changes irregularly. In the second method for generating the interpolation distance image, the interpolation distance image can be generated with the lightest load. In the third method for generating the interpolation distance image, the interpolation accuracy of the moving object can be improved compared to the second method.

[0057] From the above, in this embodiment, a 3D model of a fast-moving subject can also be generated with high accuracy.

[0058] As described above, the present invention has been described based on embodiments. It should be understood by those skilled in the art that these embodiments are illustrative, and various modifications are possible for each combination of their respective components and each processing process, and such modifications are also within the scope of the present invention.

[0059] In the second method for generating the interpolation distance image described above, based on the first detection frame A detected in the first luminance change image Fevs1 and the second detection frame B detected in the second luminance change image Fevs2, the interpolation image generation unit 25 specified at least one of the motion vector of the detection frame, the change rate of the size of the detection frame, or the change rate of the angle of the detection frame. In this regard, the interpolation image generation unit 25 may specify at least one of the motion vector of the detection frame, the change rate of the size of the detection frame, or the change rate of the angle of the detection frame based on the second detection frame B detected in the second luminance change image Fevs2 and the third detection frame C detected in the third luminance change image Fevs3.

[0060] In this case, based on at least one of the motion vector of the detection frame, the change rate of the size of the detection frame, or the change rate of the angle of the detection frame, and the depth value of the region corresponding to the detection frame of the moving object in the second distance image Ftof2, the interpolation image generation unit 25 generates the depth value of the region corresponding to the detection frame of the moving object in the interpolation distance image Ftof1'.

[0061] Note that the interpolation image generation unit 25 may compare the similarity between the first detection frame A and the second detection frame B and the similarity between the second detection frame B and the third detection frame C, and generate the depth value of the region corresponding to the detection frame of the moving object in the interpolation distance image Ftof1’ based on the combination with the higher similarity.

[0062] In the third generation method of the interpolation distance image described above, the interpolation image generation unit 25 generated the first interpolation distance image Ftof1’ by copying the first distance image Ftof1. In this regard, the interpolation image generation unit 25 may generate the first interpolation distance image Ftof1’ by copying the second distance image Ftof2. In this case, the interpolation image generation unit 25 deletes the region corresponding to the third detection frame C in the first interpolation distance image Ftof1’, and interpolates the depth value of the deleted region from the depth values of the surrounding background portions.

[0063] Also, the interpolation image generation unit 25 may generate the average value of the first distance image Ftof1 and the second distance image Ftof2 as the first interpolation distance image Ftof1’. In this case, the interpolation image generation unit 25 deletes the regions corresponding to the first detection frame A and the third detection frame C in the first interpolation distance image Ftof1’, and interpolates the depth values of the deleted regions from the depth values of the surrounding background portions.

[0064] Also, when the resolutions of the respective sensors are different, a plurality of pixels of the sensor with a higher resolution may be made to correspond to one pixel of the sensor with a lower resolution.

Explanation of Signs

[0065] 10 Sensor unit, 11 Visible light imaging unit, 12 Distance measurement sensor unit, 13 EVS unit, 20 Image processing apparatus, 21 Color image acquisition unit, 22 Distance image acquisition unit, 23 Luminance change image acquisition unit, 24 Detection frame setting unit, 25 Interpolation image generation unit.

Claims

1. A first acquisition unit that acquires a distance image at a first frame rate from a distance measurement sensor unit; A second acquisition unit that acquires a luminance change image at a second frame rate higher than the first frame rate from an event vision sensor unit; An interpolation image generation unit that generates an interpolation distance image between two temporally adjacent distance images based on motion information obtained from the luminance change image in the same time period; An image processing apparatus comprising the above.

2. Further comprising a detection frame setting unit that sets a detection frame for a moving object in the luminance change image; The interpolation image generation unit: Based on a detection frame set in one of the two distance images and the luminance change image at the same time, a detection frame set in the luminance change image at the same time as the interpolation distance image, the motion vector of the detection frame, and at least one of a change rate of the size of the detection frame or a change rate of the angle of the detection frame are specified; Based on at least one of the motion vector of the detection frame, a change rate of the size of the detection frame or a change rate of the angle of the detection frame, and a depth value of a region corresponding to the detection frame of the moving object in the one distance image, a depth value of a region corresponding to the detection frame of the moving object in the interpolation distance image is generated; The image processing apparatus according to Claim 1.

3. Further comprising a detection frame setting unit that sets a detection frame for a moving object in the luminance change image; The interpolation image generation unit: Based on detection frames set in two luminance change images at the same time as each of the two distance images, and a detection frame set in the luminance change image at the same time as the interpolation distance image, the motion vector of the detection frame, a change rate of the size of the detection frame, and a change rate of the angle of the detection frame are specified; Based on the change rate of the size of the detection frame and the change rate of the angle of the detection frame, the sizes and orientations of regions corresponding to the detection frame in the two distance images are made to correspond; Average the depth values of the regions corresponding to the detection frames of the moving object in the two distance images, corresponding to the size and orientation. Correct the depth value of the region corresponding to the detection frame of the averaged moving object according to the size and orientation of the detection frame set in the interpolation distance image and the luminance change image at the same time. Generate the depth value of the region corresponding to the detection frame of the moving object in the interpolation distance image based on the depth value of the region corresponding to the detection frame of the corrected moving object and the motion vector of the detection frame. The image processing apparatus according to claim 1.

4. Further comprising a third acquisition unit that acquires a color image from the visible light imaging unit at the second frame rate. The interpolation image generation unit Based on the correspondence between a plurality of feature points in the color image at the same time as one of the two distance images and the plurality of feature points in the color image at the same time as the interpolation distance image, and the one distance image, generate the interpolation distance image. The image processing apparatus according to claim 1.

5. A step of acquiring a distance image from the distance measurement sensor unit at the first frame rate, A step of acquiring a luminance change image from the event vision sensor unit at a second frame rate higher than the first frame rate, A step of generating an interpolation distance image between two temporally adjacent distance images based on the motion information obtained from the luminance change image in the same time period, An image processing method having the above.

Images