Image generation device, program and ranging system

Abstract

To enable performing high density processing with a higher degree of accuracy.SOLUTION: An image generation device comprises: an acquisition unit that acquires a luminance image of first resolution and a first distance image of second resolution being lower than the first resolution; a conversion unit that converts the first distance image into a sparse distance image corresponding to the luminance image; a filter processing unit that performs filter processing based on a distance value included in a predetermined area around a first pixel for the distance value of the first pixel in the sparse distance image; and a densification unit that generates a second distance image of a third resolution being higher than the second resolution by performing predetermined high density processing for the sparse distance image after filter processing.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image generation device, a program, and a distance measurement system.

Background Art

[0002] Conventionally, a distance measurement device that measures the distance from an imaging device to an object using the ToF (Time of Flight) method is known. Such a distance measurement device irradiates a subject with distance measurement light that is infrared light intensity-modulated by a predetermined irradiation pattern, and then receives the distance measurement light reflected by the object with an infrared imaging element. The distance is calculated by detecting the time difference from irradiation to reception for each pixel according to the irradiation pattern. Then, the distance measurement device collects the calculated distance values in a bitmap for each pixel and stores them as a "distance image".

[0003] In the ToF method, since color information cannot be obtained, an RGB camera is often used in combination. However, the ToF light-receiving sensor has a lower resolution than the RGB light-receiving sensor. Therefore, in order to combine color information and distance information for use in image processing or the like, it is necessary to project the distance information onto the RGB coordinate system. However, as described above, due to the difference in resolution, an image is obtained in which the distance is obtained every few pixels to several tens of pixels on the RGB image, so some kind of high-resolution processing is required.

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the prior art, upsampling is performed on the assumption that the distance to the position on the object corresponding to one pixel of the ToF sensor is pinpoint-measured for one pixel of the corresponding RGB imaging element (see Patent Document 1 below).

[0005] However, in the prior art, high-density processing cannot be performed with higher accuracy.

[0006] In order to solve the problems of the above-described prior art, an object of the present invention is to perform densification processing with higher accuracy.

Means for Solving the Problems

[0007] In order to solve the above-described problems, an image generation apparatus according to an embodiment includes: an acquisition unit that acquires a luminance image with a first resolution and a first distance image with a second resolution lower than the first resolution; a conversion unit that converts the first distance image into a sparse distance image corresponding to the luminance image; a filter processing unit that performs a filter process on the distance value of a first pixel in the sparse distance image based on distance values included in a predetermined region around the first pixel; and a densification unit that generates a second distance image with a third resolution higher than the second resolution by performing a predetermined densification process on the sparse distance image after the filter process.

Effects of the Invention

[0008] According to the image generation apparatus according to an embodiment, densification processing can be performed with higher accuracy.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, an embodiment will be described with reference to the drawings.

[0011] (Functional configuration of distance measurement device 100) FIG. 1 is a diagram showing the functional configuration of a distance measuring device 100 according to an embodiment. The distance measuring device 100 according to an embodiment is an example of a "distance measuring system". As shown in FIG. 1, the distance measuring device 100 according to an embodiment includes a light projecting unit 110, a luminance light receiving unit 120, a ToF light receiving unit 130, and a distance measurement control unit 140.

[0012] The light projecting unit 110 includes a light source 111 and a light source driving unit 112. The light projecting unit 110 irradiates the object with the irradiation light emitted from the light source 111.

[0013] The luminance light receiving unit 120 is an example of a "first light receiving unit". The luminance light receiving unit 120 has an image sensor 121 and an ADC 122. The luminance light receiving unit 120 outputs, as a "luminance image" with a first resolution, the light received by the image sensor 121 that has been changed to a digital value by the ADC 122.

[0014] The ToF light receiving unit 130 is an example of a "second light receiving unit". The ToF light receiving unit 130 has a ToF light receiving unit 130, two image sensors 131, and two ADCs 132. The ToF light receiving unit 130 outputs, as a "first distance image" with a second resolution lower than the first resolution, the phase signal (reflected light of the irradiation light reflected by the object) received by the two image sensors 131 that has been changed to a digital value by the two ADCs 132.

[0015] The distance measurement control unit 140 is an example of an "image generation device" and an "image generation unit". The distance measurement control unit 140 includes an RGB control unit 141, an RGB image storage unit 142, a ToF control unit 143, a ToF image storage unit 144, a sparse depth creation unit 145, a sparse filter processing unit 146, a densification unit 147, and an output unit 148.

[0016] The RGB control unit 141 controls the operation of the luminance light receiving unit 120. The RGB image storage unit 142 stores the luminance image output from the ADC 122. The ToF control unit 143 controls the operations of the light projecting unit 110 and the ToF light receiving unit 130. For example, the ToF control unit 143 can time-modulate (temporally control) the light emission by the light source 111 by outputting a drive signal with a predetermined voltage waveform and a predetermined light emission frequency to the light source drive unit 112 of the light projecting unit 110. The ToF image storage unit 144 stores the first distance image output from the ADC 132.

[0017] The sparse depth creation unit 145 is an example of a "conversion unit". The sparse depth creation unit 145 calculates the coordinates corresponding to the ToF distance information based on the first distance image stored in the ToF image storage unit 144 in the RGB image, converts it into a distance value as seen from the RGB image, and generates an image having the same size as the RGB image having the distance value as a pixel value as a sparse depth image (an example of a "sparse distance image"). Note that the sparse depth creation unit 145 can calculate the coordinates of the RGB image corresponding to the ToF distance information by using various parameters obtained in advance (for example, camera parameters, rotation and translation parameters between RGB and ToF, etc.).

[0018] The sparse filter processing unit 146 is an example of a "filter processing unit". The sparse filter processing unit 146 performs a predetermined filter processing on the sparse depth image while the distance information of the sparse depth image created by the sparse depth creation unit 145 is not densified.

[0019] The densification unit 147 performs a predetermined densification process on the sparse distance image after the filter process by the sparse filter processing unit 146 to generate a second distance image with a third resolution higher than the second resolution.

[0020] The output unit 148 outputs the second distance image generated by the densification unit 147.

[0021] (Hardware Configuration of the Distance Measuring Device 100) FIG. 2 is a diagram showing the hardware configuration of the distance measuring device 100 according to an embodiment.

[0022] As the light source 111, for example, a VCSEL (Vertical Cavity Surface Emitting LASER) can be used. The light source 111 emits laser light from the VCSEL as irradiation light over a wide range through, for example, a wide-angle lens or a fish-eye lens. Note that the distance measuring device 100 may use a combination of a plurality of light sources 111. For example, the distance measuring device 100 may use a combination of one or more light sources 111 that irradiate diffused light and one or more light sources 111 that irradiate spot light.

[0023] The image sensor 121 included in the luminance light receiving unit 120 is a sensor that can acquire general luminance information and outputs an RGB image or a black-and-white image.

[0024] The image sensor 131 included in the ToF light receiving unit 130 is a so-called ToF (Time of Flight) sensor that receives the reflected light of the irradiation light emitted from the light source 111 to the object. The image sensor 131 acquires, for each pixel, an electrical signal corresponding to the intensity of the received reflected light and divides it into a plurality of phase signals. The ADC 132 included in the ToF light receiving unit 130 converts the phase signals acquired for each pixel from analog signals into digital data and outputs them to the distance measurement control unit 140.

[0025] The distance measurement control unit 140 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a light source drive circuit 205, a sensor I / F (Interface) 206, a sensor I / F 207, and an input / output I / F 208. These are electrically connected to each other via a system bus.

[0026] The CPU 201 reads a program or data from a storage device such as the ROM 202 or the SSD 204 onto the RAM 203 and executes processing to control the entire distance measurement control unit 140. Note that part or all of the functions of the CPU 201 may be realized by an electronic circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0027] The ROM 202 is a non-volatile semiconductor memory (storage device) that can retain programs or data even when the power is turned off. The ROM 202 stores programs or data such as the BIOS (Basic Input / Output System) executed when the CPU 201 starts up and OS (Operating System) settings.

[0028] The RAM 203 is a volatile semiconductor memory (storage device) that temporarily holds programs or data.

[0029] The SSD 204 is a non-volatile memory in which programs or various data for executing processing by the distance measurement control unit 140 are stored. As an example, a distance measurement imaging program is stored in the SSD 204. Specifically, as will be described later, the CPU 201 controls the image sensor 131 so as to acquire, for each pixel, an electrical signal corresponding to the intensity of the received reflected light and divide it into a plurality of phase signals by executing this distance measurement imaging program. Note that another storage device such as an HDD (Hard Disk Drive) may be used instead of the SSD 204.

[0030] The sensor I / F 206 is an interface for acquiring the luminance signal from the image sensor 121. The sensor I / F 207 is an interface for acquiring the phase signal from the image sensor 131. The sensor I / F 206 and the sensor I / F 207 are examples of the "acquisition unit". The input / output I / F 208 is an interface for connecting to an external device such as a main controller device or a personal computer device.

[0031] Based on the control signal supplied from the CPU 201, the light source drive circuit 205 supplies a drive signal such as a drive voltage to the light source 111 to drive it to emit light. As the drive signal supplied to the light source 111, a rectangular wave, a sine wave, or a voltage waveform with a predetermined waveform shape can be used. The light source drive circuit 205 changes the frequency of the voltage waveform to modulate and control the frequency of the drive signal. Further, the light source drive circuit 205 can also simultaneously control the light emission of some of the plurality of light sources 111, or change the light source 111 that emits light.

[0032] (Distance measurement principle by the distance measurement device 100) FIG. 3 is a timing chart for explaining the distance measurement principle by the distance measurement device 100 according to an embodiment. FIG. 3(a) shows the timing of light projection, and FIG. 3(b) shows the timing of the reflected light obtained by the light projection. Further, FIG. 3(c) shows the timing at which a phase signal with a phase of 0 degrees is accumulated in the first charge accumulation unit among the two charge accumulation units provided in the image sensor 131, and FIG. 3(d) shows the timing at which a phase signal with a phase of 180 degrees is accumulated in the second charge accumulation unit. Further, FIG. 3(e) shows the timing at which a phase signal with a phase of 90 degrees is accumulated in the first charge accumulation unit among the two charge accumulation units provided in the image sensor 131, and FIG. 3(f) shows the timing at which a phase signal with a phase of 270 degrees is accumulated in the second charge accumulation unit.

[0033] The image sensor 131 has two charge accumulation units (a first charge accumulation unit and a second charge accumulation unit) for one light receiving element, and can rapidly switch the charge accumulation unit that accumulates charges. For this reason, the image sensor 131 can simultaneously detect two phase signals that are exactly opposite to each other for one rectangular wave. As an example, the image sensor 131 can simultaneously detect a phase signal of 0 degrees and a phase signal of 180 degrees. Further, the image sensor 131 can simultaneously detect a phase signal of 90 degrees and a phase signal of 270 degrees. This means that distance measurement is possible through two light projection and light reception processes.

[0034] Between the times indicated by the hatching in FIGS. 3(c) to 3(f), the charges of the phase signals of respective phases are accumulated in the first charge accumulation unit or the second charge accumulation unit. Specifically, as the charge of the phase signal of 0 degrees, as shown in FIG. 3(c), the charge between the pulse edge of the end of light projection and the pulse edge of the start of reception of the reflected light is accumulated in the first charge accumulation unit. As the charge of the phase signal of 180 degrees, as shown in FIG. 3(d), the charge between the completion of charge accumulation of the phase signal of 0 degrees and the pulse edge of the end of reception of the reflected light is accumulated in the second charge accumulation unit.

[0035] Similarly, as the charge of the phase signal of 90 degrees, as shown in FIG. 3(e), the charge between the pulse edge of the start of reception of the reflected light and the pulse edge of the end of charge accumulation of the pulse for performing charge accumulation control is accumulated in the first charge accumulation unit. As the charge of the phase signal of 270 degrees, as shown in FIG. 3(f), the charge between the completion of charge accumulation of the phase signal of 90 degrees and the pulse edge of the end of reception of the reflected light is accumulated in the second charge accumulation unit.

[0036] Note that, actually, in order to increase the amount of charge to be accumulated, the light projection is not a single rectangular wave but a repeating pattern of rectangular waves, and the switching control to the first and second charge accumulation units according to the timing of projecting the light of this repeating pattern is also repeatedly performed.

[0037] The four phase signals of 0 degrees (A0), 90 degrees (A90), 180 degrees (A180), and 270 degrees (A270) are phase signals that are temporally divided into four phases of 0 degrees, 90 degrees, 180 degrees, and 270 degrees with respect to the pulse period of the light (irradiation light) to be projected. Therefore, the phase difference angle φ can be obtained using the following mathematical formula (1).

[0038] φ = Arctan{(A90 - A270) / (A0 - A180)} ··· (1)

[0039] Also, from this phase difference angle φ, the delay time Td can be obtained using the following mathematical formula (2).

[0040] Td = (φ / 2π) × T (T = 2T0, T0: Pulse width of the irradiation light) ··· (2)

[0041] Further, from this delay time Td, the distance value d to the object can be obtained using the following mathematical formula (3).

[0042] d = Td × c ÷ 2 (c: Speed of light) ··· (3)

[0043] The example in Fig. 3 is an example of acquiring the phase signals at 0 degrees and 180 degrees in the first measurement. However, when there is an influence of external light, the charge amount of the second charge accumulation unit is subtracted from the charge amount of the first charge accumulation unit obtained in the first measurement, and a phase signal with reduced influence of external light is generated. In such a measurement, one phase signal is acquired by one light emission and exposure. Therefore, to acquire the phase signals for four phases, four light emissions and exposures are required, and the imaging time becomes twice that of the case without external light.

[0044] In the following description, it is assumed that the phase signal obtained by one light emission and the exposure of the reflected light is a phase signal with the influence of external light eliminated, which is calculated from the charge amounts of the first charge accumulation unit and the second charge accumulation unit.

[0045] (First example of the method for creating a sparse depth image by the sparse depth creation unit 145) Fig. 4 is a diagram for explaining a first example of the method for creating a sparse depth image by the sparse depth creation unit 145 according to an embodiment.

[0046] As shown in Fig. 4(a), when the image sensor 121 of the luminance light receiving unit 120, the image sensor 131 of the ToF light receiving unit 130, and the light source 111 of the light projecting unit 110 are arranged side by side, the sparse depth creation unit 145 can calculate the coordinates of the RGB image corresponding to the distance measurement points of ToF by a known calculation method based on the camera parameters of the image sensors 121 and 131 and the parameters of the rotation and translation between RGB and ToF. At this time, the sparse depth creation unit 145 uses, as parameters, those that have been calculated by mapping a pre-known chart pattern, etc.

[0047] The image sensor 121 (RGB sensor) has a higher resolution than the image sensor 131 (ToF sensor). Therefore, as shown in Fig. 4(b), when each of the plurality of distance values detected by the image sensor 131 is projected onto the corresponding coordinates for the RGB image, the RGB image has a state where the plurality of distance values are sparsely present. Such a state is called a "sparse structure", and a distance image in which the plurality of distance values have a sparse structure is called a "sparse depth image".

[0048] Since the sparse depth image has a lower resolution than the RGB image, the number of distance values is insufficient for the number of pixels in the RGB image, and it is difficult to use it as a 4-channel RGBD image in various image processes. Therefore, it is common to increase the density (upsampling) by some means.

[0049] As methods for increasing the density, a nearest neighbor interpolation method, a joint bilateral filter for increasing the density while referring to RGB information as a guide, etc. are used. In recent years, in the field of Depth Completion, many methods using deep learning techniques have also been proposed.

[0050] (Second example of the method for creating a sparse depth image by the sparse depth creation unit 145) Fig. 5 is a diagram for explaining a second example of the method for creating a sparse depth image by the sparse depth creation unit 145 according to an embodiment.

[0051] The distance measuring device 100 according to an embodiment may adopt a configuration in which the image sensor 121 of the luminance light receiving unit 120, the two image sensors 131a and 131b of the ToF light receiving unit 130, and the light source 111 of the light projecting unit 110 are arranged side by side as shown in Fig. 5(a).

[0052] In this case, as shown in FIG. 5(b), the sparse depth creation unit 145 projects each of a plurality of distance values detected from the "ToF light reception area 1" shown in FIG. 5(a) by one image sensor 131a and each of a plurality of detected distance values detected from the "ToF light reception area 2" shown in FIG. 5(a) by the other image sensor 131b onto corresponding coordinates with respect to the RGB image. Thereby, the sparse depth creation unit 145 can acquire ranging points over a wider range.

[0053] However, in this case, when combining a plurality of distance images received by a plurality of ToF sensors, discontinuity (step shift) occurs in the overlapping portion, so there is a risk of deterioration in accuracy. Specifically, due to various factors such as noise (calibration error, temperature error, etc.) of a single ToF camera, individual differences between cameras, projection error, etc., discontinuity (step shift) occurs in the overlapping portion of two distance images in the sparse depth image, and it may become difficult to improve the accuracy of the sparse depth image by subsequent high-precision processing.

[0054] Therefore, the distance measurement device 100 according to an embodiment performs the following filter processing on the sparse depth image in order to solve such problems.

[0055] (General filter processing using a median filter) In general filter processing using a median filter, a block region of a predetermined size (for example, 3X3 pixels) is set centering on the pixel to be processed, and the median value of all values within the block pixels is set as the new pixel value of the pixel to be processed. However, in the case of a sparse depth image obtained by projecting ToF distance values onto an RGB image, pixels having values are sparsely present, so general filter processing is not suitable. Even if the size of the block region is enlarged, only the number of 0 values increases. Therefore, the sparse filter processing unit 146 according to an embodiment performs unique filter processing using a median filter as described below with reference to FIGS. 6 and 7.

[0056] (Example of the procedure of filter processing) FIG. 6 is a flowchart showing an example of the procedure of filter processing by the sparse filter processing unit 146 according to an embodiment. Here, the procedure of filter processing when using a median filter will be exemplified.

[0057] First, the sparse filter processing unit 146 selects a pixel having a pixel value in the sparse depth image as a pixel to be processed (an example of the "first pixel"), and determines whether the pixel value of the pixel to be processed is greater than 0 (step S601).

[0058] In step S601, if it is determined that the pixel value of the pixel to be processed is not greater than 0 (step S601: NO), the sparse filter processing unit 146 proceeds to step S605.

[0059] In step S601, if it is determined that the pixel value of the pixel to be processed is greater than 0 (step S601: YES), the sparse filter processing unit 146 sets a processing target region of a predetermined size (for example, 7×7 pixels) centered on the pixel to be processed for the sparse depth image (step S602).

[0060] Next, the sparse filter processing unit 146 extracts a plurality of pixel values greater than 0 in the processing target region set in step S602, and calculates the median value of the plurality of pixel values (step S603).

[0061] Next, the sparse filter processing unit 146 determines the median value calculated in step S603 as the new pixel value of the pixel to be processed (step S604). Then, the sparse filter processing unit 146 proceeds to step S605.

[0062] In step S605, the sparse filter processing unit 146 determines whether the processing for all the pixels of the sparse depth image has been completed (step S605).

[0063] In step S605, if it is determined that the processing for all pixels of the sparse depth image has not been completed (step S605: NO), the sparse filter processing unit 146 returns the processing to step S601.

[0064] In step S605, if it is determined that the processing for all pixels of the sparse depth image has been completed (step S605: YES), the sparse filter processing unit 146 ends the series of processes shown in FIG. 6.

[0065] (Specific Example of Filter Processing) FIG. 7 is a diagram showing a specific example of filter processing by the sparse filter processing unit 146 according to an embodiment. Here, a specific example of filter processing when using a median filter will be exemplified.

[0066] In the example shown in FIG. 7(a), a processing target region of a predetermined size (for example, 7×7 pixels) centered on the processing target pixel (that is, the pixel having a pixel value) selected from the sparse depth image by the sparse filter processing unit 146 is shown.

[0067] In this case, the sparse filter processing unit 146 extracts "1", "1", "1", "2", "2", "3", "3", "4", "8" as a plurality of pixel values greater than 0 in the processing target region. Then, the sparse filter processing unit 146 calculates "2" as the median value of these plurality of pixel values.

[0068] Furthermore, as shown in FIG. 7(b), the sparse filter processing unit 146 determines the calculated median value "2" as the new pixel value of the processing target pixel.

[0069] In this way, the sparse filter processing unit 146 according to an embodiment performs, as filter processing, a process of updating the pixel value of the processing target pixel using a plurality of pixel values greater than 0 included in the processing target region centered on the processing target pixel on the sparse depth image while maintaining the sparse structure of the sparse depth image.

[0070] Note that, as the size of the processing target area used for the above filter processing, an appropriate size corresponding to the resolution of the ToF sensor and the resolution of the RGB sensor is preset in the distance measurement control unit 140.

[0071] Also, in the above example, the filter processing using the median filter was illustrated, but it is not limited to this. For example, filter processing using a joint bilateral filter guided by RGB may be performed. Also in this case, similar to the case of using the median filter, it can be calculated by referring to the RGB value at the same position as the pixel value having the distance value.

[0072] (Effect of Filter Processing) FIG. 8 is a diagram for explaining the effect of the filter processing by the sparse filter processing unit 146 according to an embodiment. As shown in FIG. 8, the distance measurement device 100 according to an embodiment can smooth two distance images in the sparse depth image by performing filter processing on the sparse depth image by the sparse filter processing unit 146. In particular, by performing filter processing on a sparse depth image in which a plurality of distance images each having an overlapping region that overlaps with each other are combined, in the overlapping portion of the two distance images in the sparse depth image, the distance value of one distance image and the distance value of the other distance image are smoothly (continuously) connected.

[0073] (Comparative Example of Effect of High-Density Processing) FIG. 9 is a diagram showing a comparative example of the effect of the high-density processing by the high-density unit 147 according to an embodiment.

[0074] As shown in FIG. 9(a), in the distance measurement device 100 according to an embodiment, if the high-density unit 147 performs high-density processing by nearest neighbor interpolation on the sparse depth image without performing filter processing, in the high-resolution distance image obtained by the high-density processing, a discontinuous connection occurs in the overlapping portion of the two distance images.

[0075] On the other hand, as shown in FIG. 9(b), in the distance measurement device 100 according to an embodiment, when the densification unit 147 performs densification processing by nearest neighbor completion after performing filter processing on the sparse depth image, in the high-resolution distance image obtained by the densification processing, a smooth (continuous) connection occurs in the overlapping portion of the two distance images.

[0076] (Effect of Occlusion) FIG. 10 is a diagram showing the principle of occlusion occurrence in the distance measurement device 100 according to an embodiment. FIG. 11 is a diagram for explaining the influence of occlusion on the distance image in the distance measurement device 100 according to an embodiment.

[0077] As shown in FIG. 10, when the positions of the image sensor 121 of the luminance light receiving unit 120 and the image sensor 131 of the ToF light receiving unit 130 are different, occlusion (detecting an object on the back side on the RGB image at the distance of the object in the front) may occur, and the accuracy may deteriorate.

[0078] For example, in the example shown in FIG. 10, the distance value of the object on the back side detected by the image sensor 131 is detected as the distance value of the edge portion of the object in the front detected by the image sensor 121.

[0079] Therefore, in the example shown in FIG. 11(a), in the RGB image after projection of the distance measurement value, the distance measurement value of the object on the back side is projected onto the edge portion of the object in the front.

[0080] As a result, in the example shown in FIG. 11(b), in the high-resolution distance image obtained by the densification processing, a defect occurs due to the projection of the distance measurement value of the object on the back side onto the edge portion of the image of the object in the front.

[0081] Therefore, the distance measurement device 100 according to an embodiment performs the deletion processing described below by the deletion unit 149 on the sparse depth image before performing filter processing in order to solve such problems.

[0082] (Overview of the deletion process by the deletion unit 149) FIG. 12 is a diagram for explaining the overview of the deletion process by the deletion unit 149 according to an embodiment.

[0083] First, as shown in FIG. 12(a), the deletion unit 149 detects the edge portion E of the foreground object with respect to the RGB image using a known technique such as a Sobel filter. Next, as shown in FIG. 12(b), the deletion unit 149 deletes the Tof distance measurement points existing on the detected edge portion E of the foreground object.

[0084] In this way, the distance measuring device 100 according to an embodiment performs filter processing and densification processing on the sparse depth image after the deletion process by the deletion unit 149, so that, as shown in FIG. 12(c), in the high-resolution distance image obtained by the densification process, it is possible to avoid the occurrence of a missing portion in the edge portion E of the image of the foreground object.

[0085] (An example of the procedure of the deletion process) FIG. 13 is a flowchart showing an example of the procedure of the deletion process by the deletion unit 149 according to an embodiment.

[0086] First, the deletion unit 149 selects a pixel in the sparse depth image as a pixel to be processed, and determines whether the pixel value of the pixel to be processed is greater than 0 (step S1301).

[0087] If it is determined in step S1301 that the pixel value of the pixel to be processed is not greater than 0 (step S1301: NO), the deletion unit 149 proceeds to step S1304 with the processing.

[0088] If it is determined in step S1301 that the pixel value of the pixel to be processed is greater than 0 (step S1301: YES), it is determined whether the pixel to be processed is a pixel corresponding to the edge portion of the object in the RGB image (step S1302).

[0089] In step S1302, when it is determined that the pixel to be processed is not a pixel corresponding to the edge portion of the object in the RGB image (step S1302: NO), the deletion unit 149 proceeds to step S1304.

[0090] In step S1302, when it is determined that the pixel to be processed is a pixel corresponding to the edge portion of the object in the RGB image (step S1302: YES), 0 is set to the distance value of the pixel to be processed in the sparse depth image (step S1303). Then, the deletion unit 149 proceeds to step S1304.

[0091] In step S1304, the deletion unit 149 determines whether the processing for all the pixels of the sparse depth image has been completed (step S1304).

[0092] In step S1304, when it is determined that the processing for all the pixels of the sparse depth image has not been completed (step S1304: NO), the deletion unit 149 returns the processing to step S1301.

[0093] In step S1304, when it is determined that the processing for all the pixels of the sparse depth image has been completed (step S1304: YES), the deletion unit 149 ends the series of processes shown in FIG. 13.

[0094] (Modification example of deletion process) FIG. 14 is a diagram for explaining an outline of a process of detecting a distance difference edge E' by the deletion unit 149 according to an embodiment. In the above-described deletion process, there is a risk of deleting distance values of portions where no occlusion occurs, such as simple patterns and boundaries between the ground and objects. Therefore, as shown in FIG. 14, the deletion unit 149 sets a processing target region centered on the pixel determined to be the edge portion E (an example of a luminance edge), and when the difference between the maximum value and the minimum value of the distance values included in the region is equal to or greater than a certain value, the pixel may be left as the "distance difference edge E'". Thereby, the deletion unit 149 can prevent the distance values of portions where no occlusion occurs from being deleted.

[0095] (Example of the procedure for detecting the distance difference edge E') FIG. 15 is a flowchart showing an example of the procedure for detecting the distance difference edge E' by the deletion unit 149 according to an embodiment.

[0096] First, the deletion unit 149 selects one pixel of the edge image as a pixel to be processed, and determines whether the pixel to be processed corresponds to the edge portion E (step S1501).

[0097] In step S1501, if it is determined that the pixel to be processed does not correspond to the edge portion E (step S1501: NO), the deletion unit 149 proceeds to step S1506.

[0098] In step S1501, if it is determined that the pixel to be processed corresponds to the edge portion E (step S1501: YES), the deletion unit 149 sets a processing target region centered on the pixel to be processed for the edge image (step S1502).

[0099] Next, the deletion unit 149 extracts the maximum value and the minimum value of pixel values of 0 or more from the processing target region set in step S1502 in the sparse depth image (step S1503).

[0100] Next, the deletion unit 149 determines whether the difference between the maximum value and the minimum value extracted in step S1503 is greater than a predetermined threshold (step S1504).

[0101] In step S1504, if it is determined that the difference between the maximum value and the minimum value is not greater than the predetermined threshold (step S1504: NO), the deletion unit 149 proceeds to step S1506.

[0102] In step S1504, when it is determined that the difference between the maximum value and the minimum value is greater than a predetermined threshold (step S1504: YES), the pixel to be processed in the edge image is determined as the "distance difference edge E'" (step S1505). Thereafter, the deletion unit 149 proceeds to step S1506.

[0103] In step S1506, the deletion unit 149 determines whether the processing for all the pixels of the edge image has been completed (step S1506).

[0104] In step S1506, when it is determined that the processing for all the pixels of the edge image has not been completed (step S1506: NO), the deletion unit 149 returns the processing to step S1501.

[0105] In step S1506, when it is determined that the processing for all the pixels of the edge image has been completed (step S1506: YES), the deletion unit 149 ends the series of processes shown in FIG. 15.

[0106] (A modified example of the configurations of the light projecting unit 110, the luminance light receiving unit 120, and the ToF light receiving unit 130) FIG. 16 is a diagram showing a modified example of the configurations of the light projecting unit 110, the luminance light receiving unit 120, and the ToF light receiving unit 130 included in the distance measuring device 100 according to an embodiment.

[0107] As shown in FIG. 16, in the distance measuring device 100 according to an embodiment, the luminance light receiving unit 120 and the set of the light projecting unit 110 and the ToF light receiving unit 130 may be provided separately from each other. In this case, as shown in FIG. 16, the set of the light projecting unit 110 and the ToF light receiving unit 130 may be configured to be changeable in position and orientation so as to be able to measure distances in both the "ToF light receiving area 1" and the "ToF light receiving area 2" shown in FIG. 16.

[0108] As another modification, in the distance measurement device 100 according to one embodiment, the combination of the light projection unit 110 and the ToF light reception unit 130 and the luminance light reception unit 120 may be configured such that by swapping their positions, both the "ToF light reception area 1" and the "ToF light reception area 2" shown in FIG. 16 can be used for distance measurement.

[0109] As another modification, in the distance measurement device 100 according to one embodiment, a plurality of combinations of the light projection unit 110 and the ToF light reception unit 130 may be provided. In this case, the plurality of combinations of the light projection unit 110 and the ToF light reception unit 130 may perform distance measurement (light projection and light reception) simultaneously.

[0110] (A modification of the functional configuration of the distance measurement control unit 140) FIG. 17 is a diagram showing a modification of the functional configuration of the distance measurement control unit 140 according to one embodiment. As shown in FIG. 17, the distance measurement control unit 140 may be implemented as a single unit without having the light projection unit 110 and the light reception units 120 and 130.

[0111] In this case, as shown in FIG. 17, the distance measurement control unit 140 includes an RGB image input unit 151 to which an RGB image is input from an external luminance light reception unit, a ToF image input unit 152 to which a distance image is input from an external ToF light reception unit, and an RGB-ToF alignment unit 153 that calculates various parameters (for example, camera parameters, rotation and translation parameters between RGB and ToF, etc.) for generating a sparse depth image based on the RGB image and the distance image.

[0112] FIG. 18 is a diagram showing an example in which the distance measurement system according to another embodiment of the present disclosure is applied to a portable information terminal. FIG. 19 is a diagram showing an example in which the distance measurement system according to another embodiment of the present disclosure is applied to an autonomous driving system of a moving body.

[0113] An application example in which the distance measuring device 100 as a distance measuring system is used in various detection systems will be described with reference to FIGS. 18 to 19. The detection systems in these application examples have respective functional blocks described later in addition to the distance measuring device 100. In FIGS. 18 to 19, functional blocks such as a determination unit included in the detection system are described outside the detection system for convenience of drawing. Each of the various detection systems shown in FIGS. 18 to 19 has a control unit that receives information from the distance measuring device 100 and controls the various detection systems based on the information from the distance measuring device 100.

[0114] FIG. 18 is an example of a shape measurement system as a detection system, and shows an application example in which the distance measuring device 100 is used for user authentication of an electronic device.

[0115] The portable information terminal 60X, which is an electronic device, has a user authentication function. The authentication function may be realized by dedicated hardware, or may be realized by a CPU (Central Processing Unit) that controls the portable information terminal 60X executing a program such as a ROM (Read Only Memory).

[0116] When performing user authentication, light is projected from the light projecting unit of the distance measuring device 100 mounted on the portable information terminal 60X toward the user 61X who uses the portable information terminal 60X.

[0117] The light reflected by the user 61X and its surroundings is received by the light receiving unit of the distance measuring device 100, and image data is generated (imaging is performed) by the image processing unit 62X. The determination unit 63X determines the degree of coincidence between the image information of the user 61X imaged by the distance measuring device 100 and the pre-registered user information, and determines whether the user is a registered user.

[0118] Specifically, the shape (contour and unevenness) of the face, ears, head, etc. of the user 61X can be measured and used as user information.

[0119] In the application example of FIG. 18, regarding the detection of the user 61X by the distance measuring device 100, distance measurement can be performed with high accuracy similar to the distance measuring device 100, and an improvement in recognition accuracy can be achieved.

[0120] FIG. 18 shows an example in which the distance measuring device 100 is mounted on the portable information terminal 60X. However, user authentication using the distance measuring device 100 can also be used for stationary personal computers, OA devices such as printers, and building security systems.

[0121] In terms of functions, it can be used not only for personal authentication functions but also for scanning three-dimensional shapes such as faces. Also in this case, high-precision scanning can be realized by mounting the distance measuring device 100.

[0122] FIG. 19 shows an application example in which the distance measuring device 100 is used in an autonomous driving system in a moving body, which is an example of a detection system.

[0123] In the application example of FIG. 19, the distance measuring device 100 is used for sensing an object outside the moving body 70X. The moving body 70X is an autonomous driving type moving body that can automatically travel while recognizing the external situation.

[0124] The distance measuring device 100 is mounted on the moving body 70X, and the distance measuring device 100 irradiates light toward the traveling direction of the moving body 70X and its peripheral area. In the room 71X, which is the moving area of the moving body 70X, a desk 72X is installed in the traveling direction of the moving body 70X.

[0125] Of the light projected from the light projecting unit of the distance measuring device 100 mounted on the moving body 70X, the light reflected by the desk 72X and its surroundings is received by the light receiving unit of the distance measuring device 100, and the electrical signal subjected to photoelectric conversion is sent to the signal processing unit 73X.

[0126] Based on the electrical signal and the like sent from the light receiving unit, the signal processing unit 73X calculates information regarding the layout of the room 71X, such as the distance to the desk 72X, the position of the desk 72X, and the peripheral situation other than the desk 72X.

[0127] Based on this calculated information, the determination unit 74X determines the moving path, moving speed, etc. of the moving body 70X, and based on the determination result of the determination unit 74X, the driving control unit 75X controls the running of the moving body 70X (such as the operation of the motor which is the driving source).

[0128] In the application example of FIG. 19, regarding the layout detection of the interior 71X by the distance measuring device 100, distance measurement can be performed with high accuracy similar to the distance measuring device 100, and an improvement in the accuracy of the autonomous driving of the moving body 70X can be realized.

[0129] FIG. 19 shows an example in which the distance measuring device 100 is mounted on the autonomous driving type moving body 70X that travels in the interior 71X, but it can also be applied to an autonomous driving type vehicle (so-called self-driving vehicle) that travels outdoors.

[0130] In addition, it is also possible to apply it to a driving support system in a moving body such as an automobile driven by a driver, rather than an autonomous driving type. In this case, the distance measuring device 100 is used to detect the surrounding situation of the moving body, and the driving of the driver can be supported according to the detected surrounding situation.

[0131] In addition to the above, the distance measuring device 100 may be applied to an article inspection system in a factory or the like. Specifically, based on the information acquired by the distance measuring device 100, the determination unit of the article inspection system determines the state of each article.

[0132] Also, the distance measuring device 100 may be applied to the operation control of a movable device.

[0133] The articulated arm as a movable device has a plurality of arms connected by bendable joints and is provided with a hand portion at the tip. The articulated arm is used, for example, on an assembly line in a factory, and grips an object with the hand portion when inspecting, transporting, or assembling the object.

[0134] The distance measuring device 100 detects an object and its surrounding area, and the determination unit of the movable device determines various information about the object, such as the distance to the object, the shape of the object, the position of the object, and the positional relationship between multiple objects when they exist, based on the information acquired by the distance measuring device 100. Then, based on the determination result of the determination unit, the drive control unit controls the articulated arm movement.

[0135] Also, the distance measuring device 100 may be applied to a driving assistance system in a moving body such as an automobile.

[0136] The distance measuring device 100 mounted inside the automobile detects the driver who drives the automobile and the surrounding area thereof. The determination unit of the driving assistance system determines information such as the face (expression) and posture of the driver based on the information acquired by the distance measuring device 100. Then, based on the determination result of the determination unit, the control unit performs appropriate driving assistance according to the situation of the driver.

[0137] The shape measurement system, the moving body, the article inspection system, the movable device, and the driving assistance system are all examples of the detection system. In the distance measuring device 100 in the present embodiment, the spatial resolution of the distance image can be improved. Therefore, in the detection system to which the distance measuring device 100 is applied, high-precision detection can be realized.

[0138] As described above, the preferred embodiments of the present invention have been described in detail, but the present invention is not limited to these embodiments, and various modifications or changes are possible within the scope of the gist of the present invention described in the claims.

Explanation of Reference Numerals

[0139] 100 Distance measuring device (distance measuring system) 110 Light projecting unit 111 Light source 112 Light source drive unit 120 Luminance light receiving unit (first light receiving unit) 121 Image sensor 122 ADC 130 ToF light receiving unit (second light receiving unit) 131, 131a, 131b Image sensors 132 ADC 140 Distance measurement control unit (image generation device, image generation unit) 141 RGB control unit 142 RGB image storage unit 143 ToF control unit 144 ToF image storage unit 145 Sparse depth creation unit (conversion unit) 146 Sparse filter processing unit (filter processing unit) 147 Densification unit 148 Output unit 149 Deletion unit 151 RGB image input unit 152 ToF image input unit 153 RGB - ToF alignment unit 201 CPU 202 ROM 203 RAM 204 SSD 205 Light source drive circuit 206, 207 Sensor I / F (acquisition unit) 208 Input / output I / F E Edge part E' Distance difference edge

Prior art documents

Patent documents

[0140]

Patent Document 1

Claims

1. An acquisition unit that acquires a luminance image with a first resolution and a first distance image with a second resolution lower than the first resolution; A conversion unit that converts the first distance image into a sparse distance image corresponding to the luminance image; A filter processing unit that performs a filter process on the distance value of a first pixel in the sparse distance image based on distance values included in a predetermined region around the first pixel; A densification unit that generates a second distance image with a third resolution higher than the second resolution by performing a predetermined densification process on the sparse distance image after the filter process; An image generation device comprising the above components.

2. The filter processing unit updates the distance value of the first pixel using a plurality of pixel values greater than 0 included in the predetermined region. The image generation device according to Claim 1.

3. The filter process is a median filter process or a joint bilateral filter process. The image generation device according to Claim 1.

4. The first distance image includes a plurality of distance images, and the plurality of distance images each have an overlapping region that overlaps with each other. The image generation device according to any one of Claims 1 to 3.

5. The filter processing unit performs the filter process on the combined plurality of distance images. The image generation device according to Claim 4.

6. Before the filter process is performed, an edge detection unit that detects an edge portion from the luminance image and deletes the distance value of a pixel corresponding to the edge portion of the sparse distance image is provided. The image generation device according to any one of Claims 1 to 3.

7. The deletion unit detects, as the edge portion, a pixel in which the difference between the maximum value and the minimum value of distance values included in a predetermined region around a pixel corresponding to a luminance edge detected from the luminance image is equal to or greater than a certain value. The image generation device according to Claim 6.

8. A computer An acquisition unit that acquires a luminance image with a first resolution and a first distance image with a second resolution lower than the first resolution; A conversion unit that converts the first distance image into a sparse distance image corresponding to the luminance image; A filter processing unit that performs a filter process on the distance value of a first pixel in the sparse distance image based on distance values included in a predetermined region around the first pixel; and A densification unit that generates a second distance image with a third resolution higher than the second resolution by performing a predetermined densification process on the sparse distance image after the filter process A program that functions as. **Claim 9** A light projecting unit that emits irradiation light, A first light receiving unit that outputs a luminance image with a first resolution, A second light receiving unit that receives the reflected light of the irradiation light reflected by the object and outputs a first distance image with a second resolution lower than the first resolution, An image generation unit Comprising The image generation unit An acquisition unit that acquires the luminance image and the first distance image, A conversion unit that converts the first distance image into a sparse distance image corresponding to the luminance image, A filter processing unit that performs a filter process on the distance value of the first pixel in the sparse distance image based on the distance values included in a predetermined region around the first pixel, A densification unit that generates a second distance image with a third resolution higher than the second resolution by performing a predetermined densification process on the sparse distance image after the filter process A distance measurement system having.

Images