Point cloud data processing device, point cloud data processing method, and program

Abstract

A point cloud data processing device (100) comprises: a first ranging unit (130) that captures a first ranging image (11) using unstructured illumination (111); a second ranging unit (140) that captures a second ranging image (12) using structured illumination (112); an image processing unit (150) that generates a difference image (13) indicating the difference between the first ranging image (11) and the second ranging image (12); a point cloud converting unit (160) that, on the basis of the first ranging image (11) captured by the first ranging unit (130), performs point-cloud conversion of distance values included in the first ranging image (11) to generate first point cloud data (21); a correction data generating unit (170) that generates correction point cloud data on the basis of the difference image (13) generated by the image processing unit (150); and a point cloud correcting unit (180) that generates blended point cloud data (22) by correcting the first point cloud data (21) on the basis of the generated correction point cloud data.

Description

Point cloud data processing device, point cloud data processing method, and program

[0001] The present disclosure relates to a point cloud data processing device, a point cloud data processing method, and a program.

[0002] ToF (Time of Flight) is a technology used to measure the distance to an object. Specifically, ToF is a technology that exposes an image sensor to light emitted from a light source and reflected by the object over multiple exposure periods with different timings, and calculates the distance to the object based on the amount of delay of the reflected light, which is calculated from the signal ratio corresponding to each exposure period.

[0003] Here, the distance measurement value obtained by ToF includes an error caused by the light emitted from the light source being reflected multiple times by the object and then exposed as reflected light, i.e., an error caused by multipath.

[0004] Patent Document 1 discloses a technique for suppressing errors contained in a ranging image that shows the results of ranging captured by a ToF camera.

[0005] U.S. Patent No. 9,329,035

[0006] An object of the present disclosure is to provide a point cloud data processing device and the like that can suppress errors caused by multipath included in point cloud data.

[0007] One aspect of a point cloud data processing device according to the present disclosure includes a first ranging unit that captures a first ranging image using unstructured illumination, a second ranging unit that captures a second ranging image using structured illumination, an image processing unit that generates a difference image showing the difference between the first ranging image and the second ranging image based on the first ranging image captured by the first ranging unit and the second ranging image captured by the second ranging unit, a point cloud conversion unit that converts the distance values ​​contained in the first ranging image into a point cloud based on the first ranging image captured by the first ranging unit and generates first point cloud data, a correction data generation unit that generates corrected point cloud data based on the difference image generated by the image processing unit, and a point cloud correction unit that generates blended point cloud data by correcting the first point cloud data based on the corrected point cloud data generated by the correction data generation unit.

[0008] Furthermore, one aspect of the point cloud data processing method according to the present disclosure includes a first ranging step of capturing a first ranging image using unstructured illumination, a second ranging step of capturing a second ranging image using structured illumination, an image processing step of generating a difference image indicating a difference between the first ranging image and the second ranging image based on the first ranging image captured by the first ranging step and the second ranging image captured by the second ranging step, a point cloud conversion step of converting distance values ​​contained in the first ranging image into point cloud data based on the first ranging image captured by the first ranging step to generate first point cloud data, a correction data generation step of generating corrected point cloud data based on the difference image generated by the image processing step, and a point cloud correction step of generating blended point cloud data by correcting the first point cloud data based on the corrected point cloud data generated by the correction data generation step.

[0009] The present disclosure can be realized not only as the point cloud data processing method, but also as a program for causing a computer to execute the point cloud data processing method, and further as a computer-readable recording medium storing the program.

[0010] The point cloud data processing device, point cloud data processing method, and program according to the present disclosure can suppress errors caused by multipath included in point cloud data.

[0011] FIG. 1 is a schematic diagram illustrating an overview of a point cloud data processing device according to an embodiment. FIG. 2 is a diagram illustrating multipath in ToF. FIG. 3 is a block diagram illustrating a functional configuration of a point cloud data processing device according to an embodiment. FIG. 4 is a diagram illustrating an example of a first ranging image according to an embodiment. FIG. 5 is a diagram illustrating an example of a second ranging image according to an embodiment. FIG. 6 is a diagram illustrating point cloud conversion. FIG. 7 is a flowchart illustrating the operation of a point cloud data processing device according to an embodiment. FIG. 8 is a diagram illustrating an example of a reduced image according to an embodiment. FIG. 9 is a diagram illustrating an example of an interpolated image according to an embodiment. FIG. 10 is a diagram illustrating an example of a difference image according to an embodiment. FIG. 11 is a diagram illustrating an example of a corrected image according to an embodiment. FIG. 12 is a diagram illustrating first point cloud data and blended point cloud data according to an embodiment. FIG. 13 is a flowchart illustrating the operation of a first modified example of a point cloud data processing device according to an embodiment. FIG. 14 is a diagram illustrating mask processing according to an embodiment. FIG. 15 is a flowchart illustrating the operation of a second modified example of a point cloud data processing device according to an embodiment. FIG. 16 is a diagram illustrating dot validation processing according to an embodiment. FIG. 17 is a diagram illustrating the time-series filtering process according to the embodiment.

[0012] (Embodiments) Hereinafter, embodiments of a point cloud data processing device, a point cloud data processing method, and a program according to the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a preferred specific example of the present disclosure. The numerical values, components, component placement and connection configurations, steps, step order, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims will be described as optional components constituting a preferred embodiment.

[0013] It should be noted that the drawings are schematic diagrams and are not necessarily strict illustrations. In addition, in the drawings, substantially the same components are denoted by the same reference numerals, and overlapping descriptions may be omitted or simplified.

[0014] Furthermore, unless otherwise specified, ordinal numbers such as "first" and "second" in this disclosure do not refer to the number or order of components, etc., but are used for the purpose of avoiding confusion and distinguishing between components, etc. of the same type.

[0015] Furthermore, unless otherwise specified, the term "frequency" in this disclosure refers to "spatial frequency." Spatial frequency is a physical quantity that is the reciprocal of distance.

[0016] [Overview] First, an overview of a point cloud data processing device according to the present embodiment will be described. Fig. 1 is a schematic diagram illustrating an overview of a point cloud data processing device 100 according to the embodiment. Fig. 2 is a diagram illustrating multipath in ToF.

[0017] 1, the point cloud data processing device 100 includes a light emitting unit 110 and an exposure unit 120. The point cloud data processing device 100 irradiates a space with irradiation light L1 from the light emitting unit 110. The irradiation light L1 irradiated into the space is reflected by an object 200 in the space, and is exposed to the exposure unit 120 as reflected light L2. At this time, the exposure unit 120 performs exposure in a plurality of exposure periods with different timings relative to the irradiation timing of the irradiation light.

[0018] The point cloud data processing device 100 can measure the distance to the object 200 by calculating the time required from when the light emitting unit 110 emits the irradiation light L1 until when the exposure unit 120 exposes the reflected light L2 based on the signal amount corresponding to each of the multiple exposure periods.

[0019] It is known that in distance measurement using ToF, the measurement result contains errors due to multipath. As shown in FIG. 2, when the light emitted by the light-emitting unit 110 travels along a direct path, where the light directly hits and reflects off the target object 200, errors from the true distance are unlikely to occur. However, in reality, multipath may exist, where part of the light emitted by the light-emitting unit 110 hits and reflects off another object 201, and then hits and reflects off the target object 200. In the example shown in FIG. 2, there is one multipath, but multiple multipaths may exist.

[0020] 2, light traveling through multiple paths returns to the point cloud data processing device 100 later than light traveling through the direct path. Therefore, the distance calculated corresponding to the travel time of light traveling through multiple paths is longer than the true distance.

[0021] For example, when measuring the distance to a wall surface using the point cloud data processing device 100, the path of light that is reflected off the floor surface, then reflected off the wall surface, and returns to the point cloud data processing device 100 is an example of multipath.

[0022] In reality, the reflected light from the target object 200 that returns to the point cloud data processing device 100 includes light that has traveled the direct path and light that has traveled the multipath. Therefore, the distance calculated by the point cloud data processing device 100 includes an error due to the multipath that makes the distance longer than the true distance.

[0023] The point cloud data processing device 100 according to the embodiment can suppress errors caused by multipath included in point cloud data indicating the results of distance measurements using ToF.

[0024] [Configuration] The following describes the configuration of the point cloud data processing device 100 according to the embodiment. Fig. 3 is a block diagram showing the functional configuration of the point cloud data processing device 100 according to the embodiment.

[0025] The point cloud data processing device 100 generates point cloud data based on a ranging image captured by ToF, the ranging image including distance values. The distance values ​​included in the ranging image indicate the distance to the target object 200. Point cloud data is data representing a collection of points in space. For example, point cloud data is three-dimensional point cloud data representing a collection of multiple points having three-dimensional coordinates. Furthermore, the point cloud data processing device 100 can suppress errors caused by multipath included in the point cloud data by performing information processing for correcting the point cloud data. Specifically, the point cloud data processing device 100 includes a light emitting unit 110, an exposure unit 120, a first ranging unit 130, a second ranging unit 140, an image processing unit 150, a point cloud conversion unit 160, a correction data generation unit 170, a point cloud correction unit 180, and a mask processing unit 190.

[0026] The light-emitting unit 110 is a light irradiator that irradiates irradiation light L1 into a space in accordance with an input light-emitting control signal. The light-emitting unit 110 irradiates pulsed light as irradiation light, for example, in accordance with the timing indicated by a light-emitting control pulse included in the input light-emitting control signal. The light-emitting unit 110 repeatedly irradiates pulsed light at a predetermined cycle in accordance with the light-emitting control signal.

[0027] The light-emitting unit 110 is composed of a light irradiator including, for example, a light-emitting diode or laser element that emits infrared light. The light-emitting unit 110 is realized by a light-emitting element that has a relatively fast response speed and is capable of high-speed blinking, and an optical system that receives light from the light-emitting element and controls the light distribution from the light-emitting element. The light-emitting unit 110 includes multiple light irradiators that irradiate a space with light of spatially different patterns. The spatially different patterns mean that the intensity distribution of the light irradiated by the light-emitting unit 110 into a space is different. Specifically, the light-emitting unit 110 includes unstructured lighting 111 and structured lighting 112.

[0028] The unstructured illumination 111 emits light with a uniform intensity distribution. The unstructured illumination 111 is also called a flood light source or a patternless light source. More specifically, the unstructured illumination 111 emits light with a uniform intensity distribution across the viewing angle of the camera (in other words, the light receiving range of the exposure unit 120).

[0029] The structured illumination 112 emits light with a non-uniform intensity distribution. An example of a non-uniform intensity distribution is a periodic dot pattern. The structured illumination 112 is also called a dot light source or a pattern light source. More specifically, the structured illumination 112 emits light in which a specific pattern, such as a dot array, exists relative to the viewing angle of the camera. In this embodiment, the structured illumination 112 emits light with a luminous intensity distribution in which areas of high light intensity are a collection of dots arranged two-dimensionally in the vertical and horizontal directions.

[0030] The exposure unit 120 is realized by, for example, a pixel array. The exposure unit 120 performs exposure in a plurality of exposure periods with different timings relative to the irradiation timing of the irradiation light emitted by the unstructured illumination 111 or the structured illumination 112.

[0031] The exposure unit 120 includes a plurality of pixels 121 arranged two-dimensionally. Each of the plurality of pixels 121 generates a signal based on incident light. For example, each of the plurality of pixels 121 converts the incident light into a signal charge and generates a signal based on the converted signal charge. The incident light received by the pixels 121 is light reflected from the object 200 and background light from the surroundings. The plurality of pixels 121 have substantially the same configuration. The plurality of pixels 121 can be exposed multiple times within one frame at the timing indicated by an exposure control signal output from the first ranging unit 130 or the second ranging unit 140. Here, a frame is a period from when the plurality of pixels 121 are exposed to when signals are read out from the plurality of pixels 121, which constitutes one cycle.

[0032] The first ranging unit 130, the second ranging unit 140, the image processing unit 150, the point cloud conversion unit 160, the correction data generation unit 170, the point cloud correction unit 180, and the mask processing unit 190 described below are functional processing units realized by, for example, a memory that stores a program and a processor that executes the program. Note that although they are shown separately in the block diagram shown in FIG. 3 , at least one of the first ranging unit 130, the second ranging unit 140, the image processing unit 150, the point cloud conversion unit 160, the correction data generation unit 170, the point cloud correction unit 180, and the mask processing unit 190 may be configured with the same memory and processor. Furthermore, at least one of the first ranging unit 130, the second ranging unit 140, the image processing unit 150, the point cloud conversion unit 160, the correction data generation unit 170, the point cloud correction unit 180, and the mask processing unit 190 may be realized by a dedicated logic circuit that performs predetermined processing.

[0033] The first ranging unit 130 performs information processing to output a first ranging image. Specifically, the first ranging unit 130 outputs a light emission control signal for driving the unstructured lighting 111. The first ranging unit 130 also outputs an exposure control signal for driving the exposure unit 120. The exposure control signal is a signal for causing each of the multiple pixels 121 to perform exposure at a timing corresponding to the timing of irradiation with the illumination light L1 by the unstructured lighting 111. The first ranging unit 130 also performs signal processing on the signals output by each of the multiple pixels 121 to calculate the distance to the object 200 and generate a first ranging image. In other words, the first ranging unit 130 captures the first ranging image by performing this information processing.

[0034] The first ranging unit 130 is realized by, for example, a drive circuit that outputs a light emission control signal for driving the unstructured lighting 111 and an exposure control signal for driving the exposure unit 120, and an information processing circuit that executes information processing for generating a ranging image based on a signal acquired from the exposure unit 120. Figure 4 is a diagram showing an example of the first ranging image 11 according to the embodiment.

[0035] As shown in FIG. 4 , the first distance measurement image 11 is a collection of distance values ​​to the object 200 calculated based on signals acquired from each of the plurality of pixels 121 .

[0036] Since the first ranging image 11 is a ranging image captured using unstructured illumination 111, distance values ​​can be calculated for all pixels 121 included in the exposure unit 120.

[0037] The second distance measuring unit 140 performs information processing to output a second distance measuring image. Specifically, the second distance measuring unit 140 outputs a light emission control signal for driving the structured illumination 112. The second distance measuring unit 140 also outputs an exposure control signal for driving the exposure unit 120. The exposure control signal is a signal for exposing each of the multiple pixels 121 at a timing corresponding to the timing of irradiation with the illumination light L1 by the structured illumination 112. The second distance measuring unit 140 also performs signal processing on the signals output by each of the multiple pixels 121 to calculate the distance to the object 200 and generate a second distance measuring image. In other words, the second distance measuring unit 140 captures the second distance measuring image by performing this information processing.

[0038] The second ranging unit 140 is realized by, for example, a drive circuit that outputs a light emission control signal for driving the structured illumination 112 and an exposure control signal for driving the exposure unit 120, and an information processing circuit that executes information processing for generating a ranging image based on a signal acquired from the exposure unit 120. Fig. 5 is a diagram showing an example of the second ranging image 12 according to the embodiment.

[0039] 5, the areas indicated by white circles represent distance values ​​calculated based on signals output by pixels 121 that received a sufficient amount of light to measure the distance. Hereinafter, these pixels 121 will be referred to as valid pixels. On the other hand, the areas indicated by dots (i.e., areas other than the white circles) in FIG. 5 represent pixels 121 that did not receive a sufficient amount of light to measure the distance.

[0040] The second ranging image 12 is a ranging image captured using structured illumination 112, and therefore reflects the intensity distribution of the light emitted by structured illumination 112. That is, in the second ranging image 12 shown in Fig. 5, the effective pixels are arranged as a set of points arranged two-dimensionally in the vertical and horizontal directions.

[0041] Here, a brief description will be given of an example of a procedure in which the first distance measuring unit 130 and the second distance measuring unit 140 perform signal processing on the signals output by each of the multiple pixels 121 to calculate the distance to the target object 200. Here, an overview of the procedure, also known as the pulse ToF method, will be given.

[0042] The drive circuit causes the light-emitting unit 110 to emit an irradiation pulse with a pulse width P1, exposing the pixel 121 to two exposure periods, exposure period A0 and exposure period A1, each having the length of the pulse width P1. The exposure period A0 starts simultaneously with the start of the light-emission control pulse. The exposure period A1 starts with a delay of the pulse width P1 from the start of the light-emission control pulse, that is, simultaneously with the end of the exposure period A0.

[0043] The reflected light from the object 200 is received by the pixel 121 with a delay of time Δt from the irradiated light, depending on the distance to the object 200. Therefore, when the object 200 is present within a distance measurement range where distance measurement is possible, the signal charges generated by the reflected light are divided into signal charges generated in the exposure period A0 and signal charges generated in the exposure period A1, depending on the distance to the object 200.

[0044] When the signal value corresponding to the exposure period A0 is A0 and the signal value corresponding to the exposure period A1 is A1, the delay amount, Δt, is calculated by the following equation (1).

[0045]

[0046] Furthermore, if the distance to the object 200 is D and the speed of light is c, the irradiated light travels a distance twice the distance D in time Δt, so the distance D is calculated using the following equation (2).

[0047]

[0048] By performing this procedure for each of the plurality of pixels 121, a distance measurement image including the distance value D can be captured.

[0049] Although the example in which the exposure period A0 starts simultaneously with the start of the light emission control pulse has been described, it may also start a predetermined time after the start of the light emission control pulse, thereby shifting the upper and lower limits of the distance measurement range in the direction of longer distances.

[0050] The procedure for calculating the distance to the object 200 is not limited to the above procedure. For example, the distance to the object 200 may be calculated using a CW (Continuous Wave)-ToF method.

[0051] The image processing unit 150 performs information processing to generate a difference image showing the difference between the first distance measurement image 11 and the second distance measurement image 12, based on the first distance measurement image 11 captured by the first distance measurement unit 130 and the second distance measurement image 12 captured by the second distance measurement unit 140. The image processing unit 150 also performs information processing on the generated difference image to generate a corrected image by performing processing to reduce frequency components equal to or higher than a predetermined first frequency. In other words, the image processing unit 150 acquires the captured distance measurement image and performs information processing on the acquired distance measurement image. The predetermined first frequency will be described in detail below.

[0052] The point cloud conversion unit 160 performs information processing to convert the distance values ​​included in the first ranging image 11 into a point cloud based on the first ranging image 11 captured by the first ranging unit 130, and generate first point cloud data. The point cloud conversion is a process of generating point cloud data by converting each of the multiple distance values ​​included in the ranging image into the coordinates of a point in the xyz coordinate system based on the ranging image and internal camera parameters.

[0053] Here, a procedure for converting distance values ​​included in a distance measurement image into a point cloud will be briefly described. Fig. 6 is a diagram for explaining point cloud conversion.

[0054] First, using the uv coordinate axes shown in Figure 6, with the center of the ranging image as the origin, the position of the pixel to be subjected to point cloud transformation in the ranging image is calculated. That is, u is the horizontal image coordinate, and v is the vertical image coordinate. If the half angle of view corresponding to the distance from the optical axis of each pixel is θ, and the rotation angle around the optical axis of each pixel position is φ, then the camera internal parameters are I, and θ and φ depend on u, v, and I. In other words, they can be expressed as θ(u, v, I) and φ(u, v, I). The camera internal parameters include the position (height) at which the ranging camera is installed, the performance of the ranging camera, etc.

[0055] Based on the calculated θ and φ, conversion coefficients are calculated. The conversion coefficients are coefficients corresponding to the x-axis, y-axis, and z-axis directions used for point cloud conversion.

[0056] Here, assuming that the conversion coefficient in the x-axis direction is paramX, the conversion coefficient in the y-axis direction is paramY, and the conversion coefficient in the z-axis direction is paramZ, the conversion coefficients are calculated by the following formula.

[0057] (paramX, paramY, paramZ) = (sin (θ) cos (φ), sin (θ) sin (φ), cos (θ))

[0058] Based on the calculated conversion coefficients and the distance values ​​"depth" of the pixels of the distance measurement image, the 3D coordinates (x, y, z) of the pixels are calculated using the following equations.

[0059] (x, y, z) = (paramX*depth, paramY*depth, paramZ*depth)

[0060] By performing this calculation for each of several pixels, the distance values ​​can be converted into 3D coordinates.

[0061] The point cloud conversion unit 160 converts the distance values ​​included in the first ranging image 11 into a point cloud using the above procedure, and generates first point cloud data.

[0062] The correction data generation unit 170 executes information processing for generating corrected point cloud data based on the difference image generated by the image processing unit 150. Specifically, the correction data generation unit 170 converts the distance values ​​included in the difference image, the corrected image, or the like into a point cloud to generate the corrected point cloud data.

[0063] The point cloud correction unit 180 executes information processing to generate blended point cloud data by correcting the first point cloud data based on the corrected point cloud data generated by the correction data generation unit 170. For example, the point cloud correction unit 180 generates blended point cloud data by combining the first point cloud data and the corrected point cloud data. The blended point cloud data is point cloud data in which errors caused by multipath during ranging have been suppressed by information processing by the point cloud data processing device 100.

[0064] The mask processing unit 190 executes information processing to extract frequency components equal to or higher than a predetermined second frequency contained in the first ranging image 11 and generate a high-frequency image. The predetermined second frequency will be described in detail later. The point cloud data processing device 100 does not necessarily have to include the mask processing unit 190.

[0065] [Operation] Next, a description will be given of the operation of the point cloud data processing device 100 according to the embodiment. Fig. 7 is a flowchart illustrating the operation of the point cloud data processing device 100 according to the embodiment.

[0066] In general, DotToF (ToF using structured illumination) can suppress errors caused by multipath compared to FloodToF (ToF using unstructured illumination).

[0067] On the other hand, DotToF has a problem in that the spatial resolution is significantly lower than that of FloodToF.

[0068] Therefore, the point cloud data processing device 100 according to the embodiment can suppress errors caused by multipath while maintaining spatial resolution by performing a process of appropriately combining DotToF and FloodToF. More specifically, the point cloud data processing device 100 according to the embodiment can generate point cloud data in which errors caused by multipath included in the captured ranging image are suppressed by capturing ranging images using different light sources including a Dot light source and a Flood light source and correcting the point cloud data obtained based on the ranging images.

[0069] First, the first distance measuring unit 130 captures the first distance measuring image 11 using the unstructured illumination 111 (S11). The first distance measuring unit 130 outputs the captured first distance measuring image 11 to the image processing unit 150 and the point cloud conversion unit 160.

[0070] Next, the second distance measuring unit 140 captures the second distance measuring image 12 using the structured illumination 112 (S12). The second distance measuring unit 140 outputs the captured second distance measuring image 12 to the image processing unit 150.

[0071] When the second ranging image 12 is acquired, the image processing unit 150 performs an interpolation process on the second ranging image 12 (S13). The interpolation process is a process for interpolating invalid distance values ​​included in the second ranging image 12 to acquire an image with the same number of pixels as the first ranging image 11. The interpolation process includes a process for extracting valid pixels included in the second ranging image 12. The interpolation process refers to, for example, a process for generating an image with the same number of pixels as the first ranging image 11 based on the second ranging image 12 by performing predetermined information processing on the valid pixels included in the second ranging image 12.

[0072] Specifically, the image processing unit 150 first generates a reduced image by extracting valid pixels included in the second ranging image 12. Fig. 8 is a diagram showing an example of the reduced image 12a according to the embodiment. As shown in Fig. 8, the reduced image 12a is an image consisting only of valid pixels included in the second ranging image 12. In other words, the reduced image 12a has a smaller number of pixels than the second ranging image 12. The number of pixels in the reduced image 12a may be the same as the number of points with high light intensity in the intensity distribution of the illumination light emitted by the structured illumination 112, for example.

[0073] Next, the image processing unit 150 generates an interpolated image based on the reduced image 12a. Specifically, the image processing unit 150 performs information processing on the reduced image 12a, as described below, to add pixels indicating appropriate distance values ​​between pixels in the reduced image 12a, thereby generating the interpolated image. Fig. 9 is a diagram showing an example of an interpolated image 12b according to the embodiment. As shown in Fig. 9, the interpolated image 12b has the same number of pixels as the first ranging image 11.

[0074] The method for generating the interpolated image 12b based on the reduced image 12a is, for example, the nearest neighbor interpolation method, but may also be the bilinear interpolation method, the bicubic interpolation method, etc. Furthermore, the generation of the interpolated image 12b may be performed by upsampling the reduced image 12a by applying a joint-bilateral filter, a guided filter, etc.

[0075] Once the interpolated image 12b is generated, the image processing unit 150 generates a difference image based on the first ranging image 11 and the interpolated image 12b (S14). Fig. 10 is a diagram showing an example of the difference image 13 according to the embodiment. The difference image 13 is an image showing the difference between the distance values ​​of the first ranging image 11 and the distance values ​​of the interpolated image 12b. For example, in the difference image 13 shown in Fig. 10, the darker the color, the greater the difference between the distance values ​​included in the first ranging image 11 and the distance values ​​included in the interpolated image 12b.

[0076] After the difference image 13 is generated, the image processing unit 150 executes a process of reducing high-frequency components of the difference image 13 (S15). The image processing unit 150 executes a process of reducing high-frequency components of the difference image 13 to generate a corrected image. Fig. 11 is a diagram showing an example of the corrected image 14 according to the embodiment.

[0077] The corrected image 14 shown in Fig. 11 is an image obtained by performing a process to reduce high-frequency components of the difference image 13 shown in Fig. 10. As shown in Fig. 11, the corrected image 14 has reduced high-frequency components and components that show spatially steep changes compared to the difference image 13 shown in Fig. 10.

[0078] Here, the difference image 13 contains error components mainly due to two causes.

[0079] The first cause is an error component resulting from low accuracy of the first ranging image 11. The decrease in accuracy of the first ranging image 11 occurs due to multipath and low-reflectivity objects, etc. This error component exhibits a relatively low frequency component.

[0080] The second cause is an error component resulting from the low resolution of the second ranging image 12, in other words, the small number of effective pixels. The low resolution of the second ranging image 12 makes it impossible to detect steep depth shapes, i.e., shapes in which the distance values ​​of adjacent pixels change significantly, such as those of thin or pointed objects, resulting in an error component. This error component exhibits relatively high frequency components.

[0081] Therefore, in the processing of step S15, the image processing unit 150 executes processing to reduce the high frequency components of the difference image 13, thereby making it possible to extract the error components caused by the first cause, i.e., the error components that indicate relatively low frequency components.

[0082] For example, the image processing unit 150 performs a frequency conversion on the difference image 13 to reduce high frequency components contained in the difference image 13 .

[0083] Specifically, the image processing unit 150 performs frequency transformation processing on the difference image 13 by discrete Fourier transform. The frequency transformation processing may be performed by discrete cosine transform, discrete wavelet transform, discrete Hadamard transform, or discrete Haar transform. The method of frequency transformation processing is not limited to the above, and is not limited here.

[0084] Next, the image processing unit 150 performs processing to reduce frequency components equal to or higher than a predetermined first frequency. The amount of light received by the structured illumination 112 is distributed according to a pattern frequency fp on an image sensor having a pixel pitch frequency fs. The predetermined first frequency is designated as fth, where fth is a value that satisfies, for example, 0.1 fp<fth<fs. Note that fth may be a value that satisfies another range, such as 0.2 fp<fth<1.2 fs.

[0085] The process of reducing the predetermined first frequency component may be, for example, a process of setting it to zero on the spectrum, or a process of reducing it based on a specific function shape such as Gaussian.

[0086] Finally, the image processing unit 150 performs an inverse frequency transform on this to generate the corrected image 14 .

[0087] This processing may be performed on all pixels included in the difference image 13 or on some of the pixels included in the difference image 13 .

[0088] Alternatively, for example, the image processing unit 150 applies a spatial filter to the difference image 13 to perform processing to reduce high frequency components.

[0089] Specifically, the image processing unit 150 applies a spatial filter designed to reduce frequency components equal to or higher than a predetermined first frequency to the difference image 13, thereby performing a process of reducing high frequency components.

[0090] The predetermined first frequency is denoted by fth, where fth is a value that satisfies, for example, 0.1 fp<fth<fs. Note that fth may be a value that satisfies another range, for example, 0.2 fp<fth<1.2 fs.

[0091] The filter size of the spatial filter is not particularly limited as long as at least one of the vertical and horizontal dimensions is greater than one pixel. For example, the vertical dimension may be one pixel and the horizontal dimension may be three pixels, or the vertical dimension may be three pixels and the horizontal dimension may be one pixel. Furthermore, the direction in which the filter extends may be either up, down, left, or right.

[0092] Alternatively, the filtering process may be performed separately vertically and horizontally. For example, a spatial filter having a size of 1 pixel by 3 pixels and a spatial filter having a size of 3 pixels by 1 pixel may be applied separately, and the applied results may be combined by some operation such as addition or multiplication to obtain a final result.

[0093] The amount of light received by the structured illumination 112 is distributed on an image sensor having a pixel pitch frequency fs according to a pattern frequency fp. A predetermined first frequency fth is a value that satisfies, for example, 0.1 fp<fth<fs. Note that fth may be a value that satisfies another range, such as 0.2 fp<fth<1.2 fs.

[0094] This processing may be performed on all pixels included in the difference image 13 or on some of the pixels included in the difference image 13 .

[0095] Alternatively, for example, the image processing unit 150 executes a process of reducing high frequency components contained in the difference image 13 using AI (Artificial Intelligence).

[0096] For example, the image processing unit 150 generates a corrected image 14 by inputting the difference image 13 to an image generation AI that has previously learned the error component caused by the first cause and the error component caused by the second cause.

[0097] The image generation AI is an AI model trained to be able to extract the error component caused by the first cause from the error components contained in the difference image 13. The image generation AI model is realized by, for example, the ControlNet or pix2pix algorithm, but may also be a model based on other algorithms.

[0098] This processing may be performed on all pixels included in the difference image 13 or on some of the pixels included in the difference image 13 .

[0099] Next, the correction data generation unit 170 generates corrected point cloud data based on the corrected image 14 (S16). Specifically, the correction data generation unit 170 generates the corrected point cloud data by performing point cloud conversion on the distance values ​​included in the corrected image 14.

[0100] The point cloud conversion unit 160 generates first point cloud data based on the first ranging image 11 (S17). Specifically, the point cloud conversion unit 160 converts the distance values ​​included in the first ranging image 11 into a point cloud to generate first point cloud data 21.

[0101] Finally, the point cloud correction unit 180 generates blended point cloud data by correcting the first point cloud data 21 using the corrected point cloud data (S18). In other words, the point cloud correction unit 180 generates blended point cloud data by executing a point cloud correction process. The point cloud correction process is a process for generating blended point cloud data based on the first point cloud data 21 obtained by point cloud conversion of the first ranging image 11 and the corrected point cloud data obtained by point cloud conversion of the corrected image 14. The point cloud correction process is executed, for example, by calculating the difference between the first point cloud data 21 and the corrected point cloud data and reflecting the difference in the first point cloud data 21. Note that the point cloud correction process may be performed by other methods.

[0102] 12 is a diagram illustrating the first point cloud data 21 and the blended point cloud data 22 according to the embodiment. Fig. 12 shows data obtained by cutting the first point cloud data 21 and the blended point cloud data 22, which are generated based on the first ranging image 11 and the second ranging image 12 captured from the upper right of an L-shaped portion, at a cross section where the L-shaped portion appears. The L-shaped portion is prone to generating multipath during distance measurement.

[0103] The ideal value 31 indicated by the L-shaped dashed line in Fig. 12 represents the actual shape captured. In other words, in Fig. 12, the closer the positions of the points constituting the point cloud data are to the ideal value 31, the more accurately the actual shape is reproduced and the less error there is in the point cloud data.

[0104] Hereinafter, in the ideal L-shape value 31 shown in FIG. 12, the vertical broken lines will be referred to as wall surfaces, and the horizontal broken lines will be referred to as floor surfaces.

[0105] 12, the first point cloud data 21 is a point cloud generated based on distance values ​​that are longer than the ideal values ​​31 and the actual distance values ​​indicated by the ideal values ​​31, both for the wall surfaces and the floor surfaces. In other words, the first point cloud data 21 includes errors caused by multipath.

[0106] On the other hand, the blended point cloud data 22 is point cloud data made up of points closer to the ideal value 31 than the first point cloud data 21. That is, the blended point cloud data 22 has distortion due to multipath interference (MPI), that is, errors caused by multipath, suppressed.

[0107] 12 , for example, as can be seen in the point cloud representing the floor surface among the first point cloud data 21, the first point cloud data 21 is a curved point cloud with a large curve, whereas the blended point cloud data 22 is a substantially linear point cloud with a relatively small curve. In other words, the blended point cloud data 22 is point cloud data with a higher degree of reproducibility of linear shapes compared to the first point cloud data 21.

[0108] Therefore, as shown in FIG. 12, the blend point cloud data 22 reflects the actual shape with higher accuracy than the first point cloud data 21.

[0109] According to the above-described operation, the point cloud data processing device 100 of the embodiment generates corrected point cloud data based on the first ranging image 11 captured using unstructured illumination 111 and the second ranging image 12 captured using structured illumination 112, and corrects the first point cloud data 21 obtained from the first ranging image 11 based on the corrected point cloud data to generate blended point cloud data 22.

[0110] Therefore, according to the point cloud data processing device 100, it is possible to generate point cloud data in which errors caused by multipath during distance measurement are suppressed.

[0111] [Variation 1] The point cloud data processing device 100 according to the above embodiment can generate blended point cloud data 22 in which errors caused by multipath are suppressed by using the corrected point cloud data to correct the first point cloud data 21. At this time, the point cloud data processing device 100 generates the corrected point cloud data by performing processing to reduce high-frequency components in order to suppress errors caused by multipath.

[0112] Here, the structured illumination 112 emits light having a light distribution in which areas of high light intensity are a collection of points arranged two-dimensionally in the vertical and horizontal directions, and therefore the distance measurement image generated by the structured illumination 112 has frequency components equal to or lower than a certain spatial frequency Fp defined by this light distribution. Therefore, it is not possible to obtain distance values ​​from the second distance measurement image 12 at locations having spatial frequency components higher than Fp. In other words, an object having a shape with a steep change in depth, such as the legs of a chair, cannot be detected from the second distance measurement image 12.

[0113] Therefore, shapes with spatial frequencies equal to or higher than Fp are not correctly reflected in the corrected image 14. Therefore, the corrected image 14 may be subjected to a masking process to protect high frequency components equal to or higher than a predetermined second frequency using a frequency filter that extracts high frequency components equal to or higher than a predetermined second frequency.

[0114] According to the point cloud data processing device 100 of Modification 1, shapes indicating high frequency components not reflected in the second ranging image 12 can be reflected in the corrected point cloud data. Modification 1 of the point cloud data processing device 100 will now be described. FIG. 13 is a flowchart illustrating the operation of Modification 1 of the point cloud data processing device 100 according to the embodiment. FIG. 14 is a diagram illustrating mask processing according to the embodiment. (a) of FIG. 14 is a diagram illustrating an example of a first ranging image 11 on which mask processing according to the embodiment is performed. (b) of FIG. 14 is a diagram illustrating an example of a high frequency image 15 for performing mask processing according to the embodiment. (c) of FIG. 14 is a diagram illustrating an example of a masking image 16 for performing mask processing according to the embodiment.

[0115] As shown in FIG. 13, first, in the process of step S15 in FIG. 7, the image processing unit 150 executes a process of reducing high frequency components of the difference image 13 that are equal to or higher than a predetermined first frequency, thereby generating a corrected image 14.

[0116] The mask processing unit 190 extracts frequency components equal to or higher than a predetermined second frequency from the first ranging image 11 to generate a high frequency image 15 (S21). The mask processing unit 190 extracts frequency components equal to or higher than the predetermined second frequency by executing processing that has the effect of a frequency filter. Examples of processing that has the effect of a frequency filter include frequency conversion processing, processing that applies a spatial filter, and processing that uses AI. Note that processing that has the effect of a frequency filter is not limited to the above, and is not limited here.

[0117] The predetermined second frequency is, for example, Fp, but may also be 1.2Fp. Specifically, as shown in Figures 14(a) and 14(b), the mask processing unit 190 generates a high-frequency image 15 by executing a process for extracting frequency components equal to or higher than the predetermined second frequency from the first ranging image 11. In the high-frequency image 15 shown in Figure 14(b), the areas indicated by solid lines are extracted areas, and the areas indicated by diagonal lines are areas not extracted. In other words, thin-shaped areas such as the legs of the chair are extracted from the first ranging image 11 captured of the chair shown in Figure 14(a), and the high-frequency image 15 is generated.

[0118] Once the high-frequency image 15 is generated, the image processing unit 150 adjusts the corrected image 14 based on the high-frequency image 15 generated by the mask processing unit 190 (S22). Specifically, as shown in (c) of FIG. 14, the mask processing unit 190 generates a masking image 16 from the high-frequency image 15. The image processing unit 150 adjusts the corrected image 14 by combining the masking image 16 and the corrected image 14 based on mask coefficients. In the masking image 16 shown in (c) of FIG. 14, the areas indicated by solid lines are areas where mask processing is performed, and the areas painted white are areas where mask processing is not performed. That is, in the masking image 16 shown in (c) of FIG. 14, mask coefficients are calculated for the areas indicated by solid lines, and mask coefficients are not calculated or invalid mask coefficients are calculated for the areas painted white.

[0119] The mask processing unit 190 calculates mask coefficients for the portions of the masking image 16 that are indicated to be subjected to mask processing, and the image processing unit 150 adjusts the corrected image 14 based on the calculated mask coefficients. For example, the mask processing unit 190 calculates the mask coefficients by normalizing pixel values ​​of the high frequency image 15 to obtain coefficients. The calculation of the mask coefficients may be performed by performing a calculation process such as multiplication or exponentiation on an image obtained by calculating the ratio between the high frequency image 15 and the first ranging image 11, and then normalizing the pixel values ​​to obtain coefficients. The method of calculating the mask coefficients is not particularly limited.

[0120] The image processing unit 150 performs a process of masking frequency components equal to or higher than a predetermined second frequency on the corrected image 14 based on the calculated mask coefficient, thereby generating an adjusted corrected image 14. The masking process may be performed by multiplying the corrected image 14 by the calculated mask coefficient. Alternatively, the masking process may be performed by calculating an average value of values ​​obtained by multiplying the corrected image 14 by the mask coefficient and replacing the image with the average value. Alternatively, the masking process may be performed by performing pixel labeling on the portion indicated by the masking image 16 using AI or connected component calculation, calculating an average value of the correction amount of the distance value for each label value, and replacing the image with the average value. The method of the masking process is not particularly limited.

[0121] Once the corrected image 14 has been adjusted, the process proceeds to step S16 in FIG. 7, where the correction data generator 170 generates corrected point cloud data based on the corrected image 14 adjusted by the image processor 150.

[0122] As described above, the point cloud data processing device 100 according to variant example 1 can reflect shapes indicating high frequency components that are not reflected in the second ranging image 12 in the corrected point cloud data, thereby improving the accuracy of the blended point cloud data 22.

[0123] [Variation 2] In the point cloud data processing device 100 according to the above embodiment, the imaging range of the first ranging image 11 is the same as the imaging range of the second ranging image 12. In other words, the ranging range of the distance measurement performed by the first ranging unit 130 using the unstructured illumination 111 is the same as the ranging range of the distance measurement performed by the second ranging unit 140 using the structured illumination 112.

[0124] However, in general, the structured illumination 112 can have a stronger light output than the unstructured illumination 111. In other words, the structured illumination 112 can measure distances farther than the unstructured illumination 111. Therefore, the second ranging image 12 can include distance values ​​farther than the first ranging image 11.

[0125] In this way, when the ToF ranging range using structured illumination 112 is wider than the ToF ranging range using unstructured illumination 111, the effective point cloud data calculated based on the second ranging image 12 captured using structured illumination 112 may be spatially combined with the blended point cloud data 22.

[0126] The point cloud data processing device 100 according to Modification 2 can generate point cloud data covering a wider spatial range by spatially combining the blended point cloud data 22 with effective point cloud data in a range farther than the range of the blended point cloud data 22. Modification 2 of the point cloud data processing device 100 will now be described. FIG. 15 is a flowchart illustrating the operation of Modification 2 of the point cloud data processing device 100 according to the embodiment. FIG. 16 is a diagram illustrating the dot validation process according to the embodiment. (a) of FIG. 16 is a diagram illustrating the difference in ranging range between the first ranging image 11 and the second ranging image 12. (b) of FIG. 16 is a diagram illustrating the range of the blended point cloud data 22. (c) of FIG. 16 is a schematic diagram of effective blended point cloud data 24.

[0127] 15, first, in the process of step S18 in Fig. 7, the point cloud correction unit 180 generates blended point cloud data. Also, the point cloud conversion unit 160 acquires the second ranging image 12 (not shown in Figs. 3 and 15).

[0128] Next, the point cloud conversion unit 160 identifies an area of ​​the imaging range of the second ranging image 12 that is not included in the imaging range of the first ranging image 11 (S31). In the first ranging image 11 shown in (a) of FIG. 16 , the dotted areas are areas that are farther than the ToF ranging range of the unstructured illumination 111 and therefore did not receive a sufficient amount of light for calculating the distance, while the white areas are areas that received a sufficient amount of light for calculating the distance. Therefore, the operation of generating the blended point cloud data 22 described using FIG. 7 is performed in the white areas of the first ranging image 11 shown in (a) of FIG. 16 . Furthermore, in the second ranging image 12 shown in (a) of FIG. 16 , black circles indicate valid pixels used to generate the blended point cloud data 22, and white circles indicate valid pixels not used to generate the blended point cloud data 22. As shown in (b) of Figure 16, the blend point cloud data 22 is generated based on the part shown in white in the first ranging image 11 shown in (a) of Figure 16 and the effective pixels shown in white circles in the second ranging image 12 shown in (a) of Figure 16.

[0129] The point cloud conversion unit 160 identifies the area of ​​the imaging range of the second ranging image 12 that is not included in the imaging range of the first ranging image 11, that is, the effective pixels indicated by white circles in the second ranging image 12 shown in (a) of Figure 16.

[0130] Next, the point cloud conversion unit 160 generates effective point cloud data 23 by performing point cloud conversion on the distance values ​​included in the specified region of the second ranging image 12 (S32). That is, the point cloud conversion unit 160 generates effective point cloud data 23 by performing point cloud conversion on the distance values ​​based on the signals output by the effective pixels indicated by white circles in the second ranging image 12 shown in Fig. 16(a). As shown in Fig. 16(c), the effective point cloud data 23 is point cloud data for a region different from that of the blended point cloud data 22.

[0131] Finally, the point cloud correction unit 180 spatially combines the generated blend point cloud data 22 and the effective point cloud data 23 generated by the point cloud conversion unit 160 to generate effective blend point cloud data 24 (S33). Spatial combination means generating point cloud data for one space by adding together point cloud data for two or more different spaces. As shown in (c) of Fig. 16, the effective blend point cloud data 24 is point cloud data obtained by spatially combining the blend point cloud data 22 and the effective point cloud data 23.

[0132] According to this operation, the point cloud data processing device 100 of Modification 2 can generate point cloud data covering a wider spatial range by spatially combining the blended point cloud data 22 with the effective point cloud data 23, which is located in a range farther than the range of the blended point cloud data 22. Furthermore, since the effective point cloud data 23 is point cloud data based on the second ranging image 12, it is less affected by error components caused by multipath. Therefore, the point cloud data processing device 100 can generate effective blended point cloud data 24 in which errors caused by multipath are suppressed.

[0133] [Variation 3] The timing at which the first ranging image 11 is captured may differ from the timing at which the second ranging image 12 is captured, depending on the configuration of the point cloud data processing device 100. For example, if the point cloud data processing device 100 includes a single exposure unit 120 and the first ranging unit 130 and the second ranging rod 140 alternately capture images using the single exposure unit 120, the first ranging image 11 and the second ranging image 12 will not be captured at the same time.

[0134] In this way, when the first ranging image 11 and the second ranging image 12 are captured at different times, for example, if the camera (particularly the exposure unit 120) or the object 200 moves between the timing of capturing the first ranging image 11 and the timing of capturing the second ranging image 12, there is a risk that false components due to movement will occur in the difference image 13.

[0135] The point cloud data processing device 100 according to the third modification of the embodiment can eliminate false components resulting from movement by performing time-series filtering on the difference image 13 in the current frame using the difference image 13 in the past frame.

[0136] The following describes the time series filtering process executed by the point cloud data processing device 100 according to the third modification of the embodiment. Fig. 17 is a diagram for explaining the time series filtering process according to the embodiment.

[0137] FIG. 17 shows an example of time-series filtering performed when the camera, which was stationary at time 1, rotates counterclockwise at time 2 in a case where the first ranging unit 130 and the second ranging unit 140 alternately capture images using one exposure unit 120. Time 1 shown in FIG. 17 is the time when the second ranging image 12 is captured by the second ranging unit 140. Time 2 shown in FIG. 17 is the frame immediately after time 1, when the third ranging image 11b is captured by the first ranging unit 130. The image processing unit 150 generates a first difference image 13a based on the first ranging image 11a and the second ranging image 12 captured by the first ranging unit 130 in the frame immediately before time 1. The image processing unit 150 generates a second difference image 13b based on the third ranging image 11b and the second ranging image 12.

[0138] Comparing the first ranging image 11a and the third ranging image 11b shown in Figure 17, the camera moved counterclockwise at time 2, so the position of the object in the space indicated by the white rectangle is shifted to the right in the third ranging image 11b. Comparing the first difference image 13a and the second difference image 13b shown in Figure 17, the second difference image 13b contains, in the area surrounded by the dashed line, a component caused by the position of the object shifting to the right, i.e., a false component caused by movement. Because false components caused by movement like this are caused by movement of the camera or object between frames, they tend to occur in a narrow spatial range and have a large rate of change in pixel values ​​between frames.

[0139] The image processing unit 150 generates a third difference image 13c by performing time-series filtering based on the first difference image 13a and the second difference image 13b. In the third difference image 13c shown in FIG. 17, the false components due to movement contained in the second difference image 13b have been eliminated in the area surrounded by the dashed line. The time-series filtering is a process for eliminating the false components due to movement contained in the difference image 13b of the current frame using the difference image 13a of a past frame. The procedure of the time-series filtering will be described below.

[0140] First, the image processing unit 150 calculates the similarity between the first difference image 13a and the second difference image 13b. The similarity indicates the degree of closeness between the pixel values ​​of the pixels in the first difference image 13a and the second difference image 13b. In other words, the smaller the difference in pixel values, the higher the calculated similarity. Therefore, the smaller the influence of the false component due to motion on a pixel, the higher the calculated similarity, and the larger the influence of the false component due to motion on a pixel, the lower the calculated similarity. The similarity is calculated for each pixel.

[0141] Specifically, the image processing unit 150 calculates the similarity by performing addition, subtraction, multiplication, division, etc. on pixel values ​​of pixels at the same position in the first difference image 13 a and the second difference image 13 b. The similarity may be calculated by AI processing. Note that the method of calculating the similarity is not limited to the above, and is not limited here.

[0142] Next, the image processing unit 150 corrects the second difference image 13b based on the calculated similarity to generate a third difference image 13c. For example, the image processing unit 150 corrects the second difference image 13b by coefficient processing based on the similarity. That is, the similarity is calculated as a coefficient ranging from 0 to 1, and the image processing unit 150 corrects the second difference image 13b by multiplying each pixel of the second difference image 13b by this coefficient. Note that the correction processing method is not limited to the above, and is not limited here.

[0143] Although the above-described time-series filtering process is performed using a plurality of difference images generated based on a plurality of different distance measurement images captured by the first distance measurement unit 130, the time-series filtering process may be performed using a plurality of difference images generated based on a plurality of different distance measurement images captured by the second distance measurement unit 140. In other words, the time-series filtering process may be performed based on a difference image generated based on the second distance measurement image 12 and a difference image generated based on a fourth distance measurement image captured by the second distance measurement unit 140 after the second distance measurement image 12.

[0144] More specifically, the second distance measurement unit 140 captures a second distance measurement image and then captures a fourth distance measurement image using structured illumination. The image processing unit 150 generates a fourth difference image based on the second distance measurement image and a fifth difference image based on the fourth distance measurement image. The image processing unit 150 generates a sixth difference image by performing time-series filtering on the fourth difference image and the fifth difference image. That is, the image processing unit 150 calculates the similarity between the fourth difference image and the fifth difference image and corrects the fifth difference image using the calculated similarity to generate the sixth difference image. The correction data generation unit 170 generates corrected point cloud data based on the sixth difference image generated by the image processing unit 150. The fourth difference image corresponds to the first difference image 13a described above, the fifth difference image corresponds to the second difference image 13b described above, and the sixth difference image corresponds to the third difference image 13c described above.

[0145] In this way, the image processing unit 150 uses multiple difference images generated based on the results of distance measurements performed at multiple times, and performs time-series filtering on the difference images to correct them, thereby eliminating false components resulting from movement.

[0146] Once the difference image in which the false components derived from movement have been eliminated has been generated, the process proceeds to step S15 in FIG. 7, where the image processing unit 150 executes a process of reducing frequency components equal to or higher than a predetermined first frequency to generate a corrected image based on the difference image in which the false components derived from movement have been eliminated, and then the correction data generation unit 170 generates corrected point cloud data based on the corrected image.

[0147] Note that the time-series filtering process may be applied to the corrected image rather than the difference image. More specifically, the image processing unit 150 generates a first corrected image by processing the first difference image 13a to reduce frequency components equal to or higher than a predetermined first frequency, and generates a second corrected image by processing the second difference image 13b to reduce frequency components equal to or higher than the predetermined first frequency. Next, the image processing unit 150 generates a third corrected image by performing the time-series filtering process described above based on the generated first and second corrected images. Finally, the correction data generation unit 170 may generate corrected point cloud data based on the third corrected image. This allows the point cloud data processing device 100 to generate a third corrected image in which false components due to movement that may be included in the second corrected image have been eliminated.

[0148] In the time series filtering process described above, the similarity is calculated by comparing with the most recent past frame, but it may also be calculated by comparing with an even earlier frame, or by comparing with two or more past frames.

[0149] The time-series filtering process may be performed on all pixels included in the second difference image 13b, or may be performed on some of the pixels included in the first difference image 13b.

[0150] 17 described above, the camera was stationary at time 1, but the camera may be moving at time 1. This is because the image processing unit 150 performs time-series filtering on the first difference image 13a obtained at time 1 using difference images older than time 1, thereby making it possible to eliminate spurious components derived from movement contained in the first difference image 13a.

[0151] In this way, the point cloud data processing device 100 according to the third modification can eliminate false components resulting from movement that may occur in the difference image 13 when the first ranging image 11 and the second ranging image 12 are captured at different times. Therefore, the point cloud data processing device 100 according to the third modification can generate blended point cloud data 22 in which errors resulting from the movement of the camera or the object 200 are suppressed.

[0152] [Effects, etc.] Techniques derived from the contents disclosed in this specification are, for example, the following techniques: Hereinafter, the techniques derived from the contents disclosed in this specification will be described together with the effects, etc. obtained by the techniques.

[0153] The technique 1 includes a first distance measurement unit 130 that captures a first distance measurement image 11 using unstructured illumination 111, a second distance measurement unit 140 that captures a second distance measurement image 12 using structured illumination 112, an image processing unit 150 that generates a difference image 13 that shows the difference between the first distance measurement image 11 and the second distance measurement image 12 based on the first distance measurement image 11 captured by the first distance measurement unit 130 and the second distance measurement image 12 captured by the second distance measurement unit 140, and an image processing unit 150 that generates a difference image 13 that shows the difference between the first distance measurement image 11 and the second distance measurement image 12 based on the first distance measurement image 11 captured by the first distance measurement unit 130. a point cloud conversion unit 160 that converts the distance values ​​included in the first ranging image 11 into a point cloud based on the difference image 13 generated by the image processing unit 150, and generates first point cloud data 21; a correction data generation unit 170 that generates corrected point cloud data based on the difference image 13 generated by the image processing unit 150; and a point cloud correction unit 180 that corrects the first point cloud data 21 based on the corrected point cloud data generated by the correction data generation unit 170, thereby generating blended point cloud data 22.

[0154] Such a point cloud data processing device 100 generates corrected point cloud data based on a first ranging image 11 captured using unstructured illumination 111 and a second ranging image 12 captured using structured illumination 112, and corrects first point cloud data 21 obtained from the first ranging image 11 based on the corrected point cloud data to generate blended point cloud data 22. Such a point cloud data processing device 100 can generate point cloud data in which errors caused by multipath during ranging are suppressed.

[0155] Technology 2 is the point cloud data processing device 100 of Technology 1, in which the image processing unit 150 generates a reduced image 12a by extracting effective pixels contained in the second ranging image 12, and generates a difference image 13 based on the first ranging image 11 and the reduced image 12a.

[0156] The point cloud data processing device 100 can generate corrected point cloud data using distance values ​​calculated based on signals output from effective pixels that receive a sufficient amount of light for distance measurement. Therefore, the point cloud data processing device 100 can generate point cloud data in which errors caused by multipath during distance measurement are further reduced.

[0157] Technique 3 is the point cloud data processing device 100 of Technique 1 or 2, in which the image processing unit 150 performs processing on the generated difference image 13 to reduce frequency components equal to or higher than a predetermined first frequency, and the correction data generation unit 170 generates corrected point cloud data based on the processed difference image. The processed difference image is an example of the corrected image 14 in the above embodiment.

[0158] Such a point cloud data processing device 100 can extract error components that indicate relatively low frequency components, which are generated by multipath and low-reflectivity objects, from the error components contained in the difference image 13, and reflect these in the corrected point cloud data.

[0159] Technique 4 is the point cloud data processing device 100 of Technique 3, in which the image processing unit 150 performs frequency conversion on the difference image 13 and executes processing.

[0160] Such a point cloud data processing device 100 can actually frequency convert the difference image 13, thereby performing processing to reduce high-frequency components while handling frequency data, thereby improving the accuracy of the processing.

[0161] Technique 5 is the point cloud data processing device 100 of Technique 3, in which the image processing unit 150 executes processing by applying a spatial filter to the difference image 13.

[0162] Such a point cloud data processing device 100 can apply a spatial filter to the difference image 13 to perform a process of reducing high-frequency components without converting the difference image 13 into frequency data, thereby reducing the amount of calculation required for the process.

[0163] Technique 6 is the point cloud data processing device 100 of Technique 3, in which the image processing unit 150 executes processing using AI.

[0164] Such a point cloud data processing device 100 can execute a process to reduce high-frequency components by inputting the difference image 13 into a pre-trained AI, thereby making the process easier.

[0165] Technique 7 is the point cloud data processing device 100 of any of Techniques 3 to 6, further comprising a mask processing unit 190 that extracts frequency components equal to or higher than a predetermined second frequency contained in the first ranging image 11 to generate a high frequency image 15, the image processing unit 150 adjusts the processed difference image based on the high frequency image 15 generated by the mask processing unit 190, and the correction data generation unit 170 generates corrected point cloud data based on the processed difference image adjusted by the image processing unit 150.

[0166] Such a point cloud data processing device 100 can reflect shapes that indicate high frequency components that are not reflected in the second ranging image 12 in the corrected point cloud data, thereby improving the accuracy of the blended point cloud data 22.

[0167] Technology 8 is a point cloud data processing device 100 according to any of Technologies 1 to 7, further comprising: a point cloud conversion unit 160 that identifies an area of ​​the imaging range of the second ranging image 12 that is not included in the imaging range of the first ranging image 11; and generates effective point cloud data 23 by converting the ranging values ​​included in the identified area of ​​the second ranging image 12 into a point cloud; and a point cloud correction unit 180 that spatially combines the generated blended point cloud data 22 and the effective point cloud data 23 generated by the point cloud conversion unit 160 to generate effective blended point cloud data 24.

[0168] The point cloud data processing device 100 can generate effective blended point cloud data 24 covering a wider spatial range by spatially combining the blended point cloud data 22 with effective point cloud data 23 covering a range farther than the range of the blended point cloud data 22. Furthermore, since the effective point cloud data 23 is point cloud data based on the second ranging image 12, it is less affected by error components caused by multipath, and therefore the point cloud data processing device 100 can generate effective blended point cloud data 24 in which errors caused by multipath are suppressed.

[0169] Technology 9 is the point cloud data processing device 100 of Technology 1 or 2, further comprising: a first ranging unit 130, after capturing a first ranging image 11a, capturing a third ranging image 11b using unstructured illumination; an image processing unit 150 generating a first difference image 13a based on the first ranging image 11a and a second difference image 13b based on the third ranging image 11b, and generating a third difference image 13c based on the first difference image 13a and the second difference image 13b; and a correction data generation unit 170 generating corrected point cloud data based on the third difference image 13c generated by the image processing unit 150.

[0170] Such a point cloud data processing device 100 can eliminate false components resulting from movement that may occur in the difference image 13 when the first ranging image 11 and the second ranging image 12 are captured at different times. Therefore, such a point cloud data processing device 100 can generate point cloud data in which errors resulting from the movement of the camera or the object 200 are suppressed.

[0171] Technique 10 is the point cloud data processing device 100 of Technique 9, in which the image processing unit 150 performs processing on the generated third difference image 13c to reduce frequency components equal to or higher than a predetermined first frequency, and the correction data generation unit 170 generates corrected point cloud data based on the processed third difference image 13c.

[0172] Such a point cloud data processing device 100 can extract error components that indicate relatively low frequency components generated by multipath and low-reflectivity objects, etc., from the error components contained in the third difference image 13c, and reflect them in the corrected point cloud data.

[0173] Technology 11 is the point cloud data processing device 100 of Technology 1 or 2, further comprising: a second ranging unit 140, after capturing the second ranging image, capturing a fourth ranging image using structured illumination; an image processing unit 150 generating a fourth difference image based on the second ranging image and a fifth difference image based on the fourth ranging image, and generating a sixth difference image based on the fourth difference image and the fifth difference image; and a correction data generation unit 170 generating corrected point cloud data based on the sixth difference image generated by the image processing unit 150.

[0174] Such a point cloud data processing device 100 can eliminate false components resulting from movement that may occur in the difference image 13 when the first ranging image 11 and the second ranging image 12 are captured at different times. Therefore, such a point cloud data processing device 100 can generate point cloud data in which errors resulting from the movement of the camera or the object 200 are suppressed.

[0175] Technique 12 is the point cloud data processing device 100 of Technique 11, in which the image processing unit 150 performs processing on the generated sixth difference image to reduce frequency components equal to or higher than a predetermined first frequency, and the correction data generation unit 170 generates corrected point cloud data based on the processed sixth difference image.

[0176] Such a point cloud data processing device 100 can extract error components that indicate relatively low frequency components generated by multipath and low-reflectivity objects, etc., from the error components contained in the sixth differential image, and reflect them in the corrected point cloud data.

[0177] Technique 13 is a point cloud data processing method including: a first ranging step of capturing a first ranging image 11 using unstructured illumination 111; a second ranging step of capturing a second ranging image 12 using structured illumination 112; an image processing step of generating a difference image 13 indicating a difference between the first ranging image 11 and the second ranging image 12 based on the first ranging image 11 captured by the first ranging step and the second ranging image 12 captured by the second ranging step; a point cloud conversion step of converting distance values ​​included in the first ranging image 11 into a point cloud based on the first ranging image 11 captured by the first ranging step to generate first point cloud data 21; a correction data generation step of generating corrected point cloud data based on the difference image 13 generated by the image processing step; and a point cloud correction step of generating blended point cloud data 22 by correcting the first point cloud data 21 based on the corrected point cloud data generated by the correction data generation step.

[0178] This point cloud data processing method generates corrected point cloud data based on a first ranging image 11 captured using unstructured illumination 111 and a second ranging image 12 captured using structured illumination 112, and corrects first point cloud data 21 obtained from the first ranging image 11 based on the corrected point cloud data to generate blended point cloud data 22. This point cloud data processing method makes it possible to generate point cloud data in which errors caused by multipath during ranging are suppressed.

[0179] Technique 14 is a program for causing a computer to execute the point cloud data processing method of Technique 13.

[0180] Such a program can assist in generating point cloud data in which errors caused by multipath during distance measurement are suppressed.

[0181] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0182] For example, in the above embodiment, in the process of step S16 in Fig. 7 , the correction data generation unit 170 generates the corrected point cloud data based on the corrected image 14, but the corrected point cloud data may also be generated based on the difference image 13. In other words, the process of step S15 in Fig. 7 may be omitted. Such a point cloud data processing device 100 can generate corrected point cloud data that includes error components caused by the low resolution of the second ranging image 12, in addition to error components caused by multipath.

[0183] Furthermore, for example, in the above embodiment, the point cloud data processing device is realized by a single device, but the point cloud data processing device may be realized by multiple devices. In this case, when the point cloud data processing device is realized by multiple devices, the components (especially functional components) of the point cloud data processing device may be allocated in any way among the multiple devices.

[0184] For example, in the above embodiment, the point cloud data processing device includes a ranging device equipped with a light emitting unit and an exposure unit, but the point cloud data processing device may also be a device that does not include a ranging device, but instead accepts input of a captured ranging image and performs information processing on the input ranging image.

[0185] Furthermore, the method of communication between the devices in the above-described embodiment is not particularly limited. In addition, a relay device (such as a gateway device) (not shown) may be involved in the communication between the devices.

[0186] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0187] In the above-described embodiments, each component may be realized by executing a software program suitable for that component, or by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0188] Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0189] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0190] The present disclosure may also be realized as a point cloud data processing method executed by a computer such as the point cloud data processing device of the above-described embodiment, or as a program for causing a computer to execute the point cloud data processing method (in other words, a computer program product).The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

[0191] In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art would think of, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope of this disclosure.

[0192] A point cloud data processing device according to the present disclosure can be used in an information processing device that generates point cloud data indicating the shape of an object based on image data of an object that actually exists.

[0193] 11, 11a First distance measurement image 11b Third distance measurement image 12 Second distance measurement image 12a Reduced image 12b Interpolated image 13 Difference image 13a First difference image 13b Second difference image 13c Third difference image 14 Corrected image 15 High frequency image 16 Masking image 21 First point cloud data 22 Blended point cloud data 23 Effective point cloud data 24 Effective blended point cloud data 31 Ideal value 100 Point cloud data processing device 110 Light emitting unit 111 Unstructured illumination 112 Structured illumination 120 Exposure unit 121 Pixel 130 First distance measurement unit 140 Second distance measurement unit 150 Image processing unit 160 Point cloud conversion unit 170 Corrected data generation unit 180 Point cloud correction unit 190 Mask processing unit 200 Object 201 Another object

Claims

1. A point cloud data processing apparatus comprising: a first distance measurement unit that captures a first distance measurement image using unstructured illumination; a second distance measurement unit that captures a second distance measurement image using structured illumination; an image processing unit that generates a difference image indicating a difference between the first distance measurement image and the second distance measurement image based on the first distance measurement image captured by the first distance measurement unit and the second distance measurement image captured by the second distance measurement unit; a point cloud conversion unit that converts distance values included in the first distance measurement image into point cloud data to generate first point cloud data based on the first distance measurement image captured by the first distance measurement unit; a correction data generation unit that generates corrected point cloud data based on the difference image generated by the image processing unit; and a point cloud correction unit that generates blended point cloud data by correcting the first point cloud data based on the corrected point cloud data generated by the correction data generation unit.

2. The point cloud data processing apparatus according to claim 1, wherein the image processing unit generates a reduced image by extracting valid pixels included in the second distance measurement image, and generates the difference image based on the first distance measurement image and the reduced image.

3. The point cloud data processing apparatus according to claim 1 or 2, wherein the image processing unit performs a process of reducing frequency components of a predetermined first frequency or higher on the generated difference image, and the correction data generation unit generates the corrected point cloud data based on the difference image on which the process has been performed.

4. The point cloud data processing apparatus according to claim 3, wherein the image processing unit performs the process by performing a frequency conversion on the difference image.

5. The point cloud data processing apparatus according to claim 3, wherein the image processing unit performs the process by applying a spatial filter to the difference image.

6. The point cloud data processing apparatus according to claim 3, wherein the image processing unit performs the process using AI (Artificial Intelligence).

7. Further, a mask processing unit is provided that extracts frequency components of a predetermined second frequency or higher included in the first distance measurement image to generate a high-frequency image, and the image processing unit adjusts the difference image on which the processing has been executed based on the high-frequency image generated by the mask processing unit. The correction data generation unit generates the corrected point cloud data based on the difference image on which the processing has been executed and adjusted by the image processing unit. The point cloud data processing apparatus according to any one of claims 3 to 6.

8. Further, the point cloud conversion unit identifies an area in which an area not included in the imaging range of the first distance measurement image is imaged within the imaging range of the second distance measurement image, and performs point cloud conversion on the distance measurement values included in the identified area of the second distance measurement image to generate valid point cloud data. The point cloud correction unit generates valid blended point cloud data by spatially synthesizing the generated blended point cloud data and the valid point cloud data generated by the point cloud conversion unit. The point cloud data processing apparatus according to any one of claims 1 to 7.

9. Further, after capturing the first distance measurement image, the first distance measurement unit captures a third distance measurement image using unstructured illumination. The image processing unit generates a first difference image based on the first distance measurement image and a second difference image based on the third distance measurement image, and generates a third difference image based on the first difference image and the second difference image. The correction data generation unit generates the corrected point cloud data based on the third difference image generated by the image processing unit. The point cloud data processing apparatus according to claim 1 or 2.

10. The image processing unit executes a process of reducing frequency components of a predetermined first frequency or higher with respect to the generated third difference image, and the correction data generation unit generates the corrected point cloud data based on the third difference image on which the process has been executed. The point cloud data processing apparatus according to claim 9.

11. Further, after imaging the second distance measurement image, the second distance measurement unit images a fourth distance measurement image using structured illumination. The image processing unit generates a fourth difference image based on the second distance measurement image and a fifth difference image based on the fourth distance measurement image, and generates a sixth difference image based on the fourth difference image and the fifth difference image. The correction data generation unit generates the corrected point cloud data based on the sixth difference image generated by the image processing unit. The point cloud data processing apparatus according to claim 1 or 2.

12. The image processing unit executes a process of reducing frequency components of a predetermined first frequency or higher with respect to the generated sixth difference image. The correction data generation unit generates the corrected point cloud data based on the sixth difference image on which the process has been executed. The point cloud data processing apparatus according to claim 11.

13. A first distance measurement step of imaging a first distance measurement image using unstructured illumination, a second distance measurement step of imaging a second distance measurement image using structured illumination, an image processing step of generating a difference image showing the difference between the first distance measurement image and the second distance measurement image based on the first distance measurement image imaged in the first distance measurement step and the second distance measurement image imaged in the second distance measurement step, a point cloud conversion step of converting distance values included in the first distance measurement image into point clouds based on the first distance measurement image imaged in the first distance measurement step to generate first point cloud data, a correction data generation step of generating corrected point cloud data based on the difference image generated in the image processing step, and a point cloud correction step of generating blended point cloud data by correcting the first point cloud data based on the corrected point cloud data generated in the correction data generation step. The point cloud data processing method includes.

14. A program for causing a computer to execute the point cloud data processing method according to claim 13.

Images