Point group data processing device, point group data processing method and program

Abstract

The point group data processing apparatus (100) includes: a first ranging unit (130) that captures a first ranging image (11) using unstructured illumination (111); a second ranging unit (140) that captures a second ranging image (12) using structured illumination (112); an image processing unit (150) that generates a difference image (13) representing the difference between the first ranging image (11) and the second ranging image (12); a point group transformation unit (160) that performs point group transformation on the distance values ​​contained in the first ranging image (11) captured by the first ranging unit (130) to generate first point group data (21); a correction data generation unit (170) that generates corrected point group data based on the difference image (13) generated by the image processing unit (150); and a point group correction unit (180) that corrects the first point group data (21) based on the generated corrected point group data, thereby generating mixed point group data (22).

Description

Technical Field

[0001] This disclosure relates to a point group data processing device, a point group data processing method, and a program. Background Technology

[0002] Time of Flight (ToF) is a technique used to determine the distance to an object. Specifically, ToF refers to the technique of calculating the distance to an object based on the amount of time lag in the reflected light. This time lag is determined by exposing the reflected light from the object to the illumination light from the light source during multiple exposures at different times in the imaging element, and then calculating the signal ratio corresponding to each exposure period.

[0003] Here, the ranging value obtained by ToF includes the error caused by the exposure of the illumination light from the light source as reflected light after being reflected multiple times by the object, i.e., the error caused by multipath.

[0004] Patent document 1 discloses a technique for suppressing errors contained in a ranging image representing a ranging result captured by a ToF camera.

[0005] Existing technical documents Patent documents Patent Document 1: US Patent No. 9329035 Summary of the Invention

[0006] The problem that the invention aims to solve The purpose of this disclosure is to provide a point group data processing device, etc., capable of suppressing errors caused by multipathing in point group data.

[0007] Methods for solving problems One embodiment of the point group data processing apparatus disclosed herein includes: a first ranging unit that captures a first ranging image using unstructured illumination; a second ranging unit that captures a second ranging image using structured illumination; an image processing unit that generates a difference image representing the difference between the first ranging image and the second ranging image based on the first ranging image captured by the first ranging unit and the second ranging image captured by the second ranging unit; a point group transformation unit that performs point group transformation on the distance values ​​contained in the first ranging image captured by the first ranging unit to generate first point group data based on the first ranging image captured by the first ranging unit; a correction data generation unit that generates corrected point group data based on the difference image generated by the image processing unit; and a point group correction unit that corrects the first point group data based on the corrected point group data generated by the correction data generation unit, thereby generating mixed point group data.

[0008] Additionally, one embodiment of the point group data processing method disclosed herein includes: a first ranging step, capturing a first ranging image using unstructured illumination; a second ranging step, capturing a second ranging image using structured illumination; an image processing step, generating a difference image representing the difference between the first ranging image and the second ranging image based on the first ranging image captured in the first ranging step and the second ranging image captured in the second ranging step; a point group transformation step, performing point group transformation on the distance values ​​contained in the first ranging image captured in the first ranging step to generate first point group data based on the first ranging image captured in the first ranging step; a correction data generation step, generating corrected point group data based on the difference image generated in the image processing step; and a point group correction unit, correcting the first point group data based on the corrected point group data generated in the correction data generation step, thereby generating mixed point group data.

[0009] Furthermore, this disclosure can be implemented not only as the aforementioned point group data processing method, but also as a program for executing the aforementioned point group data processing method on a computer. Moreover, it can also be implemented as a computer-readable recording medium storing the program.

[0010] Invention Effects The point group data processing apparatus, method, and program disclosed herein can suppress errors caused by multipathing in point group data. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating the general outline of the point group data processing apparatus according to the embodiments.

[0012] Figure 2 This is a diagram used to illustrate multipathing in Time of Flight (ToF).

[0013] Figure 3 This is a block diagram illustrating the functional structure of the point group data processing device in the implementation method.

[0014] Figure 4 This is a diagram showing an example of the first ranging image of the implementation method.

[0015] Figure 5 This is a diagram showing an example of the second ranging image of the implementation method.

[0016] Figure 6 It is a diagram used to illustrate point group transformations.

[0017] Figure 7 This is a flowchart illustrating the operation of the point group data processing device according to the implementation method.

[0018] Figure 8This is a diagram showing an example of a scaled-down image of an implementation method.

[0019] Figure 9 This is a diagram illustrating an example of an interpolated image used in an implementation method.

[0020] Figure 10 This is a diagram illustrating an example of a differential image of an implementation method.

[0021] Figure 11 This is a diagram illustrating an example of a corrected image for an implementation method.

[0022] Figure 12 This is a diagram used to illustrate the first point group data and the mixed point group data of the implementation method.

[0023] Figure 13 This is a flowchart illustrating the operation of a modified example 1 of the point group data processing apparatus according to the embodiment.

[0024] Figure 14 This is a diagram used to illustrate the masking process in the implementation method.

[0025] Figure 15 This is a flowchart illustrating the operation of a modified example 2 of the point group data processing apparatus according to the embodiment.

[0026] Figure 16 This is a diagram used to illustrate the Dot activation process in the implementation method.

[0027] Figure 17 This is a diagram used to illustrate the time series filtering process in the implementation method. Detailed Implementation

[0028] (Implementation Method) The following detailed description of the implementation of the point group data processing apparatus, point group data processing method, and program of this disclosure is provided with reference to the accompanying drawings. Furthermore, the embodiments described below represent preferred examples of this disclosure. The numerical values, constituent elements, arrangement positions of constituent elements, connection methods, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Additionally, constituent elements not described in the independent claims in the constituent elements of the following embodiments are described as arbitrary constituent elements constituting the preferred embodiments.

[0029] Furthermore, the figures are schematic diagrams and not necessarily strictly representational. Additionally, substantially identical structures are labeled with the same reference numerals across the figures, and sometimes repeated descriptions are omitted or simplified.

[0030] Furthermore, unless otherwise specified, ordinal numbers such as "first" and "second" in this disclosure do not imply the quantity or order of constituent elements, but are used for the purpose of distinguishing between similar constituent elements.

[0031] In addition, unless otherwise specified, "frequency" in this disclosure refers to "spatial frequency". Spatial frequency is a physical quantity that is the reciprocal of distance.

[0032] [summary] First, an overview of the point group data processing apparatus of this embodiment will be described. Figure 1 This is a schematic diagram illustrating the general outline of the point group data processing apparatus 100 according to the implementation method. Figure 2 This is a diagram used to illustrate multipathing in Time of Flight (ToF).

[0033] like Figure 1 As shown, the dot group data processing apparatus 100 includes a light-emitting unit 110 and an exposure unit 120. The dot group data processing apparatus 100 irradiates illumination light L1 into space from the light-emitting unit 110. The illumination light L1 irradiated into space is reflected by an object 200 in space and is exposed by the exposure unit 120 as reflected light L2. At this time, the exposure unit 120 performs exposures during multiple exposure periods with different timings relative to the illumination timing of the illumination light.

[0034] The dot group data processing device 100 calculates the time required from the illumination of the irradiation light L1 by the light-emitting unit 110 to the exposure of the reflected light L2 by the exposure unit 120 based on the signal quantities corresponding to the multiple exposure periods, thereby enabling the determination of the distance to the object 200.

[0035] In distance measurements using Time-of-Flight (ToF), it is known that the results include errors caused by multipath propagation. For example... Figure 2 As shown, when the illumination light from the light-emitting unit 110 travels along a direct path, directly illuminating the target object 200 and then reflecting, it is less likely to produce an error in distance from the actual distance. However, in reality, there may be a multipath where a portion of the illumination light from the light-emitting unit 110 illuminates other objects 201 and reflects before reaching the target object 200 and reflecting again. Figure 2 In the example shown, there is one multipath, but there may be multiple multipaths.

[0036] like Figure 2 As shown, light traveling in a multipath returns to the point group data processing device 100 later than light traveling in a direct path. Therefore, the distance calculated corresponding to the travel time of light traveling in the multipath is longer than the actual distance.

[0037] For example, when the distance to the wall is measured using the point group data processing device 100, the path of the light that is reflected from the ground, then reflected from the wall, and returns to the point group data processing device 100 is an example of a multipath.

[0038] In reality, the reflected light from the target object 200 that returns to the point group data processing device 100 includes light that travels along the direct path and light that travels along multiple paths. Therefore, the distance calculated by the point group data processing device 100 contains an error that is longer than the actual distance due to the multiple paths.

[0039] The point group data processing apparatus 100 of the embodiment is able to suppress the errors caused by multipathing contained in the point group data representing the results of distance measurement using ToF.

[0040] [structure] The structure of the point group data processing apparatus 100 of the embodiment will be described. Figure 3 This is a block diagram illustrating the functional structure of the point group data processing device 100 according to the implementation method.

[0041] The point group data processing apparatus 100 generates point group data based on a ranging image containing distance values ​​captured by a Time-of-Flight (ToF) sensor. The distance values ​​contained in the ranging image represent the distance to the object 200. Point group data is data representing a set of points in space. For example, point group data is three-dimensional point group data representing a set of multiple points with three-dimensional coordinates. Furthermore, by performing information processing for correcting the point group data, the point group data processing apparatus 100 can suppress errors caused by multipath propagation contained in the point group data. Specifically, the point group data processing apparatus 100 includes a light-emitting unit 110, an exposure unit 120, a first ranging unit 130, a second ranging unit 140, an image processing unit 150, a point group transformation unit 160, a correction data generation unit 170, a point group correction unit 180, and a mask processing unit 190.

[0042] The light-emitting unit 110 is a light irradiator that irradiates illumination light L1 into space according to an input light-emitting control signal. For example, the light-emitting unit 110 irradiates pulsed light as illumination light according to the timing indicated by the light-emitting control pulse included in the input light-emitting control signal. The light-emitting unit 110 repeatedly irradiates pulsed light at a predetermined period according to the light-emitting control signal.

[0043] The light-emitting unit 110 is, for example, a light illuminator comprising a light-emitting diode or laser element that emits infrared light. The light-emitting unit 110 is implemented using a light-emitting element with a relatively fast response speed and capable of high-speed flickering, and an optical system that receives light incident from the light-emitting element and controls the light distribution from the light-emitting element. The light-emitting unit 110 includes multiple light illuminators that irradiate light into space in different spatial forms. Different spatial forms refer to different intensity distributions of the light irradiated into space by the light-emitting unit 110. Specifically, the light-emitting unit 110 includes unstructured illumination 111 and structured illumination 112.

[0044] Unstructured illumination 111 illuminates light with a uniform intensity distribution. Unstructured illumination 111 is also referred to as a floodlight or unpatterned light source. More specifically, unstructured illumination 111 illuminates light with a uniform intensity distribution across the camera's field of view (in other words, the illuminated area of ​​the exposure section 120).

[0045] Structured illumination 112 illuminates light with a non-uniform intensity distribution. This non-uniform intensity distribution is, for example, a periodic dot pattern. Structured illumination 112 is also referred to as a dot light source or a patterned light source. More specifically, structured illumination 112 illuminates light with a specific pattern, such as an arrangement of dots, at the camera's viewing angle. In an embodiment, structured illumination 112 illuminates light with a light distribution such that the high-intensity portions of the light form a set of dots arranged in a two-dimensional pattern in both the longitudinal and transverse directions.

[0046] The exposure unit 120 is implemented, for example, by a pixel array. The exposure unit 120 performs exposures during multiple exposure periods with different timings relative to the illumination timing of the illumination light illuminating the unstructured illumination 111 or the structured illumination 112.

[0047] The exposure unit 120 includes a plurality of pixels 121 arranged in a two-dimensional configuration. Each of the plurality of pixels 121 generates a signal based on incident light. For example, the plurality of pixels 121 respectively convert the incident light into signal charge, generating a signal based on the converted signal charge. The incident light received by the pixels 121 is reflected light from the object 200 and background light from the surrounding environment. The plurality of pixels 121 have substantially the same structure as each other. The plurality of pixels 121 can be exposed multiple times within one frame at the timing indicated by the exposure control signal output from the first ranging unit 130 or the second ranging unit 140. Here, a frame is a period from the exposure of the plurality of pixels 121 until the signal is read out from the plurality of pixels 121.

[0048] The first ranging unit 130, the second ranging unit 140, the image processing unit 150, the point group transformation unit 160, the correction data generation unit 170, the point group correction unit 180, and the mask processing unit 190 described below are functional processing units implemented, for example, by a memory that stores the program and a processor that executes the program. Furthermore, in Figure 3 Although they are separated in the block diagram shown, at least one of the first ranging unit 130, the second ranging unit 140, the image processing unit 150, the point group transformation unit 160, the correction data generation unit 170, the point group correction unit 180, and the mask processing unit 190 can be configured with the same memory and processor. Furthermore, at least one of the first ranging unit 130, the second ranging unit 140, the image processing unit 150, the point group transformation unit 160, the correction data generation unit 170, the point group correction unit 180, and the mask processing unit 190 can also be implemented using dedicated logic circuits for performing specific processing.

[0049] The first ranging unit 130 performs information processing for outputting a first ranging image. Specifically, the first ranging unit 130 outputs a light emission control signal for driving the unstructured illumination 111. Additionally, the first ranging unit 130 outputs an exposure control signal for driving the exposure unit 120. The exposure control signal is a signal used to expose the plurality of pixels 121 individually at a timing corresponding to the timing of the illumination light L1 illuminating the unstructured illumination 111. Furthermore, the first ranging unit 130 performs signal processing related to the signals output by the plurality of pixels 121, calculates the distance to the object 200, and generates the first ranging image. In other words, the first ranging unit 130 captures the first ranging image by performing this information processing.

[0050] The first ranging unit 130 is implemented, for example, by a driving circuit and an information processing circuit. The driving circuit outputs a light emission control signal for driving the unstructured illumination 111 and an exposure control signal for driving the exposure unit 120. The information processing circuit performs information processing to generate a ranging image based on the signal obtained from the exposure unit 120. Figure 4 This is a diagram showing an example of the first ranging image 11 of the embodiment.

[0051] like Figure 4 As shown, the first ranging image 11 is a set of distance values ​​to the object 200 calculated based on signals obtained from multiple pixels 121 respectively.

[0052] The first ranging image 11 is a ranging image captured using unstructured illumination 111, thus enabling the calculation of distance values ​​for all pixels 121 of the exposure unit 120.

[0053] The second ranging unit 140 performs information processing for outputting a second ranging image. Specifically, the second ranging unit 140 outputs a light emission control signal for driving the structured illumination 112. Additionally, the second ranging unit 140 outputs an exposure control signal for driving the exposure unit 120. This exposure control signal is a signal used to expose each of the multiple pixels 121 at a timing corresponding to the timing of the illumination light L1 emitted by the structured illumination 112. Furthermore, the second ranging unit 140 performs signal processing related to the signals output by each of the multiple pixels 121, calculates the distance to the object 200, and generates the second ranging image. In other words, the second ranging unit 140 captures the second ranging image by performing this information processing.

[0054] The second ranging unit 140 is implemented, for example, by a driving circuit and an information processing circuit, wherein the driving circuit outputs a light emission control signal for driving the structured illumination 112 and an exposure control signal for driving the exposure unit 120, and the information processing circuit performs information processing for generating a ranging image based on the signal obtained from the exposure unit 120. Figure 5 This is a diagram illustrating an example of the second ranging image 12 of the embodiment.

[0055] exist Figure 5 In the diagram, the portion indicated by the white circle represents the distance value calculated based on the signal output by pixel 121, which receives sufficient light to measure the distance. Hereinafter, this pixel 121 will be referred to as the effective pixel. On the other hand, in... Figure 5 In the diagram, the part represented by dots (i.e., the part outside the white circle) is pixel 121 that did not receive enough light to measure the distance.

[0056] The second ranging image 12 is a ranging image taken using structured illumination 112, and therefore reflects the intensity distribution of the light illuminated by structured illumination 112. That is, in Figure 5 In the second ranging image 12 shown, the effective pixels are arranged as a set of points arranged in a two-dimensional shape in the longitudinal and transverse directions.

[0057] Here, we will briefly describe an example of the process by which the first ranging unit 130 and the second ranging unit 140 perform signal processing related to the signals output by the plurality of pixels 121 respectively, and calculate the distance to the object 200. Here, we will give an overview of the process, which is also known as the pulse ToF method.

[0058] The driving circuit causes the light-emitting part 110 to be irradiated with an irradiation pulse of pulse width P1. Pixel 121 is exposed during two exposure periods: exposure period A0, which is the length of the pulse width P1, and exposure period A1. Exposure period A0 starts simultaneously with the start of the light-emitting control pulse. Exposure period A1 is a timing delay of the pulse width P1 from the start of the light-emitting control pulse, that is, it starts simultaneously with the end of exposure period A0.

[0059] Reflected light from object 200 is received by pixel 121 after a light delay time Δt, depending on the distance to object 200. Therefore, when object 200 is within a range where the distance can be measured, the signal charge generated by the reflected light is allocated as a signal charge generated during exposure A0 and a signal charge generated during exposure A1, depending on the distance to object 200.

[0060] If the signal value corresponding to the exposure period A0 is set as A0, and the signal value corresponding to the exposure period A1 is set as A1, then the time Δt, which is the delay amount, is calculated by the following formula (1).

[0061] [Mathematical Expression 1] In addition, if the distance to object 200 is set as D and the speed of light is set as c, the illumination light takes time Δt to travel a distance of twice the distance D. Therefore, the distance D is calculated by the following formula (2).

[0062] [Mathematical Expression 2] By performing this process on multiple pixels 121 respectively, a ranging image containing the distance value D can be captured.

[0063] Furthermore, an example was given where the exposure period A0 begins simultaneously with the start of the emission control pulse, but it can also begin with a predetermined delay from the start of the emission control pulse. This allows the upper and lower limits of the ranging range to be shifted towards longer distances.

[0064] Furthermore, the process of calculating the distance to object 200 is not limited to the process described above. For example, the distance to object 200 can also be calculated using the CW (Continuous Wave)-ToF method.

[0065] The image processing unit 150 performs information processing to generate a difference image representing the difference between the first ranging image 11 and the second ranging image 12, based on the first ranging image 11 captured by the first ranging unit 130 and the second ranging image 12 captured by the second ranging unit 140. Additionally, the image processing unit 150 performs information processing to generate a corrected image by reducing frequency components at or above a predetermined first frequency on the generated difference image. In other words, the image processing unit 150 acquires the captured ranging images and performs information processing on the acquired ranging images. Details regarding the predetermined first frequency will be described later.

[0066] The point group transformation unit 160 performs information processing to perform point group transformation on the distance values ​​contained in the first ranging image 11 captured by the first ranging unit 130, thereby generating first point group data. Point group transformation refers to the process of transforming multiple distance values ​​contained in the ranging image into coordinates of points in the xyz coordinate system based on the ranging image and camera intrinsic parameters, thereby generating point group data.

[0067] Here, we will briefly explain the process of performing point group transformation on the distance values ​​contained in the ranging image. Figure 6 It is a diagram used to illustrate point group transformations.

[0068] First, use Figure 6 The u / v coordinate axes shown are used to calculate the location of pixels in the ranging image where point group transformation occurs, with the center of the ranging image as the origin. That is, u is the horizontal image coordinate, and v is the vertical image coordinate. If we define the half-field angle corresponding to the distance of each pixel from the optical axis as θ, and the rotation angle of each pixel position about the optical axis as... Then the camera's internal parameters are taken as I, θ, and It depends on u, v, and I. In other words, it can be written as θ(u, v, I). (u, v, I). The camera's internal parameters include the position (height) of the rangefinder camera and its performance.

[0069] Based on the calculated θ and To calculate the transformation coefficients. The transformation coefficients are the coefficients used for point group transformations, corresponding to the x-axis, y-axis, and z-axis directions, respectively.

[0070] Here, if the transformation coefficient in the x-axis direction is set as paramX, the transformation coefficient in the y-axis direction is set as paramY, and the transformation coefficient in the z-axis direction is set as paramZ, then the transformation coefficients are calculated by the following formula.

[0071] (paramX, paramY, paramZ) = (sin (θ) cos ( ), sin(θ) sin( ),cos(θ)) Based on the calculated transformation coefficients and the depth values ​​of multiple pixels in the ranging image, the 3D coordinates (x, y, z) of each pixel are calculated using the following formula.

[0072] (x, y, z) = (paramX) depth, paramY depth, paramZ depth) By performing this calculation on multiple pixels separately, the distance value can be transformed into 3D coordinates.

[0073] The point group transformation unit 160 performs point group transformation on the distance values ​​contained in the first ranging image 11 through such a process to generate the first point group data.

[0074] The correction data generation unit 170 performs information processing to generate correction point group data based on the difference image generated by the image processing unit 150. Specifically, the correction data generation unit 170 performs point group transformation on the distance values ​​contained in the difference image or correction image to generate correction point group data.

[0075] The point group correction unit 180 performs information processing to correct the first point group data based on the correction point group data generated by the correction data generation unit 170, thereby generating mixed point group data. For example, the point group correction unit 180 generates mixed point group data by synthesizing the first point group data and the correction point group data. The mixed point group data is point group data in which errors caused by multipath during ranging are suppressed by the information processing of the point group data processing device 100.

[0076] The mask processing unit 190 performs information processing to generate a high-frequency image by extracting frequency components of a predetermined second frequency or higher contained in the first ranging image 11. Details regarding the predetermined second frequency will be described later. The point group data processing apparatus 100 may also omit the mask processing unit 190.

[0077] [action] Next, the operation of the point group data processing device 100 according to the embodiment will be explained. Figure 7 This is a flowchart illustrating the operation of the point group data processing device 100 according to the implementation method.

[0078] Generally speaking, DotToF (ToF using structured lighting) is better at suppressing errors caused by multipathing compared to FloodToF (ToF using unstructured lighting).

[0079] On the other hand, DotToF has a significantly reduced spatial resolution compared to FloodToF.

[0080] Therefore, the point group data processing apparatus 100 of the embodiment can suppress errors caused by multipathing while maintaining spatial resolution by performing appropriate synthesis of DotToF and FloodToF. More specifically, the point group data processing apparatus 100 of the embodiment uses different light sources, including point light sources and floodlight sources, to capture ranging images and corrects the point group data obtained based on the ranging images, thereby generating point group data that suppresses the errors caused by multipathing contained in the captured ranging images.

[0081] First, the first ranging unit 130 uses unstructured illumination 111 to capture a first ranging image 11 (S11). The first ranging unit 130 outputs the captured first ranging image 11 to the image processing unit 150 and the point group transformation unit 160.

[0082] Next, the second ranging unit 140 uses structured illumination 112 to capture a second ranging image 12 (S12). The second ranging unit 140 outputs the captured second ranging image 12 to the image processing unit 150.

[0083] When the second ranging image 12 is acquired, the image processing unit 150 performs interpolation processing (S13) on the second ranging image 12. The interpolation processing is a process for obtaining an image with the same number of pixels as the first ranging image 11 by interpolating the invalid distance values ​​contained in the second ranging image 12. The interpolation processing includes the process of extracting the valid pixels contained in the second ranging image 12. For example, the interpolation processing refers to the process of generating an image with the same number of pixels as the first ranging image 11 based on the second ranging image 12 by performing prescribed information processing on the valid pixels contained in the second ranging image 12.

[0084] Specifically, firstly, the image processing unit 150 generates a reduced image by extracting the effective pixels contained in the second ranging image 12. Figure 8 This is a diagram illustrating an example of a reduced-size image 12a of an embodiment. For example... Figure 8 As shown, the reduced image 12a is an image composed only of the effective pixels contained in the second ranging image 12. That is, the reduced image 12a is an image with fewer pixels than the second ranging image 12. The number of pixels in the reduced image 12a may, for example, be the same as the number of regions with strong light intensity in the intensity distribution of the illumination light irradiated by the structured illumination 112.

[0085] Next, the image processing unit 150 generates an interpolated image based on the reduced image 12a. Specifically, the image processing unit 150 performs the information processing described later on the reduced image 12a to supplement pixels representing appropriate distance values ​​between pixels in the reduced image 12a, thereby generating an interpolated image. Figure 9 This is a diagram illustrating an example of the interpolated image 12b of the implementation method. For example... Figure 9 As shown, the interpolated image 12b is an image with the same number of pixels as the first ranging image 11.

[0086] The method for generating the interpolated image 12b from the reduced image 12a is, for example, the nearest neighbor interpolation method, but it can also be the bilinear interpolation method, or the bicubic interpolation method, etc. Alternatively, the generation of the interpolated image 12b can also be performed by upsampling the reduced image 12a using a joint-bilateral filter and a guided filter.

[0087] If the interpolated image 12b is generated, the image processing unit 150 generates a difference image based on the first ranging image 11 and the interpolated image 12b (S14). Figure 10 This is a diagram illustrating an example of the difference image 13 in the embodiment. The difference image 13 is an image representing the difference between the distance value of the first ranging image 11 and the distance value of the interpolated image 12b. For example, in... Figure 10 In the difference image 13 shown, the darker the color, the greater the difference between the distance value contained in the first ranging image 11 and the distance value contained in the interpolation image 12b.

[0088] If a difference image is generated, the image processing unit 150 performs a process to reduce the high-frequency components of the difference image 13 (S15). The image processing unit 150 performs the process to reduce the high-frequency components of the difference image 13 and generates a corrected image. Figure 11 This is a diagram illustrating an example of the corrected image 14 of the implementation method.

[0089] Figure 11 The corrected image 14 shown is the result of the reduction. Figure 10 The image shown is a processed image of the high-frequency components of the difference image 13. (Example:) Figure 11 As shown, the corrected image 14 and Figure 10 Compared to the differential image 13 shown, the high-frequency components, which exhibit abrupt spatial changes, are reduced.

[0090] Here, the difference image 13 mainly contains erroneous components caused by two reasons.

[0091] The first reason is the erroneous component caused by the low accuracy of the first ranging image 11. The reduced accuracy of the first ranging image 11 is caused by multipath propagation and low-reflectivity objects, etc. This erroneous component represents a relatively low frequency component.

[0092] The second reason is an error component caused by the low resolution of the second distance measurement image 12, that is, the small number of effective pixels. Since the resolution of the second distance measurement image 12 is low, it is impossible to detect a sharp Depth shape, that is, a shape with a large change in distance values in adjacent pixels such as a thin object or a pointed object, so an error component is generated. This error component represents a relatively high-frequency component.

[0093] Therefore, in the process of step S15, by performing a process of reducing the high-frequency component of the differential image 13 by the image processing unit 150, it is possible to extract the error component caused by the first reason, that is, the error component representing a relatively low-frequency component.

[0094] For example, the image processing unit 150 performs a frequency transformation on the differential image 13 and executes a process of reducing the high-frequency component included in the differential image 13.

[0095] Specifically, the image processing unit 150 performs a frequency transformation process on the differential image 13 through a discrete Fourier transform. The frequency transformation process can be performed through a discrete cosine transform, or through a discrete wavelet transform, or through a discrete Hadamard transform, or through a discrete Haar transform. The method of the frequency transformation process is not limited to the above and is not limited here.

[0096] Next, the image processing unit 150 executes a process of reducing the frequency components above a specified first frequency. The light reception amount based on the structured illumination 112 is distributed according to the pattern frequency fp on the image sensor with the pixel pitch frequency fs. Let the specified first frequency be fth, and fth is, for example, a value satisfying 0.1fp < fth < fs. In addition, fth can also be a value satisfying other ranges, for example, it can also be a value satisfying 0.2fp < fth < 1.2fs.

[0097] The process of reducing the specified first frequency component can be, for example, a process of setting it to 0 in the frequency spectrum, or a process of reducing it based on a specific function shape such as a Gaussian.

[0098] Finally, the image processing unit 150 executes a process of generating the corrected image 14 by performing an inverse frequency transformation on it.

[0099] In addition, this process can be performed on all pixels included in the differential image 13 or on a part of the pixels included in the differential image 13.

[0100] Alternatively, for example, the image processing unit 150 executes a process of reducing the high-frequency component by applying a spatial filter to the differential image 13.

[0101] Specifically, the image processing unit 150 performs processing to reduce high-frequency components by applying a spatial filter designed to reduce frequency components above a specified first frequency to the difference image 13.

[0102] The specified first frequency is set as fth. For example, fth is a value that satisfies 0.1fp < fth < fs. In addition, fth can also be a value that satisfies other ranges. For example, it can also be a value that satisfies 0.2fp < fth < 1.2fs.

[0103] The filter size of the spatial filter is not particularly limited as long as at least one of the vertical and horizontal sizes is larger than 1 pixel. For example, the vertical size can be 1 pixel and the horizontal size can be 3 pixels, or the vertical size can be 3 pixels and the horizontal size can be 1 pixel. Additionally, the direction in which the filter extends can be any one of up, down, left, or right.

[0104] In addition, the filtering process can also be performed separately in the vertical and horizontal directions. For example, a spatial filter with a size of 1 pixel × 3 pixels and a spatial filter with a size of 3 pixels × 1 pixel can be applied respectively, and the result obtained by summing or multiplying the applied results through some operation can be used as the final result.

[0105] The amount of received light based on structured illumination 112 is distributed according to the pattern frequency fp on an image sensor with a pixel pitch frequency fs. The specified first frequency is set as fth. For example, fth is a value that satisfies 0.1fp < fth < fs. In addition, fth can also be a value that satisfies other ranges. For example, it can also be a value that satisfies 0.2fp < fth < 1.2fs.

[0106] In addition, this processing can be performed on all pixels included in the difference image 13 or on a part of the pixels included in the difference image 13.

[0107] Alternatively, for example, the image processing unit 150 performs processing to reduce high-frequency components included in the difference image 13 using AI (Artificial Intelligence).

[0108] For example, the image processing unit 150 generates a corrected image 14 by inputting the difference image 13 to an image generation AI that has previously learned error components caused by a first cause and error components caused by a second cause.

[0109] The image generation AI is an AI model learned in such a way that it can extract error components caused by the first cause among the error components included in the difference image 13. The image generation AI model is implemented by an algorithm such as ControlNet or pix2pix, for example, but it can also be a model based on other algorithms.

[0110] Furthermore, this processing can be performed on all pixels contained in the difference image 13, or on a portion of the pixels contained in the difference image 13.

[0111] Next, the correction data generation unit 170 generates correction point group data based on the correction image 14 (S16). Specifically, the correction data generation unit 170 generates correction point group data by performing point group transformation on the distance values ​​contained in the correction image 14.

[0112] The point group transformation unit 160 generates first point group data based on the first ranging image 11 (S17). Specifically, the point group transformation unit 160 generates first point group data 21 by performing point group transformation on the distance values ​​contained in the first ranging image 11.

[0113] Finally, the point group correction unit 180 corrects the first point group data 21 using the corrected point group data, thereby generating mixed point group data (S18). In other words, the point group correction unit 180 generates mixed point group data by performing point group correction processing. Point group correction processing refers to the process of generating mixed point group data based on the first point group data 21 obtained by performing point group transformation on the first ranging image 11 and the corrected point group data obtained by performing point group transformation on the corrected image 14. Point group correction processing is performed, for example, by calculating the difference between the first point group data 21 and the corrected point group data and reflecting it in the first point group data 21. In addition, point group correction processing can also be implemented by other methods.

[0114] Figure 12 This is a diagram used to illustrate the first point group data 21 and the mixed point group data 22 of the implementation method. Figure 12 The diagram shows data obtained by cutting through the cross-section where the L-shaped shape appears, using the first ranging image 11 and the second ranging image 12, which are generated based on the L-shaped portion captured from the upper right. The L-shaped shape is a shape that easily generates multipath during ranging.

[0115] exist Figure 12 The ideal value of 31, represented by the dashed L-shape, represents the actual shape captured in the image. That is, in... Figure 12 In this context, the closer the positions of the points constituting the point group data are to the ideal value of 31, the more accurately the point group data reproduces the actual shape with fewer errors.

[0116] Below, in Figure 12 In the ideal value 31 of the L-shape shown, the dashed lines in the vertical direction are called the wall, and the dashed lines in the horizontal direction are called the ground.

[0117] like Figure 12As shown, the first point group data 21 is a point group generated based on distance values ​​that are longer than the ideal value 31 and the actual distance value shown by the ideal value 31, whether on the wall or the ground. That is, the first point group data 21 includes errors caused by multipathing.

[0118] On the other hand, the mixed point group data 22 is a point group data composed of points close to the ideal value 31 compared to the first point group data 21. That is, the mixed point group data 22 suppresses the distortion caused by MPI (Multipath Interference), that is, the error caused by multipath.

[0119] In addition, Figure 12 For example, as shown in the point group representing the ground in the first point group data 21, the first point group data 21 is a curved point group with high curvature, while the mixed point group data 22 is a roughly straight point group with relatively low curvature. That is to say, the mixed point group data 22 is a point group data with higher reproducibility of straight-line shape compared to the first point group data 21.

[0120] Therefore, such as Figure 12 As shown, the mixed point group data 22 reflects the actual shape with higher accuracy compared to the first point group data 21.

[0121] According to the operation described above, the point group data processing apparatus 100 of the embodiment generates corrected point group data based on the first ranging image 11 captured using unstructured lighting 111 and the second ranging image 12 captured using structured lighting 112, and generates mixed point group data 22 by correcting the first point group data 21 obtained from the first ranging image 11 based on the corrected point group data.

[0122] Therefore, according to such a point group data processing device 100, it is possible to generate point group data that suppresses errors caused by multipath during ranging.

[0123] [Variation Example 1] The point group data processing apparatus 100 of the above embodiment corrects the first point group data 21 by using corrected point group data, and can generate mixed point group data 22 that suppresses errors caused by multipathing. At this time, in order to suppress errors caused by multipathing, the point group data processing apparatus 100 generates corrected point group data by performing processing to reduce high-frequency components.

[0124] Here, the structured illumination 112 illuminates light with a light distribution such that the high-intensity portions of the light form a set of points arranged in a two-dimensional pattern in both the longitudinal and transverse directions. Therefore, the ranging image based on the structured illumination 112 has frequency components below a constant spatial frequency Fp, as defined by this light distribution. Consequently, it is impossible to obtain distance values ​​from the second ranging image 12 for locations with spatial frequency components greater than Fp. In other words, for example, it is impossible to detect objects with shapes exhibiting abrupt changes in depth, such as the legs of a chair, from the second ranging image 12.

[0125] Therefore, shapes with spatial frequencies above Fp cannot be accurately reflected in the corrected image 14. Consequently, the corrected image 14 can also be masked to protect the high-frequency components above the specified second frequency by using a frequency filter that extracts the high-frequency components above the specified second frequency.

[0126] According to the point group data processing apparatus 100 of Modification 1, it is possible to reflect the shape of high-frequency components that are not reflected in the second ranging image 12 in the corrected point group data. Hereinafter, Modification 1 of such a point group data processing apparatus 100 will be described. Figure 13 This is a flowchart illustrating the operation of a modified example 1 of the point group data processing apparatus 100 according to the embodiment. Figure 14 This is a diagram used to illustrate the masking process in the implementation method. Figure 14 (a) is a diagram showing an example of the first ranging image 11 performing mask processing in the implementation method. Figure 14 (b) is a diagram illustrating an example of a high-frequency image 15 used to perform mask processing in an embodiment. Figure 14 (c) is a diagram showing an example of a mask image 16 used to perform mask processing in an embodiment.

[0127] like Figure 13 As shown, firstly, in Figure 7 In step S15, the image processing unit 150 performs a process to reduce the high-frequency components above the first frequency specified in the differential image 13, and generates a corrected image 14.

[0128] The mask processing unit 190 extracts frequency components of a predetermined second frequency or higher contained in the first ranging image 11 to generate a high-frequency image 15 (S21). The mask processing unit 190 extracts the predetermined second frequency or higher frequency components by performing a process that has the effect of a frequency filter. The process that has the effect of a frequency filter is, for example, frequency transformation processing, spatial filter application processing, and processing using AI. Furthermore, the process that has the effect of a frequency filter is not limited to the above, and is not limited here.

[0129] The specified second frequency is, for example, Fp, but it could also be 1.2Fp. Specifically, such as... Figure 14 (a) and Figure 14 As shown in (b), the mask processing unit 190 generates a high-frequency image 15 by performing a process on the first ranging image 11 to extract frequency components of a predetermined second frequency or higher. Figure 14 In the high-frequency image 15 shown in (b), the solid lines indicate the extracted regions, and the diagonal lines indicate the unextracted regions. That is, from... Figure 14 The first ranging image 11 obtained by photographing the chair shown in (a) contains fine-shaped parts such as the legs of the chair, which are then used to generate a high-frequency image 15.

[0130] When the high-frequency image 15 is generated, the image processing unit 150 adjusts the correction image 14 based on the high-frequency image 15 generated by the mask processing unit 190 (S22). Specifically, as Figure 14 As shown in (c), the mask processing unit 190 generates a mask image 16 based on the high-frequency image 15. The image processing unit 150 adjusts the correction image 14 by synthesizing the mask image 16 and the correction image 14 based on mask coefficients. Figure 14 In the mask image 16 shown in (c), the solid lines indicate the areas that have undergone masking, while the white areas indicate the areas that have not undergone masking. That is, in Figure 14 In the mask image 16 shown in (c), the mask coefficient is calculated for the parts indicated by solid lines, and for the parts that are painted white, the mask coefficient is not calculated, or an invalid value of the mask coefficient is calculated.

[0131] The mask processing unit 190 calculates mask coefficients for the areas in the mask image 16 that are designated for mask processing, and the image processing unit 150 adjusts and corrects the image 14 based on the calculated mask coefficients. For example, the mask processing unit 190 calculates the mask coefficients by standardizing the pixel values ​​of the high-frequency image 15 and setting them as coefficients. The calculation of the mask coefficients can also be performed by standardizing the pixel values ​​and setting them as coefficients based on calculations such as multiplication or exponentiation of the image that yields the ratio of the high-frequency image 15 to the first ranging image 11. There are no particular limitations on the method for calculating the mask coefficients.

[0132] The image processing unit 150 performs masking processing on the corrected image 14 based on the calculated masking coefficients, masking frequency components at a predetermined second frequency and above, to generate an adjusted corrected image 14. The masking process can also be performed by multiplying the corrected image 14 by the calculated masking coefficients. Alternatively, the masking process can be performed by calculating the average value obtained by multiplying the corrected image 14 by the masking coefficients and replacing it with that average value. Furthermore, the masking process can be performed by performing pixel labeling based on AI or connection components on the regions shown in the mask image 16, calculating the average value of the distance correction for each label, and replacing it with that average value. The method of masking processing is not particularly limited.

[0133] When the calibration image 14 is adjusted, enter Figure 7 In step S16, the correction data generation unit 170 generates correction point group data based on the correction image 14 adjusted by the image processing unit 150.

[0134] As described above, the point group data processing apparatus 100 of Modified Example 1 can reflect the shape of high-frequency components that are not reflected in the second ranging image 12 in the corrected point group data, thereby improving the accuracy of the mixed point group data 22.

[0135] [Variation Example 2] In the point group data processing apparatus 100 of the above embodiment, the shooting range of the first ranging image 11 is the same as the shooting range of the second ranging image 12. In other words, the ranging range of the distance measurement performed by the first ranging unit 130 using unstructured illumination 111 is the same as the ranging range of the distance measurement performed by the second ranging unit 140 using structured illumination 112.

[0136] However, generally speaking, structured lighting 112 can enhance light output compared to unstructured lighting 111. In other words, structured lighting 112 can measure farther distances compared to unstructured lighting 111. Therefore, the second ranging image 12 can contain farther distance values ​​compared to the first ranging image 11.

[0137] Thus, when the ranging range of ToF based on structured illumination 112 is wider than that of ToF based on unstructured illumination 111, the effective point group data calculated based on the second ranging image 12 captured using structured illumination 112 and the mixed point group data 22 can be spatially synthesized.

[0138] Modification 2 of the point group data processing apparatus 100 can generate point group data with a wider spatial range by spatially combining effective point group data from a range farther than the range of the mixed point group data 22 with the mixed point group data 22. Hereinafter, Modification 2 of such a point group data processing apparatus 100 will be described. Figure 15 This is a flowchart illustrating the operation of a modified example 2 of the point group data processing apparatus 100 according to the embodiment. Figure 16 This is a diagram used to illustrate the Dot activation process in the implementation method. Figure 16 (a) is a figure illustrating the difference in the ranging range between the first ranging image 11 and the second ranging image 12. Figure 16 (b) is a diagram illustrating the range of the mixed point group data 22. Figure 16 (c) is a schematic diagram of the effective mixed point group data 24.

[0139] like Figure 15 As shown, firstly, in Figure 7 In step S18, the point group correction unit 180 generates mixed point group data. Additionally, the point group transformation unit 160 acquires the second ranging image 12 (in...). Figure 3 , Figure 15 (Not shown in the image).

[0140] Next, the point group transformation unit 160 determines the region within the shooting range of the second ranging image 12 that is not included in the shooting range of the first ranging image 11 (S31). Figure 16 In the first ranging image 11 shown in (a), the portion represented by dots is farther than the ranging range of the ToF based on unstructured illumination 111, and therefore represents the portion that did not receive enough light to calculate the distance. The portion represented by white represents the portion that received enough light to calculate the distance. Therefore, using Figure 7 The action described in the description of generating mixed point group data 22 is in Figure 16 The portion of the first ranging image 11 shown in (a) is indicated in white. Additionally, in Figure 16 In the second ranging image 12 shown in (a), black circles represent valid pixels used to generate the mixed point group data 22, and white circles represent valid pixels not used to generate the mixed point group data 22. Figure 16 As shown in (b), the mixed point group data 22 is based on Figure 16 The portion of the first ranging image 11 shown in white in (a) and in Figure 16 The effective pixels, indicated by white circles, in the second ranging image 12 shown in (a) are generated.

[0141] The point group transformation unit 160 determines the region within the shooting range of the second ranging image 12 that is not included in the shooting range of the first ranging image 11, i.e., in Figure 16 The effective pixels indicated by white circles in the second ranging image 12 shown in (a).

[0142] Next, the point group transformation unit 160 performs point group transformation on the ranging values ​​contained in the determined region of the second ranging image 12 to generate valid point group data 23 (S32). That is, the point group transformation unit 160 transforms the ranging values ​​based on the region determined in the second ranging image 12 to generate valid point group data 23 (S32). Figure 16 The distance values ​​of the signals output by the effective pixels (represented by white circles) in the second ranging image 12 shown in (a) are subjected to point group transformation to generate effective point group data 23. For example... Figure 16 As shown in (c), the effective point group data 23 is point group data about a region different from the mixed point group data 22.

[0143] Finally, the point group correction unit 180 spatially synthesizes the generated mixed point group data 22 and the effective point group data 23 generated by the point group transformation unit 160 to generate effective mixed point group data 24 (S33). Spatial synthesis means generating point group data about one space by adding point group data about two or more different spaces. Figure 16 As shown in (c), the effective mixed point group data 24 is the point group data obtained by spatially synthesizing the mixed point group data 22 and the effective point group data 23.

[0144] Based on this operation, the point group data processing apparatus 100 of Modified Example 2 can generate point group data with a wider spatial range by spatially synthesizing the effective point group data 23, which is located in a range farther than the mixed point group data 22, with the mixed point group data 22. Furthermore, since the effective point group data 23 is based on the second ranging image 12, it is less affected by errors caused by multipathing. Therefore, the point group data processing apparatus 100 can generate effective mixed point group data 24 that suppresses errors caused by multipathing.

[0145] [Variation Example 3] Depending on the structure of the point group data processing device 100, the timing of capturing the first ranging image 11 may differ from the timing of capturing the second ranging image 12. For example, when the point group data processing device 100 has an exposure unit 120, and the first ranging unit 130 and the second ranging rod 140 alternately use the same exposure unit 120 for capturing images, the first ranging image 11 and the second ranging image 12 may not be captured at the same timing.

[0146] In cases where the first ranging image 11 and the second ranging image 12 are captured at different timings, for example, due to the movement of the camera (especially the exposure unit 120) or the object 200 between the timing of capturing the first ranging image 11 and the timing of capturing the second ranging image 12, it is possible to generate motion-related pseudo-components in the differential image 13.

[0147] The point group data processing apparatus 100 of the modified embodiment 3 can eliminate spurious components originating from motion by performing time-series filtering processing on the difference image 13 in the current frame using the difference image 13 in the past frame.

[0148] The following describes the time series filtering process performed by the point group data processing apparatus 100 in a variation of this embodiment 3. Figure 17 This is a diagram used to illustrate the time series filtering process in the implementation method.

[0149] exist Figure 17 The image shows an example of time-series filtering processing performed when the camera, which is stationary at time 1, rotates counterclockwise at time 2, in a situation where the first ranging unit 130 and the second ranging unit 140 alternately use one exposure unit 120 to take pictures. Figure 17 The time shown as 1 is the time when the second ranging unit 140 captures the second ranging image 12. Figure 17 The time shown as 2 is the frame immediately following 1, and is the time when the third ranging image 11b is captured by the first ranging unit 130. The image processing unit 150 generates a first difference image 13a based on the first ranging image 11a and the second ranging image 12 captured by the first ranging unit 130 in the frame immediately preceding 1. The image processing unit 150 generates a second difference image 13b based on the third ranging image 11b and the second ranging image 12.

[0150] When Figure 17 When comparing the first ranging image 11a with the third ranging image 11b, at time 2, the camera moves counterclockwise, therefore the position of the object in the projected space, represented by the white-painted rectangle, shifts to the right in the third ranging image 11b. Figure 17 When comparing the first difference image 13a with the second difference image 13b, the second difference image 13b contains components in the area enclosed by the dashed line, resulting from the rightward deviation of the object's position—that is, motion-derived pseudo-components. These motion-derived pseudo-components are generated due to the movement of the camera or object between frames, and therefore tend to occur within a narrow spatial range with a large rate of change in pixel values ​​between frames.

[0151] The image processing unit 150 generates a third difference image 13c by performing time-series filtering based on the first difference image 13a and the second difference image 13b. Figure 17In the third difference image 13c shown, motion-related spurious components contained in the second difference image 13b are eliminated in the area enclosed by the dashed line. The time-series filtering process is used to eliminate motion-related spurious components contained in the difference image 13b of the current frame using the difference image 13a from past frames. The process of the time-series filtering process will be explained below.

[0152] First, the image processing unit 150 calculates the similarity between the first difference image 13a and the second difference image 13b. Similarity represents how close the pixel values ​​of the pixels in the first difference image 13a are to the pixel values ​​of the pixels in the second difference image 13b. That is, the smaller the difference in pixel values, the higher the similarity. Therefore, pixels with less influence from motion-related pseudo-components have a higher similarity, and pixels with greater influence from motion-related pseudo-components have a lower similarity. The similarity is calculated for each pixel.

[0153] Specifically, the image processing unit 150 calculates similarity by performing addition, subtraction, multiplication, and division on the pixel values ​​of pixels at the same location in the first difference image 13a and the second difference image 13b. Similarity can also be calculated using AI processing. Furthermore, the method for calculating similarity is not limited to the above and is not restricted here.

[0154] Next, the image processing unit 150 corrects the second difference image 13b based on the calculated similarity, thereby generating the third difference image 13c. For example, the image processing unit 150 corrects the second difference image 13b by processing a coefficient based on similarity. That is, the similarity is calculated as a coefficient in the range of 0 to 1, and the image processing unit 150 corrects the second difference image 13b by multiplying each pixel of the second difference image 13b by this coefficient. Furthermore, the method of correction processing is not limited to the above, and is not limited here.

[0155] Furthermore, in the aforementioned time-series filtering process, multiple difference images generated based on multiple different ranging images captured by the first ranging unit 130 are used. However, the time-series filtering process can also be performed using multiple difference images generated based on multiple different ranging images captured by the second ranging unit 140. In other words, the time-series filtering process can also be performed based on a difference image generated from the second ranging image 12 and a difference image generated from a fourth ranging image captured by the second ranging unit 140 after the second ranging image 12.

[0156] More specifically, after capturing the second ranging image, the second ranging unit 140 uses structured illumination to capture the fourth ranging image. The image processing unit 150 generates a fourth difference image based on the second ranging image and a fifth difference image based on the fourth ranging image. The image processing unit 150 performs time-series filtering based on the fourth and fifth difference images to generate a sixth difference image. That is, the image processing unit 150 calculates the similarity between the fourth and fifth difference images, uses the calculated similarity to correct the fifth difference image, thereby generating the sixth difference image. The correction data generation unit 170 generates correction point group data based on the sixth difference image generated by the image processing unit 150. The fourth difference image corresponds to the first difference image 13a, the fifth difference image corresponds to the second difference image 13b, and the sixth difference image corresponds to the third difference image 13c.

[0157] In this way, the image processing unit 150 uses multiple differential images generated based on the results of ranging at multiple times to perform time-series filtering on the differential images for correction, thereby eliminating spurious components originating from motion.

[0158] When generating a difference image that eliminates spurious components originating from motion, proceed... Figure 7 In step S15, the image processing unit 150 performs a process to reduce frequency components above a predetermined first frequency, generates a corrected image based on the differential image that has eliminated spurious components originating from motion, and then the corrected data generation unit 170 generates corrected point group data based on the corrected image.

[0159] Furthermore, the time-series filtering process can be applied to the corrected image instead of the difference image. More specifically, the image processing unit 150 generates a first corrected image by reducing frequency components of a predetermined first frequency or higher in the first difference image 13a, and generates a second corrected image by reducing frequency components of a predetermined first frequency or higher in the second difference image 13b. Next, the image processing unit 150 generates a third corrected image by performing the aforementioned time-series filtering process based on the generated first and second corrected images. Finally, the correction data generation unit 170 can also generate correction point group data based on the third corrected image. Thus, the point group data processing apparatus 100 is able to generate a third corrected image that eliminates motion-related spurious components that may be included in the second corrected image.

[0160] Furthermore, in the time series filtering process described above, similarity calculation is performed by comparing with the most recent past frame, but it can also be performed by comparing with an earlier frame, or by comparing with more than two past frames.

[0161] Furthermore, the time series filtering process can be performed on all pixels contained in the second difference image 13b, or on a portion of the pixels contained in the second difference image 13b.

[0162] Furthermore, in the above Figure 17 In the example shown, the camera is stationary at time 1, but it can also move at time 1. This is because the image processing unit 150 can eliminate motion-related spurious components contained in the first difference image 13a by performing time-series filtering on the first difference image 13a obtained at time 1 using a difference image earlier than time 1.

[0163] Thus, when the first ranging image 11 and the second ranging image 12 are captured at different timings, the point group data processing apparatus 100 of Modified Example 3 can eliminate motion-related spurious components that may occur in the difference image 13. Therefore, according to the point group data processing apparatus 100 of Modified Example 3, it is possible to generate mixed point group data 22 that suppresses errors caused by the movement of the camera or the object 200.

[0164] [Effects, etc.] The technologies derived from the disclosure in this specification are, for example, the following. Hereinafter, the technologies derived from the disclosure in this specification, along with the effects obtained through them, will be explained.

[0165] Technology 1 is a point group data processing apparatus 100, comprising: a first ranging unit 130 that captures a first ranging image 11 using unstructured illumination 111; a second ranging unit 140 that captures a second ranging image 12 using structured illumination 112; and an image processing unit 150 that generates a difference image 13 representing the difference between the first ranging image 11 and the second ranging image 12 based on the first ranging image 11 captured by the first ranging unit 130 and the second ranging image 12 captured by the second ranging unit 140. The point group transformation unit 160 performs point group transformation on the distance values ​​contained in the first ranging image 11 captured by the first ranging unit 130 to generate first point group data 21; the correction data generation unit 170 generates correction point group data based on the difference image 13 generated by the image processing unit 150; and the point group correction unit 180 corrects the first point group data 21 based on the correction point group data generated by the correction data generation unit 170, thereby generating mixed point group data 22.

[0166] Such a point group data processing apparatus 100 generates corrected point group data based on a first ranging image 11 captured using unstructured illumination 111 and a second ranging image 12 captured using structured illumination 112. Based on the corrected point group data, it corrects the first point group data 21 obtained from the first ranging image 11 to generate mixed point group data 22. According to such a point group data processing apparatus 100, it is possible to generate point group data that suppresses errors caused by multipathing during ranging.

[0167] In the point group data processing apparatus 100 of the first technique, the image processing unit 150 generates a reduced image 12a by extracting effective pixels contained in the second ranging image 12, and generates a difference image 13 based on the first ranging image 11 and the reduced image 12a.

[0168] Such a point group data processing apparatus 100 can generate corrected point group data using distance values ​​calculated from signals output by effective pixels that receive sufficient light to measure distance. Therefore, according to such a point group data processing apparatus 100, point group data that further suppresses errors caused by multipath during ranging can be generated.

[0169] In technique 3, within the point group data processing apparatus 100 of technique 1 or 2, the image processing unit 150 performs processing on the generated differential image 13 to reduce frequency components at or above a predetermined first frequency, and the correction data generation unit 170 generates correction point group data based on the processed differential image. The processed differential image is an example of the correction image 14 in the above embodiment.

[0170] Such a point group data processing device 100 can extract the error components in the error components contained in the difference image 13, which represent relatively low frequency components generated by multipath and low reflectivity objects, and reflect them in the corrected point group data.

[0171] In technique 4, in the point group data processing apparatus 100 of technique 3, the image processing unit 150 performs frequency transformation on the differential image 13 and performs processing.

[0172] By actually performing frequency transformation on the differential image 13, such a point group data processing device 100 can process frequency data while performing processing to reduce high-frequency components, thus improving the accuracy of the processing.

[0173] In technique 5, in the point group data processing apparatus 100 of technique 3, the image processing unit 150 performs processing by applying a spatial filter to the differential image 13.

[0174] Such a point group data processing device 100 can perform the process of reducing high-frequency components by applying a spatial filter to the difference image 13 without transforming the difference image 13 into frequency data, thus reducing the amount of computation required for the process.

[0175] Technology 6 is that in the point group data processing device 100 of Technology 3, the image processing unit 150 performs processing using AI.

[0176] Such a point group data processing device 100 can perform the process of reducing high-frequency components by pre-inputting the difference image 13 into the learned AI, thus making the process easier.

[0177] In the point group data processing apparatus 100 of any one of techniques 3 to 6, technique 7 further includes a mask processing unit 190, which extracts a predetermined frequency component of a second frequency or higher contained in the first ranging image 11 to generate a high-frequency image 15, an image processing unit 150 adjusts the processed differential image based on the high-frequency image 15 generated by the mask processing unit 190, and a correction data generation unit 170 generates correction point group data based on the processed differential image adjusted by the image processing unit 150.

[0178] Such a point group data processing device 100 can make the shape representing the high-frequency components that are not reflected in the second ranging image 12 reflected in the corrected point group data, thereby improving the accuracy of the mixed point group data 22.

[0179] In the point group data processing apparatus 100 of any one of techniques 1 to 7, technique 8 further includes a point group transformation unit 160 that determines a region within the imaging range of the second ranging image 12 that is not included in the imaging range of the first ranging image 11. Valid point group data 23 is generated by performing point group transformation on the ranging values ​​contained in the determined region of the second ranging image 12. A point group correction unit 180 generates valid mixed point group data 24 by spatially combining the generated mixed point group data 22 with the valid point group data 23 generated by the point group transformation unit 160.

[0180] This point group data processing apparatus 100 can generate effective mixed point group data 24 with a wider spatial range by spatially synthesizing effective point group data 23 from a range farther than the mixed point group data 22 with the mixed point group data 22. Furthermore, since the effective point group data 23 is based on the second ranging image 12, it is less affected by errors caused by multipathing. Therefore, the point group data processing apparatus 100 can generate effective mixed point group data 24 that suppresses errors caused by multipathing.

[0181] In the point group data processing apparatus 100 of technique 1 or 2, technique 9 further includes the following steps: after the first ranging unit 130 captures the first ranging image 11a, it captures the third ranging image 11b using unstructured illumination; the image processing unit 150 generates a first difference image 13a based on the first ranging image 11a and a second difference image 13b based on the third ranging image 11b; a third difference image 13c is generated based on the first difference image 13a and the second difference image 13b; and the correction data generation unit 170 generates correction point group data based on the third difference image 13c generated by the image processing unit 150.

[0182] When the first ranging image 11 and the second ranging image 12 are captured at different timings, such a point group data processing device 100 can eliminate motion-related spurious components that may occur in the difference image 13. Therefore, such a point group data processing device 100 can generate point group data that suppresses errors caused by the movement of the camera or the object 200.

[0183] In technique 10, in the point group data processing apparatus 100 of technique 9, the image processing unit 150 performs processing on the generated third differential image 13c to reduce frequency components above a predetermined first frequency, and the correction data generation unit 170 generates correction point group data based on the processed third differential image 13c.

[0184] Such a point group data processing device 100 can extract the error components in the third difference image 13c that represent relatively low frequency components generated by multipath and low reflectivity objects, and reflect them in the corrected point group data.

[0185] In the point group data processing apparatus 100 of technique 1 or 2, technique 11 further includes the following steps: after the second ranging unit 140 captures the second ranging image, it captures the fourth ranging image using structured illumination; the image processing unit 150 generates a fourth difference image based on the second ranging image and a fifth difference image based on the fourth ranging image; a sixth difference image is generated based on the fourth difference image and the fifth difference image; and the correction data generation unit 170 generates correction point group data based on the sixth difference image generated by the image processing unit 150.

[0186] When the first ranging image 11 and the second ranging image 12 are captured at different timings, such a point group data processing device 100 can eliminate motion-related spurious components that may occur in the difference image 13. Therefore, such a point group data processing device 100 can generate point group data that suppresses errors caused by the movement of the camera or the object 200.

[0187] In the point group data processing apparatus 100 of technique 11, the image processing unit 150 performs processing on the generated sixth differential image to reduce frequency components above a predetermined first frequency, and the correction data generation unit 170 generates correction point group data based on the processed sixth differential image.

[0188] Such a point group data processing device 100 can extract the error components in the sixth difference image that represent relatively low frequency components generated by multipath and low reflectivity objects, and reflect them in the corrected point group data.

[0189] Technique 13 is a point group data processing method, comprising: a first ranging step, capturing a first ranging image 11 using unstructured illumination 111; a second ranging step, capturing a second ranging image 12 using structured illumination 112; an image processing step, generating a difference image 13 representing the difference between the first ranging image 11 and the second ranging image 12 based on the first ranging image 11 captured by the first ranging step and the second ranging image 12 captured by the second ranging step; a point group transformation step, performing point group transformation on the distance values ​​contained in the first ranging image 11 captured by the first ranging step, generating first point group data 21 based on the first ranging image 11 captured by the first ranging step; a correction data generation step, generating corrected point group data based on the difference image 13 generated by the image processing step; and a point group correction step, correcting the first point group data 21 based on the corrected point group data generated by the correction data generation step, thereby generating mixed point group data 22.

[0190] This point group data processing method generates corrected point group data based on a first ranging image 11 captured using unstructured illumination 111 and a second ranging image 12 captured using structured illumination 112. The first point group data 21 obtained from the first ranging image 11 is then corrected based on the corrected point group data to generate mixed point group data 22. According to this point group data processing method, point group data that suppresses errors caused by multipath propagation during ranging can be generated.

[0191] Technique 14 is a program for enabling a computer to perform the point group data processing method of Technique 13.

[0192] Such a program can help generate point cluster data that suppresses errors caused by multipathing during ranging.

[0193] (Other implementation methods) The implementation methods have been described above, but this disclosure is not limited to the above-described implementation methods.

[0194] For example, in the above embodiments, in Figure 7In step S16, the correction data generation unit 170 generates correction point group data based on the correction image 14, but it can also generate correction point group data based on the difference image 13. In other words, it can be omitted. Figure 7 The processing in step S15. According to such a point group data processing device 100, it is possible to generate corrected point group data that includes not only error components caused by multipath but also error components caused by the low resolution of the second ranging image 12.

[0195] Furthermore, for example, in the above embodiment, the point group data processing apparatus is implemented by a single device, but it can also be implemented by multiple devices. In the case where the point group data processing apparatus is implemented by multiple devices, the constituent elements (especially functional constituent elements) of the point group data processing apparatus can be arbitrarily distributed among the multiple devices.

[0196] For example, in the above embodiment, the point group data processing apparatus includes a ranging device having a light-emitting part and an exposure part, but the point group data processing apparatus may not include a ranging device, but instead be an apparatus that accepts the input of a captured ranging image and performs information processing on the input ranging image.

[0197] Furthermore, the communication method between devices in the above embodiments is not particularly limited. Additionally, relay devices (gateway devices, etc.) not shown may also be present in the communication between devices.

[0198] Furthermore, in the above embodiments, the processing performed by the specific processing unit can also be executed by other processing units. Additionally, the order of multiple processes can be changed, or multiple processes can be executed in parallel.

[0199] Furthermore, in the above embodiments, each component can also be implemented by executing a software program suitable for each component. Each component can also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0200] Furthermore, each component can also be implemented using hardware. For example, each component can be a circuit (or integrated circuit). These circuits can be used as a whole to form a single circuit, or they can be separate circuits. Additionally, these circuits can be general-purpose circuits or dedicated circuits.

[0201] Furthermore, the present disclosure, in its entirety or in specific forms, can also be implemented using systems, apparatuses, methods, integrated circuits, computer programs, or recording media such as computer-readable CD-ROMs. Additionally, it can be implemented using any combination of systems, apparatuses, methods, integrated circuits, computer programs, and recording media.

[0202] Furthermore, this disclosure can be implemented as a point group data processing method executed by a computer, such as the point group data processing apparatus described in the above embodiments, or as a program (in other words, a computer program product) for causing a computer to execute the point group data processing method. Additionally, this disclosure can also be implemented as a computer-readable, non-transitory recording medium containing such a program.

[0203] Furthermore, this disclosure also includes various modifications to the embodiments that would be conceived by those skilled in the art, or methods implemented by arbitrarily combining the constituent elements and functions of the embodiments without departing from the spirit of this disclosure.

[0204] Industrial applicability The point group data processing apparatus disclosed herein is an information processing apparatus that can generate point group data representing the shape of an object by utilizing data obtained from photographing an actual existing object.

[0205] Explanation of reference numerals in the attached figures 11, 11a First distance measurement image 11b Third ranging image 12 Second Distance Measurement Image 12a Reduced image 12b interpolated image 13. Difference Image 13a First difference image 13b Second Difference Image 13c Third Difference Image 14. Correcting the image 15 High-frequency images 16 Mask Images 21. Group data at point 1 22 Mixed point cluster data 23 Effective point cluster data 24 Effective Mixed Point Cluster Data 31 Ideal value 100-point group data processing device 110 Light-emitting part 111 Unstructured lighting 112 Structured Lighting 120 Exposure Department 121 pixels 130 First Distance Measuring Unit 140 Second Distance Measuring Unit 150 Image Processing Department 160-point group transformation unit 170 Calibration Data Generation Unit 180-point group calibration department 190 Mask Processing Department 200 objects 201 Other objects

Claims

1. A point group data processing device, wherein, have: The first ranging unit uses unstructured lighting to capture the first ranging image; The second ranging unit uses structured lighting to capture the second ranging image; The image processing unit generates a difference image representing the difference between the first ranging image and the second ranging image based on the first ranging image captured by the first ranging unit and the second ranging image captured by the second ranging unit. The point group transformation unit performs point group transformation on the distance values ​​contained in the first ranging image captured by the first ranging unit to generate first point group data. The correction data generation unit generates correction point group data based on the difference image generated by the image processing unit; as well as The point group correction unit corrects the first point group data based on the correction point group data generated by the correction data generation unit, thereby generating mixed point group data.

2. The point group data processing device according to claim 1, wherein, The image processing unit generates a reduced image by extracting the effective pixels contained in the second ranging image, and generates the difference image based on the first ranging image and the reduced image.

3. The point group data processing apparatus according to claim 1 or 2, wherein, The image processing unit performs a process on the generated differential image to reduce frequency components above a predetermined first frequency. The correction data generation unit generates the correction point group data based on the differential image after the processing.

4. The point group data processing device according to claim 3, wherein, The image processing unit performs frequency transformation on the differential image and executes the processing.

5. The point group data processing device according to claim 3, wherein, The image processing unit performs the processing by applying a spatial filter to the differential image.

6. The point group data processing apparatus according to claim 3, wherein, The image processing unit uses artificial intelligence (AI) to perform the processing.

7. The point group data processing apparatus according to any one of claims 3 to 6, wherein, It also includes a mask processing unit that extracts frequency components of a predetermined second frequency or higher contained in the first ranging image to generate a high-frequency image. The image processing unit adjusts the differential image after performing the processing based on the high-frequency image generated by the mask processing unit. The correction data generation unit generates the correction point group data based on the differential image after the processing, which has been adjusted by the image processing unit.

8. The point group data processing apparatus according to any one of claims 1 to 7, wherein, Furthermore, the point group transformation unit determines the region within the shooting range of the second ranging image that is not included in the shooting range of the first ranging image, and generates valid point group data by performing point group transformation on the ranging values ​​contained in the determined region of the second ranging image. The point group correction unit generates effective mixed point group data by spatially synthesizing the generated mixed point group data and the effective point group data generated by the point group transformation unit.

9. The point group data processing apparatus according to claim 1 or 2, wherein, Furthermore, after capturing the first ranging image, the first ranging unit uses unstructured illumination to capture a third ranging image. The image processing unit generates a first difference image based on the first ranging image and a second difference image based on the third ranging image, and generates a third difference image based on the first difference image and the second difference image. The correction data generation unit generates the correction point group data based on the third difference image generated by the image processing unit.

10. The point group data processing apparatus according to claim 9, wherein, The image processing unit performs a process on the generated third differential image to reduce frequency components above a predetermined first frequency. The correction data generation unit generates the correction point group data based on the third difference image after the processing is performed.

11. The point group data processing apparatus according to claim 1 or 2, wherein, Furthermore, after capturing the second ranging image, the second ranging unit uses structured illumination to capture a fourth ranging image. The image processing unit generates a fourth difference image and a fifth difference image based on the second ranging image, and generates a sixth difference image based on the fourth difference image and the fifth difference image. The correction data generation unit generates the correction point group data based on the sixth difference image generated by the image processing unit.

12. The point group data processing apparatus according to claim 11, wherein, The image processing unit performs a process on the generated sixth differential image to reduce frequency components above a predetermined first frequency. The correction data generation unit generates the correction point group data based on the sixth difference image after the processing is performed.

13. A method for processing point cluster data, wherein, include: The first ranging step involves capturing the first ranging image using unstructured lighting; The second ranging step involves capturing the second ranging image using structured lighting; The image processing step generates a difference image representing the difference between the first ranging image and the second ranging image, based on the first ranging image captured by the first ranging step and the second ranging image captured by the second ranging step. The point group transformation step involves performing point group transformation on the distance values ​​contained in the first ranging image captured by the first ranging step to generate first point group data. The correction data generation step generates correction point group data based on the difference image generated through the image processing step. as well as The point group correction step involves correcting the first point group data based on the corrected point group data generated through the correction data generation step, thereby generating mixed point group data.

14. A program for causing a computer to perform the point group data processing method of claim 13.

Images