Signal processing device and signal processing method

Abstract

The present disclosure relates to a signal-processing device and a signal-processing method with which it is possible to correct any offset in position information of a dToF sensor. This signal-processing device comprises: a data acquisition unit for acquiring a distance histogram, which is histogram data pertaining to a radiated-light flight time at a prescribed distance measurement point of a distance-measuring sensor; a confirmation subject assessment unit for assessing whether the prescribed distance measurement point is to be subjected to confirmation of positional offset; and a coordinate correction unit for, when the prescribed distance measurement point is to be subjected to confirmation of positional offset, executing a correction process for correcting three-dimensional coordinates of the prescribed distance measurement point, as computed from the distance histogram, on the basis of the result of an assessment as to the degree of similarity between the distance histogram of the prescribed distance measurement point and the distance histogram of a distance measurement point that is near the prescribed distance measurement point. The technology according to the present disclosure can be applied to, inter alia, a signal-processing device for calculating three-dimensional coordinates of an object using a distance histogram outputted from a distance-measuring sensor.

Description

[Technical Field]

[0001] The present disclosure relates to a signal processing device and a signal processing method, and more particularly to a signal processing device and a signal processing method that are capable of correcting deviations in position information of a dToF sensor. [Background technology]

[0002] A direct ToF ToF sensor (hereinafter also referred to as a dToF sensor) uses a light-receiving element called a SPAD (Single Photon Avalanche Diode) in each light-receiving pixel to detect the reflected light of pulsed light reflected by an object. To suppress noise caused by ambient light, etc., the dToF sensor repeatedly emits pulsed light and receives the reflected light a predetermined number of times (e.g., several to several hundred times). The dToF sensor generates a histogram of the time-of-flight of the pulsed light and calculates the distance to the object from the time-of-flight corresponding to the peak of the histogram.

[0003] It is known that the signal-to-noise ratio is low and it is difficult to detect the peak position when measuring distances to subjects with low reflectivity or at a distance, or in environments where there is a strong disturbance from external light, such as outdoors. For this reason, the emitted pulsed light is made into a spot shape to extend the reach of the pulsed light, in other words, to increase the number of reflected light detections. Since spot-shaped pulsed light is generally sparse, the pixels where reflected light is detected also become sparse depending on the spot diameter and irradiation area.

[0004] In order to improve the S / N ratio and reduce power consumption by driving pixels efficiently in accordance with a sparse reflected light detection environment, a number of adjacent pixels (referred to as multi-pixels) in a pixel array are treated as one large pixel, and light reception is performed in multi-pixel units to generate a histogram. For example, Patent Document 1 discloses a method of forming a multi-pixel with any number of adjacent pixels, such as 2x3, 3x3, 3x6, 3x9, 6x3, 6x6, or 9x9, and creating a histogram using the multi-pixel signals to calculate distance, thereby improving the S / N ratio at the expense of lowering spatial resolution.

[0005] Distance measurement sensors such as dToF sensors are used together with RGB cameras in volumetric capture technology, which generates a 3D object of a subject from video images captured from multiple viewpoints and generates a virtual viewpoint image of the 3D object according to any viewing position. Distance measurement sensors such as dToF sensors are also used together with RGB cameras in SLAM (Simultaneous Localization and Mapping), which simultaneously estimates self-position and creates an environmental map. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-112443 Summary of the Invention [Problem to be solved by the invention]

[0007] When the 3D position information of the ranging point acquired by the dToF sensor is matched with the captured image obtained by the RGB camera, there may be a discrepancy in the position information acquired by the dToF sensor due to calibration errors, ranging errors, deviations in exposure timing, etc.

[0008] The present disclosure has been made in consideration of such circumstances, and makes it possible to correct deviations in position information of a dToF sensor. [Means for solving the problem]

[0009] A signal processing device according to one aspect of the present disclosure includes an acquisition unit that acquires a distance histogram, which is histogram data of the time of flight of irradiated light at a predetermined ranging point of a ranging sensor; a judgment unit that determines whether the predetermined ranging point is a ranging point for which positional deviation is to be confirmed; and a correction unit that, if the predetermined ranging point is a ranging point for which positional deviation is to be confirmed, performs a correction process to correct the three-dimensional coordinates of the predetermined ranging point calculated from the distance histogram based on the judgment result of the similarity between the distance histogram of the predetermined ranging point and the distance histograms of ranging points near the predetermined ranging point.

[0010] A signal processing method of one aspect of the present disclosure includes a signal processing device that acquires a distance histogram, which is histogram data of the time of flight of irradiated light at a predetermined ranging point of a ranging sensor, determines whether the predetermined ranging point is a ranging point for which positional deviation is to be confirmed, and, if the predetermined ranging point is a ranging point for which positional deviation is to be confirmed, performs a correction process to correct the three-dimensional coordinates of the predetermined ranging point calculated from the distance histogram based on the determination result of the similarity between the distance histogram of the predetermined ranging point and the distance histogram of a ranging point near the predetermined ranging point.

[0011] In one aspect of the present disclosure, a distance histogram, which is histogram data of the flight time of irradiated light at a predetermined ranging point of a ranging sensor, is acquired, and it is determined whether the predetermined ranging point is a ranging point for which positional deviation is to be confirmed.If the predetermined ranging point is a ranging point for which positional deviation is to be confirmed, a correction process is performed to correct the three-dimensional coordinates of the predetermined ranging point calculated from the distance histogram based on the determination result of the similarity between the distance histogram of the predetermined ranging point and the distance histogram of a ranging point near the predetermined ranging point.

[0012] The signal processing device according to one aspect of the present disclosure can be realized by causing a computer to execute a program. The program executed by a computer to realize the signal processing device according to one aspect of the present disclosure can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0013] The signal processing device may be a stand-alone device or a module that is incorporated into another device. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration example of a first embodiment of a signal processing system according to the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating the operation of the RGB camera and the dToF sensor in FIG. 1. [Figure 3] FIG. 1 is a diagram illustrating a dToF sensor. [Figure 4] 10A and 10B are diagrams illustrating a positional deviation correction function of the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating a peak region of a distance histogram. [Figure 6] 10A and 10B are diagrams illustrating a confirmation target determination process performed by a confirmation target determination unit. [Figure 7] 10A and 10B are diagrams illustrating a positional deviation correction process performed by a coordinate correction unit. [Figure 8] FIG. 10 is a diagram illustrating a method for calculating a similarity. [Figure 9] FIG. 10 is a diagram illustrating a method for calculating a similarity. [Figure 10] 4 is a flowchart illustrating a position information calculation process performed by the signal processing system of the first embodiment. [Figure 11] FIG. 10 is a block diagram showing a configuration example of a second embodiment of a signal processing system according to the present disclosure. [Figure 12] 10A and 10B are diagrams illustrating a confirmation target determination process performed by a confirmation target determination unit. [Figure 13] 10A and 10B are diagrams illustrating a confirmation target determination process performed by a confirmation target determination unit. [Figure 14]FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer that executes signal processing according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, with reference to the accompanying drawings, a description will be given of a mode for carrying out the technology of the present disclosure (hereinafter referred to as an embodiment). Note that in this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted. The description will be given in the following order. 1. First embodiment of signal processing system 2. Verification target determination process of the verification target determination unit 3. Positional deviation correction processing of the coordinate correction unit 4. Position Information Calculation Process According to the First Embodiment 5. Second embodiment of signal processing system 6. Position Information Calculation Process According to the Second Embodiment 7. Summary 8. Computer configuration example

[0016] <1. First embodiment of signal processing system> FIG. 1 is a block diagram showing a configuration example of a first embodiment of a signal processing system according to the present disclosure.

[0017] The signal processing system 1 in FIG. 1 is composed of an RGB camera 11, a dToF sensor 12, and a signal processing device 13.

[0018] The signal processing device 13 has a data acquisition unit 21, a distance calculation unit 22, a correction processing unit 23, a storage unit 24, and an output unit 25. The correction processing unit 23 includes a confirmation target determination unit 31 and a coordinate correction unit 32.

[0019] The RGB camera 11 captures an image of a predetermined object as a subject, generates an RGB image (moving image), and supplies the image to the signal processing device 13. The dToF sensor 12 is a distance measurement sensor that acquires distance information to the subject using a direct ToF method, and acquires distance information to the same object as the object captured by the RGB camera 11. The relative positional relationship between the RGB camera 11 and the dToF sensor 12 is fixed, and the imaging ranges of the RGB camera 11 and the dToF sensor 12 are calibrated. In other words, the imaging ranges of the RGB camera 11 and the dToF sensor 12 are the same, and the correspondence between each pixel of the RGB camera 11 and the dToF sensor 12 is known. In this embodiment, for simplicity of explanation, it is assumed that the difference in position between the RGB camera 11 and the dToF sensor 12 can be ignored, and the camera positions (camera orientation) of the RGB camera 11 and the dToF sensor 12 are the same.

[0020] 2, the RGB camera 11 captures an object OBJ as a subject while moving from one shooting location to another over time, and generates an RGB image. The dToF sensor 12 moves together with the RGB camera 11 and receives light reflected from the object OBJ, which is a plurality of spot lights (illumination light) emitted from a light source (not shown), to acquire a distance histogram as distance information of the object OBJ. The distance histogram is histogram data of the time of flight of the illumination light corresponding to the distance to the object OBJ.

[0021] The dToF sensor 12 will be briefly described with reference to FIG.

[0022] The dToF sensor 12 uses a light-receiving element called a SPAD (Single Photon Avalanche Diode) in each light-receiving pixel to detect reflected light that is returned when pulsed light as irradiated light is reflected by an object. In order to suppress noise due to ambient light, etc., the dToF sensor 12 generates a histogram of the time of flight of the pulsed light by repeating the emission of pulsed light and the reception of the reflected light a predetermined number of times (e.g., several to several hundred times), and outputs a distance histogram. Following the RGB camera 11, a unit for outputting a distance histogram once for the same imaging range as the RGB camera 11 is called one frame.

[0023] It is known that the signal-to-noise ratio is low and it is difficult to detect the peak position when measuring distances to subjects with low reflectivity or at a distance, or in environments where external light disturbances, such as outdoors, are a significant factor. Therefore, the emitted pulsed light is shaped like a spot to extend the reach of the pulsed light, or in other words, to increase the number of reflected light detections. Because spot-shaped pulsed light is generally sparse, the pixels where reflected light is detected also become sparse depending on the spot diameter and irradiation area. Figure 3 shows an example in which 5 x 5 = 25 spot light beams SP are irradiated onto the same imaging area as the RGB camera 11, and each spot light beam SP detects the reflected light reflected by an object.

[0024] To improve the signal-to-noise ratio and reduce power consumption by efficiently driving pixels in accordance with the sparse reflected light detection environment, the dToF sensor 12 sets up a group of adjacent pixels as multi-pixels MPs in response to the sparse light spot SP, and causes only a few of the pixels in the pixel array to perform light receiving operations, generating a histogram for each multi-pixel MP. In the example of Figure 3, a 3x3 multi-pixel MP consisting of nine pixels is set for one light spot SP.

[0025] 3, an example has been described using 5x5=25 spot lights SP and a 3x3=9-pixel multi-pixel MP, but the number and arrangement of the spot lights SP and multi-pixel MP are arbitrary. In the following description, when there is no particular distinction between the spot lights SP and the multi-pixel MP, the points corresponding to them will also be referred to as distance measurement points.

[0026] The RGB camera 11 in FIG. 1 captures a predetermined object as a subject from time to time to generate (a moving image of) RGB images and supplies the images to a signal processing device 13. The dToF sensor 12 supplies the signal processing device 13 with a distance histogram obtained by receiving light reflected from a spot light SP irradiated onto the predetermined object as a subject, as well as the camera attitude at the time the distance histogram was acquired. The distance histogram is composed of the pixel position (x, y) corresponding to the center of the spot light SP detected by the dToF sensor 12 and histogram data. The camera attitude is information on the external parameters of the dToF sensor 12 detected by an inertial measurement unit (IMU) within the dToF sensor 12. However, the dToF sensor 12 does not necessarily have to include an inertial measurement unit (IMU). In this case, the dToF sensor 12 outputs only a distance histogram, and the camera attitude of the dToF sensor 12 is calculated by, for example, calculating the three-dimensional position information of corresponding (identical) ranging points in each frame using a method such as Normal Distribution Transform in a distance calculation unit 22 in the signal processing device 13.

[0027] The signal processing device 13 acquires the RGB image captured by the RGB camera 11 and the distance histogram and camera attitude generated by the dToF sensor 12, and generates and outputs three-dimensional coordinates on a global coordinate system of a predetermined object that is a subject. In the following description, even when simply referred to as three-dimensional coordinates, this refers to three-dimensional coordinates on the global coordinate system.

[0028] That is, the signal processing device 13 is a signal processing device that performs processing to calculate the three-dimensional coordinates of a predetermined object, which is a subject, on the global coordinate system based on the distance histogram generated by the dToF sensor 12 and the camera attitude at that time. At this time, the signal processing device 13 has a position deviation correction function that corrects the position deviation when the calculated three-dimensional coordinates of the object are deviated from the position of the object on the RGB image due to a calibration error, a ranging error, a deviation in exposure timing, etc.

[0029] FIG. 4 shows a diagram in which each distance measurement point before positional deviation correction calculated based on the distance histogram acquired from the dToF sensor 12 is superimposed on an RGB image captured by the RGB camera 11.

[0030] The RGB image 51 captured by the RGB camera 11 includes a car (automobile) 41, a person (pedestrian) 42, and a tree 43 as subjects. Circles with predetermined patterns superimposed on the RGB image 51 represent ranging points K of the dToF sensor 12. The patterns attached to each ranging point K shown in FIG. 4 are divided into three types of patterns, ranging points K1, K2, and K3, according to the calculated distances (three-dimensional coordinates) to the objects. The ranging point K1 is a ranging point having a distance (three-dimensional coordinates) corresponding to the car 41. The ranging point K2 is a ranging point having a distance (three-dimensional coordinates) corresponding to the person 42. The ranging point K3 is a ranging point having a distance (three-dimensional coordinates) corresponding to a background 44 other than the car 41, the person 42, and the tree 43. The background 44 is, for example, a wall of a building.

[0031] Here, attention is focused on one predetermined ranging point K11 among the many ranging points K of the dToF sensor 12. The ranging point K11 has a distance (three-dimensional coordinates) calculated based on the distance histogram that corresponds to the car 41. However, the three-dimensional coordinates of the ranging point K11 calculated based on the distance histogram are not the position of the car 41 but the position of the background 44. Such a positional deviation of the ranging point K11 occurs, for example, due to a calibration error or ranging error of the dToF sensor 12. Alternatively, such a positional deviation can occur due to a deviation in the timing of acquiring the distance histogram (exposure timing) or movement of the object when the distance histogram of the entire shooting range of the RGB camera 11 is acquired over multiple frames because the number of ranging points K acquired by the dToF sensor 12 in one frame is small and sparse.

[0032] The signal processing device 13 executes a positional deviation correction process to correct the positional deviation of the three-dimensional coordinates of the ranging points K such as the ranging point K11, and outputs the three-dimensional coordinates of each ranging point K. In the example of the ranging point K11, the current ranging point K11 is corrected to a position in the lower left direction on the RGB image 51 in FIG. 4, and the positional deviation of the three-dimensional coordinates is corrected so that the three-dimensional coordinates become the position of the car 41.

[0033] 1, the data acquisition unit 21 of the signal processing device 13 acquires the RGB image supplied from the RGB camera 11 and the distance histogram and camera attitude supplied from the dToF sensor 12. The data acquisition unit 21 supplies the acquired RGB image to the correction processing unit 23, and also supplies the acquired distance histogram and camera attitude to the distance calculation unit 22.

[0034] Based on the distance histogram and the camera attitude from the data acquisition unit 21, the distance calculation unit 22 calculates three-dimensional coordinates (x, y, z) for each ranging point of the dToF sensor 12. More specifically, the distance calculation unit 22 detects a peak region of the count value from the histogram data of the multi-pixel MP corresponding to the spot light SP, and calculates the three-dimensional coordinates (x, y, z) from the detected peak region and the camera attitude.

[0035] Here, the peak region is detected as follows. For example, as shown in FIG. 5, a bin whose count value is equal to or greater than a predetermined threshold value Th and has the largest count value (peak) PV among multiple adjacent bins, as well as multiple bins surrounding it, are detected as the peak region. The multiple bins surrounding the peak may be defined as bins surrounding the peak that have count values ​​equal to or greater than a certain percentage of the count value of the peak PV (e.g., 0.5 PV, which is half the peak PV), or may be defined as a predetermined number of bins before and after the bin of the peak PV. In the example of FIG. 5, the three hatched bins, i.e., the bin of the peak PV and two bins surrounding it that have count values ​​equal to or greater than 0.5 PV, are detected as the peak region. Three-dimensional coordinates (x, y, z) are calculated from the detected peak region and the camera attitude.

[0036] Furthermore, as in the example of FIG. 5, the distance calculation unit 22 can detect the median EVC of the count values ​​of the bins other than the detected peak region as the ambient light intensity.

[0037] The distance calculation unit 22 supplies the three-dimensional coordinates (x, y, z) of each distance measurement point calculated from the distance histogram and the camera attitude, and the distance histogram to the correction processing unit 23. Note that the distance calculation unit 22 may supply histogram data of the peak region and ambient light intensity to the correction processing unit 23 instead of the distance histogram.

[0038] Correction processing unit 23 determines whether or not positional deviation correction is required for the calculated three-dimensional coordinates of each ranging point, using the RGB image supplied from data acquisition unit 21 and the three-dimensional coordinates (x, y, z) and distance histogram of each ranging point supplied from distance calculation unit 22. If it is determined that positional deviation correction is required, correction processing unit 23 executes positional deviation correction processing to move (correct) the three-dimensional coordinates of the ranging point.

[0039] Specifically, the confirmation target determination unit 31 and the coordinate correction unit 32 of the correction processing unit 23 set each of the plurality of ranging points included in one frame of the dToF sensor 12 as a ranging point of interest, and perform the following processing.

[0040] The confirmation target determination unit 31 determines whether the target ranging point is a ranging point for which positional deviation is to be confirmed, in other words, whether it is a ranging point for which positional deviation is likely to have occurred. If the confirmation target determination unit 31 determines that the target ranging point is a ranging point for which positional deviation is to be confirmed, it instructs the coordinate correction unit 32 to execute positional deviation correction processing for the target ranging point. On the other hand, if it determines that the target ranging point is not a ranging point for which positional deviation is to be confirmed, the confirmation target determination unit 31 supplies the three-dimensional coordinates (x, y, z) and distance histogram of the target ranging point calculated by the distance calculation unit 22 to the storage unit 24 as is (without performing positional deviation correction processing) and stores them therein.

[0041] The coordinate correction unit 32 executes the positional deviation correction process for the target ranging point when instructed by the confirmation target determination unit 31 to execute the positional deviation correction process for the target ranging point. As the positional deviation correction process, the coordinate correction unit 32 determines whether or not positional deviation correction is necessary, and if it determines that it is necessary, executes a process to correct (move) the three-dimensional coordinates (x, y, z) of the target ranging point calculated by the distance calculation unit 22. The three-dimensional coordinates (x, y, z) of the target ranging point after the positional deviation correction process and the distance histogram are supplied to the storage unit 24 and stored therein.

[0042] The correction processing unit 23 also supplies the RGB image supplied from the data acquisition unit 21 to the storage unit 24 at a predetermined timing for storage. Either the confirmation target determination unit 31 or the coordinate correction unit 32 may store the RGB image in the storage unit 24.

[0043] When the correction processing unit 23 has completed processing for all of the multiple frames sequentially supplied from the dToF sensor 12, the correction processing unit 23 supplies an end notification to the output unit 25.

[0044] When the output unit 25 receives a completion notification from the correction processing unit 23, it outputs the three-dimensional coordinates of each ranging point in all frames stored in the storage unit 24 as three-dimensional coordinates after correction processing.

[0045] The correction processing unit 23 may supply the output unit 25 with a completion notification for each frame of a plurality of frames sequentially supplied from the dToF sensor 12. In this case, the output unit 25 acquires and outputs the three-dimensional coordinates after the correction process from the storage unit 24 for each frame.

[0046] The signal processing device 13 has the above-described configuration. The correction processing unit 23 of the signal processing device 13 will be further described in detail below.

[0047] <2. Verification target determination process by the verification target determination unit> Next, the confirmation target determination process performed by the confirmation target determination unit 31 will be described.

[0048] The confirmation target determination unit 31 determines whether the target distance measurement point is a distance measurement point to be confirmed, in other words, whether it is a distance measurement point where there is a high possibility that a positional deviation has occurred.

[0049] Specifically, because positional deviations tend to occur near the boundary of an object, or because the impact of positional deviations is significant when positional deviations occur near the boundary of an object, the confirmation target determination unit 31 extracts, as an edge area, an area near the boundary of an object from the RGB image captured by the RGB camera 11. Then, when the target ranging point is included in the edge area, the confirmation target determination unit 31 determines that the target ranging point is likely to be displaced and is therefore the ranging point to be confirmed.

[0050] FIG. 6 is a diagram illustrating the confirmation target determination process performed by the confirmation target determination unit 31. In FIG.

[0051] First, the confirmation target determination unit 31 detects edges using, for example, the Canny method in the RGB image 51 captured by the RGB camera 11. The edge image 51E in Fig. 6 is an image in which the edges detected from the RGB image 51 are expressed with black pixel values.

[0052] Next, the confirmation target determination unit 31 determines an edge region of a predetermined width from the edge by performing a filter process N times (N is an integer equal to or greater than 2) on the edge image 51E in which the edge has been detected, which expands black pixels. In the example of Fig. 6, the region indicated by a dot pattern in the edge region image 51R indicates the edge region determined based on the edge image 51E.

[0053] Note that edge regions may be determined by methods other than the above-described expansion process. For example, a region in which movement has been detected from several past frames may be determined as an edge region. More specifically, the RGB image of a past frame stored in the storage unit 24 and the RGB image of the current frame may be input, and a region in which the number of expansion processes has been increased for portions with large movement in a dense optical flow image obtained by the Gunnar-Farneback method or the like may be determined as an edge region.

[0054] Next, the confirmation target determination unit 31 uses the internal camera parameters of the RGB camera 11 to find screen coordinates (u, v) on the RGB image 51 that correspond to the three-dimensional coordinates of the target ranging point. If the screen coordinates (u, v) of the target ranging point on the RGB image 51 are included in an edge region, the confirmation target determination unit 31 determines that the target ranging point is the ranging point to be confirmed. On the other hand, if the screen coordinates (u, v) of the target ranging point on the RGB image 51 are located outside the edge region, the confirmation target determination unit 31 determines that the target ranging point is not the ranging point to be confirmed. Note that the internal camera parameters of the RGB camera 11 for converting the three-dimensional coordinates of the target ranging point into screen coordinates (u, v) are known.

[0055] In the example of Figure 6, when the target ranging point is ranging point g1, the screen coordinates (u, v) of the corresponding point gs1 on the RGB image 51 of ranging point g1 are outside the edge area, so ranging point g1 is determined to not be the ranging point to be confirmed.

[0056] In contrast, when the target ranging point is ranging point g2, the screen coordinates (u, v) of the corresponding point gs2 on the RGB image 51 of ranging point g2 are within the edge area, so ranging point g2 is determined to be the ranging point to be confirmed.

[0057] <3. Position deviation correction process of the coordinate correction unit> Next, a description will be given of the positional deviation correction process that is executed by the coordinate corrector 32 when it is determined that the target distance measurement point is the distance measurement point to be confirmed.

[0058] FIG. 7 is a diagram illustrating the positional deviation correction process by the coordinate corrector 32 when it is determined that the distance measurement point K11 in the RGB image 51 shown in FIG. 4 is the distance measurement point to be confirmed.

[0059] Fig. 7 shows an enlarged area 61 obtained by enlarging the area surrounding the ranging point K11 in the RGB image 51 shown in Fig. 4. In addition to the ranging point K11, the enlarged area 61 also includes eight ranging points K21 to K28 located in the vicinity of the ranging point K11.

[0060] The coordinate corrector 32 executes positional deviation correction processing using MxM neighboring distance measurement points around the target distance measurement point K11. In the example of Fig. 7, M=3 is set, and positional deviation correction processing is executed using eight distance measurement points K21 to K28 around the target distance measurement point K11. Here, the position (position vector) of the distance measurement point K11 within the enlarged area 61 is defined as position a. Furthermore, the positions (position vectors) of the eight distance measurement points K21 to K28 are defined as positions b1 to b8, respectively.

[0061] In Figure 7, the histograms displayed near the nine distance measurement points K11 and K21 to K28 are conceptual representations of the distance histograms for each distance measurement point and are not part of the RGB image 51.

[0062] The enlarged area 61 includes an edge 62, and is divided by the edge 62 into an area 63 of the car 41 and an area 64 of the background 44. An area of ​​a predetermined width from the edge 62 is defined as an edge area 65.

[0063] The coordinate corrector 32 calculates a vector (ea) from the position a of the target distance measurement point K11 to the shortest position e of the edge 62 in the screen coordinate system of the RGB image 51.

[0064] The coordinate correction unit 32 calculates the similarity of the distance histogram between the target ranging point K11 and multiple ranging points nearby, and calculates a centroid vector of the similarity. In this example, the coordinate correction unit 32 calculates the similarity between the target ranging point K11 and eight surrounding ranging points K21 to K28 using a 3x3 array of nine ranging points centered around the target ranging point K11, and calculates a centroid vector of the similarity.

[0065] The distance histogram contains not only distance information for a single ranging point, but also additional information such as the reflectance of an object and ambient light, which are clues for determining the correspondence between surrounding ranging points and the correspondence between adjacent frames. By comparing distance histograms, the following information, for example, can be used to calculate the similarity: Movement along the distance attenuation line (distance change of the same subject) - Appearance of other peaks due to object boundaries or transparent objects such as glass Distortion of histogram waveform due to internal scattering Ambient light (detects the light source direction and uses shadow edge information) -Histogram waveform changes that occur at object boundaries

[0066] In this embodiment, as examples of calculating the similarity between distance histograms, (1) an example using correlation between distance histograms and (2) an example using distance in a manifold space will be described.

[0067] First, (1) a method for calculating similarity when the similarity is calculated using the correlation of distance histograms will be described.

[0068] Distance histogram H for the focus point K11 a consists of N bins, and the count value of the i-th bin is C ai(i=1,2,3,...,N), the distance histogram H a is H a ={C a1 ,C a2 ,···,C aN}. The distance histogram H of a predetermined distance measurement point K2b among the distance measurement points K21 to K28 in the vicinity of the target distance measurement point K11 is expressed as follows: b consists of N bins, and the count value of the i-th bin is C bi (i=1,2,3,...,N), the distance histogram H b is H b ={C b1 ,C b2 ,···,C bN}.

[0069] At this time, the distance histogram H a and the distance histogram H b Similarity r ab can be expressed by the following equation (1).

number

[0070] Similarity vector vr between the target focusing point K11 and its neighboring focusing point K2b ab is the similarity r in Eq. (1). ab Using this, it can be expressed by the following equation (2).

number

[0071] For all of the eight focus points K21 to K28 in the vicinity of the focus point K11, the similarity vector vr ab are calculated and then combined into a similar vector vr abi (i=1,2,··,8).

[0072] Similarity vectors vr between the target distance measurement point K11 and each of the eight neighboring distance measurement points K21 to K28 abi The center of gravity vector r of (i=1,2,··,8) is expressed by the following equation (3).

number

[0073] FIG. 7 shows the similarity vectors vr of eight distance measurement points K21 to K28 in the vicinity of the target distance measurement point K11. ab1 Or VR ab8 The centroid vector r of is shown.

[0074] Next, the coordinate correction unit 32 calculates the similarity vectors vr of the eight distance measurement points K21 to K28 in the vicinity of the target distance measurement point K11. ab1 Or VR ab8The angle θ formed by the center of gravity vector r of the target distance measurement point K11 and the vector (ea) pointing from the position a of the target distance measurement point K11 to the shortest position e of the edge 62 is calculated.

[0075] The coordinate correction unit 32 determines that the directions of the two vectors match when the angle θ between the center of gravity vector r and the vector (ea) is within a predetermined threshold, for example, -π / 2<θ<π / 2. Conversely, when the angle θ between the center of gravity vector r and the vector (ea) is greater than the predetermined threshold, the coordinate correction unit 32 determines that the directions of the two vectors do not match.

[0076] If the directions of the two vectors match, the coordinate corrector 32 determines that correction of the target ranging point K11 is necessary, and moves (corrects) the position a of the target ranging point K11 so that the target ranging point K11 straddles the edge 62. For example, the coordinate corrector 32 moves (corrects) the position a of the target ranging point K11 to position a' in FIG. 7, which is calculated as a' = a + 2(ea). Conversely, if the directions of the two vectors do not match, it is determined that correction of the target ranging point K11 is unnecessary, and the position a of the target ranging point K11 is not moved (corrected).

[0077] As described above, (1) the similarity vector vr calculated using the correlation of the distance histogram ab From this, the center of gravity vector r is calculated, and it is determined whether or not to correct the position a of the target distance measurement point K11, thereby making it possible to correct the positional deviation of the target distance measurement point.

[0078] Next, (2) a method for calculating similarity when the similarity is calculated using a distance in a manifold space will be described.

[0079] As shown in FIG. 8, the coordinate correction unit 32 calculates the distance histogram H of the target distance measurement point K11. a Peak count value C a and the count value (median) of ambient light D a Then, the coordinate correction unit 32 calculates the distance histogram H aThe count value of each bin is normalized by dividing it by the total count value, and converted into a probability density function, which is then used to calculate the distance histogram H a For example, the distance histogram H of the target focusing point K11 is a However, the average μ a , variance σ a 2 Normal distribution N(μ a ,σ a 2 ) is approximated as

[0080] The coordinate correction unit 32 calculates the distance histogram H of a predetermined distance measurement point K2b among the distance measurement points K21 to K28 in the vicinity of the target distance measurement point K11. b Similarly, the peak count value C b and the count value D of the ambient light b Calculate the distance histogram H and perform Gaussian fitting. b For example, the average μ b , variance σ b 2 Normal distribution N(μ b ,σ b 2 ) is approximated as

[0081] Next, as shown in FIG. 9, the coordinate corrector 32 calculates the target distance measurement point K11 and the nearby distance measurement point K2b by using the mean μ on the x-axis and the variance σ on the y-axis. 2 , and are mapped onto a non-Euclidean space with the z-axis being the peak count value C. Point P on the non-Euclidean space in FIG. 9 corresponds to the target distance measurement point K11, and p(x p )=N(μ a ,σ a 2 ), C p =C a , D p =D a Point Q corresponds to the target focus point K2b, and p(x q )=N(μ b ,σ b 2 ), C q =C b , D q =Db is.

[0082] Next, the coordinate corrector 32 calculates the pseudo distance L between the points P and Q in the non-Euclidean space corresponding to the target distance measurement point K11 and the distance measurement point K2b using the following equation (4).

number

[0083] That is, the pseudo-distance L between points P and Q in the non-Euclidean space corresponding to the target distance measurement point K11 and the distance measurement point K2b is calculated by the KL divergence between the two points P and Q and the peak count value C p and C q and the background light count value D p and D q It is calculated as a weighted sum of the differences.

[0084] Next, the coordinate correction unit 32 calculates the similarity vector vr between the target distance measurement point K11 and the nearby distance measurement point K2b. ab is calculated using the pseudorange L by the following equation (5). vr ab =1-(1 / L) ·····················(5)

[0085] The subsequent processing is the same as when the correlation of the distance histogram is used. That is, for all of the eight distance measurement points K21 to K28 in the vicinity of the target distance measurement point K11, the similarity vector vr ab is calculated, and the similarity vector vrabi (i=1, 2, . . . , 8) is calculated. Then, the center of gravity vector r is calculated using the above-mentioned formula (3), and the position a of the target distance measurement point K11 is moved (corrected) depending on whether the angle θ formed by the center of gravity vector r and the vector (ea) is within a predetermined threshold value.

[0086] When the similarity is calculated using the distance in the manifold space, the amount of data can be reduced compared to when the correlation of the distance histogram is used.

[0087] In the above example, M=3, and the eight distance measurement points K21 to K28 in the vicinity of the target distance measurement point K11 are used to determine whether or not to correct the target distance measurement point K11. However, it goes without saying that the number of nearby distance measurement points can be set arbitrarily.

[0088] <4. Position Information Calculation Process According to First Embodiment> Next, the position information calculation process performed by the signal processing system 1 according to the first embodiment will be described with reference to the flowchart in Fig. 10. This process starts, for example, when an RGB image and a distance histogram are supplied from the RGB camera 11 and the dToF sensor 12.

[0089] In the position information calculation process of FIG. 10, the coordinate correcting unit 32 will be described using an example of (1) the correlation between distance histograms as a method for calculating the similarity between distance histograms.

[0090] First, in step S1, the data acquisition unit 21 acquires an RGB image supplied from the RGB camera 11, and a distance histogram and camera attitude supplied from the dToF sensor 12. The data acquisition unit 21 supplies the acquired RGB image to the correction processing unit 23, and also supplies the acquired distance histogram and camera attitude to the distance calculation unit 22.

[0091] In step S2, distance calculation unit 22 calculates three-dimensional coordinates (x, y, z) for each distance measurement point based on the distance histogram and camera attitude from data acquisition unit 21. More specifically, distance calculation unit 22 detects a peak region of the count value from the histogram data of the multi-pixel MP corresponding to the spot light SP, and calculates three-dimensional coordinates (x, y, z) from the detected peak region and the camera attitude. The calculated three-dimensional coordinates (x, y, z) of each distance measurement point are supplied to correction processing unit 23 together with the distance histogram.

[0092] In step S3, the correction processing unit 23 determines a predetermined one of the distance measurement points included in one frame supplied from the dToF sensor 12 as a target distance measurement point, and the process proceeds to step S4.

[0093] In step S4, the confirmation target determination unit 31 of the correction processing unit 23 executes a confirmation target determination process. For example, the confirmation target determination unit 31 determines an edge area by detecting edges in the RGB image 51 and performing an expansion process, and determines whether the target ranging point is a ranging point to be confirmed based on whether the screen coordinates (u, v) of the target ranging point on the RGB image 51 are included in the edge area.

[0094] In step S5, the confirmation target determination unit 31 determines whether the target ranging point is a ranging point to be confirmed based on the result of the confirmation target determination process. If it is determined in step S5 that the target ranging point is not a ranging point to be confirmed, the process proceeds to step S6, and if it is determined that the target ranging point is a ranging point to be confirmed, the process proceeds to step S7.

[0095] In step S6, if it is determined that the target ranging point is not the ranging point to be confirmed, the confirmation target determining unit 31 supplies the three-dimensional coordinates (x, y, z) of the target ranging point calculated by the distance calculation unit 22 and the distance histogram to the storage unit 24 for storage. The RGB image supplied from the data acquisition unit 21 is also supplied to the storage unit 24 and stored therein.

[0096] On the other hand, in step S7, if it is determined that the target ranging point is the ranging point to be confirmed, the coordinate correction unit 32 calculates a vector (ea) from the position a of the target ranging point to the shortest position e of the edge 62 in the screen coordinate system of the RGB image 51.

[0097] In step S8, the coordinate correction unit 32 calculates the similarity (similarity vector) of the distance histogram between the target distance measurement point K11 and each of the multiple distance measurement points nearby, and calculates a centroid vector of the similarity. For example, using nine distance measurement points in a 3x3 array with the target distance measurement point K11 at the center, the similarity vector vr of the distance histogram between the target distance measurement point K11 and the eight distance measurement points K21 to K28 nearby is calculated as follows: ab1 Or VR ab8 is calculated, and the similarity vector vr ab1 Or VR ab8 The centroid vector r of is calculated.

[0098] In step S9, the coordinate correction unit 32 determines whether the directions of two vectors match: the centroid vector r of the similarities of the multiple distance measurement points near the target distance measurement point, and the vector (ea) pointing from the target distance measurement point to the shortest position e of the edge 62. In step S9, for example, if the angle θ formed by the centroid vector r and the vector (ea) is within a predetermined threshold, it is determined that the directions of the two vectors match; if the angle θ is equal to or greater than the predetermined threshold, it is determined that the directions of the two vectors do not match.

[0099] If it is determined in step S9 that the directions of the two vectors do not match, the process proceeds to step S6 described above. Therefore, in this case, the three-dimensional coordinates (x, y, z) of the target ranging point are not corrected and are stored as they are in the storage unit 24.

[0100] On the other hand, if it is determined in step S9 that the directions of the two vectors match, the process proceeds to step S10. In step S10, the coordinate correcting unit 32 moves the position a of the target ranging point so that the target ranging point straddles the edge 62, and stores the three-dimensional coordinates (x, y, z) after the movement in the memory unit 24 as the corrected three-dimensional coordinates (x, y, z). The distance histogram of the target ranging point and the RGB image supplied from the data acquiring unit 21 are also supplied to the memory unit 24 and stored therein.

[0101] After step S6 or S10, in step S11, the correction processing unit 23 determines whether all ranging points in one frame supplied from the dToF sensor 12 have been set as ranging points of interest. If it is determined in step S11 that all ranging points in one frame have not yet been set as ranging points of interest, the process returns to step S3, and the above-mentioned steps S3 to S11 are repeated. That is, a ranging point in one frame that has not yet been set as a ranging point of interest is set as the next ranging point of interest, and it is determined whether or not it is the ranging point to be confirmed. If it is determined that the directions of the two vectors match, the three-dimensional coordinates (x, y, z) of the ranging point of interest are corrected (moved).

[0102] On the other hand, if it is determined in step S11 that all ranging points in one frame have been set as target ranging points, the process proceeds to step S12, where the signal processing device 13 determines whether to end the process. For example, the process of steps S1 to S11 described above is performed for the ranging points in all frames supplied from the dToF sensor 12, and if a distance histogram for the next frame is not supplied from the dToF sensor 12, the signal processing device 13 determines to end the process. Conversely, if a distance histogram for the next frame is supplied from the dToF sensor 12, the signal processing device 13 determines not to end the process.

[0103] If it is determined in step S12 that the process is not yet to be ended, the process returns to step S1, and the processes of steps S1 to S12 described above are repeated.

[0104] On the other hand, if it is determined in step S12 that the processing is to be ended, the processing proceeds to step S13, and the correction processing unit 23 supplies an end notification to the output unit 25.

[0105] In step S14, the output unit 25 outputs the three-dimensional coordinates (x, y, z) of each ranging point for all frames stored in the storage unit 24, and ends the position information calculation process in Fig. 10. The three-dimensional coordinates (x, y, z) of each ranging point for each output frame include those that have been corrected and those that have not been corrected.

[0106] According to the position information calculation process of the signal processing system 1 according to the first embodiment described above, whether or not the target ranging point is the ranging point to be confirmed is determined based on whether or not the target ranging point is included in an edge area. If the target ranging point is determined to be the ranging point to be confirmed, the similarity (similarity vector) between the distance histogram of the target ranging point and the distance histograms of multiple nearby ranging points is calculated, and the position of the target ranging point is corrected to a nearby ranging point with a similar distance histogram. This makes it possible to correct any deviation in the position information of the dToF sensor 12.

[0107] <5. Second embodiment of signal processing system> FIG. 11 is a block diagram showing a configuration example of the second embodiment of the signal processing system of the present disclosure.

[0108] In the second embodiment of FIG. 11, parts corresponding to those in the first embodiment shown in FIG. 1 are given the same reference numerals, and the description of those parts will be omitted as appropriate.

[0109] The signal processing system 1 according to the second embodiment includes an RGB camera 11, a dToF sensor 12, and a signal processing device 13. The signal processing device 13 includes a data acquisition unit 21, a distance calculation unit 22, a correction processing unit 23, a storage unit 24, and an output unit 25. The correction processing unit 23 includes a confirmation target determination unit 71 and a coordinate correction unit 32.

[0110] That is, in the signal processing system 1 according to the second embodiment, the confirmation target determination unit 31 of the correction processing unit 23 in the first embodiment is changed to a confirmation target determination unit 71, and the other configurations are the same as those of the first embodiment. In other words, the second embodiment differs from the first embodiment in the determination method for determining whether or not the target ranging point is the ranging point to be confirmed, but the other points are the same as those of the first embodiment.

[0111] In the first embodiment described above, an edge area is extracted from an RGB image as an area near an object boundary, and whether or not the target ranging point is highly likely to be misaligned is determined depending on whether or not it is included in the edge area. In contrast, the confirmation target determination unit 71 in the second embodiment determines whether or not the target ranging point is in an area near an object boundary by observing changes in the distance histogram over multiple frames. If the target ranging point is determined to be in an area near an object boundary, the confirmation target determination unit 71 determines that the target ranging point is a ranging point to be confirmed, as it is a ranging point highly likely to be misaligned.

[0112] 12, consider a case where an object 81 in the background and an object 82 in the foreground are present as subjects, and a spot light SP is irradiated onto the boundary between the objects 81 and 82. Assume that the object 82 in the foreground is a moving subject, and is moving in the left direction as indicated by the arrow.

[0113] The distance histogram output by the dToF sensor 12 includes two peak regions: a peak region h1 corresponding to the object 81 and a peak region h2 corresponding to the object 82. If the area where the spot light SP hits the object 82 is larger than the area where the spot light SP hits the object 81, the peak count value in the peak region h2 corresponding to the object 82 will be larger than the peak count value in the peak region h1 corresponding to the object 81. If the area where the spot light SP hits the object 81 becomes larger than the area where the spot light SP hits the object 82 due to movement of the object 82, the peak count value in the peak region h1 corresponding to the object 81 will be larger than the peak count value in the peak region h2 corresponding to the object 82. Furthermore, although not shown in the figure, if the spot light SP is not irradiated onto the boundary between the objects 81 and 82 but onto only one of the objects 81 or 82, the distance histogram output by the dToF sensor 12 will be a histogram with one peak region with a large peak count value.

[0114] Therefore, the confirmation target determination unit 71 observes distance histograms over multiple frames, and determines that a distance measurement point where the number of peak areas in the distance histogram changes from 1 to 2 to 1 is a distance measurement point to be confirmed. Note that distance measurement points where the number of peak areas changes from 1 to 2 to 1 may change from "background to boundary between foreground and background to foreground" or from "foreground to boundary between foreground and background to background."

[0115] FIG. 13 is a diagram illustrating the confirmation target determination process performed by the confirmation target determination unit 71. In FIG.

[0116] First, the confirmation target determination unit 71 determines whether the number of peak areas in the distance histogram of the target ranging point Kx in the frame Ft at the current time t is 1 or 2 or more. If the number of peak areas in the distance histogram of the target ranging point Kx is 2 or more, the target ranging point is determined to be the ranging point to be confirmed, as the target ranging point is likely to be an object boundary and is therefore likely to be misaligned.

[0117] On the other hand, if the number of peak areas in the distance histogram of the target ranging point Kx is one, the distance histogram of the target ranging point Kx at the same position in W past frames (W is an integer greater than or equal to 2) is obtained from the storage unit 24, and the number of peak areas in the distance histogram in the past W frames and the bin position with the maximum count value are confirmed. If the number of peak areas in the distance histogram in the past W frames is one and the bin position with the maximum count value is also the same, the confirmation target determination unit 71 determines that the target ranging point is unlikely to be misaligned, and that the target ranging point is not the ranging point to be confirmed.

[0118] On the other hand, if the number of peak areas in the distance histogram in the past W frame is two or more and the bin position of the maximum count value has changed, the confirmation target determination unit 71 determines that the target ranging point is the ranging point to be confirmed, as it is a ranging point that is likely to have a positional shift.

[0119] In the example of Figure 13, since the number of peak areas in the distance histogram of the target focusing point Kx in frame Ft at the current time t is one, the distance histogram of the focusing point Kx at the same position in the past W frame is obtained from memory unit 24, and the number of peak areas in the distance histogram in the past W frame and the bin position of the maximum count value are confirmed.

[0120] Now, assuming W=3, and the number of peak regions and the bin position of the maximum count value in the distance histogram for the past three frames are confirmed, the number of peak regions is two for frame Ft-1 at time t-1 and frame Ft-2 at time t-2, and the bin position of the maximum count value changes between frame Ft-2 at time t-2 and frame Ft-1 at time t-1. In the distance histogram for each frame shown in Figure 13, the peak of the maximum count value is marked with a circle.

[0121] Therefore, the confirmation target determination unit 71 determines that the target distance measurement point Kx in the frame Ft at the current time t is a distance measurement point where there is a high possibility that a positional shift has occurred, and is a distance measurement point to be confirmed.

[0122] In the second embodiment, as described above, by observing changes in the distance histogram over the past W frames, it is determined whether the target ranging point is in an area near the object boundary, and whether it is the ranging point to be confirmed. Therefore, the position of each multi-pixel MP of the dToF sensor 12 does not change in all frames during the shooting period, and the same position must be sampled in each frame.

[0123] <6. Position Information Calculation Process According to Second Embodiment> The position information calculation process by the signal processing system 1 of the second embodiment differs in the determination method of the confirmation target determination process performed in step S4 of the flowchart of Figure 10 described in the first embodiment, but the other processes of steps S1 to S3 and steps S5 to S14 are the same.

[0124] Explaining this with reference to the flowchart in Fig. 10, in step S4 in Fig. 10, the confirmation target determination unit 71 determines whether the number of peak areas in the distance histogram of the target ranging point Kx in frame Ft at current time t is 1 or 2 or more. If the number of peak areas in the distance histogram of the target ranging point Kx is 2 or more, the confirmation target determination unit 71 determines that the target ranging point is the ranging point to be confirmed.

[0125] On the other hand, if the number of peak areas in the distance histogram of the target ranging point Kx in frame Ft at current time t is one, the confirmation target determination unit 71 further checks the number of peak areas in the distance histogram in the past W frame and the bin position with the largest count value. If the number of peak areas in the distance histogram in the past W frame is one and the bin position with the largest count value is also the same, the confirmation target determination unit 71 determines that the target ranging point Kx is not the target ranging point to be confirmed. On the other hand, if the number of peak areas in the distance histogram in the past W frame is two and the bin position with the largest count value is also swapped, the confirmation target determination unit 71 determines that the target ranging point Kx is the target ranging point to be confirmed.

[0126] According to the position information calculation process of the signal processing system 1 according to the second embodiment described above, it is determined whether the target ranging point is the ranging point to be confirmed based on whether the number of peak areas in the distance histogram of the target ranging point and the bin position of the maximum count value have changed in the past W frames. If it is determined that the target ranging point is the ranging point to be confirmed, the similarity (similarity vector) between the distance histogram of the target ranging point and distance histograms of multiple nearby ranging points is calculated, and the position of the target ranging point is corrected to a nearby ranging point with a similar distance histogram. This makes it possible to correct the deviation in the position information of the dToF sensor 12.

[0127] In the second embodiment, when the RGB camera 11 and the dToF sensor 12 are fixed in position and capture an image of a subject, a ranging point where the number of peak regions in the distance histogram changes from "1 to 2 to 1" is a point where a moving subject is captured, and the reliability of the calculated three-dimensional coordinates is expected to be low. Therefore, when the positions of the RGB camera 11 and the dToF sensor 12 are fixed, for ranging points where the number of peak regions in the distance histogram changes from "1 to 2 to 1," the measured values ​​of the three-dimensional coordinates may be deleted and not output from the signal processing device 13.

[0128] Alternatively, a reliability may be added to a ranging point where the number of peak areas in the distance histogram changes from "1 to 2 to 1" and output, allowing a downstream device to identify the ranging point as having low reliability. The reliability of a ranging point can be calculated by subtracting the reciprocal of the number of frames required for the distance histogram peak to change, such as when the number of peak areas in the distance histogram changes from "1 to 2 to 1," "1 to 2 to 2 to 1," or "1 to 2 to 2 to 2 to 1," from 1. Specifically, the reliability of a ranging point where the number of peak areas in the distance histogram changes from "1 to 2 to 1" can be calculated as "1-1 / 3 = 0.6." The reliability of a ranging point where the number of peak areas in the distance histogram changes from "1 to 2 to 2 to 1" can be calculated as "1-1 / 4 = 0.75." For example, the reliability of a distance measurement point where the number of peak regions in the distance histogram changes from "1 to 2 to 2 to 2 to 1" can be calculated as "1-1 / 5=0.8".

[0129] In a downstream device that acquires a ranging point with added reliability, the ranging point is recognized as a ranging point with low reliability, and processing can be performed to prevent afterimages, for example.

[0130] <7. Summary> The signal processing device 13 includes a data acquisition unit 21 that acquires a distance histogram, which is histogram data of the flight time of irradiated light at a predetermined ranging point of the ranging sensor; a distance calculation unit 22 that calculates the three-dimensional coordinates of the predetermined ranging point from the distance histogram of the predetermined ranging point; a confirmation target determination unit 31 that determines whether the predetermined ranging point is a ranging point to be checked for positional deviation; and a coordinate correction unit 32 that, if the predetermined ranging point is a ranging point to be checked for positional deviation, performs a positional deviation correction process to correct the three-dimensional coordinates of the predetermined ranging point calculated by the distance calculation unit 22 based on the determination result of the similarity between the distance histogram of the predetermined ranging point and the distance histograms of ranging points near the predetermined ranging point.

[0131] The confirmation target determination unit 31 of the first embodiment detects the edge area of ​​an object from the RGB image (captured image) of the subject captured by the RGB camera 11, and if a specified ranging point is included in the edge area, determines that the ranging point to be confirmed is likely to be misaligned.

[0132] On the other hand, the confirmation target determination unit 31 in the second embodiment determines whether a specific ranging point is a ranging point for which positional deviation confirmation is to be performed by observing changes in the distance histogram of the ranging point over multiple frames. For example, the confirmation target determination unit 31 determines that the ranging point is a ranging point for which confirmation is to be performed when the number of peak areas in the distance histogram changes from 1 to 2 to 1 and the bin position with the maximum count value changes.

[0133] If the centroid vector of the similarity between the distance histogram of the ranging point to be confirmed and the distance histogram of a ranging point near that ranging point matches the direction of the object boundary, the coordinate correction unit 32 determines that correction of the three-dimensional coordinate of the ranging point is necessary, and corrects (moves) the three-dimensional coordinate of the ranging point.

[0134] According to the above-described confirmation target determination process and position deviation correction process of the signal processing device 13, the deviation of the position information of the dToF sensor 12 can be corrected in the correct direction.

[0135] The signal processing device 13 may have the configuration and functions of only one of the first embodiment or the second embodiment described above, or may have the configuration and functions of both and, for example, may selectively perform one of the processes by switching between a first operation mode corresponding to the first embodiment and a second operation mode corresponding to the second embodiment.

[0136] <Application examples of this technology> The position information calculation process of the present disclosure, which can correct and output three-dimensional coordinates using histogram data acquired from the dToF sensor 12, can be applied to three-dimensional measurement in various applications, such as SLAM (Simultaneous Localization and Mapping), which simultaneously estimates self-position and creates an environmental map, robot operations that grasp objects and move or work, CG (computer graphics) modeling when generating virtual scenes and objects using CG, object recognition processing, and object classification processing. Applying the correction process of the present disclosure can improve the measurement accuracy of the three-dimensional coordinates of an object.

[0137] <8. Computer configuration example> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes microcomputers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0138] FIG. 14 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0139] In the computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0140] An input / output interface 105 is further connected to the bus 104. To the input / output interface 105, an input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected.

[0141] The input unit 106 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 107 includes a display, speaker, output terminal, etc. The storage unit 108 includes a hard disk, RAM disk, non-volatile memory, etc. The communication unit 109 includes a network interface, etc. The drive 110 drives a removable recording medium 111 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0142] In the computer configured as above, the CPU 101 performs the above-described series of position information calculation processes by, for example, loading a program stored in the storage unit 108 into the RAM 103 via the input / output interface 105 and the bus 104 and executing the program. The RAM 103 also stores data necessary for the CPU 101 to execute various processes as appropriate.

[0143] The program executed by the computer (CPU 101) can be provided by being recorded on a removable recording medium 111 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0144] In a computer, the program can be installed in the storage unit 108 via the input / output interface 105 by inserting the removable recording medium 111 into the drive 110. The program can also be received by the communication unit 109 via a wired or wireless transmission medium and installed in the storage unit 108. Alternatively, the program can be installed in the ROM 102 or the storage unit 108 in advance.

[0145] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0146] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0147] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0148] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0149] The technology of the present disclosure can have the following configurations. (1) an acquisition unit that acquires a distance histogram, which is histogram data of the time of flight of irradiated light at a predetermined distance measurement point of the distance measurement sensor; a determination unit that determines whether the predetermined distance measurement point is a distance measurement point that is a target for positional deviation confirmation; a correction unit that, when the predetermined distance measurement point is a distance measurement point to be checked for positional deviation, executes a correction process to correct the three-dimensional coordinates of the predetermined distance measurement point calculated from the distance histogram based on a determination result of the similarity between the distance histogram of the predetermined distance measurement point and the distance histogram of a distance measurement point in the vicinity of the predetermined distance measurement point; A signal processing device comprising: (2) The correction unit executes the correction process when it is determined that a centroid vector of similarity between the distance histogram of the predetermined distance measurement point and the distance histogram of a distance measurement point in the vicinity of the predetermined distance measurement point matches a direction of an object boundary. The signal processing device according to (1) above. (3) When it is determined that the centroid vector of the similarity coincides with the direction of the object boundary, the correction unit corrects the three-dimensional coordinates of the predetermined distance measurement point to a position that straddles the object boundary. The signal processing device according to (2) above. (4) The correction unit calculates a centroid vector of the similarity from the distance histogram of the predetermined distance measurement point and the distance histograms of a plurality of distance measurement points in the vicinity of the predetermined distance measurement point. The signal processing device according to (2) above. (5) The correction unit calculates a similarity between the distance histogram of the predetermined distance measurement point and the distance histogram of one of the neighboring distance measurement points from a difference between a position vector of the predetermined distance measurement point and a position vector of the neighboring distance measurement point, and a correlation coefficient between the distance histogram of the predetermined distance measurement point and the distance histogram of the neighboring distance measurement point. The signal processing device according to (4) above. (6) The correction unit calculates a similarity between the distance histogram of the predetermined distance measurement point and the distance histogram of one of the neighboring distance measurement points using a distance between probability density functions that approximate the distance histogram of the predetermined distance measurement point and the distance histogram of the neighboring distance measurement point. The signal processing device according to (4) above. (7) The determination unit determines whether the three-dimensional coordinates of the predetermined distance measurement point indicated by the distance histogram are in an area near an object boundary, and if the three-dimensional coordinates of the predetermined distance measurement point are in an area near an object boundary, determines that the predetermined distance measurement point is a distance measurement point to be checked for positional deviation. The signal processing device according to any one of (1) to (6). (8) The determination unit detects an edge area of ​​an object included in a captured image of a subject, and when the three-dimensional coordinates of the predetermined distance measurement point are included in the edge area, determines that the three-dimensional coordinates of the predetermined distance measurement point are in an area near the object boundary. The signal processing device according to (7) above. (9) The acquisition unit also acquires the captured image captured in the same range as the range sensor, The determination unit detects an edge area of ​​an object included in the captured image. The signal processing device according to (8) above. (10) The determination unit detects an area where movement is detected in a captured image of a subject as an edge area, and determines that the three-dimensional coordinates of the predetermined distance measurement point are in the area near the object boundary when the three-dimensional coordinates of the predetermined distance measurement point are included in the edge area. The signal processing device according to (7) above. (11) The determination unit determines whether the three-dimensional coordinates of the predetermined distance measurement point are in the area near the object boundary by observing changes in the distance histogram over a plurality of frames. The signal processing device according to (7) above. (12) The determining unit determines that the three-dimensional coordinates of the predetermined distance measurement point are in the area near the object boundary when the number of peak areas of the distance histogram changes in the plurality of frames. The signal processing device according to (11) above. (13) The determination unit determines that the three-dimensional coordinates of the predetermined distance measurement point are in the area near the object boundary when the bin position of the maximum count value of the distance histogram changes in the plurality of frames. The signal processing device according to (11) or (12). (14) a calculation unit that calculates three-dimensional coordinates of the predetermined distance measurement point from the distance histogram of the predetermined distance measurement point; The signal processing device according to any one of (1) to (13). (15) The signal processing device A distance histogram is obtained, which is histogram data of the time of flight of the irradiated light at a predetermined distance measurement point of the distance measurement sensor; determining whether the predetermined distance measurement point is a distance measurement point to be checked for positional deviation; When the predetermined distance measurement point is a distance measurement point for which positional deviation is to be confirmed, a correction process is performed to correct the three-dimensional coordinates of the predetermined distance measurement point calculated from the distance histogram based on a result of determining the similarity between the distance histogram of the predetermined distance measurement point and the distance histogram of a distance measurement point in the vicinity of the predetermined distance measurement point. Signal processing methods. [Explanation of symbols]

[0150] 1 signal processing system, 11 RGB camera, 12 dToF sensor, 13 signal processing device, 21 data acquisition unit, 22 distance calculation unit, 23 correction processing unit, 24 memory unit, 25 output unit, 31 confirmation target determination unit, 32 coordinate correction unit, 71 confirmation target determination unit, 101 CPU, 102 ROM, 104 bus, 105 input / output interface, 106 input unit, 107 output unit, 108 memory unit, 109 communication unit, 110 drive, 111 removable recording medium

Claims

1. an acquisition unit that acquires a distance histogram, which is histogram data of the time of flight of irradiated light at a predetermined distance measurement point of the distance measurement sensor, and an image captured by the distance measurement sensor in the same range; a first determination unit that determines whether the predetermined distance measurement point is within an edge area of ​​a predetermined width from an edge of an object in the captured image, thereby determining whether the predetermined distance measurement point is a distance measurement point that is a target for confirmation of positional deviation; a second determination unit that, when the predetermined distance measurement point is a distance measurement point to be checked for positional deviation, calculates a similarity between the distance histogram of the predetermined distance measurement point and the distance histogram of a distance measurement point nearby the predetermined distance measurement point, performs a calculation for a plurality of distance measurement points nearby the predetermined distance measurement point by multiplying a difference between a position vector of the predetermined distance measurement point and a position vector of the nearby distance measurement point by the similarity to calculate a center of gravity vector of the similarity vectors of the plurality of distance measurement points, calculates an angle formed by the center of gravity vector and a vector extending from the position of the predetermined distance measurement point to the shortest position of the edge, and determines whether the angle is greater than a predetermined threshold value; a correction unit that executes a correction process to correct three-dimensional coordinates of the predetermined distance measurement point so that the predetermined distance measurement point straddles the edge when the angle is within a predetermined threshold value; A signal processing device comprising:

2. An acquisition unit that acquires a distance histogram, which is histogram data of the time of flight of irradiated light at a predetermined distance measurement point of a distance measurement sensor, and an image captured by the same range as the distance measurement sensor; a first determination unit that determines whether the predetermined distance measurement point is within an edge area of ​​a predetermined width from an edge of an object to be measured, thereby determining whether the predetermined distance measurement point is a distance measurement point to be checked for positional deviation; a second determination unit that, when the predetermined distance measurement point is a distance measurement point to be checked for positional deviation, approximates the distance histogram of the predetermined distance measurement point and the distance histograms of distance measurement points nearby the predetermined distance measurement point to a probability density function, maps the predetermined distance measurement point and the nearby distance measurement points into a non-Euclidean space with the x-axis representing the mean of the probability density function, the y-axis representing the variance of the probability density function, and the z-axis representing the peak count value of the distance histogram to calculate pseudo distances in the non-Euclidean space, and calculates similarity vectors of the predetermined distance measurement point and the nearby distance measurement points using the pseudo distances, for a plurality of distance measurement points nearby the predetermined distance measurement point, determines a centroid vector of the similarity vectors of the plurality of distance measurement points, calculates an angle formed by the centroid vector and a vector extending from the position of the predetermined distance measurement point to the shortest position of the edge, and determines whether the angle is greater than a predetermined threshold; a correction unit that executes a correction process to correct three-dimensional coordinates of the predetermined distance measurement point so that the predetermined distance measurement point straddles the edge when the angle is within a predetermined threshold value; A signal processing device comprising:

3. The first determination unit detects an edge of the object included in the captured image and determines the edge area.

3. The signal processing device according to claim 1.

4. The first determination unit detects an area in which movement is detected in the captured image as the edge area.

3. The signal processing device according to claim 1.

5. The first determination unit, instead of determining whether the predetermined distance measurement point is within the edge area, determines whether the three-dimensional coordinates of the predetermined distance measurement point are in the area near the object boundary by observing changes in the distance histogram over a plurality of frames, thereby determining whether the predetermined distance measurement point is a distance measurement point to be checked for positional deviation.

3. The signal processing device according to claim 1.

6. The first determination unit determines that the three-dimensional coordinates of the predetermined distance measurement point are in the area near the object boundary when the number of peak areas of the distance histogram changes in the plurality of frames. The signal processing device according to claim 5 .

7. The first determination unit determines that the three-dimensional coordinates of the predetermined distance measurement point are in the area near the object boundary when a bin position of a maximum count value of the distance histogram changes in the plurality of frames. The signal processing device according to claim 5 .

8. a calculation unit that calculates three-dimensional coordinates of the predetermined distance measurement point from the distance histogram of the predetermined distance measurement point; 3. The signal processing device according to claim 1.

9. The signal processing device Obtaining a distance histogram, which is histogram data of the time of flight of irradiated light at a predetermined distance measurement point of the distance measurement sensor, and an image captured by capturing the same range as the distance measurement sensor; determining whether the predetermined distance measurement point is within an edge area of ​​a predetermined width from an edge of an object in the photographed image, thereby determining whether the predetermined distance measurement point is a distance measurement point to be checked for positional deviation; If the predetermined distance measurement point is a distance measurement point to be checked for positional deviation, a similarity between the distance histogram of the predetermined distance measurement point and the distance histogram of a distance measurement point nearby the predetermined distance measurement point is calculated, and a similarity vector is calculated by multiplying the difference between the position vector of the predetermined distance measurement point and the position vector of the nearby distance measurement point by the similarity, for a plurality of distance measurement points nearby the predetermined distance measurement point; a centroid vector of the similarity vectors of the plurality of distance measurement points is calculated; an angle formed by the centroid vector and a vector extending from the position of the predetermined distance measurement point to the shortest position of the edge is calculated; and it is determined whether the angle is greater than a predetermined threshold value; If the angle is within a predetermined threshold, a correction process is performed to correct the three-dimensional coordinates of the predetermined distance measurement point so that the predetermined distance measurement point straddles the edge. A signal processing method comprising:

10. A signal processing device comprising: Obtaining a distance histogram, which is histogram data of the time of flight of irradiated light at a predetermined distance measurement point of the distance measurement sensor, and an image captured in the same range as the distance measurement sensor; determining whether the predetermined distance measurement point is within an edge area of ​​a predetermined width from an edge of the object to be measured, thereby determining whether the predetermined distance measurement point is a distance measurement point to be checked for positional deviation; If the predetermined distance measurement point is a distance measurement point to be checked for positional deviation, the distance histogram of the predetermined distance measurement point and the distance histograms of distance measurement points nearby the predetermined distance measurement point are approximated to a probability density function, the predetermined distance measurement point and the nearby distance measurement points are mapped into a non-Euclidean space with the x-axis representing the mean of the probability density function, the y-axis representing the variance of the probability density function, and the z-axis representing the peak count value of the distance histogram to calculate pseudo distances in the non-Euclidean space, and similarity vectors of the predetermined distance measurement point and the nearby distance measurement points are calculated using the pseudo distances, and this is performed for a plurality of distance measurement points nearby the predetermined distance measurement point, a centroid vector of the similarity vectors of the plurality of distance measurement points is obtained, an angle formed by the centroid vector and a vector extending from the position of the predetermined distance measurement point to the shortest position of the edge is calculated, and it is determined whether the angle is greater than a predetermined threshold value; If the angle is within a predetermined threshold, a correction process is performed to correct the three-dimensional coordinates of the predetermined distance measurement point so that the predetermined distance measurement point straddles the edge. A signal processing method comprising:

Images