Information processing method, information processing device, and program

Abstract

The present disclosure relates to: an information processing method that makes it possible to improve range finding accuracy by excluding the influence of artifacts; an information processing device; and a program. This information processing device (1): determines whether there are a plurality of peaks in a histogram for a first pixel (pixel of interest) of a first range finding sensor (11) in histogram data for each pixel output from the first range finding sensor (11); if there are a plurality of peaks (peaks PK1-PK3) in the histogram for the first pixel, uses image data acquired from a camera (12) or a second range finding sensor to select at least one second pixel (reference pixel) of the first range finding sensor (11), the at least one second pixel being present in an area corresponding to that of the first pixel (pixel of interest); selects one peak (true peak PK2) of the plurality of peaks in the histogram for the first pixel on the basis of a peak (true peak PK11) in the histogram for the at least one second pixel; and determines a distance corresponding to the selected peak as the distance of the first pixel (pixel of interest). The technology of the present disclosure can be applied to, for example, an information processing device for generating a depth image.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program, and more particularly to an information processing method, an information processing device, and a program that can improve distance measurement accuracy by eliminating the influence of artifacts.

[0002] A direct ToF (Time-of-Flight) distance sensor measures the distance to an object by measuring the time it takes for irradiated light to be reflected by the object and return. To generate a single measurement value, the process of irradiating light and receiving reflected light is repeated a predetermined number of times. A histogram of the time it takes for the reflected light to be received is then generated, and the distance corresponding to the peak of the histogram is taken as the distance to the object.

[0003] When a highly reflective object is present within the measurement range of a ranging sensor, the reflected light can be so strong that it causes a phenomenon called flare in the captured image. Flare is a phenomenon in which strong light causes an image to appear whitish and low in contrast, but there are also other phenomena called ghosting and blooming. These are sometimes difficult to distinguish clearly, and these false images are collectively referred to as artifacts. Artifacts such as flare cause histogram peaks (false peaks) at positions (distances) where no object actually exists, obscuring the peak of the actual object (true peak), making accurate distance measurement difficult.

[0004] For example, Patent Document 1 discloses a technique for segmenting images in medical ultrasound diagnostics using machine learning and removing artifacts.

[0005] JP 2016-221264 A

[0006] The artifact removal technique of Patent Document 1 requires prior learning of the occurrence of artifacts, and is difficult to apply to subjects for which it is difficult to predict artifacts in advance.

[0007] The present disclosure has been made in view of such circumstances, and aims to improve distance measurement accuracy by eliminating the influence of artifacts.

[0008] An information processing method of one aspect of the present disclosure includes: determining whether there are multiple peaks in the histogram of a first pixel of a first ranging sensor, using histogram data for each pixel output from the first ranging sensor; if there are multiple peaks in the histogram of the first pixel, using image data acquired from a camera or a second ranging sensor, selecting at least one second pixel of the first ranging sensor that is present in an area corresponding to the first pixel; selecting one of the multiple peaks in the histogram of the first pixel based on the peaks in the histogram of one or more of the second pixels; and determining the distance corresponding to the selected peak as the distance of the first pixel.

[0009] An information processing device according to one aspect of the present disclosure includes: a determination unit that determines whether the histogram of a first pixel of a first ranging sensor has multiple peaks, based on histogram data for each pixel output from the first ranging sensor; a selection unit that, if the histogram of the first pixel has multiple peaks, selects at least one second pixel of the first ranging sensor that exists in an area corresponding to the first pixel, using image data acquired from a camera or a second ranging sensor; and a distance determination unit that selects one of the multiple peaks in the histogram of the first pixel based on the peaks in the histogram of one or more of the second pixels, and determines the distance corresponding to the selected peak as the distance of the first pixel.

[0010] A program according to one aspect of the present disclosure causes a computer to perform the following processes: determine, for histogram data for each pixel output from a first ranging sensor, whether the histogram of a first pixel of the first ranging sensor has multiple peaks; if the histogram of the first pixel has multiple peaks, select at least one second pixel of the first ranging sensor that exists in an area corresponding to the first pixel using image data acquired from a camera or a second ranging sensor; select one of the multiple peaks in the histogram of the first pixel based on the peaks in the histogram of one or more of the second pixels; and determine the distance corresponding to the selected peak as the distance of the first pixel.

[0011] In one aspect of the present disclosure, histogram data for each pixel output from a first ranging sensor is determined to determine whether there are multiple peaks in the histogram of a first pixel of the first ranging sensor, and if there are multiple peaks in the histogram of the first pixel, at least one second pixel of the first ranging sensor that exists in an area corresponding to the first pixel is selected using image data acquired from a camera or a second ranging sensor, and one of the multiple peaks in the histogram of the first pixel is selected based on the peaks in the histogram of one or more of the second pixels, and the distance corresponding to the selected peak is determined as the distance of the first pixel.

[0012] The program can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0013] The information processing device may be an independent device or an internal block constituting a single device.

[0014] FIG. 1 is a block diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating processing of a depth generation unit. FIG. 3 is a diagram illustrating processing of the depth generation unit. FIG. 4 is a block diagram illustrating a detailed configuration example of a depth generation unit. FIG. 5 is a flowchart illustrating point cloud data generation processing. FIG. 6 is a flowchart illustrating depth image generation processing. FIG. 7 is a diagram illustrating a modified example of reference pixel selection. FIG. 8 is a flowchart illustrating a modified example of depth image generation processing. FIG. 9 is a block diagram illustrating a configuration example of a moving body according to an embodiment of the present disclosure. FIG. 10 is a block diagram illustrating a configuration example of an embodiment of a computer to which the technology of the present disclosure is applied.

[0015] Hereinafter, a description will be given of a mode for carrying out the technology of the present disclosure (hereinafter referred to as an embodiment) with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are assigned the same reference numerals to avoid redundant description. The description will be given in the following order: 1. Example of the configuration of an information processing device 2. Description of the depth generation unit 3. Detailed configuration block diagram of the depth generation unit 4. Flowchart of point cloud data generation processing 5. Flowchart of depth image generation processing 6. Other examples of depth image generation processing 7. Summary of information processing device 8. Example of application to a moving body 9. Example of computer configuration

[0016] 1. Configuration Example of Information Processing Apparatus> FIG. 1 is a block diagram showing a configuration example of an information processing apparatus according to an embodiment of the present disclosure.

[0017] The information processing device 1 in FIG. 1 is a device that performs processing to generate point cloud data, which is three-dimensional position information of a subject (object), based on histogram data output as distance measurement data from a distance measurement sensor 11. The information processing device 1 generates point cloud data in which RGB color information is added to the three-dimensional position of the subject, using an RGB image, which is an image captured by an RGB camera 12. The information processing device 1 also generates point cloud data with improved accuracy by removing the effects of artifacts such as flare from the distance measurement data from the distance measurement sensor 11, using the RGB image captured by the RGB camera 12. The point cloud data, which is three-dimensional position information of the subject, can be used, for example, to estimate the self-position of a moving object, create an environmental map, and so on, as will be described later with reference to FIG. 9 .

[0018] In this embodiment, calibration of the distance measurement sensor 11 and the RGB camera 12 has been completed, and the correspondence between the pixels of the distance measurement sensor 11 and the pixels of the RGB camera 12 is known. The shooting range of the RGB camera 12 includes at least the measurement range of the distance measurement sensor 11. The shooting range of the RGB camera 12 may be adjusted to match the measurement range of the distance measurement sensor 11, and for simplicity, it is assumed below that the shooting range of the RGB camera 12 and the measurement range of the distance measurement sensor 11 are the same.

[0019] The information processing device 1 includes a depth generation unit 21 , an area division unit 22 , a point cloud data generation unit 23 , a control unit 24 , and a storage unit 25 .

[0020] The ranging sensor 11 is, for example, a depth sensor that measures the distance to a subject using a direct ToF method, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), or the like. The sensor emits laser light and measures the time it takes to receive the reflected light, thereby generating ranging data representing the distance to the subject (object being measured). The generated ranging data is output to a depth generation unit 21 of the information processing device 1. The ranging data generated by the ranging sensor 11 is composed of histogram data for each pixel in the pixel array of the ranging sensor 11. In the direct ToF method, light emission and reception of reflected light are repeated a predetermined number of times (e.g., tens to thousands of times) to generate one measurement data. The histogram data is data obtained by aggregating the frequency values ​​(count numbers) of each bin, where TDC (Time Domain Counter) values ​​representing the time it takes to receive the reflected light measured over the predetermined number of repetitions are used as bins. Since the TDC value can be converted into a distance using the speed of light or the like, the histogram data may be data obtained by aggregating the frequency values ​​of each distance converted from the TDC value in pixel units.

[0021] The RGB camera 12 is configured by a color camera using an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device), and generates an RGB image that is an image of the same subject as that captured by the distance measurement sensor 11, and outputs the image to the area dividing unit 22 of the information processing device 1. Note that the resolution of the RGB camera 12 and the resolution of the distance measurement sensor 11 may or may not match, but the correspondence between the pixels of the two is known.

[0022] The depth generation unit 21 acquires histogram data output as distance measurement data from the distance measurement sensor 11. The depth generation unit 21 generates a depth image from the acquired histogram data and supplies it to the point cloud data generation unit 23. The depth generation unit 21 generates a depth image with improved accuracy by removing data affected by artifacts. Specifically, when multiple peaks are detected in the histogram data of each pixel, the depth generation unit 21 requests the results of region division processing from the region division unit 22, removes data affected by artifacts by referring to the region division processing results acquired from the region division unit 22, and calculates the distance to the object. The depth generation unit 21 generates a depth image in which depth values ​​representing the calculated distance to the object are stored as pixel values ​​for each pixel of the distance measurement sensor 11, and supplies the depth image to the point cloud data generation unit 23.

[0023] The region dividing unit 22 performs a region dividing process to divide the RGB image acquired from the RGB camera 12 into regions for each object. In this embodiment, the region dividing unit 22 performs, for example, semantic segmentation using a convolutional neural network (CNN) as the region dividing process. Semantic segmentation is a technology that stores object identification dictionary data (trained data) based on the shapes and other feature information of various actual objects in an internal memory and identifies the objects in the image based on the degree of correspondence between this dictionary data and the objects in the RGB image. Semantic segmentation can identify objects on a pixel-by-pixel basis in the RGB image, and a label indicating the type of object is assigned to each pixel. Note that processes other than semantic segmentation may be used as the region dividing process, and any process that divides an RGB image into regions for each object may be employed.

[0024] The region dividing unit 22 supplies the segmentation result, which is the result of the region dividing process, to the point cloud data generating unit 23. Furthermore, when the depth generating unit 21 requests the result of the region dividing process, the region dividing unit 22 supplies the segmentation result to the depth generating unit 21 in response to the request. The segmentation result supplied to the depth generating unit 21 is the same as the segmentation result supplied to the point cloud data generating unit 23.

[0025] The point cloud data generation unit 23 generates point cloud data using the depth image from the depth generation unit 21 and the segmentation result from the region division unit 22, and supplies the point cloud data to the control unit 24. Specifically, the point cloud data generation unit 23 generates point cloud data by adding the RGB values ​​of the corresponding pixels in the segmentation result and a label indicating the type of object to the three-dimensional position coordinates of the object, which are made up of the pixel position (x, y) and depth value z of each pixel in the depth image, and supplies the point cloud data to the control unit 24.

[0026] The control unit 24 controls the overall operation of the information processing device 1. For example, the control unit 24 receives user instructions via an operation unit (not shown) or the like, and instructs the distance measurement sensor 11 and the RGB camera 12 to perform distance measurement and photography, and instructs the start and end of a point cloud data generation process for generating point cloud data. The control unit 24 also supplies the point cloud data supplied from the point cloud data generation unit 23 to the storage unit 25 for storage.

[0027] The storage unit 25 is configured by, for example, a hard disk or a semiconductor memory, and stores the point cloud data supplied from the control unit 24. The storage unit 25 also stores data to be temporarily stored in the control of the information processing device 1 and programs for executing the control.

[0028] The information processing device 1 is configured as described above, and the depth generation unit 21 calculates the distance to an object excluding data affected by artifacts, thereby generating a depth image with improved distance measurement accuracy. This also improves the accuracy of point cloud data generated using the depth image.

[0029] 2. Description of Depth Generation Unit Next, the processing of the depth generation unit 21, which calculates the distance to an object while excluding data affected by artifacts, will be described with reference to FIGS. 2 and 3. FIG.

[0030] 2A shows an example of an RGB image captured by the RGB camera 12. The RGB image 31 captured by the RGB camera 12 includes roads, sidewalks, people, cars, buildings, road signs, and the like.

[0031] The distance measurement sensor 11 measures distance within a measurement range that is the same as the imaging range of the RGB camera 12. Of the subjects included in the RGB image 31, the road sign 32 is made of a retroreflective material, which causes strong reflected light, making flare likely to occur in the distance measurement sensor 11, which performs measurements using active light. For example, a false peak due to flare (ghost) has occurred in the histogram data output by the distance measurement sensor 11 for an area 32F that is slightly away from the actual position of the road sign 32.

[0032] 2B shows the result of region segmentation processing performed by the region segmentation unit 22 on an acquired RGB image 31. The segmentation result 33 shown in FIG. 2B is an image in which a predetermined pattern is applied to each label assigned to each pixel of the RGB image 31. In the segmentation result 33, segments (regions) are divided into object units such as roads, sidewalks, people, cars, buildings, road signs, and road sign posts.

[0033] FIG. 3 is a diagram showing histogram data of pixel A in an area 32F where a false peak due to flare occurs and pixel B in an area 35 in the same segment as area 32F where no false peak due to flare occurs.

[0034] 3, histogram data A for pixel A in region 32F contains multiple (specifically, three) peaks PK1 to PK3 due to the influence of flare. Of peaks PK1 to PK3, peak PK2 is a true peak (true peak) corresponding to reflected light from the building, which is the actual subject, and peaks PK1 and PK3 are false peaks (false peaks) generated by the influence of flare.

[0035] On the other hand, histogram data B for pixel B in region 35 has only one peak, a true peak PK11 corresponding to the light reflected from the building, as shown in Figure 3. If there are no artifacts such as flare, the histogram data will have only one peak, as shown here.

[0036] The depth generation unit 21 sequentially sets each pixel of the distance measurement sensor 11 as a pixel of interest and performs the following processing on the histogram data of the pixel of interest (first pixel). First, the depth generation unit 21 determines whether the histogram data of the pixel of interest has multiple peaks. If the histogram data has multiple peaks, the depth generation unit 21 selects, as a reference pixel (second pixel), a pixel in the same segment as the pixel of interest that has histogram data with only one peak, based on the segmentation result by the region division unit 22. Then, the depth generation unit 21 selects, from the multiple peaks in the histogram data of the pixel of interest, a peak of the pixel of interest that corresponds to one peak in the histogram data of the reference pixel. Specifically, the peak of the pixel of interest that corresponds to one peak of the reference pixel is the peak that is closest to the distance corresponding to the one peak of the reference pixel. The depth generation unit 21 determines the peak selected from the multiple peaks as the peak of the pixel of interest, calculates the distance corresponding to the determined peak, and sets it as the depth value of the pixel of interest.

[0037] 3, when the pixel of interest is pixel A, histogram data A for pixel A includes three peaks PK1 to PK3, so pixel B, which belongs to the same segment (the building segment) as pixel A and has only one peak PK11, is selected as the reference pixel. Then, peak PK2, which is the peak in histogram data A closest to peak PK11 of pixel B, is selected as the true peak of pixel of interest A, and the distance corresponding to peak PK2 is set as the depth value of the pixel of interest.

[0038] As described above, the depth generation unit 21 determines the true peak of an object by referencing the segmentation result and excluding peaks such as flares, and calculates the distance to the object.

[0039] 3. Detailed Configuration Block Diagram of Depth Generator> FIG. 4 is a block diagram showing an example of the detailed configuration of the depth generator 21. As shown in FIG.

[0040] The depth generation unit 21 includes an acquisition unit 41 , a peak determination unit 42 , a selection unit 43 , and a distance determination unit 44 .

[0041] The acquisition unit 41 acquires histogram data output as distance measurement data from the distance measurement sensor 11 and detects histogram peaks for each pixel of the distance measurement sensor 11. For example, the acquisition unit 41 detects, as peaks, bins (distances) having a locally maximum frequency value (local maximum value) indicated by the histogram data, for example, bins having a frequency value equal to or greater than a predetermined threshold. The maximum peak indicated by the histogram data may be detected, and peaks other than the maximum peak may be limited to bins having a frequency value equal to or greater than a predetermined ratio (e.g., 1 / 2) of the frequency value of the maximum peak. Any known method may be used for peak detection, and the conditions for peak detection may be appropriately selected and determined, for example, on a setting screen or the like.

[0042] The distance measurement sensor 11 may have the function of detecting histogram peaks. If the distance measurement sensor 11 has the function of detecting peaks, the histogram data supplied from the distance measurement sensor 11 includes peak information indicating one or more detected peaks. For example, the peak information for a specific pixel of the distance measurement sensor 11 is the following data. In the following example, n peaks are detected, and for each detected peak, a bin (distance) and a frequency value are included as peak information.

[0043] Peak 1: 1.0m, 2000 counts / sec Peak 2: 5.0m, 1500 counts / sec ... Peak n: 10.0m, 1000 counts / sec

[0044] If peak information is included in the histogram data from the distance measurement sensor 11, the process of detecting the peak of the histogram for each pixel by the acquisition unit 41 is omitted. The acquisition unit 41 supplies the histogram data including the peak information for each pixel of the distance measurement sensor 11 to the peak determination unit 42. The histogram data for each pixel supplied to the peak determination unit 42 is shared within the depth generation unit 21.

[0045] The peak determination unit 42 sequentially selects each pixel of the distance measurement sensor 11 as a pixel of interest. Based on the peak information of the pixel of interest, the peak determination unit 42 determines whether there are multiple peaks in the histogram of the pixel of interest. The determination result is supplied to the selection unit 43.

[0046] If the histogram of the pixel of interest does not have multiple peaks, in other words, if the histogram data of the pixel of interest indicates only one peak, the selection unit 43 supplies the detected one peak to the distance determination unit 44.

[0047] If the histogram of the pixel of interest has multiple peaks, the selection unit 43 requests the result of the region segmentation process from the region segmentation unit 22. The selection unit 43 acquires the segmentation result supplied from the region segmentation unit 22 in response to the request. The segmentation result is the result of semantic segmentation performed on the RGB image captured by the RGB camera 12. The selection unit 43 selects a reference pixel present in the region corresponding to the pixel of interest based on the semantic segmentation result. In other words, the selection unit 43 selects a pixel in the same segment as the pixel of interest that has a histogram with only one peak as the reference pixel. The selection unit 43 supplies histogram data of the selected reference pixel and the pixel of interest to the distance determination unit 44.

[0048] If the histogram of the pixel of interest has only one peak, the distance determination unit 44 determines the distance corresponding to the bin of the peak indicated by the histogram data as the depth value of the pixel of interest. On the other hand, if the histogram of the pixel of interest has multiple peaks, the distance determination unit 44 selects one of the multiple peaks in the histogram of the pixel of interest based on the single peak indicated by the peak information of the reference pixel, and determines the distance corresponding to the bin of the selected peak as the depth value of the pixel of interest. If the bin at the peak position indicated by the peak information is expressed as a TDC value rather than a distance, the distance determination unit 44 also performs a process of calculating the distance from the TDC value using the speed of light, etc. The distance determination unit 44 outputs a depth image generated by sequentially setting each pixel of the ranging sensor 11 as the pixel of interest to the point cloud data generation unit 23.

[0049] The depth generation unit 21 is configured as described above.

[0050] 5, a description will be given of the point cloud data generation process performed by the information processing device 1. This process is started, for example, when the control unit 24 instructs each unit in the device to start the point cloud data generation process and instructs the distance measurement sensor 11 and the RGB camera 12 to perform distance measurement and image capture.

[0051] First, in step S1 , the depth generation unit 21 acquires histogram data output as distance measurement data from the distance measurement sensor 11 , generates a depth image from the acquired histogram data, and supplies it to the point cloud data generation unit 23 .

[0052] In step S2, the region dividing unit 22 acquires an RGB image from the RGB camera 12 and performs region dividing processing to divide the RGB image into regions for each object. The region dividing unit 22 performs semantic segmentation using CNN, for example, as the region dividing processing. The region dividing unit 22 supplies the segmentation result, which is the result of the region dividing processing, to the point cloud data generating unit 23. Furthermore, when the depth generating unit 21 requests the result of the region dividing processing, the region dividing unit 22 supplies the segmentation result to the depth generating unit 21 in response to the request.

[0053] In step S3, the point cloud data generation unit 23 generates point cloud data using the depth image from the depth generation unit 21 and the segmentation result from the region division unit 22, and supplies the point cloud data to the control unit 24. Specifically, the point cloud data generation unit 23 generates point cloud data by adding the RGB values ​​of the corresponding pixels in the segmentation result and a label indicating the type of object to the three-dimensional position coordinates of the object, which are made up of the pixel position (x, y) and depth value z of each pixel in the depth image, and supplies the point cloud data to the control unit 24. The generated point cloud data is stored in the storage unit 25.

[0054] The information processing device 1 repeatedly executes the point cloud data generation process consisting of steps S1 to S3 described above on the histogram data sequentially supplied from the distance measurement sensor 11. When the distance measurement sensor 11 and the RGB camera 12 are instructed to end distance measurement and photography and histogram data is no longer supplied from the distance measurement sensor 11, the point cloud data generation process ends.

[0055] 5. Flowchart of Depth Image Generation Processing Next, the depth image generation processing will be described with reference to the flowchart of Fig. 6. This processing corresponds to the detailed processing of step S1 of the point cloud data generation processing of Fig. 5.

[0056] First, in step S11 , the acquisition unit 41 acquires histogram data output from the distance measurement sensor 11 as distance measurement data.

[0057] In step S12, the acquisition unit 41 detects a histogram peak for each pixel of the distance measurement sensor 11 based on the acquired histogram data. The acquisition unit 41 supplies the histogram data for each pixel, including peak information based on the peak detection result, to the peak determination unit 42. If the distance measurement sensor 11 has a function for detecting histogram peaks and the histogram data supplied from the distance measurement sensor 11 includes peak information, the peak detection process is omitted, and the acquisition unit 41 supplies the histogram data including the peak information supplied from the distance measurement sensor 11 to the peak determination unit 42.

[0058] In step S13, the peak determination unit 42 sets the coordinates (x, y) that identify the pixel of interest to (0, 0).

[0059] In step S14, the peak determination unit 42 obtains the number n of peaks in the histogram of the pixel of interest based on the histogram data of the pixel of interest.

[0060] In step S15, the peak determination unit 42 determines whether the number of peaks in the histogram of the pixel of interest is 1, i.e., whether the histogram of the pixel of interest has one peak. The determination result is supplied to the selection unit 43.

[0061] If it is determined in step S15 that the number of peaks for the pixel of interest is n=1, the process proceeds to step S16, where the peak determination unit 42 supplies the determination result to the selection unit 43, and the selection unit 43 supplies one peak for the pixel of interest to the distance determination unit 44. The distance determination unit 44 determines the distance corresponding to the bin of the one peak as the depth value of the pixel of interest.

[0062] On the other hand, if it is determined in step S15 that the number of peaks for the pixel of interest is not n=1, that is, if there are multiple peaks in the histogram for the pixel of interest, the process proceeds to step S17, and the peak determination unit 42 supplies the determination result to the selection unit 43. The selection unit 43 requests the result of the region division process from the region division unit 22, and obtains the segmentation result supplied from the region division unit 22 in response to the request.

[0063] In step S18, the selection unit 43 selects a segment (region) corresponding to the pixel of interest based on the semantic segmentation result. Then, in step S19, the selection unit 43 selects, as reference pixels, pixels in the selected segment that have a histogram with one peak, and obtains histogram data of the selected reference pixels. The selection unit 43 supplies the histogram data of the selected reference pixels and the pixel of interest to the distance determination unit 44.

[0064] In step S20, the distance determination unit 44 detects the peak of the bin closest to the peak bin of the histogram of the reference pixel from among the multiple peaks of the pixel of interest, and then in step S21, the distance determination unit 44 determines the distance corresponding to the bin of the detected peak as the depth value of the pixel of interest.

[0065] After step S16 or S21, in step S22, the peak determination unit 42 determines whether depth values ​​have been determined for all pixels of the distance measurement sensor 11, in other words, whether all pixels of the distance measurement sensor 11 have been set as pixels of interest.

[0066] If it is determined in step S22 that depth values ​​have not yet been determined for all pixels of the distance measurement sensor 11, the process proceeds to step S23, where the peak determination unit 42 sets the next pixel of the distance measurement sensor 11 that has not yet been set as the pixel of interest as the pixel of interest. Each pixel of the distance measurement sensor 11 is selected as the pixel of interest in a predetermined order, such as raster scan order, and is set to the coordinates (x, y) that identify the pixel of interest. Once the pixel of interest has been set in step S23, the process returns to step S14. Then, the processes of steps S14 to S22 described above are performed on the newly set pixel of interest.

[0067] On the other hand, if it is determined in step S22 that depth values ​​have been determined for all pixels of the ranging sensor 11, processing proceeds to step S24, and the distance determination unit 44 outputs a depth image in which the depth values ​​of each pixel of the ranging sensor 11 are stored as pixel values ​​to the point cloud data generation unit 23.

[0068] The depth image generation process is now complete.

[0069] In the above-mentioned step S19, an example was described in which a specific pixel in the same segment as the pixel of interest that has only one peak is selected as the reference pixel, but multiple pixels that have only one peak may also be selected as reference pixels, as follows.

[0070] For example, the selection unit 43 selects, as reference pixels, multiple pixels having only one peak from multiple pixels in the four-neighborhood or eight-neighborhood around the pixel of interest shown in Fig. 7. The four-neighborhood is four pixels adjacent to the pixel of interest in the vertical and horizontal directions, and the eight-neighborhood is eight pixels including the four four-neighborhood pixels surrounding the pixel of interest.

[0071] The selector 43 then calculates the average value of the distances corresponding to the bins of the peaks of the multiple reference pixels, and detects the peak of the target pixel that is closest to the average distance of the reference pixels. The distance determiner 44 then determines the distance corresponding to the bin of the detected peak as the depth value of the target pixel.

[0072] Alternatively, the selection unit 43 compares the heights of the peaks of the multiple reference pixels, i.e., the magnitudes of the frequency values, and extracts the reference pixel with the highest peak height (highest frequency value).The selection unit 43 detects the peak of the bin closest to the bin of the peak of the extracted reference pixel from the multiple peaks of the target pixel.The distance determination unit 44 then determines the distance corresponding to the bin of the detected peak as the depth value of the target pixel.

[0073] In this way, by selecting multiple reference pixels using multiple pixels in the four or eight neighborhoods around the pixel of interest and determining the depth value of the pixel of interest, it is possible to suppress the occurrence of outliers in the distance measurement value. Note that if there is no pixel with a single peak among the eight neighborhood pixels around the pixel of interest, a reference pixel can be selected from the surrounding pixels further outside.

[0074] 6. Another Example of Depth Image Generation Processing Next, another example of depth image generation processing will be described.

[0075] 5 and 6, the RGB camera 12 performs imaging regardless of the number n of peaks in the histogram data of the distance measurement sensor 11. When there are multiple peaks in the histogram of the pixel of interest, the depth generation unit 21 obtains the results of semantic segmentation of the RGB image performed by the region division unit 22 and selects a reference pixel for determining the true peak.

[0076] In contrast, in the depth image generation process described next in FIG. 8, photography by the RGB camera 12 and region division processing by the region division unit 22 are performed only when there are multiple peaks in the histogram of the pixel of interest.

[0077] In the depth image generation process of FIG. 8, a segmentation execution flag M is introduced, which is a flag indicating whether or not the region dividing unit 22 has executed semantic segmentation.

[0078] The depth image generation process will be described with reference to the flowchart in Fig. 8. This process is started, for example, when the control unit 24 instructs each unit in the device to start the point cloud data generation process and instructs the distance measurement sensor 11 to perform distance measurement.

[0079] The processing in steps S41 to S43 in FIG. 8 is the same as steps S11 to S13 in FIG. 6, and therefore a description thereof will be omitted.

[0080] In step S44 after the pixel at coordinates (x, y) = (0, 0) is set as the pixel of interest, the peak determination unit 42 sets the segmentation execution flag M to 0 (segmentation execution flag M = 0). This segmentation execution flag M can be referenced by each unit in the depth generation unit 21.

[0081] In step S45, the peak determination unit 42 obtains the number n of peaks in the histogram of the pixel of interest based on the histogram data of the pixel of interest.

[0082] In step S46, the peak determination unit 42 determines whether the number of peaks n in the histogram of the pixel of interest is 1, i.e., whether the histogram of the pixel of interest has one peak. If it is determined in step S46 that the number of peaks n in the pixel of interest is 1, the process proceeds to step S47, where the peak determination unit 42 supplies the determination result to the selection unit 43, and the selection unit 43 supplies one peak of the pixel of interest to the distance determination unit 44. The distance determination unit 44 determines the distance corresponding to the bin of the one peak as the depth value of the pixel of interest. If it is determined that the histogram of the pixel of interest has one peak, the process is the same as the depth image generation process of FIG. 6.

[0083] On the other hand, if it is determined in step S46 that the number of peaks for the pixel of interest is not n=1, i.e., if there are multiple peaks in the histogram for the pixel of interest, the processing proceeds to step S48, and the selection unit 43 determines whether the segmentation execution flag M is 0 (M=0).

[0084] If it is determined in step S48 that the segmentation execution flag M is 0 (M=0), the process proceeds to step S49, where the selection unit 43 requests the result of the region division process from the region division unit 22. In step S50, the region division unit 22 accepts the request from the selection unit 43 and commands the RGB camera 12 to perform photography. The RGB camera 12 performs photography based on the command from the region division unit 22, and outputs the RGB image as the execution result to the region division unit 22.

[0085] In step S51, the region dividing unit 22 performs semantic segmentation as a region dividing process on the RGB image acquired from the RGB camera 12, and supplies the segmentation result to the selection unit 43 of the depth generation unit 21. The segmentation result is also supplied to the point cloud data generation unit 23.

[0086] In step S52, the selection unit 43 acquires the segmentation result supplied from the region dividing unit 22 in response to the request, and sets the segmentation execution flag M to 1 (segmentation execution flag M=1).

[0087] In step S53, the selection unit 43 selects a segment (region) corresponding to the pixel of interest based on the semantic segmentation result. Then, in step S54, the selection unit 43 selects, as reference pixels, pixels in the selected segment that have a histogram with one peak, and obtains histogram data of the selected reference pixels. The selection unit 43 supplies the histogram data of the selected reference pixels and the pixel of interest to the distance determination unit 44.

[0088] In step S55, the selection unit 43 associates the histogram data of the selected reference pixel with each pixel of the selected segment.

[0089] In step S56, the distance determination unit 44 detects, from among the multiple peaks of the pixel of interest, the peak of the bin closest to the bin of the peak of the histogram of the reference pixel. Then, in step S57, the distance determination unit 44 determines the distance corresponding to the bin of the detected peak as the depth value of the pixel of interest. The processes of steps S56 and S57 are similar to the processes of steps S20 and S21 in FIG. 6.

[0090] On the other hand, if it is determined in step S48 that the segmentation execution flag M is not 0 (M=0) but M=1, the process proceeds to step S58, where the selection unit 43 selects a segment corresponding to the pixel of interest based on the semantic segmentation result. Therefore, if the segmentation execution flag M is 1, the processes performed in steps S49 to S52, such as requesting the region segmentation processing result, taking an image using the RGB camera 12, and executing semantic segmentation, are omitted.

[0091] In step S59, the selection unit 43 determines whether histogram data is associated with the segment selected in step S58.

[0092] If it is determined in step S59 that histogram data is associated with the segment corresponding to the pixel of interest, the process proceeds to step S60, where the distance determination unit 44 detects the peak of the bin closest to the bin of the associated histogram peak among the multiple peaks of the pixel of interest. Then, in step S61, the distance determination unit 44 determines the distance corresponding to the bin of the detected peak as the depth value of the pixel of interest.

[0093] On the other hand, if it is determined in step S59 that histogram data is not associated with the segment corresponding to the pixel of interest, the process proceeds to step S54, and the processes of steps S54 to S57 described above are executed. That is, a pixel in the same segment as the pixel of interest that has a histogram with one peak is selected as the reference pixel, and the distance corresponding to the peak of the bin closest to the bin of the histogram peak of the reference pixel among the multiple peaks of the pixel of interest is determined as the depth value of the pixel of interest. Furthermore, the histogram data of the selected reference pixel is associated with each pixel in the same segment as the pixel of interest.

[0094] After step S47, S57, or S61, the process proceeds to step S62. The processes of steps S62 to S64 are the same as the processes of steps S22 to S24 in FIG.

[0095] This completes the depth image generation process of Fig. 8. Note that the modification of selecting multiple reference pixels described with reference to Fig. 7 can also be applied to the depth image generation process of Fig. 8.

[0096] In the depth image generation process of FIG. 8 , if the histogram data of the ranging sensor 11 does not have multiple peaks, the RGB camera 12 can capture the image (generate an RGB image) and the region segmentation process of the RGB image (semantic segmentation) by the region segmentation unit 22 can be omitted. The process of the point cloud data generation unit 23 adding RGB values ​​and labels to the three-dimensional position coordinates of the depth image is also omitted, and the point cloud data generation unit 23 outputs point cloud data corresponding to the three-dimensional position coordinates of the depth image to the control unit 24. Furthermore, since each pixel of the ranging sensor 11 in a segment selected as a segment corresponding to a pixel of interest is associated with histogram data of a reference pixel having only one peak, the process of detecting the reference pixel is omitted when the pixel with the associated histogram data becomes the pixel of interest. Therefore, the depth image generation process of FIG. 8 can be realized with a reduced amount of calculation, high speed, and low power consumption compared to the depth image generation process of FIG. 6 .

[0097] 7. Summary of information processing device The information processing device 1 includes a peak determination unit 42 that determines whether or not there are multiple peaks in the histogram of a first pixel (pixel of interest) of the distance measurement sensor 11, based on histogram data for each pixel output from the distance measurement sensor 11 as a first distance measurement sensor; a selection unit 43 that, if there are multiple peaks in the histogram of the first pixel, performs area division processing using image data of the RGB image acquired from the RGB camera 12, and selects at least one second pixel (reference pixel) of the distance measurement sensor 11 that exists in an area corresponding to the first pixel based on the result of the area division processing; and a distance determination unit 44 that selects one of the multiple peaks in the histogram of the first pixel based on the peaks in the histogram of one or more second pixels, and determines the distance corresponding to the selected peak as the distance of the first pixel.

[0098] The information processing device 1 executes the point cloud data generation process described with reference to Fig. 5 using the distance measurement data from the distance measurement sensor 11 and the image data of the RGB image from the RGB camera 12. This makes it possible to generate point cloud data while excluding the influence of artifacts, thereby improving distance measurement accuracy.

[0099] The distance measurement sensor 11 that generates distance measurement data that is the basis for generating point cloud data has been described as a depth sensor or LiDAR that directly measures the distance to a subject using the ToF method, but it may also be an ultrasonic sensor that emits sound waves. In other words, the distance measurement sensor 11 may be a sensor that uses light waves or sound waves and generates distance measurement data using the principle of reflection. The light waves may be visible light or invisible light.

[0100] In the above-described embodiment, the region dividing unit 22 performs region dividing processing using image data of the RGB image output by the RGB camera 12 to generate a region dividing processing result, but a sensor other than the RGB camera 12 may be used as long as it can generate image data that can be used to perform region dividing processing. For example, instead of the RGB camera 12, a monochrome camera that generates image data of a monochrome image may be used, or an indirect ToF distance measuring sensor that generates image data of a depth image may be used. A distance measuring sensor that uses electromagnetic waves other than light waves may also be used.

[0101] 8. Application Example to a Mobile Body FIG. 9 is a block diagram showing a configuration example of a mobile body according to an embodiment of the present disclosure, in which the configuration of the information processing device 1 of FIG. 1 is incorporated.

[0102] The mobile body 50 in FIG. 9 is, for example, a mobile robot capable of autonomously traveling and moving on the ground. The mobile robot may be a guide robot, cleaning robot, delivery robot, or vacuum cleaner robot that mainly travels and moves within a building. The mobile body 50 may also be a drone that flies within a space or a vehicle capable of self-driving. The mobile body 50 may also be a semi-autonomous type that includes both automatic control without human intervention and control based on human operation, instructions, etc. For example, the mobile body 50 may be a controlled mobile body equipped with a distance measurement function for map creation.

[0103] The moving body 50 has the same components as those in the configuration example of Fig. 1 : a distance measurement sensor 11, an RGB camera 12, a depth generation unit 21, an area division unit 22, and a point cloud data generation unit 23. The functions of these components are the same as those in the configuration example described above, so a description thereof will be omitted. The moving body 50 also has a control unit 71, a storage unit 72, a drive control unit 73, multiple drive units 74, and a rotation angle detection unit 75.

[0104] The control unit 71 controls the overall operation of the mobile object 50. The control unit 71 has a function of controlling the point cloud data generation process similar to that of the control unit 24 of the information processing device 1 described above. Furthermore, the control unit 71 has a function of estimating the mobile object 50's own position, a function of creating an environmental map, and the like. For example, the control unit 71 executes SLAM (Simultaneous Localization and Mapping) based on the point cloud data from the point cloud data generation unit 23 to create an environmental map showing obstacles and the like around the mobile object 50's own position and estimate the mobile object's own position. Then, the control unit 71 supplies a movement command to the drive control unit 73 to move to the destination based on the result of estimating the self-position.

[0105] In addition, the mobile body 50 may be equipped with an IMU (Inertial Measurement Unit) or a positioning sensor that receives signals from a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System), and may perform a process of estimating its own position using these sensor signals.

[0106] The storage unit 72 is configured by, for example, a hard disk or semiconductor memory, and stores point cloud data, an environment map, and the like supplied from the control unit 24. The storage unit 72 also stores data to be temporarily stored in the control of the moving body 50, and programs for executing the control.

[0107] The drive control unit 73 issues commands regarding the rotation direction, speed, torque, etc. to the multiple drive units 74 in accordance with the movement command supplied from the control unit 71. The drive control unit 73 obtains feedback such as drive current and motor temperature from the multiple drive units 74, and the rotation angle from the rotation angle detection unit 75, and controls the multiple drive units 74.

[0108] Each of the plurality of drive units 74 is configured with, for example, a motor that rotates a tire or a motor that rotates a rotor, and rotates based on commands such as rotation direction, speed, and torque supplied from the drive control unit 73. Each of the plurality of drive units 74 supplies drive current, motor temperature, and the like to the drive control unit 73 as feedback.

[0109] The rotation angle detection unit 75 is configured by, for example, an encoder or a resolver, and detects the rotation angle of a predetermined drive unit 74 and supplies it to the drive control unit 73. The same number of rotation angle detection units 75 as the number of drive units 74 may be provided.

[0110] In the moving body 50 configured as described above, the point cloud data generation process of FIG. 5 is executed using the ranging data from the ranging sensor 11 and the RGB image from the RGB camera 12 to generate point cloud data. Then, the generated point cloud data is used to estimate the self-position and control movement. By employing the point cloud data generation process of FIG. 5 or the depth image generation process of FIG. 6 or FIG. 8 as the process for generating point cloud data, it is possible to generate point cloud data with improved ranging accuracy by eliminating the influence of artifacts. By using point cloud data with improved ranging accuracy, the accuracy of movement control of the moving body 50 is also improved.

[0111] 1 can be used for the self-position estimation and movement control described above, and can also be applied to distance measurement data for any purpose using the distance measurement sensor 11. For example, the distance measurement sensor 11 can be used to measure the amount of displacement of liquids such as water and chemicals, or to measure the dimensions of products, and the information processing device 1 can also be applied to distance measurement data for such purposes.

[0112] 9. Example Computer Configuration The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes microcomputers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0113] FIG. 10 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0114] In the computer, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 are interconnected by a bus 104.

[0115] An input / output interface 105 is also connected to the bus 104. An input unit 106, an output unit 107, a storage unit 108, a communication unit 109, and a drive 110 are connected to the input / output interface 105.

[0116] The input unit 106 includes a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 107 includes a display, a speaker, an output terminal, etc. The storage unit 108 includes a hard disk, a solid state drive (SSD), a RAM disk, a non-volatile memory, etc. The communication unit 109 includes a network interface, etc. The drive 110 drives a removable recording medium 111 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0117] In a computer configured as described above, the CPU 101 performs the above-described series of processes by, for example, loading a program stored in the storage unit 108 into the RAM 103 via the input / output interface 105 and the bus 104 and executing the program. The RAM 103 also stores data necessary for the CPU 101 to execute various processes as appropriate.

[0118] The program executed by the computer (CPU 101) can be provided by being recorded on a removable recording medium 111 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0119] In a computer, the program can be installed in the storage unit 108 via the input / output interface 105 by inserting the removable recording medium 111 into the drive 110. The program can also be received by the communication unit 109 via a wired or wireless transmission medium and installed in the storage unit 108. Alternatively, the program can be installed in advance in the ROM 102 or the storage unit 108.

[0120] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0121] For example, it is possible to adopt a configuration in which some of the above-described embodiments are combined as appropriate.

[0122] For example, the technology of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0123] Each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0124] In this specification, the steps described in the flowcharts may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order, and may be performed in parallel or at any necessary timing, such as when a call is made.

[0125] The effects described in this specification are merely examples and are not intended to be limiting, and there may be effects other than those described in this specification.

[0126] The technology disclosed herein may employ the following configurations: (1) An information processing method that determines, for histogram data for each pixel output from a first ranging sensor, whether a histogram of a first pixel of the first ranging sensor has multiple peaks; if the histogram of the first pixel has multiple peaks, selects at least one second pixel of the first ranging sensor that exists in a region corresponding to the first pixel using image data acquired from a camera or a second ranging sensor; selects one of the multiple peaks of the histogram of the first pixel based on one or more peaks of the histogram of the second pixel, and determines the distance corresponding to the selected peak as the distance of the first pixel. (2) The information processing method described in (1), if the histogram of the first pixel has multiple peaks, performs region segmentation processing using the image data, and selects at least one second pixel of the first ranging sensor that exists in a region corresponding to the first pixel based on a result of the region segmentation processing. (3) The information processing method according to (2), wherein the second pixel of the first ranging sensor is present in a region corresponding to the first pixel in the result of the region segmentation processing and has a histogram with only one peak. (4) The information processing method according to (2) or (3), wherein a plurality of the second pixels are selected based on the result of the region segmentation processing, and a peak in the histogram of the first pixel in a bin closest to an average value of distances of the selected plurality of second pixels is selected. (5) The information processing method according to any of (2) to (4), wherein a plurality of the second pixels are selected based on the result of the region segmentation processing, and one of a plurality of peaks in the histogram of the first pixel is selected based on the peak in the histogram of the second pixel with the highest peak height among the selected plurality of second pixels. (6) The information processing method according to (4) or (5), wherein the plurality of second pixels are pixels adjacent to the first pixel.(7) The information processing method according to any one of (2) to (6), further comprising: a flag indicating whether the region segmentation process has been performed using the image data; and, if there are multiple peaks in a histogram of the first pixel and the flag indicates that the region segmentation process has not yet been performed, performing the region segmentation process using the image data. (8) The information processing method according to (7), further comprising: if there are multiple peaks in a histogram of the first pixel and the flag indicates that the region segmentation process has not yet been performed, instructing the camera or the second ranging sensor to acquire the image data, and performing the region segmentation process using the image data acquired from the camera or the second ranging sensor. (9) The information processing method according to any one of (2) to (7), further comprising: selecting an area corresponding to the first pixel based on a result of the area division processing; when the second pixel present in the selected area is selected, associating histogram data of the second pixel with each pixel in the selected area; and when a histogram of a third pixel of the first ranging sensor selected as the first pixel has multiple peaks and the histogram data of the second pixel is associated with the third pixel, omitting selection of the second pixel present in the area corresponding to the third pixel. (10) The information processing method according to (9), further comprising: selecting one of the multiple peaks of the histogram of the third pixel based on the peak of the histogram of the associated second pixel. (11) The information processing method according to any one of (2) to (9), wherein semantic segmentation is performed as the region division process, and at least one second pixel of the first ranging sensor that exists in a region corresponding to the first pixel is selected based on a result of the semantic segmentation. (12) The information processing method according to (1), wherein, from among a plurality of peaks in a histogram of the first pixel, a peak of a bin that is closest to a bin of a peak in a histogram of the second pixel is selected.(13) The information processing method according to any one of (1) to (12), further generating a depth image storing the distance determined based on the histogram data of the first ranging sensor as pixel values. (14) The information processing method according to (13), further generating point cloud data using the depth image and the result of the region segmentation process by performing region segmentation processing using the image data. (15) The information processing method according to any one of (1) to (14), wherein the first ranging sensor is configured as a direct ToF depth sensor or a LiDAR. (16) The information processing method according to any one of (1) to (15), wherein the image data of an RGB image is acquired from the camera when the histogram of the first pixels has multiple peaks. (17) The information processing method according to any one of (1) to (15), wherein the image data of a depth image is acquired from the second ranging sensor using an indirect ToF method when the histogram of the first pixels has multiple peaks. (18) An information processing device comprising: a determination unit that determines whether the histogram of a first pixel of a first ranging sensor has multiple peaks, based on histogram data for each pixel output from the first ranging sensor; a selection unit that, if the histogram of the first pixel has multiple peaks, selects at least one second pixel of the first ranging sensor that exists in an area corresponding to the first pixel, using image data acquired from a camera or a second ranging sensor; and a distance determination unit that selects one of the multiple peaks in the histogram of the first pixel based on the peaks in the histogram of one or more of the second pixels, and determines the distance corresponding to the selected peak as the distance of the first pixel.(19) A program for causing a computer to execute the following process: determining whether there are multiple peaks in the histogram of a first pixel of a first ranging sensor, using histogram data for each pixel output from the first ranging sensor; if there are multiple peaks in the histogram of the first pixel, using image data acquired from a camera or a second ranging sensor to select at least one second pixel of the first ranging sensor that exists in an area corresponding to the first pixel; selecting one of the multiple peaks in the histogram of the first pixel based on the peaks in the histogram of one or more of the second pixels; and determining the distance corresponding to the selected peak as the distance of the first pixel.

[0127] REFERENCE SIGNS LIST 1 Information processing device, 11 Distance measurement sensor, 12 RGB camera, 21 Depth generation unit, 22 Region division unit, 23 Point cloud data generation unit, 24 Control unit, 25 Memory unit, 41 Acquisition unit, 42 Peak determination unit, 43 Selection unit, 44 Distance determination unit, 50 Moving object, 101 CPU, 102 ROM, 103 RAM, 104 Bus, 105 Input / output interface, 106 Input unit, 107 Output unit, 108 Memory unit, 109 Communication unit, 110 Drive, 111 Removable recording medium

Claims

1. An information processing method that determines whether there are multiple peaks in the histogram of a first pixel of a first ranging sensor, using histogram data for each pixel output from the first ranging sensor; if there are multiple peaks in the histogram of the first pixel, uses image data acquired from a camera or a second ranging sensor to select at least one second pixel of the first ranging sensor that exists in an area corresponding to the first pixel; selects one of the multiple peaks in the histogram of the first pixel based on the peaks in the histogram of one or more of the second pixels; and determines the distance corresponding to the selected peak as the distance of the first pixel.

2. The information processing method according to claim 1, wherein if there are multiple peaks in the histogram of the first pixel, a region segmentation process is performed using the image data, and at least one second pixel of the first ranging sensor that exists in a region corresponding to the first pixel is selected based on the results of the region segmentation process.

3. The information processing method according to claim 2, wherein the second pixel of the first ranging sensor is a pixel that exists in an area corresponding to the first pixel in the result of the area division processing and has a histogram with only one peak.

4. The information processing method according to claim 2, further comprising: selecting a plurality of said second pixels based on the result of said region segmentation process; and selecting a peak in the histogram of said first pixels in a bin that is closest to the average value of the distances of said selected plurality of said second pixels.

5. The information processing method according to claim 2, wherein a plurality of the second pixels are selected based on the result of the region division processing, and one of the plurality of peaks in the histogram of the first pixel is selected based on the peak in the histogram of the second pixel having the highest peak height among the plurality of selected second pixels.

6. The information processing method according to claim 4, wherein the plurality of second pixels are pixels adjacent to the first pixel.

7. An information processing method according to claim 2, further comprising a flag indicating whether the region segmentation process has been performed using the image data, and performing the region segmentation process using the image data if there are multiple peaks in the histogram of the first pixel and the flag indicates that the region segmentation process has not yet been performed.

8. An information processing method as described in claim 7, wherein if the histogram of the first pixel has multiple peaks and the flag indicates that the area segmentation process has not yet been performed, the camera or the second ranging sensor is instructed to acquire the image data, and the area segmentation process is performed using the image data acquired from the camera or the second ranging sensor.

9. The information processing method of claim 2, wherein: a region corresponding to the first pixel is selected based on the result of the region division process; if the second pixel present in the selected region is selected, histogram data of the second pixel is associated with each pixel in the selected region; and if the histogram of a third pixel of the first ranging sensor selected as the first pixel has multiple peaks and the histogram data of the second pixel is associated with the third pixel, selection of the second pixel present in the region corresponding to the third pixel is omitted.

10. An information processing method as described in claim 9, wherein if there are multiple peaks in the histogram of a third pixel of the first ranging sensor selected as the first pixel and histogram data of the second pixel is associated with the third pixel, one of the multiple peaks in the histogram of the third pixel is selected based on the peak in the histogram of the associated second pixel.

11. The information processing method of claim 2, wherein semantic segmentation is performed as the region division process, and at least one second pixel of the first ranging sensor that exists in the region corresponding to the first pixel is selected based on the result of the semantic segmentation.

12. The information processing method according to claim 1, wherein the peak of the bin closest to the bin of the peak of the histogram of the second pixel is selected from among the plurality of peaks of the histogram of the first pixel.

13. The information processing method according to claim 1, further comprising generating a depth image in which the distance determined based on the histogram data of the first distance measuring sensor is stored as a pixel value.

14. The information processing method according to claim 13, further comprising: performing an area division process using the image data; and further generating point cloud data using the depth image and a result of the area division process.

15. The information processing method according to claim 1, wherein the first distance measuring sensor is a direct ToF depth sensor or a LiDAR.

16. The information processing method according to claim 1, wherein the image data of an RGB image is acquired from the camera if the histogram of the first pixel has multiple peaks.

17. The information processing method according to claim 1, wherein if the histogram of the first pixel has multiple peaks, the image data of the depth image is acquired from the second ranging sensor of an indirect ToF type.

18. An information processing device comprising: a determination unit that determines whether the histogram of a first pixel of a first ranging sensor has multiple peaks, based on histogram data for each pixel output from the first ranging sensor; a selection unit that, if the histogram of the first pixel has multiple peaks, selects at least one second pixel of the first ranging sensor that exists in an area corresponding to the first pixel, using image data acquired from a camera or a second ranging sensor; and a distance determination unit that selects one of the multiple peaks in the histogram of the first pixel based on the peaks in the histogram of one or more of the second pixels, and determines the distance corresponding to the selected peak as the distance of the first pixel.

19. A program for causing a computer to execute the following process: determining whether the histogram of a first pixel of a first ranging sensor contains multiple peaks for each pixel of histogram data output from the first ranging sensor; if the histogram of the first pixel contains multiple peaks, selecting at least one second pixel of the first ranging sensor that exists in an area corresponding to the first pixel using image data acquired from a camera or a second ranging sensor; selecting one of the multiple peaks in the histogram of the first pixel based on one or more peaks in the histogram of the second pixel; and determining the distance corresponding to the selected peak as the distance of the first pixel.

Images