Information processing device, information processing method, and information processing program

Abstract

This information processing device comprises: an acquisition unit that acquires echoes, which are reflected light due to reflection of laser light emitted from an irradiation device by an object; a determination unit that determines, on the basis of the signal intensity of the acquired echoes, whether or not to use, among the echoes, a multi-echo due to two or more round-trip reflections between the irradiation device and the object to generate an image for detecting the object; and a generation unit that generates the image on the basis of the acquired echoes and the result of the determination.

Description

Information processing device, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program.

[0002] As a technology for detecting an object, a technology is known that detects an object based on the similarity between a marker in an image generated based on an echo, which is light reflected by a marker attached to the object, and a correct pattern of the marker. For example, a technology is known that detects the position of the marker by pattern matching between a marker in an image generated based on an echo acquired by a sensor and the correct pattern of the marker, and then detects the position at which the similarity between these markers is greatest (for example, Patent Document 1).

[0003] JP 2007-114831 A

[0004] Conventional technology classifies an object based on the signal strength of the echo as either a high-reflectivity material with a reflectivity above a predetermined percentage, or a low-reflectivity material with a reflectivity below a predetermined percentage, making it easy to identify the type of object, for example, whether it is a car or a person.

[0005] However, there is room for improvement in the performance of technologies that use echo signal strength to detect objects. When the signal strength exceeds a certain value, the signal strength saturates, making it impossible to distinguish the markers in the image from the background, which can reduce the accuracy of object detection.

[0006] Therefore, an object of the present disclosure is to propose an information processing device, an information processing method, and an information processing program that can improve the performance of object detection.

[0007] The information processing device according to the present disclosure includes an acquisition unit that acquires echoes, which are reflected light of laser light irradiated from an irradiation device and reflected by an object; a determination unit that determines, based on the signal intensity of the acquired echoes, whether or not a multi-echo that has reflected two or more times back and forth between the irradiation device and the object will be used to generate an image for detecting the object; and a generation unit that generates the image based on the acquired echoes and the result of the determination.

[0008] FIG. 1 is a diagram for explaining an overview of an information processing system according to an embodiment. FIG. 2 is a diagram showing an example of a change over time in signal strength of an echo. FIG. 3 is a diagram showing an example of an image for detecting an object. FIG. 4 is a block diagram showing an example of a configuration of an information processing system according to an embodiment. FIG. 5 is a diagram showing an example of an echo acquisition. FIG. 6 is a diagram showing a second determination example. FIG. 7 is a diagram showing a third determination example. FIG. 8 is a diagram showing a fourth determination example. FIG. 9 is a diagram showing a fifth determination example. FIG. 10 is a diagram showing an example of an image generation. FIG. 11 is a diagram showing an example of similarity synthesis. FIG. 12 is a flowchart showing an example of the flow of information processing according to an embodiment. FIG. 13 is a flowchart showing an example of the flow of echo determination processing. FIG. 14 is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0010] Hereinafter, embodiments of the present disclosure will be described in the following order: 1. Embodiment 1-1. Overview of information processing system according to embodiment 1-2. Configuration of information processing system according to embodiment 1-3. Flow of information processing according to embodiment 2. Modifications 2-1. First modification 2-2. Second modification 2-3. Third modification 3. Other embodiments 4. Effects of information processing system according to the present disclosure 5. Hardware configuration 6. Supplementary information

[0011] (1. Embodiment) (1-1. Overview of Information Processing System According to Embodiment) An overview of an information processing system 1 according to an embodiment will be described using Fig. 1. Fig. 1 is a diagram for explaining the overview of an information processing system according to an embodiment.

[0012] The information processing system 1 includes an information processing device 100 and an irradiation device 200. The information processing device 100 is, for example, a server. The information processing device 100 detects the object 10 based on the similarity between the marker 21 in an image 20 generated based on an echo, which is light reflected by a marker 21 attached to the object 10 when laser light irradiated from the irradiation device 200 is reflected by the marker 21.

[0013] 1 , the information processing device 100 generates an image 20 based on the signal intensity of an echo acquired by the irradiation device 200. For example, the information processing device 100 generates an image 20 corresponding to an echo that has been reflected back and forth between the irradiation device 200 and the marker 21, among echoes whose signal intensity has a peak value.

[0014] Next, the information processing device 100 performs pattern matching between the marker 21 in the image 20 and the correct pattern of the marker. Next, the information processing device 100 calculates the similarity between these markers through pattern matching. Next, the information processing device 100 detects the position where the similarity is greatest as the position 22 of the marker.

[0015] The irradiation device 200 is a sensor that irradiates laser light and detects an echo reflected by the marker 21. For example, the irradiation device 200 is an optical sensor that irradiates laser light used in distance measurement methods such as dToF (direct Time of Flight) and FMCW (Frequency Modulated Continuous Wave) and detects a change over time in the amount of photons as an echo.

[0016] As an example, the irradiation device 200 is an APD (Avalanche Photodiode) sensor, a SPAD (Single Photon Avalanche Diode) sensor, an RGB (Red-Green-Blue) sensor, or a SiPM (Silicon Photo Multipliers) sensor.

[0017] The method of detecting echoes by the irradiation device 200 is a distance measurement method that can detect multiple echoes that have been reflected two or more times between the irradiation device 200 and the object 10, such as dToF or FMCW.

[0018] The marker 21 is a mark attached to the object 10 in order to detect the object 10. For example, the marker 21 is a reflective material that reflects the laser light emitted from the irradiation device 200 back and forth between the irradiation device 200 and the marker 21 two or more times. Specifically, the marker 21 is a retroreflective material or a highly reflective material having a reflectance equal to or greater than a predetermined percentage. The marker 21 is also a marker in which multiple types of rectangles with different reflectances are arranged in a pattern. For example, the marker 21 is a two-dimensional barcode in which white rectangles and black rectangles are arranged in a pattern.

[0019] As described above, the information processing device 100 generates the image 20 for detecting the marker 21 based on the signal intensity of the echo. However, if the signal intensity is higher than a predetermined value, the signal intensity generally tends to saturate, making it impossible to distinguish the marker 21 and the like in the image from the background, which may result in a decrease in the detection accuracy of the object 10.

[0020] The information processing device 100 according to the embodiment acquires echoes in order to solve the problem of improving the performance of detecting the object 10. Then, based on the signal strength of the acquired echoes, the information processing device 100 determines whether or not multi-echoes among the acquired echoes are to be used to generate an image for detecting the object 10. Then, the information processing device 100 generates an image based on the acquired echoes and the result of the determination.

[0021] Since the signal strength of a multi-echo is generally lower than the signal strength of an echo that has been reflected once, and therefore has a high resistance to saturation of signal strength, the information processing device 100 can strengthen the resistance to saturation of signal strength by determining whether or not to use multi-echoes for generating an image. This makes it less likely that the marker 21 in the image will become indistinguishable from the background due to signal strength saturation, and therefore the information processing device 100 can improve the performance of detecting the object 10.

[0022] Below, as an example of the above-mentioned processing, a case will be described in which the information processing device 100 performs an echo detection process to detect echoes, a determination process to determine whether or not to use multi-echoes to generate an image, a generation process to generate an image, and a detection process to detect the object 10.

[0023] (Echo Detection Processing) The echo detection processing will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of a change in signal strength of an echo over time. For example, the information processing device 100 acquires, from the irradiation device 200, a change in signal strength over time according to the amount of photons as an echo. The information processing device 100 also detects a first echo 11 reflected once between the irradiation device 200 and the marker 21, a second echo 12 reflected twice, a third echo 13 reflected three times, and a fourth echo 14 reflected four times.

[0024] (Determination Process) Based on the signal intensities of the acquired echoes, the information processing device 100 determines whether or not a multi-echo from the acquired echoes is to be used to generate an image for detecting a marker. An example of the determination process will be described below with reference to Fig. 2. For example, the information processing device 100 determines whether or not the multi-echoes, the second echo 12, the third echo 13, and the fourth echo 14, are to be used to generate an image based on the signal intensities corresponding to the amounts of photons in these multi-echoes.

[0025] In this case, the information processing device 100 determines that echoes whose signal intensity peak value is within a predetermined range are to be used for generating an image. Specifically, as shown in Fig. 2, the information processing device 100 determines that echoes whose signal intensity peak value is high, equal to or higher than the first threshold value 30, and equal to or lower than the second threshold value 40, and whose signal intensity is not saturated, are to be used for generating an image.

[0026] 2 , the peak value of the signal strength of the first echo 11 (signal strength at point 111), the peak value of the signal strength of the second echo 12 (signal strength at point 121), and the peak value of the signal strength of the third echo 13 (signal strength at point 131) are equal to or greater than the first threshold value 30 and equal to or less than the second threshold value 40. On the other hand, the peak value of the signal strength of the fourth echo 14 (signal strength at point 141) is less than the first threshold value 30. Therefore, the information processing device 100 determines that the first echo 11, the second echo 12, and the third echo 13 among the echoes will be used to generate an image.

[0027] (Generation Process) The information processing device 100 generates an image based on the acquired echoes and the determination results. An example of the generation process will be described below with reference to FIG. 3. FIG. 3 is a diagram showing an example of an image for detecting the object 10. For example, when the information processing device 100 determines that multiple echoes are to be used to generate an image, it generates multiple images corresponding to each of the multiple echoes, such as an image 20 corresponding to the first echo 11, an image 20A corresponding to the second echo 12, and an image 20B corresponding to the third echo 13.

[0028] (Detection Process) The information processing device 100 detects the object 10 based on the generated image. For example, as shown in Fig. 3, the information processing device 100 calculates the similarity between the marker 21 in the image 20, the marker 21A in the image 20A, and the marker 21B in the image 20B, and the correct marker pattern 21X, by performing pattern matching on these markers. Next, the information processing device 100 detects the position where the similarity is greatest as the position 22 of the marker in Fig. 1. An example of the above-mentioned detection process will be described in detail below with reference to Fig. 3.

[0029] First, the information processing device 100 selects an area for pattern matching with the correct marker pattern 21X for each of the generated images based on the signal intensities of the echoes. For example, the information processing device 100 selects a smaller range for the area as the signal intensities of the echoes are lower.

[0030] 2, the signal intensity at point 111, which is the peak value of the signal intensity of first echo 11, is the highest among the peak values ​​of the multiple echoes used to generate the image. In this case, as shown in FIG. 3, the information processing device 100 sets the range of region 50 in image 20 corresponding to first echo 11 to the largest range among the ranges of regions selected from image 20, image 20A, and image 20B.

[0031] Furthermore, the information processing device 100 selects, as an area for pattern matching, an area 50 of the image 20 where the signal strength is high and which includes the marker 21, the object 23, and the person 24. This is because the signal strength of the area 50 including the object 10 such as the marker 21, the object 23, and the person 24 is generally high, and therefore the area 50 is likely to include highly reflective materials such as the marker 21.

[0032] Furthermore, since the signal intensity at point 121, which is the peak value of the signal intensity of the second echo 12, is the second highest after point 111, the information processing device 100 enlarges the range of region 50A in image 20A corresponding to the second echo 12 to be the next largest after region 50. Furthermore, the information processing device 100 selects region 50A of image 20A, which has high signal intensity and includes marker 21A and object 23A, as the region for performing pattern matching.

[0033] Furthermore, since the signal intensity at point 131, which is the peak value of the signal intensity of the third echo 13, is the lowest among the peak values ​​of the signal intensity of the multiple echoes used to generate the image, the information processing device 100 sets the range of region 50B of image 20B corresponding to the third echo 13 to the smallest of the above-mentioned regions. Furthermore, the information processing device 100 selects region 50B of image 20B, which has a high signal intensity and includes marker 21B, as the region for pattern matching.

[0034] Next, the information processing device 100 determines the position 22 of the marker based on the similarity between each of the region 50, the region 50A, and the region 50B and the correct marker pattern 21X.

[0035] In this case, the information processing device 100 first calculates the similarity between the regions 50, 50A, and 50B and the correct marker pattern 21X by performing pattern matching between these patterns. For example, the information processing device 100 calculates the similarity between the correct marker pattern 21X and all pixels of these regions by searching for the correct marker pattern 21X for all pixels of these regions.

[0036] Next, the information processing device 100 combines the first similarity between the area 50 and the correct marker pattern 21X, the second similarity between the area 50A and the correct marker pattern 21X, and the third similarity between the area 50B and the correct marker pattern 21X.

[0037] For example, the information processing device 100 assigns a weight to each of the first similarity, the second similarity, and the third similarity, and then combines these weighted similarities. As an example, the information processing device 100 combines these weighted similarities such that the weight decreases as the signal strength decreases.

[0038] Specifically, the information processing device 100 combines the first similarity to the third similarity so that the weight of the first similarity corresponding to the first echo 11 with the highest signal strength is the highest, and the weight of the third similarity corresponding to the third echo 13 with the lowest signal strength is the lowest.

[0039] Next, the information processing device 100 determines the marker position 22 based on the similarity combination result. For example, the information processing device 100 determines the pixel position at which the combined similarity is greatest as the marker position 22. The information processing device 100 detects the object 10 using the determined marker position 22 as the position of the object 10.

[0040] As described above, the information processing device 100 determines, based on the signal strength of the echoes, whether or not to use multi-echoes, which generally have a lower signal strength than the signal strength of an echo that has been reflected once and therefore have a high resistance to saturation of signal strength, for generating an image for detecting the object 10. This makes it less likely that markers or the like in the image will become indistinguishable from the background due to saturation of signal strength, and therefore the information processing device 100 can improve the performance of detecting the object 10.

[0041] (1-2. Configuration of Information Processing System According to Embodiment) Next, an example of the configuration of the information processing system 1 according to the embodiment will be described with reference to FIG. 4. FIG. 4 is a block diagram showing an example of the configuration of the information processing system according to the embodiment. The information processing system 1 includes an information processing device 100 and an irradiation device 200.

[0042] (Configuration of Irradiation Device) The irradiation device 200 includes a light-emitting unit 210, a light-receiving element 220, and a communication unit 230. The light-emitting unit 210 is a light source of laser light. For example, the light-emitting unit 210 is a light source of an LED (Light Emitting Diode). The light-receiving element 220 is an element that receives echoes. For example, the light-receiving element 220 is a SPAD or the like.

[0043] The communication unit 230 is realized by, for example, a network interface card (NIC) or a network interface controller. The communication unit 230 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the information processing device 100 via the network N. The network N is realized by, for example, a wireless communication standard or method such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra Wide Band), or LPWA (Low Power Wide Area).

[0044] For example, the communication unit 230 receives light emission control information for controlling the light emission of the light emitting unit 210 from the information processing device 100 .

[0045] (Configuration of Information Processing Apparatus) The information processing apparatus 100 includes a communication unit 110 , a storage unit 120 , a detection result display unit 130 , and a control unit 140 .

[0046] (Communication Unit) The communication unit 110 is realized by, for example, a NIC, a network interface controller, etc. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the irradiation device 200 via the network N. For example, the communication unit 110 transmits light emission control information for controlling the light emission of the light-emitting unit 210 to the irradiation device 200. In addition, the communication unit 110 receives echoes received by the light-receiving element 220 of the irradiation device 200 from the irradiation device 200.

[0047] (Storage Unit) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk, a solid state drive (SSD), an optical disk, etc. For example, the storage unit 120 stores echoes and images.

[0048] (Detection Result Display Unit) The detection result display unit 130 is a desktop or the like that displays the detection results of the object 10. For example, the detection result display unit 130 displays the position 22 of the marker.

[0049] (Control Unit) The control unit 140 is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), etc. executing a program (for example, an information processing program according to the present disclosure) stored inside the information processing device 100 using a RAM or the like as a work area. The control unit 140 is a controller, and may be realized by, for example, an integrated circuit such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a micro controller unit (MCU).

[0050] The control unit 140 includes a system control unit 142 , a light emission control unit 143 , a light reception signal processing unit 144 , and an image processing unit 145 .

[0051] (System Control Unit) The system control unit 142 controls each unit, such as the communication unit 110, the storage unit 120, the detection result display unit 130, the light emission control unit 143, the light reception signal processing unit 144, and the image processing unit 145. For example, the system control unit 142 causes the detection result display unit 130 to display the object 10 detected by the image processing unit 145.

[0052] (Light Emission Control Unit) The light emission control unit 143 controls the emission of laser light by the light emission unit 210. For example, the light emission control unit 143 generates a light emission control signal that reduces the signal intensity of the laser light.

[0053] (Light-receiving signal processing unit) The light-receiving signal processing unit 144 processes echoes received by the light-receiving element 310 and echo signals. For example, the light-receiving signal processing unit 144 acquires the echoes and processes the echo signals. The light-receiving signal processing unit 144 includes an acquisition unit 1441 and a determination unit 1442.

[0054] (Acquisition Unit) The acquisition unit 1441 acquires an echo, which is light reflected by the object 10 from the laser light irradiated from the irradiation device 200. For example, the acquisition unit 1441 acquires the amount of photons as the echo.

[0055] An example of echo acquisition by the acquisition unit 1441 will be described below with reference to Fig. 5. Fig. 5 is a diagram for explaining an example of echo acquisition.

[0056] An example of how the acquisition unit 1441 acquires the echo 11A will be described using the histogram on the left of Fig. 5. The acquisition unit 1441 extracts echoes from point 151A to point 161A having signal intensities equal to or greater than threshold 31A.

[0057] 5, points 111A and 111A2 are points between points 151A and 161A where the signal intensity is equal to or greater than the threshold value 30A. Also, there are no points between points 151A and 161A where the signal intensity is less than the threshold value 30A. Therefore, the acquisition unit 1441 detects one echo 11A, which has peak values ​​of the signal intensity at points 111A and 111A2, as the echo between points 151A and 161A.

[0058] 5, an example will be described in which the acquisition unit 1441 acquires a first echo 11B, a second echo 12B, and a third echo 13B. The acquisition unit 1441 extracts echoes from point 151B to point 161B that have signal intensities equal to or greater than threshold 31B.

[0059] 5, the acquisition unit 1441 has determined that the points from point 151B to point 161B where the signal strength is equal to or greater than the threshold value of 30B (the points where the signal strength is at its peak value) are points 111B, 121B1, 121B2, and 131B. Also, the points from point 151B to point 161B where the signal strength is less than the threshold value of 30B are points 171B and 181B.

[0060] Therefore, the acquisition unit 1441 separates the echo from point 151B to point 161B into three echoes, a first echo 11B to a third echo 13B, with points 171B and 181B as boundaries. For example, the acquisition unit 1441 detects the first echo 11B as the echo from point 151B to point 171B. Similarly, the acquisition unit 1441 detects the second echo 12B as the echo from point 171B to point 181B. Similarly, the acquisition unit 1441 detects the third echo 13B as the echo from point 181B to point 161B.

[0061] (Determination Unit) Returning to the description of Fig. 4 , the determination unit 1442 determines whether or not to use multiple echoes from among the acquired echoes for generating an image, based on the signal strength of the echoes. For example, the determination unit 1442 determines that at least one of an echo that has been reflected twice between the irradiation device 200 and the object 10 and an echo that has been reflected three times between the irradiation device 200 and the object 10 is to be used for generating an image as the multiple echoes.

[0062] The determination unit 1442 also determines whether or not to use, for generating an image, a multi-echo that has reflected two or more times between the object 10 and a retroreflective material or a highly reflective material having a reflectance equal to or higher than a predetermined rate. For example, the determination unit 1442 determines whether or not to use, for generating an image, a multi-echo that has reflected two or more times between the object 10 and a marker 21 that is a retroreflective material or a highly reflective material and that has multiple types of rectangles with different reflectances arranged in a pattern.

[0063] (First Determination Example) Hereinafter, a first determination example by the determination unit 1442 will be described with reference to Fig. 2. For example, the determination unit 1442 determines whether or not to use the second echo 12, the third echo 13, and the fourth echo 14 for generating an image based on signal intensities corresponding to the amounts of photons in these echoes.

[0064] For example, the determination unit 1442 determines that an echo having a signal intensity peak value within a predetermined range is to be used for generating an image. Specifically, the determination unit 1442 determines that an echo having a signal intensity peak value that is high, equal to or higher than a first threshold value of 30, and equal to or lower than a second threshold value of 40, and whose signal intensity is not saturated, is to be used for generating an image.

[0065] 2 , the peak value of the signal intensity of the first echo 11 (signal intensity at point 111), the peak value of the signal intensity of the second echo 12 (signal intensity at point 121), and the peak value of the signal intensity of the third echo 13 (signal intensity at point 131) are equal to or greater than the first threshold value 30 and equal to or less than the second threshold value 40. On the other hand, the signal intensity of point 141 is less than the first threshold value 30. Therefore, the determination unit 1442 determines that the first echo 11, the second echo 12, and the third echo 13 among the echoes should be used to generate an image.

[0066] The determining unit 1442 also determines that an echo whose time indicating the peak position of the signal strength of the multi-echo is an integer multiple of the time indicating the peak position of the signal strength of the first echo 11 is to be used to generate an image.

[0067] 2 , the time indicating the peak position of the signal strength of the second echo 12 is twice as long as the time indicating the peak position of the signal strength of the first echo 11. The time indicating the peak position of the signal strength of the third echo 13 is three times as long as the time indicating the peak position of the signal strength of the first echo 11. Furthermore, the time indicating the peak position of the signal strength of the fourth echo 14 is four times as long as the time indicating the peak position of the signal strength of the first echo 11. However, as described above, the peak value of the signal strength of the fourth echo 14 (signal strength at point 141) is less than the first threshold value 30.

[0068] Therefore, the determination unit 1442 determines to use, for image generation, the second echo 12 and the third echo 13, whose time indicating the peak position of the signal strength of the multi-echoes is an integer multiple of the time indicating the peak position of the signal strength of the first echo 11, in addition to the first echo 11. The determination unit 1442 determines not to use, for image generation, the fourth echo 14, whose time indicating the peak position of the signal strength of the multi-echoes is four times the time indicating the peak position of the signal strength of the first echo 11, but whose peak value of the signal strength is less than the first threshold value 30.

[0069] Furthermore, the determination unit 1442 determines that the multi-echoes are to be used for generating an image when the peak value of the signal strength of the multi-echoes is equal to or greater than a predetermined percentage of the peak value of the signal strength of the first echo 11. For example, the determination unit 1442 determines that the second echo 12 and the third echo 13 are to be used for generating an image when their signal strengths are equal to or greater than 10% of the signal strength at point 111, which is the peak value of the signal strength of the first echo 11. Furthermore, the determination unit 1442 determines that the first echo 11 is to be used for generating an image, as described above.

[0070] From the above, the determining unit 1442 determines that the first echo 11, the second echo 12, and the third echo 13 are to be used to generate an image.

[0071] (Second Determination Example) A second determination example by the determination unit 1442 will be described below with reference to Fig. 6. Fig. 6 is a diagram for explaining the second determination example. In the second determination example, a case will be described in which only the first echo 11C is acquired as an echo. The determination unit 1442 determines that the first echo 11C, whose signal intensity peak value is high, equal to or higher than the first threshold value 30C, and equal to or lower than the second threshold value 40C, and whose signal intensity is not saturated, will be used to generate an image.

[0072] (Third Determination Example) A third determination example by the determination unit 1442 will be described below with reference to Fig. 7. Fig. 7 is a diagram for explaining the third determination example. The determination unit 1442 determines that the first echo 11D and the second echo 12D, whose signal intensity peak values ​​are high, equal to or higher than the first threshold value 30D, and equal to or lower than the second threshold value 40D, and whose signal intensity is not saturated, should be used to generate an image.

[0073] Furthermore, when the peak value of the signal strength of the multi-echo is equal to or greater than a predetermined ratio relative to the peak value of the signal strength of the first echo 11D, the determination unit 1442 determines that the multi-echo should be used to generate an image. For example, the determination unit 1442 determines that the second echo 12D, whose peak value of the signal strength is equal to or greater than 10% relative to the peak value of the signal strength of the first echo 11D, should be used as the multi-echo to generate an image.

[0074] From the above, the determining unit 1442 determines that the first echo 11D and the second echo 12D are to be used to generate an image.

[0075] (Fourth Determination Example) A fourth determination example by the determination unit 1442 will be described below with reference to FIG. 8. FIG. 8 is a diagram for explaining the fourth determination example. FIG. 8 is also a histogram showing the time change in signal intensity of an echo from a retroreflector detected by a SPAD element using a dToF detection method. The histogram on the left of FIG. 8 is a histogram when the distance between the object 10 and the irradiation device 200 is 250 cm. The histogram on the right of FIG. 8 is a histogram when the distance between the object 10 and the irradiation device 200 is 290 cm.

[0076] 8, the determination unit 1442 determines that the second echo 12E and the third echo 13E, whose signal intensity peak values ​​are equal to or higher than the first threshold value 30E and equal to or lower than the second threshold value 40E, and whose signal intensity is not saturated, should be used to generate an image. On the other hand, the determination unit 1442 determines that the first echo 11E, whose signal intensity peak value is higher than the second threshold value 40E, and whose signal intensity is saturated, should not be used to generate an image.

[0077] In the histogram on the right of Fig. 8, the determination unit 1442 determines that the second echo 12F, whose signal intensity peak value is high, equal to or higher than the first threshold 30F, and equal to or lower than the second threshold 40F, and whose signal intensity is not saturated, should be used to generate an image. On the other hand, the determination unit 1442 determines that the first echo 11F, whose signal intensity peak value is higher than the second threshold 40F, and whose signal intensity is saturated, should not be used to generate an image. In addition, the determination unit 1442 determines that the third echo 13F, whose signal intensity peak value is lower than the first threshold 30F, and whose signal intensity is low, should not be used to generate an image.

[0078] (Fifth Determination Example) Hereinafter, a fifth determination example by the determination unit 1442 will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining the fifth determination example. Fig. 9 is also a histogram showing the time change in signal intensity of an echo from a mirror detected by a SPAD element using a dToF detection method. The histogram on the left of Fig. 9 and the histogram on the right of Fig. 9 are histograms when the distance between the object 10 and the irradiation device 200 is different, respectively.

[0079] 9, the determination unit 1442 determines that the first echo 11G and the third echo 13G, whose signal intensity peak values ​​are high, equal to or higher than the first threshold value of 30 G, and equal to or lower than the second threshold value of 40 G, and whose signal intensity is not saturated, should be used to generate an image. On the other hand, the determination unit 1442 determines that the second echo 12G, whose signal intensity peak value is lower than the first threshold value of 30 G, and whose signal intensity is low, should not be used to generate an image.

[0080] In the histogram on the right of Figure 9, the judgment unit 1442 judges that the first echo 11H, the second echo 12H, and the third echo 13H, whose signal intensity peak values ​​are high, equal to or higher than the first threshold value of 30H, and equal to or lower than the second threshold value of 40H, and whose signal intensity is not saturated, will be used to generate the image.

[0081] (Generation Unit) Returning to the description of FIG. 4 , the generation unit 1451 generates an image based on the acquired echoes and the determination result. For example, when it is determined that multiple echoes are to be used to generate an image as multi-echoes, the generation unit 1451 generates multiple images corresponding to the multiple echoes. Specifically, as shown in FIG. 3 , the generation unit 1451 generates an image 20 corresponding to the first echo 11, an image 20A corresponding to the second echo 12, and an image 20B corresponding to the third echo 13.

[0082] An example of image generation by the generation unit 1451 will be described below with reference to Fig. 10. Fig. 10 is a diagram for explaining an example of image generation.

[0083] Image 20C in Fig. 10 is an image corresponding to the first echo. Image 20D in Fig. 10 is an image corresponding to the second echo. Point cloud 60 in Fig. 10 is a point cloud detected by irradiation device 200 and corresponding to images 20C and 20D. In Fig. 10, point cloud 60 is shown as a criterion for determining whether or not images 20C and 20D contain a plate corresponding to subject plate 61, which is a retroreflective material, in point cloud 60.

[0084] The generation unit 1451 generates an image 20C based on the acquired first echo and the result of the determination. The generation unit 1451 also generates an image 20D based on the acquired second echo and the result of the determination. As shown in FIG. 10 , in the image 20C corresponding to the first echo, the signal intensity of the first echo is saturated, and therefore a plate corresponding to the plate 61 in the point cloud 60 is not detected. In contrast, in the image 20D corresponding to the second echo, the signal intensity of the second echo is not saturated, and therefore a plate 21D corresponding to the plate 61 in the point cloud 60 is included in the image 20 in a state where it can be distinguished from the background.

[0085] (Detection Unit) Returning to the description of FIG. 4 , the detection unit 1452 detects the object 10 based on the generated image. For example, as shown in FIG. 3 , the detection unit 1452 calculates the similarity between the marker 21 in the image 20, the marker 21A in the image 20A, and the marker 21B in the image 20B, and the correct marker pattern 21X, by performing pattern matching on these markers. Next, the detection unit 1452 detects the position where the similarity is maximum as the position 22 of the marker in FIG. 1 . An example of the above-mentioned detection process will be described in detail below with reference to FIG. 3 .

[0086] First, the detection unit 1452 selects an area for pattern matching with the correct marker pattern 21X for each of the generated images based on the signal intensity of each of the multiple echoes. For example, the detection unit 1452 selects a smaller range of the area as the signal intensity of each of the multiple echoes becomes lower.

[0087] 2, the signal intensity at point 111, which is the peak value of the signal intensity of the first echo 11, is the highest among the peak values ​​of the multiple echoes used to generate the image. In this case, as shown in FIG. 3, the detection unit 1452 sets the range of region 50 in image 20 corresponding to the first echo 11 to be the largest among the ranges of regions selected from image 20, image 20A, and image 20B. Furthermore, the detection unit 1452 selects region 50 in image 20 with high signal intensity, which includes marker 21, object 23, and person 24, as the region for pattern matching.

[0088] Furthermore, since the signal intensity at point 121, which is the peak value of the signal intensity of the second echo 12, is the second highest after point 111, the detection unit 1452 increases the range of region 50A in image 20A corresponding to the second echo 12 to the next largest range after region 50. Furthermore, the detection unit 1452 selects region 50A of image 20A, which has a high signal intensity and includes marker 21A and object 23A, as the region for pattern matching.

[0089] Furthermore, since the signal intensity at point 131, which is the peak value of the signal intensity of the third echo 13, is the lowest among the peak values ​​of the signal intensities of the multiple echoes used to generate the image, the detection unit 1452 sets the range of region 50B of image 20B corresponding to the third echo 13 to the smallest of the above-mentioned regions. Furthermore, the detection unit 1452 selects region 50B of image 20B, which has a high signal intensity and includes marker 21B, as the region for pattern matching.

[0090] Next, the detection unit 1452 calculates weights to be assigned to the respective similarities between the above-mentioned regions 50 to 50B and the correct marker pattern 21X. For example, the detection unit 1452 calculates a weight to be assigned to a first similarity between the region 50 and the correct marker pattern 21X. The detection unit 1452 calculates a weight to be assigned to a second similarity between the region 50A and the correct marker pattern 21X. The detection unit 1452 also calculates a weight to be assigned to a third similarity between the region 50B and the correct marker pattern 21X.

[0091] Next, the detection unit 1452 determines the position 22 of the marker based on the similarity between each of the region 50, the region 50A, and the region 50B and the correct marker pattern 21X. For example, the detection unit 1452 determines the position 22 of the marker based on a plurality of similarity maps in which the similarity between each of these regions and the correct marker pattern 21X is graphed.

[0092] In this case, the detection unit 1452 first performs pattern matching between each of the regions 50, 50A, and 50B and the correct marker pattern 21X, thereby calculating the similarity between these patterns. For example, the detection unit 1452 calculates the similarity of the correct marker pattern 21X to all pixels in these regions by searching for the correct marker pattern 21X for all pixels in these regions. Next, the detection unit 1452 outputs a plurality of similarity maps in which the calculated similarities are each graphed.

[0093] Next, the detection unit 1452 combines the first similarity between the region 50 and the correct marker pattern 21X, the second similarity between the region 50A and the correct marker pattern 21X, and the third similarity between the region 50B and the correct marker pattern 21X. An example of combining similarities will be described below with reference to Fig. 11. Fig. 11 is a diagram for explaining an example of combining similarities.

[0094] The first similarity map 80 to the third similarity map 80B and the similarity map 90 in Fig. 11 are graphs of similarity, with the X-axis and Y-axis representing pixels corresponding to the X-axis and Y-axis pixels of the image in Fig. 3, and the Z-axis representing similarity. The similarity of the similarity map 90 is obtained by adding together the similarity of the first similarity map 80 to which a first weight is assigned, the similarity of the second similarity map 80A to which a second weight is assigned, and the similarity of the third similarity map 80B to which a third weight is assigned, and then normalizing the sum.

[0095] The detection unit 1452 assigns weights to the first similarity map 80 corresponding to the first similarity, the second similarity map 80A corresponding to the second similarity, and the third similarity map 80B corresponding to the third similarity, and then combines these weighted similarity maps. As an example, the detection unit 1452 combines these similarity maps that are weighted so that the weight decreases as the signal strength decreases.

[0096] Specifically, the detection unit 1452 combines the first similarity map 80 to the third similarity map 80B so that the first similarity map 80 has the highest weight and the third similarity map 80B has the lowest weight. As a result, the detection unit 1452 obtains a similarity map 90 that is combined based on the weights assigned to the multiple similarity maps, as a result of combining the multiple similarity maps, namely the first similarity map 80 to the third similarity map 80B.

[0097] Next, the detection unit 1452 determines, as the similarity, the position 22 of the marker based on the similarity map 90, which is a result of combining a plurality of similarity maps. For example, the detection unit 1452 determines, as the marker position 22, the position of the pixel corresponding to the point 91 where the similarity in the similarity map 90 is maximum. The detection unit 1452 detects the object 10 by determining the determined marker position 22 as the position of the object 10.

[0098] (1-3. Flow of Information Processing According to the Embodiment) An example of the flow of information processing by the information processing device 100 according to the embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of the flow of information processing according to the embodiment.

[0099] In step S1, the acquisition unit 1441 of the received light signal processing unit 144 performs an echo detection process to detect an echo. For example, the acquisition unit 1441 acquires the amount of photons as an echo. Next, the acquisition unit 1441 detects an echo from the change in the amount of photons over time based on the signal intensity corresponding to the amount of photons. In the histogram on the right of FIG. 5 , the acquisition unit 1441 detects a first echo 11B from the echoes from point 151B to point 161B based on whether the signal intensity is equal to or greater than a threshold value of 30B.

[0100] In step S2, the acquisition unit 1441 performs the same process on the N echoes. For example, in the histogram on the right of Fig. 5, the acquisition unit 1441 detects the second echo 12B and the third echo 13B from the echoes from point 151B to point 161B based on whether the signal intensity is equal to or greater than a threshold value of 30B.

[0101] In step S3, the determination unit 1442 of the received light signal processing unit 144 determines whether or not to use multiple echoes from the acquired echoes for generating an image, based on the signal intensities of the echoes. For example, the determination unit 1442 determines whether or not to use, as multiple echoes, an echo that has made two round trips between the irradiation device 200 and the object 10 or an echo that has made three round trips between the irradiation device 200 and the object 10 for generating an image.

[0102] If the determining unit 1442 determines that multi-echoes are to be used to generate an image (step S3; Yes), in step S4, the generating unit 1451 of the image processing unit 145 generates a signal intensity image from the echoes.

[0103] For example, when it is determined that a plurality of echoes are to be used as multi-echoes to generate an image, the generator 1451 generates a plurality of images corresponding to the plurality of echoes, respectively, based on the acquired echoes and the determination result. Specifically, as shown in FIG. 3 , the generator 1451 generates an image 20 corresponding to the first echo 11, an image 20A corresponding to the second echo 12, and an image 20B corresponding to the third echo 13.

[0104] In step S5, the detection unit 1452 of the image processing unit 145 performs region selection and weight calculation based on the signal intensity.

[0105] For example, as shown in FIG. 3 , the detection unit 1452 selects region 50, region 50A, and region 50B for pattern matching with the correct marker pattern 21X. Furthermore, the lower the signal intensity of each of the multiple echoes, the smaller the range of the region selected by the detection unit 1452. For example, as shown in FIG. 3 , the detection unit 1452 sets the range of region 50 in the image 20 corresponding to the first echo 11 to the largest among the above-mentioned regions. Furthermore, the detection unit 1452 sets the range of region 50B in the image 20B corresponding to the third echo 13 to the smallest.

[0106] Next, the detection unit 1452 calculates weights to be assigned to the similarities between the above-mentioned regions 50 to 50B and the correct marker pattern 21X.

[0107] In step S6, the detection unit 1452 of the image processing unit 145 performs pattern matching on the selected regions. For example, the detection unit 1452 first performs pattern matching between each of the regions 50, 50A, and 50B and the correct marker pattern 21X, thereby calculating the similarity between these patterns. Specifically, the detection unit 1452 searches for the correct marker pattern 21X for all pixels in these regions, and thereby calculates the similarity between the correct marker pattern 21X and all pixels in these regions.

[0108] Next, the detection unit 1452 outputs a plurality of similarity maps in which these similarities are each graphed. For example, as shown in FIG. 11 , the detection unit 1452 outputs a first similarity map 80 corresponding to a first similarity between the region 50 and the correct marker pattern 21X. The detection unit 1452 outputs a second similarity map 80A corresponding to a second similarity between the region 50A and the correct marker pattern 21X. The detection unit 1452 also outputs a third similarity map 80B corresponding to a third similarity between the region 50B and the correct marker pattern 21X.

[0109] In step S7, the detection unit 1452 stores the similarities. For example, the detection unit 1452 stores the first similarity map 80 to the third similarity map 80B in the storage unit 120.

[0110] After step S7, or if the determining unit 1442 determines that multi-echoes are not to be used in generating an image (step S3; No), in step S8, the detecting unit 1452 ends the echo processing.

[0111] In step S9, the detection unit 1452 combines the similarities. For example, as shown in FIG. 11 , the detection unit 1452 assigns weights to the first similarity map 80, the second similarity map 80A, and the third similarity map 80B, and then combines these weighted similarity maps.

[0112] As an example, the detection unit 1452 combines the first similarity map 80 to the third similarity map 80B so that the weight of the first similarity map 80 corresponding to the first echo 11 is the highest and the weight of the third similarity map 80B corresponding to the third echo 13 is the lowest. In this way, the detection unit 1452 obtains the similarity map 90.

[0113] In step S10, the detection unit 1452 performs a marker detection process. For example, the detection unit 1452 determines a marker position 22 as the similarity based on the similarity map 90. Specifically, the detection unit 1452 determines, as the marker position 22, the position of a pixel corresponding to a point 91 in the similarity map 90 where the similarity is maximum. The detection unit 1452 detects the object 10 by determining the determined marker position 22 as the position of the object 10.

[0114] In step S11, the detection unit 1452 outputs the detection result. For example, the detection unit 1452 causes the detection result display unit 130 to display the position 22 of the marker.

[0115] Next, an example of the flow of the echo determination process corresponding to step S3 in Fig. 12 will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of the flow of the echo determination process.

[0116] In step S21, the determination unit 1442 determines whether the peak values ​​of the signal strength of the echoes are within a predetermined range. For example, as shown in FIG. 2 , the determination unit 1442 determines whether the peak values ​​of the signal strength of the second echo 12 and the third echo 13 are equal to or greater than the first threshold value of 30 and equal to or less than the second threshold value of 40.

[0117] If the determination unit 1442 determines that the peak value of the echo signal strength is within a predetermined range (step S21; Yes), the process proceeds to step S22. For example, if the peak values ​​of the signal strength of the second echo 12 and the third echo 13 are equal to or greater than the first threshold value of 30 and equal to or less than the second threshold value of 40, the determination unit 1442 proceeds to step S22. On the other hand, if the determination unit 1442 determines that the peak value of the echo signal strength is outside the predetermined range (step S21; No), the process ends the echo determination process.

[0118] In step S22, the determination unit 1442 determines whether the peak positions of the signal strengths of the multiple echoes are an integer multiple of the peak position of the signal strength of the first echo. For example, the determination unit 1442 determines whether the peak positions of the signal strengths of the second echo 12 and the third echo 13 are an integer multiple of the peak position of the signal strength of the first echo 11.

[0119] If the determination unit 1442 determines that the peak position of the signal strength of the multi-echoes is an integer multiple of the peak position of the signal strength of the first echo (step S22; Yes), the determination unit 1442 proceeds to step S23. For example, if the time indicating the peak positions of the signal strength of the second echo 12 and the third echo 13 is an integer multiple of the time indicating the peak position of the signal strength of the first echo 11, the determination unit 1442 proceeds to step S23.

[0120] On the other hand, if the determining unit 1442 determines that the peak position of the signal strength of the multi-echo is not an integer multiple of the peak position of the signal strength of the first echo (step S22; No), it ends the echo determination process.

[0121] In step S23, the determination unit 1442 determines whether the peak value of the signal strength of the multi-echoes is x% or more of the peak value of the signal strength of the first echo. For example, the determination unit 1442 determines whether the peak values ​​of the signal strength of the second echo 12 and the third echo 13 are 10% or more of the signal strength at point 111, which is the peak value of the signal strength of the first echo 11.

[0122] If the determination unit 1442 determines that the peak value of the signal strength of the multi-echoes is x% or more of the peak value of the signal strength of the first echo (step S23; Yes), the determination unit 1442 adopts the multi-echoes for generating an image in step S24. For example, if the peak values ​​of the signal strength of the second echo 12 and the third echo 13 are 10% or more of the signal strength at point 111, which is the peak value of the signal strength of the first echo 11, the determination unit 1442 determines that the second echo 12 and the third echo 13 are to be used for generating an image.

[0123] On the other hand, if the determination unit 1442 determines that the peak value of the signal strength of the multi-echoes is less than x% of the peak value of the signal strength of the first echo (step S23; No), the determination unit 1442 ends the echo determination process. For example, if the peak values ​​of the signal strength of the second echo 12 and the third echo 13 are less than 10% of the signal strength at point 111, which is the peak value of the signal strength of the first echo 11, the determination unit 1442 ends the echo determination process.

[0124] (2. Modifications) (2-1. First Modification) In the above example, a case has been described in which the signal strength of the multiple echoes is lower than the signal strength of the echo that has been reflected once. However, depending on the imaging environment, the signal strength of the multiple echoes may be higher than the signal strength of the echo that has been reflected once. On the other hand, even in this case, the information processing device 100 can solve the problem of improving the detection of the object 10. For example, the information processing device 100 can solve the problem of improving the detection of the object 10 by determining that the multiple echoes whose signal strength is not saturated are to be used to generate an image for detecting the object 10.

[0125] (2-2. Second Modification) Furthermore, the information processing device 100 can be used, when mounted on an autonomous mobile robot, a smartphone, or a head-mounted display, to estimate the self-position of these devices based on a self-position estimation method such as SLAM (Simultaneous Localization and Mapping). In this case, the detection unit 1452 of the information processing device 100 estimates the self-position of the information processing device 100 based on the determined position of the object.

[0126] SLAM methods include, for example, a method that detects landmark markers using an RGB camera or LiDAR (Light Detection and Ranging).There are also many other SLAM methods, such as a method that uses LiDAR and an IMU (Inertial Measurement Unit), a method that uses an RGB camera and an IMU, and a method that uses LiDAR, an RGB camera, and an IMU.

[0127] (2-3. Third Modification) The marker may also be used as an AR (Augmented Reality) marker. For example, the marker is a trigger for overlaying AR content on an RGB image. In this case, the determination unit 1442 of the information processing device 100 determines whether or not to use a multi-echo that has reflected two or more times between the marker and the AR marker as a marker for generating an image.

[0128] (3. Other Embodiments) Of the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0129] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. For example, the information processing system 1 may be an information processing device 100 and an irradiation device 200 integrated together.

[0130] Furthermore, the above-described embodiments of the present disclosure can be combined as appropriate within the scope of the processing content without causing inconsistencies. Furthermore, the order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate. For example, the steps may be processed in chronological order, repeatedly, or partially in parallel.

[0131] (4. Effects of the information processing device according to the present disclosure) As described above, the information processing device according to the present disclosure (information processing device 100 in the embodiment) includes an acquisition unit (acquisition unit 1441 in the embodiment), a determination unit (determination unit 1442 in the embodiment), and a generation unit (generation unit 1451 in the embodiment).

[0132] The acquisition unit acquires echoes, which are light reflected by an object from laser light irradiated from an irradiation device (irradiation device 200 in this embodiment). The determination unit determines, based on the signal intensity of the acquired echoes, whether or not to use multi-echoes, which have been reflected two or more times between the irradiation device and the object, to generate an image for detecting the object. The generation unit generates an image based on the acquired echoes and the result of the determination.

[0133] Since the signal strength of a multi-echo is generally lower than that of an echo that has been reflected once, and therefore has a high resistance to signal strength saturation, the information processing device can strengthen its resistance to signal strength saturation by determining whether or not to use multi-echoes for image generation. This makes it less likely that a marker or the like in an image will become indistinguishable from the background due to signal strength saturation, and therefore the information processing device can improve its object detection performance.

[0134] The determination unit determines that echoes whose signal strength peak values ​​are within a predetermined range should be used to generate an image. This allows the information processing device to use echoes whose signal strength is high enough to enable object detection and that are not saturated to generate an image, thereby further improving object detection performance.

[0135] The determination unit determines that an echo whose time indicating the peak position of the signal strength of the multi-echo is an integer multiple of the time indicating the peak position of the signal strength of the echo that has been reflected once between the irradiation device and the object will be used to generate an image.

[0136] The time indicating the peak position of the signal strength of a multi-echo is generally an integer multiple of the time indicating the peak position of the signal strength of an echo that has been reflected once. Therefore, by using an echo that is an integer multiple to generate an image, even if there is a location where the signal strength is stronger than the echo that is an integer multiple, if the location is not an integer multiple, the location can be considered to be noise. This allows the information processing device to reduce the erroneous use of noise in generating an image, and therefore can use appropriate multi-echoes to generate an image.

[0137] The determination unit determines that the multi-echoes should be used to generate an image when the peak value of the signal strength of the multi-echoes is equal to or greater than a predetermined ratio of the peak value of the signal strength of the echoes that have been reflected once between the irradiation device and the object. This allows the information processing device to use echoes with signal strength high enough to detect the object for image generation, thereby further improving the object detection performance.

[0138] The determination unit determines that at least one of the echoes reflected twice between the irradiation device and the object and the echoes reflected three times between the irradiation device and the object is to be used as the multi-echo for generating the image. This allows the information processing device to use, for example, echoes that have a high enough signal strength to enable object detection and that are not saturated for generating the image, thereby further improving the object detection performance.

[0139] The determination unit determines whether or not to use, in generating an image, multi-echoes that have reflected two or more times between the object and a retroreflective material or a highly reflective material having a reflectivity equal to or higher than a predetermined rate. Since multi-echoes are generally likely to occur when the object is a retroreflective material or a highly reflective material, the information processing device can further improve object detection performance by determining whether or not to use, in generating an image, multi-echoes that have reflected between the object and a retroreflective material or a highly reflective material.

[0140] The determination unit determines whether or not to use, for generating an image, multi-echoes that have reflected back and forth two or more times between the marker and the marker, which is a retroreflective material or a highly reflective material and has a pattern of multiple types of rectangles with different reflectivities. Generally, such markers are easy to detect, so the information processing device can further improve the object detection performance by determining whether or not to use, for generating an image, multi-echoes that have reflected back to the marker.

[0141] Furthermore, by determining whether or not multi-echoes that have reflected back and forth between the marker and the device are used to generate an image, it becomes easier to identify the use of such an invention by another company. For example, if a third party's product uses a marker, it may be possible to identify the use of such an invention by another company through a direct investigation of the marker's reflectivity and pattern by the user. Furthermore, if the third party's product malfunctions when operated under the following conditions, it may be possible to identify the use of such an invention by another company.

[0142] For example, if no measures are taken to avoid saturation, such as automatically reducing the laser signal strength, even when the marker and camera are brought close enough to each other that the signal becomes saturated, it may be possible to allege that such an invention has been used by another person.

[0143] As another example, if only the white rectangle in a marker is made of a highly reflective material, but the black rectangles are also made of a highly reflective material, and the marker can be detected even if all of the first echoes of the marker are saturated, it may be possible to point out that the invention has been used by another person. This is because, since the signal strength of the first echo detected by LiDAR is generally the strongest, if the marker can be detected even if all of the first echoes of the marker are saturated, it is possible that images corresponding to the second or third echoes were used.

[0144] As another example, if a sensor used is fitted with a filter to suppress multiple reflections, it may be possible to identify the use of the invention by another person if the marker cannot be detected. The filter may be, for example, an AR coating or tilted glass.

[0145] The determination unit determines whether or not to use the multi-echo that has reflected two or more times between the marker and the AR marker as a marker for generating an image. In this way, the information processing device can also use the AR marker that cannot be detected by the naked eye or an RGB camera and is detected by a detection method suitable for dToF or the like for the determination.

[0146] The acquisition unit acquires the amount of photons as echoes, and the determination unit determines whether or not to use the multi-echoes to generate an image based on the signal strength corresponding to the amount of photons. This allows the information processing device to make a determination with higher accuracy than when making a determination based on the signal strength detected by, for example, a radar, thereby further improving the object detection performance.

[0147] When it is determined that multiple echoes are to be used to generate an image, the generator generates multiple images corresponding to the multiple echoes, respectively. This allows the information processing device to increase the number of images used for pattern matching to detect an object, for example, and therefore improves object detection performance compared to when an image corresponding to one type of echo is generated.

[0148] The information processing device further includes a detection unit that selects an area for pattern matching with a target pattern of the object for each of the generated images based on the signal strength of each of the multiple echoes. This allows the information processing device to reduce the range for pattern matching, thereby reducing the amount of processing load, such as the amount of similarity calculation. As a result, the information processing device can improve its processing speed.

[0149] The detection unit reduces the range of the region as the signal strength of each of the multiple echoes decreases, which allows the information processing device to reduce the range of pattern matching for images with lower signal strength that are less likely to contain objects such as highly reflective materials, thereby further improving processing speed.

[0150] The detection unit determines the position of the object based on the similarity between each region and the correct pattern of the object. In this way, by calculating the similarity using multiple images, the information processing device can calculate the similarity with higher accuracy than when using a single image, even if noise is present in the image, thereby further improving the object detection performance.

[0151] The detection unit determines the position of the object based on a plurality of similarity maps in which the similarity between each region and the correct pattern of the object is graphed. In this way, by detecting the position of the object based on a plurality of similarity maps, the information processing device can reduce the influence of reduced accuracy in similarity calculation due to noise in the image, thereby further improving the performance of object detection.

[0152] The detection unit determines the position of an object based on the result of combining multiple similarity maps as the similarity. In this way, by combining multiple similarity maps, the information processing device can improve its resistance to noise and calculate the similarity with higher accuracy. This allows the information processing device to further improve its object detection performance.

[0153] The detection unit determines the position of the object based on a similarity map that is a result of combining the multiple similarity maps based on weights assigned to each of the multiple similarity maps. This allows the information processing device to determine the position of the object based on a similarity map that is combined so that, for example, the lower the signal strength, the lower the weight of the similarity map, thereby particularly improving object detection performance.

[0154] The detection unit determines the pixel position at which the similarity is greatest as the object position, thereby enabling the information processing device to more appropriately determine the object position, thereby further improving the object detection performance.

[0155] The detection unit estimates the self-location of the information processing device based on the determined position of the object. In this way, the information processing device can be applied not only to object detection but also to self-location estimation, thereby improving the performance of self-location estimation.

[0156] (5. Hardware Configuration) The information processing device 100 and the like according to the embodiments of the present disclosure described above are realized by, for example, a computer 1000 configured as shown in FIG. 14 . The information processing device 100 will be described as an example. FIG. 14 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. The computer 1000 has a processing circuitry 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, an input / output interface 1600, a display unit 1700, a camera unit 1800, a microphone 1900, and a speaker 2000. The components of the computer 1000 are connected by a bus 1050.

[0157] The processing circuit 1100 operates and controls each unit based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the processing circuit 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0158] The ROM 1300 stores boot programs such as a basic input output system (BIOS) that is executed by the processing circuit 1100 when the computer 1000 is started up, and programs that depend on the hardware of the computer 1000 .

[0159] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing device 100 according to an embodiment of the present disclosure, which are examples of program data 1450.

[0160] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. The communication interface 1500 corresponds to the communication unit 110 provided in the information processing device 100. For example, the processing circuit 1100 receives data from other devices and transmits data generated by the processing circuit 1100 to other devices via the communication interface 1500.

[0161] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from an input device such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to an output device such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of the media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0162] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescence display (EL display). The display unit 1700 may also be a touch panel display device or a video projection device.

[0163] The camera unit 1800 is an interface through which the computer 1000 captures images. The microphone 1900 is an interface through which the computer 1000 captures audio. The speaker 2000 is an interface through which the computer 1000 outputs audio processed by the computer 1000. The components of the computer 1000 are connected by a bus 1050. The interfaces do not necessarily need to be provided inside the computer 1000, but may be provided outside the computer 1000 via a network or the like. Furthermore, the components constituting the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100 but by a circuit dedicated to display processing provided in the display unit 1700.

[0164] For example, when the computer 1000 functions as the information processing device 100 according to an embodiment of the present disclosure, the processing circuit 1100 of the computer 1000 functions as the control unit 140 by executing a program loaded onto the RAM 1200. The secondary storage device 1400 stores the information processing program according to the present disclosure and various data stored in the storage device 120. The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400. Alternatively, the processing circuit 1100 may obtain these programs from another device via an external network 1550. That is, the secondary storage device 1400 does not need to be located inside the computer 1000, but may also be located outside the computer 1000. The processing circuit 1100 is an example of an integrated circuit, and a CPU, an MPU, a GPU, an APU, an ASIC, and an FPGA can all be considered to be integrated circuits.

[0165] (6. Supplementary Information) The present technology may also be configured as follows. (1) An information processing device comprising: an acquisition unit that acquires echoes, which are light reflected by an object from laser light irradiated from an irradiation device; a determination unit that determines, based on signal intensities of the acquired echoes, whether or not multi-echoes that have reflected two or more times back and forth between the irradiation device and the object are to be used to generate an image for detecting the object; and a generation unit that generates the image based on the acquired echoes and a result of the determination. (2) The information processing device described in (1), wherein the determination unit determines that echoes whose peak value of signal intensity is within a predetermined range are to be used to generate the image. (3) The information processing device described in (1) or (2), wherein the determination unit determines that echoes whose time indicating the peak position of signal intensity of the multi-echoes is an integer multiple of the time indicating the peak position of signal intensity of an echo that has reflected once back and forth between the irradiation device and the object are to be used to generate the image. (4) The information processing device according to any one of (1) to (3), wherein the determination unit determines that the multi-echoes should be used to generate the image when a peak value of the signal strength of the multi-echoes is equal to or greater than a predetermined ratio of a peak value of the signal strength of an echo that has reflected once between the irradiation device and the object. (5) The information processing device according to any one of (1) to (4), wherein the determination unit determines that at least one of an echo that has reflected twice between the irradiation device and the object and an echo that has reflected three times between the irradiation device and the object should be used to generate the image as the multi-echoes. (6) The information processing device according to any one of (1) to (5), wherein the determination unit determines whether the multi-echoes that have reflected twice or more between the object and a retroreflective material or a highly reflective material having a reflectivity equal to or greater than a predetermined ratio should be used to generate the image. (7) The information processing device according to (6), wherein the determination unit determines whether or not the multi-echo that has reflected two or more times between the retroreflective material or the marker in which multiple types of rectangles with different reflectivities are arranged in a pattern as the highly reflective material is to be used for generating the image.(8) The information processing device according to (7), wherein the determination unit determines whether or not the multi-echo that has reflected two or more times back and forth between the marker and an AR marker is to be used as the marker for generating the image. (9) The information processing device according to any one of (1) to (8), wherein the acquisition unit acquires an amount of photons as the echo, and the determination unit determines whether or not the multi-echo is to be used for generating the image based on signal strength corresponding to the amount of photons. (10) The information processing device according to any one of (1) to (9), wherein, when it is determined that multiple echoes are to be used as the multi-echoes for generating the image, the generation unit generates multiple images corresponding to the multiple echoes as the images. (11) The information processing device according to (10), further comprising a detection unit that selects an area for pattern matching with a correct pattern of the object for each of the multiple generated images based on the signal strength of each of the multiple echoes. (12) The information processing device according to (11), wherein the detection unit narrows the range of the area as the signal strength of each of the multiple echoes becomes lower. (13) The information processing device according to (12), wherein the detection unit determines the position of the object based on a similarity between each of the regions and a correct pattern of the object. (14) The information processing device according to (13), wherein the detection unit determines the position of the object based on a plurality of similarity maps in which the similarity between each of the regions and a correct pattern of the object is graphed as the similarity. (15) The information processing device according to (14), wherein the detection unit determines the position of the object based on a combination result of the plurality of similarity maps. (16) The information processing device according to (15), wherein the detection unit determines the position of the object based on a similarity map combined based on weights assigned to each of the plurality of similarity maps as a combination result of the plurality of similarity maps. (17) The information processing device according to any one of (13) to (16), wherein the detection unit determines, as the position of the object, the position of a pixel at which the similarity is maximum.(18) The information processing device according to any one of (13) to (17), wherein the detection unit estimates a self-position of the information processing device based on the determined position of the object. (19) An information processing method including: a computer acquiring echoes, which are light reflected by an object from laser light irradiated from an irradiation device; determining, based on signal intensity of the acquired echoes, whether or not multi-echoes, which have been reflected two or more times between the irradiation device and the object, will be used to generate an image for detecting the object; and generating the image based on the acquired echoes and a result of the determination. (20) An information processing program for causing a computer to function as an information processing device, comprising: an acquisition unit that acquires echoes, which are light reflected by an object from laser light irradiated from an irradiation device; a determination unit that determines, based on signal intensity of the acquired echoes, whether or not multi-echoes, which have been reflected two or more times between the irradiation device and the object, will be used to generate an image for detecting the object.

[0166] REFERENCE SIGNS LIST 100 Information processing device 110 Communication unit 120 Storage unit 130 Detection result display unit 140 Control unit 142 System control unit 143 Light emission control unit 144 Light reception signal processing unit 145 Image processing unit 200 Irradiation device 210 Light emission unit 220 Light receiving element 230 Communication unit 1441 Acquisition unit 1442 Determination unit 1451 Generation unit 1452 Detection unit N Network

Claims

1. An information processing device comprising: an acquisition unit that acquires echoes, which are light reflected by an object from laser light irradiated from an irradiation device; a determination unit that determines, based on the signal intensity of the acquired echoes, whether or not a multi-echo that has reflected two or more times back and forth between the irradiation device and the object will be used to generate an image for detecting the object; and a generation unit that generates the image based on the acquired echoes and the result of the determination.

2. The information processing device according to claim 1, wherein the determination unit determines that an echo having a peak value of signal intensity of the multi-echo that is within a predetermined range is to be used for generating the image.

3. The information processing device according to claim 1, wherein the determination unit determines that an echo whose time indicating the peak position of the signal strength of the multi-echo is an integer multiple of the time indicating the peak position of the signal strength of an echo that has been reflected once between the irradiation device and the object is to be used to generate the image.

4. The information processing device of claim 1, wherein the determination unit determines that the multi-echo is to be used to generate the image when the peak value of the signal strength of the multi-echo is equal to or greater than a predetermined ratio of the peak value of the signal strength of the echo that has been reflected once between the irradiation device and the object.

5. The information processing device according to claim 1, wherein the determination unit determines that at least one of an echo that has been reflected twice between the irradiation device and the object and an echo that has been reflected three times between the irradiation device and the object is to be used as the multi-echo for generating the image.

6. The information processing device according to claim 1, wherein the determination unit determines whether or not the multi-echo is to be used to generate the image when the object is a retroreflective material or a highly reflective material having a reflectance equal to or greater than a predetermined ratio.

7. The information processing device according to claim 6, wherein the determination unit determines whether or not the multi-echo is to be used to generate the image between the retroreflective material or the marker in which multiple types of rectangles with different reflectivities are arranged in a pattern as the highly reflective material.

8. The information processing device according to claim 7, wherein the determination unit determines whether or not the multi-echo between the marker and an AR (Augmented Reality) marker is used to generate the image.

9. The information processing device according to claim 1, wherein the acquisition unit acquires the amount of photons as the echoes, and the determination unit determines whether or not the multi-echoes are to be used to generate the image based on signal strength corresponding to the amount of photons.

10. The information processing device according to claim 1, wherein, when it is determined that a plurality of echoes are to be used as the multi-echo to generate the image, the generating unit generates a plurality of images corresponding to each of the plurality of echoes as the image.

11. The information processing device according to claim 10, further comprising a detection unit that selects an area for pattern matching with the correct pattern of the object for each of the generated images based on the signal strength of each of the echoes.

12. The information processing device according to claim 11, wherein the detection unit reduces the range of the region as the signal strength of each of the plurality of echoes decreases.

13. The information processing device according to claim 12, wherein the detection unit determines the position of the object based on the similarity between each of the regions and a correct pattern of the object.

14. The information processing device according to claim 13, wherein the detection unit determines the position of the object based on a plurality of similarity maps in which the similarity between each of the regions and the correct pattern of the object is graphed as the similarity.

15. The information processing device according to claim 14, wherein the detection unit determines the position of the object based on a result of combining the plurality of similarity maps as the similarity.

16. The information processing device according to claim 15, wherein the detection unit determines the position of the object based on a similarity map that is synthesized based on weights assigned to each of the plurality of similarity maps as a result of synthesizing the plurality of similarity maps.

17. The information processing device according to claim 13, wherein the detection unit determines the position of the pixel at which the similarity is greatest as the position of the object.

18. The information processing device according to claim 13, wherein the detection unit estimates a self-position of the information processing device based on the determined position of the object.

19. An information processing method including the steps of: a computer acquiring echoes, which are light reflected by an object from laser light irradiated from an irradiation device; determining, based on the signal strength of the acquired echoes, whether or not a multi-echo, which has been reflected two or more times back and forth between the irradiation device and the object, will be used to generate an image for detecting the object; and generating the image based on the acquired echoes and the result of the determination.

20. An information processing program for causing a computer to function as an information processing device, comprising: an acquisition unit that acquires echoes, which are reflected light from an object when laser light emitted from an irradiation device is reflected; a judgment unit that judges, based on the signal intensity of the acquired echoes, whether or not multi-echoes that have reflected two or more times back and forth between the irradiation device and the object will be used to generate an image for detecting the object; and a generation unit that generates the image based on the acquired echoes and the result of the judgment.

Images