Information processing device, information processing method, and information processing program

Abstract

This information processing device comprises: a first acquisition unit that acquires an image captured by a camera mounted on a mobile body; a second acquisition unit that acquires three-dimensional point group data acquired by a sensor mounted on the mobile body; a detection unit that detects, from the image, a transparent region that includes a region in which reflected light appears; and a specification unit that, by associating the positions represented by each point of the three-dimensional point group data with the positions of each pixel in the transparent region, specifies the three-dimensional point group data corresponding to the transparent region as the three-dimensional point group data of a transparent object.

Description

Information processing device, information processing method, and information processing program

[0001] The disclosed technology relates to an information processing device, an information processing method, and an information processing program.

[0002] Conventionally, a technique has been known that uses an ultrasonic sensor, a light-emitting diode (LED), and SLAM (simultaneous localization and mapping) technology to distinguish between mirrors and glass and assign three-dimensional spatial coordinates to the mirrors and glass (for example, see Non-Patent Document 1). The technique disclosed in Non-Patent Document 1 distinguishes whether an object is a mirror or glass based on reflected waves of ultrasonic waves output from an ultrasonic sensor and reflected light of light output from an LED. The technique disclosed in Non-Patent Document 1 then assigns three-dimensional spatial coordinates to the mirrors and glass based on the discrimination results regarding the mirrors and glass and the three-dimensional spatial coordinates obtained by SLAM.

[0003] Keisuke Aoki and Masao Yanagisawa, "Mirror and Glass Mapping Method in RGB-D SLAM," IPSJ 81st National Convention 7R-03, Proceedings of the 81st National Convention 2019 (1), 227-228, 2019-02-28

[0004] The technology disclosed in Non-Patent Document 1 is based on the premise of determining only whether an object is a mirror or glass. Therefore, when an object different from a mirror surface or a glass surface is present, the technology of Non-Patent Document 1 cannot determine the boundary between the object and the mirror surface or the glass surface. Specifically, for example, in an environment where there is a wall and a glass door, the technology of Non-Patent Document 1 cannot determine where the wall ends and where the glass door ends.

[0005] The disclosed technology has been made in consideration of the above points, and aims to provide an information processing device, an information processing method, and an information processing program that can accurately acquire three-dimensional point cloud data of transparent objects.

[0006] A first aspect of the present disclosure is an information processing device including a first acquisition unit that acquires an image captured by a camera mounted on a moving body, a second acquisition unit that acquires three-dimensional point cloud data acquired by a sensor mounted on the moving body, a detection unit that detects transparent areas from the image, including areas where reflected light is captured, and an identification unit that identifies the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by correlating the positions represented by each point of the three-dimensional point cloud data with the positions of each pixel in the transparent area.

[0007] A second aspect of the present disclosure is an information processing method in which a computer executes the following process: acquires an image captured by a camera mounted on a moving body; acquires three-dimensional point cloud data acquired by a sensor mounted on the moving body; detects a transparent area from the image, including an area where reflected light is captured; and identifies the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by associating the positions represented by each point of the three-dimensional point cloud data with the positions of each pixel in the transparent area.

[0008] According to the disclosed technology, it is possible to obtain the effect of accurately acquiring three-dimensional point cloud data of a transparent object.

[0009] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to an embodiment; FIG. 2 is a block diagram showing an example of a functional configuration of an information processing device according to an embodiment; FIG. 3 is a diagram showing an example of an overall configuration of an information processing system according to an embodiment; FIG. 4 is a diagram for explaining identification of a reflective area when the Canny algorithm is used; FIG. 5 is a diagram for explaining a trained model of an embodiment; FIG. 6 is a diagram showing an example of a case where a glass surface is a transparent object and an object exists behind the glass surface; FIG. 7 is a diagram showing an example of a case where a glass surface is a transparent object and an object exists behind the glass surface; and FIG. 8 is a diagram for explaining the operation of an information processing device according to an embodiment.

[0010] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that in each drawing, the same or equivalent components and parts are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0011] First, the hardware configuration of an information processing device 10 according to this embodiment will be described with reference to FIG.

[0012] FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing apparatus 10 according to this embodiment.

[0013] 1, the information processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 18 so as to be able to communicate with each other.

[0014] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a learning program and an estimation program.

[0015] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0016] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various information to the device itself.

[0017] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

[0018] The communication interface 17 is an interface for the device itself to communicate with other external devices. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI (Fiber Distributed Data Interface) or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0019] The information processing apparatus 10 according to this embodiment is implemented as a general-purpose computer device such as a server computer or a personal computer (PC).

[0020] Next, the functional configuration of the information processing device 10 will be described with reference to FIG.

[0021] FIG. 2 is a block diagram showing an example of the functional configuration of the information processing device 10 according to this embodiment.

[0022] 2, the information processing device 10 includes, as functional components, a data storage unit 100, a control unit 101, a first acquisition unit 102, a second acquisition unit 104, a detection unit 106, and an identification unit 108. Each functional component is realized by the CPU 11 reading out an information processing program stored in the ROM 12 or the storage 14, expanding the program in the RAM 13, and executing the program.

[0023] 3 is a block diagram showing an example of the overall configuration of an information processing system 20 according to this embodiment. As shown in FIG. 3, the information processing system 20 according to this embodiment includes an information processing device 10 and a drone M, which is an example of a moving object. The drone M is equipped with a light source LS, a camera C, a sensor LA for acquiring three-dimensional point cloud data, and a position sensor such as a GPS sensor (not shown). For example, the information processing device 10 is mounted on the drone M.

[0024] The information processing system 20 according to this embodiment detects transparent objects such as glass, which are considered difficult to detect even with known technologies. Currently, drones are increasingly being introduced into surveying, security surveillance, and the like. Specific examples of drone-based surveying include flying drones to collect information on topography and land development progress. Furthermore, specific examples of drone-based security surveillance include flying drones to monitor indoor or outdoor facilities. In these applications, obstacle detection is a challenge for drones. Transparent objects such as window glass and glass doors in buildings are particularly difficult to detect.

[0025] Therefore, the information processing system 20 of this embodiment detects transparent objects such as glass windows or glass doors by using an optical camera mounted on a drone and a LiDAR device, which is an example of a sensor mounted on a drone. In this embodiment, a camera C is mounted on the drone M as the optical camera, and a sensor LA is mounted on the drone M as the LiDAR device.

[0026] 3, when the drone M flies, it captures images of the surroundings with the camera C and outputs a laser to the surroundings from the sensor LA. Note that the drone M is intended for measurement, instrumentation, or imaging purposes, and flies, for example, at a low speed while capturing images of the surroundings rather than flying at a high speed.

[0027] The light source LS outputs light. The light source LS is configured so that the wavelength, intensity, and angle of the light output from the light source LS can be changed by a control unit 101, which will be described later.

[0028] The camera C is, for example, an optical camera capable of capturing RGB images or RGB video. The camera C captures images or video around the drone M while the drone M is moving.

[0029] When light output from the light source LS is irradiated onto a transparent object such as glass, the light is reflected by the transparent object. The reflection of light by the transparent object is captured in the image captured by the camera C, improving the detection accuracy of the transparent object in the processing described below. As described below, the amount of light output from the light source LS may be adjusted, the color of the light may be changed, or dimming techniques such as PWM (Pulse Width Modulation) or phase control may be used, taking into account the amount of light around the drone M, the direction of natural light L1, or the frame rate when capturing images by the camera C.

[0030] The sensor LA acquires three-dimensional point cloud data of an object. The sensor LA is, for example, a sensor that can be employed in LiDAR (Light Detection and Ranging) technology.

[0031] As shown in FIG. 3 , the information processing system 20 of this embodiment detects the glass surface T by utilizing reflected light from the glass surface T, which is an example of a transparent object. Specifically, when natural light L1 and light L2 output from a light source LS are applied to the glass surface T, regions R1 and R2 (hereinafter simply referred to as "reflective regions") in which the reflected light is reflected are formed. The information processing system 20 of this embodiment detects the glass surface T in which the reflective regions R1 and R2 are reflected by image processing. The information processing system 20 then identifies the 3D point cloud data of the glass surface T, which is a transparent object, by associating the 3D point cloud data obtained by the laser LA1 output from the sensor LA with the detection result of the glass surface T, including the reflective regions R1 and R2. A specific description will be given below.

[0032] The data storage unit 100 stores images captured by the camera C while the drone M moves, and three-dimensional point cloud data acquired by the sensor LA while the drone M moves. As described above, the drone M is equipped with a position sensor (not shown), such as a GPS sensor. Therefore, based on the positional relationship between the position sensor (not shown) and the camera C and the attitude information of the camera C, the x- and y-coordinates, which are the two-dimensional coordinates of each pixel in the image captured by the camera C, are known. Furthermore, based on the positional relationship between the position sensor (not shown) and the sensor LA and the attitude information of the sensor LA, the x-, y-, and z-coordinates, which are the three-dimensional coordinates of each point in the three-dimensional point cloud data acquired by the sensor LA, are known. By performing a known calibration between the camera C and the sensor LA in advance, it is possible to associate the coordinates of the three-dimensional point cloud data with the coordinates of the image. The data storage unit 100 also stores a trained model, which will be described later.

[0033] The control unit 101 controls the light source LS to change at least one of the wavelength, intensity, and angle of the light output while the drone M is moving and while an image is being captured by the camera C. This causes various types of light to be irradiated onto the glass surface T, forming various reflective areas. The presence of these various reflective areas on the glass surface T enables the detection unit 106, which will be described later, to detect the glass surface T with high accuracy.

[0034] Conventionally, reflective areas have been the target of removal, and various techniques for removing reflective areas have been proposed. Reflective areas are also called "blown-out highlights." For example, Reference 1 below discloses a technique for adjusting the position of a light source when blown-out highlights occur and changing the position of the blown-out highlights in order to resolve the obscuration of objects in an image due to blown-out highlights.

[0035] Reference 1: "Correction of Overexposed and Underexposed Areas in Images Using Multiple Lighting Control for Dark Area Investigation," ITE Technical Report 39.30 (2015)

[0036] While reflective areas have traditionally been removed, the information processing system 20 of this embodiment uses the reflective areas of transparent objects to accurately detect transparent objects. Specifically, in an environment where light reflection is difficult, such as indoors, the control unit 101 of the information processing device 10 of the information processing system 20 intentionally generates light reflection on the glass surface T by outputting light from an artificial light source LS mounted on the drone M. This makes it easier to detect transparent objects. When outputting light from the light source LS, the control unit 101 of the information processing device 10 adjusts the wavelength, intensity, and angle of the light output from the light source LS.

[0037] For example, as described above, the control unit 101 of the information processing device 10 changes the wavelength of light output from the light source LS. Changing the wavelength of the light output from the light source LS changes the color of the light. In this case, for example, the control unit 101 of the information processing device 10 changes the wavelength of the light output from the light source LS to match the environment around the drone M. For example, if the walls around the drone M are white, illuminating a transparent object such as a glass door with white light would not clearly distinguish the glass door from the surrounding walls, making it difficult to identify the glass door. Therefore, the control unit 101 of the information processing device 10 outputs light that does not exist in the area, such as purple or green, from the light source LS. In this case, for example, the control unit 101 of the information processing device 10 recognizes the environment around the drone M based on an image captured by the camera C. For example, if the control unit 101 of the information processing device 10 recognizes that a white wall is captured in the image, it outputs light different from white light from the light source LS.

[0038] Furthermore, the control unit 101 of the information processing device 10 controls the light source LS to output light that clearly defines the contours of transparent objects. For example, known photography techniques such as high key or low key can be used to clearly define the contours of transparent objects. These photography techniques adjust exposure to prevent blurring of contours, primarily due to excessive whiteout. Furthermore, the dimming control of the irradiated light may be adjusted using known PWM dimming or known phase control dimming to generate whiteout that makes it easier to detect transparent objects.

[0039] When performing the above-described adjustment, the control unit 101 of the information processing device 10 may determine whether or not to output light from the light source LS, depending on the amount of light around the drone M. When performing the above-described adjustment, the control unit 101 of the information processing device 10 may adjust the wavelength, intensity, and angle of the light output from the light source LS, depending on the amount of light around the drone M. In this case, the drone M is equipped with a light sensor (not shown), and the control unit 101 of the information processing device 10 adjusts the wavelength, intensity, and angle of the light output from the light source LS, depending on the amount of light detected by the light sensor (not shown).

[0040] The first acquisition unit 102 acquires images captured by the camera C mounted on the drone M from the data storage unit 100.

[0041] The second acquisition unit 104 acquires, from the data storage unit 100, three-dimensional point cloud data acquired by the sensor LA mounted on the drone M.

[0042] The detection unit 106 detects a transparent area including a reflective area from the image acquired by the first acquisition unit 102. In the following, an example will be described in which the transparent area is a glass surface. For example, the detection unit 106 detects a glass surface including a reflective area from the image using the following various methods. Note that which of the following methods is to be adopted is determined taking into consideration, for example, the real-time nature of the detection process and the detection accuracy. The following various methods may be combined to detect a glass surface.

[0043] For example, the detection unit 106 extracts a reflection area shown in an image by performing threshold processing on the image based on a preset threshold value related to pixel values (for example, a threshold value related to color).

[0044] When light irradiates an object, the reflected light varies depending on the color of the object. For example, when light irradiates a red object, the reflected light is also red. When light irradiates a transparent object, the reflected light is the color specific to the transparent object. Therefore, by imaging the area where this reflected light appears with the camera C, the area where the reflected light specific to the transparent object appears is converted into RGB values. And a threshold value related to the color of the reflected light specific to the transparent object is preset. The detection unit 106 detects whether the reflection area shown in the image is a transparent area based on the preset threshold value. Specifically, the detection unit 106 identifies an area within the image whose RGB values are within a predetermined threshold range as a transparent area. Note that light that does not exist in nature or in the environment where the drone M operates (for example, purple light, etc.) may be output from the light source LS, the light may be reflected by the transparent area, and the reflected light may be detected to make it easier to detect the transparent area.

[0045] Therefore, the detection unit 106 extracts pixels having pixel values within a preset threshold range as a reflection area. And the detection unit 106 identifies the extracted reflection area as a glass surface.

[0046] Alternatively, for example, the detection unit 106 identifies a transparent area shown in an image by using the known Canny method. The Canny method is an edge detection algorithm devised by Mr. Canny in 1986. The Canny method is disclosed in the following Reference 2.

[0047] Reference 2: John Canny, "A Computational Approach to Edge Detection", <Internet: https: / / ieeexplore.ieee.org / document / <4767851>

[0048] The Canny algorithm is still widely used as an edge detection algorithm. The Canny algorithm roughly smooths an image using a known Gaussian filter, and then extracts potential brightness gradients (edges) using a known Sobel filter. The Canny algorithm then detects edges by adjusting the brightness gradient threshold, known as hysteresis.

[0049] More specifically, in the known Canny algorithm, a first threshold and a second threshold smaller than the first threshold are preset, and edges within an image are extracted using the preset thresholds. The detection unit 106 detects edges representing boundaries between reflective regions and other regions within the image. The detection unit 106 extracts a group of pixels having pixel values ​​greater than the first threshold as the boundary of the reflective region. On the other hand, the detection unit 106 extracts a group of pixels having pixel values ​​equal to or less than the first threshold and equal to or greater than the second threshold as a candidate boundary of the reflective region. Then, if the candidate boundary extracted using the second threshold is adjacent to the boundary extracted using the first threshold, the detection unit 106 extracts the candidate boundary extracted using the second threshold as the boundary of the reflective region.

[0050] FIG. 4 is a diagram illustrating the identification of reflective areas using the Canny algorithm. In the image shown on the left side of FIG. 4, each of the multiple reflective areas R1, R2, R3, R4, and R5 is depicted as an independent figure, but in reality, a gradation is applied, and the reflected light appears to cover the entire transparent area. As shown in FIG. 4, when the Canny algorithm is applied to image IM1, which contains multiple reflective areas R1, R2, R3, R4, and R5, image IM2 is generated, which contains a glass surface T formed by integrating the multiple reflective areas R1, R2, R3, R4, and R5. In this way, by using the Canny algorithm, the outlines of the reflective areas are extracted, and these outlines are considered to be the outlines of the transparent areas. In the process described below, three-dimensional point cloud data of the transparent areas is identified based on the two-dimensional coordinates of the outlines of the transparent areas.

[0051] Alternatively, for example, the detection unit 106 may extract the contour of the reflective area using a known Hough transform to identify the reflective area.

[0052] Alternatively, for example, the detection unit 106 may identify the reflective region using a machine learning algorithm such as deep learning. In this case, a trained model is generated in advance using a known machine learning algorithm. This trained model is specialized for detecting transparent objects including reflective regions. For example, a known convolutional neural network (CNN) model or a transformer model may be adopted as the trained model.

[0053] 5 is a diagram illustrating the trained model of this embodiment. As shown in FIG. 5, when an image IM1 showing multiple reflection regions R1, R2, R3, R4, and R5 is input to the trained model LM, the trained model LM outputs an image IM2 showing a glass surface T in which the multiple reflection regions R1, R2, R3, R4, and R5 are integrated.

[0054] To generate a learned model LM as shown in FIG. 5, various processes such as data collection, teacher image generation, and learning model training are performed.

[0055] Specifically, first, an image of the glass reflecting the light reflection area is collected. If it is difficult to prepare an image of the glass reflecting the light reflection area, a composite image of the glass reflecting the light reflection area may be used by image synthesis.

[0056] Next, a training image is created. Specifically, annotation of the light reflection area and annotation of the glass area are performed on the collected images of the glass that show the light reflection area.

[0057] Then, training of the learning model is performed. Specifically, the learning model is trained using a known machine learning algorithm, thereby generating a learned model LM as shown in Fig. 5. In this case, the learning model is trained, evaluated, and adjusted, thereby generating the learned model LM.

[0058] When identifying a glass surface using the learned model LM, the detection unit 106 inputs the image IM1 acquired by the first acquisition unit 102 to the learned model LM. In this case, the learned model LM outputs an image IM2 showing the glass surface T in which multiple reflection regions R1, R2, R3, R4, and R5 are integrated.

[0059] The identification unit 108 identifies the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by associating the positions represented by each point of the three-dimensional point cloud data with the positions of each pixel in the transparent area.

[0060] As described above, each point in the three-dimensional point cloud data acquired by the sensor LA of the drone M is assigned three-dimensional x, y, and z coordinates, and the image captured by the camera C of the drone M is assigned two-dimensional x, y coordinates. Note that the x, y coordinates are coordinates in the vertical and horizontal directions as viewed from the drone M, and the z coordinate is a coordinate in the depth direction as viewed from the drone M.

[0061] Therefore, the identification unit 108 identifies the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by associating the surface x-y coordinates of each point of the three-dimensional point cloud data with the x-y coordinates of each pixel in the transparent area.

[0062] 6 and 7 are diagrams showing an example of a case where a glass surface T, which is a transparent object, and an object B are present behind the glass surface T. As shown in Fig. 6, when a plurality of reflective regions R1, R2, R3, and R4 are present, the glass surface T is detected, for example, by image processing by the detection unit 106. The identification unit 108 projects the three-dimensional point cloud data onto the image side using the two-dimensional plane of the image as a reference, and considers the three-dimensional point cloud data present within the range of the region of the glass surface T detected on the image side to be the three-dimensional point cloud data of the glass surface T.

[0063] Also, as shown in FIG. 7 , consider a case where a laser beam LA1 output from the sensor LA is irradiated onto the glass surface T under favorable conditions, such as when the drone M and the glass surface T are positioned directly opposite each other. In this case, although the amount of laser beam RL2 reflected by the glass surface T is small, it is detected, and three-dimensional point cloud data P1 of the glass surface T is formed. Immediately after collection, it is unclear whether such three-dimensional point cloud data P1 is due to the glass surface T. For this reason, the identification unit 108 combines the transparent region of the glass surface T detected by the detection unit 106 with the collection results of the three-dimensional point cloud data to determine whether the acquired three-dimensional point cloud data is three-dimensional point cloud data of the glass surface T.

[0064] 7, when a laser beam LA1 is output from the sensor LA, a portion of the laser beam LA1 passes through the glass surface T and becomes a laser beam LA2, which is then irradiated onto the object B. The laser beam RL2 reflected from the object B returns toward the sensor LA and is detected by the sensor LA. Meanwhile, the laser beam RL2 reflected by the glass surface T returns toward the sensor LA and is detected by the sensor LA. Therefore, the sensor LA detects three-dimensional point cloud data P1 of the glass surface T and three-dimensional point cloud data P2 of the object B.

[0065] In a situation such as that shown in FIG. 7 , there is an overlapping portion of the glass surface T and the object B on the two-dimensional plane coordinates as viewed from the drone M. In such a case, the identification unit 108 distinguishes between the three-dimensional point cloud data P1 representing the glass surface T and the three-dimensional point cloud data P2 representing the object B. This distinction is made using coordinate information in the x- and y-directions and the z-direction of the three-dimensional point cloud data. First, as shown in FIG. 6 , the identification unit 108 associates the x- and y-coordinates of the transparent region in the image with the x- and y-coordinates of the three-dimensional point cloud data, thereby designating the three-dimensional point cloud data present in the transparent region as a candidate for the three-dimensional point cloud data of the transparent object. Then, as shown in FIG. 7 , the identification unit 108 considers the three-dimensional point cloud data P1 that is closest to the drone M as the three-dimensional point cloud data of the glass surface T based on the three-dimensional coordinates assigned to each point of the three-dimensional point cloud data.

[0066] The three-dimensional point cloud data of the transparent object identified in this manner has known three-dimensional coordinates, so the distance between the drone M and the transparent object that is an obstacle becomes clear.

[0067] 8 is a flowchart showing an example of the flow of processing by the information processing program according to this embodiment. The processing by the information processing program is realized by the CPU 11 of the information processing device 10 writing the information processing program stored in the ROM 12 or the storage 14 to the RAM 13 and executing it.

[0068] When the information processing device 10 receives a predetermined instruction signal, it executes the information processing of Fig. 8. Note that the information processing of Fig. 8 is executed repeatedly while the drone M is flying. Images or videos captured by the camera C of the drone M are sequentially stored in the data storage unit 100. In addition, three-dimensional point cloud data acquired by the sensor LA of the drone M is also sequentially stored in the data storage unit 100.

[0069] 8, the CPU 11, as the first acquisition unit 102, acquires an image captured by the camera C mounted on the drone M from the data storage unit 100. Also, in step S100, the CPU 11, as the second acquisition unit 104, acquires three-dimensional point cloud data acquired by the sensor LA mounted on the drone M from the data storage unit 100.

[0070] In step S102, the CPU 11 functions as the detection unit 106 to detect transparent areas including reflective areas from the image acquired in step S100.

[0071] In step S104, the CPU 11, as the identification unit 108, identifies the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by associating the positions represented by each point of the three-dimensional point cloud data acquired in step S100 with the positions of each pixel of the transparent area detected in step S102.

[0072] As described above, the information processing device of this embodiment acquires an image captured by a camera mounted on a drone, which is an example of a moving object. The information processing device acquires 3D point cloud data acquired by a sensor mounted on the drone. The information processing device detects a transparent area, including an area where reflected light is captured, from the image. The information processing device then associates the positions represented by each point in the 3D point cloud data with the positions of each pixel in the transparent area, thereby identifying the 3D point cloud data corresponding to the transparent area as 3D point cloud data of a transparent object. This makes it possible to accurately acquire 3D point cloud data of a transparent object. Specifically, the information processing device of this embodiment detects a transparent area through image processing, thereby making it possible to accurately distinguish between a transparent area and other areas. This makes it possible to accurately identify 3D point cloud data of a transparent object corresponding to the detected transparent area.

[0073] Furthermore, the drone of this embodiment is equipped with a light source, and the information processing device controls the light source to change at least one of the wavelength, intensity, and angle at which the light is output. The information processing device acquires images captured while the control is being executed. This allows various types of light to hit the transparent object, making it possible to accurately detect the transparent area based on the reflected light. Furthermore, in this embodiment, in addition to natural light, an auxiliary light source is used to expand the light reflection area on the transparent object, making it possible to accurately detect the transparent area using the reflected light.

[0074] In addition, while conventionally, light reflection has often been removed as noise, the information processing device of this embodiment detects transparent areas using light reflection and identifies the 3D point cloud data corresponding to the transparent areas as the 3D point cloud data of a transparent object, thereby enabling the 3D point cloud data of the transparent object to be identified with high accuracy.

[0075] Currently, drones and robots, which are examples of mobile objects, are being increasingly introduced in various situations such as nursing care, logistics, and customer service. However, recognition of transparent objects such as glass doors and glass windows is still difficult. In response to this, the present embodiment solves the following two problems regarding the recognition of transparent objects.

[0076] The first issue is the recognition of transparent objects. Determining whether a transparent object exists in the direction of travel of a moving body is essential for the safe movement of a robot or drone, which is an example of a moving body. The information processing device of this embodiment makes it possible to accurately acquire three-dimensional point cloud data of a transparent object, thereby enabling accurate recognition of the transparent object.

[0077] The second is identifying the three-dimensional position of a transparent object. In creating a movement algorithm for a moving object, it is important to identify not only whether or not a transparent obstacle exists in the moving direction of the moving object, but also the distance from the moving object to the obstacle. The information processing device of this embodiment makes it possible to accurately acquire three-dimensional point cloud data of a transparent object, and also to accurately recognize the distance from the moving object to the transparent object.

[0078] In the above embodiment, the moving body is a drone, but the present invention is not limited to this. Any moving body may be used. For example, a walking robot or the like may be used as the moving body.

[0079] In addition, when a mobile object is used to survey terrain or buildings, the three-dimensional point cloud data of identified transparent objects can be converted into textures and used to generate 3D images corresponding to the survey results.

[0080] In the above embodiment, the processes that the CPU 11 reads and executes each program may be executed by various processors other than the CPU 11. Examples of processors in this case include dedicated electrical circuits, such as programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and application-specific integrated circuits (ASICs) that are processors having a circuit configuration specifically designed to execute specific processes. Each process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0081] In the above embodiment, the program is pre-stored (also referred to as "installed") in the ROM 12 or the storage 14, but the present invention is not limited to this. The information processing program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0082] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0083] The following additional notes are provided regarding the above-described embodiments.

[0084] (Supplementary Item 1) An information processing device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: acquire an image captured by a camera mounted on a moving body; acquire three-dimensional point cloud data acquired by a sensor mounted on the moving body; detect a transparent area including an area where reflected light is captured from the image; and identify the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by associating the positions represented by each point of the three-dimensional point cloud data with the positions of each pixel of the transparent area.

[0085] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to perform information processing, wherein the information processing comprises: acquiring an image captured by a camera mounted on a moving body; acquiring three-dimensional point cloud data acquired by a sensor mounted on the moving body; detecting a transparent area including an area where reflected light is captured from the image; and identifying the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by associating the positions represented by each point of the three-dimensional point cloud data with the positions of each pixel in the transparent area.

[0086] 100 Data storage unit 101 Control unit 102 Acquisition unit 104 Acquisition unit 106 Detection unit 108 Identification unit

Claims

1. An information processing device comprising: a first acquisition unit that acquires an image captured by a camera mounted on a moving body; a second acquisition unit that acquires three-dimensional point cloud data acquired by a sensor mounted on the moving body; a detection unit that detects transparent areas including areas where reflected light is captured from the image; and an identification unit that identifies the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by matching the positions represented by each point of the three-dimensional point cloud data with the positions of each pixel of the transparent area.

2. The information processing device of claim 1, wherein the moving body is equipped with a light source, and further comprises a control unit that controls to change at least one of the wavelength of light output from the light source, the intensity of the light, and the angle at which the light is output, the control unit controls to change at least one of the wavelength of light output from the light source, the intensity of the light, and the angle at which the light is output, and the first acquisition unit acquires the image captured while control by the control unit is being executed.

3. An information processing method in which a computer executes the following processes: acquiring an image captured by a camera mounted on a moving object; acquiring three-dimensional point cloud data acquired by a sensor mounted on the moving object; detecting transparent areas including areas where reflected light is captured from the image; and identifying the three-dimensional point cloud data corresponding to the transparent area as three-dimensional point cloud data of a transparent object by associating the positions represented by each point of the three-dimensional point cloud data with the positions of each pixel of the transparent area.

4. An information processing program for causing a computer to function as each part of the information processing device according to claim 1 or 2.

Images