Method and device for detecting transparent obstacles based on artificial intelligence

Abstract

A device (1000) for detecting a transparent obstacle based on artificial intelligence may comprise an RGB depth camera (10) configured to generate an RGB image and a depth image, a thermal imaging camera (20) configured to generate a thermal image, and a control device (100) connected to the RGB depth camera (10) and the thermal imaging camera (20) which are synchronized with each other, wherein the control device (100) is configured to generate the RGB image and the depth image aligned with respect to the thermal image, generate an aligned RGB image, an aligned depth image, and an aligned thermal image, detect a pixel area determined to be a transparent obstacle by using an artificial intelligence-based model based on the aligned RGB image and the aligned thermal image,and estimate the depth of the transparent obstacle by using the aligned depth image and the detected pixel area when detecting the pixel area determined to be a transparent obstacle.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an artificial intelligence-based transparent obstacle detection apparatus and method. More specifically, the present disclosure relates to an artificial intelligence-based transparent obstacle detection apparatus and method capable of detecting the transparent obstacle, such as a glass door, when a robot is operating. BACKGROUND

[0002] A mobile robot can move along a fixed path or can move autonomously within a specific environment. Mobile robots can be used in various industrial and service sectors and can take various forms, such as a vehicle, a drone, an intelligent robot, or the like. The primary goal of a mobile robot can be to sense and recognize the environment and safely reach the destination.

[0003] Depth and LiDAR sensors, which can be used in conventional technologies for robot operation and obstacle detection and recognition, may not detect transparent objects such as glass as obstacles. If the robot cannot detect the transparent object during operation, there is a risk of collision during operation. Unlike visible and near-infrared light, long-wave infrared (LWIR) from thermal imaging cameras cannot penetrate glass. BRIEF EXPLANATION

[0004] The present disclosure provides an artificial intelligence-based transparent obstacle detection apparatus and method capable of implementing an image processing system using an RGB depth camera and a thermal imaging camera, detecting a transparent obstacle through deep learning with input of information obtained by the image processing system, and enabling a robot to avoid the detected transparent obstacle when applied to a mobile robot.

[0005] A device for detecting (e.g. recognizing or identifying) a transparent obstacle can comprise a red-green-blue depth camera (RGB depth camera) which is configured to generate at least one image or recording (hereinafter referred to as image) associated with the transparent obstacle, wherein the at least one image comprises red-green-blue data (RGB data) and depth data, a thermal imaging camera which is configured to generate a thermal image associated with the transparent obstacle, and a control device which is connected to the RGB depth camera and the thermal imaging camera, which are synchronized with one another, wherein the control device is configured to:with each other), to capture a pixel area determined as the transparent obstacle based on the aligned at least one image and the aligned thermal image using an artificial intelligence model, and to estimate a depth (e.g. distance, e.g. to the device) of the transparent obstacle based on the captured pixel area and based on the depth data of the aligned at least one image.

[0006] For example, the control device may be configured to: compare the estimated depth of the transparent obstacle with a collision range of a robot (e.g., a predefined area in the environment of the robot with a fixed size), based on the estimated depth being within the collision range, control the robot to prevent a collision of the robot with the transparent obstacle, and based on the estimated depth being outside the collision range, control the robot to move normally.

[0007] For example, the control device may be configured to: extract an RGB camera parameter and a depth camera parameter from the RGB depth camera through camera calibration, extract a thermal camera parameter from the thermal camera, and generate the aligned at least one image and the aligned thermal image by using the extracted RGB camera parameter, the extracted depth camera parameter, and the extracted thermal camera parameter.

[0008] For example, the control device may be configured to: align a depth image of the RGB depth camera and an RGB image of the RGB depth camera (e.g., with each other) to generate the at least one image, and align the at least one image and the thermal image to generate an aligned RGB image, an aligned depth image, and the aligned thermal image (e.g., these three images are aligned with each other).

[0009] For example, the control device may be configured to: extract a feature from each of an aligned RGB image of the RGB depth camera and an aligned thermal image of the RGB depth camera using a deep learning model, extract regions in which different images (e.g., views of the extracted feature) are formed as a pixel region based on the extracted feature, and detect pixels of the pixel region as a transparent obstacle.

[0010] For example, the controller may be configured to: divide the pixel area into a plurality of groups using a grouping algorithm; and detect from the plurality of groups a lower outer boundary line near a floor surface (e.g., on which the robot will move).

[0011] For example, the control device may be configured to stop estimating the depth of the transparent obstacle based on the non-detection of the lower outer boundary line.

[0012] For example, the control device may be configured to estimate the depth of the transparent obstacle based on the detected lower outer boundary line, wherein the depth of the transparent obstacle is estimated based on camera space information including a location of the thermal imaging camera and a direction (e.g., orientation) of the thermal imaging camera, and a depth error value measured in a depth image of the RGB depth camera.

[0013] For example, the control device may be configured to determine, based on the detected lower outer boundary line, a depth average of pixels having the same specific coordinate value as the pixel range among neighboring pixels below the lower outer boundary line as the depth of the floor surface and to estimate the determined depth of the floor surface as the depth of the transparent obstacle.

[0014] For example, the control device may be configured to send a depth query with respect to pixels of pixel regions that were not determined to be a transparent obstacle based on the undetected pixel region via an aligned depth image of the RGB depth camera.

[0015] A method for detecting a transparent obstacle may include: synchronizing a red-green-blue (RGB) depth camera and a thermal imaging camera, aligning a red-green-blue (RGB) image and a depth image generated by the RGB depth camera with a thermal image generated by the thermal imaging camera to generate an aligned RGB image, an aligned depth image, and an aligned thermal image, wherein each of the aligned RGB image, the aligned depth image, and the aligned thermal image is associated with the transparent obstacle, detecting a pixel region determined to be a transparent obstacle based on the aligned RGB image and the aligned thermal image using an artificial intelligence model, and estimating a depth of the transparent obstacle based on the detected pixel region and the aligned depth image.

[0016] For example, the method may further comprise comparing the estimated depth of the transparent obstacle with a collision range of a robot, controlling the robot to stop based on the estimated depth being within the collision range, and controlling the robot to move normally based on the estimated depth being outside the collision range.

[0017] For example, the method may further comprise extracting an RGB camera parameter and a depth camera parameter from the RGB depth camera through camera calibration, extracting a thermal camera parameter from the thermal camera, and generating the aligned RGB image, the aligned depth image, and the aligned thermal image using the extracted RGB camera parameter, the extracted depth camera parameter, and the extracted thermal camera parameter.

[0018] For example, aligning the RGB image and the depth image with the thermal image may comprise aligning the depth image with the RGB image, and aligning the aligned depth image and the aligned RGB image with the thermal image to generate the aligned RGB image, the aligned depth image, and the aligned thermal image.

[0019] For example, detecting the pixel region determined as a transparent obstacle may include: extracting a feature from each of the aligned RGB image and the aligned thermal image via a deep learning model, extracting regions in which different images are formed based on the extracted feature as the pixel region, and detecting pixels of the pixel region as the transparent obstacle.

[0020] For example, estimating the depth of the transparent obstacle may comprise: clustering the pixel area into a plurality of groups using a clustering algorithm, and detecting a lower outer boundary line (e.g., of the transparent obstacle) near a ground surface from the plurality of groups.

[0021] For example, estimating the depth of the transparent obstacle may comprise: stopping the estimation of the depth of the transparent obstacle based on the lower outer boundary line not being detected.

[0022] For example, estimating the depth of the transparent obstacle may comprise: estimating the depth of the transparent obstacle based on the detected lower outer boundary line, wherein the depth of the transparent obstacle is estimated based on camera spatial information including a location of the thermal imaging camera and a direction (e.g., orientation) of the thermal imaging camera, and a depth error value measured in the depth image.

[0023] For example, estimating the depth of the transparent obstacle may include determining a depth average of pixels with respect to neighboring pixels below the lower outer boundary line based on the detected lower outer boundary line, applying the depth average to pixels having a same coordinate value as a specific coordinate value in the pixel range, and estimating a depth of the ground surface (e.g., at the specific coordinate values) as the depth of the transparent obstacle.

[0024] For example, the method may further comprise: based on the non-detecting of the pixel region, sending a request for a depth with respect to pixels of pixel regions not determined as the transparent obstacle through the aligned depth image. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows an example of an apparatus for detecting a transparent obstacle based on artificial intelligence according to an example. Fig. 2 to Fig. 5 show examples of a method for detecting a transparent obstacle based on artificial intelligence according to an example. Fig. 6 shows an example of a drawing illustrating a computing device according to an example. DETAILED DESCRIPTION

[0025] An example of the disclosure will be described in more detail below with reference to the accompanying drawings so that those skilled in the art can easily implement the example. As those skilled in the art would appreciate, the described examples can be modified in various ways without departing from the spirit or scope of the present disclosure. For the sake of clarity of the present disclosure, parts not related to the description are omitted, and the same elements or modifications are referred to by the same reference numerals throughout the description.

[0026] Additionally or alternatively, unless expressly stated otherwise, the word "comprise" and variations such as "comprises" or "comprising" are to be understood as implying the inclusion of the specified elements, but not the exclusion of others. Terms that have a common number, such as first and second, are used to describe various components, but the components are not limited by the terms. The terms are used only to distinguish one component from other components.

[0027] In addition, the terms “unit,” “part,” “section,” “module,” etc., in the description refer to a unit that performs at least one operation or function that can be implemented by hardware, software, or a combination of hardware and software.

[0028] Examples of the present disclosure will now be described with reference to the drawings.

[0029] Fig. 1 shows an example of an artificial intelligence-based transparent obstacle detection device according to an example.

[0030] A transparent obstacle detection device 1000, for example, based on artificial intelligence, can be applied to a mobile robot and / or any mobile device (e.g., a vehicle, a drone, an autonomous vehicle, etc.). For example, the mobile robot can detect a transparent obstacle, such as a glass door, while moving through the artificial intelligence transparent obstacle detection device 1000.

[0031] The artificial intelligence-based transparent obstacle detection device 1000 may use an RGB depth camera and a thermal imaging camera together to detect a transparent obstacle. The artificial intelligence-based transparent obstacle detection device 1000 may detect the transparent glass within the image by using a difference between the RGB image and the thermal image.

[0032] The artificial intelligence-based transparent obstacle detection device 1000 may include an image processing system including an RGB depth camera and a thermal imaging camera, and may detect the transparent obstacle from the image using artificial intelligence. The artificial intelligence-based transparent obstacle detection device 1000 may, for example, send (e.g., transmit) a stop or move command to the robot based on the detected transparent obstacle.

[0033] Fig. 1 shows an example of a configuration for detecting a transparent obstacle based on artificial intelligence, which has a control device connected to an image processing system.

[0034] With reference to Fig. 1, the artificial intelligence-based transparent obstacle detection device 1000 may include an RGB depth camera (RGBD) 10, a thermal imaging camera 20, and a control device 100.

[0035] The image processing system may include the RGB depth camera 10 and the thermal imaging camera 20.

[0036] The RGB depth camera 10 can generate RGB images and / or depth images by recording.

[0037] The RGB image can contain color information of an object. The depth image can contain distance information from each pixel of an object to the camera. The depth image may be required for alignment between the RGB image and the thermal image and can be used to estimate the depth or distance of the transparent obstacle, for example, as auxiliary information.

[0038] The thermal imaging camera 20 can generate the thermal image. The thermal image can show the temperature distribution of an object.

[0039] The RGB depth camera 10 and the thermal imaging camera 20 can be attached or fixed to each other. The recording directions of the RGB depth camera 10 and the thermal imaging camera 20 can be parallel to each other. For example, a center of the lens of the thermal imaging camera 20 can be located approximately 25 mm above a center of the RGB camera 10.

[0040] The RGB depth camera 10 and the thermal imaging camera 20 can, for example, each send (e.g., transmit) individual images from simultaneously recorded images to the control device 100 connected thereto.

[0041] The control device 100 may be connected to the RGB depth camera 10 and the thermal imaging camera 20, which are synchronized with each other.

[0042] The control device 100 may be mounted on the mobile robot, and for operating the driving algorithm, a plurality of control devices 100 may be provided. At least one control device 100 may be required to implement the present disclosure.

[0043] The control device 100 may convert the captured frames to be used as input for a deep learning model and may operate the transparent obstacle detection algorithm to estimate the transparent obstacle using the deep learning model.

[0044] The control device 100 may align the RGB image and the depth image with respect to the thermal image and generate an aligned RGB image, an aligned depth image, and an aligned thermal image.

[0045] The control device 100 may detect a pixel area determined as the transparent obstacle by using the artificial intelligence model based on the aligned RGB image and / or the aligned thermal image.

[0046] When the pixel area determined to be a transparent obstacle is detected, the control device 100 may estimate the depth of the transparent obstacle using the aligned depth image and / or the detected pixel area.

[0047] The control device 100 can improve the detection performance for the transparent obstacle such as glass by effectively modeling a difference between the RGB image and the thermal image through the deep learning model.

[0048] The control device 100 may compare the estimated depth or distance of the transparent obstacle with a collision range predetermined in the robot that can currently move.

[0049] If the estimated depth is within the collision range, the control device 100 can stop the robot. If the estimated depth is outside the collision range, the control device 100 can control the robot to move normally.

[0050] The control device 100 can extract an RGB camera parameter and a depth camera parameter from the RGB depth camera through camera calibration, and can extract a thermal imaging camera parameter from the thermal imaging camera.

[0051] The control device 100 may generate the aligned RGB image, the aligned depth image, and / or the aligned thermal image using the extracted RGB camera parameter, the depth camera parameter, and / or the thermal camera parameter.

[0052] The control device 100 may generate a primary aligned depth image and a primary aligned RGB image by primarily aligning the depth image with respect to the RGB image.

[0053] The control device 100 may generate the aligned RGB image, the aligned depth image, and / or the aligned thermal image by finally aligning the primary aligned depth image and the primary aligned RGB image, respectively, with respect to the thermal image.

[0054] The control device 100 can extract a feature from both the aligned RGB image and the aligned thermal image through the deep learning model.

[0055] The control device 100 may extract areas where different shots may be formed in the image as pixel areas based on each extracted feature, and detect the pixels of the extracted pixel area as a transparent obstacle.

[0056] The control device 100 may divide the pixel area into a plurality of groups using a grouping algorithm and may detect a lower outer boundary line near a ground from the plurality of groups.

[0057] If the lower outer boundary line is not detected, the control device 100 may stop estimating the depth with respect to the transparent obstacle.

[0058] When the lower outer boundary line is detected, the control device 100 may estimate an actual depth of the transparent obstacle based on camera space information including the location and direction (e.g., orientation) of the thermal imaging camera and a false depth value measured in the depth image.

[0059] When the lower outer boundary line is detected, the control device 100 may calculate a depth average of pixels having the same specific coordinate value as the pixel range among neighboring pixels below the lower outer boundary line as a depth of a ground surface.

[0060] The control device 100 may estimate the calculated depth of the ground surface as the depth of the transparent obstacle.

[0061] When the pixel area determined as a transparent obstacle is not detected, the control device 100 may retrieve an actual depth with respect to pixels of pixel areas not determined as a transparent obstacle via the aligned depth image.

[0062] Fig. 2 to Fig. 5 show examples of a method for detecting a transparent obstacle based on artificial intelligence according to an example. A method for detecting a transparent obstacle based on artificial intelligence according to Fig. 2 to Fig. 5 can be realized by the device 1000 for detecting a transparent obstacle based on artificial intelligence according to Fig. 1 should be carried out.

[0063] Fig. 2 shows an example of a method for detecting a transparent obstacle based on artificial intelligence according to an example. Fig. 2 shows an example of a scenario in which the mobile robot according to the present disclosure can detect and avoid the transparent obstacle (e.g., glass) when moving.

[0064] The glass may be a representative example of the transparent obstacle, and the transparent obstacle of the present disclosure is not specifically limited to the glass, but may be the concept covering all transparent objects including glass.

[0065] In detail, Fig. 2 shows an exemplary process in which the device 1000 capable of detecting a transparent obstacle based on artificial intelligence can capture or acquire the individual images through the RGB depth camera 10 and the thermal camera 20, estimate the area where the transparent obstacle is located by using it as input to the deep learning model, and instruct the robot to avoid the transparent obstacle based on the estimation result.

[0066] In Fig. 2, the artificial intelligence-based transparent obstacle detection device 1000 may synchronize the two cameras (e.g., the RGB depth camera 10 and the thermal imaging camera 20) and / or photograph or capture images simultaneously in step S100.

[0067] For example, the artificial intelligence-based transparent obstacle detection device 1000 may synchronize the RGB depth camera 10 and the thermal imaging camera 20 by connecting cables to a synchronization board or to each other, or it may measure and adjust the shooting time.

[0068] When the RGB depth camera 10 and the thermal camera 20 capture or acquire the single image in step S300, the artificial intelligence-based transparent obstacle detection apparatus 1000 may adjust and align the RGB image, the depth image, and the thermal images to have the same screen coordinates.

[0069] The artificial intelligence-based transparent obstacle detection device 1000 can align the RGB image and the depth image generated by the RGB depth camera with respect to the thermal image generated by the thermal camera, and generate the aligned RGB image, the aligned depth image, and the aligned thermal image. The alignment process is described with reference to Fig. 3 described in detail.

[0070] In step S400, the artificial intelligence-based transparent obstacle detection device 1000 may perform preprocessing to use the aligned RGB image, the aligned depth image, and the aligned thermal image as input data for the deep learning model. For example, the preprocessing may include one or more processing steps that may be applied to data before the data can be used in a particular application, such as a deep learning model. The one or more operations may ensure that the data is in the correct format and / or quality for the intended analysis or use. For example, the preprocessing may include operations such as normalization, alignment, noise reduction, and other adjustments to make the RGB, depth, and thermal images suitable as input to the deep learning model.

[0071] For example, the artificial intelligence-based transparent obstacle detection apparatus 1000 may correct the data by normalization, shape conversion, or the like with respect to each image.

[0072] In step S500, the artificial intelligence transparent obstacle detection device 1000 may perform an estimation by the artificial intelligence model with respect to each pixel of the images. For example, after preprocessing, the artificial intelligence transparent obstacle detection device 1000 may detect the pixel area determined to be a transparent obstacle using the deep learning model based on the aligned RGB image and the aligned thermal image.

[0073] The deep learning model can receive the aligned RGB image and the aligned thermal image as input and, according to a model weight value, output the probability associated with each pixel region of the input images for estimating a transparent obstacle as a result array of the input image size. The deep learning model can perform image segmentation and can be deployed as a model pre-trained on transparent obstacle data.

[0074] For example, when the probability of estimating a transparent obstacle for each pixel is a predetermined value (e.g., 0.5) or more, the artificial intelligence-based transparent obstacle detection apparatus 1000 may determine the corresponding pixel as a transparent obstacle.

[0075] The artificial intelligence-based transparent obstacle detection device 1000 may determine that the transparent obstacle is present in the pixel region in which the pixels with the estimation probability of the predetermined value or more are located or present. For example, the artificial intelligence-based transparent obstacle detection device 1000 may determine that all pixels of the pixel region may include the transparent obstacle. This will be described in detail with reference to Fig. 3 described.

[0076] In step S600, the artificial intelligence-based transparent obstacle detection apparatus 1000 may determine whether the pixel area estimated as a transparent obstacle is detected.

[0077] The artificial intelligence-based transparent obstacle detection device 1000 can detect the transparent obstacle according to the presence of the pixel whose probability of estimating the transparent obstacle is equal to or higher than the predetermined value.

[0078] When the pixel area is estimated to be a transparent obstacle in step S710, the artificial intelligence-based transparent obstacle detection apparatus 1000 may estimate the actual depth of the transparent obstacle.

[0079] For example, when the pixel area determined to be a transparent obstacle is detected, the artificial intelligence-based transparent obstacle detection apparatus 1000 may estimate the depth of the transparent obstacle using the aligned depth image and the pixel area. This process will be described in detail with reference to Fig. 5 described.

[0080] If the pixel area is not judged to be a transparent obstacle in step S720, the artificial intelligence-based transparent obstacle detection apparatus 1000 may retrieve the corresponding pixel depth from the aligned depth image.

[0081] Through this operation, the artificial intelligence-based transparent obstacle detection apparatus 1000 can correct the incorrect depth value of the transparent obstacle area in the aligned depth image and perform collision detection more accurately.

[0082] In step S800, the artificial intelligence-based transparent obstacle detection apparatus 1000 may determine whether the depth of the pixel of the pixel region estimated as a transparent obstacle is within the preset collision range of the robot.

[0083] If, in step S910, the depth of the corresponding pixel is closer than the collision range (e.g., the collision range is 1 m), the artificial intelligence-based transparent obstacle detection device 1000 may send a robot stop signal to stop the robot. The artificial intelligence-based transparent obstacle detection device 1000 may display a warning on a motion monitoring screen.

[0084] Thereafter, the artificial intelligence-based transparent obstacle detection device 1000 may again search for a path to avoid the obstacle ahead in step S920.

[0085] If there is no obstacle ahead within the collision range in step S930, the artificial intelligence-based transparent obstacle detection device 1000 may send a signal to the robot to move normally on the path.

[0086] All steps from Fig. 2 can be executed in real time, and while the robot is moving, the artificial intelligence-based transparent obstacle detection device 1000 can be repeatedly operated.

[0087] Fig. Fig. 3 shows an example of a flowchart showing details of the image alignment step (step S300) of Fig. 2 according to an example.

[0088] In Fig. 3, the artificial intelligence-based transparent obstacle detection device 1000 may extract the RGB camera parameter (e.g., focal length, aperture, shutter speed, ISO sensitivity, white balance, and / or resolution, etc.) and the depth camera parameter (e.g., baseline distance, depth resolution, field of view, and / or minimum and maximum depth range, etc.) from the RGB depth camera 10 through camera calibration, and extract the thermal camera parameter (e.g., thermal sensitivity, resolution, field of view, and / or temperature range, etc.) from the thermal camera 20.

[0089] The artificial intelligence-based transparent obstacle detection device 1000 may obtain camera parameters between depth and RGB images and between RGB and thermal images before the robot moves.

[0090] The artificial intelligence-based transparent obstacle detection device 1000 can generate an aligned RGB image A11, an aligned depth image A12, and an aligned thermal image A21 using the extracted RGB camera parameter, the extracted depth camera parameter, and the extracted thermal camera parameter. For example, the parameters can include intrinsic parameters and extrinsic parameters.

[0091] For perfect alignment (e.g. of the RGB image, the depth image and / or the thermal image), information about the position and / or distance of the object may be required, and a captured depth image 12 can play this role.

[0092] The artificial intelligence-based transparent obstacle detection device 1000 may, in order to align the RGB image 11 with the thermal image 21, align the depth image 12 with the RGB image 11, and then align the aligned depth image A12 and the aligned RGB image A11 with the thermal image 21, thereby obtaining the aligned thermal image A21.

[0093] In step S310, the artificial intelligence-based transparent obstacle detection apparatus 1000 may first remove distortions from images including the RGB image 11, the depth image 12, and the thermal image 21 by intrinsic parameters of the RGB camera, intrinsic parameters of the depth camera, and intrinsic parameters of the thermal camera.

[0094] In step S320, the artificial intelligence-based transparent obstacle detection apparatus 1000 may calculate the depth of each pixel of the depth image 12 with an intrinsic parameter matrix of the depth camera and project it into a 3D space to align the depth image 12 with the RGB image 11.

[0095] In step S330, the artificial intelligence-based transparent obstacle detection device 1000 may re-project the point cloud projected into the 3D space into a 2D RGB plane by calculating with a camera extrinsic parameter matrix of depth and RGB to match the depth image 12 with the RGB coordinate.

[0096] Through the process, the artificial intelligence-based transparent obstacle detection apparatus 1000 can obtain the aligned depth image A12 on an RGB coordinate plane.

[0097] The artificial intelligence-based transparent obstacle detection device 1000 may perform similar operations to align the RGB image 11 with the coordinate plane of the thermal image 21.

[0098] In step S340, the artificial intelligence-based transparent obstacle detection apparatus 1000 may, for example, project pixels of the RGB image 11 corresponding to each pixel of the aligned depth image A12 into the 3D space.

[0099] The artificial intelligence-based transparent obstacle detection device 1000 can reproject the depth image A12 aligned with the RGB image 11 into 3D space using an intrinsic parameter matrix of the RGB camera. Each point of a projected point cloud can have a color value of the corresponding RGB image 11.

[0100] In step S350, the artificial intelligence-based transparent obstacle detection device 1000 may reproject pixels of the 3D space onto a 2D thermal (image) plane.

[0101] The artificial intelligence-based transparent obstacle detection device 1000 can back-project the point cloud of the 3D space onto the 2D thermal (image) plane using the extrinsic parameter of the RGB thermal imaging camera and obtain the aligned RGB image A11 and the aligned depth image A12 aligned with the thermal image 21. The thermal image 21 can be considered the aligned thermal image A21.

[0102] Fig. 4 shows an example of a flowchart showing details of the inference step of the deep learning model (step S500) of Fig. 2 according to an example.

[0103] In Fig. 4, the artificial intelligence-based transparent obstacle detection apparatus 1000 may extract a feature (e.g., regions, edges, textures, shapes, color patterns, and / or heat signatures, etc.) from both the aligned RGB image A11 and the aligned thermal image A21 through the deep learning model in steps S511 and S512.

[0104] For example, the artificial intelligence-based transparent obstacle detection apparatus 1000 may extract regions in which different images (e.g., different views of the extracted feature) may be formed as the pixel region of the transparent obstacle based on each extracted feature, and may detect the pixels of the extracted pixel region as the transparent obstacle.

[0105] The deep learning model for detecting the transparent obstacle can work based on the difference in optical properties between the RGB camera and the thermal imaging camera. For example, the difference in optical properties can relate to the different methods by which the RGB camera can detect visible light and color, while the thermal imaging camera can detect infrared radiation and temperature differences.

[0106] For example, the wavelength of a visible light beam used or received by the RGB camera may be approximately 0.4 µm to 0.7 µm, and the LWIR used or received by the thermal imaging camera may have a wavelength of 8 µm to 14 µm.

[0107] Transparent obstacles such as glass and plastic typically allow light with a wavelength of 4 µm or less to pass through. When glass is imaged with an RGB camera, light passes through, allowing objects behind it to be observed. These objects may not be visible to the thermal imaging camera.

[0108] The artificial intelligence-based transparent obstacle detection device 1000 can analyze each image by the deep learning model and distinguish or detect the difference in the images to estimate pixels in different areas as the transparent obstacle.

[0109] As an inference process in deep learning, the artificial intelligence-based transparent obstacle detection apparatus 1000 may receive the aligned RGB image A1 and the aligned thermal image A21 as inputs, and analyze and compare the features of each image to detect or estimate the transparent obstacle on a pixel-by-pixel basis.

[0110] For example, the artificial intelligence-based transparent obstacle detection device 1000 may extract the features related to the aligned RGB image A11 and the aligned thermal image A21 through respective networks in steps S511 and S512. During this process, the artificial intelligence-based transparent obstacle detection device 1000 may exchange features, calculate a difference for each feature, compare the two images, and thereby embed feature vectors containing information about the area where different images are formed in step S520. This step may be performed repeatedly depending on the network structure.

[0111] At step S530, the artificial intelligence-based transparent obstacle detection apparatus 1000 may use an embedding decoder network in which the features of the aligned RGB image A11 and the aligned thermal image A21 are mixed.

[0112] For example, by blending the detailed color and texture information from the aligned RGB image A11 with the temperature-based data from the aligned thermal image A21, the embedding decoder network can create a comprehensive feature map that improves the detection of transparent obstacles. This blending allows the system to detect transparent obstacles such as glass, which may be visible in the RGB image but do not (necessarily) have unique thermal signatures. The blended features can provide a more robust input to the deep learning model, enabling it to accurately identify and locate transparent obstacles under various environmental conditions.

[0113] At step S540, the artificial intelligence-based transparent obstacle detection apparatus 1000 performs the estimation of the class of the transparent obstacle on a pixel-by-pixel basis by applying a deep learning network.

[0114] Fig. Fig. 5 shows an example of a flowchart showing details of the step (step S710) of estimating the actual depth of the transparent obstacle from Fig. 2 according to an example. Fig. 5 shows an exemplary process of estimating the actual depth of the pixel area of ​​the transparent obstacle by the pixel-wise estimation result of the transparent obstacle.

[0115] In Fig. 5, the artificial intelligence-based transparent obstacle detection device 1000 may use the depth image aligned with the thermal image and the pixel-by-pixel transparent obstacle estimation result PC to estimate the depth of the pixel region of the transparent obstacle. The pixel-by-pixel transparent obstacle estimation result PC may include the pixel region where pixels estimated as the transparent obstacle are located.

[0116] If the transparent obstacle is not detected due to sensor limitations, the aligned depth image A12 may have an incorrect depth value in the transparent obstacle area (the pixel area where the transparent obstacle is located).

[0117] For example, the transparent obstacle including a glass window is typically shaped like a rectangular plane and may be fixed to the ground or adjacent to the ground. The artificial intelligence-based transparent obstacle detection device 1000 can estimate the actual depth of the transparent obstacle fixed to the ground.

[0118] In step S711, the artificial intelligence-based transparent obstacle detection apparatus 1000 may divide the pixel area obtained by the estimation result of the transparent obstacle into a plurality of groups (e.g., called “clustering”) by using a grouping algorithm (e.g., called K-means, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), hierarchical clustering, Mean Shift, Affinity Propagation, Gaussian Mixture Models (GMM), Spectral Clustering, Agglomerative Clustering, Ordering Points To Identify the Clustering Structure (OPTICS), Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH), or another suitable algorithm, etc.).

[0119] In step S712, the artificial intelligence-based transparent obstacle detection apparatus 1000 may detect an outer boundary line below the group area of ​​the plurality of groups by using a detection algorithm (e.g., referred to as Canny edge detector, Sobel operator, Hough transform, Laplacian of Gaussian (LoG), Fast Region-based Convolutional Neural Network (Fast R-CNN), Mask Region-based Convolutional Neural Network (Mask R-CNN), You Only Look Once (YOLO), or the Region-based Convolutional Neural Network (R-CNN) or another suitable algorithm, etc.).

[0120] If the outer boundary line is not detected in step S713, the artificial intelligence-based transparent obstacle detection device 1000 may stop the actual depth estimation of the transparent obstacle because there is a high probability that an obstacle closer than the glass exists (e.g., an obstacle other than glass is closer to the device 1000).

[0121] If the outer boundary line is not detected in step S714, the artificial intelligence-based transparent obstacle detection apparatus 1000 may search for pose / posture information (e.g., position and / or orientation information within the space) of each camera to correct the erroneously recorded depth of the pixel area of ​​the transparent obstacle.

[0122] The artificial intelligence-based transparent obstacle detection device 1000 may perform measurements using an inertial measurement unit (IMU) sensor (e.g., accelerometers, gyroscopes, magnetometers, etc.), or when the posture / attitude information of each camera is available and the camera is fixed, the angle of the actual transparent obstacle (e.g., glass window) may be obtained based on the posture / attitude information of the camera.

[0123] When the lower outer boundary line is detected, the artificial intelligence-based transparent obstacle detection apparatus 1000 may estimate the actual depth of the transparent obstacle based on camera space information including the location and direction (e.g., orientation) of the thermal imaging camera and a depth error value with respect to the transparent obstacle measured by the aligned depth image A12.

[0124] In step S715, the artificial intelligence transparent obstacle detection apparatus 1000 may obtain the pose / posture information in the plane of the transparent obstacle by calculating the norm vector (e.g., a normal vector) of the transparent obstacle using trigonometric calculations based on the camera space information and the false depth value measured by the depth camera, and calculate the correct depth value of the transparent obstacle area based on this.

[0125] The artificial intelligence-based transparent obstacle detection apparatus 1000 can approximate the actual depth value of the pixel area of ​​the transparent obstacle by using pixels adjacent to the pixel area of ​​the transparent obstacle even when the camera space information is not available.

[0126] For example, when the lower outer boundary line is detected, the artificial intelligence-based transparent obstacle detection apparatus 1000 may calculate a depth average with respect to the specific coordinate value of neighboring pixels below the lower outer boundary line.

[0127] The artificial intelligence-based transparent obstacle detection apparatus 1000 may apply the calculated depth average to the pixels having the same coordinate value as the specific coordinate value in the pixel area.

[0128] The artificial intelligence-based transparent obstacle detection device 1000 can estimate the depth (e.g., distance) of the ground surface as the actual depth (e.g., distance) of the transparent obstacle.

[0129] In step S716, the artificial intelligence-based transparent obstacle detection apparatus 1000 may calculate the average depth of pixels adjacent to the outer boundary line below the pixel area of ​​the transparent obstacle according to their y-coordinate values.

[0130] In step S717, the artificial intelligence-based transparent obstacle detection apparatus 1000 may apply the calculated average to the pixel area of ​​the transparent obstacle having the same y-coordinate value, and may replace the depth of the pixel area of ​​the transparent obstacle according to the y-coordinate values ​​with the average depth of the pixels on the ground surface where the transparent obstacle is set.

[0131] In step S718, the artificial intelligence-based transparent obstacle detection apparatus 1000 may estimate the depth of the pixel region of the transparent obstacle based on the calculation result.

[0132] The artificial intelligence-based transparent obstacle detection device 1000 can estimate that the pixel area of ​​the transparent obstacle has the same depth as the ground surface, and based on this, can control the robot to avoid the transparent obstacle fixed to the ground during navigation.

[0133] The artificial intelligence-based transparent obstacle detection device 1000 according to one example can effectively detect and avoid transparent obstacles such as glass, which may not be detected by depth cameras and LiDAR, when the robot moves in both indoor and outdoor environments. For example, the artificial intelligence-based transparent obstacle detection device 1000 can detect obstacles such as glass swing doors, revolving doors, automatic doors, glass walls, or the like.

[0134] The artificial intelligence-based transparent obstacle detection device 1000 can significantly reduce the risk of collisions in buildings with a lot of glass (e.g., glass fixtures), such as hotels.

[0135] The artificial intelligence-based transparent obstacle detection device 1000 can also be used for functions such as human detection, fire detection, or the like, using the same sensors (e.g., camera, blind spot monitoring sensor, lane departure warning sensor, parking sensor, light sensor, rain sensor, traction control sensor, anti-lock braking system sensor, tire pressure monitoring sensor, seat belt sensor, airbag sensor, fuel sensor, emission sensor, throttle position sensor, etc.).

[0136] The artificial intelligence-based transparent obstacle detection device 1000 can detect transparent objects as obstacles during the creation of environmental maps using a mobile robot, thus enabling the creation of maps that can more accurately determine whether they are suitable for movement on them.

[0137] The artificial intelligence-based transparent obstacle detection device 1000 can also be used for data collection with a mobile robot, and RGB, depth, and thermal images can be easily captured and / or acquired using the image processing system.

[0138] A device for detecting a transparent obstacle based on artificial intelligence may comprise an RGB depth camera configured to generate an RGB image and a depth image, a thermal imaging camera configured to generate a thermal image, and a control device connected to the synchronized RGB depth camera and the thermal imaging camera, wherein the control device is configured to align the RGB image and the depth image with respect to the thermal image to generate an aligned RGB image, an aligned depth image, and an aligned thermal image, detect a pixel area determined as a transparent obstacle based on the aligned RGB image and the aligned thermal image by using an artificial intelligence-based model, and estimate the depth of the transparent obstacle.by using the aligned depth image and the detected pixel area when detecting the pixel area determined to be a transparent obstacle.

[0139] The control device may be configured to compare the estimated depth of the transparent obstacle with a collision range predetermined in the robot currently moving to stop the robot when the depth is within the collision range, and to control the robot to move normally when the depth is outside the collision range.

[0140] The control device may be configured to extract an RGB camera parameter and a depth camera parameter from the RGB depth camera through camera calibration and extract the thermal camera parameter from the thermal camera and generate the aligned RGB image, the aligned depth image, and the aligned thermal image using the extracted RGB camera parameter, the extracted depth camera parameter, and the extracted thermal camera parameter.

[0141] The control device may be configured to primarily (e.g., first) align the depth image with respect to the RGB image to generate a primarily aligned depth image and a primarily aligned RGB image, and finally align the respective primarily aligned depth image and the primarily aligned RGB image with respect to the thermal image to generate the aligned RGB image, the aligned depth image, and the aligned thermal image.

[0142] The control device may be configured to extract a feature from each of the aligned RGB image and the aligned thermal image by a deep learning model, extract areas in which different images (e.g., views) are formed as a pixel area based on the extracted feature, and detect pixels of the extracted pixel area as a transparent obstacle.

[0143] The control device may be configured to divide the pixel area into a plurality of groups by using a grouping algorithm and to detect a lower outer boundary line near a ground surface from the plurality of groups.

[0144] If the lower outer boundary line is not detected, the control device may be configured to stop estimating the depth with respect to the transparent obstacle.

[0145] When the lower outer boundary line is detected, the control device may be configured to estimate the depth of the transparent obstacle based on camera space information including the location and / or direction (e.g., orientation) of the thermal imaging camera and / or a depth error value measured in the depth image.

[0146] When the lower outer boundary line is detected, the control device may be configured to calculate a depth average of pixels among adjacent pixels below the lower outer boundary line having the same specific coordinate value as the pixel range as a depth of the ground surface and estimate the calculated depth of the ground surface as the depth of the transparent obstacle.

[0147] If the pixel area determined as a transparent obstacle is not detected, the control device may be configured to query a depth with respect to pixels of pixel areas that cannot be determined as a transparent obstacle by the aligned depth image.

[0148] A method for detecting a transparent obstacle based on artificial intelligence may include: synchronizing an RGB depth camera and a thermal imaging camera, aligning an RGB image and a depth image generated by the RGB depth camera with respect to a thermal image generated by the thermal imaging camera to generate an aligned RGB image, an aligned depth image, and an aligned thermal image, preprocessing the aligned RGB image, the aligned depth image, and the aligned thermal image, detecting a pixel region determined to be a transparent obstacle using an artificial intelligence-based model based on the aligned RGB image and the aligned thermal image after the preprocessing, and estimating the depth of the transparent obstacle using the aligned depth image and the pixel region when the pixel region determined to be a transparent obstacle is detected or detected.is determined.

[0149] A method for detecting a transparent obstacle based on artificial intelligence may further comprise: comparing the estimated depth of the transparent obstacle with a collision range predetermined in or for the currently moving robot, controlling the robot to stop when the depth is within the collision range, and controlling the robot to move normally when the depth is outside the collision range.

[0150] Aligning the RGB image and the depth image with respect to the thermal image may further comprise: extracting an RGB camera parameter and a depth camera parameter from the RGB depth camera through camera calibration and extracting a thermal camera parameter from the thermal camera, and generating the aligned RGB image, the aligned depth image, and the aligned thermal image using the extracted RGB camera parameter, the extracted depth camera parameter, and the extracted thermal camera parameter.

[0151] Aligning the RGB image and the depth image with respect to the thermal image may further comprise: primarily aligning the depth image with respect to the RGB image to generate a primarily aligned depth image and a primarily aligned RGB image, and finally aligning the primarily aligned depth image and the primarily aligned RGB image with respect to the thermal image, respectively, to generate the aligned RGB image, the aligned depth image, and the aligned thermal image.

[0152] Detecting the pixel region determined to be a transparent obstacle may further include extracting a feature from each of the aligned RGB image and the aligned thermal image by a deep learning model, extracting regions in which different images are formed as a pixel region based on the extracted feature, and detecting pixels of the extracted pixel region as a transparent obstacle.

[0153] Estimating the depth of the transparent obstacle may further comprise dividing or grouping the pixel area into a plurality of groups using a grouping algorithm and detecting a lower outer boundary line near a ground surface from the plurality of groups.

[0154] Estimating the depth of the transparent obstacle may further comprise terminating the estimation of the depth with respect to the transparent obstacle when the lower outer boundary line is not detected.

[0155] Estimating the depth of the transparent obstacle may further comprise estimating the depth of the transparent obstacle based on camera spatial information including the position and direction (e.g., orientation) of the thermal imaging camera and a depth error value measured in the depth image when the lower outer boundary line is detected.

[0156] Estimating the depth of the transparent obstacle may further comprise: calculating a depth average of specific coordinate values ​​with respect to neighboring pixels below the lower outer boundary line when the lower outer boundary line is detected, applying the calculated depth average to pixels having the same coordinate value as the specific coordinate value in the pixel area, and estimating the depth of the ground surface as the actual depth of the transparent obstacle.

[0157] A method for detecting a transparent obstacle based on artificial intelligence may further comprise that, when the pixel region determined as a transparent obstacle is not detected, a depth with respect to pixels of pixel regions that cannot be determined as a transparent obstacle is retrieved through the aligned depth image.

[0158] An artificial intelligence-based transparent obstacle detection apparatus and method according to an example can effectively detect the transparent obstacles, such as glass, that may not be detected by the depth camera and / or the LIDAR while the robot is moving.

[0159] An artificial intelligence-based transparent obstacle detection apparatus and method according to an example may configure an image processing system using an RGB depth camera and a thermal imaging camera, detect a transparent obstacle through deep learning with input of information obtained by the image processing system, and enable a robot to avoid the detected transparent obstacle when applied to a mobile robot.

[0160] Fig. 6 shows an example of a drawing illustrating a computing device according to an example.

[0161] With reference to Fig. 6, a method and apparatus for detecting the transparent obstacle based on artificial intelligence according to an example may be implemented by using a computing device 900.

[0162] Computing device 900 may include at least a processor 910, a memory 930, a user interface input device 940, a user interface output device 950, and a storage device 960 communicating via a bus 920. For example, storage device 960 may include random access memory (RAM), an embedded multimedia card (eMMC), a data scratchpad RAM (DSPR), a data local storage unit (DLMU), a local storage unit (LMU), or a standard application memory (DAM), etc. Computing device 900 may also include a network interface 970 electrically connected to a network 90. ​​Network interface 970 may send or receive signals with other devices via network 90.

[0163] The processor 910 may be implemented in various ways, such as a microcontroller (MCU), application processor (AP), central processing unit (CPU), graphics processor (GPU), neural processing unit (NPU), and the like, and may be any type of semiconductor device capable of executing instructions stored in the memory 930 or the storage device 960. The processor 910 may be configured to perform the functions and methods described above with respect to Fig. 1 to Fig. 6 to be implemented.

[0164] Memory 930 and storage device 960 may include various types of volatile or non-volatile storage media. For example, memory may include read-only memory (ROM) 931 and random access memory (RAM) 932. In this example, memory 930 may be located inside or outside of processor 910, and memory 930 may be connected to processor 910 via various known means.

[0165] In some examples, at least some configurations or functions of an artificial intelligence-based transparent obstacle detection apparatus and method according to an example may be implemented as a program or software executable by the computing device 900, and the program or software may be stored in a computer-readable medium.

[0166] In some examples, at least some configurations or functions of an artificial intelligence-based transparent obstacle detection apparatus and method according to an example may be implemented using hardware or circuitry of the computing device 900 or may also be implemented as separate hardware or circuitry that may be electrically coupled to the computing device 900.

[0167] This disclosure has been described in the context of what are presently considered to be practical examples, but it is to be understood that the disclosure is not limited to the disclosed examples, but on the contrary is intended to cover various modifications and equivalent arrangements within the spirit and scope of the appended claims.

Claims

[1] A device (1000) for detecting a transparent obstacle, the device (1000) comprising: a red-green-blue depth camera, RGB depth camera (10) configured to generate at least one image associated with the transparent obstacle, the at least one image comprising red-green-blue (RGB) data and depth data, a thermal imaging camera (20) configured to generate a thermal image (21) associated with the transparent obstacle, and a control device (100) coupled to the RGB depth camera (10) and the thermal imaging camera (20), which are synchronized with each other, wherein the control device (100) is arranged to: align the at least one image and the thermal image (21), based on the aligned at least one image and the aligned thermal image (21), to detect a pixel area determined as the transparent obstacle by using an artificial intelligence model, and to estimate a depth of the transparent obstacle based on the detected pixel area and based on the depth data of the aligned at least one image. [2] Device (1000) according to claim 1, wherein the control device (100) is arranged to: to compare the estimated depth of the transparent obstacle with a collision zone of a robot, based on the estimated depth being within the collision range, to control the robot to prevent a collision of the robot with the transparent obstacle, and based on the estimated depth being outside the collision range, to control the robot to move normally. [3] Device (1000) according to claim 1 or 2, wherein the control device (100) is arranged to: extract an RGB camera parameter and a depth camera parameter from the RGB depth camera (10) by camera calibration, extract a thermal imaging camera parameter from the thermal imaging camera (20) and to generate the aligned at least one image and the aligned thermal image (21) using the extracted RGB camera parameter, the extracted depth camera parameter and the extracted thermal imaging camera parameter. [4] Device (1000) according to any one of the preceding claims, wherein the control device (100) is arranged to: align a depth image of the RGB depth camera (10) and an RGB image (11) of the RGB depth camera (10) to generate the at least one image, and align the at least one image and the thermal image (21) to produce an aligned RGB image (11), an aligned depth image (12) and the aligned thermal image (21). [5] Device (1000) according to any one of the preceding claims, wherein the control device (100) is arranged to: to extract a feature from each of an aligned RGB image (11) of the RGB depth camera (10) and an aligned thermal image (21) of the RGB depth camera (10) using a deep learning model, to extract areas in which different images are formed as pixel areas based on the extracted feature, and pixels of the pixel area as the transparent obstacle. [6] Device (1000) according to claim 5, wherein the control device (100) is arranged to: to divide the pixel area into a plurality of groups using a grouping algorithm and to detect a lower outer boundary line near a ground surface from the plurality of groups. [7] The device (1000) according to claim 6, wherein the control device (100) is configured to stop estimating the depth of the transparent obstacle based on the non-detection of the lower outer boundary line. [8] Device (1000) according to claim 6 or 7, wherein the control device (100) is arranged to estimate the depth of the transparent obstacle based on the detection of the lower outer boundary line, where the depth of the transparent obstacle is estimated based on: Camera space information including a location of the thermal imaging camera (20) and a direction of the thermal imaging camera (20), and a depth error value measured in a depth image (12) of the RGB depth camera (10). [9] Device (1000) according to any one of claims 6 to 8, wherein the control device (100) is arranged to: to determine, on the basis of the detected lower outer boundary line, a depth average of pixels with the same specific coordinate value as the pixel area under neighboring pixels below the lower outer boundary line as the depth of the ground surface and to estimate the determined depth of the ground surface as the depth of the transparent obstacle. [10] Device (1000) according to any one of the preceding claims, wherein the control device (100) is arranged to send a request for a depth with respect to pixels of pixel regions that were not determined as a transparent obstacle based on the non-detection of the pixel region by an aligned depth image (12) of the RGB depth camera (10). [11] A method for detecting a transparent obstacle, the method comprising: Synchronizing (S100) a red-green-blue depth camera, RGB depth camera (10), and a thermal imaging camera (20), Aligning (S300) a red-green-blue image, RGB image (11), and a depth image (12) generated by the RGB depth camera (10) with a thermal image (21) generated by the thermal camera (20) to generate an aligned RGB image (11), an aligned depth image (12), and an aligned thermal image (21), each of the aligned RGB image (11), the aligned depth image (12), and the aligned thermal image (21) being associated with the transparent obstacle, Detecting (S600) a pixel area determined to be a transparent obstacle based on the aligned RGB image (11) and the aligned thermal image (21) using an artificial intelligence model, and Estimating (S710) a depth of the transparent obstacle based on the detected pixel area and based on the aligned depth image (12). [12] The method of claim 11, further comprising: Comparing (S800) the estimated depth of the transparent obstacle with a collision area of ​​a robot, based on the estimated depth being within the collision range, controlling the robot to stop, and Based on the estimated depth being outside the collision range, control the robot to move normally. [13] The method according to claim 11 or 12, further comprising: Extracting an RGB camera parameter and a depth camera parameter from the RGB depth camera (10) by camera calibration, Extracting a thermal imaging camera parameter from the thermal imaging camera (20) and using the extracted RGB camera parameter, the extracted depth camera parameter and the extracted thermal camera parameter, generating the aligned RGB image (11), the aligned depth image (12) and the aligned thermal image (21). [14] A method according to any one of claims 11 to 13, wherein aligning the RGB image (11) and the depth image (12) with the thermal image (21) comprises: Align the depth image (12) with the RGB image (11) and Aligning the aligned depth image (12) and the aligned RGB image (11) with the thermal image (21) to generate the aligned RGB image (11), the aligned depth image (12) and the aligned thermal image (21). [15] A method according to any one of claims 11 to 14, wherein detecting the pixel region determined to be a transparent obstacle comprises: Extracting a feature from each of the aligned RGB image (11) and the aligned thermal image (21) via a deep learning model, Extracting regions in which different images are formed as a pixel region based on the extracted feature, and Detecting pixels of the pixel area as the transparent obstacle. [16] The method of claim 15, wherein estimating the depth of the transparent obstacle comprises: Dividing the pixel area into a plurality of groups using a grouping algorithm and Detecting a lower outer boundary line near a floor surface from the plurality of groups. [17] The method of claim 16, wherein estimating the depth of the transparent obstacle comprises: Stop estimating the depth of the transparent obstacle based on the lower outer boundary line not being detected. [18] A method according to claim 16 or 17, wherein estimating the depth of the transparent obstacle comprises: Estimating the depth of the transparent obstacle on the basis that the lower outer boundary line is detected, where the depth of the transparent obstacle is estimated based on Camera room information including a location of the thermal imaging camera (20) and a direction of the thermal imaging camera (20), and a depth error value measured in the depth image (12). [19] A method according to any one of claims 16 to 18, wherein estimating the depth of the transparent obstacle comprises: based on the detected lower outer boundary line, determining a depth average of pixels with respect to neighboring pixels below the lower outer boundary line, Apply the depth average to pixels that have the same coordinate value as a specific coordinate value in the pixel range, and Estimating a depth of the ground surface as the depth of the transparent obstacle. [20] A method according to any one of claims 11 to 19, further comprising: Based on the non-detection of the pixel area, sending a request for a depth with respect to pixels of pixel areas not determined as a transparent obstacle through the aligned depth image (12).

Images