Method and device for detecting transparent obstacle based on artificial intelligence

Abstract

The invention relates to a device and a method for detecting a transparent obstacle based on artificial intelligence. The apparatus may include an RGB-depth camera configured to generate an RGB image and a depth image; a thermal imaging camera configured to generate a thermal image; and a controller connected to the RGB-depth camera and the thermal imaging camera synchronized with each other, where the controller may be configured to: align the RGB image and the depth image relative to the thermal image to generate an aligned RGB image, an aligned depth image, and an aligned thermal image; detecting a pixel region determined as a transparent obstacle by using an artificial intelligence model based on the aligned RGB image and the aligned thermal image; and when a pixel region determined as a transparent obstacle is detected, estimating the depth of the transparent obstacle by using the aligned depth image and the detected pixel region.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Korean Patent Application No. 10-2024-0038552, filed on Mar. 20, 2024, which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure relates to an apparatus and method for detecting transparent obstacles based on artificial intelligence. More specifically, the present disclosure relates to an apparatus and method for detecting transparent obstacles based on artificial intelligence, which can detect transparent obstacles such as glass doors when a robot is driving. Background Art

[0004] Mobile robots can move along a designated path or navigate autonomously within a given environment. They are used in a variety of industries and service sectors and can take various forms, such as vehicles, drones, and intelligent robots. The core goal of mobile robots can be to detect and identify their environment and safely reach their destination.

[0005] Depth and lidar sensors, traditional technologies used in robot navigation and obstacle detection, cannot identify transparent objects such as glass as obstacles. If a robot cannot detect transparent objects during navigation, there is a risk of collision. Unlike visible and near-infrared light, long-wave infrared (LWIR) light from thermal imaging cameras cannot pass through glass. Summary of the Invention

[0006] The present disclosure attempts to provide an apparatus and method for detecting transparent obstacles based on artificial intelligence. The apparatus and method can configure a visual system using an RGB-depth camera and a thermal imaging camera, detect transparent obstacles through deep learning using information obtained by the visual system as input, and if applied to a mobile robot, can enable the robot to bypass the detected transparent obstacles.

[0007] An apparatus for detecting a transparent obstacle may include: a red, green, blue, and depth (RGB-depth) camera configured to generate at least one image associated with a transparent obstacle, wherein the at least one image includes red, green, and blue (RGB) data and depth data; a thermal imaging camera configured to generate a thermal image associated with the transparent obstacle; and a controller coupled to the RGB-depth camera and the thermal imaging camera that are synchronized with each other, wherein the controller is configured to: align the at least one image with the thermal image; detect, based on the aligned at least one image and the aligned thermal image, a pixel area determined to be a transparent obstacle by utilizing an artificial intelligence model; and estimate the depth of the transparent obstacle based on the detected pixel area and based on the depth data of the aligned at least one image.

[0008] The controller can be configured to: compare the estimated depth of the transparent obstacle with the collision range of the robot; based on the estimated depth being within the collision range, control the robot to prevent the robot from colliding with the transparent obstacle; and based on the estimated depth being outside the collision range, control the robot to drive normally.

[0009] The controller can be configured to: extract RGB camera parameters and depth camera parameters from the RGB-depth camera through camera calibration; extract thermal imaging camera parameters from the thermal imaging camera; and generate at least one aligned image and an aligned thermal image by utilizing the extracted RGB camera parameters, the extracted depth camera parameters, and the extracted thermal imaging camera parameters.

[0010] The controller may be configured to: align the depth image of the RGB-depth camera with the RGB image of the RGB-depth camera to generate at least one image; and align the at least one image with the thermal image to generate an aligned RGB image, an aligned depth image, and an aligned thermal image.

[0011] The controller can be configured to: extract features of each of the aligned RGB image of the RGB-depth camera and the aligned thermal image of the thermal imaging camera through a deep learning model; extract areas forming different images as pixel areas based on the extracted features; and detect pixels of the pixel areas as transparent obstacles.

[0012] The controller may be configured to: divide the pixel area into a plurality of groups by using a clustering algorithm; and detect a lower outer boundary line close to the bottom surface from the plurality of groups.

[0013] The controller may be configured to stop estimating the depth of the transparent obstacle based on failure to detect the lower outer boundary line.

[0014] The controller can be configured to estimate the depth of the transparent obstacle based on detecting the lower outer boundary line, wherein the depth of the transparent obstacle is estimated based on the following: camera space information including the position of the thermal imaging camera and the direction of the thermal imaging camera; and an erroneous depth value measured in the depth image of the RGB-depth camera.

[0015] The controller can be configured to: based on detecting the lower outer boundary line, determine the depth average of pixels among adjacent pixels below the lower outer boundary line that have the same specific coordinate value as the pixel area as the depth of the bottom surface; and estimate the determined depth of the bottom surface as the depth of the transparent obstacle.

[0016] The controller may be configured to query depths of pixels of the pixel region that is not determined to be a transparent obstacle through the aligned depth image of the RGB-depth camera based on the undetected pixel region.

[0017] A method for detecting a transparent obstacle may include: synchronizing a red, green, blue, and depth (RGB-depth) camera with a thermal imaging camera; aligning a red, green, and blue (RGB) image and a depth image generated by the RGB-depth camera with a thermal image generated by the thermal imaging camera to generate an aligned RGB image, an aligned depth image, and an aligned thermal image, wherein each of the aligned RGB image, the aligned depth image, and the aligned thermal image is associated with a transparent obstacle; detecting a pixel area determined to be a transparent obstacle by utilizing an artificial intelligence model based on the aligned RGB image and the aligned thermal image; and estimating the depth of the transparent obstacle based on the detected pixel area and based on the aligned depth image.

[0018] The method may further include: comparing the estimated depth of the transparent obstacle with a collision range of the robot; controlling the robot to stop based on the estimated depth being within the collision range; and controlling the robot to travel normally based on the estimated depth being outside the collision range.

[0019] The method may further include: extracting RGB camera parameters and depth camera parameters from the RGB-depth camera through camera calibration; extracting thermal imaging camera parameters from the thermal imaging camera; and generating an aligned RGB image, an aligned depth image, and an aligned thermal image by using the extracted RGB camera parameters, the extracted depth camera parameters, and the extracted thermal imaging camera parameters.

[0020] Aligning the RGB image and the depth image with the thermal image may include: aligning the depth image with the RGB image; and aligning the aligned depth image and the aligned RGB image with the thermal image to generate an aligned RGB image, an aligned depth image, and an aligned thermal image.

[0021] Detecting a pixel area determined to be a transparent obstacle may include: extracting features of each of the aligned RGB image and the aligned thermal image through a deep learning model; extracting an area forming a different image as a pixel area based on the extracted features; and detecting pixels of the pixel area as a transparent obstacle.

[0022] Estimating the depth of the transparent obstacle may include: dividing the pixel region into a plurality of groups by using a clustering algorithm; and detecting lower outer boundary lines close to the bottom surface from the plurality of groups.

[0023] Estimating the depth of the transparent obstacle may include stopping estimating the depth of the transparent obstacle based on failure to detect the lower outer boundary line.

[0024] Estimating the depth of the transparent obstacle may include: estimating the depth of the transparent obstacle based on detecting the lower outer boundary line, wherein the depth of the transparent obstacle is estimated based on the following: camera space information including the position of the thermal imaging camera and the direction of the thermal imaging camera; and an erroneous depth value measured in the depth image.

[0025] Estimating the depth of the transparent obstacle may include: determining a depth average of pixels of adjacent pixels below the lower outer boundary line based on detecting the lower outer boundary line; applying the depth average to pixels in the pixel area having the same coordinate value as the specific coordinate value; and estimating the depth of the bottom surface as the depth of the transparent obstacle.

[0026] The method may further include querying depths of pixels of the pixel region that is not determined to be a transparent obstacle through the aligned depth image based on the undetected pixel region. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An example of an apparatus for detecting a transparent obstacle based on artificial intelligence according to an example is shown.

[0028] Figures 2 to 5 An example of a method for detecting a transparent obstacle based on artificial intelligence according to an example is shown.

[0029] Figure 6 An example of a diagram illustrating a computing device is shown according to an example. DETAILED DESCRIPTION

[0030] Examples of the present disclosure will be described more fully below with reference to the accompanying drawings so that those skilled in the art can easily implement the examples. As will be appreciated by those skilled in the art, the described examples can be modified in various ways without departing from the spirit or scope of the present disclosure. To clarify the present disclosure, parts not related to the description will be omitted, and the same reference numerals will be used throughout the specification to refer to the same elements or equivalents.

[0031] Additionally or alternatively, unless explicitly described to the contrary, the word "include" and variations such as "comprises" or "comprising" will be understood to imply the inclusion of the elements stated but not the exclusion of any other elements. Commonly numbered terms such as first and second are used to describe various constituent elements, but these constituent elements are not limited by these terms. These terms are only used to distinguish one component from other components.

[0032] In addition, the terms "unit", "part" or "section", "device" and "module" in this specification refer to a unit that processes at least one function or operation, and the unit can be implemented by hardware, software or a combination of hardware and software.

[0033] Examples of the present disclosure will be described below with reference to the accompanying drawings.

[0034] Figure 1 An example of an apparatus for detecting a transparent obstacle based on artificial intelligence according to an example is shown.

[0035] For example, the device 1000 for detecting transparent obstacles based on artificial intelligence can be applied to mobile robots and / or any mobile devices (e.g., vehicles, drones, autonomous vehicles, etc.). For example, the mobile robot can detect transparent obstacles such as glass doors while driving using the device 1000 for detecting transparent obstacles based on artificial intelligence.

[0036] The device 1000 for detecting transparent obstacles based on artificial intelligence can use an RGB-depth camera and a thermal imaging camera simultaneously to detect transparent obstacles. The device 1000 for detecting transparent obstacles based on artificial intelligence can detect transparent glass within an image by utilizing the difference between an RGB image and a thermal image.

[0037] The device 1000 for detecting transparent obstacles based on artificial intelligence may include a vision system including an RGB-depth camera and a thermal imaging camera, and may detect transparent obstacles from an image using artificial intelligence. For example, the device 1000 for detecting transparent obstacles based on artificial intelligence may send (e.g., transmit) a stop or travel command to a robot based on the detected transparent obstacle.

[0038] Figure 1 An example of a configuration of an apparatus for detecting a transparent obstacle based on artificial intelligence is shown, the apparatus including a controller linked to a vision system.

[0039] Reference Figure 1 , an apparatus 1000 for detecting transparent obstacles based on artificial intelligence may include an RGB-depth (RGBD) camera 10 , a thermal imaging camera 20 , and a controller 100 .

[0040] The vision system may include an RGB-depth camera 10 and a thermal imaging camera 20 .

[0041] The RGB-depth camera 10 may generate an RGB image and / or a depth image by photographing.

[0042] The RGB image can include the color information of the object. The depth image can include the distance information from each pixel of the object to the camera. The depth image can be required for alignment between the RGB image and the thermal image, and can be used for depth estimation of transparent obstacles, for example, as auxiliary information.

[0043] The thermal imaging camera 20 can generate a thermal image, which can show the temperature distribution of an object.

[0044] The RGB-depth camera 10 and the thermal imaging camera 20 can be fixed to each other. The shooting directions of the RGB-depth camera 10 and the thermal imaging camera 20 can be parallel to each other. For example, the center of the lens of the thermal imaging camera 20 can be located approximately 25 mm above the center of the RGB-depth camera 10.

[0045] The RGB-depth camera 10 and the thermal imaging camera 20 may respectively send (eg, transmit) simultaneously captured image frames to the controller 100 connected to the RGB-depth camera 10 and the thermal imaging camera 20 .

[0046] The controller 100 may be connected to the RGB-depth camera 10 and the thermal imaging camera 20 which are synchronized with each other.

[0047] The controller 100 may be mounted on the mobile robot, and in order to drive the algorithm operation, the number of the controller 100 may be set to plural. For the operation of the present disclosure, at least one controller 100 may be required.

[0048] The controller 100 may convert the photographed frames to be used as input to the deep learning model, and may operate a transparent obstacle detection algorithm to estimate transparent obstacles by utilizing the deep learning model.

[0049] The controller 100 may align the RGB image and the depth image with respect to the thermal image, and may generate an aligned RGB image, an aligned depth image, and an aligned thermal image.

[0050] The controller 100 may detect a pixel area determined to be a transparent obstacle by utilizing an artificial intelligence model based on the aligned RGB images and / or the aligned thermal images.

[0051] If a pixel area determined to be a transparent obstacle is detected, the controller 100 may estimate the depth of the transparent obstacle by using the aligned depth image and / or the detected pixel area.

[0052] The controller 100 can improve the detection performance of transparent obstacles such as glass by effectively modeling the difference between RGB images and thermal images through a deep learning model.

[0053] The controller 100 may compare the estimated depth of the transparent obstacle with a predetermined collision range in which the robot may currently be traveling.

[0054] If the estimated depth is within the collision range, the controller 100 may stop the robot. If the estimated depth is outside the collision range, the controller 100 may control the robot to travel normally.

[0055] The controller 100 may extract RGB camera parameters and depth camera parameters from the RGB-depth camera through camera calibration, and may extract thermal imaging camera parameters from the thermal imaging camera.

[0056] The controller 100 may generate an aligned RGB image, an aligned depth image, and / or an aligned thermal image by using the extracted RGB camera parameters, depth camera parameters, and / or thermal imaging camera parameters.

[0057] The controller 100 may generate a preliminarily aligned depth image and a preliminarily aligned RGB image by preliminarily aligning the depth image with respect to the RGB image.

[0058] The controller 100 may generate an aligned RGB image, an aligned depth image, and / or an aligned thermal image by finally aligning the preliminary aligned depth image and the preliminary aligned RGB image with respect to the thermal image, respectively.

[0059] The controller 100 may extract features of each of the aligned RGB image and the aligned thermal image through a deep learning model.

[0060] The controller 100 may extract a region in the image that may form a different image as a pixel region based on each extracted feature, and may detect pixels of the extracted pixel region as a transparent obstacle.

[0061] The controller 100 may divide the pixel area into a plurality of groups by using a clustering algorithm, and may detect a lower outer boundary line close to the bottom surface from the plurality of groups.

[0062] If the lower outer boundary line is not detected, the controller 100 may stop estimating the depth of the transparent obstacle.

[0063] If the lower outer boundary line is detected, the controller 100 may estimate the actual depth of the transparent obstacle based on camera space information including the position and direction of the thermal imaging camera and an erroneous depth value measured in the depth image.

[0064] If the lower outer boundary line is detected, the controller 100 may calculate an average value of depths of pixels having the same specific coordinate value as the pixel region among adjacent pixels below the lower outer boundary line as the depth of the bottom surface.

[0065] The controller 100 may estimate the calculated depth of the bottom surface as the depth of the transparent obstacle.

[0066] If the pixel area determined to be the transparent obstacle is not detected, the controller 100 may inquire actual depths of pixels of the pixel area not determined to be the transparent obstacle through the aligned depth image.

[0067] Figures 2 to 5 An example of a method for detecting a transparent obstacle based on artificial intelligence according to an example is shown. Figures 2 to 5 The AI-based method for detecting transparent obstacles can be achieved by Figure 1 The device 1000 for detecting transparent obstacles based on artificial intelligence is executed.

[0068] Figure 2 An example of a method for detecting a transparent obstacle based on artificial intelligence according to an example is shown. Figure 2 An example of a scenario showing that the mobile robot of the present disclosure can detect and navigate around transparent obstacles (e.g., glass) while driving.

[0069] Glass may be a representative example of a transparent obstacle, and the transparent obstacle of the present disclosure is not particularly limited to glass, but may be a concept covering all transparent objects including glass.

[0070] In more detail, Figure 2 The following example process is shown: the device 1000 that can detect transparent obstacles based on artificial intelligence can shoot or capture image frames through the RGB-depth camera 10 and the thermal imaging camera 20, estimate the area where the transparent obstacles exist by utilizing the image frames as input of the deep learning model, and issue a command to the robot to bypass the transparent obstacle based on the estimation result.

[0071] exist Figure 2 In step S100 , the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may synchronize two cameras (eg, an RGB-depth camera 10 and a thermal imaging camera 20 ) and / or may cause the two cameras to shoot or capture frames simultaneously.

[0072] For example, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence can synchronize the RGB-depth camera 10 and the thermal imaging camera 20 by connecting cables to a synchronization board or connecting the cables to each other, or can synchronize the RGB-depth camera 10 and the thermal imaging camera 20 by measuring and adjusting the shooting time.

[0073] In step S300 , if the RGB-depth camera 10 and the thermal imaging camera 20 shoot or capture frames, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may match and align the RGB image, the depth image, and the thermal image to have the same screen coordinates.

[0074] The apparatus 1000 for detecting transparent obstacles based on artificial intelligence can align the RGB image and the depth image generated by the RGB-depth camera with respect to the thermal image generated by the thermal imaging camera, and can generate an aligned RGB image, an aligned depth image, and an aligned thermal image. The alignment process will refer to Figure 3 Provide a detailed description.

[0075] In step S400, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may perform preprocessing to use the aligned RGB images, aligned depth images, and aligned thermal images as input data for a deep learning model. For example, preprocessing may include one or more processing steps that may be applied to the data before it can be used in a specific application such as a deep learning model. The one or more processing steps may ensure that the data is in the correct format and / or of the correct quality for the intended analysis or use. For example, preprocessing may include operations such as normalization, alignment, noise reduction, and other adjustments to make the RGB images, depth images, and thermal images suitable as input for a deep learning model.

[0076] For example, the device 1000 for detecting transparent obstacles based on artificial intelligence can correct the data for each image through normalization, form conversion, etc.

[0077] In step S500, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may perform an estimation for each pixel of the image using an artificial intelligence model. For example, after pre-processing, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may detect pixel regions determined to be transparent obstacles based on the aligned RGB image and the aligned thermal image using a deep learning model.

[0078] The deep learning model can receive the aligned RGB image and the aligned thermal image as input and, based on the model weight values, can output the estimated probability of transparent obstacles associated with each pixel region of the input image as a result of the input image size arrangement. The deep learning model can perform image segmentation and can be provided as a model pre-trained on data on transparent obstacles.

[0079] For example, if the transparent obstacle estimation probability of each pixel is a predetermined value (eg, 0.5) or greater, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may determine the corresponding pixel as a transparent obstacle.

[0080] The device 1000 for detecting transparent obstacles based on artificial intelligence can determine that a transparent obstacle exists in a pixel area where pixels having an estimated probability of a predetermined value or greater are arranged. For example, the device 1000 for detecting transparent obstacles based on artificial intelligence can determine that all pixels in the pixel area may include a transparent obstacle. This will refer to Figure 4 Provide a detailed description.

[0081] In step S600 , the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may determine whether a pixel region estimated to be a transparent obstacle is detected.

[0082] The apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may determine a transparent obstacle based on whether there are pixels whose estimated probability of a transparent obstacle existing is a predetermined value or more.

[0083] In step S710 , if the pixel region is estimated to be a transparent obstacle, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may estimate the actual depth of the transparent obstacle.

[0084] For example, if a pixel area determined to be a transparent obstacle is detected, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence can estimate the depth of the transparent obstacle by using the aligned depth image and the pixel area. Figure 5 Provide a detailed description.

[0085] In step S720 , if the pixel region is not estimated as a transparent obstacle, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may query the depth of the corresponding pixel from the aligned depth image.

[0086] Through this process, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence can correct erroneous depth values ​​of the transparent obstacle region in the aligned depth images and perform collision detection more accurately.

[0087] In step S800 , the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may determine whether the depth of pixels of a pixel area estimated to be a transparent obstacle is within a preset collision range of the robot.

[0088] In step S910, if the depth of the corresponding pixel is closer than the collision range (for example, the collision range is 1 meter), the device 1000 for detecting transparent obstacles based on artificial intelligence can send a robot stop signal to stop the robot. The device 1000 for detecting transparent obstacles based on artificial intelligence can display a warning on the screen to monitor driving.

[0089] Subsequently, in step S920 , the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may re-search a path to bypass the obstacle ahead.

[0090] In step S930, if there is no obstacle ahead within the collision range, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may transmit a signal to the robot to drive normally along the path.

[0091] Figure 2 All steps can be executed in real time, and the device 1000 for detecting transparent obstacles based on artificial intelligence can repeat the operation while the robot is driving.

[0092] Figure 3 Shown according to the example Figure 2 An example of a flowchart of details of the image alignment step (step S300 ).

[0093] exist Figure 3 In the apparatus 1000 for detecting transparent obstacles based on artificial intelligence, RGB camera parameters (e.g., focal length, aperture, shutter speed, ISO sensitivity, white balance and / or resolution, etc.) and depth camera parameters (e.g., baseline distance, depth resolution, field of view and / or minimum and maximum depth ranges, etc.) can be extracted from the RGB-depth camera 10 through camera calibration, and thermal imaging camera parameters (e.g., thermal sensitivity, resolution, field of view and / or temperature range, etc.) can be extracted from the thermal imaging camera 20.

[0094] The device 1000 for detecting transparent obstacles based on artificial intelligence can obtain camera parameters between a depth image and an RGB image and between an RGB image and a thermal image before the robot travels.

[0095] The apparatus 1000 for detecting transparent obstacles based on artificial intelligence can generate an aligned RGB image A11, an aligned depth image A12, and an aligned thermal image A21 by utilizing the extracted RGB camera parameters, the extracted depth camera parameters, and the extracted thermal imaging camera parameters. For example, the parameters may include intrinsic parameters and extrinsic parameters.

[0096] For perfect alignment, information about the position and / or distance of the subject may be needed, and the captured depth image 12 may play a role.

[0097] The device 1000 for detecting transparent obstacles based on artificial intelligence can align the depth image 12 with the RGB image 11 in order to align the RGB image 11 with the thermal image 21, and then align the aligned depth image A12 and the aligned RGB image A11 with the thermal image 21 to obtain an aligned thermal image A21.

[0098] In step S310, the device 1000 for detecting transparent obstacles based on artificial intelligence can first eliminate distortion from the images including the RGB image 11, the depth image 12 and the thermal image 21 through the intrinsic parameters of the RGB camera, the intrinsic parameters of the depth camera, and the intrinsic parameters of the thermal imaging camera.

[0099] In step S320 , in order to align the depth image 12 with the RGB image 11 , the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may calculate the depth of each pixel of the depth image 12 using the intrinsic parameter matrix of the depth camera, and project the depth of each pixel of the depth image 12 into a 3D space.

[0100] In step S330 , in order to match the depth image 12 with the RGB coordinates, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may reproject the point cloud projected into the 3D space into the 2D RGB plane by calculating the extrinsic parameter matrices of the depth and RGB cameras.

[0101] Through this process, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence can obtain an aligned depth image A12 on the RGB coordinate plane.

[0102] The apparatus 1000 for detecting transparent obstacles based on artificial intelligence may perform a similar process to match the coordinate planes of the RGB image 11 and the thermal image 21 .

[0103] In step S340 , for example, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may project pixels of the RGB image 11 corresponding to each pixel of the aligned depth image A12 into a 3D space.

[0104] The apparatus 1000 for detecting transparent obstacles based on artificial intelligence can project the aligned depth image A12 aligned with the RGB image 11 back into the 3D space by utilizing the intrinsic parameter matrix of the RGB camera. Each point of the projected point cloud can have a color value of the corresponding RGB image 11.

[0105] In step S350 , the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may reproject the pixels of the 3D space onto a 2D thermal plane.

[0106] The apparatus 1000 for detecting transparent obstacles based on artificial intelligence can reproject the point cloud of the 3D space onto the 2D thermal plane by utilizing the extrinsic parameters of the RGB camera, and can obtain an aligned RGB image A11 and an aligned depth image A12 aligned with the thermal image 21. The thermal image 21 can be considered as an aligned thermal image A21.

[0107] Figure 4 Shown according to the example Figure 2 An example of a flowchart of details of the deep learning model inference step (step S500).

[0108] exist Figure 4 In steps S511 and S512, the device 1000 for detecting transparent obstacles based on artificial intelligence can extract features (e.g., area, edge, texture, shape, color pattern and / or thermal features, etc.) of each of the aligned RGB image A11 and the aligned thermal image A21 through a deep learning model.

[0109] For example, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may extract an area that may form a different image as a pixel area of ​​a transparent obstacle based on each extracted feature, and may detect pixels of the extracted pixel area as a transparent obstacle.

[0110] Deep learning models for detecting transparent obstacles can operate based on the differences in optical properties between RGB cameras and thermal cameras. For example, the optical property differences can refer to the different ways in which RGB cameras can capture visible light and color, and thermal cameras can capture infrared radiation and temperature differences.

[0111] For example, the wavelength of visible light used by an RGB camera may be approximately 0.4 μm to 0.7 μm, while the wavelength of LWIR used by a thermal imaging camera may be 8 μm to 14 μm.

[0112] Transparent barriers such as glass and plastic typically allow light with a wavelength of 4 μm or less to pass through. If the glass is imaged by an RGB camera, the light passes through, allowing objects behind the glass to be observed. These objects may be invisible to the thermal imaging camera.

[0113] The apparatus 1000 for detecting transparent obstacles based on artificial intelligence can analyze each image through a deep learning model, and can distinguish differences in the images to estimate pixels in different areas as transparent obstacles.

[0114] As a deep learning inference process, the device 1000 for detecting transparent obstacles based on artificial intelligence can receive the aligned RGB image A11 and the aligned thermal image A21 as input, and analyze and compare the features of each image, thereby detecting or estimating transparent obstacles pixel by pixel.

[0115] For example, in steps S511 and S512, the AI-based transparent obstacle detection apparatus 1000 can extract features from the aligned RGB image A11 and the aligned thermal image A21 through their respective networks. During this process, in step S520, the AI-based transparent obstacle detection apparatus 1000 can exchange features, calculate the difference in each feature, and compare the two images, thereby embedding a feature vector containing information about the regions forming the different images. This step can be repeated depending on the network structure.

[0116] In step S530 , the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may use an embedded decoder network that mixes features of the aligned RGB image A11 and the aligned thermal image A21 .

[0117] For example, by blending the detailed color and texture information from aligned RGB image A11 with the temperature-based data from aligned thermal image A21, the embedding decoder network can create a comprehensive feature map that enhances transparent obstacle detection. This blending enables the system to detect transparent obstacles, such as glass, that may be visible in the RGB image but lack a distinct thermal signature. The blended features provide more robust input to the deep learning model, enabling it to accurately identify and localize transparent obstacles under various environmental conditions.

[0118] In step S540 , the apparatus 1000 for detecting transparent obstacles based on artificial intelligence performs transparent obstacle category estimation pixel by pixel by applying a deep learning network.

[0119] Figure 5 Shown according to the example Figure 2 An example of a flowchart of details of the step of estimating the actual depth of the transparent obstacle (step S710). Figure 5 An example process of estimating the actual depth of a pixel area of ​​a transparent obstacle using pixel-by-pixel transparent obstacle estimation results is shown.

[0120] exist Figure 5 In the apparatus 1000 for detecting transparent obstacles based on artificial intelligence, the aligned depth image aligned with the thermal image and the pixel-by-pixel transparent obstacle estimation result PC may be used to estimate the depth of a pixel region of a transparent obstacle. The pixel-by-pixel transparent obstacle estimation result PC may include a pixel region where pixels estimated to be transparent obstacles are arranged.

[0121] Since the transparent obstacle is not detected due to sensor limitations, the aligned depth image A12 may have erroneous depth values ​​in the transparent obstacle area (the pixel area where the transparent obstacle exists).

[0122] For example, transparent obstacles including glass windows are mostly in the form of rectangular planes and may be fixed to or near the bottom surface. The apparatus 1000 for detecting transparent obstacles based on artificial intelligence can estimate the actual depth of the transparent obstacles fixed to the bottom surface.

[0123] In step S711, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may divide the pixel areas obtained through the transparent obstacle estimation result into a plurality of groups by utilizing a clustering algorithm (e.g., K-means, density-based spatial clustering with noise applications (DBSCAN), hierarchical clustering, mean shift, affinity propagation, Gaussian mixture model (GMM), spectral clustering, agglomerative clustering, ordered point identification cluster structure (OPTICS), balanced iterative reduction and clustering using hierarchies (BIRCH: Balanced Iterative Reducing and Clustering using Hierarchies), or any other suitable algorithm, etc.).

[0124] In step S712, the device 1000 for detecting transparent obstacles based on artificial intelligence can detect the outer boundary lines below the group areas of multiple groups by utilizing a detection algorithm (for example, a Canny edge detector, a Sobel operator, a Hough transform, a Laplacian of Gaussian operator (LoG), a region-based fast convolutional neural network (Fast R-CNN), a region-based Mask convolutional neural network (Mask R-CNN), YOLO (You Only Look Once) or a region-based convolutional neural network (R-CNN) and any other suitable algorithm, etc.).

[0125] In step S713 , if the outer boundary line is not detected, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may stop actual depth estimation of the transparent obstacle because the possibility that an obstacle closer than the glass exists may be high.

[0126] In step S714 , if the outer boundary line is detected, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may search for posture information (eg, position information in space) of each camera to correctly correct the depth of the pixel area of ​​the mistakenly photographed transparent obstacle.

[0127] The device 1000 for detecting transparent obstacles based on artificial intelligence can perform measurements by utilizing an inertial measurement unit (IMU) sensor (e.g., an accelerometer, a gyroscope, a magnetometer, etc.), or if posture information of each camera exists and the camera is fixed, the angle of the actual transparent obstacle (e.g., a glass window) can be obtained based on the posture information of the camera.

[0128] If the lower outer boundary line is detected, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence can estimate the actual depth of the transparent obstacle based on camera space information including the position and direction of the thermal imaging camera and the erroneous depth value of the transparent obstacle measured by the aligned depth image A12.

[0129] In step S715, the device 1000 for detecting transparent obstacles based on artificial intelligence can obtain posture information in the transparent obstacle plane by calculating the normal vector (normvector) of the transparent obstacle using triangulation calculation based on camera space information and the erroneous depth value measured by the depth camera, and based on this, the correct depth value of the transparent obstacle area can be calculated.

[0130] Even if camera space information is unavailable, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence can approximate the actual depth value of the pixel region of the transparent obstacle by utilizing pixels adjacent to the pixel region of the transparent obstacle.

[0131] For example, if a lower outer boundary line is detected, the apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may calculate a depth average of specific coordinate values ​​of adjacent pixels below the lower outer boundary line.

[0132] The apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may apply the calculated depth average value to pixels having the same coordinate value as a specific coordinate value in the pixel area.

[0133] The apparatus 1000 for detecting a transparent obstacle based on artificial intelligence may estimate the depth of the bottom surface as the actual depth of the transparent obstacle.

[0134] In step S716, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may calculate the average depth of pixels adjacent to the outer boundary line below the transparent obstacle pixel area according to the y-coordinate values ​​of the pixels adjacent to the outer boundary line below the transparent obstacle pixel area.

[0135] In step S717, the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may propagate the calculated average value to the transparent obstacle pixel area having the same y-coordinate value, and replace the depth of the transparent obstacle pixel area according to the y-coordinate value with the average depth of the pixels on the bottom surface to which the transparent obstacle is fixed.

[0136] In step S718 , the apparatus 1000 for detecting transparent obstacles based on artificial intelligence may estimate the depth of the transparent obstacle pixel region based on the calculation result.

[0137] The apparatus 1000 for detecting transparent obstacles based on artificial intelligence can estimate that the pixel area of ​​the transparent obstacle has the same depth as the bottom surface, and based on this, control the robot to bypass the transparent obstacle fixed to the bottom surface during navigation.

[0138] Regardless of whether the robot is traveling in an indoor or outdoor environment, the device 1000 for detecting transparent obstacles based on artificial intelligence can effectively detect and navigate around transparent obstacles such as glass that cannot be detected by depth cameras and lidar. For example, the device 1000 for detecting transparent obstacles based on artificial intelligence can detect obstacles such as glass swing doors, revolving doors, automatic doors, and glass walls.

[0139] The device 1000 for detecting transparent obstacles based on artificial intelligence can greatly reduce the risk of collision in buildings with a lot of glass, such as hotels.

[0140] The device 1000 for detecting transparent obstacles based on artificial intelligence can also be applied to functions such as personnel detection and fire detection that use the same sensors (for example, cameras, blind spot monitoring sensors, lane departure warning sensors, parking sensors, light sensors, rain sensors, traction control sensors, anti-lock braking system sensors, tire pressure monitoring sensors, seat belt sensors, airbag sensors, fuel sensors, emission sensors, throttle position sensors, etc.).

[0141] The device 1000 for detecting transparent obstacles based on artificial intelligence can detect transparent objects as obstacles when generating an environment map using a mobile robot, thereby generating a map that can more accurately determine whether it is drivable.

[0142] The device 1000 for detecting transparent obstacles based on artificial intelligence can also be used for data collection using a mobile robot, and through a vision system, RGB, depth, and thermal images can be easily captured and / or collected.

[0143] An apparatus for detecting transparent obstacles based on artificial intelligence may include: an RGB-depth camera configured to generate an RGB image and a depth image; a thermal imaging camera configured to generate a thermal image; and a controller connected to the RGB-depth camera and the thermal imaging camera that are synchronized with each other, wherein the controller may be configured to: align the RGB image and the depth image relative to the thermal image to generate an aligned RGB image, an aligned depth image, and an aligned thermal image; based on the aligned RGB image and the aligned thermal image, detect a pixel area determined to be a transparent obstacle by utilizing an artificial intelligence model; and if a pixel area determined to be a transparent obstacle is detected, estimate the depth of the transparent obstacle by utilizing the aligned depth image and the detected pixel area.

[0144] The controller may be configured to: compare the estimated depth of the transparent obstacle with a predetermined collision range in the currently traveling robot; control the robot to stop if the depth is within the collision range; and control the robot to travel normally if the depth is outside the collision range.

[0145] The controller can be configured to: extract RGB camera parameters and depth camera parameters from the RGB-depth camera, and extract thermal imaging camera parameters from the thermal imaging camera through camera calibration; and generate an aligned RGB image, an aligned depth image, and an aligned thermal image by utilizing the extracted RGB camera parameters, the extracted depth camera parameters, and the extracted thermal imaging camera parameters.

[0146] The controller can be configured to: preliminarily align the depth image relative to the RGB image to generate a preliminarily aligned depth image and a preliminarily aligned RGB image; and finally align the preliminarily aligned depth image and the preliminarily aligned RGB image relative to the thermal image to generate an aligned RGB image, an aligned depth image, and an aligned thermal image, respectively.

[0147] The controller can be configured to: extract features of each of the aligned RGB image and the aligned thermal image through a deep learning model; extract areas forming different images as pixel areas based on the extracted features; and detect pixels of the extracted pixel areas as transparent obstacles.

[0148] The controller may be configured to: divide the pixel area into a plurality of groups by using a clustering algorithm; and detect a lower outer boundary line close to the bottom surface from the plurality of groups.

[0149] The controller may be configured to stop estimating the depth of the transparent obstacle if the lower outer boundary line is not detected.

[0150] The controller may be configured to estimate the depth of the transparent obstacle based on camera space information including a position and / or orientation of the thermal imaging camera and / or an erroneous depth value measured in the depth image if the lower outer boundary line is detected.

[0151] The controller can be configured to: if a lower outer boundary line is detected, calculate the depth average of pixels having the same specific coordinate value as the pixel area among adjacent pixels below the lower outer boundary line as the depth of the bottom surface; and estimate the calculated depth of the bottom surface as the depth of the transparent obstacle.

[0152] The controller may be configured to, if the pixel region determined to be the transparent obstacle is not detected, query the depths of pixels of the pixel region that cannot be determined to be the transparent obstacle through the aligned depth image.

[0153] A method for detecting transparent obstacles based on artificial intelligence may include: synchronizing an RGB-depth camera with a thermal imaging camera; aligning an RGB image and a depth image generated by the RGB-depth camera relative to a thermal image generated by the thermal imaging camera to generate an aligned RGB image, an aligned depth image, and an aligned thermal image; preprocessing the aligned RGB image, the aligned depth image, and the aligned thermal image; detecting a pixel area determined to be a transparent obstacle by utilizing an artificial intelligence model based on the aligned RGB image and the aligned thermal image after preprocessing; and if a pixel area determined to be a transparent obstacle is detected, estimating the depth of the transparent obstacle by utilizing the aligned depth image and the pixel area.

[0154] The method for detecting transparent obstacles based on artificial intelligence may further include: comparing the estimated depth of the transparent obstacle with a predetermined collision range in the currently traveling robot; if the depth is within the collision range, controlling the robot to stop; and if the depth is outside the collision range, controlling the robot to travel normally.

[0155] Aligning the RGB image and the depth image relative to the thermal image may further include: extracting RGB camera parameters and depth camera parameters from the RGB-depth camera and extracting thermal imaging camera parameters from the thermal imaging camera through camera calibration; and generating an aligned RGB image, an aligned depth image, and an aligned thermal image by utilizing the extracted RGB camera parameters, the extracted depth camera parameters, and the extracted thermal imaging camera parameters.

[0156] Aligning the RGB image and the depth image relative to the thermal image may further include: preliminarily aligning the depth image relative to the RGB image to generate a preliminarily aligned depth image and a preliminarily aligned RGB image; and finally aligning the preliminarily aligned depth image and the preliminarily aligned RGB image relative to the thermal image, respectively, to generate an aligned RGB image, an aligned depth image, and an aligned thermal image.

[0157] Detecting a pixel area determined to be a transparent obstacle may further include: extracting features of each of the aligned RGB image and the aligned thermal image through a deep learning model; extracting an area forming a different image as a pixel area based on each extracted feature; and detecting pixels of the extracted pixel area as a transparent obstacle.

[0158] Estimating the depth of the transparent obstacle may further include: dividing the pixel region into a plurality of groups by using a clustering algorithm; and detecting lower outer boundary lines close to the bottom surface from the plurality of groups.

[0159] Estimating the depth of the transparent obstacle may further include stopping estimating the depth of the transparent obstacle if the lower outer boundary line is not detected.

[0160] Estimating the depth of the transparent obstacle may further include estimating the depth of the transparent obstacle based on camera space information including a position and a direction of the thermal imaging camera and an erroneous depth value measured in the depth image if the lower outer boundary line is detected.

[0161] Estimating the depth of the transparent obstacle may further include: if a lower outer boundary line is detected, calculating a depth average of specific coordinate values ​​of adjacent pixels below the lower outer boundary line; applying the calculated depth average to pixels in the pixel area having the same coordinate value as the specific coordinate value; and estimating the depth of the bottom surface as the actual depth of the transparent obstacle.

[0162] The method for detecting transparent obstacles based on artificial intelligence may further include: if a pixel area determined to be a transparent obstacle is not detected, querying the depths of pixels of the pixel area that cannot be determined to be a transparent obstacle through the aligned depth image.

[0163] The apparatus and method for detecting transparent obstacles based on artificial intelligence according to the example can effectively detect transparent obstacles such as glass that cannot be detected by depth cameras and / or lidars when the robot is driving.

[0164] According to the example, the apparatus and method for detecting transparent obstacles based on artificial intelligence can configure a visual system by utilizing an RGB-depth camera and a thermal imaging camera, detect transparent obstacles through deep learning with information obtained by the visual system as input, and if applied to a mobile robot, enable the robot to bypass the detected transparent obstacles.

[0165] Figure 6 An example of a diagram illustrating a computing device is shown according to an example.

[0166] Reference Figure 6 According to an example, the method and apparatus for detecting transparent obstacles based on artificial intelligence can be implemented by utilizing the computing device 900 .

[0167] The computing device 900 may include at least one of a processor 910, a memory 930, a user interface input device 940, a user interface output device 950, and a storage device 960 that communicate via a bus 920. For example, the storage device 960 may include a random access memory (RAM), an embedded multimedia card (eMMC), a data scratch pad RAM (DSPR), a data local memory unit (DLMU), a local memory unit (LMU), or a default application memory (DAM). The computing device 900 may also include a network interface 970 electrically connected to the network 90. ​​The network interface 970 may send or receive signals with other entities via the network 90.

[0168] The processor 910 may be implemented in various types, such as a microcontroller unit (MCU), an application processor (AP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), etc., and the processor 910 may be any type of semiconductor device capable of executing instructions stored in the memory 930 or the storage device 960. The processor 910 may be configured to implement the above-mentioned Figures 1 to 6 Describes the functions and methods.

[0169] The memory 930 and the storage device 960 may include various types of volatile or non-volatile storage media. For example, the memory may include a read-only memory (ROM) 931 and a random access memory (RAM) 932. In this example, the memory 930 may be located inside or outside the processor 910, and the memory 930 may be connected to the processor 910 through various known means.

[0170] In some examples, at least some configurations or functions of the apparatus and method for detecting transparent obstacles based on artificial intelligence according to the example can be implemented as a program or software executable by the computing device 900, and the program or software can be stored in a computer-readable medium.

[0171] In some examples, at least some configurations or functions of the apparatus and method for detecting transparent obstacles based on artificial intelligence according to the example can be implemented by utilizing the hardware or circuit of the computing device 900, or can also be implemented as separate hardware or circuit that can be electrically connected to the computing device 900.

[0172] While the disclosure has been described in connection with what are presently considered to be practical examples, it will be understood that the disclosure is not limited to the disclosed examples, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A device for detecting transparent obstacles, comprising: a red, green, and blue (RGB)-depth camera, generating at least one image associated with the transparent obstacle, the at least one image including red, green, and blue (RGB) data and depth data; a thermal imaging camera that generates a thermal image associated with the transparent obstacle; as well as a controller coupled to the RGB-depth camera and the thermal imaging camera in synchronization with each other, Wherein, the controller: aligning the at least one image with the thermal image; Based on the aligned at least one image and the aligned thermal image, detecting a pixel area determined to be the transparent obstacle by utilizing an artificial intelligence model; and The depth of the transparent obstacle is estimated based on the detected pixel areas and based on the depth data of the aligned at least one image.

2. The device according to claim 1, wherein The controller: comparing the estimated depth of the transparent obstacle with the collision range of the robot; controlling the robot to prevent a collision between the robot and the transparent obstacle based on the estimated depth within the collision range; and Based on the estimated depth being outside the collision range, the robot is controlled to travel normally.

3. The device according to claim 1, wherein The controller: Extracting RGB camera parameters and depth camera parameters from the RGB-depth camera through camera calibration; extracting thermal imaging camera parameters from the thermal imaging camera; and The aligned at least one image and the aligned thermal image are generated by utilizing the extracted RGB camera parameters, the extracted depth camera parameters, and the extracted thermal imaging camera parameters.

4. The device according to claim 1, wherein The controller: aligning the depth image of the RGB-depth camera with the RGB image of the RGB-depth camera to generate at least one image; and At least one image is aligned with the thermal image to generate an aligned RGB image, an aligned depth image, and the aligned thermal image.

5. The device according to claim 1, wherein The controller: extracting features of each of the aligned RGB image of the RGB-depth camera and the aligned thermal image of the thermal imaging camera through a deep learning model; Extracting a region forming a different image based on the extracted features as the pixel region; and Pixels in the pixel area are detected as the transparent obstacles.

6. The device according to claim 5, wherein The controller: dividing the pixel region into a plurality of groups by utilizing a clustering algorithm; and A lower outer boundary line close to the bottom surface is detected from the plurality of groups.

7. The device according to claim 6, wherein The controller: Based on the failure to detect the lower outer boundary line, estimating the depth of the transparent obstacle is stopped.

8. The device according to claim 6, wherein The controller: estimating the depth of the transparent obstacle based on detecting the lower outer boundary line, and The depth of the transparent obstacle is estimated based on: camera spatial information including a position of the thermal imaging camera and an orientation of the thermal imaging camera; as well as Incorrect depth values ​​measured in the depth image of the RGB-depth camera.

9. The device according to claim 6, wherein The controller: Based on the detection of the lower outer boundary line, determining an average depth value of pixels having the same specific coordinate value as the pixel area among adjacent pixels below the lower outer boundary line as the depth of the bottom surface; and The determined depth of the bottom surface is estimated as the depth of the transparent obstacle.

10. The device according to claim 1, wherein The controller: Based on the failure to detect the pixel region, depths of pixels in the pixel region that are not determined to be the transparent obstacle are queried using the aligned depth image of the RGB-depth camera.

11. A method for detecting a transparent obstacle, comprising: Synchronize the RGB-depth camera with the thermal imaging camera; aligning a red, green, and blue image (RGB image) and a depth image generated by the RGB-depth camera with a thermal image generated by the thermal imaging camera to generate an aligned RGB image, an aligned depth image, and an aligned thermal image, wherein each of the aligned RGB image, the aligned depth image, and the aligned thermal image is associated with the transparent obstacle; detecting a pixel area determined to be the transparent obstacle by utilizing an artificial intelligence model based on the aligned RGB image and the aligned thermal image; and The depth of the transparent obstacle is estimated based on the detected pixel areas and based on the aligned depth images.

12. The method according to claim 11, further comprising: comparing the estimated depth of the transparent obstacle with the collision range of the robot; controlling the robot to stop based on the estimated depth being within the collision range; as well as Based on the estimated depth being outside the collision range, the robot is controlled to travel normally.

13. The method according to claim 11, further comprising: Extracting RGB camera parameters and depth camera parameters from the RGB-depth camera through camera calibration; extracting thermal imaging camera parameters from the thermal imaging camera; as well as The aligned RGB image, the aligned depth image, and the aligned thermal image are generated by utilizing the extracted RGB camera parameters, the extracted depth camera parameters, and the extracted thermal imaging camera parameters.

14. The method according to claim 11, wherein Aligning the RGB and depth images with the thermal image involves: aligning the depth image with the RGB image; and The aligned depth image and the aligned RGB image are aligned with the thermal image to generate the aligned RGB image, the aligned depth image, and the aligned thermal image.

15. The method according to claim 11, wherein Detecting the pixel area determined to be the transparent obstacle includes: extracting features of each of the aligned RGB image and the aligned thermal image through a deep learning model; Extracting a region forming a different image as the pixel region based on the extracted features; and Pixels in the pixel area are detected as the transparent obstacles.

16. The method according to claim 15, wherein Estimating the depth of the transparent obstacle includes: dividing the pixel region into a plurality of groups by using a clustering algorithm; and A lower outer boundary line close to the bottom surface is detected from the plurality of groups.

17. The method according to claim 16, wherein Estimating the depth of the transparent obstacle includes: Based on the failure to detect the lower outer boundary line, estimating the depth of the transparent obstacle is stopped.

18. The method according to claim 16, wherein Estimating the depth of the transparent obstacle includes: estimating the depth of the transparent obstacle based on detecting the lower outer boundary line, and The depth of the transparent obstacle is estimated based on: Camera space information including a position of the thermal imaging camera and an orientation of the thermal imaging camera; and Erroneous depth values ​​measured in the depth image.

19. The method according to claim 16, wherein Estimating the depth of the transparent obstacle includes: determining, based on detecting the lower outer boundary line, an average depth value of pixels of adjacent pixels below the lower outer boundary line; applying the depth average value to pixels in the pixel region having the same coordinate value as a specific coordinate value; and The depth of the bottom surface is estimated as the depth of the transparent obstacle.

20. The method of claim 11, further comprising: Based on the fact that the pixel region is not detected, depths of pixels in the pixel region that are not determined to be the transparent obstacle are queried through the aligned depth image.

Images