Real-time total illuminance object detection method and robot capable of real-time total illuminance operation using the same

Abstract

Embodiments disclose a robot capable of real-time total illuminance operation, comprising a main body unit, a transport unit configured to move the main body unit, and a vision unit including an RGB camera configured to capture an image in a visible light band and a depth camera configured to capture an image in an infrared band and configured to generate an RGB image captured by the RGB camera and a depth image captured by the depth camera and transmit the RGB image and the depth image to the main body unit.

Description

REAL-TIME TOTAL ILLUMINANCE OBJECT DETECTION METHOD AND ROBOT CAPABLE OF REAL-TIME TOTAL ILLUMINANCE OPERATION USING THE SAME

[0001] One embodiment of the present invention relates to an object detection method and a robot using the same, and more specifically, to an object detection method that is less sensitive to a change in illuminance and is capable of real-time detection, and a robot using the same.

[0002] In large stores or factories, many types of goods should be moved all the time. Since it is inefficient for people to move objects without using tools or devices, it is common to borrow tools such as carts or the power of powered devices such as forklifts and pallet devices (hereinafter referred to as "transportation devices, etc.").

[0003] Since these conventional transportation devices, etc. require people to manipulate or drive the transportation devices, etc. to change locations, then get off and display objects on a stand or put the objects into a process, people spend a lot of time getting on and off the transportation devices, etc., and thus it is inefficient in environments in which many objects should be transported to many places, such as large stores.

[0004] Therefore, the need for logistics transport robots that track people is increasing, and elements that have been previously technically difficult to implement are gradually becoming possible thanks to recent developments in related technologies such as image recognition, light detection and ranging (LiDAR), Radar, and machine learning.

[0005] Recently, an example of such a human-tracking logistics transport robot has been disclosed, and FIG. 1 is a flowchart showing a method of allowing the conventional human-tracking logistics transport robot to track people. The conventional human-tracking logistics transport robot performs user tracking (S2) when succeeding (S1) in tracking the user in a captured image, performs user detection (S3) when failing, and moves while avoiding an obstacle when the obstacle is identified in a process of moving to a set target point according to the success or failure of the user tracking (S4).

[0006] However, the conventional human-tracking logistics transport robot may detect objects in an environment in which clear images may be detected, that is, in an environment with good illuminance, but as the illuminance is low, object detection becomes more difficult. In particular, due to the nature of the conventional object tracking method in which the object is momentarily lost when the illuminance is suddenly decreased and which performs current detection with predictions based on previous object detection, there is a problem in that object tracking is stopped.

[0007] To solve this problem, the resolution of the image may be increased or the number of frames per unit time may be increased, but in this case, there is a problem in that real-time object tracking becomes impossible due to an increase in computation as image data increases or high-performance and expensive calculation devices should be provided.

[0008] One embodiment of the present invention has been made in efforts to solve the problems and is directed to providing a robot capable of object detection in the total illuminance in real time and a method of detecting the same.

[0009] The objects of the embodiments are not limited to the above-described objects, and other objects that are not mentioned will be able to be clearly understood by those skilled in the art from the tracking description.

[0010] A robot capable of a real-time total illuminance operation using line clustering according to the present invention includes a main body unit, a transport unit configured to move the main body unit, and a vision unit including an RGB camera configured to capture an image in a visible light band and a depth camera configured to capture an image in an infrared band and configured to generate an RGB image captured by the RGB camera and a depth image captured by the depth camera, wherein the main body unit includes a motor configured to generate a motion necessary for moving the robot, a motor driving board configured to control the motor, a battery, a power supply board configured to control the battery, and a PC configured to receive vision data from the vision unit to set a route and control the motor driving board and the power supply board by generating an operation command, and the PC includes a YOLO module configured to determine a bounding box that is the smallest rectangle including an object in the RGB image, a sampling module configured to extract spatial data corresponding to some of pixels in the bounding box, a density-based spatial clustering of applications with noise (DBSCAN) module configured to group the sampled pieces of spatial data based on density, a first object median derivation module configured to derive a median of a cluster grouped by the DBSCAN module, a line initialization module configured to determine a first line on which the line clustering is performed of the depth image, a line clustering module configured to form a line cluster by grouping spatial data in the determined row of the depth image, a second object median derivation module configured to derive a median of the derived line cluster, and an object location determination module configure to determine a location of the object based on any one of the medians derived from the first object median derivation module and the second object median derivation module.

[0011] A real-time total illuminance object detection method using line clustering according to the present invention as the object detection method using an RGB image and a depth image includes a bounding box determining operation of determining a bounding box that is the smallest rectangle including an object in the RGB image, a sampling operation of extracting spatial data corresponding to some of pixels in the bounding box, an object clustering operation of grouping the sampled pieces of spatial data based on density, a first object median deriving operation of deriving a median of the grouped cluster, a line initializing operation of determining a first line on which line clustering is performed of the depth image, a line clustering operation of forming a line cluster by grouping spatial data in the determined row of the depth image, a second object median deriving operation of deriving a median of the derived line cluster, and an object location determining operation of determining the location of the object based on any one of the medians derived from the first object median derivation module and the second object median derivation module, wherein the bounding box determining operation, the sampling operation, the object clustering operation, and the first object median deriving operation are repeatedly performed by constituting a high-illuminance loop, and the line clustering operation and the second object median deriving operation are repeatedly performed by constituting a low-illuminance loop.

[0012] According to the robot capable of the real-time total illuminance operation using line clustering and a real-time total illuminance object detection method, the object detection of the robot is possible in the total illuminance including low illuminance.

[0013] In addition, the real-time object detection is possible by significantly reducing the amount of calculation in the low illuminance.

[0014] The effects of the present invention are not limited to the above-described effects, and other effects that are not described will be able to be clearly understood by those skilled in the art to which the present invention pertains from the tracking description.

[0015] FIG. 1 is a flowchart showing a method of allowing a conventional human-tracking logistics transport robot to track people.

[0016] FIG. 2 is a perspective view of a robot capable of real-time full illuminance operation using line clustering according to the present invention.

[0017] FIG. 3 is a configuration diagram of the robot capable of real-time total illuminance operation using line clustering according to the present invention.

[0018] FIG. 4 is a configuration diagram of a module assembly that detects and tracks people in a control device.

[0019] FIG. 5 is a flowchart of a detection method performed in the module assembly.

[0020] FIG. 6 is an example of line clustering.

[0021] FIG. 7 is an exemplary view showing the movement of center rows by band clustering.

[0022] Hereinafter, a robot capable of a real-time total illuminance operation using line clustering and a real-time total illuminance object detection method according to the present invention will be described in detail with reference to the accompanying drawings.

[0023] FIG. 2 is a perspective view of a robot capable of real-time full illuminance operation using line clustering according to the present invention, and FIG. 3 is a configuration diagram of the robot capable of real-time total illuminance operation using line clustering according to the present invention.

[0024] Referring to FIGS. 2 and 3, a robot capable of real-time total illuminance operation using line clustering according to the present invention includes a main body unit 100, a transport unit 200 for moving the main body unit 100, and a vision unit 300 provided in the main body unit 100.

[0025] When the vision unit 300 detects a set target, such as a carrier, in a three-dimensional space and generates vision data, the main body unit 100 may process the vision data and detect the target, and the transport unit 200 may move the main body unit 100 according to a control command.

[0026] The main body unit 100 may detect, in real time, a location and speed of a target (a plurality of targets may be present) that is present in a space (hereinafter referred to as "work space" and may be inside a warehouse, inside a factory, etc.) in which a robot capable of a real-time total illuminance operation using line clustering according to the present invention from the vision data collected from the vision unit 300 moves.

[0027] The transport unit 200 is a component that is composed of wheels, caterpillars, etc. to transport the main body unit 100 under the control of the main body unit 100. According to an embodiment, the transport unit 200 is not limited to wheels or caterpillars and may be applied without limitation as long as it is a component for moving the main body unit 100.

[0028] The vision unit 300 may include an RGB camera 310 for capturing images in a visible light band and a depth camera 320 for capturing images in an infrared band. The vision unit 300 may generate an RGB image captured by the RGB camera 310 and a depth image captured by the depth camera 320 (hereinafter referred to as "vision data") and transmit the images to the main body unit 100.

[0029] The vision unit 300 may detect reflection points of objects defined in a three-dimensional space. Even when visible light or infrared rays (hereinafter referred to as "light") are reflected from an object and a point or light captured by the vision unit 300 is reflected, a point reflecting light that is weak enough to be determined to be a zero intensity may not become a reflection point. However, considering that robots are rarely used in an environment without any light, a case where there is no reflection point does not need to be considered in the present invention.

[0030] The vision unit 300 may set a predetermined region in front of the robot as a region of interest (ROI) and collect the vision data only in the ROI.

[0031] The main body unit 100 may include a motor 110 for generating the motion necessary for robot movement such as wheels or caterpillars, a motor driving board 120 for controlling the motor 110, a battery 130, a power supply board 140 for controlling the battery, and a control device 150 for setting a route by receiving the vision data from the vision unit 300 and controlling the motor driving board 120 and the power supply board 140 by generating an operation command. The control device may include one or more processors / microprocessor(s) / computer coupled to a computer-readable recording medium for storing computer-readable code / algorithm / software.

[0032] According to hardware implementation, the embodiments described herein may be implemented by using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, microcomputers, personal computers (PCs), and other electrical units for performing functions. Such embodiments may be implemented in the control device.

[0033] The main body unit 100 may further include a sensor 160 such as an infrared sensor or an ultrasonic sensor, and an MCU board 170 for collecting data sensed by the sensor 160. Data and control commands may be transmitted and received between the control device 150, the motor driving board 120, and the MCU board 170 using a method such as RS232 or CAN communication, but the present embodiments are not necessarily limited thereto.

[0034] The control device 150 may detect a target by processing the sensed vision data, and such target detection may be performed by using firmware or software. In the present invention, the concept of a module (which may be implemented in hardware or software, but they are the same in terms of the technical spirit, and thus are not distinguished neither) may be introduced without distinguishing firmware and software.

[0035] FIG. 4 is a configuration diagram of a module assembly that detects and tracks people in a control device, and FIG. 5 is a flowchart of a detection and tracking method performed in the module assembly, which is a flowchart of a real-time total illuminance object detection method using line clustering according to the present invention.

[0036] In the present invention, the control device 150 may include a YOLO module 151, a sampling module 152, a density-based spatial clustering of applications with noise (DBSCAN) module 153, a first object median derivation module 154, a line initialization module 155, a line clustering module 156, a second object median derivation module 157, and an object location determination module 158.

[0037] The control device 150 may include a first derivation module 150A including the YOLO module 151, the sampling module 152, the DBSCAN module 153, the first object median derivation module 154, a second derivation module 150B including the line initialization module 155, the line clustering module 156, the second object median derivation module 157, and the object location determination module 158, but is not necessarily limited thereto.

[0038] The YOLO module 151 may perform a bounding box determining operation (S10) of determining a bounding box, which is the smallest rectangle including an object, by applying a YOLO algorithm, which is an object detection algorithm, to the RGB image captured by the RGB camera 310.

[0039] When the YOLO module 151 learned with a COCO dataset with 80 classes, which provides images and labels, receives RGB images and divides the RGB images into grids, three bounding boxes may be predicted for each grid.

[0040] For each box, information may be stored in a total of 85 channels including 4 channels for a location and size of the box, 1 channel for probability of object presence, and 80 channels for probability of being a specific class, and thus an output matrix of 255 (=3Х85) channels may be derived.

[0041] The output matrix may be derived as a 13Х13 matrix used for easily detecting large objects and a 26Х26 matrix used for detecting small objects, and each bounding box and object probability may be derived from the results of the output matrix.

[0042] Bounding boxes in which the probability that an object is present in each bounding box is smaller than or equal to a predetermined first reference value may be removed, and bounding boxes in which a value obtained by multiplying the probability that the object is present by the probability that it is a human class is smaller than or equal to a predetermined second reference value may also be removed.

[0043] Thereafter, in a case where there are two or more bounding boxes pointing the same object among the remaining bounding boxes, when an intersection of union (IOU) of any two bounding boxes defined by [Equation 1] below is calculated and the IOU is greater than or equal to a reference value, a bounding box may be finally specified by performing non-maximum suppression (NMS) work in a manner of removing boxes with a low probability of being people.

[0044] [Equation 1]

[0045] IOU=

[0046] The sampling module 152 is a component for performing a sampling operation (S20) of extracting spatial data corresponding to some of pixels in the bounding box, and the reason for sampling is that, when an object is detected in consideration of all pixels in the bounding box determined in the YOLO module 151, the amount of calculation is too high to make real-time detection difficult, and thus it is necessary to reduce the amount of data.

[0047] The spatial data extracted from the sampling module 152 is composed of (x, y) coordinates of RGB video images and a depth value d of a depth video image corresponding to the coordinates (x, y) of each RGB video image. Therefore, the form of sampled spatial data may be described as (x, y, d).

[0048] Sampling may be performed, for example, on a case where an x-coordinate value and a y-coordinate value are odd, even, or a multiple of 3.

[0049] Objects such as people should be detected from sampled spatial data, and since pieces of the spatial data sampled from the objects are close, it is necessary to distinguish the pieces of the spatial data adjacent to each other.

[0050] The DBSCAN module 153 may perform an object clustering operation (S30) of grouping the pieces of the sampled spatial data by applying a DBSCAN algorithm, which is a density-based clustering algorithm.

[0051] After undergoing the object clustering operation (S30), the pieces of the sampled spatial data are divided into one or two or more object clusters. In order for a robot to track an object, first, representative spatial data should be determined for each of the divided object clusters (question of which to track among the detected objects is left aside), and its criterion can be defined in various ways as a median or mean of all of the pieces of the spatial data constituting a cluster, a median or mean of pieces of spatial data excluding an outermost cluster to rule out extreme or erroneous data, etc.

[0052] In the present invention, the median of all of the pieces of the spatial data constituting the cluster is adopted as the representative spatial data of the object cluster, and the reason is that, when the average is adopted, the influence of extreme values or erroneous values increases, and the outside of the object may be derived as the representative spatial data.

[0053] The first object median derivation module 154 may perform a first object median deriving operation (S40) of deriving a median of clusters representing an object in the same manner as above.

[0054] In this operation, the median may be, for example, a median of a spatial concept defined as coordinates of the spatial data in a sorted central portion after sorting all of the pieces of the spatial data constituting the cluster in an ascending order based on a y-coordinate and sorting the same in a descending order based on an x-coordinate. Since the median of this spatial concept is adopted, the median of the cluster derived by the first object median derivation module 154 is always present in the object.

[0055] When the illuminance is sufficient, the object may be detected by undergoing the bounding box determining operation (S10), the sampling operation (S20), the object clustering operation (S30), and the first object median deriving operation (S40), and after the object is detected, a tracking algorithm such as Kalman filtering may be performed to predict and track a location of the object. That is, in the present invention, the bounding box determining operation (S10), the sampling operation (S20), the object clustering operation (S30), and the first object median deriving operation (S40) form one loop (hereinafter referred to as "high-illuminance loop") to detect an object in a high-illuminance environment.

[0056] The line initialization module 155 is a component for performing a line initializing operation (S50) of determining a first line on which line clustering is performed of a depth image. In the present invention, the line clustering refers to grouping performed on a specific line of a depth image, and a cluster may be identified in the line (hereinafter referred to as "line cluster").

[0057] Considering that spatial data caused by objects such as people have a less longitudinal change than a transverse change, it is preferable that the line is determined as a specific row of the depth image. Since the number of cases in which people move horizontally is much larger than the number of cases in which people move vertically, and vertical movement is due to actions such as squatting or running, and thus the limits of movement may be relatively clear.

[0058] The line initialization module 155 may determine that a specific line among the depth images of a person (robot operator) captured during an initial operation of a robot capable of a real-time total illuminance operation according to the present invention, for example, a first line is a row including a pixel at the exact center of the depth image, a row including pixels of a central portion of a region estimated to be the person's shoulder or waist, etc. The line set as initially given in this way may be adaptively changed to a line advantageous for object tracking by the line clustering module 156 and the second object median derivation module 157, which will be described below.

[0059] The line clustering module 156 may perform a line clustering operation (S60) of forming a line cluster by performing line clustering that groups spatial data of the determined row of the depth image.

[0060] The line clustering method is similar to the object clustering operation (S30) and has an advantage in that the amount of calculation is very small. The reason why the amount of calculation is significantly reduced is that the clustering in the object clustering operation (S30) should determine whether to include specific spatial data in the cluster by reflecting density of spatial data and requires distance calculation and comparison between three-dimensional spatial coordinates. However, in the case of the line clustering, since a distance between pixels of the depth image in one row is fixed to 1 pixel and only the depth value d is changed, only whether a difference in depth values of a specific pixel and a next pixel is greater than or equal to a threshold value while sequentially reading the depth value of the pixel from x=1 is determined (when the difference is greater than or equal to the threshold value, it is determined to be a different line cluster).

[0061] For example, when the line clustering is performed on 20 pixels horizontally as shown in FIG. 6, and when the threshold value is 3, three line clusters A, B, and C are generated (a number inside the pixel is a depth value).

[0062] Since the line clustering is performed on the entirety of the specific line (spatial data) of the depth image without sampling, there is an advantage in that the amount of calculation of the line clustering module 156 is small.

[0063] The second object median derivation module 157 may perform a second object median deriving operation (S70) of deriving a median of the derived line clusters. The calculation here is simpler. This is because a row of pixels constituting each cluster is determined by the line clustering (a y-coordinate of the pixel is determined), and only an x-coordinate of a pixel at the center of the row is derived.

[0064] In the case of FIG. 6, a median of the line cluster A is a pixel a with a depth value of 3 (when the number n of pixels constituting the line cluster is even, a (n / 2)thpixel is adopted as a median, and the same applies hereafter), a median of the line cluster B is a pixel b with a depth value of 14, and a median of the line cluster C is a pixel c with a depth value of 8.

[0065] The line clustering operation (S60) and the second object median deriving operation (S70) form one loop (hereinafter referred to as "low-illuminance loop") to enable object detection even when luminance is low.

[0066] As described above, the line for detecting the object may be adaptively determined, and this is possible by performing clustering (band clustering) on a predetermined band composed of a specific line and a predetermined number of rows under and above the specific line. When the clustering is performed on bands, it is more likely that the median of the cluster will be determined at a line with a larger number of pixels constituting the cluster among lines under or above the specific line, and this means that a row with a larger number of objects in the depth image becomes a center row of next band clustering, and thus it becomes more advantageous for object detection.

[0067] FIG. 7 is an exemplary view showing the movement of a center row by band clustering, in which shaded portions are pixels of an object in a depth image. For simplicity, it is assumed that the band is composed of a center row and one row under and above the center row (3 rows). As shown in FIG. 7A, when the initial line is determined to be a row Lkwith a low proportion of objects, band clustering is performed on lines Lk-1, Lk, and Lk+1, and its median is d (fourth pixel among 7 shaded pixels), and a center row of next band clustering becomes Lk+1. The next band clustering is performed on the lines Lk, Lk+1, and Lk-1as shown in FIG. 7B, and its median is d (fourth pixel among 7 shaded pixels), and a center row of next band clustering becomes Lk+1.

[0068] The center row is gradually moved to a row with a higher proportion of objects by the band clustering by the line clustering module 156 and the band cluster median derivation by the second object median derivation module 157, and as the low illuminance loop is repeated, there is a high possibility that the median of the band cluster representing the object is present in the moved-up center row, thereby improving the reliability of the object detection.

[0069] The object location determination module 158 may perform an object location determining operation (S80) of determining a location of an object based on one of the medians derived from the first object median derivation module 154 and the second object median derivation module 157.

[0070] Since the motivation for introducing line clustering in the present invention is to improve detection ability in low illuminance, the median derived from the first object median derivation module 154 or the median derived from the second object median derivation module 157 may be adopted based on illuminance. For example, in a case of a specific illuminance value or more, the median derived from the first object median derivation module 154 is adopted, and in a case of the specific illuminance value or less, the median derived from the second object median value derivation module 157 is adopted. For this configuration, a separate illuminance sensor may be provided. When the illuminance measured by the illuminance sensor is greater than or equal to a predetermined illuminance value, the object location determination module 158 may determine the location of the object by adopting the median derived from the first object median derivation module 154.

[0071] Alternatively, the location of the object may be determined through comparison with the results of tracking such as Kalman filtering performed after determining the location of the object. In this case, it is preferable to prioritize object location determination at the median derived by a high-illuminance loop that detection is easy due to sufficient illuminance. That is, when the median derived from the first object median derivation module 154 is reliable within a predetermined error range (predetermined) or less from a predicted value derived from a tracking module, the object location determination module 158 may determine the median derived from the first object median derivation module 154 as the location of the object.

[0072] In addition, when the median derived from the first object median value derivation module 154 exceeds the predetermined error range from the predicted value derived from the tracking module and the median derived from the second object median derivation module 157 is within the predetermined error range from the predicted value derived from the tracking module, the object location determination module 158 may determine the median derived from the second object median derivation module 157 as the location of the object.

[0073] In addition, when there is a difference between both the medians derived from the first object median derivation module 154 and the second object median derivation module 157 due to exceeding the predetermined error range from the predicted values derived from the tracking module, the medians derived from the object median derivation modules 154 and 157, which has been just before adopted by the object location determination module 158 that was used just before and thus has a relatively low possibility of erroneously detecting an object, may be determined to be the location of the object.

[0074] The above-described present invention may be implemented as computer-readable codes on a medium on which a program is recorded. The computer-readable recording medium includes any type of recording media in which data that may be read by a computer system are stored. Examples of the computer-readable media include a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a read only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and also include one implemented in the form of carrier waves (e.g., transmission via Internet).

[0075] Although the present invention has been described with reference to the embodiments shown in the drawings, these are merely illustrative, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical scope of the present invention should be determined by the technical spirit of the appended claims.

Claims

1.A robot capable of real-time total illuminance operation, comprising:a main body unit;a transport unit configured to move the main body unit; anda vision unit including an RGB camera configured to capture an image in a visible light band and a depth camera configured to capture an image in an infrared band and configured to generate an RGB image captured by the RGB camera and a depth image captured by the depth camera and transmit the RGB image and the depth image to the main body unit,wherein the main body unit includes a control device configured to receive vision data from the vision unit to set a route and generate an operation command, andthe control device is configured to:derive a first object median based on the RGB image and derive a second object median based on depth information; anddetermine a location of an object using each of the derived first object median and second object median.2.The robot of claim 1, wherein the control device determines that the first object median is the location of the object when a measured illuminance value is a predetermined value or more and determines that the second object median is the location of the object when the measured illuminance value is smaller than the predetermined value.3.The robot of claim 1, wherein the control device includes a tracking module configured to predict a moving location of the object, andthe control device determines that a value matching results of tracking predicted by the tracking module among the first object median and the second object median is the location of the object.4.The robot of claim 1, wherein the control device includes:a first derivation module configured to derive the first object median based on the RGB image; anda second derivation module configured to derive the second object median based on the depth information.5.The robot of claim 4, wherein the first derivation module includes:a YOLO module configured to determine a bounding box that is the smallest rectangle including an object in the RGB image;a sampling module configured to extract spatial data corresponding to some of pixels in the bounding box;a density-based spatial clustering of applications with noise (DBSCAN) module configured to group the sampled pieces of spatial data based on density; anda first object median derivation module configured to derive a median of a cluster grouped by the DBSCAN module.6.The robot of claim 5, wherein the second derivation module includes:a line initialization module configured to determine a first line on which line clustering is performed of the depth image;a line clustering module configured to form a line cluster by grouping spatial data in the determined row of the depth image; anda second object median derivation module configured to derive a median of the derived line cluster.7.The robot of claim 6, wherein the control device includes an object location determination module configured to determine the location of the object based on any one of the medians derived from the first object median derivation module and the second object median derivation module.8.The robot of claim 1, wherein the main body unit includes a motor configured to generate a motion necessary for moving the robot, a motor driving board configured to control the motor, a battery, and a power supply board configured to control the battery.9.The robot of claim 7, wherein the median value derived from the first object median derivation module is coordinates of spatial data at a sorted central portion after sorting all of pieces of the spatial data constituting the cluster according to x and y coordinate values.10.The robot of claim 7, wherein the line initialization module determines that a specific row of a depth image of a person captured upon an initial driving of the robot or a row including pixels of a central portion of a region estimated as the person's shoulder or waist is a first line.11.An object detection method using an RGB image and a depth image, comprising:a bounding box determining operation of determining a bounding box that is the smallest rectangle including an object in the RGB image;a sampling operation of extracting spatial data corresponding to some of pixels in the bounding box;an object clustering operation of grouping the sampled pieces of spatial data based on density;a first object median deriving operation of deriving a median of the grouped cluster;a line initializing operation of determining a first line on which line clustering is performed of the depth image;a line clustering operation of forming a line cluster by grouping spatial data in the determined row of the depth image;a second object median deriving operation of deriving a median of the derived line cluster; andan object location determining operation of determining the location of the object based on any one of the medians derived from the first object median derivation module and the second object median derivation module,wherein the bounding box determining operation, the sampling operation, the object clustering operation, and the first object median deriving operation are repeatedly performed by constituting a high-illuminance loop, and the line clustering operation and the second object median deriving operation are repeatedly performed by constituting a low-illuminance loop.12.The robot of claim 11, wherein the line clustering operation includes determining that the line cluster is a different line cluster when a difference in depth values of a specific pixel and a next pixel is greater than or equal to a threshold value and determining that the line cluster is the same line cluster when the difference is smaller than the threshold value, while sequentially reading the depth value of the pixel of the depth image.

Images