Robot and controlling method thereof
The robot uses a depth camera to generate 3D points at a preset height for accurate mapping and position estimation, addressing sensor limitations in existing technologies by providing cost-effective and miniaturized solutions with enhanced accuracy.
Patent Information
- Application Number
- KR1020200121772
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-21
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2040-09-21
AI Technical Summary
Existing self-driving robots face challenges with position estimation due to the limitations of vision sensors requiring high computation and causing image blur, and LiDAR sensors being costly, bulky, and difficult to miniaturize, especially 2D LiDAR's limited detection capability and multi-channel 3D LiDAR's high expense and size.
A robot utilizing a depth camera to generate 3D points based on depth images, identifying points at a preset height relative to the floor surface, and controlling its movement based on these points to create accurate maps and estimate its position.
Enables cost-effective and miniaturized sensor solutions for acquiring map data at various heights with reduced computational load and improved positional accuracy, minimizing image blur.
Smart Images

Figure 112020100292224-PAT00003_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a robot and a method for controlling the same, and more specifically, to a robot that moves using a depth image and a method for controlling the same. Background Technology
[0002] Recently, self-driving robots and their application technologies are being developed. To move autonomously, robots use Simultaneous Localization and Mapping (SLAM) technology, which estimates the robot's position (Localization) and creates a map (Mapping) in real time.
[0003] To this end, the robot can perform position estimation using data acquired through various sensors. In this case, for position estimation, vision sensors that acquire images projected onto a plane of the space within the camera's field of view and distance sensors such as LiDAR (Light Detection And Ranging), which acquire data scanning the distance to an object in a 360-degree direction, can generally be used.
[0004] Vision sensors, such as cameras, have the advantages of being inexpensive and capable of miniaturization, but they have the disadvantages of requiring a large amount of computation for data processing and causing blur in images acquired when the robot moves or rotates at high speeds.
[0005] Distance sensors such as LiDAR have disadvantages such as a short lifespan, high cost, and difficulty in miniaturizing the sensor, due to reasons such as rotating the sensor using a rotating mechanism like a motor to detect distance in a 360-degree direction. In particular, 2D (two-dimensional) LiDAR has difficulty detecting various environments or objects around a robot because it can only detect distance at a fixed height (the height at which the LiDAR is mounted), and multi-channel 3D (three-dimensional) LiDAR has the disadvantages of being very expensive and having large sensor size and weight.
[0006] A method is required to perform robot position estimation by compensating for the shortcomings of such sensors. The problem to be solved
[0007] The present disclosure was devised due to the aforementioned necessity, and the purpose of the present disclosure is to provide a robot that drives using a depth image and a method for controlling the same. means of solving the problem
[0008] A robot according to one embodiment of the present disclosure for achieving the above objective may include a depth camera, a driving unit, and a processor that acquires a depth image by taking a picture through the depth camera, generates a plurality of 3D points in a three-dimensional space corresponding to a plurality of pixels based on depth information of a plurality of pixels of the depth image, identifies a plurality of 3D points among the plurality of 3D points having a preset height value based on the floor surface on which the robot travels in the three-dimensional space, and controls the driving unit to drive the robot based on the identified plurality of 3D points.
[0009] The processor can determine the floor surface on which the robot travels in 3D space based on the distribution of multiple 3D points in 3D space.
[0010] The processor can rotate a plurality of 3D points in 3D space so that the determined floor surface is mapped to a preset plane in 3D space, and identify a plurality of 3D points among the rotated plurality of 3D points that have a preset height value based on the preset plane.
[0011] The preset plane corresponds to the xz plane in 3D space defined by the XYZ axes, and the processor can identify a plurality of rotated 3D points in which the value of the Y-axis has a preset value.
[0012] The processor can identify a plurality of 3D points having height values within a preset threshold range that includes a preset height value based on the bottom surface among a plurality of 3D points, and control a drive unit to drive a robot based on the identified plurality of 3D points.
[0013] The processor can convert the identified multiple 3D points into 2D data based on the X-axis values and Z-axis values of the identified multiple 3D points, and control the drive unit to drive the robot based on the 2D data.
[0014] The preset height value may be a height value set based on the robot's height value.
[0015] A method for controlling a robot according to one embodiment of the present disclosure may include the steps of: acquiring a depth image by taking a picture through a depth camera provided in the robot; generating a plurality of 3D points in a three-dimensional space corresponding to a plurality of pixels based on depth information of a plurality of pixels of the depth image; identifying a plurality of 3D points among the plurality of 3D points having a preset height value based on the floor surface on which the robot travels in the three-dimensional space; and controlling a driving unit provided in the robot to drive the robot based on the identified plurality of 3D points.
[0016] The identification step may include a step of determining the floor surface on which the robot travels in 3D space based on the distribution of a plurality of 3D points in 3D space.
[0017] The identification step involves rotating a plurality of 3D points in 3D space so that the determined floor surface is mapped to a preset plane in 3D space, and identifying a plurality of 3D points among the rotated plurality of 3D points that have a preset height value based on the preset plane.
[0018] The preset plane corresponds to the XZ plane in 3D space defined by the XYZ axes, and the identification step may include identifying a plurality of 3D points among a plurality of rotated 3D points where the Y-axis value has a preset value.
[0019] The identification step can identify a plurality of 3D points having height values within a preset threshold range that includes a preset height value based on the bottom surface among a plurality of 3D points.
[0020] The identification step may include converting the identified multiple 3D points into 2D data based on the X-axis values and Z-axis values of the identified multiple 3D points, and the control step may include controlling a drive unit to drive the robot based on the 2D data.
[0021] The preset height value may be a height value set based on the robot's height value. Effects of the invention
[0022] According to various embodiments of the present disclosure as described above, a robot that drives using a depth image and a method for controlling the same can be provided.
[0023] According to one embodiment of the present disclosure, there are advantages such as the ability to acquire map data for various heights without special limitations compared to LiDAR, cost-effectiveness, and miniaturization of the sensor.
[0024] According to one embodiment of the present disclosure, the amount of computation for estimating the position of a robot can be reduced. In addition, the accuracy of estimating the position of a robot can be improved by utilizing a depth image that minimizes the occurrence of blur. Brief explanation of the drawing
[0025] FIG. 1 is a drawing for explaining a robot according to one embodiment of the present disclosure. FIG. 2 is a block diagram for explaining the configuration of a robot according to one embodiment of the present disclosure. FIG. 3a is a drawing for explaining a method of generating 3D points through a depth image according to one embodiment of the present disclosure. FIG. 3b is a drawing for explaining a method of generating 3D points through a depth image according to one embodiment of the present disclosure. FIG. 4a is a drawing for explaining a method of identifying a bottom surface according to one embodiment of the present disclosure. FIG. 4b is a side view illustrating a method for identifying a bottom surface according to one embodiment of the present disclosure. FIG. 5a is a drawing for explaining a method of aligning a plurality of 3D points according to one embodiment of the present disclosure. FIG. 5b is a front view illustrating a method for aligning a plurality of 3D points according to one embodiment of the present disclosure. FIG. 5c is a side view illustrating a method for aligning a plurality of 3D points according to one embodiment of the present disclosure. FIG. 6a is a drawing for explaining a method for identifying a 3D point of a specific height according to one embodiment of the present disclosure. FIG. 6b is a front view and a side view for illustrating a method for identifying a 3D point of a specific height according to one embodiment of the present disclosure. FIG. 7 is a drawing for explaining a method for generating 2D data according to one embodiment of the present disclosure. FIG. 8 is a block diagram illustrating an additional configuration of a robot according to one embodiment of the present disclosure. FIG. 9 is a drawing for explaining a flowchart according to one embodiment of the present disclosure. Specific details for implementing the invention
[0026] In describing the present disclosure, detailed descriptions of related known functions or configurations are omitted if it is determined that such descriptions may unnecessarily obscure the essence of the present disclosure. Additionally, the following embodiments may be modified in various different forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.
[0027] It should be understood that the technology described in this disclosure is not intended to limit the technology to specific embodiments and includes various modifications, equivalents, and / or alternatives to the embodiments of this disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.
[0028] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.
[0029] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.
[0030] In this disclosure, singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “consisting of” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0031] When it is stated that a component (e.g., a first component) is "(operatively or communicatively) coupled with" or "connected to" another component (e.g., a second component), it should be understood that the component may be directly connected to the other component or connected through another component (e.g., a third component). On the other hand, when it is stated that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between the component and the other component.
[0032] The expression “configured to” as used in this disclosure may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” in conjunction with other devices or components. For example, the phrase “processor configured to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in a memory device.
[0033] FIG. 1 is a drawing for explaining a robot according to one embodiment of the present disclosure.
[0034] Referring to FIG. 1, a robot (100) according to one embodiment of the present disclosure is a device capable of driving in a specific area. In this case, the robot (100) can provide a specific service to a user while moving in a specific area. For example, the robot (100) can be implemented as a device capable of providing various services, such as a cleaning robot, a repair robot, an exploration robot, a transport robot, etc.
[0035] The robot (100) can drive according to various driving methods based on various sensing data. The sensing data is data that senses the surrounding environment of the robot (100) and can be used as map data, and the sensing data may include depth images. Additionally, the sensing data may be used to estimate the position of the robot (100) or to generate a map. The driving method may include at least one of a walking type that uses legs like a human or animal, a wheeled type (or caterpillar type) that uses the rotation of wheels, or a flying type that uses the rotation of wings or fuel injection.
[0036] A robot (100) according to one embodiment of the present disclosure can generate a map of a specific height based on a depth image and drive based on the map.
[0037] Specifically, the robot (100) can obtain a depth image by taking a picture of the surrounding environment through a depth camera (110). Here, the depth image may include depth information mapped to each of a plurality of pixels. The depth information may represent the distance between the position of the depth camera (110) and the position of the object (200). For example, assuming the case where the robot (100) and the object (200) are placed on the same axis as in FIG. 1, the depth image may include depth information representing the distance (e.g., d1, d2, d3) between the position of the robot (100) (e.g., position at height y2) and the position of the object (200) (e.g., position at height y1, y2, y3). In this case, the object (200) refers to an object or animal existing in the surrounding environment of the robot (100), and may mean, for example, a wall, a door, a table, a chair, a carpet, a cat, a person, etc.
[0038] Additionally, the robot (100) can generate multiple three-dimensional (3D) points based on a depth image. Here, the 3D points are defined by coordinates in a virtual 3D space (e.g., x, y, z coordinates), and the coordinates of the 3D points may correspond to locations in real space. Furthermore, the 3D points generated (or converted) based on the depth image may indicate information that an object exists at a corresponding location. That is, by using the distance to the object (200) mapped to multiple pixels of the depth image and the coordinates of each pixel, the object (200) can be generated as multiple 3D points, each having a specific location in 3D space. In this way, the robot (100) can convert the depth image into multiple 3D points. In this case, the multiple 3D points may be named a point cloud.
[0039] And, the robot (100) can identify a plurality of 3D points for a preset height relative to the floor surface among a plurality of 3D points, and drive based on the identified plurality of 3D points.
[0040] Here, the floor surface can be used as a reference plane to generate map data for a preset height. The preset height may be the height of the robot (100), but this is merely one example, and it may be set to a height that takes into account the environment in which the robot (100) travels. For example, if the robot (100) is a cleaning robot traveling in an environment such as a home, the preset height may be set to the height of a door threshold among the structures within the home.
[0041] Multiple identified 3D points can be used to generate a map of a preset height or to estimate the position of the robot (100) on the map. For example, when the robot (100) is driving, if depth images are continuously acquired over time through the depth camera (110), the robot (100) can repeatedly perform the aforementioned operation to identify multiple 3D points of a preset height corresponding to the depth images, and generate a map of a preset height by aligning the multiple 3D points identified over time into a single 3D space. Additionally, the robot (100) can identify multiple 3D points of a preset height corresponding to the most recently acquired depth image, and estimate the position and orientation of the robot (100) by determining the position and orientation of the part that matches the multiple 3D points identified in the generated map.
[0042] According to various embodiments of the present disclosure as described above, a robot (100) that drives using a depth image and a method for controlling the same can be provided. In addition, according to one embodiment of the present disclosure, map data for various heights can be acquired without special restrictions compared to LiDAR, and there are advantages such as cost-effectiveness and miniaturization of the sensor.
[0043] The present disclosure will be described in more detail below with reference to the attached drawings.
[0044] FIG. 2 is a block diagram for explaining the configuration of a robot according to one embodiment of the present disclosure.
[0045] Referring to FIG. 2, a robot (100) according to one embodiment of the present disclosure may include a depth camera (110), a driving unit (120), and a processor (130).
[0046] The depth camera (110) can acquire a depth image by taking a picture of an area within the field of view (FoV) of the depth camera (110).
[0047] Here, the depth image may include multiple pixels in which the real 3D space is projected onto a 2D plane (i.e., an image plane). The multiple pixels may be arranged in a matrix form on the 2D plane, and the position or coordinates of each pixel may indicate a direction in 3D space centered on the position of the depth camera (110). Additionally, depth information may be mapped to each of the multiple pixels, and the depth information may indicate the distance to an object (200) existing in a direction corresponding to the position of the pixel centered on the position of the depth camera (110).
[0048] To this end, the depth camera (110) may include a lens that focuses visible light or a signal received by being reflected by an object (200) onto an image sensor, and an image sensor capable of detecting visible light or a signal. Here, the image sensor may include a 2D pixel array divided into a plurality of pixels.
[0049] The depth camera (110) can be implemented using a stereo method, a Time-Of-Flight (ToF) method, a structured light method, etc.
[0050] The stereo method represents a method of calculating the distance (or depth) to an object (200) by using the disparity (i.e., the difference in position of the same object (200) included in the multiple images) of multiple images obtained by simultaneously capturing an object (200) from different positions using two cameras (or more cameras), similar to the human eye. The ToF method represents a method of calculating the distance (or depth) to an object (200) by using the difference between the time a signal (e.g., infrared, ultrasound, laser, etc.) is emitted and the time the emitted signal is reflected by the object (200) and detected, and the speed of the signal. In particular, the ToF method has the advantage of being able to identify the distance to an object (200) over a long distance, consuming less power, and having a small volume, which enables the miniaturization of the product. The structured light method represents a method of irradiating an object (200) with structured light that is distinct from ambient lighting, such as visible light, infrared, or lasers, and calculating the distance to the object (200) by detecting the distortion of the structured light reflected by the object (200).
[0051] Additionally, the depth camera (110) may be implemented as an RGB-D camera capable of acquiring a depth image in which depth information and color information (i.e., color values of Red, Green, and Blue) are mapped for each of the multiple pixels.
[0052] Meanwhile, the depth camera (110) can sequentially acquire multiple depth images through continuous shooting. At this time, the depth image or metadata separate from the depth image may include information regarding at least one of the time, frame rate, viewpoint, angle of view, pixel resolution, and pixel pitch captured by the depth camera (110). The frame rate indicates the number of frames (number of images) acquired per second (or per minute), and the angle of view may indicate a value determined by the focal length of the lens of the depth camera (110) and the size (e.g., diagonal length) of the image sensor of the depth camera (110). Additionally, the viewpoint may be detected by a sensor (e.g., gyroscope, accelerometer, etc.) provided inside or outside the depth camera (110). However, this is merely an example of an embodiment, and the viewpoint can be identified by identifying a portion that matches a plurality of 3D points generated based on a depth image on a map generated by aligning a plurality of 3D points, and by using the degree of misalignment thereof. Pixel resolution indicates the number of pixels arranged in the horizontal direction (e.g., 640) and the number of pixels arranged in the vertical direction (e.g., 480), such as 640 * 480, and pixel pitch may indicate the distance between adjacent pixels (e.g., 50 µm, etc.).
[0053] Meanwhile, a depth camera (110) according to one embodiment of the present disclosure can acquire a depth image by performing shooting according to a global shutter method. A global shutter is a method of performing shooting by simultaneously exposing the entire image sensor and then closing it all at once, and since the shooting time of a frame is the same, distortion due to parallax may not occur. That is, it is possible to prevent distortion including blur from occurring in the acquired depth image.
[0054] In contrast, a rolling shutter is a method of taking photos by sequentially varying the exposure for each horizontal line or vertical line of an image sensor, and in the case of fast-moving objects, the depth image obtained through this method may be distorted. However, the present disclosure does not exclude the application of such a rolling shutter method to a depth camera (110) for various reasons such as cost-effectiveness and miniaturization.
[0055] The drive unit (120) is a device capable of driving the robot (100), and the drive unit (120) can adjust the driving direction and driving speed according to the control of the processor (130). To this end, the drive unit (120) may include a power generation device that generates power for the robot (100) to drive (e.g., a gasoline engine, a diesel engine, an LPG (liquefied petroleum gas) engine, an electric motor, etc., depending on the fuel (or energy source) used), a steering device for adjusting the driving direction (e.g., manual steering, hydraulics steering, electronic control power steering (EPS), etc.), and a driving device that drives the robot (100) according to the power (e.g., wheels, propellers, etc.). Here, the drive unit (120) may be modified according to the driving type of the robot (100) (e.g., wheel type, walking type, flying type, etc.).
[0056] The processor (130) can obtain a depth image by taking a picture through a depth camera (110), generate a plurality of 3D points in a three-dimensional space corresponding to a plurality of pixels based on depth information of a plurality of pixels of the depth image, identify a plurality of 3D points among the plurality of 3D points that have a preset height value based on the floor surface where the robot (100) travels in the three-dimensional space, and control the drive unit (120) so that the robot (100) travels based on the identified plurality of 3D points.
[0057] Specifically, the processor (130) can acquire a depth image by taking a picture through the depth camera (110). The processor (130) can generate multiple 3D points in 3D space corresponding to multiple pixels based on the depth information of multiple pixels of the depth image. This will be explained in detail with reference to FIGS. 3a and 3b.
[0058] And, the processor (130) can identify a plurality of 3D points having a preset height value based on the floor surface where the robot (100) travels in 3D space among a plurality of 3D points.
[0059] As a specific embodiment, the processor (130) can determine the floor surface on which the robot (100) travels in 3D space based on the distribution of a plurality of 3D points in 3D space. This will be explained in detail with reference to FIGS. 4a and 4b.
[0060] Here, the processor (130) may rotate a plurality of 3D points in 3D space so that the determined floor surface is mapped to a preset plane in 3D space. In this case, the preset plane may correspond to the XZ plane in 3D space defined by the XYZ axes. This will be explained in detail with reference to FIGS. 5a to 5c.
[0061] Here, the processor (130) can identify a plurality of rotated 3D points having a preset height value based on a preset plane. In this case, the processor (130) can identify a plurality of rotated 3D points having a preset value for the Y-axis. This will be explained in detail with reference to FIGS. 6a and 6b.
[0062] And, the processor (130) can control the drive unit (120) so that the robot (100) drives based on a plurality of identified 3D points.
[0063] As an example of one embodiment, the processor (130) can convert the identified multiple 3D points into 2D data based on the X-axis values and Z-axis values of the identified multiple 3D points. Then, the processor (130) can control the drive unit (120) so that the robot (100) drives based on the 2D data. This will be explained in detail with reference to FIG. 7.
[0064] Meanwhile, as an embodiment of the present disclosure, the processor (130) can identify a plurality of 3D points having height values within a preset threshold range that includes a preset height value based on a floor surface among a plurality of 3D points. In this case, the processor (130) can control the drive unit (120) so that the robot (100) drives based on the identified plurality of 3D points.
[0065] FIGS. 3a and 3b are drawings for explaining a method of generating 3D points through a depth image according to one embodiment of the present disclosure.
[0066] Referring to (1) in FIG. 3a, a processor (130) according to one embodiment of the present disclosure can obtain a depth image (310) by taking a picture through a depth camera (110). Here, the depth image (310) may include depth information mapped in pixel units.
[0067] As an example of one embodiment, the depth image (310) may be implemented as data arranged in a matrix form. Here, in the depth image (310), columns may represent the x-axis coordinates of pixels, and in the depth image (310), rows may represent the y-axis coordinates of pixels. Values mapped to specific rows and columns in the depth image (310) may be depth information mapped to pixels having the corresponding coordinates. For example, among multiple pixels (e.g., 640 * 480 pixels) included in the depth image (310), depth information such as 1 meter for pixel (1, 1) and 1.2 meters for pixel (1, 2) may be mapped and stored.
[0068] Referring to (2) in FIG. 3a, the processor (130) can generate a plurality of 3D points (320) in 3D space based on depth information of a plurality of pixels included in the depth image (310) and the coordinates of the pixels. Here, each of the plurality of 3D points (320) may correspond to each pixel of the depth image (310). The 3D points are located in 3D space having a coordinate system of X-axis, Y-axis, and Z-axis, and the location of the 3D points may correspond to the location where the object (200) exists.
[0069] Hereinafter, the process of generating a first 3D point, which is one of the multiple 3D points (320), will be described as a representative process of generating multiple 3D points (320). For example, the processor (130) can generate a first 3D point having a position (X1, Y1, Z1) according to the following mathematical formula 1, based on coordinates (e.g., (x1, y1)) mapped to a first pixel included in the depth image (310) and depth information (e.g., d1).
[0070]
[0071] Here, the 3D space is set with the horizontal direction of the depth camera (110) as the X-axis, the vertical direction of the depth camera (110) as the Y-axis, and the optical axis direction as the Z-axis, based on the center of the lens of the depth camera (110).
[0072] Here, x1 and y1 represent the row and column coordinates of a pixel in the depth image (310), and d1 represents the depth information mapped to the corresponding pixel. Also, f x , f y represents the focal length of the depth camera (110) for the X-axis and Y-axis, and c x , c y represents the location of the principal point with respect to the X-axis and Y-axis, and this may be information stored in memory inside or outside the depth camera (110).
[0073] By such an operation, a plurality of 3D points (320F) of (2) in FIG. 3b can be generated according to the coordinates and depth information mapped to a plurality of pixels included in the depth image (1) of FIG. 3b. Here, the plurality of 3D points (320F) can be implemented in the form of an image that visually represents the depth information mapped to each pixel, as in (1) of FIG. 3b. At this time, the depth information may include a brightness value (or grayscale) corresponding to the distance from the object (200).
[0074] Meanwhile, the above-described example is merely one embodiment, and the processor (130) can generate 3D points based on the coordinates of pixels and depth information included in the depth image (310) through various geometric relationships or various algorithms. Additionally, although the above-described operation is described as being performed by the processor (130), this is merely one embodiment, and the operation of generating multiple 3D points based on the depth image (310) can also be performed inside the depth camera (110).
[0075] Meanwhile, the height of the aforementioned multiple 3D points can generally be the Y-axis value. However, given that there is a possibility that the floor surface is tilted or that the height of the floor surface may be measured as a value other than 0 meters, a process of identifying the floor surface that serves as a standard for determining the height may be required to extract a 3D point of a specific height. Below, a method for identifying the floor surface will be explained in detail with reference to FIGS. 4a and 4b.
[0076] FIGS. 4a and 4b are drawings for illustrating a method of identifying a bottom surface according to an embodiment of the present disclosure. FIG. 4b is a side view of the first plane (430G) of FIG. 4a.
[0077] Referring to FIG. 4a, the processor (130) can determine the floor surface (430G) on which the robot (100) travels in 3D space based on the distribution of a plurality of 3D points (420) in 3D space.
[0078] For example, the processor (130) can randomly select a plurality of 3D points (e.g., 3 3D points) from a plurality of 3D points (420). Then, the processor (130) can identify a first plane (430G) through the locations of the 3 3D points and the following mathematical formula 2.
[0079]
[0080] Here,<A, B, C> can represent a normal vector n perpendicular to the first plane (430G). The values of A, B, C, and D can be obtained in various ways. For example, assuming the case where the first 3D point is located at (X1, Y1, Z1), the second 3D point is located at (X2, Y2, Z2), and the third 3D point is located at (X3, Y3, Z3), the values of A, B, C, and D can be obtained by solving the system of equations by substituting the first to third 3D points into X, Y, and Z of Equation 2, respectively.
[0081] Referring to FIG. 4b, the processor (130) may consider a 3D point among a plurality of 3D points (420) located within a preset distance (e.g., t) from the first plane (430G) as an inlier, and consider the remaining 3D points excluding these as outliers. The processor (130) may calculate the distribution (i.e., number) of 3D points identified as inliers and store it in memory.
[0082] The processor (130) can repeat the above-described operation to calculate the distribution of normal points for randomly identified second to nth planes (n is a natural number greater than or equal to 2). Then, the processor (130) can determine the plane having the largest distribution among the distributions of normal points for the first plane (430G) to the nth plane as the bottom plane.
[0083] Meanwhile, when a processor (130) according to one embodiment of the present disclosure randomly selects a 3D point, it identifies a 3D point among a plurality of 3D points (420) in which the Y value is less than or equal to a preset value, and randomly selects a 3D point from among the identified 3D points. This has the effect of reducing the number of operations for the bottom surface determination process, taking into account that the bottom surface is generally located mostly on the lower side.
[0084] Meanwhile, the above-described embodiment is merely one example, and the processor (130) can determine the bottom surface through various algorithms such as RANSAC (Random sample consensus) and eigenvalue analysis.
[0085] In addition, the processor (130) can align multiple 3D points relative to the bottom surface to identify 3D points having a preset height value relative to the bottom surface. That is, all 3D points can be aligned so that the 3D points corresponding to the bottom surface have a Y-axis value of 0. This will be explained in detail with reference to FIGS. 5a to 5c.
[0086] FIGS. 5a to 5c are drawings for explaining a method of aligning a plurality of 3D points according to one embodiment of the present disclosure.
[0087] Referring to (1) in FIG. 5a, the processor (130) can rotate a plurality of 3D points (520) in 3D space so that the determined bottom surface (530G) is mapped to a preset plane (G') in 3D space.
[0088] Here, the pre-set plane (G') may correspond to the XZ plane in 3D space defined by the XYZ axes. Here, the XZ plane is a plane where the value of the Y-axis is 0, such as Y=0.
[0089] Specifically, the processor (130) can calculate the angle θ between the normal vector n and the direction vector of the Y-axis by using various methods such as the dot product or cross product of the normal vector n perpendicular to the determined floor surface (530G) and the direction vector of the Y-axis. Then, the processor (130) can rotate all of the plurality of 3D points (520) so that the angle θ between the normal vector n and the direction vector of the Y-axis becomes 0.
[0090] In this case, multiple points (540) and a bottom surface (540G) can be aligned by rotation as in (2) of FIG. 5a. At this time, the value of the Y-axis of the 3D point can represent the height of the object relative to the bottom surface (540G).
[0091] Referring to FIG. 5b and FIG. 5c, an example in which a plurality of 3D points are aligned according to an embodiment of the present disclosure will be described. FIG. 5b shows a plurality of 3D points in the front direction of an object, and FIG. 5c shows a plurality of 3D points in the side direction of an object.
[0092] For example, assuming that a bottom surface (530G) is determined among a plurality of 3D points (520F, 520S) generated based on a depth image as in (1) of FIG. 5b and (1) of FIG. 5c, the bottom surface (530G) can be mapped to a plane having a Y-axis value of 0. In this case, as in (2) of FIG. 5b and (2) of FIG. 5c, a plurality of 3D points (540F, 540S) can be aligned with the bottom surface (540G) mapped to a plane having a Y-axis value of 0.
[0093] And, the processor (130) can identify a plurality of 3D points having a preset height value based on the floor surface where the robot (100) travels in 3D space among a plurality of 3D points. This will be explained in detail with reference to FIGS. 6a and 6b.
[0094] FIGS. 6a and 6b are drawings for illustrating a method for identifying a 3D point of a specific height according to one embodiment of the present disclosure.
[0095] Referring to FIG. 6a, the processor (130) can identify a plurality of rotated 3D points (640) having a preset height value based on a preset plane (G'). At this time, the plurality of 3D points (640) are rotated so that the bottom surface (640G) is mapped to a preset plane (G') in 3D space.
[0096] Here, the preset plane (G') may correspond to the same XZ plane (plane where Y=0) as the bottom plane (640G). The identified multiple 3D points may represent the position of an object in a plane having a preset height value relative to the bottom plane of the robot (100).
[0097] That is, the processor (130) can identify a plurality of rotated 3D points (640) in which the Y-axis value has a preset value. For example, a plurality of 3D points in which the Y-axis value has a preset value h are 3D points located on the H plane (650H) where Y=h.
[0098] An example of identifying 3D points located on an H plane (650H) having a preset height value based on the floor surface (640G) described above may be shown as illustrated in (1) and (2) of FIG. 6b. (1) of FIG. 6b shows a plurality of 3D points in the front direction of an object (640F), and (2) of FIG. 6b shows a plurality of 3D points in the side direction of an object (640S).
[0099] Meanwhile, as an example embodiment, a preset height value may be a value based on the height of the robot (100). For example, the preset height value may be a value within a range of 0 or greater and less than or equal to the height value of the robot (100). This is for estimating the position of the robot (100) or determining the position of an object that may collide with the robot (100).
[0100] As another embodiment, the preset height value may be a height value based on a person's height. For example, the preset height value may be a Y value corresponding to a height value greater than a typical person's height (e.g., 2 meters). This is to perform position estimation of the robot (100) more accurately by excluding dynamic objects and identifying the indoor structure.
[0101] Furthermore, although it has been explained that the preset height value is set as a single value, this is merely an example, and the preset height value may be set as a plurality of values. In this case, the 3D points for each height value can be used to estimate the position of the robot (100).
[0102] Meanwhile, the embodiments regarding the aforementioned preset height values are merely examples, and various modified embodiments are possible.
[0103] Meanwhile, as an embodiment of the present disclosure, the processor (130) can identify a plurality of 3D points having height values within a preset threshold range that includes a preset height value based on a bottom surface. For example, the height value within the preset threshold range may represent a height value between ha (a, obtained by subtracting a, a, from a preset height value h) and h+a (a, obtained by adding a, a, to a preset threshold value h). That is, the height value within the preset threshold range may be a value in the range greater than or equal to ha and less than h+a.
[0104] And, the processor (130) can control the drive unit (120) so that the robot (100) drives based on a plurality of identified 3D points. This will be explained together with reference to FIG. 7.
[0105] FIG. 7 is a drawing for explaining a method for generating 2D data according to an embodiment of the present disclosure. (1) of FIG. 7 is a top view projected onto an XZ plane of a plurality of aligned 3D points (740T), and (2) of FIG. 7 is a top view projected onto an XZ plane of a plurality of 3D points (750T) having a preset height.
[0106] Referring to (1) and (2) of FIG. 7, the processor (130) can convert a plurality of identified 3D points (750T) into 2D data based on the X-axis and Z-axis values of a plurality of 3D points (750T) having a preset height identified among a plurality of aligned 3D points (740T). For example, the processor (130) can convert a 2D point having X-axis and Z-axis values (X, Z) by omitting (or removing) the Y-axis value h among the (X, h, Z) values of a 3D point having a preset height h. That is, the 2D data may include a plurality of 2D points. Additionally, the 2D data may include information about height. This 2D data can be utilized as 2D LiDAR-based SLAM by converting it into the same format as the output value (or input value, etc.) of a 2D LiDAR sensor.
[0107] And, the processor (130) can control the drive unit (120) so that the robot (100) drives based on 2D data.
[0108] Specifically, the processor (130) can generate a plurality of 3D points based on depth images acquired periodically, identify a plurality of 3D points having a preset height among the aligned plurality of 3D points, and convert the identified plurality of 3D points into 2D data. In this case, the processor (130) can periodically align the converted 2D data to create (or update) a 2D map.
[0109] Additionally, the processor (130) can generate a plurality of 3D points based on a depth image acquired at the current (or most recent) time, identify a plurality of 3D points having a preset height among the aligned plurality of 3D points, and convert the identified plurality of 3D points into 2D data. In this case, the processor (130) can compare the 2D data converted at the current (or most recent) time with a 2D map to identify the position of the robot (100) on the 2D map.
[0110] A processor (130) according to one embodiment of the present disclosure can simultaneously perform the operation of generating the above-described 2D map and the operation of estimating the location.
[0111] And, the processor (130) can determine (or plan) a driving path from the position of the robot (100) to the destination on a 2D map. At this time, various path search algorithms may be used, such as an algorithm for finding the shortest driving path and an algorithm for finding a path that minimizes changes in driving direction.
[0112] And, the processor (130) can control the drive unit (120) to drive along the driving path to the destination.
[0113] According to various embodiments of the present disclosure as described above, a robot that drives using a depth image and a method for controlling the same can be provided.
[0114] In addition, since depth images are utilized, map data for various heights can be acquired without special restrictions compared to LiDAR. Also, the depth camera (110) capable of acquiring depth images has the advantage of being more cost-effective than LiDAR and allowing for the miniaturization of the sensor.
[0115] In addition, the amount of computation can be reduced by processing multiple 3D points with a specific height rather than processing all of the multiple 3D points generated based on the depth image.
[0116] FIG. 8 is a block diagram illustrating an additional configuration of a robot according to one embodiment of the present disclosure.
[0117] Referring to FIG. 8, a robot (100) according to one embodiment of the present disclosure may further include at least one of an input interface (140), an output interface (150), a memory (160), a sensor (170), a communication unit (180), and a power supply unit (190), in addition to a depth camera (110), a driving unit (120), and a processor (130). Content that overlaps with that described in FIG. 2 will be omitted.
[0118] The processor (130) can be implemented as a general-purpose processor such as a CPU (Central Processing Unit) or AP (Application Processor), a graphics-dedicated processor such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU (Neural Processing Unit). Additionally, the processor (130) may include volatile memory for loading at least one instruction or module.
[0119] The input interface (140) can receive various user commands and transmit them to the processor (130). That is, the processor (130) can recognize user commands input by the user through the input interface (140). Here, user commands can be implemented in various ways, such as user touch input (touch panel), key (keyboard) or button (physical button or mouse, etc.) input, user voice (microphone), etc.
[0120] The output interface (150) is a configuration capable of outputting information and can be implemented, for example, as a display or a speaker. A display is a device that outputs information or data in a visual form. The display can display image frames in one area or the entire area of the display, which can be driven by pixels. At least a portion of the display may be combined in the form of a flexible display in at least one of the front area, side area, and rear area of the robot (100). The flexible display may be characterized by being able to bend, curve, or roll without damage through a paper-thin and flexible substrate. The speaker can directly output various notification sounds or voice messages as sound, as well as various audio data for which various processing operations such as decoding, amplification, and noise filtering have been performed by an audio processing unit (not shown).
[0121] The memory (160) is configured to store an operating system (OS) for controlling the overall operation of the components of the robot (100) and various data related to the components of the robot (100).
[0122] To this end, memory (160) may be configured as hardware that stores data or information temporarily or permanently. For example, memory (160) may be implemented as at least one of non-volatile memory, volatile memory, flash memory, hard disk drive (HDD) or solid state drive (SSD), RAM, ROM, etc.
[0123] At least one instruction, program, or data required for the operation of the robot (100) or processor (130) may be stored in the memory (160). Here, the instruction is a unit of code that directs the operation of the robot (100) or processor (130) and may be written in machine language, which is a language that a computer can understand. The program may be an instruction set, which is a set of instructions that perform a specific task of a task unit. The data may be state information in bit or byte units that can represent characters, numbers, images, etc.
[0124] The sensor (170) can be implemented as various sensors such as a camera, microphone, proximity sensor, light sensor, motion sensor, ToF sensor, GPS sensor, etc. For example, a camera can distinguish light in pixel units and detect the intensity of light for R (Red), G (Green), and B (Blue) colors for each pixel, convert the light intensity into an electrical signal, and obtain data expressing the color, shape, brightness, etc. of an object. At this time, the type of data may be an image having R, G, and B color values for each of multiple pixels. A microphone can detect sound waves such as a user's voice and convert the sound waves into an electrical signal to obtain data. At this time, the type of data may be an audio signal of various formats. A proximity sensor can detect the presence of surrounding objects and obtain data regarding the presence or proximity of surrounding objects. A light sensor can detect the amount of light (or brightness) of the surrounding environment of the robot (100) and obtain data regarding the light intensity. The motion sensor can detect the movement distance, direction of movement, tilt, etc. of the robot (100). To this end, the motion sensor can be implemented as a combination of an accelerometer, a gyro sensor, a geomagnetic sensor, etc. The Time of Flight (TOF) sensor can obtain data regarding the distance (or position) to an object by detecting the flight time of various electromagnetic waves (e.g., ultrasound, infrared, laser, UWB, etc.) having a specific speed and returning. The Global Positioning System (GPS) sensor can obtain data regarding the current position of the robot (100) by receiving radio signals from multiple satellites, calculating the distance to each satellite using the transmission time of the received signals, and using triangulation of the calculated distance.However, the above-described example of the implementation of the sensor (170) is merely one example and is not limited thereto, and it is possible to implement it with various types of sensors.
[0125] The communication unit (180) can transmit and receive various types of data by communicating with various types of external devices according to various types of communication methods. The communication unit (180) is a circuit that performs various types of wireless communication and may include at least one of an Ethernet module, USB module, HDMI (High Definition Multimedia Interface), DP (DisplayPort), D-SUB (D-subminiature), DVI (Digital Visual Interface), Thunderbolt, and components that perform wired communication with a Bluetooth module (Bluetooth method), a Wi-Fi module (Wi-Fi method), a wireless communication module (cellular method such as 3G, 4G, 5G), an NFC module (NFC method), an IR module (infrared method), a Zigbee module (Zigbee method), and an ultrasonic module (ultrasonic method).
[0126] The power supply unit (190) can supply power to each component of the robot (100). For example, the power supply unit (190) may include a battery that can be charged by an external commercial power source.
[0127] FIG. 9 is a drawing for explaining a flowchart according to one embodiment of the present disclosure.
[0128] Referring to FIG. 9, a control method for a robot (100) may include the step of obtaining a depth image by taking a picture through a depth camera (110) provided in the robot (100) (S910); the step of generating a plurality of 3D points in a three-dimensional space corresponding to a plurality of pixels based on depth information of a plurality of pixels of the depth image (S920); the step of identifying a plurality of 3D points among the plurality of 3D points having a preset height value based on the floor surface where the robot (100) travels in the three-dimensional space (S930); and the step of controlling a driving unit (120) provided in the robot (100) so that the robot (100) travels based on the identified plurality of 3D points (S940).
[0129] Specifically, the control method of the robot (100) can first obtain a depth image by taking a picture through a depth camera (110) equipped on the robot (100) (S910).
[0130] Next, based on the depth information of multiple pixels of a depth image, multiple 3D points in a three-dimensional space corresponding to multiple pixels can be generated (S920).
[0131] Next, among the plurality of 3D points, a plurality of 3D points having a preset height value based on the floor surface where the robot (100) travels in 3D space can be identified (S930). Here, the preset height value may be a height value set based on the height value of the robot.
[0132] As an example of one embodiment, the identifying step may include the step of determining the floor surface on which the robot (100) travels in 3D space based on the distribution of a plurality of 3D points in 3D space.
[0133] The identification step here involves rotating a plurality of 3D points in 3D space so that the determined floor surface is mapped to a preset plane in 3D space, and identifying a plurality of 3D points among the rotated plurality of 3D points that have a preset height value based on the preset plane.
[0134] Meanwhile, the preset plane may correspond to the XZ plane in 3D space defined by the XYZ axes. In this case, the identification step may include identifying a plurality of 3D points among a plurality of rotated 3D points in which the Y-axis value has a preset value.
[0135] Here, the identification step can convert the identified multiple 3D points into 2D data based on the X-axis values and Z-axis values of the identified multiple 3D points.
[0136] Meanwhile, the identification step can identify a plurality of 3D points having height values within a preset threshold range that includes a preset height value based on the bottom surface among a plurality of 3D points.
[0137] Next, a drive unit (120) equipped in the robot (100) can be controlled to drive the robot (100) based on a plurality of identified 3D points (S940).
[0138] Here, the control step can control the drive unit (120) so that the robot (100) drives based on 2D data.
[0139] Various embodiments of the present disclosure may be implemented as software containing instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device (e.g., a robot (100)) according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When the instructions are executed by a processor, the processor may perform the function described in the instructions directly or using other components under the control of the processor. The instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily on the storage medium.
[0140] Methods according to various embodiments may be provided as part of a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.
[0141] Each component (e.g., module or program) according to various embodiments may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the components prior to integration. The operations performed by the module, program, or other components according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added. Explanation of the symbols
[0142] 100: Robot
Claims
Claim 1 In a robot, depth camera; drive unit; A processor comprising: acquiring a depth image by performing shooting through the depth camera; generating a plurality of 3D points in a three-dimensional space corresponding to a plurality of pixels based on depth information of a plurality of pixels of the depth image; identifying a plurality of 3D points among the plurality of 3D points having a preset height value based on the floor surface on which the robot travels in the 3D space; and controlling the drive unit to drive the robot based on the identified plurality of 3D points; wherein the processor rotates a plurality of 3D points in the 3D space so that the floor surface is mapped to an XZ plane in the 3D space defined by XYZ axes; identifies a plurality of 3D points among the rotated plurality of 3D points having a preset height value based on the XZ plane; obtains a plurality of 2D points having X-axis and Z-axis values by removing the Y-axis value from the coordinate values of the identified plurality of 3D points; converts the identified plurality of 3D points into 2D data; A robot that identifies the position of the robot on a 2D map based on 2D data and controls the drive unit to drive the robot based on the identified position. Claim 2 In claim 1, the processor determines the floor surface on which the robot travels in the 3D space based on the distribution of the plurality of 3D points in the 3D space. Claim 3 delete Claim 4 In claim 1, the processor is a robot that identifies a plurality of 3D points among the rotated plurality of 3D points having a Y-axis value that is preset. Claim 5 delete Claim 6 delete Claim 7 In paragraph 1, the robot, wherein the preset height value is a height value set based on the height value of the robot. Claim 8 A method for controlling a robot comprises: a step of acquiring a depth image by performing shooting through a depth camera provided in the robot; a step of generating a plurality of 3D points in a three-dimensional space corresponding to a plurality of pixels based on depth information of a plurality of pixels of the depth image; and a step of identifying a plurality of 3D points among the plurality of 3D points having a preset height value based on the floor surface on which the robot travels in the three-dimensional space. The control method comprises: a step of controlling a drive unit equipped in the robot to drive the robot based on the identified plurality of 3D points; wherein the identifying step comprises rotating a plurality of 3D points in the 3D space so that the floor surface is mapped to an XZ plane in the 3D space defined by XYZ axes, identifying a plurality of 3D points having a preset height value based on the XZ plane among the rotated plurality of 3D points, obtaining a plurality of 2D points having X-axis values and Z-axis values by removing the Y-axis value from the coordinate values of the identified plurality of 3D points, and converting the identified plurality of 3D points into 2D data; and the controlling step comprises identifying the position of the robot on a 2D map based on the 2D data and controlling the drive unit to drive the robot based on the identified position. Claim 9 In claim 8, the identifying step comprises determining a floor surface on which the robot travels in the 3D space based on the distribution of the plurality of 3D points in the 3D space; a control method. Claim 10 delete Claim 11 A control method according to claim 8, wherein the identifying step comprises identifying a plurality of 3D points among the rotated plurality of 3D points having a Y-axis value that is preset. Claim 12 delete Claim 13 delete Claim 14 A control method according to claim 8, wherein the above-mentioned preset height value is a height value set based on the height value of the robot.
Citation Information
Patent Citations
Moving robot and controlling method thereof
KR1020170103556A
