Visual control device and method of wearable robot

Abstract

The invention relates to a visual control device and method of a wearable robot. A visual control device of a wearable robot may include a transceiver configured to communicate with at least one controller of the wearable robot; at least one processor; and a memory. The memory may store instructions that, when executed by the at least one processor, are configured to cause the visual control device to: receive, via the receiver of the transceiver, an indication of a current robot foot position from the at least one controller of the wearable robot; detecting, via a depth camera of the wearable robot, features of terrain around the wearable robot; generating point cloud-based geometric information associated with the terrain based on the detected features of the terrain; based on the current robot foot position and the geometric information based on the point cloud, determining a subsequent robot foot position; and transmit, via a transmitter of the transceiver, the determined subsequent robot foot position to at least one controller of the wearable robot.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from Korean Patent Application No. 10-2024-0053870, filed on April 23, 2024, which is hereby incorporated by reference in its entirety for all purposes. Technical Field

[0003] The present disclosure relates to a visual control device and method for a wearable robot, and more particularly, to a visual control device and method for a wearable robot that can respond instantly to various walking environments. Background Art

[0004] If a robot is worn for walking for medical and rehabilitation purposes such as rebuilding lower limb muscles or restoring joint movement to patients or disabled people (e.g., a medical wearable robot), the robot without a device for environmental perception will have difficulty moving on stairs with steps or height changes.

[0005] Therefore, the introduction of vision control technology may be useful in helping robots move more safely and flexibly in various terrain environments and / or improving the convenience and stability of users controlling the robot. Summary of the Invention

[0006] The present disclosure attempts to provide a visual control device and method for a wearable robot, which can perceive the terrain environment including flat ground, stairs, and inclined roads through a camera, and calculate the arrangement path of the robot's feet in response to the perceived environment and provide it to the robot.

[0007] The present disclosure attempts to provide a visual control device and method for a wearable robot, which can estimate the height and normal of the terrain through a grid-type elevation map, thereby generating geometric information and perceiving the terrain environment including flat ground, stairs, and inclined roads.

[0008] A visual control device of a wearable robot can include a transceiver configured to communicate with at least one controller of the wearable robot, at least one processor, and a memory storing instructions that, when executed by the at least one processor, are configured to cause the visual control device to receive, via a receiver of the transceiver, an indicator of a current robot foot position from the at least one controller of the wearable robot, detect, via a depth camera of the wearable robot, a feature of a terrain surrounding the wearable robot, generate point cloud-based geometry information associated with the terrain based on the detected feature of the terrain, determine a subsequent robot foot position based on the current robot foot position and the point cloud-based geometry information, and transmit, via a transmitter of the transceiver, the determined subsequent robot foot position to the at least one controller of the wearable robot.

[0009] The at least one processor can include a red-green-blue-depth (RGB-depth) preprocessor configured to convert depth information of the depth camera into a point cloud form to generate the point cloud-based geometry information associated with the terrain, wherein the depth camera comprises an RGB-depth camera.

[0010] The RGB-depth preprocessor can be configured to perform point cloud filtering to remove noise and adjust data size from the converted point cloud form.

[0011] The device can further include an inertial measurement unit (IMU)-based point cloud corrector configured to align the filtered point cloud in a gravity direction based on one or more measurements of at least one IMU sensor.

[0012] The instructions, when executed by the at least one processor, can be configured to cause the visual control device to set a region of interest, align the filtered point cloud within the region of interest, segment the aligned point cloud within the region of interest into a plurality of grids, determine an elevation value of each portion of the terrain corresponding to each grid of the plurality of grids, and generate an elevation map by combining the determined elevation values.

[0013] Each of the determined elevation values is determined based on an average of length values associated with the point cloud in the gravity direction, the length values being input for each grid of the plurality of grids.

[0014] The instructions, when executed by the at least one processor, can be configured to cause the visual control device to generate a plurality of clusters by clustering the point cloud based on a distance between the point cloud in the elevation map and a normal vector estimated for each of the point cloud.

[0015] The instructions, when executed by the at least one processor, can be configured to cause the visual control device to: classify respective point clouds for which an angle difference between normal vectors of point clouds adjacent to each other based on a distance is determined to be within a threshold level into a same cluster.

[0016] The instructions, when executed by the at least one processor, can be configured to cause the visual control device to: extract geometry information of a terrain from the plurality of clusters, respectively.

[0017] The plurality of clusters can include geometry information about at least one of a flat ground, a stair, an uphill, or a downhill.

[0018] A control method of a wearable robot can include receiving an indication of a current robot foot position from at least one controller of the wearable robot, detecting a feature of a terrain around the wearable robot via a depth camera of the wearable robot, generating point cloud-based geometry information associated with the terrain based on the detected feature of the terrain, determining a subsequent robot foot position based on the current robot foot position and the point cloud-based geometry information, transmitting the determined subsequent robot foot position to the at least one controller of the wearable robot, and controlling the wearable robot based on the determined subsequent robot foot position.

[0019] The control method can further perform one or more operations described herein.

[0020] The visual control device and method of a wearable robot according to an example can predict a movement path of the robot through geometry information of each perceived terrain and implement a movement response to various environments.

[0021] The visual control device and method of a wearable robot according to an example can estimate a height and a normal of a terrain through a grid-type elevation map and extract geometry information (a height, a width, a slope of a terrain) to perceive a flat ground, a stair, a sloped road terrain, thereby providing a fast processing speed and reducing an error. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A visual control device of a wearable robot according to an example and a visual system of a wearable robot including the visual control device of the wearable robot are schematically illustrated.

[0023] Figure 2 and Figure 3 A flowchart illustrating a visual-based walking scenario of a robot by a visual system of a wearable robot is illustrated.

[0024] Figure 4 A flowchart illustrating a visual control method of a wearable robot according to an example is illustrated.

[0025] Figure 5 FIG. 1 shows a diagram illustrating an elevation map generation step according to an example. Figure 4

[0026] Figure 6 FIG. 2 shows a diagram illustrating a clustering step according to an example. Figure 4

[0027] Figure 7 FIG. 3 shows a diagram illustrating a geometry information extraction step according to an example. Figure 4

[0028] Figure 1 FIG. 4 shows a diagram illustrating a result provided according to a visual control method of a wearable robot according to an example. Figure 8A

[0029] Figure 8B FIG. 5 shows a diagram illustrating a wearable robot according to an example. DETAILED DESCRIPTION

[0030] Examples of the present disclosure are described more fully hereinafter with reference to the accompanying drawings, in which examples of the present disclosure are shown. As those skilled in the art will understand, the described examples can be modified in various different ways, all without departing from the spirit or scope of the present disclosure. To clarify the present disclosure, portions unrelated to the description will be omitted, and the same elements or equivalents are designated by the same reference numerals throughout the specification.

[0031] In addition, unless explicitly described to the contrary, the word "comprise" and variations such as "comprises" or "comprising" will be understood to imply the inclusion of stated elements but not the exclusion of any other elements. The terms including ordinal numbers such as first and second are used to describe various constituent elements, but the constituent elements are not limited by the terms. The terms are used only to distinguish one component from another component.

[0032] In addition, the terms "unit", "part", or "portion", "device", and "module" in the present specification refer to a unit that processes at least one function or operation, which can be implemented by hardware, software, or a combination of hardware and software.

[0033] Hereinafter, examples of the present disclosure will be described with reference to the accompanying drawings.

[0034] Figure 9 A visual control device of a wearable robot according to an example and a visual system of a wearable robot including the visual control device of the wearable robot.

[0035] ​​​​The vision system of the wearable robot can be a vision system of a medical wearable robot (lower limb wearable robot) installed with an RGB-depth camera and an inertial measurement unit (IMU) sensor for environment perception.

[0036] For example, the RGB-depth camera can include a device that simultaneously captures both color images (RGB) and depth information. The RGB-depth camera can combine the functions of a conventional color camera, which can capture the visual appearance of a scene using red, green, and blue channels, and a depth sensor, which can measure the distance between the camera and objects in the scene. This dual data capture can enable the camera to create a 3D representation of the environment, making it useful for applications such as 3D modeling, robotics, augmented reality, and / or gesture recognition. The RGB-depth camera can be useful in scenarios where both color and spatial structure of a scene can be needed.

[0037] For example, the IMU sensor can include a device that can measure and / or report the specific force, angular rate, and / or magnetic field of a subject using a combination of accelerometers, gyroscopes, and / or magnetometers. The IMU sensor can track the motion and orientation of an object in 3D space, providing data about acceleration, rotation, and sometimes direction. The IMU can be useful in applications that require precise motion tracking and stability, such as in smartphones, drones, virtual reality systems, and / or autonomous vehicles. By integrating this motion data, the IMU can enable a device to navigate, stabilize, and interact with its environment more effectively.

[0038] In Figure 1 The vision system of the wearable robot can include a vision controller 100 and a robot controller 200.

[0039] The vision controller 100 can perform initial calibration between the robot and the sensor (e.g., the IMU sensor) by using the RGB image information obtained through the RGB-depth camera, and can obtain information about the environment around the robot refined in the form of a point cloud.

[0040] For example, a point cloud can include a collection of data points in a three-dimensional coordinate system representing the outer surface of an object or environment. Each point in the cloud can have its own set of X, Y, and Z coordinates and / or additional information (e.g., color or intensity). Point clouds can be commonly generated by 3D scanners, LiDAR, or photogrammetry techniques, and can be used in various applications such as 3D modeling, computer vision, and / or robotics. Point clouds can provide highly detailed and / or accurate representations of complex surfaces and / or structures, making them ideal for tasks such as object recognition, environment mapping, and / or digital reconstruction.

[0041] The vision controller 100 can estimate the height and / or normal of the terrain through a meshed elevation map, which is generated by using depth information obtained through an RGB-depth camera.

[0042] For example, a meshed elevation map can include a representation of a terrain or surface topography that elevation data can be organized in a grid form. Each cell in the grid can correspond to a specific geographic location, and can contain a value representing the elevation at that point. The grid structure can allow for easier processing and / or manipulation of the elevation data, making it useful for tasks such as path planning, terrain analysis, and / or environment simulation.

[0043] The vision controller 100 can extract geometric information including the height, width, and / or slope of the terrain, and based on the information, can perceive a walking environment such as a flat ground, a stair, and a sloped road.

[0044] The vision controller 100 can estimate a coordinate path of a robot foot by utilizing information about a walking environment perceived through the extracted geometric information. The vision controller 100 can calculate coordinates of subsequent positions of a left foot and a right foot of the robot in a terrain such as a flat ground, a sloped road, a stair, etc.

[0045] For example, the vision controller 100 can use NVIDIA Jetson Nano. The vision controller 100 can calculate coordinates of subsequent positions (e.g., subsequent robot foot positions) of the robot foot within 10 fps. The vision controller 100 can detect a 80cm x 80cm area in front of the robot's body ahead.

[0046] The robot controller 200 can control the motion of the robot including walking. The robot can include a wearable robot. The robot can include a medical robot. The robot can not be particularly limited, and can be any robot of various forms, types, and purposes capable of walking.

[0047] The robot controller 200 can include a control module such as a cane for moving the robot according to a user input.

[0048] The robot controller 200 can issue a walking instruction to instruct the robot to move its feet to a preset position by a control module. Here, the preset position can be received from the vision controller 100.

[0049] The robot controller 200 can transmit (e.g., emit) the current state including the current position of the robot feet (e.g., current robot feet position) and the walking instruction to the vision controller 100.

[0050] The robot controller 200 can receive a coordinate path of the robot feet estimated by the vision controller 100, and based on this, can control the robot.

[0051] The vision control device 100 of the wearable robot of the present disclosure can be implemented as a vision controller 100. Hereinafter, the vision controller 100 can be referred to as the vision control device 100 of the wearable robot.

[0052] The vision control device 100 of the wearable robot can include a receiver 110, a geometric information generator 120, an RGB-depth preprocessor 121, an IMU-based point cloud corrector 122, a robot path calculator 130, and / or an emitter 140.

[0053] The receiver 110 can receive the current position of the robot feet from the robot controller 200 (see Figure 1 ). The receiver 110 can receive a walking instruction of the robot from the robot controller 200. The walking instruction can include an instruction to provide a subsequent position of the robot feet (e.g., a subsequent robot feet position).

[0054] The geometric information generator 120 can perceive a terrain by an RGB-depth camera mounted on the robot and generate geometric information based on a point cloud.

[0055] The geometric information generator 120 can generate geometric information including height, width, inclination, etc. related to the perceived terrain based on the point cloud.

[0056] The RGB-depth preprocessor 121 can convert depth information obtained from the RGB-depth camera into a point cloud form.

[0057] The RGB-depth preprocessor 121 can perform point cloud filtering to remove noise from the converted point cloud and adjust the data size.

[0058] The IMU-based point cloud corrector 122 can align the filtered point cloud in the direction of gravity based on an IMU.

[0059] The geometry information generator 120 can set an area of interest and can segment a point cloud aligned within the area of interest into a plurality of grids. The area of interest can be set as an arbitrary area.

[0060] The geometry information generator 120 can calculate an elevation value regarding a terrain corresponding to each grid and can generate an elevation map by combining the calculated elevation values.

[0061] The geometry information generator 120 can generate a grid-based elevation map. The grid can be 80 cm x 80 cm.

[0062] For example, the elevation value can be calculated as an average value of length values (or z-axis values) in a gravity direction of the point cloud input for each grid.

[0063] The geometry information generator 120 can generate a plurality of clusters by clustering the point cloud based on distances between the point cloud in the elevation map and normal vectors (normal values) estimated for each point cloud.

[0064] The geometry information generator 120 can determine whether an angle difference of normal vectors between point clouds adjacent to each other at a certain criterion is within a threshold level.

[0065] The geometry information generator 120 can classify point clouds, for which the angle difference of normal vectors between point clouds adjacent to each other at a certain criterion is determined to be within the threshold level, into the same cluster.

[0066] The geometry information generator 120 can extract geometry information of a terrain from the plurality of clusters, respectively.

[0067] The plurality of clusters can include geometry information of a terrain corresponding to a walking environment of one of flat ground, a stair, an uphill, or a downhill.

[0068] The robot path calculator 130 can calculate a subsequent robot foot position based on a current robot foot position and the calculated geometry information.

[0069] The robot path calculator 130 can calculate a subsequent robot foot position through a subsequent foot position planning (e.g., a next step planning) algorithm of the robot and can update the walking information.

[0070] The transmitter 140 can transmit (e.g., emit) the calculated subsequent robot foot position to the robot controller 200.

[0071] Figure 1 and Figure 2 A flowchart illustrating a robot vision-based walking scenario through a vision system of a wearable robot is shown.

[0072] Figure 3A flowchart showing a walking scenario of a robot in a vision-based walking mode is shown.

[0073] Figure 2 A flowchart showing a walking scenario of a robot in a vision-based walking mode is shown. Figure 3 Figure 2 Differently, steps performed by a vision control device 100 (see Figure 3 ) of a wearable robot implemented as a vision controller and by a robot controller 200 (see Figure 1 ) are shown.

[0074] Figure 1 A vision control method of a wearable robot can be included. Figure 3 The vision control method of a wearable robot can be performed by a vision control device 100 of the wearable robot.

[0075] At step S100, in Figure 3 and Figure 2 , if a walking signal is input by the robot controller 200, the vision system of the wearable robot can send (e.g., transmit) a walking instruction to the vision control device 100 of the wearable robot.

[0076] At step S150, if a user inputs a specific button on the crutch, the robot controller 200 can send (e.g., transmit) the current robot foot position information and the walking instruction to the vision control device 100 of the wearable robot.

[0077] The vision control device 100 of the wearable robot can calculate a subsequent robot foot position based on RGB-depth camera information. That is, at step S200, the vision control device 100 of the wearable robot can perceive a terrain based on RGB information and depth information, generate geometric information about the terrain, and calculate a subsequent robot foot position based on the geometric information.

[0078] At step S250, the vision control device 100 of the wearable robot can send (e.g., transmit) the calculated subsequent robot foot position to the robot controller 200.

[0079] The robot controller 200 can check the received subsequent robot foot position. At step S300, the robot controller 200 can output the received subsequent robot foot position to a display of the robot so that a user can confirm it.

[0080] At step S350, the user can send a walking instruction to the robot through a control module such as a crutch based on the received subsequent robot foot position.

[0081] ​At step S400, if the user inputs a specific button on the crutch, the robot controller 200 can move the robot foot to a subsequent robot foot position.

[0082] Figure 3 A flowchart showing a visual control method of a wearable robot according to an example is shown. Figure 4 A flowchart showing a visual control method of a wearable robot according to an example is shown. Figure 4 and Figure 2 geometric information generation step and a subsequent robot foot position calculation step.

[0083] In Figure 3 At step S210, the visual control device 100 of the wearable robot can convert the depth information of the camera into a point cloud form.

[0084] At step S220, the visual control device 100 of the wearable robot can perform point cloud filtering to remove noise and / or adjust data size.

[0085] At step S230, the visual control device 100 of the wearable robot can align the point cloud along the z-axis based on the IMU, which can be the direction of gravity.

[0086] At step S240, the visual control device 100 of the wearable robot can generate a region of interest (RoI) and / or a grid-based elevation map.

[0087] For example, the RoI can include a specific region within an image, video, or dataset, which can be selected for detailed analysis or processing due to its relevance to the task at hand. The visual control device 100 of the wearable robot can arbitrarily set the RoI, and can segment the aligned point cloud within the region of interest into multiple grids.

[0088] For example, the grid-based elevation map can include a digital representation of a terrain, the surface of which can be divided into grids, each cell in the grid containing an elevation value corresponding to the height of the terrain at that specific location. The grid-based elevation map can provide a structured way of visualizing and / or analyzing the topography of an area, making it useful for applications such as environmental modeling and / or robot navigation. The grid form can allow for easier computation and / or manipulation of elevation data, enabling tasks such as slope analysis, watershed modeling, and / or terrain visualization.

[0089] The visual control device 100 of the wearable robot can calculate the elevation value regarding the terrain corresponding to each grid. The visual control device 100 of the wearable robot can generate an elevation map by combining the calculated elevation value of each grid.

[0090] The elevation value of each mesh can be calculated as an average of the length values of the point clouds input for each mesh in the z-axis direction of gravity.

[0091] In step S250, the wearable robot vision control device 100 can cluster by using the distance between each point cloud in the elevation map and the normal vector.

[0092] The wearable robot vision control device 100 can cluster the point clouds based on the distance between the point clouds in the elevation map and the normal vector estimated for each point cloud, and generate a plurality of clusters.

[0093] The wearable robot vision control device 100 can determine whether the distance between adjacent point clouds is within a certain criterion.

[0094] The wearable robot vision control device 100 can determine whether the angle difference of the normal vectors between the point clouds adjacent to each other whose distance is within the certain criterion is within a threshold level.

[0095] The wearable robot vision control device 100 can classify the adjacent point clouds whose angle difference of the normal vectors is determined to be within the threshold level into the same cluster.

[0096] In step S260, the wearable robot vision control device 100 can extract geometric information from each cluster.

[0097] The wearable robot vision control device 100 can extract geometric information from a plurality of clusters, respectively, the plurality of clusters including geometric information about one of a flat ground, a stair, an uphill, or a downhill.

[0098] In step S270, the wearable robot vision control device 100 can update the walking information according to the geometric information through the next step planning of the robot.

[0099] The wearable robot vision control device 100 can perceive a walking environment based on the geometric information of the terrain, and based on this, calculate the subsequent foot position of the robot through a robot foot position calculation algorithm.

[0100] Figure 4 A graph showing an elevation map generation step according to an example is shown. Figure 5 A graph showing an elevation map generation step according to an example is shown.

[0101] Figure 4 A region of interest (RoI) and a point cloud falling within the region of interest and aligned in the direction of gravity (z-axis alignment) are shown.

[0102] In Figure 5The wearable robotic vision control device 100 can quantize the point of the point cloud into the inside of a grid of a preset size (e.g., 80x80), and can input a length value on the z-axis to each grid to generate a grid map (GM). For example, the grid map can include a representation of a physical space, the area of which can be divided into a uniform grid of cells or squares, each cell corresponding to a specific location in the real world. Each cell in the grid can contain data or attributes such as whether the area is occupied, free, or has a specific feature such as elevation or cost. The simplicity of the grid structure makes it easy to process and / or apply algorithms for movement, exploration, and mapping in 2D and 3D environments. The grid can be represented in 2 dimensions with respect to the X and Y axes.

[0103] The wearable robotic vision control device 100 can calculate the average value of the points input in each grid. The wearable robotic vision control device 100 can calculate an elevation value as a length value in the direction of gravity represented for each grid based on the calculated average value.

[0104] The wearable robotic vision control device 100 can calculate the average value of the height of the points of each grid and represent it as a representative elevation value.

[0105] The wearable robotic vision control device 100 can perform clustering based on the x, y coordinates and the elevation value (z value) of each grid and generate a grid-based elevation map (GEM). For example, the GEM can include a digital representation of a terrain, the surface of which can be divided into a grid, each cell in the grid containing an elevation value corresponding to the height of the terrain at that particular location. The GEM can provide a structured way of visualizing and / or analyzing the topography of an area, making it useful for applications such as environment modeling and / or robot navigation. The grid form can allow for easier calculation and / or manipulation of elevation data, enabling tasks such as slope analysis, watershed modeling, and / or terrain visualization.

[0106] Figure 5 A graph showing a clustering step of a method according to an example is shown. Figure 6

[0107] The wearable robotic vision control device 100 can perform clustering using the distance and normal vector between each point in the elevation map.

[0108] The wearable robotic vision control device 100 can cluster the point cloud by conditional region growing clustering (CRGC).

[0109] For example, the wearable robotic vision control device 100 can determine whether the distance between the point clouds is within a certain threshold (e.g., 1 cm).

[0110] ​The wearable robot's vision control device 100 can compare the normal angle between point clouds and determine whether it is within a certain threshold angle.

[0111] If the normal becomes similar to the ground normal vector, the wearable robot's vision control device 100 can classify it into a cluster CL1.

[0112] Figure 4 A figure showing a geometry information extraction step according to an example is shown. Figure 7

[0113] After clustering (flat ground, stair perception, etc.), the wearable robot's vision control device 100 can extract angle information using the normal information of each cluster through principal component analysis (PCA).

[0114] In Figure 4 , the right lower arrow and the right upper arrow can correspond to the normal to the ground, and the left lower arrow and the left upper arrow can correspond to the normal estimated to be a plane.

[0115] Figure 7 and Figure 8A A figure showing a result provided according to a wearable robot's vision control method according to an example is shown.

[0116] In Figure 8B and Figure 8A , the wearable robot's vision system including the wearable robot's vision control device 100 shows the robot foot position calculation results 81a and 81b predicted by the wearable robot's vision control method (vision algorithm) through a display.

[0117] Figure 8B and Figure 8A RGB images 82a and 82b and depths 83a and 83b obtained through an RGB-depth camera are shown.

[0118] The robot foot position calculation results can include a point cloud PC, a non-terrain recognition part (NS), a right foot next position (NRF), a left foot next position (NLF), and a holder recognition part (HG).

[0119] Figure 8B A point cloud form 84 and a shadow form 85 of the geometry information shown in the display are shown. Figure 8B The geometry information of

[0120] Figure 8B Figure 9 A figure showing a wearable robot according to an example is shown. The wearable robot can be a medical robot. ​

[0121] The vision controller 91 can be installed in an area near the chest of the medical wearable robot.

[0122] Additionally or optionally, the camera 92 can be installed in an area near the chest, and can obtain a field of view (FoV) regarding the terrain, thereby determining a movement path.

[0123] For example, the FoV can refer to an observable area that a camera, sensor, or human eye can capture at any given moment. The FoV can be typically measured as an angle, representing the extent of a scene that can be seen horizontally, vertically, or diagonally. In the case of cameras and sensors, a wider FoV can allow more of the surrounding environment to be captured in a single image or scan, which can be useful for applications such as photography, video recording, virtual reality, and / or autonomous navigation, etc. The FoV can be influenced by various factors such as lens design, sensor size, and / or distance between the observer or device and the object being observed.

[0124] The camera 92 can include an RGB-depth camera. The camera 92 can be installed on the lower side of the vision controller 91.

[0125] The marker 93 includes a marker for initial calibration of the robot and the camera, and can be installed to be attachable and detachable. The marker 93 can be used only for initial calibration and detached since it is not used thereafter.

[0126] A vision control device of a wearable robot can include a receiver configured to receive a current robot foot position from the wearable robot, a geometry information generator configured to perceive a terrain through an RGB-depth camera installed on the wearable robot and generate point cloud-based geometry information, a robot path calculator configured to calculate a subsequent robot foot position based on the current robot foot position and the geometry information, and a transmitter configured to transmit the calculated subsequent robot foot position to the wearable robot.

[0127] The vision control device of the wearable robot can further include an RGB-depth preprocessor configured to convert depth information of the RGB-depth camera into a point cloud form.

[0128] The RGB-depth preprocessor can be configured to perform point cloud filtering to remove noise from the converted point cloud and adjust data size.

[0129] The vision control device of the wearable robot can further include an inertial measurement unit (IMU)-based point cloud corrector configured to align the filtered point cloud in a gravity direction based on an IMU.

[0130] The geometry information generator can be configured to set an area of interest, and segment the point cloud aligned within the area of interest into a plurality of grids, and calculate elevation values about terrains corresponding to each grid, and generate an elevation map by combining the calculated elevation values.

[0131] The elevation values can be calculated as an average of length values in a gravity direction of the input point cloud for each grid.

[0132] The geometry information generator can be configured to generate a plurality of clusters by clustering the point cloud based on distances between the point clouds in the elevation map and normal vectors estimated for each point cloud.

[0133] If an angle difference of normal vectors between point clouds adjacent to each other within a certain criterion is determined to be within a threshold level, the geometry information generator can be configured to classify the respective point clouds for which the angle difference is determined to be within the threshold level into the same cluster.

[0134] The geometry information generator can be configured to extract geometry information of terrains from the plurality of clusters, respectively.

[0135] The plurality of clusters can include geometry information about one of a flat ground, a stair, an uphill, or a downhill.

[0136] A vision control method of a wearable robot can include receiving a current robot foot position from the wearable robot, perceiving a terrain and generating geometry information based on a point cloud through an RGB-depth camera mounted on the wearable robot, calculating a subsequent robot foot position based on the current robot foot position and the geometry information, and transmitting the calculated subsequent robot foot position to the wearable robot.

[0137] Generating the geometry information can include converting depth information of the RGB-depth camera into a point cloud form.

[0138] Generating the geometry information can further include point cloud filtering to remove noise from the converted point cloud and adjust a data size.

[0139] Generating the geometry information can further include aligning the filtered point cloud in a gravity direction based on an inertial measurement unit (IMU).

[0140] Generating the geometry information can further include setting an area of interest, and segmenting the point cloud aligned within the area of interest into a plurality of grids, calculating elevation values about terrains corresponding to each grid, and generating an elevation map by combining the calculated elevation values.

[0141] The elevation values can be calculated as an average of length values in a gravity direction of the input point cloud for each grid.

[0142] Generating the geometry information can include generating a plurality of clusters by clustering the point clouds based on distances between the point clouds in the elevation map and normal vectors estimated for each of the point clouds.

[0143] Generating the plurality of clusters can include determining whether distances between adjacent point clouds are within a particular criterion, determining whether angle differences of normal vectors between point clouds adjacent to each other whose distances are determined to be within the particular criterion are within a threshold level, and classifying respective point clouds whose angle differences are determined to be within the threshold level into a same cluster.

[0144] Generating the geometry information can further include extracting geometry information of a terrain from the plurality of clusters, respectively.

[0145] The plurality of clusters can include geometry information about one of a flat ground, a stair, an uphill, or a downhill.

[0146] While this disclosure has been described in connection with what is presently considered to be the most practical example embodiments, it is to be understood that the disclosure is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A vision control device for a wearable robot, comprising: a transceiver in communication with at least one controller of the wearable robot; at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the vision control device to: receive, via a receiver of the transceiver, an indication of a current robot foot position from the at least one controller of the wearable robot; detect, via a depth camera of the wearable robot, features of a terrain surrounding the wearable robot; generate, based on the detected features of the terrain, point cloud-based geometry information associated with the terrain; determine, based on the current robot foot position and the point cloud-based geometry information, a subsequent robot foot position; and transmit, via a transmitter of the transceiver, the determined subsequent robot foot position to the at least one controller of the wearable robot.

2. The vision control device of claim 1, wherein the at least one processor comprises: a red-green-blue-depth pre-processor (RGB-depth pre-processor) to convert depth information of the depth camera into a point cloud form to generate the point cloud-based geometry information associated with the terrain, the depth camera comprising an RGB-depth camera.

3. The vision control device of claim 2, wherein the RGB-depth pre-processor performs point cloud filtering to remove noise and adjust data size from the converted point cloud form.

4. The vision control device of claim 3, further comprising: an inertial measurement unit-based point cloud corrector (IMU-based point cloud corrector) to align the filtered point cloud in a gravity direction based on one or more measurements of at least one IMU sensor.

5. The vision control device of claim 4, wherein the instructions, when executed by the at least one processor, cause the vision control device to: set an area of interest; align the filtered point cloud within the area of interest; segment the aligned point cloud within the area of interest into a plurality of grids; determine an elevation value for each portion of the terrain corresponding to each grid of the plurality of grids; and generate an elevation map by combining the determined elevation values.

6. The vision control device of claim 5, wherein each of the determined elevation values is determined based on an average of length values associated with the point cloud in the gravity direction, the length values being inputted for each grid of the plurality of grids.

7. The vision control device of claim 5, wherein the instructions, when executed by the at least one processor, cause the vision control device to: generate a plurality of clusters by clustering the point cloud based on distances between the point cloud in the elevation map and estimated normal vectors for each of the point cloud.

8. The vision control device of claim 7, wherein the instructions, when executed by the at least one processor, cause the vision control device to: ​ ​ ​ An angle difference between normal vectors of point clouds that are adjacent to each other based on a distance within a certain criterion is determined to be within a threshold level, and respective point clouds for which the angle difference is determined to be within the threshold level are classified into a same cluster.

9. The visual control device of claim 7, wherein the instructions, when executed by the at least one processor, cause the visual control device to: extract geometry information of the terrain from the plurality of clusters, respectively.

10. The visual control device of claim 9, wherein the plurality of clusters include geometry information about at least one of a flat ground, a stair, an uphill, or a downhill.

11. A control method of a wearable robot, the control method comprising: receiving, from at least one controller of the wearable robot, an indication of a current robot foot position; detecting, via a depth camera of the wearable robot, a feature of a terrain surrounding the wearable robot; generating, based on the detected feature of the terrain, point cloud-based geometry information associated with the terrain; determining, based on the current robot foot position and the point cloud-based geometry information, a subsequent robot foot position; transmitting, to the at least one controller of the wearable robot, the determined subsequent robot foot position; and controlling, based on the determined subsequent robot foot position, the wearable robot.

12. The control method of claim 11, wherein the generating the point cloud-based geometry information comprises: converting depth information of the depth camera into a point cloud form, the depth camera including a red-green-blue-depth camera, i.e., an RGB-depth camera.

13. The control method of claim 12, wherein the generating the point cloud-based geometry information further comprises: point cloud filtering to remove noise and adjust a data size from the converted point cloud form.

14. The control method of claim 13, wherein the generating the point cloud-based geometry information further comprises: aligning the filtered point cloud in a gravity direction based on one or more measurements of at least one inertial measurement unit sensor, i.e., an IMU sensor. The generating the point cloud-based geometry information further comprises:

15. The control method according to claim 14, wherein setting an area of interest; segmenting the aligned filtered point cloud within the area of interest into a plurality of grids; determining an elevation value for each portion of the terrain corresponding to each grid of the plurality of grids; and generating an elevation map by combining the determined elevation values.

16. The control method of claim 15, wherein each of the determined elevation values is determined based on an average of length values associated with the point cloud in the gravity direction, the length values being input for each of the plurality of grids.

17. The control method of claim 15, wherein the generating the point cloud-based geometry information comprises: generating a plurality of clusters by clustering the point cloud based on distances between the point cloud in the elevation map and normal vectors estimated for each of the point cloud.

18. The control method of claim 17, wherein the generating the plurality of clusters comprises: determining that distances between adjacent point clouds are within a certain criterion; ​ determining whether an angle difference of normal vectors between adjacent point clouds is within a threshold level; and classifying respective point clouds for which the angle difference is determined to be within the threshold level into a same cluster.

19. The control method of claim 17, wherein generating geometry information based on the point clouds further comprises: extracting geometry information of the terrain from the plurality of clusters, respectively.

20. The control method of claim 19, wherein the plurality of clusters include geometry information about at least one of a flat ground, a stair, an uphill, or a downhill.

Images