Interaction control method, device and equipment of legged robot and medium

Abstract

The present disclosure provides an interactive control method, device, equipment and medium of a legged robot, comprising: determining local cumulative point cloud data according to pose transformation information and registration point cloud data; judging the passability of the legged robot through the local cumulative point cloud data to obtain a local passable domain; determining low obstacle point cloud data and impassable obstacle point cloud data of the legged robot in the local passable domain according to the local cumulative point cloud data; obtaining a low obstacle representation shape by clustering the low obstacle point cloud data; obtaining an impassable obstacle representation shape by clustering the impassable obstacle point cloud data; and rendering the low obstacle representation shape and the impassable obstacle representation shape into the local passable domain displayed in the first display interface of the handle end of the legged robot in real time. Thus, the user experience is effectively improved.

Description

Interactive control methods, devices, equipment and media for legged robots Technical Field

[0001] The embodiments of this disclosure relate to the field of artificial intelligence technology, and more specifically, to an interactive control method, apparatus, device, and medium suitable for a legged robot. Background Technology

[0002] Legged robots are robots that achieve movement and interaction through a multi-legged limb structure. They typically have multiple independently driven leg actuators and can perform actions such as walking, crossing, and obstacle avoidance in complex terrain environments.

[0003] Legged robots integrate mechanical design, motion control, sensor technology, and artificial intelligence algorithms. They can be equipped with diverse task modules, such as robotic arms and environmental perception units, to meet various practical needs, including inspection, rescue, and scientific research. However, in natural interaction scenarios, legged robots struggle to achieve truly intelligent collaboration and natural interaction, resulting in low efficiency in user control and a poor user experience. Summary of the Invention

[0004] The embodiments described herein provide an interactive control method, apparatus, device, and medium for a legged robot that overcomes the aforementioned problems.

[0005] Firstly, according to the content of this disclosure, an interactive control method for a legged robot is provided, comprising:

[0006] The optimal state of the legged robot is estimated based on laser point cloud data and IMU data to obtain the pose transformation information of the legged robot at the current moment.

[0007] Based on the pose transformation information of the legged robot at the current moment and the registration point cloud data of the legged robot corresponding to the body coordinate system, the local cumulative point cloud data of the legged robot in the surrounding area of ​​the target is determined;

[0008] The legged robot is assessed for navigability using the local accumulated point cloud data to obtain the local navigable area of ​​the legged robot within the surrounding area of ​​the target.

[0009] Based on the local cumulative point cloud data of the legged robot within the target's perimeter, determine the low-lying obstacle point cloud data and impassable obstacle point cloud data of the legged robot within the local passable area;

[0010] The low obstacle point cloud data is clustered to obtain the low obstacle representation shape that wraps the point cloud cluster; and the impassable obstacle point cloud data is clustered to obtain the impassable obstacle representation shape that wraps the point cloud cluster.

[0011] The shapes representing low-lying obstacles and impassable obstacles are rendered in real time onto the partially traversable area displayed on the first display interface of the footed robot's handle, so that the user can interactively control the footed robot within the partially traversable area displayed on the first display interface through the handle.

[0012] Secondly, according to the present disclosure, an interactive control device for a legged robot is provided, comprising:

[0013] The estimation module is used to perform optimal state estimation of the legged robot based on laser point cloud data and IMU data, and obtain the pose transformation information of the legged robot corresponding to the current moment.

[0014] The first determining module is used to determine the local cumulative point cloud data of the legged robot within the target's perimeter based on the pose transformation information of the legged robot at the current moment and the registration point cloud data of the legged robot corresponding to the body coordinate system.

[0015] The judgment module is used to judge the drivability of the legged robot by the local accumulated point cloud data, and obtain the local drivable domain of the legged robot in the area surrounding the target;

[0016] The second determining module is used to determine the low obstacle point cloud data and impassable obstacle point cloud data of the legged robot in the local passable area based on the local cumulative point cloud data of the legged robot in the area surrounding the target.

[0017] The clustering module is used to obtain the low obstacle representation shape that surrounds the point cloud cluster by clustering the low obstacle point cloud data; and to obtain the impassable obstacle representation shape that surrounds the point cloud cluster by clustering the impassable obstacle point cloud data.

[0018] The rendering module is used to render the shapes of the low-lying obstacles and the shapes of the impassable obstacles in real time to the partially passable area displayed on the first display interface of the handle of the legged robot, so that the user can interactively control the legged robot in the partially passable area displayed on the first display interface through the handle.

[0019] Thirdly, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the interactive control method for a legged robot as described in any of the above embodiments.

[0020] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, and when executed by a processor, the computer program implements the steps of the interactive control method for a legged robot as described in any of the above embodiments.

[0021] The interactive control method for a legged robot provided in this application embodiment estimates the optimal state of the legged robot based on laser point cloud data and IMU data to obtain the pose transformation information of the legged robot at the current moment; based on the pose transformation information of the legged robot at the current moment and the registered point cloud data of the legged robot corresponding to the body coordinate system, it determines the local cumulative point cloud data of the legged robot within the target's perimeter; it performs a passability judgment on the legged robot using the local cumulative point cloud data to obtain the local passable domain of the legged robot within the target's perimeter; and it further determines the local traversable domain of the legged robot within the target's perimeter based on the local cumulative point cloud data. Data is used to determine the point cloud data of low-lying obstacles and impassable obstacles within the locally traversable region of the legged robot. The low-lying obstacle point cloud data is clustered to obtain the shape representing the low-lying obstacle that encloses the point cloud clusters; similarly, the impassable obstacle point cloud data is clustered to obtain the shape representing the impassable obstacle that encloses the point cloud clusters. These shapes are then rendered in real-time onto the locally traversable region displayed on the first display interface of the legged robot's handle, allowing the user to interactively control the robot within this region. This process, by accumulating point cloud data to generate traversable areas within the legged robot's movement capabilities, effectively determines the robot's walkable range. By rendering different types of obstacles within this range, the obstacle information is clearly displayed to the user, facilitating real-time and efficient interactive control of the robot within the corresponding walkable area via the handle, thus significantly improving the user experience.

[0022] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein:

[0024] Figure 1 is a flowchart illustrating an interactive control method for a legged robot provided in this disclosure.

[0025] Figure 2 is a schematic diagram of the structure of an interactive control device for a legged robot provided in this disclosure.

[0026] Figure 3 is a schematic diagram of the structure of a computer device provided in this disclosure.

[0027] It should be noted that the elements in the attached diagram are schematic and not drawn to scale. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0029] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, the statement of “connecting” or “coupling” two or more parts together shall mean that these parts are directly joined together or joined through one or more intermediate components.

[0030] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists, A and B exist simultaneously, or B exists. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Terms such as "first" and "second" are only used to distinguish one component (or part of a component) from another component (or another part of a component).

[0032] In the description of this application, unless otherwise stated, "multiple" means two or more (including two), and similarly, "multiple groups" means two or more (including two groups).

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0034] Figure 1 is a flowchart illustrating an interactive control method for a legged robot according to an embodiment of this disclosure. As shown in Figure 1, the specific process of the interactive control method for the legged robot includes:

[0035] S110. Based on the laser point cloud data and IMU data, perform optimal state estimation on the legged robot to obtain the pose transformation information of the legged robot at the current moment; based on the pose transformation information of the legged robot at the current moment and the registered point cloud data of the legged robot corresponding to the body coordinate system, determine the local cumulative point cloud data of the legged robot within the target's surrounding area.

[0036] Specifically, based on 10Hz lidar point cloud data and 200Hz IMU (Inertial Measurement Unit) data, the current state estimation can be iteratively optimized through a tightly coupled IEKF framework to obtain a laser odometry. The laser odometry outputs the pose change information of the legged robot corresponding to the current moment.

[0037] Specifically, 200Hz IMU data is continuously stored in a queue. Based on the optimal state estimate from the previous moment and the current IMU measurement, a rough robot state estimate is calculated through IMU integration. Simultaneously, the error and corresponding covariance matrix are calculated in real time as prior information for subsequent processing. Considering the movement of the legged robot, directly using radar point cloud data would introduce motion distortion and errors. Therefore, this embodiment uses high-frequency IMU prediction to compensate for the radar point cloud in each frame, achieving distortion correction. For example, based on the robot pose predicted by IMU integration, each radar point cloud is converted to the same sampling time. Then, feature extraction is performed from the distortion-corrected point cloud, extracting planar and edge point features, thereby reducing the amount of computational data. The distortion-free point cloud features are matched with the existing global map. For example, the extracted point cloud features are transformed to the global coordinate system through coordinate transformation, and the nearest neighbor points are searched using the global map to calculate the residuals. A maximum a posteriori estimation problem is constructed using all the errors. The optimal state correction is solved by iteratively linearizing the Kalman filter to correct the prior state predicted by the IMU. Finally, the optimal state estimate after radar observation optimization is output, which is the laser odometry.

[0038] Simultaneously, a global map is maintained during the aforementioned iteration process. For example, using the estimated optimal pose, the optimized point cloud of the current frame is transformed to the world coordinate system; after voxel filtering downsampling, it is incrementally added to the global map. Then, based on the pose transformation information provided by the laser odometry and the point cloud registered to the robot's body coordinate system, a local cumulative point cloud within a certain range around the robot can be obtained through methods such as time synchronization, pose transformation, and point cloud filtering. The target's surrounding range can be the global or local range currently scannable by the legged robot.

[0039] In some embodiments, the local cumulative point cloud data of the legged robot within the target's perimeter is determined based on the pose transformation information of the legged robot at the current moment and the registered point cloud data of the legged robot corresponding to the body coordinate system. This includes: synchronizing the pose transformation information of the legged robot at the current moment and the registered point cloud data of the legged robot corresponding to the body coordinate system to obtain the current frame point cloud data and the robot pose of the legged robot at the current moment; calculating the relative coordinate transformation between the robot pose of the legged robot at the current moment and the robot pose at the previous moment; and performing coordinate transformation on the previous frame point cloud data based on the relative coordinate transformation to obtain the historical frame point cloud data of the legged robot; and superimposing the historical frame point cloud data and the current frame point cloud data of the legged robot to obtain the local cumulative point cloud data of the legged robot within the target's perimeter.

[0040] For example, the output of the laser odometry (10 Hz) and the radar point cloud (10 Hz) are synchronized to obtain the point cloud and the corresponding robot pose at the same timestamp, and stored. In the next synchronization time, the relative coordinate transformation between the current robot pose and the pose saved in the previous synchronization time is calculated, and the radar point cloud saved in the previous synchronization time is transformed to the current position through this coordinate transformation to obtain the representation point cloud of the radar point cloud saved in the previous synchronization time at the current position, that is, the historical frame point cloud data.

[0041] Furthermore, considering that historical radar point clouds may contain dynamic obstacles, the current frame point cloud is used to optically trace the historical frame radar point cloud to eliminate dynamic obstacle point clouds in the historical frames. The processed historical frame point cloud is then superimposed with the current frame point cloud, downsampled using voxel filtering, and clipped to a fixed size. The resulting point cloud is also saved and accumulated until the next synchronization time. Through this iterative accumulation process, the accumulated point cloud data within a certain range around the legged robot can be maintained.

[0042] In some embodiments, after performing coordinate transformation on the previous frame point cloud data according to relative coordinate transformation to obtain the historical frame point cloud data of the legged robot, the method further includes: performing voxelization operation on the current frame point cloud data and the historical frame point cloud data at the same resolution; calculating the ray from each point cloud data in the current frame point cloud data to the origin of the lidar, and performing dynamic obstacle point cloud removal on the historical frame point cloud data according to the distance between each ray and the voxel center of the historical frame point cloud.

[0043] The process involves voxelizing historical frame point clouds and current frame point clouds at the same resolution. A ray from each current frame point cloud to the lidar origin is calculated. Each historical frame point cloud voxel traversed by a ray is considered a voxel to be eliminated, and the points within that voxel are then eliminated. The distance from each ray to the voxel center is calculated. If the distance is greater than or equal to a preset threshold, the ray is considered to have struck the voxel at its edge, and the voxel is considered not to need to be eliminated. If the distance is less than the preset threshold, the voxel is considered to need to be eliminated.

[0044] Alternatively, retain one or more point cloud data points from historical frames whose distance from the lidar origin is less than or equal to a set threshold. For example, historical points very close to the lidar are easily penetrated by the ray, resulting in false positives. Therefore, the ray is terminated at a certain distance from the lidar (i.e., the corresponding point cloud is removed) to avoid such false positives.

[0045] S120. By using locally accumulated point cloud data, the legged robot is judged to be passable, and the local passable domain of the legged robot within the target's surrounding area is obtained.

[0046] Compared to wheeled robots, legged robots can traverse complex terrains such as stairs and elevated platforms. Therefore, it is necessary to perform traversability analysis to identify these complex terrains as areas that the robot can walk on. An elevation map can be obtained through cumulative point cloud computing, and traversability can then be determined based on factors such as height differences, ultimately resulting in a local traversable area within a certain range around the robot.

[0047] In some embodiments, the legged robot is assessed for drivability using local accumulated point cloud data to obtain a locally drivable region around the target. This process includes: performing ground segmentation on the local accumulated point cloud data to distinguish between ground point cloud data and non-ground point cloud data; constructing a height map model based on the ground point cloud data; each grid cell in the height map model corresponds to a physical region around the target; traversing each grid cell in the height map model; if there exists a target grid cell whose difference between the maximum and minimum ground height is less than or equal to a drivable height threshold, and the height change rate between the target grid cell and its adjacent grid cells is less than or equal to a slope threshold, then the target grid cell is determined to be a drivable area; and merging all target grid cells to obtain the locally drivable region around the target for the legged robot.

[0048] Among them, ground segmentation processing can adopt a region-growing-based algorithm or a random sample consensus algorithm (RANSAC). By setting seed points and growth conditions, point clouds with similar geometric characteristics (such as normal vectors, slope, etc.) are clustered into ground point cloud data, thereby effectively separating ground and non-ground objects.

[0049] When constructing a height map model, the three-dimensional space surrounding the target can be divided into a regular grid array according to a preset resolution (such as 0.1 m × 0.1 m). Each grid cell records the elevation information of all ground point cloud data within its covered physical area, such as the maximum and minimum ground height values. The difference between the maximum and minimum ground height reflects the flatness of the terrain within a grid cell. When traversing each grid cell in the heightmap model, for each grid cell, the difference between its maximum and minimum ground height is calculated. If this difference is within the passable height threshold (e.g., set to 0.05 meters based on the leg lift height of the legged robot and the ground clearance of the body), it indicates that the terrain inside the grid is relatively flat and initially meets the passability conditions. Further, the height change between the grid cell and its eight adjacent grid cells (or four neighbors selected based on the actual terrain complexity) is determined. The height change rate can be obtained by calculating the ratio of the average height difference between adjacent grid cells to the grid spacing. If this change rate is less than or equal to the preset slope threshold (e.g., set to 30 degrees, the maximum slope that the legged robot can stably climb), it indicates that the transition between the grid cell and the surrounding terrain is smooth, and the robot is less likely to tip over or get stuck during movement. Finally, the grid cell is determined to be a passable area. By sequentially executing the above judgment logic on all grid cells, target grid cells that meet the conditions are selected, and spatially continuous target grid cells are merged to form one or more continuous regions. This constitutes the local traversable area in which the legged robot can safely walk in the current target's surrounding environment. Thus, by digitally analyzing the terrain surrounding the target environment of the legged robot, the area in which the robot can safely move can be accurately identified.

[0050] S130. Based on the local cumulative point cloud data of the legged robot within the target's surrounding area, determine the point cloud data of low obstacles and impassable obstacles within the local passable area of ​​the legged robot.

[0051] Point clouds of low-lying obstacles and impassable obstacles can be obtained through point cloud segmentation, clustering, and edge extraction. For example, local cumulative point cloud data can be transferred to a gravity coordinate system and projected onto a two-dimensional XY plane. A two-dimensional matrix on the XY plane can be constructed according to a certain resolution, and the point cloud height corresponding to each matrix cell can be calculated to construct a height map. Based on the height difference between adjacent matrix cells and the height information calculated for each matrix cell, the point cloud can be divided into impassable obstacle point clouds and low-lying obstacle point clouds.

[0052] In some embodiments, based on the local cumulative point cloud data of the legged robot within the target's surrounding area, the point cloud data of low-lying obstacles and impassable obstacles within the locally passable domain of the legged robot are determined. This includes: dividing the local cumulative point cloud data into ground point cloud data and non-ground point cloud data using a preset height threshold; performing cluster analysis on the non-ground point cloud data to obtain multiple obstacle point cloud clusters; calculating the minimum bounding box height of each obstacle point cloud cluster; if there exists a first obstacle point cloud cluster whose minimum bounding box height is less than or equal to a preset low-lying obstacle height threshold, then the first obstacle point cloud cluster is determined as low-lying obstacle point cloud data; if there exists a second obstacle point cloud cluster whose minimum bounding box height is greater than the preset low-lying obstacle height threshold, then the second obstacle point cloud cluster is determined as impassable obstacle point cloud data.

[0053] The preset height threshold can be dynamically adjusted or pre-set based on the legged robot's structural parameters (such as the minimum leg clearance and the height of the bottom of the robot body from the ground) and typical terrain features. For example, it can be set to 0.1 meters to accurately distinguish between ground points and non-ground points above the ground, thus precisely dividing the ground point cloud data into non-ground point cloud data. When performing cluster analysis on the non-ground point cloud data, a clustering algorithm based on Euclidean distance can be used. By setting an appropriate neighborhood radius and minimum number of cluster points, spatially close non-ground points are aggregated into an obstacle point cloud cluster, thereby dividing the discrete non-ground point cloud into different obstacle entities.

[0054] The minimum bounding box is the smallest cube structure that can completely enclose all points in the obstacle point cloud cluster. The height of the minimum bounding box is the dimension of the cube along the Z-axis in the gravitational coordinate system, i.e., the difference between the Z-coordinate values ​​of the top and bottom faces of the cube. A preset height threshold for low obstacles can be determined based on the obstacle-crossing ability of the legged robot. For example, for a small legged robot, it can be set to 0.3 meters. When the height of the minimum bounding box of an obstacle point cloud cluster is at or below this threshold, it is determined to be a low obstacle that the legged robot can cross by taking steps or adjusting its gait. When the height exceeds this threshold, it is determined to be an impassable obstacle that the legged robot cannot directly cross and must be avoided in subsequent path planning. Thus, low obstacles and impassable obstacles in the interactive environment can be accurately identified.

[0055] S140. The low obstacle representation shape of the wrapped point cloud cluster is obtained by clustering the low obstacle point cloud data; and the impassable obstacle representation shape of the wrapped point cloud cluster is obtained by clustering the impassable obstacle point cloud data.

[0056] This can be achieved by clustering these two types of point clouds separately, grouping nearby point clouds of the same type into the same category; for each cluster of point clouds, by calculating the edges, several corner points of the point cloud cluster are obtained in the form of corner points, and these corner points are connected to obtain the polygon that encloses the point cloud cluster.

[0057] In some embodiments, the low-lying obstacle point cloud data is clustered to obtain the low-lying obstacle representation shape that surrounds the point cloud cluster; and the impassable obstacle point cloud data is clustered to obtain the impassable obstacle representation shape that surrounds the point cloud cluster, including: aggregating low-lying obstacle point cloud data with a spatial distance less than a first preset distance threshold and a density that meets a first preset requirement into independent low-lying obstacle point cloud sub-clusters; for each low-lying obstacle point cloud sub-cluster, constructing a structure that can tightly surround all points in the low-lying obstacle point cloud sub-cluster based on the corner point information of the corresponding point cloud sub-cluster. The geometric shape of the cloud data is used to obtain the low obstacle representation shape corresponding to each low obstacle point cloud sub-cluster; the impassable obstacle point cloud data with a spatial distance less than a second preset distance threshold and a density that meets the second preset requirement are aggregated into independent impassable obstacle point cloud sub-clusters; for each impassable obstacle point cloud sub-cluster, a three-dimensional geometry that can completely enclose all point cloud data in the impassable obstacle point cloud sub-cluster is constructed based on the boundary point information of the corresponding point cloud sub-cluster, thus obtaining the impassable obstacle representation shape corresponding to each impassable obstacle point cloud sub-cluster.

[0058] The first preset distance threshold and the second preset distance threshold can be dynamically adjusted according to the distribution density of obstacles and the perception accuracy of the legged robot in the actual application scenario. For example, in an environment with densely distributed obstacles, the first preset distance threshold and the second preset distance threshold can be appropriately reduced to avoid mis-aggregation of point cloud subclusters of different obstacles; in an environment with sparsely distributed obstacles, the first preset distance threshold and the second preset distance threshold can be appropriately increased to ensure that the discrete point clouds of the same obstacle can be accurately aggregated.

[0059] The density conditions in the first and second preset requirements can be defined by statistically analyzing the number of point cloud data per unit volume. When the number of point cloud data per unit volume is greater than or equal to the corresponding preset density threshold, the point cloud density of that region is determined to meet the requirements, thereby ensuring that the aggregated point cloud subclusters have a sufficient number of point cloud data to accurately reflect the actual shape of the obstacle.

[0060] When acquiring corner information of a point cloud sub-cluster of low-lying obstacles, a corner detection algorithm based on curvature analysis can be used. This algorithm traverses the neighborhood set of each point in the sub-cluster, calculates the curvature value of each point, and identifies points with larger curvature values ​​that meet a preset curvature threshold as corners. Then, the detected corners are connected sequentially according to their spatial relationships to form convex or concave polygons, ensuring that the polygons fit the contour of the low-lying obstacle point cloud sub-cluster to the maximum extent possible.

[0061] When acquiring boundary point information for impassable obstacle point cloud subclusters, the boundary points can be selected by calculating the normal vector of the subcluster and filtering out points where the normal vector direction changes significantly. Alternatively, a region growing algorithm can be used to grow from the edge of the subcluster to determine the range of boundary points. Then, a 3D shape such as a minimum circumscribed cuboid, circumscribed sphere, or convex hull can be selected as the corresponding 3D geometry for the impassable obstacle point cloud subcluster, facilitating the provision of accurate obstacle space occupancy data for path planning of legged robots.

[0062] S150. The shapes representing low obstacles and impassable obstacles are rendered in real time to the local passable area displayed on the first display interface of the handle of the legged robot, so that the user can interactively control the legged robot through the local passable area displayed on the first display interface.

[0063] The partially passable area displayed in the first display interface can intuitively reflect the terrain structure within a certain range around the robot. When the shape of an obstacle is rendered into this area, it can be drawn according to its relative spatial position and size ratio in the real environment. For example, the polygonal outline of a low obstacle can be superimposed on the ground area it actually occupies, while the three-dimensional geometry of an impassable obstacle can be presented in a semi-transparent or specific color highlighting manner to show its three-dimensional occupancy in space.

[0064] When operating the controller, users can clearly perceive the distance and orientation between the legged robot and various obstacles by observing the first display interface. This allows for more accurate judgments when planning the robot's movement path or performing obstacle avoidance actions, such as preventing the robot's feet from stepping into the polygonal area of ​​low obstacles or avoiding the space occupied by the three-dimensional geometry of impassable obstacles.

[0065] In some embodiments, the method further includes: responding to a target following instruction triggered by a user via a handle on a corresponding second display interface, controlling the legged robot to follow the moving target in real time based on the movement information of the moving target; and / or: responding to a robot movement instruction triggered by a user via a handle on a corresponding second display interface, controlling the legged robot to move from the current position to the target position specified in the robot movement instruction according to a preset path.

[0066] In this system, users can long-press on a location in the video, and the legged robot can autonomously navigate to that location. Alternatively, users can activate the target-following function via a button. The video stream will recognize human figures in real time, and clicking on a person will set that person as the follow target, allowing the robot to follow them in real time. Specifically, the position of the follow target is marked in the camera video stream in real time, and the pixel coordinates of the follow target are calculated. The point cloud is projected onto the image coordinate system using the extrinsic parameters of the camera and LiDAR. Using the point cloud data provided by the LiDAR, the robot obtains the specific position information of the corresponding pixel coordinates in the robot's coordinate system, allowing the robot to use this position information as the follow target point for real-time following. Furthermore, users can switch the follow target by clicking on other people during the following process.

[0067] The controller has two display interfaces: a top-down view (the first display) and a video stream interface (the second display). When one interface is the primary display, the other will appear as a small window in the lower right corner. Users can freely switch between the two interfaces by clicking the window. During operation, users can obtain real-time obstacle information about the robot's surroundings through the controller interface, facilitating remote control. Users can long-press to select a point on the controller, and the robot will autonomously navigate to that point. Simultaneously, the controller interface will display the currently planned navigation route and update the robot's surrounding environment in real-time. Users can also cancel tasks or assign new tasks at any time to interrupt the current task.

[0068] In this embodiment, optimal state estimation of the legged robot is performed based on laser point cloud data and IMU data to obtain the pose transformation information of the legged robot at the current moment; based on the pose transformation information of the legged robot at the current moment and the registered point cloud data of the legged robot corresponding to the body coordinate system, the local cumulative point cloud data of the legged robot in the surrounding area of ​​the target is determined; the legged robot is judged to be passable based on the local cumulative point cloud data to obtain the local passable domain of the legged robot in the surrounding area of ​​the target; based on the local cumulative point cloud data of the legged robot in the surrounding area of ​​the target, the legged robot is determined to be passable. The robot uses point cloud data of low-lying obstacles and impassable obstacles within a locally traversable region. The low-lying obstacle point cloud data is clustered to obtain the shape representing the low-lying obstacle that encloses the point cloud clusters; similarly, the impassable obstacle point cloud data is clustered to obtain the shape representing the impassable obstacle that encloses the point cloud clusters. These shapes are then rendered in real-time onto the locally traversable region displayed on the first display interface of the legged robot's handle. This allows the user to interactively control the legged robot within this locally traversable region displayed on the first display interface via the handle. By accumulating point cloud data to generate traversable areas within the legged robot's movement capabilities, the robot's walkable range can be effectively determined. Different types of obstacles are then rendered within this walkable range, clearly displaying obstacle information to the user. This facilitates real-time and efficient interactive control of the legged robot within the corresponding walkable area via the handle, effectively improving the user experience.

[0069] Figure 2 is a schematic diagram of the structure of an interactive control device for a legged robot provided in this embodiment. The interactive control device for the legged robot may include:

[0070] The estimation module 210 is used to perform optimal state estimation of the legged robot based on laser point cloud data and IMU data, and obtain the pose transformation information of the legged robot corresponding to the current moment.

[0071] The first determining module 220 is used to determine the local cumulative point cloud data of the legged robot within the target's perimeter based on the pose transformation information of the legged robot at the current moment and the registration point cloud data of the legged robot corresponding to the body coordinate system.

[0072] The judgment module 230 is used to judge the legged robot's accessibility by using local accumulated point cloud data, and to obtain the local accessible domain of the legged robot within the target's surrounding area.

[0073] The second determining module 240 is used to determine the point cloud data of low obstacles and impassable obstacles in the local passable domain of the legged robot based on the local cumulative point cloud data of the legged robot in the area surrounding the target.

[0074] Clustering module 250 is used to obtain the low obstacle representation shape that wraps the point cloud cluster by clustering the low obstacle point cloud data; and to obtain the impassable obstacle representation shape that wraps the point cloud cluster by clustering the impassable obstacle point cloud data.

[0075] The rendering module 260 is used to render the shapes of low-lying obstacles and impassable obstacles in real time to the local passable area displayed on the first display interface of the handle of the legged robot, so that the user can interactively control the legged robot through the local passable area displayed on the first display interface.

[0076] In this embodiment, optionally, the first determining module 220 is specifically used for:

[0077] The pose transformation information of the legged robot corresponding to the current moment and the registration point cloud data of the legged robot corresponding to the body coordinate system are synchronized in time to obtain the current frame point cloud data and the robot pose of the legged robot corresponding to the current moment; the relative coordinate transformation between the robot pose of the legged robot corresponding to the current moment and the robot pose corresponding to the previous moment is calculated; and the coordinate transformation of the previous frame point cloud data is performed according to the relative coordinate transformation to obtain the historical frame point cloud data of the legged robot; the historical frame point cloud data and the current frame point cloud data of the legged robot are superimposed to obtain the local cumulative point cloud data of the legged robot in the surrounding area of ​​the target.

[0078] In this embodiment, optionally, a removal module is also included.

[0079] The removal module is used to perform voxelization operations at the same resolution on the current frame point cloud data and the historical frame point cloud data; calculate the ray from each point cloud data in the current frame point cloud data to the origin of the lidar, and perform dynamic obstacle point cloud removal on the historical frame point cloud data according to the distance between each ray and the voxel center of the historical frame point cloud; or; retain one or more point cloud data in the historical frame point cloud data whose distance from the lidar origin is less than or equal to a set threshold.

[0080] In this embodiment, optionally, the judgment module 230 is specifically used for:

[0081] Ground segmentation is performed on the local cumulative point cloud data to distinguish between ground point cloud data and non-ground point cloud data; a height map model is constructed based on the ground point cloud data; each grid cell in the height map model corresponds to a physical region within the target's perimeter; each grid cell in the height map model is traversed; if there is a target grid cell whose difference between the maximum and minimum ground height is less than or equal to the passable height threshold, and the height change rate between the target grid cell and its adjacent grid cells is less than or equal to the slope threshold, then the target grid cell is determined to be a passable area; all target grid cells are merged to obtain the local passable domain of the legged robot within the target's perimeter.

[0082] In this embodiment, optionally, the second determining module 240 is specifically used for:

[0083] Local cumulative point cloud data is divided into ground point cloud data and non-ground point cloud data by using a preset height threshold; cluster analysis is performed on the non-ground point cloud data to obtain multiple obstacle point cloud clusters; the height of the minimum bounding box of each obstacle point cloud cluster is calculated; if there is a first obstacle point cloud cluster whose minimum bounding box height is less than or equal to the preset height threshold of low obstacles, then the first obstacle point cloud cluster is identified as low obstacle point cloud data; if there is a second obstacle point cloud cluster whose minimum bounding box height is greater than the preset height threshold of low obstacles, then the second obstacle point cloud cluster is identified as impassable obstacle point cloud data.

[0084] In this embodiment, optionally, the clustering module 250 is specifically used for:

[0085] Point cloud data of low-lying obstacles whose spatial distance is less than a first preset distance threshold and whose density meets the first preset requirement are aggregated into independent low-lying obstacle point cloud subclusters. For each low-lying obstacle point cloud subcluster, a geometric shape that can tightly wrap all point cloud data in the low-lying obstacle point cloud subcluster is constructed based on the corner point information of the corresponding point cloud subcluster, thus obtaining the low-lying obstacle representation shape corresponding to each low-lying obstacle point cloud subcluster. Point cloud data of impassable obstacles whose spatial distance is less than a second preset distance threshold and whose density meets the second preset requirement are aggregated into independent impassable obstacle point cloud subclusters. For each impassable obstacle point cloud subcluster, a three-dimensional geometry that can completely wrap all point cloud data in the impassable obstacle point cloud subcluster is constructed based on the boundary point information of the corresponding point cloud subcluster, thus obtaining the impassable obstacle representation shape corresponding to each impassable obstacle point cloud subcluster.

[0086] In this embodiment, optionally, a control module may also be included.

[0087] The control module is used to respond to a target following instruction triggered by the user via the handle on the corresponding second display interface, and to control the legged robot to follow the moving target in real time according to the motion information of the moving target; and / or, in response to a robot movement instruction triggered by the user via the handle on the corresponding second display interface, to control the legged robot to move from the current position to the target position specified in the robot movement instruction according to a preset path.

[0088] The interactive control device for the legged robot provided in this disclosure can execute the above-described method embodiments. Its specific implementation principle and technical effects can be found in the above-described method embodiments, and will not be repeated here.

[0089] This application also provides a computer device. Please refer to Figure 3 for details. Figure 3 is a basic structural block diagram of the computer device of this embodiment.

[0090] The computer device includes a memory 310 and a processor 320 that are interconnected via a system bus. It should be noted that only a computer device with memory 310 and processor 320 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0091] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0092] The memory 310 includes at least one type of readable storage medium, including non-volatile memory or volatile memory, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. RAM may include static RAM or dynamic RAM. In some embodiments, the memory 310 may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory 310 may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the computer device. Of course, the memory 310 may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the method described above. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

[0093] Processor 320 is typically used to perform overall operations of a computer device. In this embodiment, memory 310 is used to store program code or instructions, including computer operation instructions, and processor 320 is used to execute the program code or instructions stored in memory 310 or process data, such as program code that runs the methods described above.

[0094] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus system can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0095] Another embodiment of this application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads computer-readable program code stored in the computer-readable medium, enabling the processor to execute the functional actions specified in each step or combination of steps in the above method; and to generate means for implementing the functional actions specified in each block or combination of blocks in the block diagram.

[0096] Computer-readable media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any suitable combination thereof, wherein the memory is used to store program code or instructions, the program code including computer operation instructions, and the processor is used to execute the program code or instructions of the above-described methods stored in the memory.

[0097] The definitions of memory and processor can be found in the description of the foregoing computer device embodiments, and will not be repeated here.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0099] In the various embodiments of this application, the functional units or modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0100] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" as described in this application does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims listing several means, several units of these means may be embodied by the same item of hardware. The use of "first," "second," and "third," etc., does not indicate any order and these words should be interpreted as names. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.

[0102] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An interactive control method for a legged robot, characterized in that, include: The optimal state of the legged robot is estimated based on laser point cloud data and IMU data to obtain the pose transformation information of the legged robot at the current moment. Based on the pose transformation information of the legged robot at the current moment and the registered point cloud data of the legged robot corresponding to the body coordinate system, the local cumulative point cloud data of the legged robot within the target's perimeter is determined; the legged robot is assessed for drivability using the local cumulative point cloud data to obtain the local drivable region of the legged robot within the target's perimeter; based on the local cumulative point cloud data of the legged robot within the target's perimeter, the point cloud data of low-lying obstacles and impassable obstacles within the local drivable region are determined; the low-lying obstacle representation shape is obtained by clustering the low-lying obstacle point cloud data to enclose the point cloud clusters; The impassable obstacle point cloud data is clustered to obtain the impassable obstacle representation shape that wraps the point cloud cluster. The low obstacle representation shape and the impassable obstacle representation shape are rendered in real time to the local passable area displayed in the first display interface of the footed robot's handle, so that the user can interactively control the footed robot in the local passable area displayed in the first display interface through the handle.

2. The method according to claim 1, characterized in that, Based on the pose transformation information of the legged robot corresponding to the current moment and the registered point cloud data of the legged robot corresponding to the body coordinate system, the local cumulative point cloud data of the legged robot within the target's perimeter is determined, including: synchronizing the pose transformation information of the legged robot corresponding to the current moment and the registered point cloud data of the legged robot corresponding to the body coordinate system to obtain the current frame point cloud data and the robot pose of the legged robot corresponding to the current moment; calculating the relative coordinate transformation between the robot pose of the legged robot corresponding to the current moment and the robot pose corresponding to the previous moment; and performing coordinate transformation on the previous frame point cloud data according to the relative coordinate transformation to obtain the historical frame point cloud data of the legged robot; and superimposing the historical frame point cloud data and the current frame point cloud data of the legged robot to obtain the local cumulative point cloud data of the legged robot within the target's perimeter.

3. The method according to claim 2, characterized in that, After performing coordinate transformation on the previous frame point cloud data according to the relative coordinate transformation to obtain the historical frame point cloud data of the legged robot, the method further includes: performing voxelization operation at the same resolution on the current frame point cloud data and the historical frame point cloud data; calculating the ray from each point cloud data in the current frame point cloud data to the origin of the lidar, and performing dynamic obstacle point cloud removal on the historical frame point cloud data according to the distance between each ray and the voxel center of the historical frame point cloud; or; retaining one or more point cloud data in the historical frame point cloud data whose distance from the lidar origin is less than or equal to a set threshold.

4. The method according to claim 1, characterized in that, The legged robot is assessed for navigability using the local accumulated point cloud data to determine its local navigable area within the target's perimeter. This process includes: performing ground segmentation on the local accumulated point cloud data to distinguish between ground point cloud data and non-ground point cloud data; constructing a height map model based on the ground point cloud data; each grid cell in the height map model corresponds to a physical region within the target's perimeter; traversing each grid cell in the height map model; if there exists a target grid cell where the difference between the maximum and minimum ground height is less than or equal to a navigable height threshold, and the height change rate between the target grid cell and its adjacent grid cells is less than or equal to a slope threshold, then the target grid cell is determined to be a navigable area; and merging all the target grid cells to obtain the legged robot's local navigable area within the target's perimeter.

5. The method according to claim 1, characterized in that, Based on the local cumulative point cloud data of the legged robot within the target's surrounding area, the point cloud data of low-lying obstacles and impassable obstacles within the locally passable domain of the legged robot are determined, including: dividing the local cumulative point cloud data into ground point cloud data and non-ground point cloud data using a preset height threshold; performing cluster analysis on the non-ground point cloud data to obtain multiple obstacle point cloud clusters; calculating the minimum bounding box height of each obstacle point cloud cluster; if there exists a first obstacle point cloud cluster whose minimum bounding box height is less than or equal to a preset low-lying obstacle height threshold, then the first obstacle point cloud cluster is determined as the low-lying obstacle point cloud data; if there exists a second obstacle point cloud cluster whose minimum bounding box height is greater than the preset low-lying obstacle height threshold, then the second obstacle point cloud cluster is determined as the impassable obstacle point cloud data.

6. The method according to claim 1, characterized in that, The low obstacle point cloud data is clustered to obtain the low obstacle representation shape that wraps the point cloud cluster; The shape of the impassable obstacle representing the point cloud cluster is obtained by clustering the impassable obstacle point cloud data, including: aggregating the low obstacle point cloud data with a spatial distance less than a first preset distance threshold and a density that meets a first preset requirement into independent low obstacle point cloud sub-clusters; For each low-lying obstacle point cloud sub-cluster, a geometric shape is constructed based on the corner point information of the corresponding point cloud sub-cluster to tightly enclose all point cloud data in the low-lying obstacle point cloud sub-cluster, thus obtaining the low-lying obstacle representation shape corresponding to each low-lying obstacle point cloud sub-cluster; the impassable obstacle point cloud data with a spatial distance less than a second preset distance threshold and a density that meets a second preset requirement are aggregated into independent impassable obstacle point cloud sub-clusters; for each impassable obstacle point cloud sub-cluster, a three-dimensional geometry is constructed based on the boundary point information of the corresponding point cloud sub-cluster to completely enclose all point cloud data in the impassable obstacle point cloud sub-cluster, thus obtaining the impassable obstacle representation shape corresponding to each impassable obstacle point cloud sub-cluster.

7. The method according to claim 1, characterized in that, Also includes: In response to a target following instruction triggered by the user via the handle on the corresponding second display interface, the legged robot is controlled to follow the moving target in real time based on the movement information of the moving target; and / or, in response to a robot movement instruction triggered by the user via the handle on the corresponding second display interface, the legged robot is controlled to move from the current position to the target position specified in the robot movement instruction according to a preset path.

8. An interactive control device for a legged robot, characterized in that, include: The estimation module is used to perform optimal state estimation of the legged robot based on laser point cloud data and IMU data, and obtain the pose transformation information of the legged robot corresponding to the current moment. The first determining module is used to determine the local cumulative point cloud data of the legged robot within the target's perimeter based on the pose transformation information of the legged robot at the current moment and the registration point cloud data of the legged robot corresponding to the body coordinate system; the judging module is used to judge the legged robot's drivability based on the local cumulative point cloud data to obtain the local drivable domain of the legged robot within the target's perimeter. The second determining module is used to determine the low obstacle point cloud data and impassable obstacle point cloud data of the legged robot in the local passable domain based on the local cumulative point cloud data of the legged robot in the target surrounding area; the clustering module is used to obtain the low obstacle representation shape of the low obstacle that surrounds the point cloud cluster by clustering the low obstacle point cloud data. The shape of the impassable obstacle representation that encloses the point cloud cluster is obtained by clustering the impassable obstacle point cloud data. The rendering module is used to render the shapes of the low-lying obstacles and the shapes of the impassable obstacles in real time to the partially passable area displayed on the first display interface of the handle of the legged robot, so that the user can interactively control the legged robot in the partially passable area displayed on the first display interface through the handle.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the interactive control method of the legged robot as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the interactive control method for the legged robot as described in any one of claims 1 to 7.

Images