Methods, apparatus, devices, and media for target following by a footed robot

Abstract

The present disclosure provides a method, device, equipment and medium for target following of a foot-type robot, comprising: performing a bounding box representation on a following target to obtain a real-time bounding box; matching a center point coordinate of a real-time detection box with a center point coordinate of the real-time bounding box to obtain a following point coordinate; performing a coordinate transformation on the following point coordinate to obtain a point cloud center coordinate; taking each point in a point cloud in a passable area as a vertex, setting a weight of an edge between the vertices, and constructing a passable point cloud graph; taking a point corresponding to the point cloud center coordinate as a source point and a preset background point as a sink point, identifying a minimum cut from the passable point cloud graph, and obtaining a point cloud clustering cluster; and controlling the foot-type robot to follow the following target in real time through the point cloud clustering cluster corresponding to the following target. Thus, the robot can not only accurately lock the following target, but also autonomously avoid obstacles in a dynamically changing environment, ensuring the stability and safety of the following process.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of artificial intelligence technology, and more specifically, to a method, apparatus, device, and medium suitable for target following by a legged robot. Background Technology

[0002] Legged robots are robots that achieve movement and interaction through a multi-legged limb structure. They typically have multiple independently driven leg actuators and can perform actions such as walking, crossing, and obstacle avoidance in complex terrain environments.

[0003] In related technologies, legged robots, when performing target following tasks, mainly rely on acquired environmental and target information and drive leg movements through preset control algorithms to achieve target tracking. However, existing technical solutions lack robustness and adaptability in complex dynamic environments, leading to significant positional deviations or even following failures during the following process. Summary of the Invention

[0004] The embodiments described herein provide a method, apparatus, device, and medium for target following by a legged robot that overcomes the aforementioned problems.

[0005] In a first aspect, according to the present disclosure, a method for target following by a legged robot is provided, comprising:

[0006] The following target in the previous video frame acquired by the legged robot is represented by a bounding box to obtain the real-time bounding box of the following target;

[0007] Match the center point coordinates of the real-time detection box of the current video frame with the center point coordinates of the real-time bounding box to obtain the coordinates of the following point corresponding to the current video frame;

[0008] The coordinates of the following point are transformed to the radar coordinate system to obtain the coordinates of the point cloud center in the radar coordinate system corresponding to the coordinates of the following point.

[0009] Obtain the passable area of ​​the legged robot in the current video frame; use each point in the point cloud within the passable area as a vertex, and set the weight of the edges between vertices to construct a passable point cloud map;

[0010] Using the point corresponding to the center coordinates of the point cloud as the source point and the preset background point as the sink point, the minimum cut is identified from the passable point cloud map to perform target segmentation on the passable point cloud map, thereby obtaining the point cloud cluster corresponding to the following target;

[0011] The legged robot is controlled to follow the target in real time by using the point cloud clusters corresponding to the target.

[0012] Secondly, according to the present disclosure, a device for target following by a legged robot is provided, comprising:

[0013] The representation module is used to represent the bounding box of the target being followed in the previous video frame acquired by the legged robot, and to obtain the real-time bounding box of the target being followed.

[0014] The matching module is used to match the center point coordinates of the real-time detection box of the current video frame with the center point coordinates of the real-time bounding box to obtain the coordinates of the following point corresponding to the current video frame.

[0015] The conversion module is used to perform coordinate transformation on the coordinates of the following point to convert the coordinates of the following point to the radar coordinate system, so as to obtain the coordinates of the center of the point cloud corresponding to the coordinates of the following point in the radar coordinate system.

[0016] A construction module is used to obtain the passable area of ​​the legged robot in the current video frame; and to construct a passable point cloud map by taking each point in the point cloud within the passable area as a vertex and setting the weight of the edges between the vertices.

[0017] The segmentation module is used to take the point corresponding to the center coordinate of the point cloud as the source point and the preset background point as the sink point, identify the minimum cut from the passable point cloud map, and perform target segmentation on the passable point cloud map to obtain the point cloud cluster corresponding to the following target;

[0018] The control module is used to control the legged robot to follow the target in real time by means of the point cloud cluster corresponding to the target.

[0019] Thirdly, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the legged robot target following method as described in any of the above embodiments.

[0020] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the legged robot target following method as described in any of the above embodiments.

[0021] The target following method for a legged robot provided in this application embodiment represents the target being followed in the previous video frame acquired by the legged robot by representing it with a bounding box, thus obtaining the real-time bounding box of the target being followed; it matches the center point coordinates of the real-time detection box in the current video frame with the center point coordinates of the real-time bounding box to obtain the coordinates of the following point corresponding to the current video frame; it performs coordinate transformation on the following point coordinates to convert them to the radar coordinate system, thus obtaining the center coordinates of the point cloud corresponding to the following point coordinates in the radar coordinate system; it obtains the passable area corresponding to the legged robot in the current video frame; it constructs a passable point cloud map by using each point in the point cloud within the passable area as a vertex and setting the weights of the edges between vertices; it identifies the minimum cut from the passable point cloud map by using the point corresponding to the center coordinates of the point cloud as the source point and the preset background point as the sink point, and divides the passable point cloud map into target clusters to obtain the point cloud clusters corresponding to the target being followed; it controls the legged robot to follow the target in real time by using the point cloud clusters corresponding to the target being followed. In this way, by matching the current frame with the previous frame, the detection box of the object being identified can be stably obtained, avoiding the problem of tracking failure. At the same time, by target segmentation, the point cloud clusters with the highest correlation with the target being followed can be adaptively segmented from the complex point cloud environment, thereby controlling the movement of the legged robot. This allows the robot to accurately lock onto the target being followed in a dynamically changing environment and autonomously avoid obstacles, ensuring the stability and safety of the following process.

[0022] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein:

[0024] Figure 1 This is a flowchart illustrating a method for target following using a legged robot, as disclosed in this publication.

[0025] Figure 2 This is a schematic diagram of the structure of a legged robot target following device provided in this disclosure.

[0026] Figure 3 This is a schematic diagram of the structure of a computer device provided in this disclosure.

[0027] It should be noted that the elements in the attached diagram are schematic and not drawn to scale. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0029] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, the statement of “connecting” or “coupling” two or more parts together shall mean that these parts are directly joined together or joined through one or more intermediate components.

[0030] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists, A and B exist simultaneously, or B exists. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Terms such as "first" and "second" are only used to distinguish one component (or part of a component) from another component (or another part of a component).

[0032] In the description of this application, unless otherwise stated, "multiple" means two or more (including two), and similarly, "multiple groups" means two or more (including two groups).

[0033] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0034] Figure 1 This is a flowchart illustrating a method for target following by a legged robot according to an embodiment of this disclosure, as shown below. Figure 1 As shown, the specific process of the target following method for legged robots includes:

[0035] S110. Represent the target being followed in the previous video frame acquired by the legged robot using a bounding box to obtain the real-time bounding box of the target being followed; match the center point coordinates of the real-time detection box in the current video frame with the center point coordinates of the real-time bounding box to obtain the coordinates of the following point corresponding to the current video frame.

[0036] The legged robot can acquire environmental and target information at the current moment through a wide-angle camera installed in it, thereby obtaining raw image data containing the target being followed. The horizontal field of view of the wide-angle camera effectively reduces blind spots during target following, allowing for earlier detection of changes in the target's position when the legged robot turns or moves across complex terrain.

[0037] In some embodiments, the following involves representing the target in the previous video frame captured by the legged robot with a bounding box to obtain the real-time bounding box of the target. This includes: extracting features and identifying the target in the previous video frame using a preset target detection algorithm to obtain pixel-level position information of the target in the previous video frame; and generating a bounding box that can completely enclose the target based on the pixel-level position information to obtain the real-time bounding box of the target.

[0038] The preset target detection algorithm can be such as the YOLO series. For example, the YOLO algorithm extracts low-level features (such as edges and textures) and high-level semantic features (such as target parts and overall contours) layer by layer from video frame images, and enhances the detection accuracy of small targets or partially occluded targets through a feature fusion mechanism. At the same time, the extracted features are matched with a preset target category model. When the matching confidence exceeds a set threshold (e.g., 0.7), it is determined that there is a following target in the region, and its pixel-level coordinate range is recorded to obtain the pixel-level position information of the following target.

[0039] The pixel-level position information of the target includes the minimum x-coordinate, maximum x-coordinate, minimum y-coordinate, and maximum y-coordinate of the target pixel. These coordinates can be used as the position parameters of the left, right, bottom, and top edges of the bounding box, respectively. This ensures the bounding box closely fits the outer contour of the target, preventing excessive background interference due to an overly large bounding box or missing target portions due to an overly small bounding box.

[0040] In some embodiments, matching the center point coordinates of the real-time detection boxes and the center point coordinates of the real-time bounding boxes in the current video frame to obtain the coordinates of the following point corresponding to the current video frame includes: matching the center point coordinates of each real-time detection box and the center point coordinates of the real-time bounding boxes in the current video frame, calculating the similarity cost between the real-time bounding boxes and each real-time detection box; constructing a cost matrix based on the similarity cost between the real-time bounding boxes and each real-time detection box; solving the cost matrix using the Hungarian algorithm to obtain the optimal match between the real-time bounding boxes and the real-time detection boxes; and determining the center point coordinates of the real-time detection box that successfully matches the real-time bounding box in the optimal matching result as the coordinates of the following point corresponding to the current video frame.

[0041] The similarity cost between the real-time bounding box and the real-time detection box can be determined based on the location cost. The location cost is represented by calculating the Euclidean distance between the center coordinates of the real-time bounding box and the center coordinates of the real-time detection box; the smaller the distance, the lower the location cost, and the closer the real-time bounding box and the real-time detection box are in the image.

[0042] The Hungarian algorithm is a combinatorial optimization algorithm that solves the assignment problem in polynomial time. In this embodiment, the cost matrix, composed of the similarity costs between real-time bounding boxes and each real-time detection box, can be considered a mathematical model of the assignment problem. The matrix rows represent real-time bounding boxes (the "tasks" to be assigned), and the matrix columns represent each real-time detection box (the "resources" for executing tasks). Each element in the matrix corresponds to the "cost" of assigning a real-time detection box to a real-time bounding box. The Hungarian algorithm can gradually find the optimal match in the cost matrix by methods such as finding independent zero elements or alternating paths, thereby determining the real-time detection box with the lowest similarity cost (i.e., the highest overall matching degree) with the real-time bounding box. Then, the center point coordinates of this real-time detection box are used as the following point coordinates to achieve stable tracking of the target.

[0043] S120. Perform coordinate transformation on the following point coordinates to convert the following point coordinates to the radar coordinate system, and obtain the center coordinates of the point cloud in the radar coordinate system corresponding to the following point coordinates.

[0044] For example, the two-dimensional coordinates of the following point in the image coordinate system are ( u , v ), combined with the camera intrinsic parameter matrix ( u , v Transform from the image coordinate system to the camera coordinate system to obtain three-dimensional coordinates. Xc , Yc , Zc Then, using the extrinsic parameter matrix between the camera and the radar, ( Xc , Yc , ZcFurther transformation to the radar coordinate system yields the coordinates of the point cloud center. Xr , Yr , Zr This facilitates the effective elimination of spatial deviations between different sensor coordinate systems.

[0045] S130. Obtain the passable area of ​​the legged robot in the current video frame; use each point in the point cloud within the passable area as a vertex, and set the weight of the edges between vertices to construct the passable point cloud map.

[0046] Each point in the point cloud can be represented as a vertex in the graph, and edges are added between vertices. The weight of the edges is set based on the similarity between the points. The greater the similarity, the closer the two points are in spatial location, and the weight value of the edges is set to a larger value accordingly.

[0047] In some embodiments, obtaining the passable area corresponding to the legged robot in the current video frame includes: determining the local cumulative point cloud of the legged robot in the current field of view based on the pose transformation information of the legged robot at the current position point and the registration point cloud of the legged robot corresponding to the body coordinate system; and performing passability determination on the legged robot through the local cumulative point cloud to obtain the passable area corresponding to the legged robot in the current video frame.

[0048] The pose transformation information of the legged robot at its current position can be obtained by performing optimal state estimation of the legged robot based on laser point cloud and IMU data. For example, based on 10Hz laser radar point cloud and 200Hz IMU (Inertial Measurement Unit) data, the current state estimation is iteratively optimized through a tightly coupled IEKF framework to obtain a laser odometry, which outputs the pose transformation information of the legged robot corresponding to the current moment. Specifically, the 200Hz IMU data is continuously stored in a queue, and based on the optimal state estimation of the previous moment and the current IMU measurement value, a rough robot state estimate is calculated through IMU integration. At the same time, the error and the corresponding covariance matrix are calculated in real time as prior information for subsequent processing. Considering that the legged robot itself is in motion, directly using the radar point cloud will introduce motion distortion and errors. Therefore, this embodiment uses high-frequency IMU prediction to compensate for the radar point cloud of each frame to achieve distortion correction. For example, based on the robot pose predicted by IMU integration, each radar point cloud is transformed to the same sampling time. Then, feature extraction is performed from the distortion-free point cloud, extracting planar and edge point features to reduce the amount of computational data. The distortion-free point cloud features are matched with an existing global map. For example, the extracted point cloud features are transformed to the global coordinate system using coordinate transformation, and the residual is calculated by searching for nearest neighbors using the global map. A maximum a posteriori estimation problem is constructed using all errors, and an optimal state correction is solved through iterative Kalman filtering and repeated linearization to correct the prior state predicted by the IMU. Finally, the optimal state estimate optimized by radar observation is output, i.e., the laser odometry.

[0049] Simultaneously, a global map is maintained during the aforementioned iteration process. For example, using the estimated optimal pose, the optimized point cloud of the current frame is transformed to the world coordinate system; after voxel filtering downsampling, it is incrementally added to the global map. Subsequently, based on the pose transformation information provided by the laser odometry and the point cloud registered to the robot's body coordinate system, the local cumulative point cloud of the robot within the current field of view can be obtained through methods such as time synchronization, pose transformation, and point cloud filtering.

[0050] When determining the local cumulative point cloud, the current frame point cloud and the robot pose at the current moment can be obtained by synchronizing the pose transformation information of the legged robot corresponding to the current moment and the registered point cloud of the legged robot corresponding to the body coordinate system. The relative coordinate transformation between the robot pose at the current moment and the robot pose at the previous moment can be calculated. Based on the relative coordinate transformation, the previous frame point cloud can be transformed to obtain the historical frame point cloud of the legged robot. The historical frame point cloud and the current frame point cloud are then superimposed to obtain the local cumulative point cloud of the legged robot within the current field of view. For example, the output of the laser odometry (10 Hz) and the radar point cloud (10 Hz) can be synchronized to obtain the point cloud and the corresponding robot pose at the same timestamp, and then stored. In the next synchronization time, the relative coordinate transformation between the current robot pose and the pose saved in the previous synchronization time is calculated, and the radar point cloud saved in the previous synchronization time is transformed to the current position through this coordinate transformation to obtain the representation point cloud of the radar point cloud saved in the previous synchronization time at the current position, that is, the historical frame point cloud.

[0051] Furthermore, considering that historical radar point clouds may contain dynamic obstacles, the current frame point cloud is used to optically trace the historical frame radar point cloud to eliminate dynamic obstacle point clouds in the historical frames. The processed historical frame point cloud is then superimposed with the current frame point cloud, downsampled using voxel filtering, and clipped to a fixed size. The resulting point cloud is also saved and accumulated until the next synchronization time. Through this iterative accumulation process, the local accumulated point cloud of the legged robot within the current field of view can be maintained.

[0052] After performing coordinate transformation on the previous frame point cloud according to the relative coordinate transformation to obtain the historical frame point cloud of the legged robot, the process also includes: performing voxelization operation on the current frame point cloud and the historical frame point cloud at the same resolution; calculating the ray from each point cloud in the current frame point cloud to the origin of the lidar, and performing dynamic obstacle point cloud removal on the historical frame point cloud based on the distance between each ray and the voxel center of the historical frame point cloud.

[0053] The process involves voxelizing historical frame point clouds and current frame point clouds at the same resolution. A ray is calculated from each current frame point cloud to the lidar origin. Historical frame point cloud voxels traversed by each ray are considered voxels to be eliminated, and the point clouds within these voxels are removed. The distance from each ray to the voxel center is calculated. If the distance is greater than or equal to a preset threshold, the ray is considered to have struck the voxel at its edge, and the voxel does not need to be removed. If the distance is less than the preset threshold, the voxel needs to be removed. Alternatively, one or more point cloud data points from historical frames whose distance from the lidar origin is less than or equal to a set threshold are retained. For example, historical points very close to the lidar are easily traversed by rays, leading to false positives. Therefore, the ray is terminated at a certain distance from the lidar (i.e., the corresponding point cloud is removed) to avoid such false positives.

[0054] When determining the passable area of ​​a legged robot in the current video frame, ground segmentation can be performed on the local cumulative point cloud to distinguish between ground point clouds and non-ground point clouds; a height map model is then constructed based on the ground point cloud; each grid cell in the height map model corresponds to a physical region within the current field of view; each grid cell in the height map model is traversed; if there is a target grid cell whose difference between the maximum and minimum ground height is less than or equal to the passable height threshold, and the height change rate between the target grid cell and its adjacent grid cells is less than or equal to the slope threshold, then the target grid cell is determined to be a passable area; all target grid cells are merged to obtain the local passable domain of the legged robot in the current video frame.

[0055] Among them, the ground segmentation process can adopt a region-growing algorithm or a random sample consensus algorithm (RANSAC). By setting seed points and growth conditions, point clouds with similar geometric characteristics (such as normal vectors, slope, etc.) are clustered into ground point clouds, thereby effectively identifying non-ground objects.

[0056] When constructing a height map model, the three-dimensional space within the current field of view can be divided into a regular grid array according to a preset resolution (such as 0.1 m × 0.1 m). Each grid cell records the elevation information of all ground point clouds within its covered physical area, such as the maximum and minimum ground height values. The difference between the maximum and minimum ground height reflects the flatness of the terrain within a grid cell. When traversing each grid cell in the heightmap model, for each grid cell, the difference between its maximum and minimum ground height is calculated. If this difference is within the passable height threshold (e.g., set to 0.05 meters based on the leg lift height of the legged robot and the ground clearance of the body), it indicates that the terrain inside the grid is relatively flat and initially meets the passability conditions. Further, the height change between the grid cell and its eight adjacent grid cells (or four neighbors selected based on the actual terrain complexity) is determined. The height change rate can be obtained by calculating the ratio of the average height difference between adjacent grid cells to the grid spacing. If this change rate is less than or equal to the preset slope threshold (e.g., set to 30 degrees, the maximum slope that the legged robot can stably climb), it indicates that the transition between the grid cell and the surrounding terrain is gentle, and the legged robot is less likely to tip over or get stuck during movement. Finally, the grid cell is determined to be a passable area. By sequentially executing the above judgment logic on all grid cells, target grid cells that meet the conditions are selected, and spatially continuous target grid cells are merged to form one or more continuous regions, which constitute the local traversable domain in which the legged robot can safely walk within its current field of vision. Therefore, by digitally analyzing the terrain of the legged robot's current field of vision, the area in which the robot can safely move can be accurately identified.

[0057] S140. Using the point corresponding to the center coordinates of the point cloud as the source point and the preset background point as the sink point, identify the minimum cut from the passing point cloud map to divide the passing point cloud map into target clusters, and obtain the point cloud clusters corresponding to the target.

[0058] In this model, the source point represents the foreground point, and the sink point represents the background point. By performing target partitioning on the passable point cloud using the minimum cut, the passable point cloud can be divided into two disjoint sets: the set connected to the source point (points belonging to the target, i.e., point cloud clusters) and the set connected to the sink point (points not belonging to the target).

[0059] In some embodiments, the point corresponding to the center coordinates of the point cloud is used as the source point, and a preset background point is used as the sink point. The minimum cut is identified from the passable point cloud graph to perform target partitioning on the passable point cloud graph and obtain the point cloud cluster corresponding to the target. This includes: using the point corresponding to the center coordinates of the point cloud as the source point and connecting the source point to all foreground points in the passable point cloud graph; using the preset background point as the sink point and connecting the sink point to all background points in the passable point cloud graph to obtain a directed graph corresponding to the passable point cloud graph; solving the maximum flow from the source point to the sink point in the directed graph using the maximum flow algorithm to obtain the minimum cut; and performing target partitioning on the passable point cloud graph using the minimum cut to obtain the point cloud cluster corresponding to the target.

[0060] The connection weight between the source point and the foreground point can be determined based on the spatial distance or feature similarity between the foreground point and the center coordinates of the point cloud. The closer the distance or the higher the similarity, the greater the connection weight; the farther the distance or the lower the similarity, the smaller the connection weight. The connection weight between the sink point and the background point can be set to a fixed value or dynamically adjusted according to the confidence level of the background point to enhance the ability to distinguish the background region.

[0061] When solving for the maximum flow from the source to the sink in a directed graph using the maximum flow algorithm, the optimal flow path from the source to the sink can be found based on the weights of each edge. When the algorithm converges, the two sets separated by the minimum cut can accurately reflect the division between the following target and the background region, thus enabling the point cloud cluster to completely contain the point cloud information of the following target.

[0062] S150. By following the point cloud clusters corresponding to the target, the legged robot is controlled to follow the target in real time.

[0063] In some embodiments, the legged robot is controlled to follow the target in real time by clustering the point cloud corresponding to the target. This includes: calculating the real-time position coordinates and attitude information of the target in three-dimensional space based on the point cloud clusters; using the real-time position coordinates and attitude information as the following reference benchmark for the legged robot; and adjusting the motion parameters of each joint of the legged robot in real time through a pre-generated target following path to drive the legged robot to follow the target in real time.

[0064] The real-time location coordinates are determined by the average of the three-dimensional coordinates of all points within the point cloud cluster. For example, each point in the point cloud cluster is traversed to obtain its corresponding coordinates. X , Y , Z Three-dimensional coordinate values, for all points X Coordinate values Y Coordinate values Z The arithmetic mean of the coordinate values ​​is used to obtain the three average values, which are the real-time position coordinates of the target in three-dimensional space, making it easy to accurately reflect the overall spatial position center of the target.

[0065] Attitude information is used to describe the orientation vector obtained from the principal component analysis results of point cloud clusters. For example, by performing principal component analysis on point cloud data within a point cloud cluster, calculating the covariance matrix and solving for its eigenvalues ​​and eigenvectors, the eigenvector corresponding to the largest eigenvalue represents the main direction of the point cloud cluster, i.e., the orientation vector of the target being followed. This facilitates a precise description of the target's attitude in three-dimensional space, such as the target's frontal orientation or walking direction.

[0066] The control system of the legged robot can adjust the motion parameters such as the angle, angular velocity, and angular acceleration of each joint, such as the hip, knee, and ankle joints, in real time according to the deviation between the following reference and the current position and posture of the legged robot. By controlling the output torque of each joint actuator, the control system drives the legs of the legged robot to make corresponding steps, turns, speed adjustments, and other actions, thereby achieving stable and accurate real-time following of the target. This ensures that the legged robot can flexibly avoid obstacles and always maintain a reasonable relative position with the target during the following process.

[0067] In some embodiments, the method further includes: during the process of controlling the legged robot to follow the target in real time, displaying the real-time bounding box of the target and the real-time pose information of the target on the handle end; when the real-time bounding box is detected to exceed the preset safe field of view, generating a corresponding warning prompt, so as to provide risk warning to the operator of the legged robot by means of audible and visual alarm or sending a warning signal to the control terminal.

[0068] The system distinguishes different target states by setting different colored real-time bounding boxes on the handle. For example, when the target is within the normal following range, the real-time bounding box is green; when the target approaches the edge of the safe field of vision, it is yellow as a pre-warning; and when the real-time bounding box exceeds the preset safe field of vision, it is red, accompanied by vibration of the handle and a rapid beeping of the buzzer, forming a multi-dimensional audible and visual alarm. Simultaneously, the handle's display screen automatically displays specific warning information, such as "Target is about to leave the field of vision, please adjust the robot's position" or "Target has exceeded the safe field of vision," and simultaneously sends this warning signal to the operator's control terminal (such as a compatible tablet or smartphone app) via a wireless communication module, preventing interruptions to the following task or accidents such as collisions due to target loss.

[0069] In this embodiment, the following target in the previous video frame acquired by the legged robot is represented by a bounding box to obtain the real-time bounding box of the following target; the center point coordinates of the real-time detection box in the current video frame are matched with the center point coordinates of the real-time bounding box to obtain the coordinates of the following point corresponding to the current video frame; the coordinates of the following point are transformed to the radar coordinate system to obtain the center coordinates of the point cloud in the radar coordinate system corresponding to the following point coordinates; the passable area corresponding to the legged robot in the current video frame is obtained; each point in the point cloud in the passable area is used as a vertex, and the weights of the edges between vertices are set to construct a passable point cloud map; the point corresponding to the center coordinates of the point cloud is used as the source point, and the preset background point is used as the sink point, and the minimum cut is identified from the passable point cloud map to divide the passable point cloud map into target clusters to obtain the point cloud clusters corresponding to the following target; the legged robot is controlled to follow the following target in real time through the point cloud clusters corresponding to the following target. In this way, by matching the current frame with the previous frame, the detection box of the object being identified can be stably obtained, avoiding the problem of tracking failure. At the same time, by target segmentation, the point cloud clusters with the highest correlation with the target being followed can be adaptively segmented from the complex point cloud environment, thereby controlling the movement of the legged robot. This allows the robot to accurately lock onto the target being followed in a dynamically changing environment and autonomously avoid obstacles, ensuring the stability and safety of the following process.

[0070] Figure 2 This is a schematic diagram of a target-following device for a legged robot provided in this embodiment. The target-following device for a legged robot may include:

[0071] The representation module 210 is used to represent the bounding box of the target being followed in the previous video frame acquired by the legged robot, so as to obtain the real-time bounding box of the target being followed.

[0072] The matching module 220 is used to match the center point coordinates of the real-time detection box and the center point coordinates of the real-time bounding box of the current video frame to obtain the coordinates of the following point corresponding to the current video frame.

[0073] The conversion module 230 is used to perform coordinate transformation on the coordinates of the following point to convert the coordinates of the following point to the radar coordinate system, so as to obtain the coordinates of the center of the point cloud in the radar coordinate system corresponding to the coordinates of the following point.

[0074] The construction module 240 is used to obtain the passable area of ​​the legged robot in the current video frame; using each point in the point cloud within the passable area as a vertex and setting the weight of the edges between vertices, a passable point cloud map is constructed.

[0075] The segmentation module 250 is used to take the point corresponding to the center coordinate of the point cloud as the source point and the preset background point as the sink point, identify the minimum cut from the passing point cloud map, and perform target segmentation on the passing point cloud map to obtain the point cloud cluster corresponding to the target.

[0076] The control module 260 is used to control the legged robot to follow the target in real time by clustering the point cloud corresponding to the target.

[0077] In this embodiment, optionally, the matching module 220 is specifically used for:

[0078] The center point coordinates of each real-time detection box in the current video frame are matched with the center point coordinates of the real-time bounding box, and the similarity cost between the real-time bounding box and each real-time detection box is calculated. A cost matrix is ​​constructed based on the similarity cost between the real-time bounding box and each real-time detection box. The cost matrix is ​​solved using the Hungarian algorithm to obtain the optimal match between the real-time bounding box and the real-time detection box. The center point coordinates of the real-time detection box that successfully matches the real-time bounding box in the optimal matching result are determined as the coordinates of the following point corresponding to the current video frame.

[0079] In this embodiment, optionally, the construction module 240 is specifically used for:

[0080] Based on the pose transformation information of the legged robot at its current position and the registration point cloud of the legged robot corresponding to the body coordinate system, the local cumulative point cloud of the legged robot within the current field of view is determined; the legged robot is then judged to be passable using the local cumulative point cloud, and the passable area corresponding to the legged robot in the current video frame is obtained.

[0081] In this embodiment, optionally, the partitioning module 250 is specifically used for:

[0082] The point corresponding to the center coordinates of the point cloud is taken as the source point, and the source point is connected to all foreground points in the passable point cloud map. The preset background point is taken as the sink point, and the sink point is connected to all background points in the passable point cloud map to obtain the directed graph corresponding to the passable point cloud map. The maximum flow algorithm is used to solve the maximum flow from the source point to the sink point in the directed graph to obtain the minimum cut. The passable point cloud map is then divided into target clusters using the minimum cut to obtain the point cloud clusters corresponding to the target.

[0083] In this embodiment, optionally, the control module 260 is specifically used for:

[0084] The real-time position coordinates and attitude information of the target in three-dimensional space are calculated based on point cloud clustering. The real-time position coordinates are determined by the mean of the three-dimensional coordinates of all points in the point cloud cluster, and the attitude information is used to describe the orientation vector obtained from the principal component analysis results of the point cloud cluster. The real-time position coordinates and attitude information are used as the following reference benchmark for the legged robot, and the motion parameters of each joint of the legged robot are adjusted in real time through the pre-generated target following path to drive the legged robot to follow the target in real time.

[0085] In this embodiment, optionally, the representation module 210 is specifically used for:

[0086] The previous video frame is used to extract features and identify targets by using a preset target detection algorithm to obtain the pixel-level position information of the following target in the previous video frame; a bounding box that can completely wrap the following target is generated based on the pixel-level position information to obtain the real-time bounding box of the following target.

[0087] In this embodiment, optionally, a detection module may also be included.

[0088] The detection module is used to display the real-time bounding box and pose information of the target in real time via the handle during the process of controlling the legged robot to follow the target. When the real-time bounding box is detected to be outside the preset safe field of view, a corresponding warning prompt is generated to provide risk warning to the operator of the legged robot through audible and visual alarms or by sending a warning signal to the control terminal.

[0089] The legged robot target following device provided in this disclosure can execute the above method embodiments. Its specific implementation principle and technical effects can be found in the above method embodiments, and will not be repeated here.

[0090] This application also provides a computer device. Please refer to the following for details. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0091] The computer device includes a memory 310 and a processor 320 that are interconnected via a system bus. It should be noted that only a computer device with memory 310 and processor 320 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0092] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0093] The memory 310 includes at least one type of readable storage medium, including non-volatile memory or volatile memory, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. RAM may include static RAM or dynamic RAM. In some embodiments, the memory 310 may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory 310 may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the computer device. Of course, the memory 310 may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the method described above. In addition, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

[0094] Processor 320 is typically used to perform overall operations of a computer device. In this embodiment, memory 310 is used to store program code or instructions, including computer operation instructions, and processor 320 is used to execute the program code or instructions stored in memory 310 or process data, such as program code that runs the methods described above.

[0095] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus system can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0096] Another embodiment of this application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads computer-readable program code stored in the computer-readable medium, enabling the processor to execute the functional actions specified in each step or combination of steps in the above method; and to generate means for implementing the functional actions specified in each block or combination of blocks in the block diagram.

[0097] Computer-readable media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any suitable combination thereof, wherein the memory is used to store program code or instructions, the program code including computer operation instructions, and the processor is used to execute the program code or instructions of the above-described methods stored in the memory.

[0098] The definitions of memory and processor can be found in the description of the foregoing computer device embodiments, and will not be repeated here.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0100] In the various embodiments of this application, the functional units or modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" as described in this application does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims listing several means, several units of these means may be embodied by the same item of hardware. The use of "first," "second," and "third," etc., does not indicate any order and these words should be interpreted as names. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.

[0103] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for target following by a legged robot, characterized in that, include: The following target in the previous video frame acquired by the legged robot is represented by a bounding box to obtain the real-time bounding box of the following target; Match the center point coordinates of the real-time detection box of the current video frame with the center point coordinates of the real-time bounding box to obtain the coordinates of the following point corresponding to the current video frame; The coordinates of the following point are transformed to the radar coordinate system to obtain the coordinates of the point cloud center in the radar coordinate system corresponding to the coordinates of the following point. Obtain the passable area of ​​the legged robot in the current video frame; use each point in the point cloud within the passable area as a vertex, and set the weight of the edges between vertices to construct a passable point cloud map; Using the point corresponding to the center coordinates of the point cloud as the source point and the preset background point as the sink point, the minimum cut is identified from the passable point cloud map to perform target segmentation on the passable point cloud map, thereby obtaining the point cloud cluster corresponding to the following target; The legged robot is controlled to follow the target in real time by using the point cloud clusters corresponding to the target.

2. The method according to claim 1, characterized in that, Matching the center point coordinates of the real-time detection bounding box of the current video frame with the center point coordinates of the real-time bounding box to obtain the coordinates of the following point corresponding to the current video frame includes: Match the center point coordinates of each real-time detection box in the current video frame with the center point coordinates of the real-time bounding box, and calculate the similarity cost between the real-time bounding box and each real-time detection box. A cost matrix is ​​constructed based on the similarity cost between the real-time bounding box and each of the real-time detection boxes; and the cost matrix is ​​solved using the Hungarian algorithm to obtain the optimal match between the real-time bounding box and the real-time detection box. The coordinates of the center point of the real-time detection box that successfully matches the real-time bounding box in the optimal matching result are determined as the coordinates of the following point corresponding to the current video frame.

3. The method according to claim 1, characterized in that, Obtaining the passable area of ​​the legged robot in the current video frame includes: Based on the pose transformation information of the legged robot at the current position and the registration point cloud of the legged robot corresponding to the body coordinate system, the local cumulative point cloud of the legged robot in the current field of view is determined. The legged robot is assessed for passability using the local cumulative point cloud, thus obtaining the passable area corresponding to the legged robot in the current video frame.

4. The method according to claim 1, characterized in that, Using the point corresponding to the center coordinates of the point cloud as the source point and the preset background point as the sink point, the minimum cut is identified from the traffic point cloud map to perform target segmentation on the traffic point cloud map, resulting in the point cloud cluster corresponding to the following target, including: The point corresponding to the center coordinates of the point cloud is taken as the source point, and the source point is connected to all the foreground points in the passage point cloud map. The preset background point is taken as the sink point, and the sink point is connected to all the background points in the passage point cloud map to obtain the directed graph corresponding to the passage point cloud map. The maximum flow from the source to the sink in the directed graph is solved by the maximum flow algorithm to obtain the minimum cut; and the minimum cut is used to divide the passing point cloud into target clusters to obtain the point cloud clusters corresponding to the following targets.

5. The method according to claim 1, characterized in that, By controlling the legged robot to follow the target in real time through the point cloud clusters corresponding to the target, including: The real-time position coordinates and attitude information of the following target in three-dimensional space are calculated based on the point cloud clusters; the real-time position coordinates are determined by the mean of the three-dimensional coordinates of all points in the point cloud clusters, and the attitude information is used to describe the orientation vector obtained from the principal component analysis results of the point cloud clusters. The real-time position coordinates and the posture information are used as the following reference benchmark for the legged robot, and the motion parameters of each joint of the legged robot are adjusted in real time through a pre-generated target following path to drive the legged robot to follow the target in real time.

6. The method according to claim 1, characterized in that, The following target in the previous video frame acquired by the legged robot is represented by a bounding box to obtain the real-time bounding box of the target, including: The previous video frame is subjected to feature extraction and target recognition by a preset target detection algorithm to obtain the pixel-level position information of the target following the previous video frame. Based on the pixel-level location information, a bounding box that can completely enclose the following target is generated to obtain the real-time bounding box of the following target.

7. The method according to claim 1, characterized in that, Also includes: During the process of controlling the legged robot to follow the target in real time, the real-time bounding box of the target and the real-time pose information of the target are displayed on the handle end. When the real-time bounding box is detected to be outside the preset safe field of view, a corresponding warning is generated to provide a risk warning to the operator of the legged robot through an audible and visual alarm or by sending a warning signal to the control terminal.

8. A device for target following in a legged robot, characterized in that, include: The representation module is used to represent the bounding box of the target being followed in the previous video frame acquired by the legged robot, and to obtain the real-time bounding box of the target being followed. The matching module is used to match the center point coordinates of the real-time detection box of the current video frame with the center point coordinates of the real-time bounding box to obtain the coordinates of the following point corresponding to the current video frame. The conversion module is used to perform coordinate transformation on the coordinates of the following point to convert the coordinates of the following point to the radar coordinate system, so as to obtain the coordinates of the center of the point cloud corresponding to the coordinates of the following point in the radar coordinate system. A construction module is used to obtain the passable area of ​​the legged robot in the current video frame; and to construct a passable point cloud map by taking each point in the point cloud within the passable area as a vertex and setting the weight of the edges between the vertices. The segmentation module is used to take the point corresponding to the center coordinate of the point cloud as the source point and the preset background point as the sink point, identify the minimum cut from the passable point cloud map, and perform target segmentation on the passable point cloud map to obtain the point cloud cluster corresponding to the following target; The control module is used to control the legged robot to follow the target in real time by means of the point cloud cluster corresponding to the target.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the target following method of the legged robot as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the method for target following of a legged robot as described in any one of claims 1 to 7.

Images