Quadruped robot motion information determination method and apparatus, and quadruped robot

Abstract

The present disclosure relates to a quadruped robot motion information determination method and device and a quadruped robot. The method comprises: acquiring data collected by a multi-modal sensor, wherein the multi-modal sensor comprises a binocular camera and a depth camera, the binocular camera is arranged on the front side of the quadruped robot, and the depth camera is symmetrically arranged on the front side and the rear side of the quadruped robot; determining a BEV feature corresponding to a current time according to the data collected by the multi-modal sensor; acquiring a BEV feature corresponding to a historical time; determining a time sequence BEV feature according to the BEV feature corresponding to the historical time and the BEV feature corresponding to the current time; determining a multi-modal fusion feature after spatial alignment at the current time according to the data collected by the multi-modal sensor; and determining an obstacle avoidance path and gait parameters of the quadruped robot according to the time sequence BEV feature and the multi-modal fusion feature after spatial alignment at the current time.

Description

Technical Field

[0001] This disclosure relates to robot control technology, and more specifically, to a method, apparatus, and quadruped robot for determining motion information. Background Technology

[0002] Current quadruped robot mapping and localization rely on high-precision LiDAR for pre-map scanning and deployment, resulting in high hardware costs and limited applicability to various scenarios. Existing algorithms for determining motion information in quadruped robots struggle to balance real-time performance with the efficiency of multimodal data fusion. Summary of the Invention

[0003] One objective of this invention is to provide a new technical solution for determining motion information of a quadruped robot.

[0004] According to a first aspect of the present invention, a method for determining motion information of a quadruped robot is provided, comprising:

[0005] Data collected by a multimodal sensor is acquired, wherein the multimodal sensor includes a binocular camera and a depth camera, the binocular camera is disposed on the front side of the quadruped robot, and the depth camera is symmetrically disposed on the front and rear sides of the quadruped robot;

[0006] Based on the data collected by the multimodal sensor, determine the BEV characteristics corresponding to the current moment;

[0007] Obtain the BEV features corresponding to historical moments;

[0008] The time-series BEV features are determined based on the BEV features corresponding to the historical time and the BEV features corresponding to the current time.

[0009] Based on the data collected by the multimodal sensor, determine the spatially aligned multimodal fusion features at the current time.

[0010] Based on the temporal BEV features and the spatially aligned multimodal fusion features at the current time, the obstacle avoidance path and gait parameters of the quadruped robot are determined.

[0011] Optionally, determining the obstacle avoidance path and gait parameters of the quadruped robot based on the temporal BEV features and the spatially aligned multimodal fusion features at the current time step includes:

[0012] The motion information of dynamic obstacles is extracted from the temporal BEV features, and the spatial location information of dynamic obstacles, the spatial location information of static obstacles, and environmental information are extracted from the spatially aligned multimodal fusion features at the current time.

[0013] A cost map is constructed based on the motion information of the dynamic obstacles, the spatial location information of the dynamic obstacles, and the spatial location information of the static obstacles;

[0014] Obtain the current position information of the quadruped robot and the position information of the target point;

[0015] Based on the current position information and target point position information of the quadruped robot, the obstacle avoidance path of the quadruped robot is determined from the cost map, and based on the environmental information, the gait parameters of the quadruped robot at each foot point on the obstacle avoidance path are determined.

[0016] Optionally, determining the gait parameters of the quadruped robot at each foothold on the obstacle avoidance path based on the environmental information includes:

[0017] Obtain the speed information of the quadruped robot;

[0018] Based on the speed information of the quadruped robot, determine the landing point of the quadruped robot on the obstacle avoidance path;

[0019] Determine the environmental information corresponding to the landing point of the quadruped robot on the obstacle avoidance path;

[0020] Based on the environmental information corresponding to the landing points of the quadruped robot on the obstacle avoidance path, the gait parameters of the quadruped robot at each landing point on the obstacle avoidance path are determined.

[0021] Optionally, determining the BEV characteristics corresponding to the current moment based on the data collected by the multimodal sensor includes:

[0022] The data collected by the multimodal sensor is input into the MLP to perform nonlinear fusion of the data collected by the multimodal sensor and map it to the BEV space to obtain the BEV feature corresponding to the current time.

[0023] Optionally, determining the time-series BEV features based on the BEV features corresponding to the historical time and the BEV features corresponding to the current time includes:

[0024] The BEV features corresponding to the historical time and the BEV features corresponding to the current time are input into the Transformer encoder to obtain the temporal BEV features determined by temporal splicing, wherein the temporal BEV features record the motion information of dynamic obstacles.

[0025] Optionally, determining the spatially aligned multimodal fusion features at the current time based on the data collected by the multimodal sensor includes:

[0026] The data collected by the multimodal sensor is input into the MLP to fuse the data collected by the multimodal sensor and obtain the spatially aligned multimodal fusion feature at the current time. The spatially aligned multimodal fusion feature at the current time records the spatial location information of dynamic obstacles, the spatial location information of static obstacles, and environmental information.

[0027] Optionally, the multimodal sensor also includes a fisheye camera, a lidar, and a telephoto camera. The fisheye camera is located on the left, right, and rear sides of the quadruped robot, the lidar is located on the top of the quadruped robot, and the telephoto camera is located on the front side of the quadruped robot.

[0028] According to a second aspect of the present invention, a device for determining motion information of a quadruped robot is provided, comprising:

[0029] The first acquisition module is used to acquire data collected by the multimodal sensor, wherein the multimodal sensor includes a binocular camera and a depth camera. The binocular camera is disposed on the front side of the quadruped robot, and the depth camera is symmetrically disposed on the front and rear sides of the quadruped robot.

[0030] The BEV feature determination module is used to determine the BEV features corresponding to the current moment based on the data collected by the multimodal sensor.

[0031] The second acquisition module is used to acquire BEV features corresponding to historical moments.

[0032] The time-series BEV feature determination module is used to determine the time-series BEV features based on the BEV features corresponding to the historical time and the BEV features corresponding to the current time.

[0033] The multimodal fusion feature determination module is used to determine the spatially aligned multimodal fusion features at the current time based on the data collected by the multimodal sensor.

[0034] The motion information determination module is used to determine the obstacle avoidance path and gait parameters of the quadruped robot based on the temporal BEV features and the spatially aligned multimodal fusion features of the current time.

[0035] According to a third aspect of the present invention, a quadruped robot motion information determination device is provided, comprising a memory and a processor, the memory storing a computer program for controlling the processor to operate in order to perform the method according to any one of the first aspects.

[0036] According to a fourth aspect of the present invention, a quadruped robot is provided, including a quadruped robot motion information determination device as described in the second or third aspect.

[0037] The method for determining the motion information of a quadruped robot disclosed herein uses a binocular camera and a depth camera as multimodal sensors on the quadruped robot, which are low in cost. In addition, based on the data collected by the multimodal sensors, the parameters required for the motion information of the quadruped robot are determined from two branches: one branch is to determine the temporal BEV features, and the other branch is to determine the multimodal fusion features after spatial alignment at the current time. This improves the accuracy of the quadruped robot's operation information and ensures the stability of the quadruped robot's motion.

[0038] The features and advantages of the embodiments of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of these embodiments.

[0040] Figure 1 This is a flowchart illustrating a method for determining motion information of a quadruped robot according to an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of a multimodal sensor mounted on a quadruped robot according to an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of a multimodal sensor mounted on a quadruped robot according to an embodiment of the present invention.

[0043] Figure 4 This is a schematic block diagram of a quadruped robot motion information determination device according to an embodiment of the present invention.

[0044] Figure 5 This is a structural block diagram of a quadruped robot motion information determination device according to an embodiment of the present invention.

[0045] Figure 6 This is a schematic diagram of the structure of a quadruped robot according to an embodiment of the present invention. Detailed Implementation

[0046] Various exemplary embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0047] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the embodiments of this specification or their application or use.

[0048] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0049] To address the aforementioned technical issues, the method for determining the motion information of a quadruped robot provided in this disclosure utilizes a binocular camera and a depth camera as multimodal sensors mounted on the quadruped robot, which are cost-effective. Furthermore, based on the data collected by the multimodal sensors, the parameters required for the quadruped robot's motion information are determined from two branches: one branch determines the temporal BEV features, and the other branch determines the spatially aligned multimodal fusion features at the current moment. This improves the accuracy of the quadruped robot's operational information and ensures the stability of the quadruped robot's motion.

[0050] In one embodiment of the present invention, a method for determining motion information of a quadruped robot is provided. According to... Figure 1 As shown, the method for determining the motion information of a quadruped robot in this embodiment includes the following steps S110 to S140.

[0051] Step S110: Acquire data collected by multimodal sensors, wherein the multimodal sensors include a binocular camera and a depth camera. The binocular camera is set on the front side of the quadruped robot, and the depth camera is symmetrically set on the front and rear sides of the quadruped robot.

[0052] The dual-lens camera has a resolution of 1920×1080. See also... Figure 2 The field of view is 120° × 90°, the baseline distance is 15cm, and the detection distance is 30m. (See also...) Figure 2 The robot has two depth cameras, symmetrically positioned on the front and rear sides of the quadruped robot, with a field of view of 120°×90° and a detection range of 0.1-5m.

[0053] In some embodiments, the multimodal sensor further includes a fisheye camera, a lidar sensor, and a telephoto camera. The fisheye camera is positioned on the left, right, and rear sides of the quadruped robot, the lidar sensor is positioned on the top of the quadruped robot, and the telephoto camera is positioned on the front of the quadruped robot. This allows the multimodal sensor to collect richer data, providing more comprehensive and accurate data for determining the quadruped robot's motion information.

[0054] See details Figure 3 The system consists of three sets of fisheye cameras with a horizontal field of view ≥180° and a detection range of 0-10m. The lidar is an 8-line lidar with a vertical angular resolution of 2°, a horizontal angular resolution of 0.1°, and a detection range of 0.2-20m. The telephoto camera has a resolution of 1920×1080, a field of view of 120°×30°, and a detection range of 100m.

[0055] In the case of a multimodal sensor including a binocular camera and a depth camera, the data acquired by the multimodal sensor includes RGB image data and depth image data. The RGB image data is acquired by the binocular camera, and the depth image data is acquired by the depth camera.

[0056] When a multimodal sensor includes a binocular camera, a depth camera, a fisheye camera, a LiDAR, and a telephoto camera, the data acquired by the multimodal sensor includes RGB image data, depth image data, and point cloud data. RGB image data is acquired by the binocular camera, fisheye camera, and telephoto camera; depth image data is acquired by the depth camera; and point cloud data is acquired by the LiDAR.

[0057] RGB image data I t ∈R H×W×3 In this tensor, H represents the image height in pixels, W represents the image width in pixels, 3 represents the color channels (RGB, red, green, and blue), t represents the timestamp (the current frame is the t-th frame in the time-series data), and R represents the pixel value, usually normalized to [0, 1] or [0, 255]. For example, an RGB image with a resolution of 1080×1920 has a tensor shape of R. 1080×1920×3 .

[0058] Depth image data D t ∈R H×W In the diagram, H and W represent the same height and width as the RGB image, with the aim of alignment with the RGB image. D t Each pixel value represents the depth value at the corresponding location, in meters or millimeters. The depth map is single-channel data and contains no color information. To align with RGB images, pixel coordinates can be mapped to three-dimensional space using camera intrinsics.

[0059] Point cloud data P t ∈R N×3 In the diagram, N represents the number of points in the point cloud, 3 represents the three-dimensional coordinates (X, Y, Z) of each point in meters, and R represents the coordinate value, which is a real number (including positive and negative values, depending on the specific location in the coordinate system).

[0060] Step S120: Determine the BEV characteristics corresponding to the current moment based on the data collected by the multimodal sensor.

[0061] In some embodiments, before performing step S120 above, the method further includes: normalizing and histogram equalizing the RGB image data to obtain preprocessed RGB image data; filling invalid values ​​in the depth image data, mapping single-channel depth data to three-channel color data, and adjusting the resolution corresponding to the depth image data to be consistent with the resolution corresponding to the RGB image data to obtain preprocessed depth image data; performing noise reduction filtering on the point cloud data, performing voxelization on the point cloud data, and retaining the point cloud data corresponding to non-empty voxels to obtain preprocessed point cloud data; and aligning the preprocessed RGB image data, preprocessed depth image data, and preprocessed point cloud data with timestamps to obtain aligned RGB image data, depth image data, and point cloud data.

[0062] Normalization is performed on RGB image data, specifically by converting the original pixel values ​​I... raw (x, y, c) maps from the integer range (0-255) to the floating-point range (0-1).

[0063]

[0064] Where (x, y) are the pixel coordinates, and c∈{R,G,B}, specifically a certain color channel.

[0065] Histogram equalization is performed on the RGB image data, specifically by dividing the RGB image into an 8×8 grid, calculating the histogram for each grid independently, and applying histogram equalization to each grid while limiting the contrast increase (Clip Limit = 2.0).

[0066] I eq (x,y,c)=T(I norm (x,y,c))

[0067] Where T(·) is the mapping function of the cumulative distribution function (CDF).

[0068] Normalization and histogram equalization of RGB image data improve the accuracy of RGB image data and enhance the features of low-light areas, such as obstacles in shadows.

[0069] Bilateral filtering interpolation is used to fill invalid values ​​in the depth image data, specifically as follows:

[0070]

[0071] Where (x, y) are the coordinates of the pixel to be filled, (i, j) are the coordinates of all neighboring pixels centered at (x, y), and D filled (x, y) represents the depth value corresponding to the pixel to be filled, Draw (i, j) represents the depth value corresponding to a neighboring pixel centered at (x, y), w spatial For the spatial Gaussian kernel (σ) s =5 pixels), w range For depth-difference Gaussian kernels (σ r =0.1 meters), W is the normalization coefficient.

[0072] Invalid values ​​are filled into the depth image data to reduce interference from invalid areas.

[0073] Jet color mapping is used to map single-channel depth data into three-channel color data, specifically...

[0074]

[0075] Where (x, y) are the coordinates of the pixel to be mapped in the depth image, and D filled (x, y) represents the depth value corresponding to the pixel to be mapped, c is the color channel, representing the R, B, and G channels, and d... min =0.2m, d max =10m is the effective detection range of the depth camera.

[0076] If the depth map resolution is lower than the RGB image resolution, bilinear interpolation upsampling is used to adjust the resolution of the depth image data to match the resolution of the RGB image data.

[0077] The point cloud data is subjected to noise reduction filtering. Specifically, noise points are removed by statistical outlier methods for each point p. i Calculate the mean μ and standard deviation σ of points within its neighborhood (e.g., radius r = 0.1 m), and remove points that satisfy the following formula:

[0078] |p i -μ|>3σ.

[0079] The point cloud data is voxelized, retaining only the point cloud data corresponding to non-empty voxels. Specifically, the voxel grid size is defined (e.g., 0.05m × 0.05m × 0.05m), and the point cloud data is divided into an X × Y × Z voxel space. Sparse tensor storage is used, retaining only the point cloud data corresponding to non-empty voxels.

[0080]

[0081] Among them, (x i y i , z i f i (x) i y i , z i) represents the location information corresponding to the voxel, fi represents the feature corresponding to the voxel, and N v f represents the number of non-empty voxels, and 10 represents the feature dimension. i The feature corresponding to a voxel is the density feature, specifically...

[0082] f density =log(1+N) points )

[0083] Where, N points The number of points within a voxel.

[0084] The point cloud data is denoised and filtered, and the point cloud data corresponding to non-empty voxels is retained, thereby reducing the computational complexity of the point cloud data.

[0085] The preprocessed RGB image data, preprocessed depth image data, and preprocessed point cloud data are timestamped to ensure temporal consistency of multimodal data and avoid errors caused by temporal misalignment during multimodal data fusion.

[0086] In some embodiments, step S120 specifically includes: inputting the data collected by the multimodal sensor into the MLP to perform nonlinear fusion of the data collected by the multimodal sensor and map it to the BEV space to obtain the BEV features corresponding to the current time.

[0087] Specifically, the data collected by the multimodal sensors are projected onto the BEV (Bird's-Eye View) coordinate system. Then, using MLP, the data collected by the multimodal sensors projected onto the BEV coordinate system are nonlinearly fused and mapped onto the BEV space to obtain the BEV features corresponding to the current time.

[0088] Before projecting the data collected by the multimodal sensors onto the BEV (Bird's-Eye View) coordinate system, feature extraction is performed on the data collected by the multimodal sensors.

[0089] For RGB image data, the original input dimension is H×W×3. To preserve semantics and reduce computational cost, the processed dimension is H. f ×W f ×256. Since ResNet-50 is chosen as the backbone network for feature extraction, the last residual block (layer 4) of ResNet-50 outputs a feature map with 2048 channels. Further dimensionality reduction using 1×1 convolution compresses the number of channels to 256, thus maximizing feature preservation while reducing subsequent computation. The RGB image data features extracted by ResNet-50 are:

[0090]

[0091] Among them, H f =H / 16,W f =W / 16.

[0092] For depth image data, the original input dimension is H×W×1, and the processed dimension is H after aligning with RGB image data. f ×W f ×256. The features of the depth image data extracted based on ResNet-50 are:

[0093]

[0094] For point cloud data, the original input dimension is N×3, and the feature H is extracted from the point cloud data through MLP and pooling operations. B ×W B ×64, where H B W B This represents the spatial resolution of the BEV mesh, corresponding to the division of the bird's-eye view. Specifically, the point cloud data is divided into H... B ×W B For each columnar voxel, local features are extracted using MLP for the point cloud within each voxel.

[0095] f local =MLP([x,y,z,r])∈R 64

[0096] Where (x, y, z) are the coordinates of the point cloud data, and r is the reflection intensity. Then, global pooling is used to max-pool the features of all point cloud data within each voxel, resulting in a single 64-dimensional feature. This 64-dimensional feature is sufficient to encode geometric and density information and can be aligned with the 256 channels of RGB image data features and depth image data features via MLP projection.

[0097] Point cloud data features can be specifically represented as:

[0098]

[0099] Among them, H B W B 64 represents the spatial resolution of the BEV mesh and the number of channels.

[0100] Step S130: Obtain the BEV features corresponding to the historical time.

[0101] The determination of the BEV feature corresponding to a historical moment can refer to the BEV feature corresponding to the current moment mentioned above. The BEV feature corresponding to a historical moment can be a single historical moment's BEV feature, or it can be a BEV feature corresponding to multiple historical moments.

[0102] Step S140: Determine the time-series BEV features based on the BEV features corresponding to historical time points and the BEV features corresponding to the current time point.

[0103] In some embodiments, step S140 specifically includes: inputting the BEV features corresponding to the historical time and the BEV features corresponding to the current time into the Transformer encoder to obtain the temporal BEV features determined by temporal splicing, wherein the temporal BEV features record the motion information of the dynamic obstacle.

[0104] Specifically, the BEV features at the current time and the BEV features corresponding to the k historical time steps are input into the same multi-head self-attention module. The purpose is to enable the Transformer encoder to learn the correlation between the BEV features at the current time and the BEV features corresponding to the k historical time steps.

[0105] The architecture of the Transformer encoder consists of key-value pair construction, linear projection, multi-head self-attention, residual connections, and layer normalization. Specifically, the input data is projected onto three sets of vectors: Query, Key, and Value. Then, multi-head splitting is performed, that is, the large channel dimension after projection is cut into h segments (e.g., h=8). Attention is calculated separately for each segment, then concatenated, and finally linear mapping is performed to obtain the temporal BEV features.

[0106] Step S150: Determine the spatially aligned multimodal fusion features at the current moment based on the data collected by the multimodal sensor.

[0107] In some embodiments, step S150 specifically includes: inputting the data collected by the multimodal sensor into the MLP to fuse the data collected by the multimodal sensor to obtain the spatially aligned multimodal fusion feature at the current time, wherein the spatially aligned multimodal fusion feature at the current time records the spatial location information of dynamic obstacles, the spatial location information of static obstacles, and environmental information.

[0108] Specifically, the input data consists of RGB image data features, depth image data features, and point cloud data features, while the output data consists of multimodal fusion features spatially aligned at the current time.

[0109] RGB image data features are represented as follows: The features of depth image data are represented as follows: Point cloud data features are represented as follows: The spatially aligned multimodal fusion features at the current time step are represented as follows: Among them, H f W fC represents the spatial resolution of the spatially aligned multimodal fusion features (typically 1 / 16 of the input image), and C represents the number of fusion feature channels, typically set to 256. The extraction methods for RGB image data features, depth image data features, and point cloud data features can be referred to the above embodiments, and will not be elaborated further here.

[0110] Step S160: Determine the obstacle avoidance path and gait parameters of the quadruped robot based on the temporal BEV features and the multimodal fusion features after spatial alignment at the current time.

[0111] In some embodiments, step S160 includes steps S161 to S164.

[0112] Step S161: Extract motion information of dynamic obstacles from temporal BEV features, and extract spatial location information of dynamic obstacles, spatial location information of static obstacles, and environmental information from the multimodal fusion features after spatial alignment at the current time.

[0113] The motion information of dynamic obstacles includes the velocity and acceleration of the dynamic obstacles.

[0114] Environmental information includes road surface type information. The road surface type information is one of the following: flat road surface, sloping road surface, and stepped road surface.

[0115] Step S162: Construct a cost map based on the motion information of dynamic obstacles, the spatial location information of dynamic obstacles, and the spatial location information of static obstacles.

[0116] The cost map provided in this embodiment is divided into two-dimensional cost grids. Each two-dimensional cost grid stores a cost value, representing the difficulty or risk for the quadruped robot to traverse that area. A higher cost value indicates a greater difficulty or risk for the quadruped robot to traverse that area.

[0117] Step S163: Obtain the current position information of the quadruped robot and the target point position information.

[0118] Step S164: Based on the current position information of the quadruped robot and the position information of the target point, determine the obstacle avoidance path of the quadruped robot from the cost map, and determine the gait parameters of the quadruped robot at each landing point on the obstacle avoidance path based on the environmental information.

[0119] Using static environment optimization algorithms (e.g., A* algorithm) and dynamic environment optimization algorithms (e.g., D*Lite algorithm), the obstacle avoidance path of the quadruped robot from its current position to the target point is determined.

[0120] In this embodiment, a high-precision obstacle avoidance path is provided for the quadruped robot by using the motion information of dynamic obstacles, the spatial position information of dynamic obstacles, and the spatial position information of static obstacles. At the same time, the gait parameters of the quadruped robot are adjusted in real time based on environmental information to ensure the stability of the quadruped robot's movement.

[0121] In this embodiment, step S164, which determines the gait parameters of the quadruped robot at each landing point on the obstacle avoidance path based on environmental information, specifically includes: acquiring the speed information of the quadruped robot; determining the landing points of the quadruped robot on the obstacle avoidance path based on the speed information of the quadruped robot; determining the environmental information corresponding to the landing points of the quadruped robot on the obstacle avoidance path; and determining the gait parameters of the quadruped robot at each landing point on the obstacle avoidance path based on the environmental information corresponding to the landing points of the quadruped robot on the obstacle avoidance path.

[0122] The speed information of the quadruped robot is pre-stored and can be obtained directly.

[0123] The landing points of the quadruped robot on the obstacle avoidance path form a sequence of landing points.

[0124] Gait parameters include at least one of gait cycle, stride length, and stride height. Gait cycle is the time required to complete one full gait cycle. Stride length is the distance a single leg travels in one cycle. Stride height is the maximum height the swinging leg reaches off the ground.

[0125] After determining the obstacle avoidance path and gait parameters of the quadruped robot, the quadruped robot is controlled to move according to the obstacle avoidance path and gait parameters.

[0126] With multimodal sensors mounted on a quadruped robot, including binocular cameras and depth cameras, the quadruped robot is suitable for navigation on urban sidewalks. With multimodal sensors mounted on a basic robot, including binocular cameras, depth cameras, fisheye cameras, LiDAR, and telephoto cameras, the quadruped robot is not only suitable for navigation on urban sidewalks, but also for indoor multi-level scenarios, such as disaster relief in ruins, logistics delivery in multi-story buildings, and security patrol in complex lighting environments.

[0127] An embodiment of the present invention also provides a device for determining motion information of a quadruped robot. According to... Figure 4 As shown, the quadruped robot motion information determination device 400 includes a first acquisition module 410, a BEV feature determination module 420, a second acquisition module 430, a temporal BEV feature determination module 440, a multimodal fusion feature determination module 450, and a motion information determination module 460.

[0128] The first acquisition module 410 is used to acquire data collected by the multimodal sensor, wherein the multimodal sensor includes a binocular camera and a depth camera. The binocular camera is set on the front side of the quadruped robot, and the depth camera is symmetrically set on the front and rear sides of the quadruped robot.

[0129] The BEV feature determination module 420 is used to determine the BEV features corresponding to the current moment based on the data collected by the multimodal sensor.

[0130] The second acquisition module 430 is used to acquire the BEV features corresponding to historical moments.

[0131] The time-series BEV feature determination module 440 is used to determine the time-series BEV features based on the BEV features corresponding to historical time points and the BEV features corresponding to the current time point.

[0132] The multimodal fusion feature determination module 450 is used to determine the spatially aligned multimodal fusion features at the current time based on the data collected by the multimodal sensor.

[0133] The motion information determination module 460 is used to determine the obstacle avoidance path and gait parameters of the quadruped robot based on the temporal BEV features and the multimodal fusion features after spatial alignment at the current time.

[0134] In some embodiments, the motion information determination module 460 is further configured to extract motion information of dynamic obstacles from temporal BEV features, and extract spatial location information of dynamic obstacles, spatial location information of static obstacles, and environmental information from the multimodal fusion features after spatial alignment at the current time; construct a cost map based on the motion information of dynamic obstacles, the spatial location information of dynamic obstacles, and the spatial location information of static obstacles; obtain the current position information and target point position information of the quadruped robot; determine the obstacle avoidance path of the quadruped robot from the cost map based on the current position information and target point position information of the quadruped robot, and determine the gait parameters of the quadruped robot at each footing point on the obstacle avoidance path based on the environmental information.

[0135] In some embodiments, the motion information determination module 460 is further configured to acquire the speed information of the quadruped robot; determine the landing point of the quadruped robot on the obstacle avoidance path based on the speed information of the quadruped robot; determine the environmental information corresponding to the landing point of the quadruped robot on the obstacle avoidance path; and determine the gait parameters of the quadruped robot at each landing point on the obstacle avoidance path based on the environmental information corresponding to the landing point of the quadruped robot on the obstacle avoidance path.

[0136] In some embodiments, the BEV feature determination module 420 is used to input the data collected by the multimodal sensor into the MLP to perform nonlinear fusion of the data collected by the multimodal sensor and map it to the BEV space to obtain the BEV features corresponding to the current time.

[0137] In some embodiments, the temporal BEV feature determination module 440 is used to input the BEV features corresponding to historical moments and the BEV features corresponding to the current moment into the Transformer encoder to obtain temporal BEV features determined by temporal splicing, wherein the temporal BEV features record the motion information of dynamic obstacles.

[0138] In some embodiments, the multimodal fusion feature determination module 450 is used to input the data collected by the multimodal sensor into the MLP to fuse the data collected by the multimodal sensor to obtain the spatially aligned multimodal fusion feature at the current time. The spatially aligned multimodal fusion feature at the current time records the spatial location information of dynamic obstacles, the spatial location information of static obstacles, and environmental information.

[0139] In some embodiments, the multimodal sensor further includes a fisheye camera, a lidar, and a telephoto camera. The fisheye camera is disposed on the left, right, and rear sides of the quadruped robot, the lidar is disposed on the top of the quadruped robot, and the telephoto camera is disposed on the front side of the quadruped robot.

[0140] An embodiment of the present invention also provides a device for determining motion information of a quadruped robot, such as... Figure 5 As shown. The quadruped robot motion information determination device 500 includes a memory 520 and a processor 510. The memory 520 stores a computer program that controls the processor 510 to operate and execute the quadruped robot motion information determination method provided according to any of the above embodiments.

[0141] An embodiment of the present invention also provides a quadruped robot, such as Figure 6 As shown, it includes a quadruped robot motion information determination device as provided in any of the above embodiments.

[0142] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0143] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0144] Embodiments of this specification may be systems, methods, and / or computer program products. A computer program product may include a computer-readable storage medium having computer instructions stored thereon for causing a processor to implement various aspects of the embodiments of this specification.

[0145] Computer-readable storage media can be tangible devices capable of holding and storing computer instructions for use by computer instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing computer instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0146] The computer instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer instructions from the network and forwards them to computer-readable storage media within the respective computing / processing device.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this specification. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of computer instructions, which contains one or more executable computer instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0148] Various embodiments of this specification have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining motion information of a quadruped robot, characterized in that, include: Data collected by a multimodal sensor is acquired, wherein the multimodal sensor includes a binocular camera and a depth camera, the binocular camera is disposed on the front side of the quadruped robot, and the depth camera is symmetrically disposed on the front and rear sides of the quadruped robot; Based on the data collected by the multimodal sensor, determine the BEV characteristics corresponding to the current moment; Obtain the BEV features corresponding to historical moments; The time-series BEV features are determined based on the BEV features corresponding to the historical time and the BEV features corresponding to the current time. Based on the data collected by the multimodal sensor, determine the spatially aligned multimodal fusion features at the current time. Based on the temporal BEV features and the spatially aligned multimodal fusion features at the current moment, the obstacle avoidance path and gait parameters of the quadruped robot are determined, including: extracting motion information of dynamic obstacles from the temporal BEV features, and extracting spatial position information of dynamic obstacles, spatial position information of static obstacles, and environmental information from the spatially aligned multimodal fusion features at the current moment; constructing a cost map based on the motion information of dynamic obstacles, the spatial position information of dynamic obstacles, and the spatial position information of static obstacles; obtaining the current position information and target point position information of the quadruped robot; determining the obstacle avoidance path of the quadruped robot from the cost map based on the current position information and target point position information of the quadruped robot; and determining the gait parameters of the quadruped robot at each footing point on the obstacle avoidance path based on the environmental information.

2. The method according to claim 1, characterized in that, The step of determining the gait parameters of the quadruped robot at each foothold on the obstacle avoidance path based on the environmental information includes: Obtain the speed information of the quadruped robot; Based on the speed information of the quadruped robot, determine the landing point of the quadruped robot on the obstacle avoidance path; Determine the environmental information corresponding to the landing point of the quadruped robot on the obstacle avoidance path; Based on the environmental information corresponding to the landing points of the quadruped robot on the obstacle avoidance path, the gait parameters of the quadruped robot at each landing point on the obstacle avoidance path are determined.

3. The method according to claim 1, characterized in that, The step of determining the BEV characteristics corresponding to the current moment based on the data collected by the multimodal sensor includes: The data collected by the multimodal sensor is input into the MLP to perform nonlinear fusion of the data collected by the multimodal sensor and map it to the BEV space to obtain the BEV feature corresponding to the current time.

4. The method according to claim 1, characterized in that, The step of determining the time-series BEV features based on the BEV features corresponding to the historical time and the BEV features corresponding to the current time includes: The BEV features corresponding to the historical time and the BEV features corresponding to the current time are input into the Transformer encoder to obtain the temporal BEV features determined by temporal splicing, wherein the temporal BEV features record the motion information of dynamic obstacles.

5. The method according to claim 1, characterized in that, The step of determining the spatially aligned multimodal fusion features at the current time based on the data collected by the multimodal sensor includes: The data collected by the multimodal sensor is input into the MLP to fuse the data collected by the multimodal sensor and obtain the spatially aligned multimodal fusion feature at the current time. The spatially aligned multimodal fusion feature at the current time records the spatial location information of dynamic obstacles, the spatial location information of static obstacles, and environmental information.

6. The method according to any one of claims 1-5, characterized in that, The multimodal sensor also includes a fisheye camera, a lidar, and a telephoto camera. The fisheye camera is located on the left, right, and rear sides of the quadruped robot, the lidar is located on the top of the quadruped robot, and the telephoto camera is located on the front side of the quadruped robot.

7. A device for determining motion information of a quadruped robot, characterized in that, include: The first acquisition module is used to acquire data collected by the multimodal sensor, wherein the multimodal sensor includes a binocular camera and a depth camera. The binocular camera is disposed on the front side of the quadruped robot, and the depth camera is symmetrically disposed on the front and rear sides of the quadruped robot. The BEV feature determination module is used to determine the BEV features corresponding to the current moment based on the data collected by the multimodal sensor. The second acquisition module is used to acquire BEV features corresponding to historical moments. The time-series BEV feature determination module is used to determine the time-series BEV features based on the BEV features corresponding to the historical time and the BEV features corresponding to the current time. The multimodal fusion feature determination module is used to determine the spatially aligned multimodal fusion features at the current time based on the data collected by the multimodal sensor. The motion information determination module is used to determine the obstacle avoidance path and gait parameters of the quadruped robot based on the temporal BEV features and the spatially aligned multimodal fusion features at the current time. This includes: extracting motion information of dynamic obstacles from the temporal BEV features; extracting spatial location information of dynamic obstacles, spatial location information of static obstacles, and environmental information from the spatially aligned multimodal fusion features at the current time; constructing a cost map based on the motion information of dynamic obstacles, the spatial location information of dynamic obstacles, and the spatial location information of static obstacles; obtaining the current position information and target point position information of the quadruped robot; determining the obstacle avoidance path of the quadruped robot from the cost map based on the current position information and target point position information; and determining the gait parameters of the quadruped robot at each landing point on the obstacle avoidance path based on the environmental information.

8. A device for determining motion information of a quadruped robot, characterized in that, It includes a memory and a processor, the memory storing a computer program for controlling the processor to operate in order to perform the method according to any one of claims 1 to 6.

9. A quadruped robot, characterized in that, Includes the quadruped robot motion information determination device as described in claim 7 or 8.

Images