Robot state determination method, apparatus, controller, and robot

Abstract

The application relates to a robot state determination method, device, controller and robot. The method comprises the following steps: acquiring a navigation map, the navigation map being constructed based on environment data collected for the robot; in the process of movement of the robot based on the navigation map, tracking the state based on multi-dimensional sensor data collected by the robot at a historical time to obtain a predicted pose state of the robot at a current time; determining an observation update quantity corresponding to each dimension of sensor data based on the multi-dimensional sensor data collected by the robot at the current time, the navigation map and the predicted pose state; and correcting the predicted pose state based on the observation update quantity corresponding to each dimension of sensor data to obtain a target pose state of the robot at the current time. The method can improve the positioning accuracy of the robot.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, apparatus, controller and robot for determining robot state. Background Technology

[0002] With the rapid development of robotics technology, autonomous movement and positioning have become key to the intelligence of mobile robots. The core of this is to autonomously perceive the location and achieve autonomous movement in complex environments.

[0003] In related technologies, dead reckoning, inertial navigation, manual marking, or single sensor positioning are commonly used to achieve positioning through motion integration, environmental marker matching, or local feature mapping. However, these methods generally suffer from insufficient positioning accuracy, resulting in low positioning accuracy for robots in actual operations. Summary of the Invention

[0004] Therefore, it is necessary to provide a robot state determination method, device, controller, and robot that can improve the robot's positioning accuracy in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for determining the state of a robot, including:

[0006] Obtain a navigation map, which is constructed based on environmental data collected for the robot;

[0007] During the robot's movement based on the navigation map, state tracking is performed based on multi-dimensional sensor data collected by the robot at historical moments to obtain the robot's predicted pose state at the current moment.

[0008] Based on the multi-dimensional sensor data collected by the robot at the current moment, the navigation map, and the predicted pose state, the observation update amount corresponding to each dimension of sensor data is determined.

[0009] Based on the observation update amounts corresponding to the sensor data of each dimension, the predicted pose state is corrected to obtain the target pose state of the robot at the current time.

[0010] By constructing a navigation map based on environmental data collected from the robot, state tracking is achieved during robot movement by combining multi-dimensional sensor data collected at historical moments to obtain the predicted pose state. Finally, the observation update amount corresponding to each dimension of sensor data is determined by combining the multi-dimensional sensor data collected at the current moment, the navigation map, and the predicted pose state. The pose state is then corrected using the observation update amount. In the above process, by combining multi-sensor data and navigation map constraints, the impact of single sensor errors and state prediction deviations is effectively reduced. By making full use of environmental constraint information and real-time perception data for mutual verification and correction, the accuracy of robot state determination is significantly improved.

[0011] In an optional embodiment of the first aspect, determining the observation update amount corresponding to each dimension of the sensor data based on the multi-dimensional sensor data collected by the robot at the current moment, the navigation map, and the predicted pose state includes:

[0012] Based on the robot's inertial measurement sensor data at the current moment and the predicted pose state, overrange detection and residual calculation are performed to obtain the inertial measurement sensor observation update results.

[0013] Based on the image data of the robot at the current moment and the rendered image block, the photometric error is calculated to obtain the visual observation update result; wherein, the rendered image block is rendered based on the visual map points corresponding to the navigation map in the predicted pose state;

[0014] Based on the point cloud data of the robot at the current moment, planar point clouds are filtered, and the distance residual between the filtered planar point clouds and the plane in the navigation map is calculated to obtain the point cloud observation update result.

[0015] In an optional embodiment of the first aspect, the complementary constraints of multi-dimensional observation updates are utilized to effectively reduce the observation error of a single sensor, further improve the accuracy of subsequent pose state correction, and thus enhance the accuracy of robot state determination.

[0016] In an optional embodiment of the first aspect, the robot is a humanoid robot, and the step of determining the observation update amount corresponding to each dimension of sensor data based on the multi-dimensional sensor data collected by the robot at the current moment, the navigation map, and the predicted pose state further includes:

[0017] The foot support result is determined based on the robot's foot pressure sensors;

[0018] When single-foot landing is determined based on the foot support results, the odometry observation update result is determined based on the robot's motor data at the current moment and the robot's historical landing foot posture.

[0019] When both feet are landed based on the foot support results, the ground height is updated based on the robot's foot reference points and the ground plane in the navigation map.

[0020] In an optional embodiment of the first aspect, for a humanoid robot, by distinguishing between two support states—one-foot landing and two-foot landing—the odometer is updated using motor data and historical landing postures, and the ground height is updated using foot reference points and ground plane constraints on the navigation map, thereby calibrating the robot's height and posture errors and enhancing the accuracy of humanoid robot positioning.

[0021] In an optional embodiment of the first aspect, determining the odometry observation update result based on the robot's motor data at the current moment and the robot's historical landing foot pose includes:

[0022] The current foot position is determined based on the motor data at the current moment;

[0023] The residual is calculated based on the current foot position and the historical foot position of the machine to obtain the program observation update result.

[0024] In an optional embodiment of the first aspect, the current foot position is calculated by real-time motor data, and the residual is calculated by comparing it with the historical stable landing position. The stability of the landing foot position is used as a truth constraint to achieve the accuracy of the odometer observation.

[0025] In an optional embodiment of the first aspect, the environmental data includes image data and point cloud data; the method further includes: constructing an initial voxel map based on a preset resolution; the initial voxel map is constructed in an octree structure;

[0026] Based on the point cloud data, point cloud mapping is performed on the initial voxel map to obtain a point cloud voxel map.

[0027] Optical flow calculation is performed based on the image data, and the image recognition result is determined based on the optical flow calculation result; the image recognition result is used to determine the dynamic voxels and static voxels in the point cloud voxel map;

[0028] The static voxels in the point cloud voxel map are initialized with Gaussian points to obtain the navigation map.

[0029] In an optional embodiment of the first aspect, an initial voxel map is constructed by using an octree structure, and a point cloud voxel map is formed by combining point cloud data mapping. At the same time, the optical flow calculation of the image data is used to distinguish between dynamic voxels and static voxels. Only the static voxels are initialized with Gaussian points to generate a navigation map. This not only effectively reduces the amount of map data storage and computational complexity by using the octree structure, but also accurately removes dynamic interference through optical flow information, ensuring the stability and reliability of the navigation map.

[0030] In an optional embodiment of the first aspect, the step of performing optical flow calculation based on the image data and determining the image recognition result based on the optical flow calculation result includes:

[0031] The image data is divided into multiple image regions;

[0032] Optical flow calculation is performed for each of the image regions. If the optical flow calculation result for the image region is less than a preset threshold, the voxel corresponding to the image region is determined to be a static voxel.

[0033] If the optical flow calculation result corresponding to the image region is not less than a preset threshold, the voxel corresponding to the image region is determined to be a dynamic voxel.

[0034] Based on the optical flow calculation results of each of the image regions, the image recognition results are obtained.

[0035] In an optional embodiment of the first aspect, by dividing the image data into multiple image regions and performing optical flow calculations for each region, and using the comparison result of the optical flow calculation with a preset threshold as the criterion, the static and dynamic regions in the image can be quickly determined, thereby determining the static voxels and dynamic voxels.

[0036] In an optional embodiment of the first aspect, after correcting the predicted pose state based on the observation update amounts corresponding to the sensor data of each dimension to obtain the target pose state of the robot at the current moment, the method further includes:

[0037] Based on the robot's target pose state at the current moment, the Gaussian points in the navigation map are optimized to update the navigation map.

[0038] In an optional embodiment of the first aspect, after the robot target pose state is determined, the Gaussian points in the navigation map are optimized and updated based on the real-time pose. This can rely on accurate pose information to constrain the geometric and texture attributes of the Gaussian points, thereby enhancing the consistency between pose estimation and scene modeling.

[0039] Secondly, this application also provides a robot state determination device, comprising:

[0040] An acquisition module is used to acquire a navigation map, which is constructed based on environmental data collected from the robot.

[0041] The prediction module is used to perform state tracking based on multi-dimensional sensor data collected by the robot at historical moments during the robot's movement based on the navigation map, and obtain the predicted pose state of the robot at the current moment.

[0042] The observation update module is used to determine the observation update amount corresponding to each dimension of sensor data based on the multi-dimensional sensor data collected by the robot at the current moment, the navigation map, and the predicted pose state.

[0043] The correction module is used to correct the predicted pose state based on the observation update amount corresponding to the sensor data of each dimension, so as to obtain the target pose state of the robot at the current time.

[0044] Thirdly, this application also provides a controller, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the methods described above.

[0045] Fourthly, this application also provides a robot, including a robot body and the aforementioned controller.

[0046] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above aspects.

[0047] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above aspects.

[0048] Regarding the beneficial effects of any of the technical solutions in the second to sixth aspects mentioned above, refer to the beneficial effects of the corresponding technical solutions in the first aspect; repeated examples will not be listed here. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1This is a schematic diagram of an optional application environment for a robot state determination method in one embodiment.

[0051] Figure 2 This is a schematic diagram of an optional flowchart of a robot state determination method in one embodiment;

[0052] Figure 3 This is a schematic diagram of an optional flowchart for determining the observation update amount for each dimension in one embodiment;

[0053] Figure 4 This is a schematic diagram of an optional structure of a robot state determination device in one embodiment;

[0054] Figure 5 This is a schematic diagram of an optional internal structure of the controller in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0057] The robot state determination method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, robot 101 includes a robot body, a controller, and at least one sensor. Robot 101 can acquire sensor data through the sensor, and the controller determines the pose state of the robot body by executing the method provided in this application.

[0058] For example, before the robot 101 begins to move, it acquires a navigation map, which is constructed based on environmental data collected by the robot 101. During the movement of the robot 101 based on the navigation map, state tracking is performed based on multi-dimensional sensor data collected by the robot 101 at historical moments to obtain the predicted pose state of the robot 101 at the current moment. Based on the multi-dimensional sensor data, navigation map and predicted pose state collected by the robot 101 at the current moment, the observation update amount corresponding to each dimension of sensor data is determined. Based on the observation update amount corresponding to each dimension of sensor data, the predicted pose state is corrected to obtain the target pose state of the robot 101 at the current moment.

[0059] Robot 101 may be, but is not limited to, humanoid robots, wheeled mobile robots, tracked robots, service robots, inspection robots, or warehouse robots.

[0060] In one exemplary embodiment, such as Figure 2 As shown, a robot state determination method is provided, which can be applied to... Figure 1 The following steps, 201 to 204, are used as an example to illustrate the process. Wherein:

[0061] Step 201: Obtain the navigation map, which is built based on the environmental data collected for the robot.

[0062] In some embodiments, the environmental data collected by the robot can be obtained by the robot through its own deployed environmental data collection devices; or it can be obtained by other environmental data collection devices. The specific method is not limited here.

[0063] The environmental data may include at least one of laser point cloud data, image data, and inertial measurement data.

[0064] In some embodiments, the navigation map may be a raster map, a voxel map, a point cloud map, etc.; wherein, the navigation map construction process includes: preprocessing the collected environmental data, such as data denoising, time synchronization, spatial coordinate calibration and point cloud registration; then, based on the preprocessed environmental data, a map is constructed using a map construction algorithm to obtain the navigation map.

[0065] Step 202: During the robot's movement based on the navigation map, state tracking is performed based on the multi-dimensional sensor data collected by the robot at historical moments to obtain the robot's predicted pose state at the current moment.

[0066] The multi-dimensional sensor data may include at least two of the following: data collected by inertial measurement unit (such as acceleration and angular velocity), data collected by odometry (such as displacement and rotational speed), point cloud data collected by lidar, and image feature data collected by visual sensor.

[0067] In some embodiments, the predicted pose state at the current moment includes the estimated position, estimated posture, and estimated motion parameters at the current moment, thereby characterizing the robot's predicted state at the current moment.

[0068] In some embodiments, state tracking can be achieved through a tracking algorithm (such as a Kalman filter algorithm); for example, the system state equation of the robot is constructed using multi-dimensional sensor data from historical moments, and the state vector is recursively predicted based on the robot motion model to obtain the predicted pose state at the current moment.

[0069] In other embodiments, historical multi-dimensional sensor data can be fused in time to obtain fused data, and the state prediction model can then predict the robot's predicted pose state at the current moment based on the fused data.

[0070] In some embodiments, before performing state tracking based on multi-dimensional sensor data collected from historical moments, the multi-dimensional sensor data can be aligned. For example, the acquisition times of different sensor data can be synchronized, and the coordinate systems of different sensors can be unified in space, thereby eliminating state tracking errors caused by time deviations and differences in spatial coordinate systems, and ensuring the consistency of multi-dimensional sensor data in terms of time and space.

[0071] Step 203: Based on the multi-dimensional sensor data, navigation map, and predicted pose state collected by the robot at the current moment, determine the observation update amount corresponding to each dimension of sensor data.

[0072] Among them, the observation update quantity of sensor data is used to describe the difference between the actual observation value and the theoretical prediction value of the sensor.

[0073] In some embodiments, the multi-dimensional sensor data collected at the current moment represents the actual data collected by the robot at that moment, while the predicted pose state describes the robot's predicted state at the current moment. Both types of information inevitably contain certain measurement or recursive errors. The navigation map, constructed from the real environment, uses static environmental elements as stable and reliable constraint benchmarks. Therefore, the current sensor data, navigation map, and predicted pose state can be jointly calculated to eliminate the error influence of individual information by using the static environmental elements in the navigation map as a benchmark, thus more accurately extracting the observation update quantities corresponding to each dimension of the sensor.

[0074] In some embodiments, for sensor data of different dimensions, independent observation error calculation methods can be established according to their sensing principles and observation models. For example, for lidar point cloud data, map features can be projected onto the current sensor coordinate system based on the predicted pose state, and the point cloud matching error can be calculated as the observation update quantity. For visual feature data, reprojection can be performed based on the target position and predicted pose state in the map, and the reprojection error can be calculated based on the reprojected image and the currently acquired image to obtain the feature reprojection error as the observation update quantity. For inertial measurement data, the inertial residual can be calculated by combining the motion model and map spatial constraints as the observation update quantity, thereby realizing independent and decoupled calculation of the sensor observation update quantity of each dimension.

[0075] Step 204: Based on the observation update amount corresponding to the sensor data of each dimension, perform state correction on the predicted pose state to obtain the target pose state of the robot at the current moment.

[0076] The target pose state refers to the optimal pose estimation result determined by the robot at the current moment after the robot has completed the fusion correction of the predicted pose state based on the observation update amounts corresponding to the sensor data of each dimension.

[0077] In some embodiments, state correction can be performed by combining the observation update amounts corresponding to the sensor data of each dimension using the extended sparse Kalman filter algorithm. For example, for multi-dimensional sensors such as LiDAR, vision sensors, and inertial sensors, corresponding nonlinear observation equations are established respectively. Based on the predicted pose state and the actual observation data of each dimension sensor, the observation residuals are calculated. The nonlinear observation equations are linearized, and the sparse Jacobian matrix corresponding to the observation equations is solved. The extended sparse Kalman filter gain is calculated by combining the prediction covariance matrix, the sparse Jacobian matrix, and the observation noise matrix. The predicted pose state is corrected using the filter gain and the observation residuals to obtain the optimal target pose state at the current moment. At the same time, the state covariance matrix is ​​updated synchronously to complete one filtering iteration correction. The pose state after correction by the extended sparse Kalman filter is the target pose state of the robot at the current moment.

[0078] In some embodiments, different weights can be assigned to sensor data of different dimensions based on the confidence levels corresponding to the sensor data of different dimensions. Then, the observation update quantities corresponding to each dimension of the sensor are multiplied by their respective weights to obtain weighted observation update quantities. The weighted observation update quantities are then fused to obtain comprehensive observation update quantities. The predicted pose state is then corrected based on the comprehensive observation update quantities, with high-confidence sensor data as the dominant correction direction to suppress the errors introduced by low-confidence sensors, and finally the target pose state of the robot at the current moment is obtained.

[0079] In the above-described robot state determination method, a navigation map is constructed based on environmental data collected from the robot. During the robot's movement, state tracking is achieved by combining multi-dimensional sensor data collected at historical moments to obtain the predicted pose state. Finally, the observation update amount corresponding to each dimension of sensor data is determined by combining the multi-dimensional sensor data collected at the current moment, the navigation map, and the predicted pose state. The pose state is then corrected using the observation update amount. In the above process, by combining multi-sensor data and navigation map constraints, the impact of single sensor errors and state prediction deviations is effectively reduced. The environmental constraint information and real-time perception data are fully utilized for mutual verification and correction, thereby significantly improving the accuracy of robot state determination.

[0080] In one exemplary embodiment, such as Figure 3 As shown, step 203 includes steps 301 to 303. Wherein:

[0081] Step 301: Based on the robot's inertial measurement sensor data and predicted pose state at the current moment, perform overrange detection and residual calculation to obtain the inertial measurement sensor observation update results.

[0082] Among them, over-range detection is based on the range of the inertial measurement sensor to judge the validity of the inertial measurement data collected at the current moment, thereby eliminating abnormal data that exceeds the normal measurement range.

[0083] The residual is the difference between the current inertial measurement sensor data and the predicted data corresponding to the predicted pose state. The predicted pose state is determined by recursive prediction based on sensor data from historical moments, and it includes predicted state data for the inertial measurement sensor data.

[0084] Inertial measurement sensor data can include acceleration, angular velocity, acceleration deviation, angular velocity deviation, etc.

[0085] For example, after filtering outliers by over-range detection of the inertial measurement sensor data at the current moment, the residual between the predicted inertial sensor data and the measured inertial sensor data at the current moment is calculated, and the inertial measurement sensor observation update result is obtained based on the residual and the corresponding Jacobian matrix, providing inertial dimension constraints for subsequent multi-sensor fusion optimization.

[0086] For example, the residuals of inertial measurement sensor data can be calculated using the following formula:

[0087] .

[0088] in, Represents the acceleration at the current moment. Represents the acceleration at the current moment. This represents the predicted angular velocity corresponding to the predicted pose state. This represents the predicted bias value of the gyroscope. This represents the predicted acceleration corresponding to the predicted pose state. This represents the predicted acceleration bias value. This represents the projected component of gravitational acceleration in the predicted pose state.

[0089] Step 302: Calculate the photometric error based on the robot's image data and the rendered image block at the current moment to obtain the visual observation update result; wherein, the rendered image block is obtained by rendering based on the visual map points corresponding to the navigation map in the predicted pose state.

[0090] Here, luminosity is the brightness (or grayscale) value of each pixel in the image, used to characterize the camera sensor's perception of scene lighting; luminosity error refers to the difference in luminosity between the real image pixel acquired at the current moment and the corresponding pixel in the rendered image block. It is used to quantify the deviation between the projection rendering result under predicted pose and the actual observation.

[0091] Visual map points in a navigation map refer to map points with defined three-dimensional parameters that are maintained in the navigation map. Visual map points are obtained by triangulation of historical visual observations and can contain information such as three-dimensional coordinates and photometric features. For example, during the construction of a navigation map, the first object is collected from different angles. Thus, the position of the first object in the world coordinate system can be determined through triangulation calculation. In the navigation map, the map point corresponding to the first object is also a visual map point.

[0092] In some embodiments, since the visual map points in the navigation map have three-dimensional spatial positions, when the predicted pose state is determined, the visual map points in the navigation map can be projected onto the camera imaging plane corresponding to the predicted pose state according to the coordinate system transformation relationship between the world coordinate system and the predicted pose state. Then, the image is rendered according to the photometric information of the visual map points to obtain local image patches. The obtained local image patches can simulate the scene that the camera should observe under the predicted pose. Finally, the rendered image patches are matched with the corresponding regions of the current frame image, the photometric error is calculated, and the visual observation update result is obtained.

[0093] Step 303: Based on the point cloud data of the robot at the current moment, perform planar point cloud filtering, and calculate the distance residual between the filtered planar point cloud and the plane in the navigation map to obtain the point cloud observation update result.

[0094] Among them, planar point cloud screening refers to selecting effective point clouds with planar structures from the point cloud data collected by the lidar at the current moment through methods such as planar detection and planar fitting.

[0095] In some embodiments, the selected planar point clouds can be projected onto the normal vector direction of the corresponding plane in the navigation map, thereby calculating the vertical distance from each point cloud to the corresponding map plane. This distance is used as a residual to characterize the positional deviation between the current point cloud observation and the map plane.

[0096] In some embodiments, when performing planar point cloud screening, only a few point clouds can be retained for the same plane to reduce the amount of computation and improve processing efficiency. Since point cloud acquisition is a continuous process, keeping a single frame of point cloud sparse can ensure real-time performance, while by continuously accumulating and stitching together multiple frames of point clouds, the density and integrity of multiple frames of point clouds can still be ensured.

[0097] In some embodiments, before performing planar point cloud filtering based on the robot's point cloud data at the current moment, the point cloud data can also be processed to reduce the impact of motion distortion caused by the robot's motion.

[0098] In this embodiment, the complementary constraints of multi-dimensional observation updates effectively reduce the observation error of a single sensor, further improve the accuracy of subsequent pose state correction, and thus enhance the accuracy of robot state determination.

[0099] In some embodiments, the robot is a humanoid robot, and step 203 may further include:

[0100] The robot's foot pressure sensors determine the foot support result.

[0101] The foot support result indicates whether the foot has landed, including both landed and not landed.

[0102] In some embodiments, the foot support result can be determined as landing if the pressure data of the foot pressure sensor is greater than a preset pressure threshold; and the foot support result can be determined as not landing if the pressure data of the foot pressure sensor is not greater than the preset pressure threshold. Herein, not landing can include various situations such as sliding or lifting.

[0103] In other embodiments, the foot support result can also be determined by the torque of the leg motor. For example, during the movement, the motor torque and angle of a single leg over a historical period are collected (the number of motors depends on the structure), and different labels are assigned at the time of landing and when not landing. This data is then fed into a neural network (e.g., a multilayer perceptron) for supervised learning. In this way, the trained neural network can be used to determine whether to land based on the real-time motor torque and angle.

[0104] In some embodiments, the humanoid robot is a bipedal robot, with each leg having a foot pressure sensor to detect the pressure on each leg.

[0105] When single-foot landing is determined based on foot support results, the odometry observation update results are determined based on the robot's motor data at the current moment and the robot's historical landing foot poses.

[0106] The robot's foot pose is determined by the pose of the inertial measurement sensor and the relative pose between the inertial measurement sensor and the foot. For example, if the pose of the inertial measurement sensor is T, the relative pose of the left foot to the inertial measurement sensor is ∆T1, and the relative pose of the right foot to the inertial measurement sensor is ∆T2, then the left foot's landing pose is... Right foot landing position Among them, the relative pose relationship between the inertial measurement sensor and the feet is the relative pose relationship under landing conditions.

[0107] In some embodiments, the feet are not always on the ground during the robot's movement. In this case, the relative pose relationship between the inertial measurement sensor and the feet can be updated based on real-time motor data. The foot pose during the movement can be determined based on the pose of the inertial measurement sensor and the updated relative pose relationship.

[0108] Among them, the robot's historical landing foot pose refers to the foot pose when the robot landed last time. Obviously, as a bipedal robot, the humanoid robot usually lands alternately on its left and right feet. In other words, if the right foot lands at the current moment, the historical landing foot pose is the pose of the left foot when it landed last time, and vice versa.

[0109] For example, at the current moment, the robot determines that its right foot will land based on the foot support result. At this time, the pose of the right foot is fixed. The pose of the inertial measurement sensor at the current moment can be determined by the predicted pose state at the current moment. The relative pose relationship can be updated by the motor data at the current moment (such as the length of the motor linkage and the angle of the motor), thereby determining the predicted pose of the left and right feet at the current moment. At this time, the residual can be calculated based on the predicted pose of the left foot at the current moment and the pose of the historical landing foot (i.e., the pose of the left foot when it landed last time), and the odometry observation update result can be obtained. If the left foot is determined to land, the processing type is different. The difference is that when the left foot lands, the odometry observation update result needs to be determined using the right foot.

[0110] When the landing of both feet is determined based on the foot support results, the ground height update result is determined based on the robot's foot reference points and the ground plane in the navigation map.

[0111] Among them, the foot reference point is the lowest point of the robot's foot. In some embodiments, the robot's foot reference point can be determined by the robot's own structure and motor data. For example, based on the motor angle and linkage structure, the coordinates of each key point at the foot end in the world coordinate system are calculated by forward kinematics, and the point with the smallest coordinates is selected as the foot reference point.

[0112] In some embodiments, the robot's foot reference point can be mapped onto the navigation map to calculate the distance between the robot's foot reference point and the ground plane in the navigation map, thereby obtaining the ground height update result.

[0113] In the above embodiments, for humanoid robots, by distinguishing between two support states—one-foot landing and two-foot landing—the odometer observation is updated using motor data and historical landing postures, and the ground height is updated using foot reference points and ground plane constraints on the navigation map. This calibrates the robot's height and posture errors and enhances the accuracy of humanoid robot positioning.

[0114] In some embodiments, the odometry observation update result is determined based on the robot's motor data at the current moment and the robot's historical landing foot poses, including:

[0115] The current foot position is determined based on the motor data at the current moment.

[0116] For example, the current motor data (such as motor angle) is obtained, and the relative pose relationship between the left and right feet and the inertial measurement sensor is updated based on the robot's structural information (such as the length of the link) and the motor data. The current foot pose is determined by using the updated relative pose relationship and the pose of the inertial measurement sensor.

[0117] The residuals are calculated based on the current foot position and the machine's historical landing foot position, and the updated process observation results are obtained.

[0118] For example, taking the left foot as an example, the residual ,in, This refers to the historical landing foot position, specifically the position of the left foot when it landed last time. Given the current foot position, based on the relative positional relationship... pose of inertial measurement sensor Sure.

[0119] In the above embodiment, the current foot position is calculated by real-time motor data, and the residual is calculated by comparing it with the historical stable landing position. The stability of the landing foot position is used as the truth constraint to achieve the accuracy of the odometer observation.

[0120] In some embodiments, environmental data includes image data and point cloud data; the robot state determination method further includes:

[0121] An initial voxel map is constructed based on a preset resolution; the initial voxel map is constructed using an octree structure.

[0122] Here, a voxel refers to a basic unit in three-dimensional space, used to discretize and represent a continuous three-dimensional space; constructing an initial voxel map at a preset resolution means dividing the three-dimensional space of the target scene into a finite number of voxel units; it can be understood that the higher the preset resolution, the smaller the voxel size and the larger the data volume; the lower the preset resolution, the larger the voxel size and the larger the data volume.

[0123] Among them, the octree is a hierarchical data structure used for three-dimensional spatial data management. It achieves efficient spatial indexing, compressed storage, and fast querying by recursively dividing the space into eight equal-volume sub-cubes.

[0124] For example, the root node of the octree can be the entire navigation scene, the intermediate nodes can be a specific area or device, and the leaf nodes can be the environmental details of the specified area or the surface details of the device. Each node stores the corresponding voxel information. Of course, in actual deployment, the maximum depth and splitting threshold of the octree can be dynamically adjusted according to the scene.

[0125] Point cloud voxel maps are obtained by mapping the initial voxel map based on point cloud data.

[0126] Point cloud mapping refers to mapping point cloud data to voxels to obtain point cloud data corresponding to each voxel. Then, the occupancy status of the voxel can be determined based on the corresponding point cloud data in the voxel. If the corresponding point cloud data exists in the voxel, the voxel is determined to be occupied; otherwise, the voxel is determined to be idle.

[0127] Optical flow calculations are performed based on image data, and the image recognition results are determined based on the optical flow calculation results. The image recognition results are used to determine the dynamic and static voxels in the point cloud voxel map.

[0128] Optical flow calculation refers to solving the motion trajectory of the same pixel in two consecutive frames of images to obtain the moving speed and direction of each pixel, thereby determining whether each region in the image is in motion.

[0129] In some embodiments, the image recognition result may include the optical flow calculation results of each image region in the image. Thus, if the optical flow calculation result of an image region indicates that the optical flow of the image region changes significantly, it means that the image region is a region of dynamic objects (such as pedestrians, animals, moving obstacles, etc.) in the environment, and the voxel corresponding to the image region is a dynamic voxel; if the optical flow calculation result of an image region indicates that the optical flow of the image region changes basically, it means that the image region is a region of stationary scene structures (such as ground, walls, railings, fixed obstacles, etc.) in the environment, and the voxel corresponding to the image region is a static voxel.

[0130] Initialize the static voxels in the point cloud voxel map with Gaussian points to obtain the navigation map.

[0131] Among them, Gaussian point initialization refers to describing the characteristics of voxels through Gaussian distribution (Gaussian points); Gaussian point initialization of static voxels is to assign corresponding Gaussian distribution feature parameters to each static voxel, thereby transforming the structured static voxel map into a Gaussian map that can be directly used for positioning and matching, and finally forming a stable and reliable navigation map.

[0132] In the above embodiments, an initial voxel map is constructed by using an octree structure, and a point cloud voxel map is formed by combining point cloud data mapping. At the same time, the optical flow calculation of image data is used to distinguish between dynamic voxels and static voxels. Only the static voxels are initialized with Gaussian points to generate a navigation map. This not only effectively reduces the amount of map data storage by using the octree structure, but also accurately removes dynamic interference through optical flow information, ensuring the stability and reliability of the navigation map.

[0133] In some embodiments, optical flow calculation is performed based on image data, and image recognition results are determined based on the optical flow calculation results, including:

[0134] The image data is divided into multiple image regions.

[0135] In some embodiments, the image data can be divided into multiple image regions according to a preset number.

[0136] In other embodiments, the image data can be divided equally according to a preset size to obtain multiple image regions; or, the image data can be divided equally according to a preset size to obtain a reference image region. For each image region, the texture complexity of the image region is calculated. If the texture complexity is higher than the complexity threshold, the image region is divided again. The above steps are repeated until the texture complexity of each image region is not higher than the complexity threshold. If the texture complexity is not higher than the complexity threshold, no further division is performed.

[0137] In some embodiments, considering that optical flow calculation is based on two adjacent frames of image data, the two adjacent frames of image data need to be divided in the same way to ensure that each image region corresponds one-to-one and to ensure the accuracy of the optical flow calculation results.

[0138] Optical flow is calculated for each image region. If the optical flow calculation result for the image region is less than a preset threshold, the voxel corresponding to the image region is determined to be a static voxel.

[0139] Obviously, if the optical flow calculation result is less than the preset threshold, it means that the image region is a fixed object such as a wall or the ground, and the parameters such as position, color, and scale do not change over time. Therefore, the voxel corresponding to the image region is a static voxel. If the optical flow calculation result of the image region is not less than the preset threshold, it means that the image region is a dynamic object such as a pedestrian or a moving obstacle. Therefore, the voxel corresponding to the image region is a dynamic voxel.

[0140] Image recognition results are obtained based on the optical flow calculation results of each image region.

[0141] In some embodiments, the optical flow calculation results of the image region indicate static and dynamic objects in the image. It is necessary to associate the static and dynamic states identified at the image level with the corresponding three-dimensional voxels through the mapping relationship between the image and point cloud voxels, so as to obtain the image recognition results used to indicate the dynamic and static voxels in the point cloud voxel map.

[0142] For example, based on the optical flow calculation results of each image region, static image regions and dynamic image regions are determined. Then, based on the point cloud voxel map, the set of voxels containing the point cloud is determined. The voxels in the voxel set are traversed, and based on the current pose and the voxel's in-voxel normal vector, the eight vertices of the voxel are divided into front and back sides. The vertices of the front side are projected into the image. If all the vertices of the front side are projected outside the static image region, it means that the voxel does not belong to the static voxel, and the voxel is removed from the set. If there are some areas in the current static image region that do not have voxel mapping, ray projection is performed using the center of the region to find the nearest static voxel, and it is added to the set of static voxels in the current field of view, thereby determining the static voxel.

[0143] In some embodiments, in order to reduce computing power requirements, the initialization process does not perform full-range initialization, but only initializes the key image blocks with the most obvious color features in the static image area; the pixels in the key image blocks are calculated by ray projection and the intersection point is calculated with the nearest voxel projected into the area. The intersection point is used as the position parameter of the Gaussian point, the pixel color is used as the color parameter of the Gaussian point, and the normal vector and covariance parameters follow the corresponding parameters of the voxel. The position obtained by projecting the center of the image block is called the visual map point.

[0144] In the above embodiments, by dividing the image data into multiple image regions and performing optical flow calculations for each region, and using the comparison results of the optical flow calculations with a preset threshold as the criterion, the static and dynamic regions in the image can be quickly determined, thereby identifying static voxels and dynamic voxels.

[0145] In some embodiments, after correcting the predicted pose state based on the observation update amounts corresponding to the sensor data of each dimension to obtain the target pose state of the robot at the current moment, the robot state determination method further includes:

[0146] Based on the robot's target pose state at the current moment, Gaussian points in the navigation map are optimized to update the navigation map.

[0147] In some embodiments, the position, color, and opacity of Gaussian points are adjusted using multi-view photometric consistency constraints, based on the target pose state of the robot at the current moment. For example, the Gaussian points are projected onto images from different viewpoints based on the current target pose, and their positions are adjusted to improve geometric accuracy. Based on the lighting information of the surrounding environment in the target pose state, their color and opacity are optimized to make them more consistent with the actual scene. Through multi-view photometric consistency constraints, the color and geometry of the reconstructed building model can be more accurate in different viewpoints, improving the detail and realism of static scene modeling.

[0148] In the above embodiments, after the robot target pose state is determined, the Gaussian points in the navigation map are optimized and updated based on the real-time pose. This can rely on accurate pose information to constrain the geometric and texture attributes of the Gaussian points, thereby enhancing the consistency between pose estimation and scene modeling.

[0149] To facilitate understanding, an application example of a humanoid robot will be given below. The humanoid robot includes a vision sensor (fisheye camera, wide-angle camera, depth camera, etc.), a lidar, an inertial unit, a waist motor, and a neck motor.

[0150] Among them, the motor is used to acquire the joint angle and torque data of the main body, the lidar is used to acquire the three-dimensional point cloud data of the environment, the inertial unit is used to measure the acceleration and angular velocity information of the device in real time, and the vision sensor acquires environmental image information. In some embodiments, a hardware time synchronization module or a software time synchronization algorithm can be used to achieve precise time alignment of the data of the motor, lidar, inertial unit and vision sensor, ensuring the consistency of data from different sensors in the time dimension, and providing a foundation for subsequent fusion processing.

[0151] In some embodiments, data preprocessing can also be performed on various types of sensor data. For example, point cloud data acquired by lidar can be denoised and voxel filtered to remove outliers and redundant points, reduce data volume, and improve data quality. Filtering algorithms, such as adaptive filtering algorithms, can also be used to better preserve the detailed information of the point cloud. Inertial unit data can be integrated and filtered to perform over-range judgment and improve the accuracy of attitude and motion information. Images acquired by visual sensors can be processed by distortion correction and grayscale conversion to enhance image quality and prepare for subsequent feature extraction and analysis. More effective image enhancement algorithms, such as deep learning-based image enhancement algorithms, can be used to improve image clarity and feature representation capabilities.

[0152] After the robot acquires multi-sensor data, an octree structure using spatial hash indexing is constructed as a global static Gaussian map based on the LiDAR point cloud data. For example, according to the set spatial resolution and geometric features, the three-dimensional space is divided into multiple octree nodes, each corresponding to a certain spatial region. On this basis, optical flow is calculated using two consecutive frames of images. Dynamic voxels and static voxels are distinguished based on the amplitude of the optical flow vector. Dynamic voxels are not initialized with Gaussian points, while only static voxels are initialized with Gaussian points. After initialization, if the Gaussian point data in the leaf node is less than the threshold, it is inserted into the leaf voxel. During insertion, it is necessary to determine whether there is a visual map point in the voxel. If the current map point is close to an existing map point, the two map points are merged, the average of the new position is calculated, and Gaussian points that are too far away from the merged map point are deleted to obtain the navigation map. In subsequent positioning based on the navigation map, only static voxels are used to avoid interference and influence of dynamic targets on positioning.

[0153] During the robot's motion, the Extended Sparse Kalman Filter (ESFK) algorithm is used as the core algorithm for the tracking thread. Utilizing preprocessed multi-sensor data, the ESFK algorithm is used to estimate and update the robot's attitude and position in real time, calculating the robot's motion transformation between consecutive frames to obtain real-time odometry information. The state vector used by the ESFK algorithm mainly includes basic states (3D position and 3D attitude) and enhanced states (3D linear velocity, 3D angular velocity, 3D acceleration, 3D gyroscope bias, 3D accelerometer bias, angular velocity random process noise standard deviation, and acceleration random process noise standard deviation).

[0154] For example, first, the state variables are initialized, the inertial cell data over a period of time is analyzed to determine whether the system is stationary. If stationary, the gravity direction is aligned, and the global gravity Gg is initialized simultaneously. The state variables, covariance, etc., are also initialized. After that, the thread mainly executes the following steps:

[0155] Given the acquisition period ∆t of each sensor, a high-frequency timer is created, the start time t1 is recorded, the time t2 in the latest data packet of the sensor is obtained, and the time of all sensor data is waited until t2≤t1<t2+∆t, ensuring that all sensors have acquired data at time t1, and then proceeding to the next step. The data packets from the previous time t0 to the current time t1 are sorted according to time. After iteratively processing the sorted data, a prediction is made in the time interval t3-t4 based on the current state and covariance to obtain the predicted pose state, and the current prediction time t3 and the next prediction time t4 are recorded.

[0156] Different observation updates are performed based on the sensor type of the data packet. Specifically, for inertial units (IMUs), overrange detection and residual calculation are performed. For visual sensors, visible visual map points are extracted based on the predicted pose state, and these map points are rendered as Gaussian points onto the camera at the current pose to obtain a rendered image patch. This rendered image patch is then matched with the corresponding region in the current frame, and the visual map points in the map are projected into the current frame. The photometric error between the projected patch and the pixels in the current image is calculated. For lidar, distortion correction is first performed using the pose residuals of the IMU. Then, planar point clouds are filtered, and a plane is determined from the navigation map. The distance residual from the planar point cloud to the map plane is calculated. For motors and odometers, if there are jointed motors or other structures between the IMU and the visual sensor or radar, this part is required. The external parameter state vector is updated based on the data from the motors. After the odometer starts, the moment the foot lands is used as the trigger. The condition for the foot landing can be a change in the sensor values ​​such as the pressure on the sole of the foot or a change in the torque value of the leg motor. After the right foot lands, it is first determined whether the left foot has slid or lifted (in scenarios such as jumping or running). The judgment logic is whether the pressure or torque of the left foot is lower than the threshold from the last time the left foot landed to the time the right foot landed. If so, the current leg motor angle value is obtained. The relative pose of the left and right feet to the relational unit can be calculated based on the length of the link and the angle of the motor. The residual is calculated based on the pose of the left foot at the previous time. The same logic applies to the landing of the left foot. In addition, when it is determined that both feet have landed, the lowest point of both feet needs to be on the plane in the point cloud map. This part is similar to the distance from the point to the map plane.

[0157] After calculating the above observation update, the state gain calculation, state vector update and error covariance matrix update are completed by extending the sparse Kalman filter, thereby realizing the state correction of the predicted pose state and obtaining the target pose state of the robot at the current moment.

[0158] Finally, the navigation map is locally updated based on the robot's current target pose. For example, multi-view photometric consistency constraints are used to optimize the position, color, and opacity of static Gaussian points. Depending on the scene, images from visual sensors are selected to optimize the Gaussian points; for instance, the optimization frequency and the number of visual sensors are reduced when geometric features are prominent, while data from visual sensors in other directions are added for optimization when geometric features are degraded or severely occluded. Furthermore, to reduce computational demands, the optimization process does not perform full-range image optimization; only key areas, such as regions with prominent color features, are optimized.

[0159] In some embodiments, the number and density of Gaussian points can be dynamically increased based on the needs of the scene to better fit the geometric and textural details of the scene, and Gaussian points that contribute little to the model or do not meet the requirements can be removed to reduce computation and memory usage.

[0160] In some embodiments, since the navigation map is stored in the form of an octree, the fast positioning capability of the spatial hash index can be utilized during positioning. By calculating the spatial hash index of a specific area, the root voxel and leaf voxel corresponding to the global Gaussian map can be quickly located, thereby retrieving the Gaussian point in the specific area and providing data support for operations such as positioning, map updating, and matching with the current observation data.

[0161] In other embodiments, a sliding window can be maintained, and operations are preferentially performed within the sliding window during data retrieval and processing. When the area to be processed is not within the sliding window, Gaussian points within the relevant voxels are loaded into the sliding window using the index structure of the global Gaussian map. Simultaneously, if vertices within a voxel are no longer within the sliding window, all Gaussian points within that voxel need to be deleted to reduce memory usage and improve processing efficiency.

[0162] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0163] Based on the same inventive concept, this application also provides a robot state determination device for implementing the robot state determination method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more robot state determination device embodiments provided below can be found in the limitations of the robot state determination method described above, and will not be repeated here.

[0164] In one exemplary embodiment, such as Figure 4 As shown, a schematic diagram of a robot state determination device is provided. The robot state determination device 500 includes:

[0165] The acquisition module 501 is used to acquire the navigation map, which is constructed based on the environmental data collected for the robot.

[0166] The prediction module 502 is used to track the state of the robot based on multi-dimensional sensor data collected by the robot at historical moments during the robot's movement based on the navigation map, and obtain the predicted pose state of the robot at the current moment.

[0167] The observation update module 503 is used to determine the observation update amount corresponding to each dimension of sensor data based on the multi-dimensional sensor data, navigation map and predicted pose state collected by the robot at the current moment.

[0168] The correction module 504 is used to correct the predicted pose state based on the observation update amount corresponding to the sensor data of each dimension, so as to obtain the target pose state of the robot at the current moment.

[0169] In some embodiments, the observation update module 503 is used to perform overrange detection and residual calculation based on the robot's inertial measurement sensor data and predicted pose state at the current moment to obtain the inertial measurement sensor observation update result; to perform photometric error calculation based on the robot's image data and rendered image blocks at the current moment to obtain the visual observation update result; wherein, the rendered image blocks are rendered based on the visual map points corresponding to the navigation map in the predicted pose state; and to perform planar point cloud filtering based on the robot's point cloud data at the current moment, and to perform distance residual calculation between the filtered planar point cloud and the plane in the navigation map to obtain the point cloud observation update result.

[0170] In some embodiments, the robot is a humanoid robot, and the observation update module 503 is further configured to determine the foot support result based on the robot's foot pressure sensor; if the foot support result indicates that the robot has landed on one foot, the module determines the odometer observation update result based on the robot's motor data at the current moment and the robot's historical foot poses; if the foot support result indicates that the robot has landed on both feet, the module determines the ground height update result based on the robot's foot reference point and the ground plane in the navigation map.

[0171] In some embodiments, the observation update module 503 is further configured to determine the current foot position posture based on the motor data at the current moment; calculate the residual based on the current foot position posture and the machine's historical landing foot position posture to obtain the programmable observation update result.

[0172] In some embodiments, the environmental data includes image data and point cloud data. The robot state determination device 500 further includes a map building module for constructing an initial voxel map based on a preset resolution. The initial voxel map is constructed using an octree structure. Point cloud mapping is performed on the initial voxel map based on the point cloud data to obtain a point cloud voxel map. Optical flow calculation is performed based on the image data, and the image recognition result is determined based on the optical flow calculation result. The image recognition result is used to determine the dynamic and static voxels in the point cloud voxel map. Gaussian points are initialized on the static voxels in the point cloud voxel map to obtain a navigation map.

[0173] In some embodiments, the map construction module is further configured to divide the image data into multiple image regions; perform optical flow calculation for each image region; determine the voxel corresponding to the image region as a static voxel if the optical flow calculation result of the image region is less than a preset threshold; determine the voxel corresponding to the image region as a dynamic voxel if the optical flow calculation result of the image region is not less than the preset threshold; and obtain the image recognition result based on the optical flow calculation results of each image region.

[0174] In some embodiments, the robot state determination device 500 further includes a map update module, which optimizes the Gaussian points in the navigation map based on the robot's target pose state at the current moment, so as to update the navigation map.

[0175] The modules in the aforementioned robot state determination device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0176] In one exemplary embodiment, a controller is provided, the internal structure of which can be as follows: Figure 5As shown, the controller includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores data related to the robot's operation. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a robot state determination method.

[0177] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the controller to which the present application is applied. A specific controller may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0178] In one exemplary embodiment, a controller is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0179] In one embodiment, a robot is provided, including a robot body and the aforementioned controller.

[0180] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0181] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described above.

[0182] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0183] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0184] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0185] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for determining the state of a robot, characterized in that, The method includes: Obtain a navigation map, which is constructed based on environmental data collected for the robot; During the robot's movement based on the navigation map, state tracking is performed based on multi-dimensional sensor data collected by the robot at historical moments to obtain the robot's predicted pose state at the current moment. Based on the multi-dimensional sensor data collected by the robot at the current moment, the navigation map, and the predicted pose state, the observation update amount corresponding to each dimension of sensor data is determined. Based on the observation update amounts corresponding to the sensor data of each dimension, the predicted pose state is corrected to obtain the target pose state of the robot at the current time.

2. The method according to claim 1, characterized in that, The determination of the observation update amount corresponding to each dimension of sensor data based on the multi-dimensional sensor data collected by the robot at the current moment, the navigation map, and the predicted pose state includes: Based on the robot's inertial measurement sensor data at the current moment and the predicted pose state, overrange detection and residual calculation are performed to obtain the inertial measurement sensor observation update results. Based on the image data of the robot at the current moment and the rendered image block, the photometric error is calculated to obtain the visual observation update result; wherein, the rendered image block is rendered based on the visual map points corresponding to the navigation map in the predicted pose state; Based on the point cloud data of the robot at the current moment, planar point clouds are filtered, and the distance residual between the filtered planar point clouds and the plane in the navigation map is calculated to obtain the point cloud observation update result.

3. The method according to claim 2, characterized in that, The robot is a humanoid robot. The step of determining the observation update amount corresponding to each dimension of sensor data based on the multi-dimensional sensor data collected by the robot at the current moment, the navigation map, and the predicted pose state further includes: The foot support result is determined based on the robot's foot pressure sensors; When single-foot landing is determined based on the foot support results, the odometry observation update result is determined based on the robot's motor data at the current moment and the robot's historical landing foot posture. When both feet are landed based on the foot support results, the ground height is updated based on the robot's foot reference points and the ground plane in the navigation map.

4. The method according to claim 3, characterized in that, The process of determining the odometry observation update result based on the robot's motor data at the current moment and the robot's historical landing foot poses includes: The current foot position is determined based on the motor data at the current moment; The residual is calculated based on the current foot position and the historical foot position of the machine to obtain the program observation update result.

5. The method according to claim 1, characterized in that, The environmental data includes image data and point cloud data; the method further includes: An initial voxel map is constructed based on a preset resolution; the initial voxel map is constructed using an octree structure. Based on the point cloud data, point cloud mapping is performed on the initial voxel map to obtain a point cloud voxel map. Optical flow calculation is performed based on the image data, and the image recognition result is determined based on the optical flow calculation result; the image recognition result is used to determine the dynamic voxels and static voxels in the point cloud voxel map; The static voxels in the point cloud voxel map are initialized with Gaussian points to obtain the navigation map.

6. The method according to claim 5, characterized in that, The step of performing optical flow calculation based on the image data and determining the image recognition result based on the optical flow calculation result includes: The image data is divided into multiple image regions; Optical flow calculation is performed for each of the image regions. If the optical flow calculation result for the image region is less than a preset threshold, the voxel corresponding to the image region is determined to be a static voxel. If the optical flow calculation result corresponding to the image region is not less than a preset threshold, the voxel corresponding to the image region is determined to be a dynamic voxel. Based on the optical flow calculation results of each of the image regions, the image recognition results are obtained.

7. The method according to any one of claims 1 to 6, characterized in that, After correcting the predicted pose state based on the observation update amounts corresponding to the sensor data of each dimension to obtain the target pose state of the robot at the current moment, the method further includes: Based on the robot's target pose state at the current moment, the Gaussian points in the navigation map are optimized to update the navigation map.

8. A robot state determination device, characterized in that, The device includes: An acquisition module is used to acquire a navigation map, which is constructed based on environmental data collected from the robot. The prediction module is used to perform state tracking based on multi-dimensional sensor data collected by the robot at historical moments during the robot's movement based on the navigation map, and obtain the predicted pose state of the robot at the current moment. The observation update module is used to determine the observation update amount corresponding to each dimension of sensor data based on the multi-dimensional sensor data collected by the robot at the current moment, the navigation map, and the predicted pose state. The correction module is used to correct the predicted pose state based on the observation update amount corresponding to the sensor data of each dimension, so as to obtain the target pose state of the robot at the current time.

9. A controller comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A robot, characterized in that, It includes the robot body and the controller as described in claim 9.

Images