Method and apparatus for generating odometry information of robot, and device and product

Abstract

The present disclosure relates to a method and apparatus for generating odometry information of a robot, and a device and a computer program product. The method comprises acquiring a plurality of pieces of sensor data from a plurality of different types of sensors. The method also comprises: on the basis of the plurality of pieces of sensor data, generating a plurality of token embeddings corresponding to the different types. The method further comprises: on the basis of the plurality of token embeddings, using an attention-based machine learning model to generate a fused odometry embedding. In addition, the method further comprises generating odometry information on the basis of the fused odometry embedding. In this way, the accuracy and robustness of the generated odometry information can be improved.

Description

Methods, apparatus, devices, and products for generating odometry information for robots. Technical Field

[0001] This disclosure relates to the field of robotics, and more specifically to methods, apparatus, devices, and computer program products for generating odometry information for robots. Background Technology

[0002] Robotic odometry is a technique used to estimate the position and orientation of a robot in its operating environment. Odometry typically relies on sensor data that measures the robot's own motion, such as lidar, wheeled odometry, inertial measurement units, and cameras.

[0003] Wheel odometry estimates a robot's pose by measuring wheel rotation. LiDAR acquires point cloud data of the environment and uses algorithms to calculate the robot's movement and position changes within that environment. An inertial measurement unit (IMU), consisting of accelerometers and gyroscopes, measures the robot's acceleration and angular velocity, helping to estimate its attitude changes (e.g., tilt and rotation). Cameras acquire images of the surrounding environment and estimate the robot's pose by analyzing successive image frames.

[0004] Summary of the Invention

[0005] In a first aspect of the embodiments of this disclosure, a method for generating odometry information for a robot is provided. The method includes acquiring multiple sensor data from multiple sensors of multiple types. The method further includes generating multiple labeled embeddings corresponding to the multiple types based on the multiple sensor data. The method also includes generating a fused odometry embedding based on the multiple labeled embeddings using an attention-based machine learning model. Furthermore, the method includes generating odometry information based on the fused odometry embedding.

[0006] In a second aspect of the embodiments of this disclosure, an apparatus for generating odometry information for a robot is provided. The apparatus includes a sensor data acquisition unit configured to acquire multiple sensor data from multiple sensors of multiple types. The apparatus also includes a label embedding generation unit configured to generate multiple label embeddings corresponding to multiple types based on the multiple sensor data. Furthermore, the apparatus includes an odometry embedding generation unit configured to generate a fused odometry embedding based on the multiple label embeddings using an attention-based machine learning model. Additionally, the apparatus includes an odometry information generation unit configured to generate odometry information based on the fused odometry embedding.

[0007] In a third aspect of embodiments of this disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a method for generating odometry information for a robot. The method includes acquiring multiple sensor data from multiple sensors of multiple types. The method further includes generating multiple labeled embeddings corresponding to the multiple types based on the multiple sensor data. The method also includes generating a fused odometry embedding based on the multiple labeled embeddings using an attention-based machine learning model. Furthermore, the method includes generating odometry information based on the fused odometry embedding.

[0008] In a fourth aspect of embodiments of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to implement a method for generating odometry information for a robot. The method includes acquiring multiple sensor data from multiple sensors of multiple types. The method also includes generating multiple labeled embeddings corresponding to the multiple types based on the multiple sensor data. The method further includes generating a fused odometry embedding based on the multiple labeled embeddings using an attention-based machine learning model. Furthermore, the method includes generating odometry information based on the fused odometry embedding.

[0009] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 illustrates a schematic diagram of an example environment in which various embodiments of the present disclosure may be implemented;

[0012] Figure 2 shows a flowchart of a method for generating odometry information for a robot according to some embodiments of the present disclosure;

[0013] Figure 3 illustrates a schematic diagram of an example of generating odometer information according to some embodiments of the present disclosure;

[0014] Figure 4 illustrates a schematic diagram of an example of identifying a failed sensor among multiple sensors when generating odometer information according to some embodiments of the present disclosure.

[0015] Figure 5 illustrates a schematic diagram of an example of generating odometer information based on multiple sets of sensor data over a period of time, according to some embodiments of the present disclosure;

[0016] Figure 6 shows a flowchart of an example process for training a model using two stages according to some embodiments of the present disclosure;

[0017] Figure 7 shows a block diagram of an apparatus for generating odometry information for a robot according to some embodiments of the present disclosure; and

[0018] Figure 8 shows a block diagram of a device capable of implementing several embodiments of the present disclosure. Detailed Implementation

[0019] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0022] Mobile robots need to accurately estimate their pose (i.e., odometry) in order to navigate correctly, avoid obstacles, and perform various tasks. Pose includes the robot's position (e.g., coordinates in a plane or three-dimensional space) and orientation (e.g., facing and tilt angles). Odometry information can be calculated based on data captured by various sensors equipped on the robot itself. Examples of such sensors include wheeled odometry, LiDAR, inertial measurement units (IMUs), and cameras. Wheeled odometry calculates the distance and angle the robot has traveled by measuring the rotation angle of its wheels. LiDAR generates three-dimensional point cloud data of the environment by emitting laser pulses and receiving reflected signals. An inertial measurement unit may contain accelerometers and gyroscopes to measure the robot's acceleration and rotational speed. Cameras capture images of the robot's surrounding environment to estimate its pose.

[0023] During robot operation, each sensor may fail under certain conditions. For example, wheel slippage can cause errors in wheeled odometry, and cameras may be heavily obstructed, preventing them from acquiring effective images. To improve the accuracy and reliability of odometry, information from multiple sensors can be combined to determine odometry data. This allows for pose estimation to continue using data from other sensors even if one sensor fails. Furthermore, even when all sensors are functioning correctly, data from different sensors may be inconsistent, necessitating the effective fusion of this information.

[0024] In some related technologies, a series of pre-set rules and conditions can be used to detect whether a sensor is malfunctioning or its data is inaccurate. For example, if wheel slippage causes anomalies in the data from a wheel odometer, the system can determine that the data is unreliable based on a pre-set threshold. However, this method relies on manually defined rules, has low flexibility and robustness, and cannot be applied to complex and variable environments. Furthermore, some related technologies utilize prior information and handwritten rules to set a covariance matrix for appropriately weighting information from different sensors when fusing them. However, this method also relies on human experience and requires extensive experimentation and adjustments, consuming resources without achieving optimal results.

[0025] Therefore, embodiments of this disclosure provide a scheme for generating odometry information for a robot. In this scheme, the robot can be configured with various types of sensors, such as wheeled odometry, LiDAR, inertial measurement units, cameras, or other types of sensors. The scheme can generate multiple labeled embeddings for each type of sensor data. Then, the scheme can utilize an attention-based machine learning model to fuse these labeled embeddings to generate a fused odometry embedding. Finally, the scheme can generate odometry information based on the fused odometry embedding.

[0026] In this way, the attention mechanism can capture the relationships and dependencies between different sensor data, helping the model better understand and fuse multiple types of sensor data. Through training, the attention-based machine learning model can also dynamically adjust the weights of various types of sensor data according to changes in individual sensor data, thereby improving the accuracy and robustness of the fused odometry information.

[0027] Figure 1 illustrates a schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. As shown in Figure 1, environment 100 includes a robot 101 having a control unit 102, sensors 103, 104, and 105, wherein sensors 103, 104, and 105 are different types of sensors. For example, sensor 103 may be an inertial measurement unit, sensor 104 may be a wheeled odometer, and sensor 105 may be a lidar. It should be noted that although only three types of sensors are shown in environment 100, this is not intended to limit the number of sensor types, and embodiments of the present disclosure may include fewer or more types of sensors.

[0028] In environment 100, control unit 102 can acquire sensor data 113 from sensor 103, sensor data 114 from sensor 104, and sensor data 115 from sensor 105. For example, when the sensor is an inertial measurement unit, the sensor data may include the robot's linear acceleration along the X, Y, and Z axes, as well as angular velocities related to roll, pitch, and yaw angles. When the sensor is a wheeled odometry system, the sensor data may include, for example, the rotation angle and rotational speed of each wheel. When the sensor is a lidar system, the sensor data may include, for example, 3D point cloud data of the surrounding environment. When the sensor is a camera, the sensor data may include, for example, images of the surrounding environment captured by the camera.

[0029] As shown in Figure 1, after acquiring sensor data 113, 114, and 115, the control unit 102 can generate corresponding labeled embeddings 123, 124, and 125 based on this sensor data. For example, in some embodiments, the control unit 102 can treat the sensor data as tokens and generate corresponding labeled embeddings using techniques such as linear layers, convolutional layers, or encoders. The control unit 102 can then input the labeled embeddings 123, 124, and 125 into an attention-based machine learning model 130. For example, the attention-based machine learning model 130 can be a Transformer encoder. The attention-based machine learning model 130 can utilize attention mechanisms to capture the relationships and dependencies between the information contained in the labeled embeddings 123, 124, and 125, and generate a fused odometry embedding 132, which can indicate the fused robot pose. Then, the control unit 102 can generate odometry information 134 based on the fused odometry embedding 132. The odometry information 134 may include, for example, the robot's two-dimensional or three-dimensional coordinates, roll angle, pitch angle, and yaw angle.

[0030] In this way, the attention-based machine learning model 130 is able to better understand and fuse sensor data 113, 114, and 115 from sensors 103, 104, and 105. Furthermore, the model can dynamically adjust the weights of various types of sensor data based on changes in the individual sensor data, thereby improving the accuracy and robustness of the generated odometer information 134.

[0031] Figure 2 illustrates a flowchart of a method 200 for generating odometry information for a robot according to some embodiments of the present disclosure. Method 200 can be performed, for example, by the control unit 102 shown in Figure 1. As shown in Figure 2, at block 202, method 200 can acquire multiple sensor data from multiple sensors of various types. For example, in the environment 100 shown in Figure 1, control unit 102 can acquire sensor data 113, 114, and 115 from sensors 103, 104, and 105. Sensors 103, 104, and 105 are different types of sensors. For example, sensor 103 could be an inertial measurement unit, sensor 104 could be a wheeled odometry system, and sensor 105 could be a lidar system.

[0032] In box 204, method 200 can generate multiple labeled embeddings corresponding to various types based on multiple sensor data. For example, in environment 100 as shown in FIG1, control unit 102 can generate labeled embedding 123 based on sensor data 113, labeled embedding 124 based on sensor data 114, and labeled embedding 125 based on sensor data 115. For example, control unit 102 can generate corresponding labeled embeddings based on sensor data by utilizing techniques such as linear layers, convolutional layers, or encoders.

[0033] In box 206, method 200 can generate a fused odometry embedding based on multiple labeled embeddings using an attention-based machine learning model. For example, in environment 100 as shown in Figure 1, control unit 102 can input labeled embeddings 123, 124, and 125 into attention-based machine learning model 130. Attention-based machine learning model 130 can use attention mechanisms to capture the relationships and dependencies between the information contained in labeled embeddings 123, 124, and 125 and generate a fused odometry embedding 132, which can indicate a fused robot pose.

[0034] In box 208, method 200 can generate odometry information based on fused odometry embedding. For example, in environment 100 as shown in FIG1, control unit 102 can generate odometry information 134 based on fused odometry embedding 132, odometry information 134 may include, for example, the robot's two-dimensional or three-dimensional coordinates, roll angle, pitch angle, and yaw angle.

[0035] In this way, attention-based machine learning models can better understand and fuse multiple sensor data from various types of sensors. Furthermore, the model can dynamically adjust the weights of different sensor data types based on changes in the individual sensor data, thereby improving the accuracy and robustness of the generated odometry information.

[0036] In some embodiments, the multiple types of sensors include at least two of LiDAR, wheel odometry, inertial measurement units, and cameras. In some embodiments, to generate a labeled embedding, a first type of cumulative pose estimation can be generated based on multiple first-type sensor data. Then, a first-type labeled embedding can be generated based on the first-type cumulative pose estimation. In some embodiments, a first-type labeled embedding can also be generated based on multiple first-type sensor data and the first-type cumulative pose estimation. In some embodiments, a second-type labeled embedding can be generated based on multiple second-type sensor data. Furthermore, a first-type embedding can be generated based on the first type, and a second-type embedding can be generated based on the second type. Then, a fused odometry embedding can be generated based on the first-type labeled embedding, the first-type embedding, the second-type labeled embedding, and the second-type embedding.

[0037] Figure 3 illustrates a schematic diagram of an example 300 for generating odometer information according to some embodiments of the present disclosure. As shown in Figure 3, in example 300, an inertial measurement unit (IMU) marker 301 is generated based on sensor data acquired from an IMU. A wheel odometer marker 302 is generated based on sensor data acquired from a wheel odometer. A lidar marker 303 is generated based on sensor data acquired from a lidar. In some embodiments, since different sensors can perform data sampling at different frequencies, the period for generating fused odometer information can be determined based on the sampling frequency of each sensor. This allows for the fusion of multiple frames of data from a subset of multiple sensors within a single period, thereby improving the accuracy of the fused odometer information.

[0038] For example, the data sampling frequency of the lidar could be 10 Hz (i.e., 10 data frames sampled per second), the data sampling frequency of the inertial measurement unit (IMU) could be 200 Hz, and the data sampling frequency of the wheeled odometer could be 100 Hz. In Example 300, the lidar sampling frequency is the slowest, so the lidar sampling frequency can be determined as the frequency for generating the fused odometer information. Within one period (e.g., 0.1 seconds corresponding to 10 Hz), lidar data from two adjacent frames can be acquired, and the relative pose can be calculated using the Iterative Closest Point (ICP) algorithm. The calculated relative pose can be used as lidar marker 303. The ICP algorithm can be used for point cloud registration, which can find the optimal registration by iteratively optimizing to minimize the distance between two sets of point clouds. Furthermore, the cumulative pose estimate of the IMU can be determined based on 20 frames of IMU data, and the cumulative pose estimate of the wheeled odometer can be determined based on 10 frames of wheeled odometer data. In some embodiments, the cumulative pose estimate of the determined inertial measurement unit (IMU) can be used as IMU marker 301, and the cumulative pose estimate of the determined wheel odometer can be used as wheel odometer marker 302. In some embodiments, 20 frames of raw IMU data and the corresponding cumulative pose estimates can be used together as IMU marker 301, and 10 frames of raw wheel odometer data and the corresponding cumulative pose estimates can be used together as wheel odometer marker 302. This enriches the information of IMU marker 301 and wheel odometer marker 302, thereby enhancing their expressive power and improving the accuracy of the fused odometer information.

[0039] In Example 300, an inertial measurement unit (IMU) label embedding 311 can be generated based on IMU label 301, a wheel odometer label embedding 312 can be generated based on wheel odometer label 302, and a lidar label embedding 313 can be generated based on lidar label 303. In some embodiments, label embeddings can be generated based on labels using linear or convolutional layers. In some embodiments, the raw data can be discretized into a vocabulary, and then label embeddings can be generated based on the vocabulary. This approach facilitates data standardization, enabling data from different sensors to be processed and compared within the same framework.

[0040] In Example 300, multiple type embeddings for multiple sensor types can also be generated, enabling the model to identify which sensor the data originates from. As shown in Figure 3, type embedding 321 corresponds to the inertial measurement unit (IMU), type embedding 322 corresponds to the wheeled odometer, and type embedding 323 corresponds to the lidar. Then, the attention-based machine learning model 330 can generate a fused odometer embedding 332 based on the IMU label embedding 311, type embedding 321, wheeled odometer embedding 312, type embedding 322, lidar label embedding 313, and type embedding 323. Then, based on the fused odometer embedding 332, a multilayer perceptron 336 can be used to generate odometer information 334, which may include the robot's two-dimensional or three-dimensional coordinates, roll angle, pitch angle, and yaw angle, etc.

[0041] In some embodiments, multiple sensor embeddings corresponding to various types can be generated using an attention-based machine learning model based on multiple labeled embeddings. Then, a failed sensor among the multiple sensors can be determined based on these multiple sensor embeddings. In some embodiments, once the failed sensor among the multiple sensors is determined, a prompt message indicating the failed sensor can be generated.

[0042] Figure 4 illustrates a schematic diagram of Example 400, which identifies a failed sensor among multiple sensors when generating odometer information according to some embodiments of the present disclosure. As shown in Figure 4, in Example 400, an inertial measurement unit (IMU) marker 401, a wheel odometer marker 402, and a lidar marker 403 have been generated based on data from multiple sensors. In Example 400, an IMU marker embedding 411 can be generated based on the IMU marker 401, a wheel odometer marker embedding 412 can be generated based on the wheel odometer marker 402, and a lidar marker embedding 413 can be generated based on the lidar marker 403. Furthermore, a type embedding 421 corresponding to the IMU, a type embedding 422 corresponding to the wheel odometer, and a type embedding 423 corresponding to the lidar can also be generated.

[0043] In Example 400, the attention-based machine learning model 430 can be a Transformer encoder. When inputting the label embedding and its corresponding type embedding into the attention-based machine learning model 430, a classification label embedding 414 can be simultaneously input, generated based on a classification label 404 (e.g., a [CLS] label). The classification label 404 does not contain explicit semantic information compared to other sensor labels. The attention-based machine learning model can then output an output embedding corresponding to the label embeddings of the multiple inputs. As shown in Figure 4, the attention-based machine learning model 430 can generate an inertial measurement unit (IMU) embedding corresponding to the IMU label embedding 411, a wheel odometer embedding 432 corresponding to the wheel odometer label embedding 412, a lidar embedding 433 corresponding to the lidar label embedding 413, and a fused odometer embedding 434 corresponding to the classification label embedding 414. The multilayer perceptron 436 can generate odometer information 438 based on the fused odometer embedding 434. In this way, the fused odometer embedding 434 can more fairly fuse the semantic information in the various sensor tags, thereby better representing the overall semantics of the various sensor tags and improving the accuracy of the odometer information 438.

[0044] In Example 400, the inertial measurement unit (IMU) embedding 431, the wheeled odometer embedding 432, and the lidar embedding 433 can indicate whether the data from each sensor is abnormal. The classification module 440 can include classifiers for each sensor, which can generate classification results indicating whether a specific sensor has failed based on the input embeddings. For example, the classification module 440 can generate classification result 441 based on the IMU embedding 431, indicating whether the IMU has failed (or malfunctioned). The classification module 440 can also generate classification result 442 based on the wheeled odometer embedding 432, indicating whether the wheeled odometer has failed. Furthermore, the classification module 440 can generate classification result 443 based on the lidar embedding 433, indicating whether the lidar has failed. If a classification result indicates that the corresponding sensor has failed, a prompt indicating the failed sensor can be generated. The generated prompt can be displayed on the robot's display device or sent to a management application to remind the user to replace or repair the failed sensor in a timely manner.

[0045] In this way, data from various sensors can be combined to identify faulty sensors without the need for manually setting individual sensor failure rules, thereby improving the accuracy and robustness of sensor failure detection.

[0046] In some embodiments, multiple sets of sensor data over multiple time periods can be acquired. Then, multiple sets of labeled embeddings can be generated based on the multiple sets of sensor data. Finally, a fused odometry embedding can be generated based on the multiple sets of labeled embeddings.

[0047] Figure 5 illustrates a schematic diagram of Example 500, which generates odometer information based on multiple sets of sensor data over a period of time according to some embodiments of the present disclosure. As shown in Figure 5, the inertial measurement unit (IMU) marker 501 and the wheel odometer marker 502 are sensor markers generated based on data acquired from the IMU and the wheel odometer within one time period (e.g., 0.1 seconds). Thus, two sensor markers can be generated every 0.1 seconds, and twenty sensor markers can be generated per second. In Example 500, a period of time T can be continuously sampled, thereby obtaining a marker sequence of length 20*T. As shown in Figure 5, the IMU marker 503 and the wheel odometer marker 504 are sensor markers generated based on data acquired from the IMU and the wheel odometer within the next time period. It should be understood that, for simplicity, only the markers of two sensors within two time periods are shown in Figure 5, but embodiments of the present disclosure may include markers of more sensors within more time periods.

[0048] In Example 500, inertial measurement unit (IMU) marker embeddings 511 and 513 can be generated based on IMU markers 501 and 503, and wheel odometer marker embeddings 512 and 514 can be generated based on wheel odometer markers 502 and 504. In Example 500, type embeddings 521 and 523 correspond to IMU type embeddings, and type embeddings 522 and 524 correspond to wheel odometer type embeddings. In Example 500, a position embedding 531 can be generated for IMU marker embedding 511, and the position embedding 531 is associated with the time when the sensor data used to generate IMU marker embedding 511 was sampled. Furthermore, a position embedding 532 can be generated for wheel odometer marker embedding 512. Since the sampling time of the sensor data used to generate IMU marker embedding 511 and wheel odometer marker embedding 512 is the same, position embedding 531 and position embedding 532 can be the same. In addition, position embeddings 533 and 534 can be generated for inertial measurement unit marker embedding 513 and wheel odometer marker embedding 514.

[0049] In Example 500, a classification label embedding 515 can be generated based on the classification label 505. Then, the inertial measurement unit (IMU) label embedding 511, wheel odometer label embedding 512, IMU label embedding 513, wheel odometer label embedding 514, corresponding type embeddings 521, 522, 523, and 524, and corresponding position embeddings 531, 532, 533, and 534 can be input into an attention-based machine learning model 540 to generate IMU embedding 541, wheel odometer embedding 542, IMU embedding 543, wheel odometer embedding 544, and a fused odometer embedding 545. The fused odometer embedding 545 can fuse information contained in multiple sets of sensor data over a period of time. A multilayer perceptron 546 can generate odometer information 548 based on the fused odometer embedding 545. In addition, the classification module 550 can generate a classification result 551 indicating whether the inertial measurement unit is malfunctioning based on the inertial measurement unit embeddings 541 and 543, and generate a classification result 552 indicating whether the wheel odometer is malfunctioning based on the wheel odometer embeddings 542 and 544.

[0050] By sampling multiple sets of sensor data over a period of time, the instantaneous noise and errors of individual sensors can be smoothed out, thereby improving the accuracy of pose estimation. Furthermore, by analyzing data over a period of time, abnormal sensor patterns and failure characteristics can be more easily identified, reducing false alarms and missed alarms.

[0051] In Example 500, the attention-based machine learning model 540 can be a Transformer encoder. To enable the model to better learn the relationships between different types of data, different attention masks can be applied at certain layers of the model. Attention masks can control which parts of the input sequence should be focused on and which parts should be ignored. In some embodiments, attention masks in some layers of the model can be applied only between different sensors at the same time. In some embodiments, attention masks in some layers of the model can be applied between different times of the same sensor. In some embodiments, causal attention masks can be applied at certain layers of the model. Causal attention masks can improve the accuracy and consistency of the fused odometry embedding generated by the model by allowing each time step to see only information from the current and previous time steps during sequence generation, while excluding future information.

[0052] This approach helps the model allocate attention more effectively when processing different types of input data, thereby improving the accuracy of the generated odometer information.

[0053] In some embodiments, the training process of an attention-based machine learning model may include an unsupervised pre-training phase and a supervised fine-tuning phase. Figure 6 illustrates a flowchart of an example process 600 for training a model using two phases according to some embodiments of the present disclosure. As shown in Figure 6, at block 602, the model can be pre-trained unsupervised. For example, the network can be pre-trained in a supervised manner using masked sensor modeling. No additional supervisory signal is required during the pre-training phase, thereby reducing the amount of ground truth pose required. During the pre-training phase, multiple training data from multiple sensors can be acquired, and multiple training label embeddings can be generated based on the multiple training data. Masked multiple training label embeddings can then be generated by masking a portion of the multiple training label embeddings, and the attention-based machine learning model can be pre-trained based on the multiple training label embeddings and the masked multiple training label embeddings. In some embodiments, the masked multiple training label embeddings can be generated by masking all training label embeddings associated with a specific type of sensor within a specific time period. In some embodiments, noise can be generated based on anomalous sensor data from a failed sensor, and the masked multiple training label embeddings can be generated by adding noise to a portion of the multiple training label embeddings. In some embodiments, multiple masked training label embeddings can be generated by masking all training label embeddings associated with a particular type of sensor.

[0054] In box 602, supervised fine-tuning of the model can be performed to achieve odometry information fusion and failure detection. During the fine-tuning phase, multiple labeled poses for multiple training data can be obtained, and the pre-trained attention-based machine learning model (e.g., attention-based machine learning model 540 in Figure 5), classification module (e.g., classification module 550 in Figure 5), and multilayer perceptron (e.g., multilayer perceptron 546 in Figure 5) can be fine-tuned based on the multiple training data and multiple labeled poses.

[0055] In this way, the amount of labeled odometer information required can be reduced, thereby saving the cost of acquiring training data.

[0056] Figure 7 shows a block diagram of an apparatus 700 for generating odometry information for a robot according to some embodiments of the present disclosure. As shown in Figure 7, the apparatus 700 includes a sensor data acquisition unit 702 configured to acquire multiple sensor data from multiple sensors of multiple types. The apparatus 700 also includes a label embedding generation unit 704 configured to generate multiple label embeddings corresponding to multiple types based on the multiple sensor data. The apparatus 700 also includes an odometry embedding generation unit 706 configured to generate a fused odometry embedding based on the multiple label embeddings using an attention-based machine learning model. Furthermore, the apparatus 700 includes an odometry information generation unit 708 configured to generate odometry information based on the fused odometry embedding.

[0057] In some embodiments, the various types of sensors include at least two of lidar, wheel odometer, inertial measurement unit, and camera.

[0058] In some embodiments, the plurality of sensor data includes a plurality of first-type sensor data, and the tag embedding generation module includes: a cumulative pose estimation generation module configured to generate a first-type cumulative pose estimation based on the plurality of first-type sensor data; and a cumulative pose estimation usage module configured to generate a first-type tag embedding based on the first-type cumulative pose estimation.

[0059] In some embodiments, the cumulative pose estimation module includes a first type of label embedding generation module configured to generate a first type of label embedding based on the plurality of first type sensor data and the first type of cumulative pose estimation.

[0060] In some embodiments, the plurality of sensor data further includes a plurality of second-type sensor data, and the odometer embedding generation module includes: a second-type tag embedding generation module configured to generate a second-type tag embedding based on the plurality of second-type sensor data; a first-type embedding generation module configured to generate a first-type embedding based on the first type; a second-type embedding generation module configured to generate a second-type embedding based on the second type; and a type embedding usage module configured to generate a fused odometer embedding based on the first-type tag embedding, the first-type embedding, the second-type tag embedding, and the second-type embedding.

[0061] In some embodiments, the odometer embedding generation module includes: a multi-set sensor data acquisition module configured to acquire multiple sets of sensor data over multiple time periods, wherein one set of sensor data in the multiple sets of sensor data includes the multiple sets of sensor data; a multi-set tag embedding generation module configured to generate multiple sets of tag embeddings based on the multiple sets of sensor data; and a multi-set tag embedding usage module configured to generate the fused odometer embedding based on the multiple sets of tag embeddings.

[0062] In some embodiments, the apparatus 700 further includes: a sensor embedding generation module configured to generate multiple sensor embeddings corresponding to the multiple types using the attention-based machine learning model based on the multiple labeled embeddings; and a failed sensor determination module configured to determine a failed sensor among the multiple sensors based on the multiple sensor embeddings.

[0063] In some embodiments, the apparatus 700 further includes a prompting information generation module configured to generate a prompting information indicating the failed sensor in response to determining the failed sensor among the plurality of sensors.

[0064] In some embodiments, the apparatus 700 further includes: a training data acquisition module configured to acquire multiple training data from the plurality of sensors; a training label embedding generation module configured to generate multiple training label embeddings based on the plurality of training data; a first label embedding masking module configured to generate masked multiple training label embeddings by masking a portion of the plurality of training label embeddings; and a pre-training module configured to pre-train the attention-based machine learning model based on the plurality of training label embeddings and the masked multiple training label embeddings.

[0065] In some embodiments, the first label embedding masking module includes a second label embedding masking module configured to generate the masked plurality of training label embeddings by masking all training label embeddings associated with a particular type of sensor over a specific time period.

[0066] In some embodiments, the first label embedding masking module includes: a noise generation module configured to generate noise based on abnormal sensor data from a failed sensor; and a third label embedding masking module configured to generate the masked plurality of training label embeddings by adding the noise to a portion of the plurality of training label embeddings.

[0067] In some embodiments, the first label embedding masking module includes a fourth label embedding masking module configured to generate the masked plurality of training label embeddings by masking all training label embeddings associated with a particular type of sensor.

[0068] In some embodiments, the apparatus 700 further includes: a labeled pose acquisition module configured to acquire a plurality of labeled poses for the plurality of training data; and a labeled pose usage module configured to fine-tune the pre-trained attention-based machine learning model based on the plurality of training data and the plurality of labeled poses.

[0069] It is understood that by utilizing the apparatus 700 of this disclosure, at least one of the many advantages achievable by the methods or processes described above can be realized. For example, by utilizing the apparatus 700, attention-based machine learning models can better understand and fuse multiple sensor data from various types of sensors. Furthermore, the model can dynamically adjust the weights of various types of sensor data according to changes in the individual sensor data, thereby improving the accuracy and robustness of the generated odometer information.

[0070] Figure 8 shows a block diagram of a device 800 capable of implementing various embodiments of the present disclosure. Device 800 may, for example, be a control unit 102 as shown in Figure 1. As shown in Figure 8, device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. Various programs and data required for the operation of device 800 may also be stored in RAM 803. The CPU / GPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804. Although not shown in Figure 8, device 800 may also include a coprocessor.

[0071] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0072] The various methods or processes described above can be executed by CPU / GPU 801. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU / GPU 801, one or more steps or actions in the methods or processes described above can be performed.

[0073] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0074] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0075] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0076] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0077] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0078] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0080] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating odometry information for a robot, comprising: Acquire multiple sensor data from multiple sensors of various types; Based on the data from the multiple sensors, generate multiple tag embeddings corresponding to the multiple types; Based on the multiple labeled embeddings, an attention-based machine learning model is used to generate a fused odometry embedding; as well as The odometer information is generated based on the fused odometer embedding.

2. The method of claim 1, wherein the plurality of sensors includes at least two of lidar, wheel odometer, inertial measurement unit, and camera.

3. The method of claim 1, wherein the plurality of sensor data includes a plurality of sensor data of a first type, and generating the plurality of tag embeddings corresponding to the plurality of types based on the plurality of sensor data includes: A first type of cumulative pose estimation is generated based on the multiple first type of sensor data; as well as The first type of labeled embedding is generated based on the cumulative pose estimation of the first type.

4. The method of claim 3, wherein generating the first type of labeled embedding based on the first type of cumulative pose estimation comprises: The first type of label embedding is generated based on the multiple first type of sensor data and the first type of cumulative pose estimation.

5. The method of claim 3, wherein the plurality of sensor data further comprises a plurality of second-type sensor data, and generating the fused odometry embedding using the attention-based machine learning model based on the plurality of labeled embeddings comprises: A second type of tag embedding is generated based on the multiple second-type sensor data; Generate a first type embedding based on the first type; Generate a second type of embedding based on the second type; as well as Based on the first type of tag embedding, the first type of embedding, and the second type The first type of marker embedding and the second type of embedding are used to generate a fused odometer embedding.

6. The method of claim 1, wherein generating the fused odometry embedding using the attention-based machine learning model based on the plurality of labeled embeddings comprises: Acquire multiple sets of sensor data over multiple time periods, wherein one set of sensor data includes all the multiple sets of sensor data. Multiple sets of tag embeddings are generated based on the aforementioned multiple sets of sensor data; as well as The fused odometer embedding is generated based on the multiple sets of labeled embeddings.

7. The method according to claim 1, further comprising: Based on the multiple labeled embeddings, the attention-based machine learning model is used to generate multiple sensor embeddings corresponding to the multiple types; as well as The failure sensor among the multiple sensors is determined based on the embedding of the multiple sensors.

8. The method according to claim 7, further comprising: In response to identifying the failed sensor among the plurality of sensors, a prompt message indicating the failed sensor is generated.

9. The method according to claim 1, wherein the training process of the attention-based machine learning model includes: Acquire multiple training data from the multiple sensors; Multiple training label embeddings are generated based on the multiple training data; Multiple masked training label embeddings are generated by masking a portion of the multiple training label embeddings; as well as The attention-based machine learning model is pre-trained based on the multiple training label embeddings and the multiple masked training label embeddings.

10. The method of claim 9, wherein generating the masked plurality of training label embeddings by masking a portion of the plurality of training label embeddings comprises: The masked training label embeddings are generated by masking all training label embeddings associated with a specific type of sensor within a specific time period.

11. The method of claim 9, wherein generating the masked plurality of training label embeddings by masking a portion of the plurality of training label embeddings comprises: Noise is generated based on abnormal sensor data from failed sensors; as well as The masked training label embeddings are generated by adding the noise to a portion of the plurality of training label embeddings.

12. The method of claim 9, wherein generating the masked plurality of training label embeddings by masking a portion of the plurality of training label embeddings comprises: The masked training label embeddings are generated by masking all training label embeddings associated with a specific type of sensor.

13. The method according to claim 9, wherein the training process of the attention-based machine learning model further comprises: Obtain multiple labeled poses for the multiple training data; as well as The attention-based machine learning model is fine-tuned based on the multiple training data and the multiple labeled poses.

14. An apparatus for generating odometry information for a robot, comprising: The sensor data acquisition module is configured to acquire multiple sensor data from multiple sensors of various types; The tag embedding generation module is configured to generate multiple tag embeddings corresponding to the multiple types based on the multiple sensor data; The odometer embedding generation module is configured to generate a fused odometer embedding based on the multiple labeled embeddings using an attention-based machine learning model. as well as The odometer information generation module is configured to generate the odometer information based on the fused odometer embedding.

15. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 13.

16. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 13.

Images