A method for top-view pose estimation of indoor mobile robots based on distilled large models

Abstract

The present invention discloses a method for estimating the top-view pose of an indoor mobile robot based on a distilled large model, which relates to the field of computer vision, and specifically comprises the following steps: using a sensor to collect an RGB image of an indoor ceiling view; then, distilling a CLIP large model, and after inputting the original RGB image, obtaining the text or semantic information corresponding to each image as a discrete feature. Next, the visual odometer network is trained to obtain the estimated pose features corresponding to each image; finally, the discrete features of each RGB image and the corresponding estimated pose features are fused to obtain fused features containing semantic information and pose information, and the global feature information is aggregated through the class token, added to the beginning of the patch, and input into the MLP network with the real pose as a label, and supervised training is performed to output the 6-DoF of the final camera estimated pose; after relative pose conversion, the global coordinates are restored to obtain trajectory prediction. The present invention realizes reliable and accurate pose estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for estimating the top-view pose of an indoor mobile robot based on a distilled large model. Background Art

[0002] With the continuous advancement of computer science, simulating human perception has become an important research field.

[0003] Accurate positioning is key to autonomous navigation and environmental perception. In order to achieve the successful application of mobile robots and other autonomous navigation systems, it is necessary to accurately estimate their position and attitude in space. Accurate position and attitude or odometer is a key element to achieve precise positioning, which is particularly important for mobile robots and other applications that require autonomous positioning and navigation technology.

[0004] Ensuring that a robot or moving object always knows its position accurately is the basis for autonomous navigation. Although researchers and engineers have developed a variety of technologies for positioning, including wheel odometers, lidar odometers, global positioning systems (GPS), inertial navigation systems (INS) and visual odometers (VO), these technologies currently have some shortcomings that need to be addressed:

[0005] Among them, VO visual positioning technology uses the camera to obtain image data to infer the position and posture of the robot or drone. There are currently two types of visual odometers: based on traditional methods and based on deep learning methods. The former has better pose estimation results in clear scenes, but the pose calculation results will be seriously degraded in challenging scenes. In comparison, the deep learning-based method has more powerful potential when facing challenging scenes.

[0006] In recent years, large model pre-training technology has been widely used. Large model pre-training can usually provide better generalization performance and convert complex large models into lightweight models to adapt to their own tasks. By extracting knowledge from pre-trained large models to help models such as Convolutional Neural Networks (CNN) and Vision Transformer (ViT) learn better visual representations and achieve better task performance, it can help overcome the limitations of traditional positioning technology.

[0007] At present, VO mainly uses forward-looking cameras to collect image data of real scenes. However, in real life scenes, it often faces a large number of dynamic moving targets and occlusion problems, such as crowded pedestrians in shopping malls, or objects blocking the camera. This will seriously affect the precise positioning of the mobile robot, resulting in the problem of positioning loss. Summary of the invention

[0008] Taking into account the actual crowded or dynamic obstacle scenarios, the present invention proposes a method for top-view pose estimation of an indoor mobile robot based on a distilled large model. By using the ceiling view and the pre-trained weights of the CLIP large model, a deep learning-based algorithm framework is designed to realize robot positioning and assist the visual odometry in pose estimation, thereby achieving reliable and accurate pose estimation.

[0009] The following steps are involved:

[0010] Step 1: Install the sensor on the robot and point it toward the ceiling to collect the RGB image of the indoor ceiling view;

[0011] Step 2: Distill the CLIP large model and input the original RGB image to obtain the text or semantic information corresponding to each image as discrete features.

[0012] The CLIP large model consists of two modules: Text Encoder and Image Encoder, which extract text and image features respectively, and then use the contrastive learning algorithm to let the model learn the text-image matching relationship.

[0013] Select part of the original RGB image and distill the CLIP large model:

[0014] First, the current original RGB image is divided into several sub-regions Patches using the ViT algorithm to form a sequence Encode the corresponding feature sequence

[0015] The size of the patch is adjusted according to the requirements of the specific task and dataset; N represents the number of patches in a single image;

[0016] Then, for the feature h of the i-th Patch i , find the nearest feature from the codebook, and calculate the relationship between all embedded features in the codebook and the feature h i The minimum distance between the features h i The mean square error with the nearest feature is used as the codebook embedding loss L emb :

[0017]

[0018] The embedding feature of the codebook is represented as {e 1 ,...e j ,...e |v|}.

[0019] Next, the feature h of the i-th patch i After clipping the large model, output feature t i , calculate the feature h i The nearest neighbor feature o found in the codebook i And the output feature t i The cosine value of all features is taken as the distillation feature reconstruction loss L rec :

[0020]

[0021] Finally, the loss function of the entire network is obtained:

[0022]

[0023] Step 3: Train the visual odometry network and input the original RGB image to obtain the estimated pose features corresponding to each image;

[0024] The loss function L is used during training vo is the mean square error between each predicted element and its true pose, defined as follows:

[0025]

[0026] Where B is the number of samples selected for training, and T is the number of frames from which a sample is selected. i,n is the true value of the 6-DOF pose of the camera in the original RGB image, γ i,n is the predicted value output by the visual odometry network.

[0027] Step 4: Fuse the discrete features of each RGB image with the corresponding estimated pose features to obtain fused features containing semantic information and pose information;

[0028] Step 5: Aggregate the fused features through the class token to perform global feature information, collect information from all patches, add it to the beginning of the patch, and use the real pose as the label;

[0029] Step 6: Input the features with the true pose labels into the MLP network for supervised training and output the 6-DoF of the final camera estimated pose.

[0030] Step 7: Perform relative pose transformation on the 6-DoF of the camera’s estimated pose to restore the global coordinates and perform trajectory prediction.

[0031] The advantages of the present invention are:

[0032] A method for top-view pose estimation of an indoor mobile robot based on a distilled large model avoids dense crowds or dynamic obstacles in indoor scenes through a ceiling dataset. The distilled large model can be used to train the network without manually labeled labels, providing semantic prior information for the pose estimation network. Finally, reliable and accurate pose estimation is achieved through a pose estimation network based on deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a method for estimating top-view pose of an indoor mobile robot based on a distilled large model according to the present invention;

[0034] Figure 2 The real 6-DOF pose data representation of the ceiling data set collected by the continuous time frame of the present invention;

[0035] Figure 3 The VQKD algorithm network structure of the distillation CLIP large model of the present invention;

[0036] Figure 4 This is the VO network structure of the trajectory prediction of the present invention. DETAILED DESCRIPTION

[0037] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail and in depth below with reference to the accompanying drawings.

[0038] This paper proposes a method for estimating the top-view pose of an indoor mobile robot based on a distilled large model. The method extracts features by performing unsupervised training on a continuous image frame sequence with the help of the CLIP large model, and uses the monocular visual odometry as an understanding task in conjunction with downstream tasks to estimate the 6-DOF pose of the camera through supervised training. Figure 1 As shown, the following steps are included:

[0039] Step 1: Install the sensor on the robot and point it toward the ceiling to collect the RGB image of the indoor ceiling view;

[0040] The real pose information is manually calibrated through laser and map matching.

[0041] Step 2: Distill the CLIP large model and input the original RGB image to obtain the text or semantic information corresponding to each image as discrete features.

[0042] The CLIP (Contrastive Language–Image Pre-training) large model is a pre-training method based on contrasting text-image pairs. It specifically includes two modules, Text Encoder and Image Encoder, which extract text and image features respectively, and then use the contrastive learning algorithm to let the model learn the matching relationship between text and image.

[0043] The CLIP large model discretizes the semantic space through the VQKD algorithm to obtain discrete features of each RGB image; in the prior art, large-scale data (400 million text-image pairs) is usually used for training. Based on massive data, the CLIP large model can learn more general visual semantic information to provide help for downstream tasks; however, the amount of data required is too large; in the present invention, the distilled CLIP large model is used, and there is no need to manually obtain large-scale image texts for training. The corresponding semantic features can be obtained from small-scale image data, and the semantic features are discrete.

[0044] Specifically, some original RGB images are selected and the CLIP large model is distilled. The loss function L used is:

[0045] First, the current original RGB image X∈R T×C×H×W , which consists of T frames of size C×H×W, is decomposed into N sub-region patches of size P×P, and N=HW / P 2 non-overlapping blocks, forming a sequence Send it to the encoder of the ViT model to get the corresponding feature sequence

[0046] The size of the patch is adjusted according to the requirements of the specific task and dataset; N represents the number of patches in a single image;

[0047] Then, for the feature h of the i-th Patch i , find the nearest neighbor feature from the codebook, and calculate the embedding features of all features in the codebook and the feature h i The minimum distance between the features h i The mean square error with the nearest feature is used as the codebook embedding loss L emb :

[0048]

[0049] The embedding feature of the codebook is represented as {e 1 ,...e j ,...e |v|}.

[0050] Next, the feature h of the i-th patchi After clipping the large model, output feature t i , calculate the feature h i The nearest neighbor feature o found in the codebook i And the output feature t i The cosine value of all features is taken as the distillation feature reconstruction loss L rec :

[0051]

[0052] Finally, the loss function of the entire network is obtained:

[0053]

[0054] Step 3: Train the visual odometry network and input the original RGB image to obtain the estimated pose features corresponding to each image;

[0055] The loss function L is used during training vo is the mean square error between each predicted element and its true pose, defined as follows:

[0056]

[0057] Where B is the number of samples selected for training, and T is the number of frames from which a sample is selected. i,n is the true value of the 6-DOF pose of the camera in the original RGB image, γ i,n is the predicted value output by the visual odometry network.

[0058] Step 4: Fuse the discrete features of each RGB image with the corresponding estimated pose features to obtain fused features containing semantic information and pose information;

[0059] Since a single RGB image is first encoded by ViT, the dimensions of the obtained discrete features and estimated pose features are N*768 and N*32 respectively. The Concat operation is performed to splice them into vector features with a dimension of N*800.

[0060] Based on the existing Tsformer_vo network, it is improved by integrating relevant semantic features, which improves the network positioning accuracy.

[0061] Step 5: Aggregate the fused features through the class token to perform global feature information, collect information from all patches, add it to the beginning of the patch, and use the real pose as the label;

[0062] The feature of each patch after encoding in the Vit algorithm is called a patch token, which aggregates global information to a certain extent, but is mainly the own feature of each patch. The class token is randomly initialized and continuously updated as the network is trained, which can encode the statistical characteristics of the entire data set; the class token aggregates the information on all other patch tokens (global feature aggregation), and because it is not based on the image content itself, it can avoid bias towards a specific patch token). Collect information from all patches and add it to the beginning of the patch.

[0063] Step 6: Input the features with the true pose labels into the MLP network for supervised training and output the 6-DoF of the final camera estimated pose.

[0064] Step 7: Perform relative pose transformation on the 6-DoF of the camera’s estimated pose to restore the global coordinates and perform trajectory prediction.

[0065] The global coordinates are recovered by mathematically inverting the matrix, converting the Euler angles back into a rotation matrix, and reusing the statistics used in normalizing the data.

[0066] Example:

[0067] Step 1: Dataset preparation: The study intends to use the laboratory to manually collect RGB images of indoor ceiling views. Specifically, the sensor is installed on the robot and pointed at the ceiling to take RGB pictures.

[0068] The reason why the ceiling dataset is prepared in step 1 is that VO currently mainly uses forward-looking cameras to collect image data of real scenes. However, in real life scenes, there are often a large number of dynamic moving targets and occlusion problems, which will seriously affect the precise positioning of the mobile robot, resulting in the problem of positioning loss. Since static targets usually exist in indoor ceiling views, the problem of dynamic targets affecting pose estimation can be avoided. In addition, by adjusting the height of the upward camera, the problem of the camera being blocked by obstacles in front can also be solved. Therefore, the problem of positioning loss caused by moving targets or camera occlusion can be solved through ceiling views. At present, there is no suitable ceiling dataset available in the public data set. Therefore, based on the RGB camera, the upward ceiling view is collected in large indoor environments such as shopping malls and teaching buildings by setting appropriate angles and directions.

[0069] Step 2: Distill the large model to obtain discrete features: Distill the clip large model and convert the complex large model into a lightweight model to adapt to the task. By extracting knowledge from the pre-trained large model, it helps models such as CNN or ViT learn better visual representations and achieve better task performance.

[0070] In step 2, the large model is distilled and the VQKD algorithm is introduced, which discretizes the semantic space. Without manually labeling any labels, the loss function can be used to optimize the learning objectives and obtain the text or semantic information corresponding to the image. In this process, the loss function L is defined as:

[0071] L=L emb +L rec

[0072] Where L emb is the codebook embedding loss, that is, the loss in the process of converting continuous features into continuous features, which is determined by the minimum cosine distance. rec To distill the feature reconstruction loss, VQ-KD uses CLIP as the teacher system, and then uses the features generated by CLIP as the output optimization target to train the model. i Represents the features generated by the teacher system on the i-th Patch.

[0073] Step 3: Supervised learning of the visual odometry network to estimate pose: The preprocessed two consecutive frames are concatenated with the features obtained by the encoder-decoder of the Transformer model and the discrete features obtained in step 2, and all feature information is collected through a special vector called class token (cls), and through the final MLP head, an estimated 6-DoF pose is output.

[0074] Step 4: Post-processing: The pose is estimated by the visual odometry network, and the global coordinates are restored from the predicted relative coordinates. The global coordinates are restored by mathematical methods such as inverse matrix, converting Euler angles back to rotation matrix, and reusing the statistics used in the normalized data.

[0075] Step 1 first prepares the upward ceiling dataset. The dataset images are taken manually, and the data text information is calibrated and edited. The acquisition device is an RGB camera. The camera is installed on the robot, and the camera points to the ceiling. The resolution is 640*480 and the acquisition frequency is 20Hz. The real pose of the dataset is obtained manually, and the real pose value is obtained by laser and map matching. Including tx, ty, tz (3 floating point numbers) to give the position of the optical center of the camera relative to the world origin defined by the motion capture system; use roll (Roll) around the X-axis, pitch (Pitch) around the Y-axis and yaw (Yaw) around the Z-axis, that is, R, P, Y to describe the posture change of the object or robot, indicating the rotational motion of the object or robot around the three coordinate axes. Examples of datasets obtained are as follows: Figure 2 As shown in the table below. The camera calibration parameters are also given, including the focal length (fx / fy) and optical center (cx / cy) of each camera.

[0076] Picture Frames tx ty tz R P Y 2277 15.0715 -6.49108 0 0 0 2.77639 2278 14.9782 -6.45463 0 0 0 2.76628 2279 14.9005 -6.4255 0 0 0 2.75865 2280 14.871 -6.41539 0 0 0 2.75434 2281 14.8169 -6.39291 0 0 0 2.74946 2282 14.7582 -6.36962 0 0 0 2.74443 2283 14.6716 -6.33337 0 0 0 2.74327 2284 14.6045 -6.30605 0 0 0 2.74665

[0077] The specific network architecture in step 2 is as follows Figure 3 As shown, CLIP is used as a teacher network. It is a powerful multimodal model that can understand images and semantic text at the same time. First, download the pre-trained CLIP model weights and use its rich knowledge learned on large-scale data. Next, use the codec to input the RGB image into the CLIP model and obtain the encoding vector of the image. Then, optimize by maximizing the cosine similarity between the codec output vector and the teacher model output vector. This process aims to enable the model of this embodiment to learn a representation similar to the teacher model, thereby capturing the correlation between images and semantics. In this way, discrete features of the image are obtained, which have meaningful representations in the semantic space. This enables the model of this embodiment to better understand images and show better performance in specific tasks.

[0078] Step 3 is as follows: First, data preprocessing, input X∈R T×C×H×W , normalized by subtracting the mean and dividing by the standard deviation.

[0079] The preprocessed data consists of T frames of size C×H×W. Each fragment is decomposed into N patches of size P×P, and N=HW / p 2Non-overlapping blocks. It is fed into the Transformer encoder and the features of each frame are obtained and concatenated with the corresponding discrete features of each frame. The feature information from the concatenation is collected through a special vector of class tokens (cls), where class tokens (cls) are usually added to the beginning of the patch sequence to learn the information features of the entire picture frame. The information of all patches learned by the class tokens (cls) is passed through the MLP head, and the real pose is used as the label through supervised training to finally output the 6-DoF of the estimated pose.

[0080] For the real pose data, the real pose data is converted into relative coordinates. In the visual odometer, the relative motion matrix Tk∈R4×4 is defined as:

[0081]

[0082] where k is the current time step, R∈SO(3) is the rotation matrix describing the camera rotation, and t∈R 3×1 is the translation vector. Both the rotation matrix and the translation vector describe the motion from time k-1 to k. To estimate the relative motion between consecutive frames, assume that there is a ground local coordinate motion T k-1 and T k The camera’s posture at time k relative to time k-1 is denoted by T k-1，k , is given by:

[0083]

[0084] For the rotation matrix R∈SO(3), the rotations around the x, y, and z axes are called roll, pitch, and yaw, and are denoted by α, θ, and The rotation matrix R is expressed using these rotations as follows:

[0085]

[0086] Euler angles are expressed as follows:

[0087] α=atan2(r 21 ,r 11 ),

[0088]

[0089]

[0090] The entire pose estimation network structure is shown in the figure Figure 4As shown. Specifically, the Transformer encoder includes a time domain encoder and a spatial encoder. As the name implies, the spatial encoder refers to applying attention along the spatial axis on the same frame. The time domain encoder uses markers with the same spatial index to apply attention along the time axis. The MLP head outputs the final estimated pose. Since two frames are required to estimate a pose, the T-1 pose is estimated from the T frame. The simplified relationship of the entire network structure is as follows:

[0091]

[0092] The encoder blocks include layer normalization (LN), multi-head self-attention (MHSA), residual connections, fully connected layers (FC), and multi-layer perceptrons (MLP).

[0093] The entire network is supervised by training the real relative pose and the estimated relative pose. The loss function L used in this work is vo It is the mean square error between each predicted element and its true pose. The final estimated result is the relative pose between consecutive frames.

[0094] Step 4 is as follows: first, restore the six-degree-of-freedom relative pose results to the value before normalization through mathematical calculation; second, convert the Euler angle into a rotation matrix to construct a rotation and translation matrix; then, multiply the absolute pose matrix of the previous frame by the rotation and translation matrix to obtain the absolute pose matrix of this frame; finally, draw the real two-dimensional coordinate trajectory based on the translation vector result in the absolute pose matrix.

Claims

1. A method for estimating the top-view pose of an indoor mobile robot based on a distilled large model, characterized in that: Specifically include: Step 1: Install the sensor on the robot and point it toward the ceiling to collect the RGB image of the indoor ceiling view; Step 2: Distill the CLIP model and input the original RGB image to obtain the text or semantic information corresponding to each image as discrete features; Distill the CLIP large model, specifically: First, the current original RGB image is divided into several sub-regions Patches using the ViT algorithm to form a sequence Encode the corresponding feature sequence The size of the patch is adjusted according to the requirements of the specific task and dataset; N represents the number of patches in a single image; Then, for the feature h of the i-th Patch i , find the nearest feature from the codebook, and calculate the relationship between all embedded features in the codebook and the feature h i The minimum distance between the features h i The mean square error with the nearest feature is used as the codebook embedding loss L emb : The embedding feature of the codebook is represented as {e1,...e j ,...e |v| }; Next, the feature h of the i-th Patch i After clipping the large model, output feature t i , calculate the feature h i The nearest neighbor feature o found in the codebook i And the output feature t i The cosine value of all features is taken as the distillation feature reconstruction loss L rec : Finally, the loss function of the entire network is obtained: Step 3: Train the visual odometry network and input the original RGB image to obtain the estimated pose features corresponding to each image; The loss function L is used during training vo is the mean square error between each predicted element and its true pose, defined as follows: Where B is the number of samples selected for training, T is the number of frames selected for a sample, and y i,n is the true value of the 6-DOF pose of the camera in the original RGB image, γ i,n is the predicted value output by the visual odometry network; Step 4: Fuse the discrete features of each RGB image with the corresponding estimated pose features to obtain fused features containing semantic information and pose information; Step 5: Aggregate the fused features through the class token to perform global feature information, collect information from all patches, add it to the beginning of the patch, and use the real pose as the label; Step 6: Input the features with the true pose labels into the MLP network for supervised training and output the 6-DoF of the final camera estimated pose.

2. The method for estimating the top view pose of an indoor mobile robot based on a distilled large model as claimed in claim 1, characterized in that: The CLIP large model includes two modules, Text Encoder and Image Encoder, which extract text and image features respectively, and then let the model learn the text-image matching relationship based on the contrastive learning algorithm; The CLIP large model discretizes the semantic space through the VQKD algorithm to obtain the discrete features of each RGB image.

3. The method for estimating the top view pose of an indoor mobile robot based on a distilled large model as claimed in claim 1, characterized in that: The fusion feature refers to a vector feature obtained by performing a Concat operation on the discrete features obtained from the RGB image and the estimated pose features.

Images