A streaming control method for mobile robot visual odometry based on deep learning

Abstract

The present invention relates to a streaming control method for a mobile robot visual odometry based on deep learning, and belongs to the technical field of computer vision and positioning control. First, the RGB image and Depth image of the scene are obtained and aligned. The mobile target in the scene is identified by a method combining semantic segmentation and target detection, and a mask is generated. Afterwards, the visual odometry is used to estimate the depth camera pose, and the depth camera predicted pose is obtained using a prediction method and output continuously. The system is then set to parallel execution of threads, and the information shared between threads is set as shared variables. Finally, the streaming performance of the system is calculated and a streaming evaluation is performed. Based on the evaluation results, the visual navigation system is optimized and controlled. This method can obtain continuous and stable depth camera pose output, meet the streaming processing control requirements of the visual odometry, and obtain better streaming performance than the non-real-time visual odometry based on considering the algorithm time delay.

Description

Technical Field

[0001] The present invention relates to a mobile robot visual odometer streaming processing method, and in particular to a mobile robot visual odometer streaming processing method based on deep learning, belonging to the field of computer vision and positioning control technology. Background Art

[0002] In recent years, odometry has been widely used in fields such as robotics and autonomous vehicles. Odometry technology is a key enabler for mobile robotic systems, autonomous vehicle systems, and other systems to achieve real-time positioning using sensors. It is fundamental to path planning, autonomous navigation, and other tasks. Computer vision technology plays an increasingly important role in navigation and positioning. Visual odometry (VO) is primarily responsible for positioning, estimating the pose of a depth camera (a camera that can detect the physical distance between objects in a scene) using data collected by sensors.

[0003] VO is often combined with deep learning methods, leveraging semantic information from the scene to improve the positioning accuracy of mobile robots' visual navigation systems in dynamic environments. However, deep learning-based VO requires high computing power and often requires high-performance graphics processing units (GPUs) for acceleration to achieve high-precision, high-frame-rate real-time processing and operation. Due to size and power consumption limitations, this hinders the widespread adoption of VO on mobile devices and prevents them from meeting the requirements of streaming processing.

[0004] Researchers at home and abroad have done a lot of work on deep learning-based visual odometry, a problem with high application value. Generally, according to different deep learning methods, deep learning-based visual odometry can be roughly divided into two categories: visual odometry based on object detection and visual odometry based on semantic segmentation. Among them, visual odometry based on object detection is mainly based on semantic monocular SLAM in dynamic street scene based on multiple object tracking[C] / / 2017IEEE International Conference on Cybernetics and Intelligent Systems(CIS)and IEEE Conference on Robotics,Automation and Mechatronics(RAM).IEEE,2017:599-604.) and dynamic SLAM based on object detection and improved octomap[J].IEEE Access, 2018, 6:75545-75559.) The former uses object detection rectangles combined with prior knowledge to identify dynamic objects in the scene, then removes all feature points within the rectangle before performing visual odometry. However, because the rectangle is larger than the dynamic object, static feature points around the target are mistakenly removed. Furthermore, the real-time performance of the object detection network is limited. The latter uses the Yolo object detection network to obtain semantic information and remove unstable feature points on moving objects, but it cannot effectively remove feature points of potentially dynamic objects.Visual odometry based on semantic segmentation is represented by DS-SLAM (Yu C, Liu Z, Liu XJ, et al. DS-SLAM: A semantic visual SLAM towards dynamic environments[C] / / 2018IEEE / RSJ International Conference on IntelligentRobots and Systems(IROS). IEEE, 2018: 1168-1174.) and DynaSLAM (Bescos B, Fácil JM, Civera J, et al. DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes[J]. IEEE Robotics and Automation Letters, 2018, 3(4): 4076-4083.). The former uses the SegNet semantic segmentation network and motion consistency check to remove dynamic feature points, and uses the remaining feature points for visual odometry to estimate the camera pose, which can construct a semantic octree map. However, semantic segmentation is time-consuming and cannot achieve real-time results. The latter uses the Mask RCNN semantic segmentation network and multi-view geometry method to eliminate dynamic feature points and complete background repair in the absence of dynamic targets, but it cannot be processed in real time due to the slow network inference speed.

[0005] This shows that when combined with deep learning methods, visual odometry can achieve high positioning accuracy and stability in dynamic scenes. However, deep learning significantly increases the amount of computation required, resulting in poor real-time performance and the inability to implement streaming processing. Summary of the Invention

[0006] The present invention addresses the problems and shortcomings of existing technologies and proposes a streaming computation method to address the poor real-time and stability of mobile robot visual navigation systems, which prevents them from implementing streaming control. This method enables the system to obtain continuous and stable camera pose output, meeting the requirements of visual odometry streaming. This method facilitates deployment on relatively low-performance hardware devices and, while accounting for algorithmic time delays, achieves better streaming performance than non-real-time visual odometry systems.

[0007] The technical solutions adopted in the present invention are as follows.

[0008] First, obtain the RGB image and Depth image of the scene and perform registration;

[0009] Then, by combining semantic segmentation and target detection, the moving targets in the scene are identified and a mask is generated;

[0010] After that, the visual odometry is used to estimate the camera pose, and based on the estimated depth camera pose, the prediction method is used to obtain the depth camera predicted pose and output it continuously;

[0011] Then, the system is set up for parallel execution of threads, and the information shared between threads is set up as shared variables;

[0012] Finally, the system's flow performance is calculated and evaluated. Based on the evaluation results, the visual navigation system is optimized and controlled to implement autonomous positioning and path navigation of the mobile robot.

[0013] It should be noted that the method of the present invention is not limited to application in mobile robot control, but can also be applied to intelligent equipment systems such as drones, unmanned vehicles, and automatic driving.

[0014] Beneficial effects

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] 1. Traditional deep learning-based visual odometry methods estimate the pose of a moving camera in non-real time. The entire process is a three-step, sequential process: first, inputting the scene's RGB and depth maps, then using deep learning methods to identify dynamic objects, and finally, using visual odometry to estimate and output the camera pose. This sequential approach results in low system efficiency. Furthermore, the deep learning framework relies on high-performance graphics processing unit (GPU) acceleration. Without hardware support, the system runtime will be significantly extended.

[0017] The present invention overcomes the above problems by modularizing the system into three threads: main thread, mask thread and track thread. Each thread is responsible for different tasks, and the threads share information and execute in parallel, greatly improving the efficiency and real-time performance of the system.

[0018] 2. Traditional deep learning-based visual odometry methods obtain very accurate camera poses on offline video datasets, but they do not consider the algorithm's time delay and cannot implement streaming processing. When using streaming accuracy indicators that balance time delay and accuracy to evaluate system performance, the results are found to be poor.

[0019] The present invention realizes streaming processing, and ensures that the system can continuously and stably output the camera pose through initialization and prediction of the camera pose, runs in a streaming manner, and can be carried on actual depth cameras and portable computing devices for application in industrial production.

[0020] 3. Traditional deep learning-based visual odometry methods do not require real-time camera pose output and directly use the estimated camera pose as the system output. This is not suitable for streaming systems because the output pose of the visual odometry is always "outdated" at the current moment. If the "outdated" time interval is short, it can be assumed that the camera motion is not large, and the visual odometry output pose can be directly used as the system output. However, if the "outdated" time interval is long, then directly outputting the result is not reasonable.

[0021] The present invention uses different prediction methods to predict the camera pose at the current moment by judging the length of the "outdated" time interval, thereby reducing the inaccuracy of the output pose caused by "outdated" and making the predicted pose closer to the actual camera pose. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the method of the present invention.

[0023] Figure 2 In the figure, Figure a is the original RGB image of the balloon in the BONN dataset, and Figure b is the original Depth image of the balloon in the BONN dataset.

[0024] Figure 3 In the figure, Figure a is the original RGB image captured by the Kinect V2 depth camera, Figure b is the original Depth image captured by the Kinect V2 depth camera, and Figure c is the Depth image after registration.

[0025] Figure 4 This is a schematic diagram of the mask result obtained by combining semantic segmentation and target detection to identify the dynamic target of the balloon in the BONN dataset. DETAILED DESCRIPTION

[0026] The specific implementation of the method of the present invention is described in detail below with reference to the accompanying drawings and examples.

[0027] like Figure 1 As shown, a mobile robot visual odometry streaming processing method based on deep learning includes the following steps:

[0028] Step 1: Get the RGB image and Depth image (depth image) of the scene and align them.

[0029] Specifically, first, obtain from the dataset or use a depth camera to collect RGB images, depth images, and true values of the depth camera pose with the same timestamp.

[0030] Then, the RGB image and Depth image mapping matrices are calculated, and the pixels with missing depth values are filled to obtain the aligned RGB image and Depth image. The calibration method is used to calculate the true pose of the depth camera.

[0031] Step 2: Identify dynamic targets in the scene image and generate a mask.

[0032] Perform pixel-level semantic segmentation on the image to obtain a binary mask image of potential moving targets in the scene. Perform object detection on the image to obtain the bounding rectangle and category label of the moving target.

[0033] Based on the binary mask output by semantic segmentation, the target detection results are further optimized. For all rectangular boxes, the pixel values inside the box are kept unchanged, and the pixel values outside the box are assigned to 0.

[0034] Step 3: Use visual odometry to estimate the camera pose.

[0035] Specifically, a grayscale image is generated based on the RGB image in step 1, and ORB (Oriented FAST and Rotated BRIEF) feature points are extracted.

[0036] According to the mask in step 2, remove the ORB feature points covered by the mask and the mask boundary. Use the uniform motion model to track between frames and estimate the depth camera pose.

[0037] Step 4: Predict the current pose of the depth camera based on the historical poses of the depth camera.

[0038] Specifically, according to the two closest camera poses obtained in step 3, the velocity matrix Velocity of the depth camera is calculated, which is expressed as:

[0039] Velocity=T nl =T nw ×T wl

[0040] Among them, T wl is the previous camera pose, T nw is the last camera pose, T nl is the velocity matrix of the camera motion.

[0041] According to the velocity matrix Velocity, predict the current camera pose T fw , expressed as:

[0042]

[0043] Among them, Δt is the sequence number of the image frame to be predicted and the next camera pose T nw The difference in the sequence numbers of the processed image frames.

[0044] Step 5: Continuously output camera poses.

[0045] When the system is turned on, the initial camera pose is output. According to step 4, after the system has obtained the camera pose at least once, it uses the prediction method to output the predicted camera pose. Ultimately, the output camera pose is represented as [tx, ty, tz, qx, qy, qz, qw], which corresponds to the camera's position in the world [tx, ty, tz] and direction [qx, qy, qz, qw], respectively. The output timestamp is also obtained. Among them, tx represents the x-axis coordinate of the camera position, ty represents the y-axis coordinate of the camera position; tz represents the z-axis coordinate of the camera position, qx represents the first value of the imaginary part of the camera direction quaternion, qy represents the second value of the imaginary part of the camera direction quaternion, qz represents the third value of the imaginary part of the camera direction quaternion, and qw represents the real value of the camera direction quaternion.

[0046] Step 6: Set the system to multi-threaded parallel mode.

[0047] According to steps 1, 4, and 5, the scene image input, depth camera pose prediction, and continuous output of camera pose are merged into the main thread.

[0048] According to step 2, semantic segmentation is performed as a separate mask thread.

[0049] According to step 3, run the visual odometry as a separate tracking thread.

[0050] Set the main thread, mask thread, and track thread to execute in parallel, and share required information between threads.

[0051] Step 7: Perform a streaming evaluation of the system.

[0052] According to the real depth camera pose and input timestamp input in step 1, it is expressed as:

[0053]

[0054] Among them, gt i is the input depth camera real pose, t i is the input timestamp, i is the input frame number, and M is the number of input frames.

[0055] According to the depth camera predicted pose and output timestamp output in step 5, its formal representation is:

[0056]

[0057] Among them, op j is the output depth camera predicted pose, s j is the output timestamp, j is the output result sequence number, N is the number of output poses, and N<M.

[0058] For each frame input i, an output result is found corresponding to it, so that the input timestamp and output timestamp meet the requirements of streaming evaluation, which is expressed as:

[0059]

[0060] Among them, μ(i) represents the starting frame number of the evaluation, is the jth output result number corresponding to the i-th frame input. Finally, the computational flow performance τ is expressed as:

[0061]

[0062] Among them, loss is the loss function for calculating the true pose and the predicted pose. represents the output corresponding to the i-th frame input. M is the number of input frames.

[0063] Based on the evaluation results, the visual navigation system is optimized and controlled to carry out autonomous positioning and path navigation of the mobile robot.

[0064] Example

[0065] Below, a specific embodiment of the method of the present invention is given, and the steps are as follows:

[0066] Step 1: Obtain the RGB image and Depth image of the scene and align them.

[0067] Obtain RGB images and depth images with the same timestamp from the dataset or use a depth camera to collect them, as well as the actual camera pose.

[0068] The RGB images and Depth images in the dataset have the same size and do not require registration operations. Figure 2 As shown in sub-figures a and b, the true camera pose can be obtained directly.

[0069] The RGB image and Depth image captured by the depth camera have different sizes, as shown in the following example: Figure 3 As shown in sub-images a and b, it is necessary to calculate the mapping matrix of the RGB image and the Depth image, and then fill the pixels with missing depth values to obtain the RGB image and Depth image of the same size, as shown in the following example: Figure 3 As shown in sub-figure c, the true camera pose can be calculated offline using the calibration method;

[0070] Step 2: Identify dynamic targets in the scene and generate a mask

[0071] Perform pixel-level semantic segmentation on the image to obtain masks and class labels for potential moving objects in the scene. Input the m×n×3 RGB image from step 1 into the semantic segmentation network, which returns an m×n×K matrix, where K is the number of identified moving objects. For each output channel k∈K, a binary mask of size m×n is generated, assigning 1 to pixels containing moving objects and 0 to pixels containing background objects. Combine the K masks into a single binary mask output.

[0072] Perform object detection on the image to obtain the bounding rectangle and category label of the moving object. Input the RGB image of size m×n×3 in step 1 into the object detection network, and return an l×5 matrix, where l is the number of dynamic objects identified. Each row of the rectangle is formally represented as:

[0073] [x′,y′,w′,h′,label]

[0074] Among them, x′, y′ are the coordinates of the upper left corner of the rectangular box, w′, h′ are the width and height of the upper left corner of the rectangular box, and label is the category label.

[0075] Based on the binary mask output by semantic segmentation, the background that is mistakenly identified as a moving target in the mask is removed in combination with the result of target detection. For all rectangular boxes, the pixel values inside the box are kept unchanged, and the pixel values outside the box are assigned to 0, and the mask result is further optimized. Figure 4 shown.

[0076] Step 3: Use visual odometry to estimate the camera pose.

[0077] Generate a grayscale image based on the RGB image in step 1, use the FAST method to extract the key points in the grayscale image, use the BRIEF (Binary Robust Independent Elementary Features) feature descriptor to describe the image area around the key points, and the key points and descriptors together constitute the ORB feature points. According to the mask in step 2, remove the ORB feature points covered by the mask and the mask boundary, and obtain the depth values of the remaining ORB feature points according to the Depth image in step 1. Use the uniform motion model for inter-frame tracking, estimate the camera pose, and use the light speed adjustment method to optimize the camera pose. The camera pose is finally represented as a 4×4 transformation matrix T wc , which is expressed as:

[0078]

[0079] Where R is the rotation matrix, which represents the rotation transformation between the two coordinate systems; t is the translation vector, which represents the translation transformation between the two coordinate systems. wc It is the transformation matrix from the camera coordinate system to the world coordinate system.

[0080] Step 4: Predict the current camera pose based on historical camera poses.

[0081] According to the two closest historical camera poses obtained in step 3, the camera's velocity matrix Velocity is calculated, which is expressed as:

[0082] Velocity=T nl =T nw ×T wl

[0083] Among them, l is last_pose, which represents the camera coordinate system of the first of the two most recent poses; n is new_pose, which represents the camera coordinate system of the second of the two most recent poses; w is world, which represents the world coordinate system; T wl is the previous camera pose, T nw is the last camera pose, T nl Is the velocity matrix of the camera motion. Predict the current camera pose T based on the velocity matrix fw , which is expressed as:

[0084]

[0085] Where f is forecast_pose, which represents the camera position to be predicted; Δt represents the difference between the image frame number to be predicted and the image frame number of the last camera pose processed. The prediction method will vary depending on the value of Δt.

[0086] Step 5: Continuously output camera poses.

[0087] When the system is just turned on, it outputs the initialized camera pose; according to step 4, when the system obtains the camera pose at least once, it uses the prediction method to output the predicted camera pose. The initialized camera pose and the predicted camera pose ensure that the system can output the camera pose continuously and stably in the overall time series. According to the predicted camera pose T fw , expressed as:

[0088]

[0089] Inverse Expressed as:

[0090]

[0091] in, Represents the transformation matrix from the camera coordinate system to the world coordinate system, that is, the direction of the camera relative to the world origin; -R T t is a 3×1 translation vector, simplified as [tx,ty,tz]; R T is a 3×3 rotation matrix, simplified as a quaternion [qx,qy,qz,qw], T represents the transpose of the matrix, and the quaternion q is formally expressed as:

[0092] q=w+xi+yj+zk=[w,v] T

[0093] Where w is the real part of the quaternion q, v is the imaginary part of the quaternion q, v = [x, y, z] T , T represents the transpose of the matrix. The conversion formula from rotation matrix to quaternion is expressed as:

[0094] R=vv T +w 2 I+2wv^+(v^) 2

[0095] Where I is the identity matrix, T represents the transpose of the matrix; v^ is an antisymmetric matrix, expressed as:

[0096]

[0097] Finally, the output camera pose is represented as a seven-tuple [tx, ty, tz, qx, qy, qz, qw], which corresponds to the position and orientation of the camera respectively. The moment of outputting the camera pose is used as the output timestamp.

[0098] Step 6: Set the system to multi-threaded parallel mode.

[0099] According to step 1, step 4, and step 5, the scene image input, camera pose prediction, and continuous output of the camera pose are merged into the same thread, which is the main thread; according to step 2, the semantic segmentation is used as a separate thread, which is the mask thread; according to step 3, the visual odometry is used as a separate thread, which is the track thread. The main thread, mask thread, and track thread are set to execute in parallel, and the required information is shared between the threads. For example, the main thread and mask thread share RGB images, the mask thread and track thread share mask masks, and the main thread and track thread share RGB images and depth images. The overall structural framework of the system is as follows: Figure 1 shown.

[0100] Step 7: Perform a streaming evaluation of the system.

[0101] According to the real camera pose and input timestamp input in step 1, its formal representation is:

[0102]

[0103] Among them, gt i is the input camera real pose, t i is the input timestamp, i is the input frame number, and M is the number of input frames.

[0104] According to the camera predicted pose and output timestamp output in step 5, its formal representation is:

[0105]

[0106] Among them, op j is the output camera predicted pose, s j is the output timestamp, j is the output result sequence number, N is the number of output poses, N<M.

[0107] For each frame input i, an output result is found corresponding to it, so that the input timestamp and output timestamp meet the requirements of streaming evaluation. Its formal expression is:

[0108]

[0109] Among them, μ(i) represents the starting frame number of the evaluation. Since at least one historical camera pose is required for prediction, the initial few frames of input have no corresponding prediction results. Therefore, the initial few frames are ignored and the initialized camera pose is not considered. The evaluation starts from μ(i); is the jth output result number corresponding to the i-th frame input. Finally, the computational flow performance τ is formally expressed as:

[0110]

[0111] Among them, loss is the loss function for calculating the true pose and the predicted pose, and the smaller the flow performance τ, the better.

[0112] Based on the evaluation results, the visual navigation system is optimized and controlled to carry out autonomous positioning and path navigation of the mobile robot.

[0113] This specification provides a specific embodiment to illustrate the present invention and its implementation. The details included in this embodiment are not intended to limit the scope of the claims, but rather to facilitate understanding of the present invention. Those skilled in the art will appreciate that various modifications, variations, and substitutions of the steps of the preferred embodiment are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiment and the accompanying drawings.

Claims

1. A mobile robot visual odometry streaming processing method based on deep learning, characterized in that: The following steps are involved: Step 1: Obtain the RGB image and Depth image of the scene and perform registration; Step 2: Identify dynamic targets in the scene image and generate a mask; Step 3: Use visual odometry to estimate the depth camera’s pose. Step 4: Predict the current pose of the depth camera based on the historical pose of the depth camera; According to the two closest poses of the depth camera obtained in step 3, the velocity matrix Velocity of the depth camera is calculated, which is expressed as: Velocity=T nl =T nw ×T wl Among them, T wl is the previous camera pose, T nw is the last camera pose, T nl is the velocity matrix of the camera motion; According to the velocity matrix Velocity, predict the current camera pose T fw , expressed as: Among them, Δt is the sequence number of the image frame to be predicted and the next camera pose T nw The difference in the sequence number of the processed image frames; Step 5: Continuously output the depth camera pose; Step 6: Set the system to multi-threaded parallel mode; According to steps 1, 4, and 5, the scene image input, depth camera pose prediction, and continuous output of the depth camera pose are merged into the main thread; According to step 2, semantic segmentation is used as a separate mask thread; According to step 3, the visual odometry is used as a separate tracking thread; Set the main thread, mask thread, and track thread to execute in parallel, and share required information between threads; Step 7: Perform streaming evaluation of the system; According to the real depth camera pose and input timestamp input in step 1, it is expressed as: Among them, gt i is the input depth camera real pose, t i is the input timestamp, i is the input frame number, and M is the number of input frames; The predicted pose and output timestamp of the depth camera output from step 5 are expressed as: Among them, op j is the output depth camera predicted pose, s j is the output timestamp, j is the output result sequence number, N is the number of output poses, and N<M; For each frame input i, an output result is found corresponding to it, so that the input timestamp and output timestamp meet the requirements of streaming evaluation, which is expressed as: Among them, μ(i) represents the starting frame number of the evaluation, is the jth output result sequence number corresponding to the i-th frame input; finally, the computational flow performance τ is expressed as: Among them, loss is the loss function for calculating the true pose and the predicted pose; represents the output result corresponding to the i-th frame input; M is the number of input frames; Based on the evaluation results, the visual navigation system is optimized and controlled to carry out autonomous positioning and path navigation of the mobile robot.

2. A mobile robot visual odometry streaming processing method based on deep learning as claimed in claim 1, characterized in that: In step 1, first obtain the true value of the RGB image, Depth image and depth camera pose with the same timestamp, including obtaining them from the dataset and using the depth camera; Then, the RGB image and Depth image mapping matrices are calculated, and the pixels with missing depth values are filled to obtain the aligned RGB image and Depth image, and then the true pose of the depth camera is calculated.

3. A mobile robot visual odometry streaming processing method based on deep learning as claimed in claim 1, characterized in that: In step 2, pixel-level semantic segmentation is performed on the image to obtain a binary mask image of potential moving targets in the scene; object detection is performed on the image to obtain the bounding rectangle and category label of the moving target; Based on the binary mask output by semantic segmentation, the target detection results are further optimized. For all rectangular boxes, the pixel values inside the box are kept unchanged, and the pixel values outside the box are assigned to 0.

4. A mobile robot visual odometry streaming processing method based on deep learning as claimed in claim 3, characterized in that: When performing target detection on an image and obtaining the bounding rectangle and category label of a moving target, the RGB image is input into the target detection network, which returns an l×5 matrix, where l is the number of dynamic targets identified. Each row of the rectangle is formally represented as: [x′,y′,w′,h′,label] Among them, x′, y′ are the coordinates of the upper left corner of the rectangular box, w′, h′ are the width and height of the upper left corner of the rectangular box, and label is the category label.

5. The method for streaming processing of mobile robot visual odometry based on deep learning according to claim 1, characterized in that: In step 3, a grayscale image is generated based on the RGB image in step 1, and ORB feature points are extracted; based on the mask in step 2, the ORB feature points covered by the mask and the mask boundary are removed; and a uniform motion model is used for inter-frame tracking to estimate the depth camera's pose.

6. A mobile robot visual odometry streaming processing method based on deep learning as claimed in claim 5, characterized in that: Use the FAST method to extract the key points in the grayscale image and use the BRIEF feature descriptor to describe the image area around the key points. The key points and descriptors together constitute the ORB feature points. The camera pose is optimized using the light speed adjustment method. The camera pose is represented as a 4×4 transformation matrix T wc , which is expressed as: Among them, R is the rotation matrix, which represents the rotation transformation between the two coordinate systems; t is the translation vector, which represents the translation transformation between the two coordinate systems; the camera pose T wc It is the transformation matrix from the camera coordinate system to the world coordinate system.

7. The method for streaming processing of mobile robot visual odometry based on deep learning according to claim 1, characterized in that: In step 4, the camera's velocity matrix Velocity is calculated based on the two closest historical camera poses obtained in step 3, which is expressed as: Velocity=T nl =T nw ×T wl Among them, l is last_pose, which represents the camera coordinate system of the first of the two most recent poses; n is new_pose, which represents the camera coordinate system of the second of the two most recent poses; w is world, which represents the world coordinate system; T wl is the previous camera pose, T nw is the last camera pose, T nl is the velocity matrix of the camera motion; predict the current camera pose T based on the velocity matrix fw , which is expressed as: Where f is forecast_pose, which represents the camera position to be predicted; Δt represents the difference between the image frame number to be predicted and the image frame number of the next camera pose processing.

8. The method for streaming processing of mobile robot visual odometry based on deep learning according to claim 1, characterized in that: Step 5 is implemented using the following method: When the system is turned on, the initialized camera pose is output; according to step 4, when the system obtains the camera pose at least once, the predicted camera pose is output using a prediction method; finally, the output depth camera pose is expressed as [tx, ty, tz, qx, qy, qz, qw], which corresponds to the position [tx, ty, tz] and direction [qx, qy, qz, qw] of the depth camera in the world, and the output timestamp is obtained at the same time; among them, tx represents the x-axis coordinate of the camera position, ty represents the y-axis coordinate of the camera position; tz represents the z-axis coordinate of the camera position, qx represents the first value of the imaginary part of the quaternion of the camera direction, qy represents the second value of the imaginary part of the quaternion of the camera direction, qz represents the third value of the imaginary part of the quaternion of the camera direction, and qw represents the value of the real part of the quaternion of the camera direction.

9. A mobile robot visual odometry streaming processing method based on deep learning as claimed in claim 8, characterized in that: In step 5, the predicted camera pose T fw Expressed as: Inverse Expressed as: in, Represents the transformation matrix from the camera coordinate system to the world coordinate system, that is, the direction of the camera relative to the world origin; -R T t is a 3×1 translation vector, simplified as [tx,ty,tz]; R T is a 3×3 rotation matrix, simplified as a quaternion [qx,qy,qz,qw], T represents the transpose of the matrix, and the quaternion q is formally expressed as: q=w+xi+yj+zk=[w,v] T Where w is the real part of the quaternion q, v is the imaginary part of the quaternion q, v = [x, y, z] T , T represents the transpose of the matrix; the conversion formula from rotation matrix to quaternion is expressed as: R=vv T +w 2 I+2wv^+(v^) 2 Where I is the identity matrix, T represents the transpose of the matrix; v^ is an antisymmetric matrix, expressed as: Finally, the output camera pose is represented as a seven-tuple [tx, ty, tz, qx, qy, qz, qw], which corresponds to the position and orientation of the camera respectively, and the moment of outputting the camera pose is used as the output timestamp.

Images