A Visual-Inertial SLAM System Based on a Multi-Task Feature Extraction Network

Abstract

A visual inertial SLAM system based on a multi-task feature extraction network disclosed by the present invention includes a multi-task feature extraction network and a three-dimensional map construction module; image data information is acquired and input into the multi-task feature extraction network to detect and track features. After the sensor data processing is completed, it is checked whether the system has been initialized. If not, visual inertial joint initialization is performed on the system; after the initialization is completed, a sliding window is used to optimize the poses and IMU biases of a fixed number of key frames, thereby performing pose estimation. The three-dimensional map construction module will combine the camera poses estimated by the system and the camera video stream to complete three-dimensional reconstruction using the surfel model and the deformation map.

Description

Technical Field

[0001] The present invention relates to a visual inertial SLAM system, and more specifically, to a visual inertial SLAM system based on a multi-task feature extraction network. Background Art

[0002] In recent years, with the research and development of artificial intelligence technology, intelligent mobile robots have increasingly appeared in many fields such as industrial manufacturing, agricultural production, transportation, social services, medical rehabilitation, and space exploration, and have played an important role. In the application of intelligent mobile robots, the robot can sense its position in the three-dimensional space and the surrounding environmental structure through sensors in an unknown environment, so as to realize functions such as autonomous positioning, map construction, and path planning. The realization of the above functions is called the Simultaneous Localization and Mapping technology (SLAM for short). Therefore, the SLAM technology is the core technology for intelligent mobile robots to work in an unknown environment.

[0003] The sensors used in the early SLAM technology were mainly lidar. After decades of development, lidar has shown excellent performance in terms of accuracy and stability. However, lidar is generally expensive and difficult to be applied to intelligent mobile devices with high requirements for weight, volume, power consumption, etc., which increases the difficulty of popularizing and using the SLAM technology. In recent years, with the development of camera technology and computer vision, and the continuous improvement of computer hardware technology, the visual SLAM technology using cameras as sensors has made great breakthroughs. The visual sensors relied on by the visual SLAM technology can obtain rich image information from the surrounding environment, and the low price and light weight of the visual sensors enable the visual SLAM technology to have a wider range of applications. At the same time, the maturity of CPU and GPU technologies has greatly improved the computer's ability to process images. Therefore, the visual SLAM technology has become a research hotspot in academia.

[0004] The traditional feature extraction algorithms relied on by traditional SLAM systems rely on artificial design, which is effective and practical in certain specific scenarios. However, in the face of complex and changeable scenarios such as insufficient light, over-illumination, and texture loss, their performance is not stable and may even be unable to run. As the most important branch of machine learning, deep learning has made rapid development in many fields of computer vision such as feature extraction and matching. This provides a new direction for the development of visual SLAM.

[0005] In view of the problem that traditional feature extraction algorithms are difficult to handle complex scenes such as low texture, this application uses a simplified multi-task convolutional neural network for feature detection. The feature points and descriptors generated by this network have better robustness than traditional methods, which makes the pose estimation results of the SLAM system more accurate and stable, and has good real-time performance. In view of the problem that the visual sensor has blurred images and too few overlapping areas between frames when moving too fast, in order to make up for the shortcomings of the visual sensor in this regard, this application adopts a strategy of fusing visual sensors and inertial sensors.

[0006] As technology develops and applications enter a new stage, the scenes faced by SLAM are more complex and changeable, which puts forward new requirements for the original technology. The traditional feature extraction algorithms that traditional SLAM systems rely on rely on manual design, which is effective and practical in certain specific scenarios. However, when faced with complex and changeable scenes such as texture loss, their performance is not stable and may even fail to operate. In addition, pure visual SLAM systems are difficult to achieve the requirements of precise positioning when the camera moves too fast, resulting in blurred images and too few overlapping areas between frames. Summary of the invention

[0007] In order to solve at least one of the above technical problems, the present invention proposes a visual-inertial SLAM system based on a multi-task feature extraction network.

[0008] A first aspect of the present invention provides a visual inertial SLAM system based on a multi-task feature extraction network, comprising: a multi-task feature extraction network and a three-dimensional map construction module;

[0009] Obtain image data information, input the image data information into the multi-task feature extraction network, detect and track the features,

[0010] After the sensor data processing is completed, check whether the system has been initialized. If not, perform visual-inertial joint initialization on the system.

[0011] After initialization, a sliding window is used to optimize the pose and IMU deviation of a fixed number of key frames to perform pose estimation.

[0012] The 3D map construction module combines the camera pose estimated by the system with the camera video stream and uses the surfel model and deformation map to complete the 3D reconstruction.

[0013] In a preferred embodiment of the present invention, three-dimensional reconstruction is performed using a surfel model and a deformation map, specifically including: obtaining an accurate camera pose by pose estimation and optimization,

[0014] Project the pixel points of each image obtained by the binocular camera into the world coordinate system, and obtain the surfel 3D map through point cloud data fusion.

[0015] In a preferred embodiment of the present invention, the multi-feature extraction network consists of a shared backbone network and two sub-modules. The two sub-modules include a position module and a descriptor module. The position module includes two convolutional layers, one of which uses the ReLU activation function and the other uses the sigmoid activation function. The descriptor module receives the image processed by the backbone network and has two convolutional layers with the number of channels being 256 each. After each convolutional layer is the ReLU activation function. According to the relative position coordinates P of the feature points output by the position module relative , use bicubic interpolation to generate the corresponding descriptor D image .

[0016] In a preferred embodiment of the present invention, the backbone network includes four convolutional layers. The number of channels in the four convolutional layers is 32-64-128-256. There is a max pooling layer between each convolutional layer, for a total of three max pooling layers. The stride and kernel size of each max pooling layer are both 2. After each max pooling layer, the number of channels of the subsequent convolutional layer will double. Therefore, after being processed by the backbone network, one pixel of the output image is 8x8 pixels of the input image.

[0017] In a preferred embodiment of the present invention, the position module predicts the relative position coordinates P of the feature points of the input image relative , from the relative position coordinates P relative The mapping to the image pixel coordinates P image is calculated by the following formula:

[0018] P image,x =(c + P relative,x )·f

[0019] P image,y =(r + P relative,y )·f

[0020] where c is the column input index of the x coordinate, r is the row input index of the y coordinate. f is the downsampling factor, and f = 8.

[0021] In a preferred embodiment of the present invention, the surfel model includes a number of surfel patches, and the deformation map includes a number of nodes.

[0022] In a preferred embodiment of the present invention, each node σ n contains the rotation matrix σ R , the translation matrix σ t , time and position σ gThe position of the face element affected by the deformed graph is given by the following formula:

[0023]

[0024] where ω n (M S ) represents the weight of the influence of node σ n on the face element, then ω n (M S ) can be expressed as:

[0025]

[0026] where d max represents the Euclidean distance from M S to the nearest face element.

[0027] In a preferred embodiment of the present invention, it further includes a self-supervised training framework. The self-supervised training framework includes a number of images for training. Each image for training is divided into two. One is the original image A that remains unchanged, and the other is the image B after being transformed by a transformation matrix and randomly non-spatially image-enhanced;

[0028] The feature points and descriptors of images A and B are respectively detected through a multi-task network, and then point correspondences are established from images A and B;

[0029] The point correspondences are used to train the model in the loss function;

[0030] Assume that there are N point pairs in images A and B, and the distance of each point pair is represented by the Euclidean distance:

[0031]

[0032] where, represents the position of the feature point in image A, represents the position of the feature point in image B, T represents the transformation matrix, and the transformation matrix T is the same as the transformation matrix from image A to image B.

[0033] The above technical solution of the present invention has the following advantages compared with the prior art:

[0034] (1) This application proposes a simplified multi-task feature extraction network to replace the traditional feature extraction algorithm for feature detection. This network has better feature extraction accuracy, and the generated features maintain the same descriptor format as the ORB features, with good portability, and basically meet the real-time requirements of the SLAM system.

[0035] (2) This application adopts a strategy of fusing visual sensors and inertial sensors. By performing pre-integration on the IMU data in a popular way, a method for loosely coupling and determining the initial system values is realized at the front end of the system. At the same time, the reference coordinate system in vision is aligned with the inertial coordinate system, and the poses of a fixed number of key frames and IMU errors are optimized in the sliding window by minimizing the combined energy function, thus effectively overcoming problems such as camera jitter and too fast scene changes, and making up for the deficiencies of visual sensors in this regard.

[0036] (3) To ensure the global consistency of the 3D map model during the 3D reconstruction process and make the reconstructed model cover the 3D environment as densely as possible, through the Surfel model, the system can obtain the individual information of the point cloud. The point cloud is divided into the point cloud just completed reconstruction and the previously reconstructed point cloud according to time nodes, and the two point cloud regions are aligned to achieve point cloud fusion. Then, the sum of four cost functions is used to optimize the point cloud. Brief Description of the Drawings

[0037] Figure 1 is a block diagram of a visual inertial SLAM system based on a multi-task feature extraction network according to an embodiment of the present invention.

[0038] Figure 2 is the surface element of the surfel model according to an embodiment of the present invention.

[0039] Figure 3 is a deformed diagram according to an embodiment of the present invention. Detailed Embodiment

[0040] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0041] Many specific details are set forth in the following description in order to provide a thorough understanding of the present invention. However, the present invention may be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0042] As Figures 1 - 3 , shown, the present invention provides a visual inertial SLAM system based on a multi-task feature extraction network, including: a multi-task feature extraction network and a 3D map construction module;

[0043] Obtain image data information, input the image data information into the multi-task feature extraction network, and detect and track the features.

[0044] After the sensor data is processed, check whether the system has been initialized. If not, perform visual-inertial joint initialization on the system.

[0045] After initialization, use a sliding window to optimize the poses and IMU biases of a fixed number of key frames for pose estimation.

[0046] The 3D map construction module will combine the camera poses estimated by the system and the camera video stream to complete 3D reconstruction using the surfel model and the deformation map.

[0047] Specifically, after the sensor data is processed, check whether the system has been initialized. If not, perform visual-inertial joint initialization on the system. The main purpose of initialization is to obtain the parameters and initial values required for system optimization. Since the visual-inertial SLAM system is a highly nonlinear system, the choice of initial values will directly affect the tracking accuracy of the entire system. Therefore, initialization is required to provide the correct parameters and initial values. The initialized parameters remain unchanged during the operation of the system, such as the absolute scale and gravitational acceleration. The initial values include the poses and velocity information of the first few frames, the 3D feature positions, and the biases of the IMU accelerometer and gyroscope. Through visual-inertial joint initialization, the inertial pose and the visual pose are combined to obtain the initial estimate of the system.

[0048] After initialization, use a sliding window to optimize the poses and IMU biases of a fixed number of key frames to achieve high-precision pose estimation. The sliding window limits the number of key frames to be optimized to control the scale of optimization, and marginalization is used to remove some older or unsatisfactory key frames to keep the window size unchanged.

[0049] Finally, the 3D map construction module will combine the camera poses estimated by the system and the camera video stream to complete 3D reconstruction using the surfel model and the deformation map.

[0050] According to the embodiments of the present invention, completing 3D reconstruction using the surfel model and the deformation map specifically includes: obtaining accurate camera poses through pose estimation and optimization.

[0051] Project the pixel points of each image obtained by the binocular camera into the world coordinate system, and through point cloud data fusion, obtain the surfel 3D map.

[0052] The multi-feature extraction network consists of a shared backbone network and two sub-modules. The two sub-modules include a location module and a descriptor module. The location module includes two convolutional layers, one of which uses the ReLU activation function and the other uses the sigmoid activation function. The descriptor module receives the image processed by the backbone network and has two convolutional layers, both with 256 channels. After each convolutional layer is the ReLU activation function. According to the relative position coordinates P of the feature points output by the location module relative , bicubic interpolation is used to generate the corresponding descriptor D image .

[0053] The input of this multi-task network is a single image. The backbone network has four convolutional layers, and the number of channels in the four convolutional layers is 32 - 64 - 128 - 256. The choice of the number of channels is determined empirically by setting several groups of candidate values and selecting the best value according to the experimental results. To speed up the convergence rate and make it easier to solve the gradient, after each convolutional layer is the ReLU activation function. There is a max-pooling layer between each pair of convolutional layers, a total of three max-pooling layers, and the stride and kernel size of each max-pooling layer are both 2. After each max-pooling layer, the number of channels of the subsequent convolutional layer will double. Therefore, after being processed by the backbone network, one pixel of the output image is 8x8 pixels of the input image.

[0054] The sub-module for generating feature points, that is, the location module, receives the image processed by the backbone network. The location module has two convolutional layers with 256 and 256 channels. After the previous convolutional layer is the ReLU activation function, and in order to limit the location prediction within the interval [0,1], after the latter convolutional layer is the sigmoid activation function. Here, regression is used for feature point detection to achieve self-supervised training. Also, because the input image of the location module is 1 / 64 of the original image, to a certain extent, it avoids the aggregation of feature points and realizes the uniform distribution of feature points. For a network with 3 pooling layers, the location module can predict the relative position coordinates P of the feature points of the input image relative , from the relative position coordinates P relative to the mapping of the image pixel coordinates P image is calculated by the following formula:

[0055] P image,x =(c + P relative,x )·f

[0056] P image,y =(r + P relative,y )·f (1)

[0057] where c is the column input index of the x coordinate, r is the row input index of the y coordinate. f is the downsampling factor, and f = 8.

[0058] The sub-module for generating descriptors, namely the descriptor module. Similar to the position module, it receives the image processed by the backbone network, has two convolutional layers with the number of channels both being 256, and is followed by a ReLU activation function after each convolutional layer. According to the relative position coordinates P of the feature points output by the position module relative , bicubic interpolation is used to generate the corresponding descriptor D image .

[0059] To achieve self-supervised feature detection, a self-supervised training framework is adopted. In this framework, each training image is divided into two. One is the original image that remains unchanged, denoted as A, and the other is the image transformed by a transformation matrix (such as rotation, scaling, and perspective transformation) and random non-spatial image enhancement (such as brightness and noise), denoted as B. To calculate the loss function, a multi-task network structure is needed to detect the feature points and descriptors of both images A and B respectively, and then establish point correspondences between images A and B. Finally, the point correspondences are used to train the model in the loss function. Then, assuming there are N point pairs in images A and B, the distance of each point pair is represented by the Euclidean distance:

[0060]

[0061] where represents the position of the feature point in image A, represents the position of the feature point in image B. T represents the transformation matrix, and the transformation matrix T is the same as the transformation matrix from image A to image B, mapping the feature points in image A to image B.

[0062] The loss function for training feature points and descriptors consists of two loss terms:

[0063] L total =α pt L pt +α desc L desc (3)

[0064] where, the former loss L pt is the feature point loss, and the latter L desc is the descriptor loss. Each loss term has a corresponding weight term, which are α pt and α desc .

[0065] The feature point loss L pt can ensure that the same feature points can be detected under different angles and different light intensities, and the specific representation form is:

[0066]

[0067] In the first term, αposition is the corresponding weight term. To ensure that two corresponding feature points predicted in Image A and Image B are the same feature point, it can be simplified to the expression given in Equation (2):

[0068]

[0069] In the second term is used to ensure the credibility of the feature points. The higher the score, the stronger the repeatability of the corresponding feature points, specifically expressed as:

[0070]

[0071] Among them, represents the average distance between all point pairs:

[0072]

[0073] Initially, the network generates feature points at random positions. As the number of training times increases, the distance between point pairs will gradually decrease, thereby improving the localization of feature points. Therefore, the network can learn uniformly distributed and accurate feature points based on the provided training data and the transformed data.

[0074] The descriptor loss L in the total loss desc , is determined using the Hinge loss function. First, define a homography-induced correspondence S ij :

[0075]

[0076] Among them, g ij represents the pixel interval between the corresponding two points.

[0077] Then each group of descriptor losses can be expressed as:

[0078]

[0079] Among them, m p is the positive margin in the Hinge loss, and m n is the negative margin in the Hinge loss. The weight term λ d is used to balance corresponding points and non-corresponding points.

[0080] Finally, the descriptor loss L desc can be expressed as:

[0081]

[0082] The method for constructing a 3D map based on surfels is as follows:

[0083] Precise camera poses can be obtained through pose estimation and optimization. Then, the pixel points of each image obtained by the binocular camera are projected into the world coordinate system. Finally, through point cloud data fusion, an accurate surfel 3D map is obtained. In the representation of surfel, the position information of 3D points can be obtained from the depth data measured by the binocular camera.

[0084] Figure 2 Denotes a surfel patch, where each circle represents a surfel patch, and the surface of the 3D map is composed of multiple patches; Figure 3 Denotes a deformation graph, where points represent nodes and lines represent edges. The position M of each surfel patch S Will be affected by the nodes in the deformation graph I(M S ,σ), and many nodes and edges can together form a deformation graph. Each node σ n Contains the rotation matrix σ R , the translation matrix σ t , time And position σ g . Then, the position of the patch affected by the deformation graph Is given by the following formula:

[0085]

[0086] Where ω n (M S ) Represents the weight of the influence of node σ n On the patch, then ω n (M S ) Can be expressed as:

[0087]

[0088] Where d max Represents the Euclidean distance from M S To the nearest patch. The point cloud is divided into a point cloud that has just completed reconstruction and a point cloud that has been reconstructed before according to time nodes, and the two point cloud regions are aligned to achieve point cloud fusion. The correspondence between the two fused points can be expressed as:

[0089]

[0090] Where Represents the position of the target point cloud, Represents the position of the current frame point cloud, And Respectively represent And The corresponding timestamps. Next, the sum of four cost functions is used to optimize the point cloud.

[0091] The first cost function uses the Frobenius norm to represent the pose of each node, expressed as:

[0092]

[0093] The second cost function uses regularization to ensure the continuity of the deformation map, expressed as:

[0094]

[0095] The third cost function is a constraint term that minimizes the error in q, where Solved from Equation (11), it can be expressed as:

[0096]

[0097] Finally, to add the new point cloud data to the 3D map, we use the fourth cost function to fix the point cloud in the inactive area and optimize the target points:

[0098]

[0099] Based on the above four cost functions, the total cost function can be obtained, expressed as:

[0100] E total = ω rot E rot + ω reg E reg + ω con E con + ω pin E pin (18)

[0101] Among them, the corresponding weights are set as ω rot = 1, ω reg = 10, ω con = 100 and ω pin = 100.

[0102] Therefore, an accurate new pose can be obtained through global camera pose optimization. The new camera pose affects node optimization, thereby affecting the surfel model. The surfel model and the deformation map are used to fuse and optimize the point cloud, and finally an accurate 3D map is obtained.

[0103] In terms of the hardware of this application: mainly use the Xiaomi D1010 binocular vision inertial camera, which has a six-axis IMU built-in; the main configuration of the computer is an Intel Core i7-9700KF CPU, a GeForce RTX 2080 Super GPU, and 16GB of running memory;

[0104] Software programming: mainly in C / C++ and Python, and experimental programming is mainly carried out under the Ubuntu system;

[0105] Pose estimation and 3D reconstruction strategy: Drawing on current advanced SLAM algorithms, a real-time SLAM experimental platform for complex scenarios is designed. This experimental platform integrates real-time data acquisition, real-time data processing, and post-optimization processing of data, and can perform pose estimation and 3D reconstruction for complex scenarios.

[0106] The strategy of this application is to use a handheld binocular visual inertial camera to collect data, and use the system of this application and three advanced visual inertial odometers, namely ROVIO, S-MSCKF, and S-VIORB, to estimate poses and make comparisons. Since the exact true trajectory of the camera cannot be obtained in a real scenario, when the system runs in ROS, the 3D visualization tool rviz in ROS is used to subscribe to the pose data published by the system, so that the trajectory route can be drawn and displayed in real time. To verify the accuracy and stability of the visual inertial SLAM system of this application in 3D reconstruction, experiments are carried out in different scenarios. The system uses a multi-task feature extraction network to extract feature points, which can well handle difficult scenes with low texture, thus ensuring that the system completes 3D reconstruction under accurate camera pose estimation. In addition, the surfel model can directly obtain the individual information of the point cloud, align the point cloud regions, and achieve point cloud optimization and fusion.

[0107] The technical solution of the present invention has the following beneficial effects:

[0108] This application proposes a simplified multi-task feature extraction network to replace the traditional feature extraction algorithm for feature detection. This network has better feature extraction accuracy, and the generated features maintain the same descriptor format as ORB features, with good portability, and basically meet the real-time requirements of the SLAM system.

[0109] This application adopts the strategy of fusing visual sensors and inertial sensors. By performing popular pre-integration on IMU data, a method for loosely coupling and determining the initial value of the system is realized at the front end of the system. At the same time, the reference coordinate system in vision and the inertial coordinate system are aligned, and the pose of a fixed number of key frames and the IMU error are optimized in the sliding window by minimizing the combined energy function, thereby effectively overcoming problems such as camera jitter and too fast scene changes, and making up for the deficiencies of visual sensors in this regard.

[0110] To ensure the global consistency of the 3D map model during the 3D reconstruction process and enable the reconstructed model to cover the 3D environment as densely as possible, this application proposes a 3D reconstruction method based on the Surfel model. Through the Surfel model, the system can obtain the individual information of the point cloud, divide the point cloud into the point cloud that has just completed reconstruction and the point cloud that has been reconstructed before according to time nodes, align the two point cloud regions to achieve point cloud fusion, and then use the sum of four cost functions to optimize the point cloud.

[0111] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0112] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to the above embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the above embodiments shown herein, but rather should be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0113] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A visual-inertial SLAM system based on a multi-task feature extraction network, comprising: a multi-task feature extraction network and a three-dimensional map construction module; characterized in that, acquire image data information, input the image data information into the multi-task feature extraction network, detect and track the features, after the sensor data processing is completed, check whether the system has been initialized. If not, perform visual-inertial joint initialization on the system; after the initialization is completed, use a sliding window to optimize the poses of a fixed number of key frames and the IMU biases, so as to perform pose estimation, the three-dimensional map construction module will combine the camera pose estimated by the system and the camera video stream, and use the surfel model and the deformation map to complete three-dimensional reconstruction; complete three-dimensional reconstruction using the surfel model and the deformation map, specifically including: obtaining the accurate camera pose through pose estimation and optimization, project the pixel points of each image obtained by the binocular camera into the world coordinate system, obtain the surfel three-dimensional map through point cloud data fusion; The multi-feature extraction network consists of a shared backbone network and two sub-modules. The two sub-modules include a location module and a descriptor module. The location module includes two convolutional layers, one of which uses the ReLU activation function and the other uses the sigmoid activation function. The descriptor module receives the image processed by the backbone network and has two convolutional layers, both with 256 channels. After each convolutional layer is the ReLU activation function. According to the relative position coordinates P of the feature points output by the location module relative , bicubic interpolation is used to generate the corresponding descriptor D image ; The backbone network includes four convolutional layers. The number of channels in the four convolutional layers is 32-64-128-256. There is a max pooling layer between each convolutional layer. There are a total of three max pooling layers. The stride and kernel size of each max pooling layer are both 2. After each max pooling layer, the number of channels of the subsequent convolutional layer will double. Therefore, after being processed by the backbone network, one pixel of the output image is 8x8 pixels of the input image; It also includes a self-supervised training framework. The self-supervised training framework includes several images for training. Each image for training will be divided into two. One of them is the original image A that remains unchanged, and the other is the image B after being transformed by the transformation matrix and random non-spatial image enhancement; detect the feature points and descriptors of images A and B respectively through the multi-task network, and then establish point correspondences from images A and B; use the point correspondences in the loss function to train the model; Assume that there are N point pairs in images A and B, and the distance of each point pair is represented by the Euclidean distance: Among them, represents the position of the feature points in Image A, represents the position of the feature points in Image B, T represents the transformation matrix, and the transformation matrix T is the same as the transformation matrix from Image A to Image B.

2. The visual-inertial SLAM system based on a multi-task feature extraction network according to claim 1, characterized in that, The position module predicts the relative position coordinates P of the feature points of the input image relative , and the mapping from the relative position coordinates P relative to the image pixel coordinates P image is calculated by the following formula: P image,x = (c + P relative,x ) · f P image,y = (r + P relative,y )·f where c is the column input index of the x coordinate, r is the row input index of the y coordinate; f is the downsampling factor, and f = 8.

3. The visual-inertial SLAM system based on a multi-task feature extraction network according to claim 1, characterized in that, the surfel model includes several surfel patches, and the deformation map includes several nodes.

4. The visual-inertial SLAM system based on a multi-task feature extraction network according to claim 3, characterized in that, Each node σ n contains the rotation matrix σ R , the translation matrix σ t , time and position σ g , and the position of the element surface affected by the deformation map is given by the following formula: where ω n (M S ) represents the weight of the influence of node σ n on the face element, then ω n (M S ) can be expressed as: where d max represents the S Euclidean distance from M to the nearest facet.

Images