A visual-inertial slam system optimization method and apparatus

Abstract

The application discloses a visual inertial SLAM system optimization method and device, comprising the following steps: S1, estimating the pixel offset size and direction between image frames through IMU pre-integration to obtain a pixel drift matrix, inputting the pixel offset matrix into an adaptive blur kernel estimator to realize adaptive blur kernel estimation of each sub-region in the image, and realizing pre-processing of a blur-degraded image through feature information matrix multiplication operation to extract shallow feature information; S2, further reconstructing a recovered image stage by stage through a lightweight Transformer network according to the pre-processed feature information in step S1; and S3, performing visual inertial SLAM positioning through visual information enhancement according to the recovered image. The technical scheme of the application can optimize positioning accuracy while having low network inference delay.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a visual-inertial SLAM system optimization method and device. BACKGROUND

[0002] Simultaneous localization and mapping (SLAM) is a key technology for mobile unmanned devices to provide accurate positioning and environment mapping in unknown environments and plays a crucial role in emerging fields such as autonomous driving and augmented reality. SLAM algorithms based on monocular vision sensors have gradually become a research topic in the SLAM field due to their low cost and high precision. Although visual SLAM technology has been developed and improved in recent years, the positioning performance of traditional visual SLAM algorithms such as vins-mono is easily significantly reduced due to image degradation.

[0003] With the development of deep learning in recent years, due to the emergence of large-scale data sets composed of blur-clear image pairs, image deblurring methods based on deep learning have made significant progress on artificial simulation data sets. Such methods design various effective network modules based on multi-scale convolutional neural networks (CNNs), achieving deblurring from coarse to fine. To address the problem of insufficient global semantic information capture capability of CNNs, a large number of related works have proposed methods based on the Transformer architecture. Such methods propose a multi-head self-attention mechanism suitable for the image processing field, achieving more reliable image restoration effects than traditional CNNs. However, such methods often only focus on the restoration capability of the network model for degraded images, without considering the feature extraction step in the visual SLAM front end or utilizing the intermediate pose information generated in the back-end optimization of the visual SLAM algorithm, which results in a large parameter quantity and high forward propagation delay of most network models in this field, making them unsuitable for visual SLAM performance optimization. SUMMARY

[0004] The technical problem to be solved by the application is to provide a visual-inertial SLAM system optimization method and device that can optimize positioning accuracy while having low network inference delay.

[0005] To achieve the above object, the application adopts the following technical solution:

[0006] A visual-inertial SLAM system optimization method, comprising:

[0007] Step S1, estimate the pixel shift size and direction between image frames through IMU pre-integration, obtain a pixel shift matrix, then input the pixel shift matrix into an adaptive blur kernel estimator to realize adaptive blur kernel estimation for each sub-region in the image, and realize image pre-processing and shallow feature information extraction through feature information matrix multiplication operation;

[0008] Step S2, further reconstruct the recovered image through a lightweight Transformer network according to the pre-processed feature information in step S1;

[0009] Step S3, perform visual information enhancement on the recovered image to realize visual-inertial SLAM positioning.

[0010] Preferably, the adaptive blur kernel estimator comprises two convolution layers and a modulation deformation convolution, which adaptively adjusts the position and weight of each sampling point in the modulation deformation convolution according to the pixel shift matrix and the input image, and the pre-processed feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulation convolution with the output obtained by the convolution layer.

[0011] Preferably, in step S2, the surrounding environment information is input into the image enhancement network to obtain the recovered image; wherein the image enhancement network comprises a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filter module HFM, the first two encoders and all the encoding layers of the image enhancement network are LCM, the third layer encoder and the intermediate layer are a deep feature extraction unit composed of 8 LTMs, the LCM is designed based on the inverse residual structure in the CNN network, the encoders and the decoders interact feature information through the skip connection, and in particular, the third layer encoder and the fifth layer decoder are connected through the HFM to realize the screening of high-frequency feature information.

[0012] Preferably, step S3 comprises:

[0013] The recovered image between the previous frame and the current frame is used to track the feature points, and the sparse optical flow is calculated by using the optical flow pyramid method to obtain multi-dimensional visual image observation;

[0014] According to the multi-dimensional visual image observation and the IMU data, the inter-frame IMU pre-integration measurement is obtained;

[0015] According to the inter-frame IMU pre-integration measurement, the optimized unmanned aerial vehicle attitude is estimated.

[0016] The application also provides a visual-inertial SLAM system optimization device, comprising:

[0017] The preprocessing module is configured to estimate pixel shift size and direction between image frames through IMU pre-integration, obtain a pixel shift matrix, input the pixel shift matrix into the adaptive blur kernel estimator, realize adaptive blur kernel estimation for each sub-region in the image, and realize image preprocessing and shallow feature information extraction through feature information matrix multiplication.

[0018] The processing module is configured to realize stage-by-stage reconstruction of the restored image through a lightweight Transformer network according to the preprocessed feature information in step S1.

[0019] The positioning module is configured to perform visual-inertial SLAM positioning based on the restored image.

[0020] Preferably, the adaptive blur kernel estimator comprises two convolution layers and a modulation deformation convolution, which adaptively adjusts the position and weight of each sampling point in the modulation deformation convolution according to the pixel shift matrix and the input image, and the preprocessed feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulation convolution with the output obtained by the convolution layer.

[0021] Preferably, the processing module inputs surrounding environment information into an image enhancement network to obtain the restored image, wherein the image enhancement network comprises a lightweight CNN module (LCM), a lightweight Transformer module (LTM) and a high-pass filter module (HFM), the first two encoders and all the encoding layers of the image enhancement network are LCMS, the third layer encoder and the intermediate layers are a deep feature extraction unit composed of eight LTMs, the LCM is designed based on the inverted residual structure in the CNN network, the encoders and the decoders thereof interact feature information through a skip connection, the third layer encoder and the fifth layer decoder are connected through the HFM to realize filtering of high-frequency feature information.

[0022] Preferably, the positioning module comprises:

[0023] The first calculation unit is configured to track feature points by using the restored image between the previous frame and the current frame, and calculate a sparse optical flow by using an optical flow pyramid method to obtain multi-dimensional visual image observations.

[0024] The second calculation unit is configured to obtain inter-frame IMU pre-integration measurements according to the multi-dimensional visual image observations and IMU data.

[0025] The third calculation unit is configured to estimate an optimized UAV attitude according to the inter-frame IMU pre-integration measurements.

[0026] The application is based on lightweight Transformer image enhancement for visual-inertial SLAM system optimization, and the application combines the characteristics of visual factor extraction in visual-inertial SLAM, adopts a differential high-frequency filter module (HFM), a linear self-attention mechanism (LSA) and a self-adaptive PSF estimator (SAPE), so that the improved SLAM framework has better positioning performance and smaller image processing delay effect. Therefore, the unmanned device can be more robust in a dynamic blur scene, and more accurate positioning accuracy can be achieved, and the image enhancement algorithm has high lightweight degree, high image processing efficiency, and can be better deployed on edge devices. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0028] Figure 1 is a flow chart of the visual-inertial SLAM system optimization method of the present application;

[0029] Figure 2 is a self-adaptive blur kernel estimator;

[0030] Figure 3 is a lightweight Transformer architecture diagram;

[0031] Figure 4 is a differential high-pass filter;

[0032] Figure 5 is a linear self-attention module diagram;

[0033] Figure 6 is an epipolar constraint diagram;

[0034] Figure 7 is an inertial sensor pre-integration diagram;

[0035] Figure 8 is a motion curve diagram in the upper three-axis direction of V2_03_diffcult;

[0036] Figure 9 is an error curve diagram in the upper three-axis direction of V2_03_diffcult;

[0037] Figure 10 For testing equipment and environment;

[0038] Figure 11 This is a schematic diagram of the two-dimensional trajectory motion in the actual experiment;

[0039] Figure 12 The diagram shows the motion trajectory along the three axes of the actual experiment;

[0040] Figure 13 This is a graph showing the actual experimental error. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Example 1:

[0044] like Figure 1 As shown, this embodiment of the invention provides a method for optimizing a visual inertial SLAM system, achieving precise positioning in a dynamically blurred environment. This method enables autonomous positioning of unmanned equipment, perception of unknown environments, and ensures real-time image processing for positioning optimization. The method includes the following steps:

[0045] Step S1: Adaptive fuzzy kernel estimator based on IMU pre-integration

[0046] This invention uses a monocular vision sensor and an IMU (Inertial Measurement Unit) to acquire real-time information about the surrounding environment, as well as the changes in the angular velocity and acceleration of the unmanned equipment itself. For example... Figure 2 As shown, firstly, the pose transformation of the visual sensor within two consecutive frames is calculated based on IMU pre-integration, and then the pixel offset of each pixel in the image is calculated using this relative pose. Secondly, a two-channel pixel drift matrix is ​​constructed according to the offset magnitude and direction of each pixel. Finally, the pixel offset matrix is ​​merged with the input image and input into this module to perform preprocessing of the blurred input image. The pixel drift matrix is ​​obtained based on the Homography matrix, and its calculation formula is:

[0047]

[0048] in, pixel coordinate of a point k in the image at time θ0, pixel coordinate of a point k in the image at time θ τ pixel coordinate of a point k in the image at time θ0, K is the camera intrinsic parameter, t is the camera displacement between adjacent image frames, n is the plane normal vector, d is the scene depth value, and H is the Homography matrix. Since the interval time of adjacent frames is short, the above formula can be simplified as:

[0049]

[0050] Then the pixel offset size of the pixel point k is:

[0051]

[0052] From the above formula, the offset size of each pixel point in the image coordinate system along two different axes can be calculated. See Figure 2 According to the offset size of each pixel point in different axis directions, a pixel offset matrix can be constructed.

[0053] As shown in Figure 2 , the adaptive blur kernel estimator includes two convolution layers and a modulation convolution to form a modulation deformation convolution, which can adaptively adjust the position and weight of each sampling point according to the pixel drift matrix and the input image. Finally, in the output part of the adaptive blur kernel estimator, the preprocessed feature can be obtained by multiplying the output obtained by the modulation convolution with the output obtained by the convolution layer. This module can be regarded as:

[0054]

[0055] wherein, indicates that the entire formula is calculated in the frequency domain, indicates the blur kernel distribution feature matrix of the degraded image estimated by the MDCN; indicates the preprocessed image in the frequency domain; indicates the blurred image in the frequency domain.

[0056] Step S2, image enhancement network based on lightweight Transformer

[0057] As shown in Figure 3As shown, the image enhancement network includes three basic modules: a lightweight CNN module (LCM), a lightweight Transformer module (LTM), and an HFM, the first two encoders and all the encoding layers of the overall network are LCMs, and the third layer encoder and the intermediate layer are a deep feature extraction unit composed of 8 LTMs. The encoder is mainly used to further extract feature information and reduce the size of the feature map to form high-dimensional semantic information. The intermediate layer is mainly used to establish the semantic information dependence of long-distance pixels and improve the network's ability to extract overall image information. The decoder decodes the semantic information obtained from the intermediate layer to restore the abstract semantic information to a reconstructed image with the same dimension and scale as the original input fuzzy image. For the preprocessed feature The preprocessed feature information is encoded by three bottom-up encoders and intermediate layers to obtain semantic information at a low scale size, and three top-down decoders gradually decode the semantic information into a reconstructed image The encoder and the decoder exchange feature information at different scale feature maps through a skip connection to improve the network's ability to extract multi-scale information. In the bottom-up path, the spatial dimension of the feature map is halved after each encoder channel doubling, and the opposite is true for each decoder. During downsampling and upsampling, convolution with a kernel size of 2 and a step size of 2 and Pixelshuffle are used, respectively. In addition, since low-dimensional feature maps contain more rich content and detail information, and high-dimensional feature maps contain more rich edge and texture information, different filtering methods are used for the skip connection between the encoder and the decoder: the skip connection in the first three stages uses a convolution with a kernel size of 1, and the fourth stage uses an HFM to extract high-frequency feature information.

[0058] The structure diagram of the HFM is as shown in Figure 4 Assuming that the input feature map is First, get F c by a fully connected layer, then use an average pooling operation with a kernel size of 2 to get the average pixel value size of each sliding window region, thereby obtaining the feature map Then upsample F avg to obtain the feature map which has the same spatial dimension as the input. Finally, subtract F up from F in element by element, and take the absolute value of both sides to obtain the high-frequency feature information F high . The complete operation process can be represented as:

[0059] F high = |H up (H avgpool (Hlinear (F in ),2))-F in .

[0060] where H up , H avgpool , H linear respectively represent the up-sampling operation, the average pooling operation and the fully connected layer operation.

[0061] LCM is designed based on the inverted residual structure in the CNN network, and mainly functions to process content information in a high-resolution feature map, but the traditional CNN network cannot well obtain global feature information due to the limitation of the local receptive field size, and the multi-head attention mechanism in the traditional Transformer consumes a large amount of calculation in forward reasoning, therefore the present application designs a lightweight Transformer module, a structural diagram of which is shown in Figure 5 The module is composed of two parts: multi-head linear self-attention and a feedforward network. For the layer normalized tensor First, it needs to undergo two linear transformations to obtain three token amounts, i.e.

[0062]

[0063] Subsequently, the global attention vector a v is obtained through the element-by-element product between and , and the multi-head linear self-attention calculation result is obtained through the element-by-element product between a v and , and finally the output is obtained through the forward feedback unit. The whole process can be written as:

[0064]

[0065] wherein, and are obtained after the original tensor is reshaped, and a v represents the attention vector after expansion.

[0066] Finally, the shape of the output is reshaped to

[0067] Step S3, visual inertial SLAM positioning of visual information enhancement

[0068] After obtaining the rectified images through steps S1 and S2, the VINS-Mono algorithm as an algorithm to be optimized can be used to track feature points by using the rectified images between the last frame and the current frame, and sparse optical flow is calculated using the optical flow pyramid method. This produces enhanced multi-dimensional visual image observations, which, combined with IMU data, can calculate inter-frame IMU pre-integral measurements. By tightly coupling the optimization of these two observations, the optimized UAV attitude can be estimated.

[0069] The multi-dimensional visual image observations are obtained by tracking feature points using the rectified images between the last frame and the current frame, and calculating sparse optical flow using the optical flow pyramid method, and the calculation process is as follows:

[0070] As shown in Figure 6 , for two key frame images I F1 and I F2 , the motion from the first frame image to the second frame image is set as a rotation matrix R and a translation matrix t, and the centers of the two cameras are O1 and O2 respectively. For a pixel feature point p1 corresponding to the projection of a laser radar in image I F1 , the corresponding feature point in image I F2 is p2, and the corresponding matching points between the two frames of images have been given in the previous part. According to the point-plane principle, points O1, O2 and a point p in space can jointly form a plane, i.e. an epipolar plane. The two intersection points of the line connecting O1 and O2 with the two image planes are the epipoles e1 and e2 respectively. In the above figure, l1 and l2 are epipolar lines, i.e. the lines connecting the corresponding pixel feature points and epipoles. In the coordinate system of the first frame image, the spatial position of p is:

[0071] P=[x,y,z]

[0072] The positions of the two pixel points p1 and p2 are:

[0073]

[0074] where K is the camera intrinsic matrix, s1 and s2 are pixels, and R and t are the camera motion in the two coordinate systems. Using homogeneous coordinates, the above equation is rewritten as an equation that is multiplied by a non-zero parameter:

[0075]

[0076] Taking the normalized plane coordinates x1 and x2 of the two pixel points, we have:

[0077]

[0078] and

[0079] x2=Rx1+t

[0080] Both sides simultaneously with t do the outer product, and simultaneously left multiplication Then we have:

[0081]

[0082] Where is zero, then we have:

[0083]

[0084] Re-enter p1, p2, we get:

[0085]

[0086] This formula is called epipolar constraint, which contains both translation and rotation. The middle part is recorded as two matrices: the fundamental matrix F and the essential matrix E, so the epipolar constraint can be further simplified as:

[0087]

[0088] SVD decomposition of the essential matrix E gives:

[0089] E = U∑V T

[0090] Where U, V are orthogonal matrices, and ∑ is a singular matrix. According to the intrinsic properties of E, we have Σ = diag(σ,σ,0). In the SVD decomposition, for any E, there are two groups of possible solutions t and R corresponding to it:

[0091]

[0092] The four solutions are The depth estimate value of each solution is obtained, and the correct pose transformation of the adjacent two frames t-1 and t cameras is obtained:

[0093]

[0094] Where, is the correct pose estimation transformation matrix of the adjacent two frames of cameras.

[0095] Considering the actual camera distortion and other observation error factors, the estimated poses exist errors. In order to reduce the error of the estimated pose transformation , set n three-dimensional space points P = [P1, P2, …, P i ,…P n ] T and their projection points in the t-1 time and the projection points observed by the camera in the t time , the estimated projection points in the t time The difference between the observed projection point and the estimated projection point is taken as the error objective function, i.e.:

[0096]

[0097] If the demand optimal pose transformation matrix The error objective function that needs to be minimized is:

[0098]

[0099] According to the inter-frame IMU pre-integration measurement, the optimized UAV pose is estimated, and the calculation process is as follows:

[0100] As shown in Figure 7 Because the data frequency of the inertial measurement unit (IMU) is higher than the image data frequency, the IMU data between the kth frame and the k+1th frame of the image can be integrated to estimate the position, velocity and rotation (PVQ) relative change size of the k+1th frame relative to the kth frame. In addition, because the inter-frame time of the camera is short, the UAV state quantity estimated in the time interval is relatively accurate. Next, the pre-integration formula of high-precision micro-inertia is derived.

[0101] 1. Velocity pre-integration derivation

[0102] The velocity arrangement in the traditional strapdown inertial navigation arrangement has the following form:

[0103]

[0104] For the above model, after adding the influence of accelerometer zero offset and noise, the following can be obtained:

[0105]

[0106] Multiply The coordinate conversion matrix represents the coordinate conversion matrix of the navigation system at time k+1 to the machine system at time k, and the following can be obtained:

[0107]

[0108] The velocity pre-integration term is obtained:

[0109]

[0110] After using the high-precision inertial pre-integration method, the carrier velocity is updated as follows:

[0111]

[0112] 2. Position pre-integration derivation

[0113] The position navigation in traditional SINS navigation system has the following form:

[0114]

[0115] For the above model, after adding the influence of accelerometer zero offset and noise, we can get:

[0116]

[0117] The position information is expressed by latitude and longitude height, and expressed in polar coordinates:

[0118]

[0119] Among them The physical meaning represents the change of latitude and longitude height. Expand to get:

[0120]

[0121] Among them: Expand to get:

[0122]

[0123] Multiply on the left and right sides to get:

[0124]

[0125] Thus the position pre-integration term is:

[0126]

[0127] After using high-precision inertial pre-integration, the carrier velocity is updated as follows:

[0128] Angle pre-integration derivation

[0129] The ideal value of the gyroscope is:

[0130]

[0131] The attitude update adopts the form of quaternion:

[0132]

[0133] Substitute the specific expression of into to get:

[0134]

[0135] Further, since:

[0136]

[0137] Therefore, the attitude updating formula can be arranged as follows:

[0138]

[0139] wherein, is the relative attitude change between the carrier coordinate system and the navigation system at the k time point expressed by a quaternion, is the relative attitude change between the carrier coordinate system at the t time point and the carrier coordinate system at the k time point expressed by a quaternion

[0140] The left and right sides are simultaneously multiplied by The quaternion represents the relative attitude change between the navigation system and the machine system at the k time point, and the angle pre-integration term can be obtained

[0141]

[0142] wherein:

[0143]

[0144] In order to verify the effectiveness of the visual-inertial SLAM system optimization method based on lightweight Transformer image enhancement, the present application uses the famous public dataset Euroc, takes the two subsets with the most serious image blur (V1_03_diffculth and V2_03_diffcult) as the algorithm verification set, and selects eight image restoration algorithms in recent years as the comparison, each algorithm is performed for 20 times, and Table 1 is the positioning accuracy comparison after image enhancement by each algorithm, wherein the time consumption is the time consumption of each image enhancement algorithm processing a 752*480 picture.

[0145] Table 1

[0146]

[0147] From Table 1, it can be seen that the embodiments of the present application have the smallest positioning error and forward reasoning delay on the two subsets, and the positioning error is 0.01m and the forward reasoning delay is 0.01s, which are the smallest among the eight algorithms. Figure 8 and Figure 9It can be seen that the difference between the embodiment of the application and the true value is minimum, and the error value is closest to 0 on most time stamps. Compared with the Vins-Mono algorithm before optimization, the positioning optimization precision of the embodiment of the application in the high dynamic blur environment is also significantly improved: the standard deviation (Standard deviation, S.D.) in the V1_03_difficult subset decreases by 34.1%, the root mean squared error (Root mean squared error, RMSE) decreases by 31.8%, and the sum of squares due to error (The sum of squares due to error, SSE) decreases by 52.9%; the S.D. in the V2_03_difficult subset decreases by 58.0%, the RMSE decreases by 58.6%, and the SSE decreases by 82.8%. Figure 9 For the three-axis trajectory error comparison chart of each algorithm on the V2_03_diffcult subset, the maximum positioning error of the three axes of the embodiment of the application is 0.149 meters, 0.155 meters and 0.133 meters, respectively.

[0148] In order to further illustrate the effectiveness of the application, experiments are also carried out for actual blur scenes. As shown in Figure 10 , the application adopts a four-wheel intelligent robot as a carrier platform, uses a total station as a pose reference true value, and carries out experiments in an actual uneven light distribution underground garage environment. The two-dimensional trajectory motion curve measured by the total station is shown in Figure 11 , and the test results are shown in Table 2.

[0149] Table 2

[0150]

[0151] As can be seen from Table 2, the application does not achieve the best performance index in the S.D. index, but achieves the best positioning effect in the remaining positioning indexes. The processing size of the 640x480 picture is only 9m, and Figure 12 and Figure 13 It can be seen that, except for the vertical direction, the error value of the embodiment of the application fluctuates the least and is closest to 0. Compared with the Vins-Mono algorithm before optimization, the positioning optimization precision of the embodiment of the application in the actual underground garage blur scene is also significantly improved: the standard deviation (Standard deviation, S.D.) decreases by 45.27%, the root mean squared error (Root mean squared error, RMSE) decreases by 29.87%, and the sum of squares due to error (The sum of squares due to error, SSE) decreases by 50.44%. Figure 13For the three-axis trajectory error comparison chart of each algorithm in the actual underground garage scene, the maximum positioning errors of the three axes of the embodiment of the application are 1.01 meters, 1.64 meters and 0.785 meters respectively.

[0152] In view of the problem that the traditional visual inertial SLAM algorithm cannot well adapt to a dynamic blur scene, a visual inertial SLAM system optimization method based on a lightweight Transformer image enhancement is proposed. First, in order to fully utilize the intermediate attitude information generated in the back-end optimization process of the visual SLAM, an adaptive blur kernel estimation module based on IMU pre-integration is proposed. Secondly, in order to improve the feature point extraction capability and reconstruct more high-frequency edge information, a differential image high-frequency filter is proposed. Finally, in order to reduce the parameter quantity of the model, the forward reasoning delay and the extraction capability of long-distance pixel information, a lightweight transformer module based on linear self-attention is proposed. The visual SLAM optimization algorithm proposed in the present application effectively reduces the forward reasoning time consumption of the model, and in addition, has a significant optimization effect on the positioning performance of the visual SLAM.

[0153] Embodiment 2:

[0154] The embodiment of the application also provides a visual inertial SLAM system optimization device, which comprises:

[0155] The preprocessing module is used for estimating the pixel offset size and direction between image frames through IMU pre-integration to obtain a pixel drift matrix, inputting the pixel offset matrix into an adaptive blur kernel estimator, realizing adaptive estimation of the blur kernel of each sub-region in the image, and realizing pre-processing of the blurred image and extraction of shallow feature information through feature information matrix multiplication operation.

[0156] The processing module is used for realizing stage-by-stage reconstruction of the restored image through the lightweight Transformer network according to the pre-processed feature information in step S1.

[0157] The positioning module is used for visual inertial SLAM positioning based on visual information enhancement of the restored image.

[0158] As an embodiment of the application, the adaptive blur kernel estimator comprises two convolution layers and a modulation deformation convolution. According to the pixel drift matrix and the input image, the position and weight of each sampling point in the modulation deformation convolution are adaptively adjusted. The pre-processed feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulation convolution with the output quantity obtained by the convolution layer.

[0159] As an embodiment of the present application, the processing module inputs the surrounding environment information into the image enhancement network to obtain the restored image; wherein the image enhancement network comprises: a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filter module HFM, the first two encoders and all the encoding layers of the image enhancement network are LCM, the third layer encoder and the intermediate layer are a deep feature extraction unit composed of 8 LTMs, the LCM is designed based on the inverse residual structure in the CNN network, the encoder and the decoder interact feature information through the skip connection, the third layer encoder and the fifth layer decoder are connected through the HFM, and the filtering of the high-frequency feature information is realized.

[0160] As an embodiment of the present application, the positioning module comprises:

[0161] The first calculation unit is configured to track the feature points by using the restored image between the last frame and the current frame, and calculate the sparse optical flow by using the optical flow pyramid method to obtain the multi-dimensional visual image observation.

[0162] The second calculation unit is configured to obtain the inter-frame IMU pre-integration measurement according to the multi-dimensional visual image observation and the IMU data.

[0163] The third calculation unit is configured to estimate the optimized UAV attitude according to the inter-frame IMU pre-integration measurement.

[0164] The above-described embodiments are only descriptions of the preferred modes of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A visual-inertial SLAM system optimization method, characterized in that, Comprise: Step S1, estimate the pixel offset size and direction between image frames by IMU pre-integration, obtain the pixel drift matrix, then input the pixel offset matrix to the adaptive blur kernel estimator, realize the adaptive estimation of the blur kernel of each sub-region in the image, and realize the pre-processing of the blurred image by feature information matrix multiplication operation to extract shallow feature information; Step S2, according to the pre-processed feature information in step S1, realize the stage-by-stage reconstruction of the restored image through the lightweight Transformer network; Step S3, according to the restored image, the visual information enhanced visual-inertial SLAM positioning is performed; The adaptive blur kernel estimator comprises two convolution layers and a modulation deformation convolution, which adaptively adjusts the position and weight of each sampling point in the modulation deformation convolution according to the pixel drift matrix and the input image, and the pre-processed feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulation convolution with the output obtained by the convolution layer; In step S2, the surrounding environment information is input into the image enhancement network to obtain the restored image; wherein the image enhancement network comprises a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filter module HFM, the first two encoders and all the encoding layers of the image enhancement network are LCM, the third layer encoder and the intermediate layer are a deep feature extraction unit composed of 8 LTMs, the LCM is designed based on the inverse residual structure in the CNN network, the encoder and the decoder interact feature information through the skip connection, the third layer encoder and the fifth layer decoder are connected through the HFM, and the filtering of high-frequency feature information is realized; Step S3 comprises: The restored image between the last frame and the current frame is used to track feature points, and a sparse optical flow is calculated using an optical flow pyramid method to obtain multi-dimensional visual image observations; According to the multi-dimensional visual image observation and the IMU data, the inter-frame IMU pre-integration measurement is obtained; According to the inter-frame IMU pre-integration measurement, the optimized unmanned aerial vehicle attitude is estimated.

2. A visual-inertial SLAM system optimization apparatus, characterized by, Comprise: The preprocessing module is used for estimating the pixel offset size and direction between image frames by IMU pre-integration, obtaining the pixel drift matrix, then inputting the pixel offset matrix to the adaptive blur kernel estimator, realizing the adaptive estimation of the blur kernel of each sub-region in the image, and realizing the pre-processing of the blurred image by feature information matrix multiplication operation to extract shallow feature information; The processing module is used for realizing the stage-by-stage reconstruction of the restored image through the lightweight Transformer network according to the pre-processed feature information in step S1; The positioning module is used for visual-inertial SLAM positioning of visual information enhancement according to the restored image; The adaptive blur kernel estimator comprises two convolution layers and a modulation deformation convolution, which adaptively adjusts the position and weight of each sampling point in the modulation deformation convolution according to the pixel drift matrix and the input image, and the pre-processed feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulation convolution with the output obtained by the convolution layer; The processing module inputs the surrounding environment information into an image enhancement network to obtain a restored image; wherein the image enhancement network comprises: a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filter module HFM, the first two encoders and all the encoding layers of the image enhancement network are the LCM, the third layer encoder and the intermediate layer are a deep feature extraction unit composed of 8 LTMs, the LCM is designed based on the reverse residual structure in the CNN network, the encoders and the decoders thereof interact feature information through a skip connection, the third layer encoder and the fifth layer decoder are connected through the HFM, and the filtering of high-frequency feature information is realized; The positioning module comprises: A first calculation unit is configured to track feature points using the restored images between the previous frame and the current frame, and calculate sparse optical flow using an optical flow pyramid method to obtain multi-dimensional visual image observations; A second calculation unit is configured to obtain inter-frame IMU pre-integration measurements based on the multi-dimensional visual image observations and the IMU data; A third calculation unit is configured to estimate an optimized unmanned aerial vehicle attitude based on the inter-frame IMU pre-integration measurements.

Images