Visual inertia SLAM system optimization method and device

Abstract

The invention discloses a visual inertia SLAM system optimization method and device, and the method comprises the steps: S1, estimating the size and direction of pixel offset between image frames through IMU pre-integration, obtaining a pixel drift matrix, inputting the pixel offset matrix into a self-adaptive blurring kernel estimator, achieving the self-adaptive estimation of a blurring kernel of each sub-region in an image, and obtaining a blurring kernel of each sub-region in the image; secondly, through feature information matrix multiplication operation, the fuzzy degraded image is preprocessed, and shallow feature information is extracted; step S2, according to the preprocessing feature information in the step S1, further realizing stage-by-stage reconstruction and restoration of the image through a lightweight Transform network; and S3, performing visual inertial SLAM positioning with enhanced visual information according to the restored image. By adopting the technical scheme provided by the invention, the positioning precision can be optimized, and meanwhile, the network reasoning delay is relatively low.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a visual inertial SLAM system optimization method and device. Background Art

[0002] Simultaneous localization and mapping (SLAM) is a key technology that provides accurate positioning and environmental mapping for mobile unmanned devices in unknown environments and plays a vital role in emerging fields such as autonomous driving and augmented reality. Among them, the SLAM algorithm based on monocular vision sensors has gradually become a research topic of concern in the SLAM field due to its low cost and high precision. Although visual SLAM technology has been fully developed and improved in recent years, the positioning performance of traditional visual SLAM algorithms such as vins-mono is easily significantly reduced due to image degradation.

[0003] With the development of deep learning in recent years, due to the emergence of large-scale datasets composed of blurry-clear image pairs, deep learning-based image deblurring methods have made significant progress on artificial simulated datasets. This type of method designs a variety of effective network modules based on multi-scale convolutional neural networks (CNNs) to achieve coarse-to-fine deblurring. In order to solve the problem that CNN is not able to capture global semantic information, a large number of related works have proposed methods based on the Transformer architecture. This type of method proposes a multi-head self-attention mechanism suitable for the field of image processing, achieving a more reliable image restoration effect than traditional CNN. However, this type of method often only focuses on the ability of the network model to restore degraded images, without considering the feature extraction step of the visual SLAM front end, nor using the intermediate posture information generated by the visual SLAM algorithm in the back-end optimization. This results in most network models in this field having large parameters and high forward propagation delay, which is not suitable for visual SLAM performance optimization. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a visual inertial SLAM system optimization method and device, which can optimize positioning accuracy while having lower network inference delay.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A visual inertial SLAM system optimization method, comprising:

[0007] Step S1, estimate the pixel offset size and direction between image frames through IMU pre-integration to obtain a pixel drift matrix, then input the pixel offset matrix into an adaptive blur kernel estimator to achieve adaptive estimation of the blur kernel for each sub-region in the image, and then perform feature information matrix multiplication operation to achieve preprocessing of the blurred degraded image and extract shallow feature information;

[0008] Step S2: Based on the preprocessed feature information in step S1, the restored image is further reconstructed stage by stage through a lightweight Transformer network;

[0009] Step S3: Perform visual inertial SLAM positioning with visual information enhancement based on the restored image.

[0010] Preferably, the adaptive blur kernel estimator includes: two convolution layers and a modulated deformed convolution, which adaptively adjusts the position and weight of each sampling point in the modulated deformed convolution according to the pixel drift matrix and the input image, and the preprocessing feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulated convolution with the output obtained by the convolution layer.

[0011] Preferably, in step S2, the surrounding environment information is input into the image enhancement network to obtain a restored image; wherein the image enhancement network includes: a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filter module HFM, the first two encoders and all encoding layers of the image enhancement network are LCM, the third encoder and the middle layer are deep feature extraction units composed of 8 LTMs, LCM is based on the inverted residual structure design in the CNN network, the encoder and the decoder exchange feature information through jump connections, especially the third encoder and the fifth decoder are connected through HFM to achieve the screening of high-frequency feature information.

[0012] Preferably, step S3 comprises:

[0013] The restored image between the previous frame and the current frame is used to track feature points, and the sparse optical flow is calculated using the optical flow pyramid method to obtain multi-dimensional visual image observations;

[0014] Based on the multi-dimensional visual image observation and IMU data, the inter-frame IMU pre-integration measurement is obtained;

[0015] Estimate the optimized drone attitude based on inter-frame IMU pre-integrated measurements.

[0016] The present invention also provides a visual inertial SLAM system optimization device, comprising:

[0017] The preprocessing module is used to estimate the pixel offset size and direction between image frames through IMU pre-integration to obtain the pixel drift matrix, and then input the pixel offset matrix into the adaptive blur kernel estimator to realize the adaptive estimation of the blur kernel of each sub-region in the image, and then realize the preprocessing of the blurred degraded image and extract the shallow feature information through the feature information matrix multiplication operation;

[0018] A processing module, used to reconstruct the restored image stage by stage through a lightweight Transformer network according to the pre-processed feature information in step S1;

[0019] The positioning module is used for visual-inertial SLAM positioning with visual information enhancement based on the restored image.

[0020] Preferably, the adaptive blur kernel estimator includes: two convolution layers and a modulated deformed convolution, which adaptively adjusts the position and weight of each sampling point in the modulated deformed convolution according to the pixel drift matrix and the input image, and the preprocessing feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulated convolution with the output obtained by the convolution layer.

[0021] Preferably, the processing module inputs the surrounding environment information into the image enhancement network to obtain a restored image; wherein the image enhancement network includes: a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filter module HFM, the first two encoders and all encoding layers of the image enhancement network are LCM, the third encoder and the middle layer are deep feature extraction units composed of 8 LTMs, LCM is based on the inverted residual structure design in the CNN network, the encoder and the decoder exchange feature information through jump connections, and the third encoder and the fifth decoder are connected through HFM to achieve screening of high-frequency feature information.

[0022] Preferably, the positioning module includes:

[0023] A first computing unit is used to track feature points using a restored image between a previous frame and a current frame, and to calculate sparse optical flows using an optical flow pyramid method to obtain a multi-dimensional visual image observation;

[0024] A second computing unit, for obtaining inter-frame IMU pre-integrated measurements based on the multi-dimensional visual image observations and the IMU data;

[0025] The third computing unit is used to estimate the optimized drone attitude based on the inter-frame IMU pre-integration measurement.

[0026] The present invention optimizes the visual inertial SLAM system based on lightweight Transformer image enhancement. The present invention combines the characteristics of visual factor extraction in visual inertial SLAM, and adopts a differential high-pass filter module (Differentia High-pass Filter Module, HFM), a linear self-attention mechanism (Linear Self-Attention, LSA) and an adaptive fuzzy kernel estimation module (Self-Adaptive PSF Estimator, SAPE), so that the improved SLAM framework obtains better positioning performance while having a smaller image processing delay effect. Therefore, unmanned equipment can be more robust in dynamic blur scenes to achieve more accurate positioning accuracy, while the image enhancement algorithm has a high degree of lightweight, high efficiency of image processing, and can be better deployed on edge devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0028] Figure 1 It is a flow chart of the visual inertial SLAM system optimization method of the present invention;

[0029] Figure 2 is an adaptive fuzzy kernel estimator;

[0030] Figure 3 This is a diagram of the lightweight Transformer architecture;

[0031] Figure 4 It is a differential high-pass filter;

[0032] Figure 5 is a linear self-attention module graph;

[0033] Figure 6 Schematic diagram of the pole constraint;

[0034] Figure 7 This is a schematic diagram of inertial sensor pre-integration;

[0035] Figure 8 The motion curves of V2_03_diffcult in three axes.

[0036] Fig. 9 It is the error curve diagram of three axis directions on V2_03_diffcult;

[0037] Fig.10 For testing equipment and environment;

[0038] Fig.11 It is a schematic diagram of the two-dimensional trajectory motion of the actual experiment;

[0039] Fig.12 It is the motion trajectory diagram in the three-axis directions of the actual test;

[0040] Fig.13 This is the actual test error diagram. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Embodiment 1:

[0044] like Figure 1 As shown, the embodiment of the present invention provides a visual inertial SLAM system optimization method, which realizes visual inertial SLAM precise positioning in a dynamic fuzzy environment, can meet the autonomous positioning of unmanned equipment, perceive unknown environments, and ensure the real-time positioning optimization of image processing, including the following steps:

[0045] Step S1: Adaptive fuzzy kernel estimator based on IMU pre-integration

[0046] The present invention uses a monocular vision sensor and an IMU inertial measurement unit to obtain real-time information about the surrounding environment, as well as the angular velocity and acceleration changes of the unmanned equipment itself. Figure 2 As shown in the figure, first, the posture transformation of the visual sensor in the previous and next two frames is calculated based on the IMU pre-integration, and then the pixel offset size of each pixel on the image is calculated through the relative posture. Secondly, a two-channel pixel drift matrix is ​​constructed according to the offset size and direction of each pixel. Finally, the pixel offset matrix is ​​merged with the input image and input into the module to realize the preprocessing of the input blurred image. The pixel drift matrix is ​​obtained based on the Homography matrix, and its calculation formula is:

[0047]

[0048] in, is a point k in the image at time θ 0 The pixel coordinates at is a point k in the image at time θ τ The pixel coordinates at the time, K is the camera internal parameter, t is the camera displacement between adjacent image frames, n is the plane normal vector, d is the scene depth value, and H is the Homography matrix. Since the interval between adjacent frames is short, the above formula can be simplified to:

[0049]

[0050] Then the pixel offset size for pixel k is:

[0051]

[0052] From the above formula, we can know the offset size of each pixel along two different axes in the image coordinate system. Then we can calculate the offset size of each pixel on the image. Figure 2 , the pixel offset matrix can be constructed according to the offset size of each pixel point in different axis directions.

[0053] like Figure 2 As shown in the figure, the adaptive blur kernel estimator contains two convolution layers and a modulated convolution to form a modulated deformation convolution, which can adaptively adjust the position and weight of each sampling point according to the pixel drift matrix and the input image. Finally, in the output part of the adaptive blur kernel estimator, the preprocessing feature can be obtained by multiplying the output obtained by the modulated convolution with the output obtained by the convolution layer. This module can be regarded as:

[0054]

[0055] in, Indicates that the entire formula is calculated in the frequency domain. represents the blur kernel distribution feature matrix of the degraded image estimated by MDCN; Represents the preprocessed image in the frequency domain; Represents a blurred image in the frequency domain.

[0056] Step S2: Image enhancement network based on lightweight Transformer

[0057] like Figure 3As shown in the figure, the image enhancement network includes three basic modules: Lightweight CNN Module (LCM), Lightweight Transformer Module (LTM) and HFM. The first two encoders and all encoding layers of the overall network are LCM, and the third encoder and the middle layer are deep feature extraction units composed of 8 LTMs. The encoder is mainly used to further extract feature information from the input features and reduce the scale of the feature map to form high-dimensional semantic information. The middle layer is mainly used to establish semantic information dependency of long-distance pixels and improve the network's ability to extract overall image information. The decoder decodes the semantic information obtained from the middle layer and restores the abstract semantic information to a reconstructed image of the same dimension and scale as the blurred image originally input to the network. For the preprocessed features obtained in step S1 The preprocessed feature information is encoded through three bottom-up encoders and intermediate layers to obtain semantic information at a low scale, and three top-down decoders gradually decode the semantic information into a reconstructed image. The encoder and decoder exchange feature information of feature maps of different scales through skip connections, improving the network's ability to extract multi-scale information. In the bottom-up path, the spatial dimension of the feature map is halved every time the encoder channel is doubled, and the opposite is true every time it passes through the decoder. When downsampling and upsampling, convolution with a kernel size of 2 and a stride of 2 and Pixelshuffle are used respectively. In addition, since low-dimensional feature maps contain richer content and detail information, while high-dimensional feature maps contain richer edge and texture information, the skip connections between the encoder and decoder use different filtering methods: the skip connections in the first three stages use convolution with a kernel size of 1, while the fourth stage uses HFM to extract high-frequency feature information.

[0058] The structural diagram of HFM is as follows Figure 4 As shown, assuming that the input feature map is First, we get F through the fully connected layer c , and then use the average pooling operation with a kernel size of 2 to obtain the average pixel value size of each sliding window area, thereby obtaining the feature map Then for F avg Upsample to get the feature map This is the same spatial dimension as the input. Finally, from F in Subtract F element by element up And take the absolute value on both sides to get the high-frequency feature information F high The complete operation process can be expressed as:

[0059] F high =|H up (H avgpool (Hlinear (F in ),2))-F in |.

[0060] Among them, H up , H avgpool , H linear They are respectively represented as upsampling operation, average pooling operation and fully connected layer operation.

[0061] LCM is based on the inverted residual structure design in the CNN network, and its main function is to process the content information in the high-resolution feature map. However, the traditional CNN network is limited by its local receptive field size and cannot obtain global feature information well. The multi-head attention mechanism in the traditional Transformer consumes a large amount of computation during forward reasoning. Therefore, the present invention designs a lightweight Transformer module, and its structure diagram is shown in the figure. Figure 5 As shown. This module consists of two parts: multi-head linear self-attention and feedforward network. For the tensor after layer normalization First, two linear transformations are required to obtain three token quantities, namely,

[0062]

[0063] Then through and The element-wise product between them obtains the global attention vector a v , and then through a v and The element-by-element product between is used to obtain the multi-head linear self-attention calculation result, and finally the output is obtained through the forward feedback unit. The whole process can be written as:

[0064]

[0065] in, and The matrix is ​​obtained by reshaping the original tensor, a v Represents the attention vector after dimensional expansion.

[0066] Finally, the output is reshaped into

[0067] Step S3: Visual-inertial SLAM positioning with visual information enhancement

[0068] Taking the VINS-Mono algorithm as the algorithm to be optimized, after obtaining the restored image through steps S1 and S2, the restored image between the previous frame and the current frame can be used to track the feature points, and the sparse optical flow can be calculated using the optical flow pyramid method. This produces enhanced multi-dimensional visual image observations, and combined with IMU data, inter-frame IMU pre-integrated measurements can be calculated. By tightly coupling the optimization of these two observations, the optimized drone attitude can be estimated.

[0069] The restored image between the previous frame and the current frame is used to track the feature points, and the optical flow pyramid method is used to calculate the sparse optical flow to obtain multi-dimensional visual image observations. The calculation process is as follows:

[0070] like Figure 6 As shown, for two key frame images I F1 and I F2 , let the movement from the first frame image to the second frame image be the rotation matrix R and the translation matrix t, and the centers of the two cameras are O 1 , O 2 For image I F1 A laser radar corresponds to the projected pixel feature point p 1 , it is in image I F2 The corresponding feature point in is p 2 , finding the corresponding matching points between two frames of images has been given in the previous part. According to the point-plane principle, point O 1 , O 2 and a point p in space can together form a plane, namely the polar plane. 1 , O 2 The two intersection points of the line connecting the two points and the two image planes, i.e., the poles, are e 1 、e 2 In the above figure, 1 and l 2 is the polar line, that is, the line between the corresponding pixel feature point and the pole. In the coordinate system of the first frame image, let the spatial position of p be:

[0071] P = [x, y, z]

[0072] Two pixels p 1 、p 2 The location is:

[0073]

[0074] Where K is the camera intrinsic parameter matrix, s 1 and 2 is the pixel, R and t are the camera motion in the two coordinate systems. Using homogeneous coordinates, the above equation can be rewritten as an equation that is valid when multiplied by a non-zero parameter:

[0075]

[0076] Take the normalized plane coordinates x of the two pixels 1 、x 2 Then we have:

[0077]

[0078] and

[0079] x 2 =Rx 1 +t

[0080] Take the outer product of both sides with t and multiply them on the left at the same time Then we have:

[0081]

[0082] in is zero, then:

[0083]

[0084] Re-introduce p 1 、p 2 have to:

[0085]

[0086] This formula is called the epipolar constraint, which includes both translation and rotation. The middle part is recorded as two matrices: the basic matrix F and the essential matrix E, and the epipolar constraint can be further simplified as:

[0087]

[0088] Perform SVD decomposition on the essential matrix E to obtain:

[0089] E=U∑V T

[0090] Where U and V are orthogonal matrices, and ∑ is a singular matrix. Based on the intrinsic properties of E, Σ=diag(σ,σ,0). In the SVD decomposition, for any E, there are two possible sets of solutions t and R corresponding to it:

[0091]

[0092] There are four solutions, namely By detecting the depth estimation value of each solution, we can obtain the correct camera pose transformation of two adjacent frames t-1 and t:

[0093]

[0094] in, To solve the correct transformation matrix for the camera pose estimation between two adjacent frames.

[0095] At the same time, considering that the actual camera has certain distortion and other observation error factors, the estimated There is an error. To reduce the estimated pose transformation Error, set n three-dimensional space points P = [P 1 ,P 2 ,…,P i ,…P n ] T and its projection point at time t-1 And the projection point observed by the camera at time t The estimated projection point at time t is obtained The difference between the observed projection point and the estimated projection point is used as the error objective function, that is:

[0096]

[0097] If the optimal pose transformation matrix is ​​required Then the error objective function that needs to be minimized is:

[0098]

[0099] Based on the inter-frame IMU pre-integration measurement, the optimized drone attitude is estimated. The calculation process is as follows:

[0100] like Figure 7 As shown in the figure, because the data frequency of the Inertial Measurement Unit (IMU) is higher than the image data frequency, the IMU data between the kth frame and the k+1th frame of the image can be integrated to estimate the relative change in position, velocity, and rotation (PVQ) of the k+1th frame relative to the kth frame. In addition, since the time between camera frames is short, the estimation of the drone state quantity in this time interval is more accurate. Next, the pre-integration formula for high-precision micro-inertia is derived.

[0101] 1. Speed ​​pre-integration derivation

[0102] The speed arrangement in traditional strapdown inertial navigation arrangement is as follows:

[0103]

[0104] For the above model, after adding the influence of accelerometer zero bias and noise, we can get:

[0105]

[0106] Multiply both sides The coordinate transformation matrix represents the coordinate transformation matrix from the navigation system at time k+1 to the machine system at time k, and we can get:

[0107]

[0108] The velocity pre-integration term is obtained:

[0109]

[0110] After adopting the high-precision inertia pre-integration method, the carrier speed is updated in the following form:

[0111]

[0112] 2. Position pre-integration derivation

[0113] The position arrangement in traditional strapdown inertial navigation arrangement is as follows:

[0114]

[0115] For the above model, after adding the influence of accelerometer zero bias and noise, we can get:

[0116]

[0117] The location information is expressed in latitude, longitude and height, and expressed in polar coordinates:

[0118]

[0119] in Its physical meaning represents the change in latitude and longitude. Expand to get:

[0120]

[0121] in: Then Expand to get:

[0122]

[0123] Multiply both sides available:

[0124]

[0125] So the position pre-integration term is obtained as:

[0126]

[0127] After adopting the high-precision inertia pre-integration method, the carrier speed is updated in the following form:

[0128] Derivation of angle pre-integration

[0129] Ideal values ​​for the gyroscope are:

[0130]

[0131] The attitude update takes the form of a quaternion:

[0132]

[0133] Will Substituting the specific expression of into the equation, we can get:

[0134]

[0135] And because:

[0136]

[0137] Therefore, the posture update formula can be organized into the following form:

[0138]

[0139] in, is the relative attitude change between the carrier coordinate system and the navigation system at time k, expressed by quaternions. It is the relative attitude change between the carrier coordinate system at time t and the carrier coordinate system at time k expressed by quaternion

[0140] Multiply both sides by The quaternion represents the relative attitude change between the navigation system and the aircraft system at time k, and the angle pre-integration term can be obtained

[0141]

[0142] in:

[0143]

[0144] In order to verify the effectiveness of the visual-inertial SLAM system optimization method based on lightweight Transformer image enhancement, the present invention adopts the well-known public dataset Euroc, and takes the two subsets (V1_03_diffculth and V2_03_diffcult) with the most serious image blur as the algorithm verification set, and selects eight image restoration algorithms in recent years as comparison. Each algorithm conducts 20 experiments in total. Table 1 shows the comparison of positioning accuracy after image enhancement of each algorithm, where the time consumption is the time consumption of each image enhancement algorithm to process a 752×480 picture.

[0145] Table 1

[0146]

[0147] It can be seen from Table 1 that the embodiment of the present invention achieves the minimum positioning error and forward reasoning delay on both sides. Figure 8 and Fig. 9 It can be seen that the difference between the embodiment of the present invention and the true value is the smallest, and the error value is closest to 0 in most time stamps. Compared with the unoptimized Vins-Mono algorithm, the positioning optimization accuracy of the embodiment of the present invention in a high dynamic fuzzy environment is also significantly improved: the standard deviation (SD) in the V1_03_difficult subset is reduced by 34.1%, the root mean squared error (RMSE) is reduced by 31.8%, and the sum of squares due to error (SSE) is reduced by 52.9%; the SD in the V2_03_difficult subset is reduced by 58.0%, the RMSE is reduced by 58.6%, and the SSE is reduced by 82.8%. Fig. 9 3-axis trajectory error comparison diagram of each algorithm on the V2_03_diffcult subset. The maximum positioning errors of the three axes of the embodiment of the present invention are 0.149 meters, 0.155 meters and 0.133 meters respectively.

[0148] To further illustrate the effectiveness of the present invention, an experimental verification is also carried out on an actual blur scene. Fig.10 The present invention adopts a four-wheel drive intelligent robot as a carrier platform and a total station as a reference value of the posture. The experiment is carried out in an underground garage environment with uneven light distribution. The two-dimensional trajectory motion curve measured by the total station is shown in Fig.11 , the test results are shown in Table 2.

[0149] Table 2

[0150]

[0151] From Table 2, we can see that except for the SD index, the present invention has achieved the best performance index, while the other positioning indexes have achieved the best positioning effect. It only takes 9m to process a 640×480 image. Fig.12 and Fig.13It can be seen that, except for the vertical direction, the error value of the embodiment of the present invention has the smallest fluctuation and is closest to 0. Compared with the unoptimized Vins-Mono algorithm, the positioning optimization accuracy of the embodiment of the present invention in the actual underground garage fuzzy scene is also significantly improved: the standard deviation (SD) is reduced by 45.27%, the root mean squared error (RMSE) is reduced by 29.87%, and the sum of squares due to error (SSE) is reduced by 50.44%. Fig.13 3 is a comparison chart of the three-axis trajectory errors of various algorithms in an actual underground garage scene. The maximum positioning errors of the three axes of the embodiment of the present invention are 1.01 meters, 1.64 meters, and 0.785 meters, respectively.

[0152] Aiming at the problem that traditional visual-inertial SLAM algorithms cannot adapt well to dynamic blur scenes, a visual-inertial SLAM system optimization method based on lightweight Transformer image enhancement is proposed. First, in order to make full use of the intermediate posture information generated by visual SLAM in the back-end optimization process, an adaptive blur kernel estimation module based on IMU pre-integration is proposed. Secondly, in order to improve the ability to extract feature points and reconstruct more high-frequency edge information, a differential image high-frequency filter is proposed. Finally, in order to reduce the number of model parameters, forward reasoning delay and the ability to extract long-distance pixel information, a lightweight transformer module based on linear self-attention is proposed. The visual SLAM optimization algorithm proposed in this paper effectively reduces the forward reasoning time of the model. In addition, it has a significant optimization effect on the visual SLAM positioning performance.

[0153] Embodiment 2:

[0154] The embodiment of the present invention further provides a visual inertial SLAM system optimization device, comprising:

[0155] The preprocessing module is used to estimate the pixel offset size and direction between image frames through IMU pre-integration to obtain the pixel drift matrix, and then input the pixel offset matrix into the adaptive blur kernel estimator to realize the adaptive estimation of the blur kernel of each sub-region in the image, and then realize the preprocessing of the blurred degraded image and extract the shallow feature information through the feature information matrix multiplication operation;

[0156] A processing module, used to reconstruct the restored image stage by stage through a lightweight Transformer network according to the pre-processed feature information in step S1;

[0157] The positioning module is used for visual-inertial SLAM positioning with visual information enhancement based on the restored image.

[0158] As an implementation method of an embodiment of the present invention, the adaptive blur kernel estimator includes: two convolution layers and a modulated deformed convolution. The position and weight of each sampling point in the modulated deformed convolution are adaptively adjusted according to the pixel drift matrix and the input image. The preprocessing feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulated convolution with the output obtained by the convolution layer.

[0159] As an implementation method of an embodiment of the present invention, the processing module inputs the surrounding environment information into the image enhancement network to obtain a restored image; wherein, the image enhancement network includes: a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filtering module HFM, the first two encoders and all encoding layers of the image enhancement network are LCM, the third encoder and the middle layer are deep feature extraction units composed of 8 LTMs, LCM is designed based on the inverted residual structure in the CNN network, the encoder and the decoder exchange feature information through a jump connection, and the third encoder and the fifth decoder are connected through HFM to achieve the screening of high-frequency feature information.

[0160] As an implementation manner of the embodiment of the present invention, the positioning module includes:

[0161] A first computing unit is used to track feature points using a restored image between a previous frame and a current frame, and to calculate sparse optical flows using an optical flow pyramid method to obtain a multi-dimensional visual image observation;

[0162] A second computing unit, for obtaining inter-frame IMU pre-integrated measurements based on the multi-dimensional visual image observations and the IMU data;

[0163] The third computing unit is used to estimate the optimized drone attitude based on the inter-frame IMU pre-integration measurement.

[0164] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A visual inertial SLAM system optimization method, characterized in that: include: Step S1, estimate the pixel offset size and direction between image frames through IMU pre-integration to obtain a pixel drift matrix, then input the pixel offset matrix into an adaptive blur kernel estimator to achieve adaptive estimation of the blur kernel for each sub-region in the image, and then perform feature information matrix multiplication operation to achieve preprocessing of the blurred degraded image and extract shallow feature information; Step S2, based on the preprocessed feature information in step S1, a lightweight Transformer network is used to reconstruct the restored image stage by stage; Step S3: Perform visual inertial SLAM positioning with visual information enhancement based on the restored image.

2. The visual-inertial SLAM system optimization method as claimed in claim 1, wherein The adaptive blur kernel estimator includes: two convolution layers and a modulated deformable convolution. According to the pixel drift matrix and the input image, the position and weight of each sampling point in the modulated deformable convolution are adaptively adjusted. The preprocessing feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulated convolution with the output of the convolution layer.

3. The visual-inertial SLAM system optimization method as claimed in claim 2, wherein: In step S2, the surrounding environment information is input into the image enhancement network to obtain a restored image; wherein the image enhancement network includes: a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filter module HFM, the first two encoders and all encoding layers of the image enhancement network are LCM, the third encoder and the middle layer are deep feature extraction units composed of 8 LTMs, LCM is based on the inverted residual structure design in the CNN network, the encoder and the decoder exchange feature information through a jump connection, and the third encoder and the fifth decoder are connected through HFM to achieve the screening of high-frequency feature information.

4. The visual-inertial SLAM system optimization method as claimed in claim 2, wherein: Step S3 includes: The restored image between the previous frame and the current frame is used to track feature points, and the sparse optical flow is calculated using the optical flow pyramid method to obtain multi-dimensional visual image observations; Based on the multi-dimensional visual image observation and IMU data, the inter-frame IMU pre-integration measurement is obtained; Estimate the optimized drone attitude based on inter-frame IMU pre-integrated measurements.

5. A visual inertial SLAM system optimization device, characterized in that: include: The preprocessing module is used to estimate the pixel offset size and direction between image frames through IMU pre-integration to obtain the pixel drift matrix, and then input the pixel offset matrix into the adaptive blur kernel estimator to realize the adaptive estimation of the blur kernel of each sub-region in the image, and then realize the preprocessing of the blurred degraded image and extract the shallow feature information through the feature information matrix multiplication operation; A processing module, used to reconstruct the restored image stage by stage through a lightweight Transformer network according to the pre-processed feature information in step S1; The positioning module is used for visual-inertial SLAM positioning with visual information enhancement based on the restored image.

6. The visual inertial SLAM system optimization device as claimed in claim 5, characterized in that: The adaptive blur kernel estimator includes: two convolution layers and a modulated deformable convolution. According to the pixel drift matrix and the input image, the position and weight of each sampling point in the modulated deformable convolution are adaptively adjusted. The preprocessing feature can be obtained by multiplying the blur kernel distribution matrix obtained by the modulated convolution with the output of the convolution layer.

7. The visual-inertial SLAM system optimization device as claimed in claim 6, characterized in that: The processing module inputs the surrounding environment information into the image enhancement network to obtain a restored image; the image enhancement network includes: a lightweight CNN module LCM, a lightweight Transformer module LTM and a high-pass filter module HFM. The first two encoders and all encoding layers of the image enhancement network are LCMs, and the third encoder and the middle layer are deep feature extraction units composed of 8 LTMs. LCM is based on the inverted residual structure design in the CNN network. The encoder and decoder exchange feature information through jump connections. The third encoder and the fifth decoder are connected through HFM to achieve the screening of high-frequency feature information.

8. The visual-inertial SLAM system optimization device as claimed in claim 7, characterized in that: The positioning module includes: A first computing unit is used to track feature points using a restored image between a previous frame and a current frame, and to calculate sparse optical flows using an optical flow pyramid method to obtain a multi-dimensional visual image observation; A second computing unit, for obtaining inter-frame IMU pre-integrated measurements based on the multi-dimensional visual image observations and the IMU data; The third computing unit is used to estimate the optimized drone attitude based on the inter-frame IMU pre-integrated measurement.

Images