Lightweight visual inertia fusion positioning method and system

Abstract

The invention discloses a lightweight visual inertia fusion positioning method and system, and relates to the technical field of autonomous positioning. The problems that a traditional visual inertial odometer is not robust enough under the condition of data damage, a sensor fused with many deep learning models in an existing deep learning method is huge in size and needs more computing resources during operation, so that deployment on an embedded system or mobile equipment with limited resources becomes difficult, and the cost is low are solved. And further application of the visual inertial odometer is limited. The method comprises the following steps: acquiring image data and IMU data; extracting visual feature information from the image data by using a FlowNetSimple network, and extracting inertial feature information from the IMU data by using a one-dimensional convolutional network; fusing the visual feature information and the inertial feature information by adopting a hard attention mechanism; a recurrent neural network is used for time modeling, and an output transformation network is used for converting the output of the model into a final prediction result. The method is applied to automatic driving.

Description

Technical Field

[0001] The present invention relates to the field of autonomous positioning technology, and in particular to a lightweight visual-inertial fusion positioning method. Background Art

[0002] The purpose of positioning technology is to accurately obtain position information from the environment, which is an important prerequisite for the carrier to achieve subsequent planning and control calculations. VIO technology is a low-cost, highly robust, and high-application-value positioning method in SLAM technology. The use of multiple information sources also makes it more suitable for fields such as autonomous driving. Visual inertial odometry (VIO) combines complementary visual information and inertial information. It is currently a low-cost, effective, and broadly applicable autonomous positioning technology. Deep learning methods are currently widely discussed in the VIO field because of their powerful feature extraction capabilities, which can effectively cope with complex environments such as large-scale occlusion and sensor failure that traditional VIO algorithms are not good at.

[0003] Traditional visual inertial odometry is not robust enough in the face of data corruption, and existing methods for studying deep learning sensor fusion. Many deep learning models are bulky and require more computing resources when running, which makes it difficult to deploy on resource-constrained embedded systems or mobile devices. At the same time, deep learning models usually consume more computing energy, which is not conducive to the long-term operation of mobile devices, resulting in their lack of lightweightness, limiting the further application of visual inertial odometry. Summary of the invention

[0004] The present invention aims at the problem that the traditional visual inertial odometer is not robust enough in the face of data corruption, and many deep learning models of existing deep learning method sensor fusion are bulky and require more computing resources when running, which makes it difficult to deploy on resource-constrained embedded systems or mobile devices, limiting the further application of visual inertial odometer. A lightweight visual inertial fusion positioning method is proposed, and the method includes:

[0005] Step S1: Acquire image data and IMU data;

[0006] Step S2: Use the FlowNetSimple network to extract visual feature information from the image data, and use a one-dimensional convolutional network to extract inertial feature information from the IMU data;

[0007] Step S3: Use hard attention mechanism to fuse visual feature information and inertial feature information;

[0008] Step S4: Use a recurrent neural network for temporal modeling and use an output transformation network to convert the output of the model into the final prediction result.

[0009] Furthermore, a preferred method is proposed, in which the FlowNetSimple network is used to extract visual feature information from the image data in step S2, including: the visual encoding part of the FlowNetSimple network includes a neural network structure composed of 10 layers of CNN, and the neural network structure composed of 10 layers of CNN outputs motion feature information, which is then pooled and sent to a three-layer fully connected network, and finally the visual feature information is output.

[0010] Furthermore, a preferred method is proposed, in which a one-dimensional convolutional network is used to extract inertial feature information from IMU data in step S2, including: obtaining angular velocity and acceleration data from the IMU data, and the input angular velocity and acceleration data are respectively passed through two one-dimensional convolutional layers and a maximum pooling layer, the outputs are connected in series, and then input into a fully connected layer to output the inertial feature information.

[0011] Furthermore, a preferred method is proposed, wherein step S3 comprises:

[0012] Step S31: Generate a hard mask by resampling from a Bernoulli distribution;

[0013] Step S32: The generated hard mask and visual feature vector a V and the inertial eigenvector a I Multiply element by element to get a new reweighted vector;

[0014] Step S33: resampling the mask using the Gumbel-Max technique;

[0015] Step S34: concatenate the re-weighted visual features and inertial features to form a final fused feature vector.

[0016] Furthermore, a preferred method is proposed, wherein step S32 comprises:

[0017] g hard (a V ,a I )=[a V es V ; a I es I ]

[0018] Among them, s V and I A hard mask.

[0019] Furthermore, a preferred method is proposed, wherein step S33 comprises:

[0020] Combine the random variable sampled from the Gumbel distribution with the class probability to form a new random variable;

[0021] By selecting the maximum value of the new random variable, the optimal category is obtained and a one-hot encoded mask is generated;

[0022] Use the Softmax function to approximate the maximization operation in the Gumbel-Max process.

[0023] Furthermore, a preferred method is proposed, wherein step S4 comprises:

[0024] The fused visual feature information and inertial feature information are input into the posture output part of the two-layer bidirectional LSTM layer. A fully connected layer is used as a posture regressor to map the features into posture transformations and output a six-degree-of-freedom displacement rotation vector to represent the motion transformation within a time window.

[0025] Based on the same inventive concept, the present invention also proposes a lightweight visual-inertial fusion positioning system, the system comprising:

[0026] A data acquisition unit, used for acquiring image data and IMU data;

[0027] A feature extraction unit, used to extract visual feature information from image data using a FlowNetSimple network, and to extract inertial feature information from IMU data using a one-dimensional convolutional network;

[0028] A fusion unit is used to fuse visual feature information and inertial feature information using a hard attention mechanism;

[0029] The output unit is used to perform temporal modeling using a recurrent neural network, and an output transformation network is used to convert the output of the model into the final prediction result.

[0030] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes a lightweight visual-inertial fusion positioning method described in any one of the above items.

[0031] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a lightweight visual-inertial fusion positioning method as described in any one of the above are executed.

[0032] The present invention is beneficial in that:

[0033] In a lightweight visual-inertial fusion positioning method proposed in the present invention, lightweight network architectures such as FlowNetSimple and one-dimensional convolutional networks are used to reduce the computational complexity and memory usage of the model, making the method suitable for deployment on embedded systems or mobile devices. By fusing visual features and inertial features, it is possible to effectively deal with the problem of data loss or corruption, improve the robustness of the system, and ensure reliable positioning in various environments. The FlowNetSimple network can quickly extract optical flow features from images, and the one-dimensional convolutional network can efficiently process IMU data. This separate processing can optimize the feature extraction process for different data types. By fusing visual and inertial features through a hard attention mechanism, the model's attention to important information can be effectively enhanced, thereby improving the final positioning accuracy. Using a recurrent neural network for time series modeling can capture the temporal relationship in the data and improve the ability to adapt to changes in a dynamic environment.

[0034] The present invention is applied to the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a lightweight visual-inertial fusion positioning method according to implementation mode 1;

[0036] Figure 2 This is a flow chart of visual feature extraction described in Implementation Example 11;

[0037] Figure 3 This is a flow chart of inertial feature information extraction according to the eleventh embodiment;

[0038] Figure 4 This is a schematic diagram of the hard fusion principle described in the eleventh embodiment;

[0039] Figure 5 This is a schematic diagram of the positioning trajectory of the 07 and 10 sequences in a complex environment according to the eleventh embodiment, wherein: Figure 5 (a) represents the 07 sequence positioning trajectory in a complex environment; Figure 5 (b) represents 10 sequence positioning trajectories in a complex environment, Ground Truth represents the true trajectory, and Start Point represents the starting point;

[0040] Figure 6 This is a comparison diagram of the model efficiency of the eleventh embodiment described in the eleventh embodiment, Figure 6 (a) represents the total number of model parameters, Figure 6 (b) represents the number of multiply-add-accumulate operations. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0042] Implementation method 1, see Figure 1 The embodiment of the present invention is a lightweight visual-inertial fusion positioning method, the method comprising:

[0043] Step S1: Acquire image data and IMU data;

[0044] Step S2: Use the FlowNetSimple network to extract visual feature information from the image data, and use a one-dimensional convolutional network to extract inertial feature information from the IMU data;

[0045] Step S3: Use hard attention mechanism to fuse visual feature information and inertial feature information;

[0046] Step S4: Use a recurrent neural network for temporal modeling and use an output transformation network to convert the output of the model into the final prediction result.

[0047] In a lightweight visual-inertial fusion positioning method proposed in this embodiment, lightweight network architectures such as FlowNetSimple and one-dimensional convolutional networks are used to reduce the computational complexity and memory usage of the model, making the method suitable for deployment on embedded systems or mobile devices. By fusing visual features and inertial features, it is possible to effectively deal with the problem of data loss or corruption, improve the robustness of the system, and ensure reliable positioning in various environments. The FlowNetSimple network can quickly extract optical flow features from images, and the one-dimensional convolutional network can efficiently process IMU data. This separate processing can optimize the feature extraction process for different data types. By fusing visual and inertial features through a hard attention mechanism, the model's attention to important information can be effectively enhanced, thereby improving the final positioning accuracy. Using a recurrent neural network for time series modeling can capture the temporal relationship in the data and improve the ability to adapt to changes in a dynamic environment.

[0048] The lightweight visual-inertial fusion positioning method proposed in this embodiment has the following principles:

[0049] Data acquisition (step S1): First, image data is acquired from the camera, and inertial data is acquired from the IMU sensor. These two data sources complement each other and provide a basis for subsequent feature extraction.

[0050] Feature extraction (step S2): The FlowNetSimple network is used to extract visual features from the image, which can capture the motion information between objects. The one-dimensional convolutional network is used to extract inertial features such as acceleration and angular velocity from the IMU data.

[0051] Feature fusion (step S3): The extracted visual features and inertial features are fused using a hard attention mechanism. This mechanism can emphasize important information at a specific moment and improve the expressiveness of the fused features.

[0052] Time modeling (step S4): Use a recurrent neural network to process the fused features and capture the impact of historical information on the current state through time series modeling. Finally, use the output transformation network to transform the output of the model into the final positioning result to ensure the availability and accuracy of the model output.

[0053] Implementation method 2. This implementation method further limits the lightweight visual-inertial fusion positioning method described in implementation method 1. In step S2, the FlowNetSimple network is used to extract visual feature information from the image data, including: the visual encoding part of the FlowNetSimple network includes a neural network structure composed of 10 layers of CNN, and the neural network structure composed of 10 layers of CNN outputs motion feature information, which is pooled and then sent to a three-layer fully connected network, and finally outputs the visual feature information.

[0054] The design of the FlowNetSimple network in this embodiment focuses on optical flow estimation, which can effectively capture motion information in image sequences. This is particularly important for positioning in dynamic environments and helps to identify object movement and scene changes. The 10-layer CNN structure enables the model to extract image features layer by layer through multi-layer convolution operations, from low-level features (such as edges, textures) to high-level features (such as shapes, motion patterns), providing richer and more detailed visual information. The pooling layer effectively reduces the feature dimension, reduces the computational complexity, and enhances the abstraction ability of features. By compressing information and retaining key features, the efficiency of subsequent fully connected network processing is improved. The pooled features are sent to a three-layer fully connected network to further integrate and process motion feature information. This structure allows the model to perform more complex mapping at the feature level and generate more accurate visual feature information. The use of multi-layer structures and pooling operations helps the model generalize and reduce the risk of overfitting on training data. This can improve the adaptability of the model in real applications, especially under different environmental conditions.

[0055] This implementation still maintains the advantage of being lightweight, and at the same time ensures positioning accuracy through reasonable network design and feature extraction methods, which makes the method more suitable for running on resource-constrained devices and broadens the scope of application.

[0056] Implementation method three: This implementation method is a further limitation of the lightweight visual-inertial fusion positioning method described in implementation method one. In step S2, a one-dimensional convolutional network is used to extract inertial feature information from IMU data, including: obtaining angular velocity and acceleration data from the IMU data, and the input angular velocity and acceleration data are respectively passed through two one-dimensional convolutional layers and a maximum pooling layer, and the outputs are connected in series, and then input into a fully connected layer to output the inertial feature information.

[0057] In this embodiment, a one-dimensional convolutional network is used to extract inertial features from IMU data, which can effectively capture the temporal features of angular velocity and acceleration. This convolutional layer can learn local features and reduce the data dimension through the maximum pooling layer, thereby improving computational efficiency.

[0058] Compared with traditional feature extraction methods, convolutional neural networks (CNNs) significantly reduce the amount of computation and storage requirements under the mechanism of parameter sharing. This makes this method suitable for resource-constrained embedded systems, especially on mobile devices. By combining visual data with IMU data, a more comprehensive understanding of the environment can be obtained. Visual data provides rich spatial information, while IMU data provides real-time feedback on the state of motion. The fusion of the two improves positioning accuracy and robustness. IMU sensors may perform poorly under certain conditions (such as insufficient light or environmental changes), but after fusing visual information, the system can maintain stability in different environments and improve overall performance. The architecture of this method can be adjusted according to different application scenarios. For example, the network parameters can be optimized according to different motion modes (such as walking, running, or driving) to improve positioning accuracy. Through lightweight design, this method can achieve real-time processing to meet the needs of real-time positioning and navigation, and is very suitable for scenarios that require fast response, such as autonomous driving and drones.

[0059] Embodiment 4: This embodiment further limits the lightweight visual-inertial fusion positioning method described in Embodiment 1, and step S3 includes:

[0060] Step S31: Generate a hard mask by resampling from a Bernoulli distribution;

[0061] Step S32: The generated hard mask and visual feature vector a V and the inertial eigenvector a I Multiply element by element to get a new reweighted vector;

[0062] Step S33: resampling the mask using the Gumbel-Max technique;

[0063] Step S34: concatenate the re-weighted visual features and inertial features to form a final fused feature vector.

[0064] In this embodiment, by resampling from the Bernoulli distribution to generate a hard mask, important features can be effectively selected. This mechanism enables the network to focus on the most relevant information, suppress unnecessary noise, and improve the accuracy of feature fusion. Visual features and inertial features are complementary in expressing environmental information and motion status. After reweighting by element-by-element multiplication, the advantages of the two can be better integrated to form a more comprehensive feature vector and improve positioning accuracy. The mask is resampled using the Gumbel-Max technique, making the fusion process random and flexible. This flexibility allows the model to adaptively select important features under different input conditions, thereby enhancing the robustness of the model.

[0065] The hard attention mechanism only focuses on a part of the features in feature selection, reducing the complexity of subsequent calculations. Compared with the traditional soft attention mechanism, this method significantly reduces the computational cost while retaining performance, making it suitable for real-time applications.

[0066] Embodiment 5: This embodiment further limits the lightweight visual-inertial fusion positioning method described in Embodiment 4. Step S32 includes:

[0067] g hard (a V ,a I )=[a V es V ; a I es I ]

[0068] Among them, s V and I A hard mask.

[0069] Embodiment 6: This embodiment further limits the lightweight visual-inertial fusion positioning method described in embodiment 4. Step S33 includes:

[0070] Combine the random variable sampled from the Gumbel distribution with the class probability to form a new random variable;

[0071] By selecting the maximum value of the new random variable, the optimal category is obtained and a one-hot encoded mask is generated;

[0072] Use the Softmax function to approximate the maximization operation in the Gumbel-Max process.

[0073] This implementation method can form a more discriminative random variable by combining the random variable sampled from the Gumbel distribution with the category probability. This method effectively improves the accuracy of selecting the optimal category, thereby enhancing the effect of feature fusion. By maximizing the new random variable, the generated one-hot encoding mask can clearly indicate which features are the most important in the current decision. The use of the Softmax function to approximate the maximization operation in the Gumbel-Max process avoids the mutation caused by hard selection, makes the model smoother during training, reduces the instability during training, and helps convergence. The calculation of the Softmax function is relatively simple, which can effectively reduce the computational overhead in the feature selection process. This enables the method to still run efficiently in a resource-constrained environment, meeting the requirements of lightweight design. Combining the mechanism of random sampling with Softmax enables the model to adapt to different input conditions. This flexibility can dynamically adjust feature selection in complex environments and enhance the robustness of the system.

[0074] Embodiment 7: This embodiment further limits the lightweight visual-inertial fusion positioning method described in Embodiment 1, and step S4 includes:

[0075] The fused visual feature information and inertial feature information are input into the posture output part of the two-layer bidirectional LSTM layer. A fully connected layer is used as a posture regressor to map the features into posture transformations and output a six-degree-of-freedom displacement rotation vector to represent the motion transformation within a time window.

[0076] In this implementation, the bidirectional LSTM (Long Short-Term Memory Network) can effectively capture long-distance dependencies in time series data. By considering past and future contextual information, the model performs better in understanding motion patterns and temporal changes, improving the accuracy of positioning. Combining visual features and inertial features into the LSTM layer allows the model to comprehensively analyze these two information sources, enhancing the complementarity of features. This fusion helps to more fully understand the motion state and improve positioning results.

[0077] The gating mechanism of LSTM enables the network to dynamically adjust the importance of different input information, and can adaptively optimize feature processing under changing environmental conditions to improve the robustness of the model. By mapping features to posture transformations through a fully connected layer, the transformation from abstract features to specific postures can be effectively achieved. This simple and efficient regression method provides a clear path for posture output and reduces complexity. Outputting a six-degree-of-freedom displacement and rotation vector can fully describe the motion state of the object, providing rich information, and is suitable for various application scenarios, such as robot navigation and augmented reality. The bidirectional LSTM structure can reduce the consumption of computing resources while ensuring performance compared to traditional deep networks, meeting the needs of lightweight applications.

[0078] Embodiment 8: A lightweight visual-inertial fusion positioning system described in this embodiment includes:

[0079] A data acquisition unit, used for acquiring image data and IMU data;

[0080] A feature extraction unit, used to extract visual feature information from image data using a FlowNetSimple network, and to extract inertial feature information from IMU data using a one-dimensional convolutional network;

[0081] A fusion unit is used to fuse visual feature information and inertial feature information using a hard attention mechanism;

[0082] The output unit is used to perform temporal modeling using a recurrent neural network, and an output transformation network is used to convert the output of the model into the final prediction result.

[0083] Embodiment 9. A computer device described in this embodiment includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a lightweight visual-inertial fusion positioning method described in any one of embodiments 1 to 7.

[0084] Embodiment 10. A computer-readable storage medium described in this embodiment stores a computer program, and when the computer program is executed by a processor, the steps of a lightweight visual-inertial fusion positioning method as described in any one of embodiments 1 to 7 are executed.

[0085] Implementation Method 11: See Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6This embodiment provides a specific example of a lightweight visual-inertial fusion positioning method described in the first embodiment, and is also used to explain the second to seventh embodiments. Specifically:

[0086] Get image data and IMU data, such as Figure 2 As shown in the figure, the FlowNetSimple network extracts visual feature information from image data. The visual encoding part of the FlowNetSimple network is designed with a neural network structure consisting of 10 layers of CNN. The motion feature information output by the structure is then pooled and sent to a three-layer fully connected network, and finally the visual feature information is output. Figure 3 As shown in the figure, the number of input angular velocity and acceleration is determined according to the IMU sampling frequency. The input angular velocity and acceleration data are respectively passed through two one-dimensional convolutional layers and a maximum pooling layer, and the outputs are connected in series. Then, they are input into a fully connected layer to output the inertial feature information.

[0087] Hard fusion learns a random function to generate a binary, each feature mask propagates features or blocks features. Random layers cannot be trained directly through backpropagation because gradients do not propagate through discrete latent variables. In this implementation, a lightweight method Gumbel-Softmax resampling is used to infer random layers, so that hard fusion can also be trained in an end-to-end manner. The hard fusion principle structure diagram is shown in the figure. Figure 4 shown.

[0088] Assume that the visual feature vector output by the visual front-end network is a V , the inertial feature vector output by the inertial front-end network is a I , hard mask s V and I Instead of deterministically learning the mask from the features, we resample from a Bernoulli distribution with parameter α, which is conditioned on the features but adds random noise:

[0089]

[0090] The features are element-wise multiplied with their corresponding hard masks as new reweighted vectors, and the random hard fusion function is modeled as

[0091] g hard (a V ,a I )=[a V es V ; a I es I ] (2)

[0092] A pair of probability variables α V and α Iis restricted to the concatenation of visual and inertial feature vectors [a V ; a I ]superior:

[0093] α V =Sigmoid V ([a V ; a I ])

[0094] α I =Sigmoid I ([a V ; a I ]) (3)

[0095] Among them, the probability variable is an n-dimensional vector α=[π1,...,π n ], indicating the probability of each feature being selected or not selected at position n. The Sigmoid function reweights each vector in the range [0,1].

[0096] The Gumbel-max technique allows to i and the random variable ε i Sample s is extracted from the category distribution in the case of , and then one-hot encoding is performed to binarize the category:

[0097] s = one_hot(argmax i [ε i +logπ i ]) (4)

[0098] For any have:

[0099]

[0100] It can be viewed as a process of adding independent Gumbel perturbations to discrete probability variables. In practice, the random variable ε i Sampling from a Gumbel distribution, which is a continuous distribution over a simplex that can approximate categorical samples:

[0101] ε=-log(-log(u)),u:Uniform(0,1) (6)

[0102] Since the argmax operation is not differentiable, the Softmax function is used as an approximation, where τ>0 is the degree of regulation of the resampling process.

[0103]

[0104] The fused features are temporally modeled using a recurrent neural network (two-layer bidirectional LSTM). The basic principle of ego-motion estimation requires modeling temporal dependencies to obtain accurate posture regression. The pose output part uses a fully connected layer as a posture regressor after the recurrent network to map features to posture transformations and output a six-degree-of-freedom displacement rotation vector to represent the motion transformation within a time window.

[0105] The loss function is used for training, and the loss of each frame represents the loss between the estimated pose and the true pose between two frames. Let the true pose between each two frames at the i-th frame be The predicted pose is where p i and represents displacement, and represents rotation, β is the scale factor that balances the per-frame loss between translation and rotation, and using the root mean square error, the per-frame loss at the i-th frame is:

[0106]

[0107] The final loss function is:

[0108]

[0109] Among them, B is the amount of data passed into the neural network at a single time during the training process; k represents the dimension of relative displacement or relative rotation predicted by the network, which is 3.

[0110] In this embodiment, the KIITI dataset is also selected for experimentation to perform data corruption processing to simulate a complex environment. The results obtained by training and testing the deep learning VIO algorithm proposed in the present invention are similar to the trajectory results obtained by training and testing the VINet algorithm. Figure 5 As shown, the test sequences are 07 and 10. The left picture shows the test result of sequence 07, and the right picture shows the test result of sequence 10. It can be intuitively found that the algorithm proposed in the present invention can be closer to the real trajectory. In complex environments, compared with the VINet algorithm, the algorithm proposed in the present invention can achieve positioning more accurately.

[0111] Use the thop tool to evaluate, the results are as follows Figure 6 As shown, it can be found that the algorithm model proposed in the present invention has great improvements in memory consumption and computing requirements compared to the classic VINet.

[0112] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure rather than to limit its protection scope. Although the present disclosure has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present disclosure, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the disclosed claims to be approved.

Claims

1. A lightweight visual-inertial fusion positioning method, characterized in that: The method comprises: Step S1: Acquire image data and IMU data; Step S2: Use the FlowNetSimple network to extract visual feature information from the image data, and use a one-dimensional convolutional network to extract inertial feature information from the IMU data; Step S3: Use hard attention mechanism to fuse visual feature information and inertial feature information; Step S4: Use a recurrent neural network for temporal modeling and use an output transformation network to convert the output of the model into the final prediction result.

2. A lightweight visual-inertial fusion positioning method according to claim 1, characterized in that: In step S2, the FlowNetSimple network is used to extract visual feature information from the image data, including: the visual encoding part of the FlowNetSimple network includes a neural network structure composed of 10 layers of CNN, the neural network structure composed of 10 layers of CNN outputs motion feature information, which is then pooled and sent to a three-layer fully connected network, and finally the visual feature information is output.

3. The lightweight visual-inertial fusion positioning method according to claim 1, characterized in that: In step S2, a one-dimensional convolutional network is used to extract inertial feature information from IMU data, including: obtaining angular velocity and acceleration data from the IMU data, the input angular velocity and acceleration data are respectively passed through two one-dimensional convolutional layers and a maximum pooling layer, the outputs are connected in series, and then input into a fully connected layer to output the inertial feature information.

4. The lightweight visual-inertial fusion positioning method according to claim 1, characterized in that: The step S3 comprises: Step S31: Generate a hard mask by resampling from a Bernoulli distribution; Step S32: The generated hard mask and visual feature vector a V and the inertial eigenvector a I Multiply element by element to get a new reweighted vector; Step S33: resampling the mask using the Gumbel-Max technique; Step S34: concatenate the re-weighted visual features and inertial features to form a final fused feature vector.

5. A lightweight visual-inertial fusion positioning method according to claim 4, characterized in that: The step S32 comprises: g hard (the V ,the I )=[a V es V ;the I es I ] Among them, s V and I A hard mask.

6. A lightweight visual-inertial fusion positioning method according to claim 4, characterized in that: The step S33 comprises: Combine the random variable sampled from the Gumbel distribution with the class probability to form a new random variable; By selecting the maximum value of the new random variable, the optimal category is obtained and a one-hot encoded mask is generated; Use the Softmax function to approximate the maximization operation in the Gumbel-Max process.

7. The lightweight visual-inertial fusion positioning method according to claim 1, characterized in that: The step S4 comprises: The fused visual feature information and inertial feature information are input into the posture output part of the two-layer bidirectional LSTM layer. A fully connected layer is used as a posture regressor to map the features into posture transformations and output a six-degree-of-freedom displacement rotation vector to represent the motion transformation within a time window.

8. A lightweight visual-inertial fusion positioning system, characterized in that: The system comprises: A data acquisition unit, used for acquiring image data and IMU data; A feature extraction unit, used to extract visual feature information from image data using a FlowNetSimple network, and to extract inertial feature information from IMU data using a one-dimensional convolutional network; A fusion unit is used to fuse visual feature information and inertial feature information using a hard attention mechanism; The output unit is used to perform temporal modeling using a recurrent neural network, and an output transformation network is used to convert the output of the model into the final prediction result.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes a lightweight visual-inertial fusion positioning method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a lightweight visual-inertial fusion positioning method as described in any one of claims 1 to 7 are executed.

Images