Pedestrian visual-inertial combined navigation method and device based on flexible attention mechanism
Through the pedestrian visual inertial combination navigation method based on flexible attention mechanism, visual and inertial feature vectors are extracted and fused, and the problem of insufficient fusion capability of multi-source data in pedestrian navigation is solved, and navigation performance is improved, especially in complex environments. sex.
Patent Information
- Application Number
- CN202210482460.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-05
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2042-05-05
AI Technical Summary
The existing pedestrian visual inertial combined navigation algorithms have poor performance in multi-sensor information fusion, especially when the camera is blocked, insufficient light, and large inertial sensor noise or drift, the navigation performance is degraded, and the deep neural network cannot be effectively fused. Multi-sensor information.
Using a pedestrian visual inertial combination navigation method based on a flexible attention mechanism, by obtaining visual and inertial data, using convolutional neural networks and long-term memory networks to extract feature vectors, and screening and fusion of feature vectors based on a flexible attention mechanism, introducing Time series modeling to estimate pedestrian position and pose.
It effectively solves the problem of insufficient multi-source data fusion capability in pedestrian navigation, improves navigation performance, especially in complex environments, where position and posture can be accurately estimated.
Smart Images
Figure CN114894184B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of pedestrian navigation technology, and in particular to a pedestrian vision-inertial combined navigation method and device based on a flexible attention mechanism. Background Art
[0002] Pedestrian navigation technology is widely used in firefighting and disaster relief, personal location services, monitoring and tracking in large buildings, and individual combat. When satellite signals are missing or blocked, such as indoors, in blocks with tall buildings, and in tunnels, visual and inertial sensors are the two main sensors for pedestrian autonomous navigation. Both are low-cost, and through the collected images and inertial measurements, the pedestrian navigation system can estimate the user's own movement and location information.
[0003] Existing pedestrian visual-inertial integrated navigation algorithms usually include feature detection and tracking, sensor fusion, and backend optimization. These algorithms rely on manually designed features and use filtering or nonlinear optimization for information fusion. However, in actual use, problems such as camera occlusion, insufficient light, excessive inertial sensor noise or drift, and time asynchrony between multi-sensor information streams may occur, resulting in reduced navigation performance. How to effectively fuse multi-source navigation information has become a core issue. In recent years, deep neural networks have gradually been used to solve navigation problems. Although deep neural networks are good at automatically extracting features for pose estimation, these deep learning-based navigation models do not consider how to effectively fuse multi-sensor information, and the performance is still not ideal. Summary of the invention
[0004] In view of the deficiencies in the above-mentioned prior art, the present invention provides a pedestrian visual-inertial combined navigation method and device based on a flexible attention mechanism, which can effectively solve the problem of insufficient multi-source data fusion capability in pedestrian navigation.
[0005] To achieve the above object, the present invention provides a pedestrian visual-inertial integrated navigation method based on a flexible attention mechanism, comprising the following steps:
[0006] Step 1, obtaining visual data and inertial data generated by pedestrian movement, and obtaining a visual feature vector and an inertial feature vector based on the visual data and the inertial data respectively;
[0007] Step 2: Based on the flexible attention mechanism, the visual feature vector and the inertial feature vector are screened. According to the current environment dynamics and the characteristics of the self-motion, the feature vectors that are helpful for pose estimation are screened out, and the high-noise and unreliable features are excluded to obtain the fused feature vector.
[0008] Step 3: Perform time series modeling on the fused feature vector to obtain a fused feature vector that takes time characteristics into account, and then map the fused feature vector to a posture transformation to obtain the position and posture information of the pedestrian.
[0009] In another embodiment, the visual data is an image, and the inertial data includes acceleration data and gyroscope data.
[0010] In another embodiment, in step 1, the visual feature vector and the inertial feature vector are both extracted by a pedestrian visual inertial navigation model based on a deep neural network;
[0011] The pedestrian visual inertial navigation model based on deep neural network includes convolutional neural network and long short-term memory network, and its training process is as follows:
[0012] Based on the visual data and inertial data generated by pedestrian movement, the actual position and posture of pedestrians in motion are marked as label data, which is used as a dataset for model training and testing;
[0013] An end-to-end training method is adopted to train a pedestrian visual inertial navigation model based on a deep neural network on the dataset to obtain a trained pedestrian navigation model.
[0014] In another embodiment, in step 1, the process of acquiring the visual feature vector is: extracting the visual feature vector from two consecutive image frames using a convolutional neural network;
[0015] The process of acquiring the inertial feature vector is as follows: using a long short-term memory network to extract the inertial feature vector from the inertial data between two consecutive image frames.
[0016] In another embodiment, in step 2, the process of obtaining the fused feature vector is:
[0017] Connect the visual feature vector and the inertial feature vector together to form a feature vector x, and calculate the dot product similarity of two different encoding mapping functions of the feature vector x;
[0018] Based on the dot product similarity, the flexible attention variable is obtained, and the attention vector is further obtained. The fused feature vector is obtained by adding the attention vector to the feature vector x.
[0019] In another embodiment, in step 2, the process of obtaining the fused feature vector is specifically as follows:
[0020] The visual feature vector a corresponding to two consecutive image frames V and the inertial eigenvector a I Connect together to form a feature vector x, and calculate the encoding mapping function θ(xi ) and Ф(x j )’s dot product similarity:
[0021] S(x i ,x j )=θ(x i ) T Ф(x j )
[0022] In the formula, S(x i ,x j ) represents the dot product similarity, θ(x i )=W θ x i and Ф(x j )=W Ф x j Map the feature vector x at position i and position j into two feature spaces respectively, where W θ Denotes the transformation matrix that maps the feature vector to the feature space θ, W Ф represents the transformation matrix that maps the feature vector to the feature space Φ;
[0023] The normalization factor is defined as Accumulate all features at position j and define another linear transformation function g(x j )=W g x j , where W g Represents the transformation matrix that maps the feature vector to the feature space g;
[0024] That is, the flexible attention variable y can be obtained as:
[0025]
[0026] In the formula, the flexible attention variable y i Indicates the eigenvector x at position i i How much weight should be given when integrating;
[0027] Finally, the calculation of the flexible attention mechanism can be expressed as:
[0028]
[0029] The residual channel is introduced, and the linearly transformed attention vector is added to the feature vector x to obtain:
[0030] Fusion(x)=α(y)+x
[0031] α(y)=W α y
[0032] In the formula, Fusion(x) is the fused feature vector, α(y) is the attention vector, and W α represents the linear transformation matrix that maps the attention vector to the feature space α;
[0033] In this way, by introducing a non-local flexible attention fusion function, the visual and inertial features are adjusted to reweight each feature and obtain the fused feature vector Fusion(x).
[0034] To achieve the above object, the present invention also provides a pedestrian visual-inertial combined navigation device based on a flexible attention mechanism, comprising:
[0035] Data acquisition module: constructs a data set for model training based on visual data and inertial data generated by pedestrian movement, wherein the visual data is an image and the inertial data includes acceleration data and gyroscope data;
[0036] Visual feature extraction module: uses a convolutional neural network to extract visual feature vectors from two consecutive monocular image frames;
[0037] Inertial feature extraction module: uses long short-term memory network to extract inertial feature vectors from inertial measurement sequences;
[0038] Multi-source navigation information fusion module: Based on the flexible attention mechanism, the visual feature vector and the inertial feature vector are screened, and the feature vectors that are helpful for posture estimation are screened according to the current environment dynamics and the characteristics of the vehicle's own motion, and high-noise and unreliable features are excluded to obtain the fused feature vector;
[0039] Time series modeling module: performs time series modeling on the fused feature vector to obtain a fused feature vector that takes time characteristics into account;
[0040] Pose estimation module: Use the fully connected layer network to map the fused feature vector to the pose transformation to obtain the position and pose information of the pedestrian.
[0041] The present invention provides a pedestrian visual-inertial combined navigation method and device based on a flexible attention mechanism. The visual and inertial information of pedestrians in motion are collected and prepared as a data set, and a navigation model based on a deep neural network is trained end-to-end. Feature vectors are extracted from images and inertial data respectively using a visual feature extraction module and an inertial feature extraction module. Then, feature selection and fusion are performed based on a flexible attention mechanism, and the visual and inertial feature vectors are re-weighted to obtain a fused feature vector. Then, the time correlation is modeled through a time series modeling module, and accurate position and posture estimation is obtained based on the fused features, thereby solving the problem of insufficient multi-source data fusion capability in pedestrian navigation. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0043] Figure 1 A schematic diagram of a flow chart of a pedestrian vision-inertial combined navigation method according to an embodiment of the present invention;
[0044] Figure 2 It is a schematic diagram of the structure of the flexible attention mechanism in an embodiment of the present invention;
[0045] Figure 3 Schematic diagram of the modules of the pedestrian vision-inertial combined navigation device in an embodiment of the present invention.
[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0049] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0050] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0051] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0052] This embodiment discloses a pedestrian visual-inertial combined navigation method based on a flexible attention mechanism. The method collects visual and inertial information of pedestrians during movement and produces a data set, and performs end-to-end training on a navigation model based on a deep neural network; the visual feature extraction module and the inertial feature extraction module are used to extract feature vectors from the image and inertial data respectively; then, feature selection and fusion are performed based on the flexible attention mechanism, and the visual and inertial feature vectors are re-weighted to obtain a fused feature vector; the time correlation is modeled through a time series modeling module, and accurate position and posture estimation is obtained based on the fused features, thereby solving the problem of insufficient multi-source data fusion capabilities in pedestrian navigation. Reference Figure 1 The pedestrian visual-inertial combined navigation method in this embodiment specifically includes the following steps 1-3.
[0053] Step 1: Obtain visual data and inertial data generated by pedestrian movement, and obtain visual feature vectors and inertial feature vectors based on the visual data and inertial data, respectively. The visual data is an RGB image, which can be obtained through a camera device; the inertial data includes acceleration data and gyroscope data, and the collection of inertial data can be achieved through a micro inertial measurement unit (MIMU).
[0054] In this embodiment, the visual feature vector and the inertial feature vector are extracted by a pedestrian visual inertial navigation model based on a deep neural network. The pedestrian visual inertial navigation model based on a deep neural network includes a convolutional neural network and a long short-term memory network, and its training process is:
[0055] Based on the visual data and inertial data generated by pedestrian movement, the actual position and posture of pedestrians in motion are marked as label data, which is used as a dataset for model training and testing;
[0056] An end-to-end training method is adopted to train a pedestrian visual inertial navigation model based on a deep neural network on the dataset to obtain a trained pedestrian navigation model.
[0057] Visual feature vector a V The acquisition process is: use a convolutional neural network to obtain data from two consecutive image frames x V Extract the visual feature vector a V The specific implementation process is as follows:
[0058] Use the convolutional neural network-based optical flow estimation network as the visual feature extraction module f 视觉 Perform visual feature extraction. The visual feature extraction module sets the number of network input layer channels to 6 and receives RGB images of two adjacent frames on the time axis. The visual feature extractor consists of nine convolutional layers, and the size of the receptive field gradually decreases from 7×7 to 5×5 and finally to 3×3. The step size of the first 6 layers is 2. Except for the last layer, each layer is followed by a ReLU nonlinear activation function. In this embodiment, the feature a of the last convolutional layer is used. V As a visual feature, that is, a V =f 视觉 (x V ).
[0059] Inertial eigenvector a I The acquisition process is: use the long short-term memory network to obtain the inertial data x between two consecutive image frames I The inertia eigenvector is extracted from . The specific implementation process is:
[0060] A two-layer bidirectional long short-term memory network (LSTM) is used as the inertial feature extraction module. The inertial measurement result x between each two images I The inertial feature extraction module is used to extract the inertial feature vector. When designing the inertial feature extraction module, considering that the inertial data stream has a strong time characteristic and the frequency is much higher than the image stream frequency, the inertial feature extraction module f 惯性 A two-layer long short-term memory network (LSTM) with 256 hidden states is used. The inertial measurement sequence x between each two images I Use the inertial feature extractor to extract the inertial feature vector a I , that is, a I =f 惯性 (x I ).
[0061] Step 2: Based on the flexible attention mechanism, the visual feature vector and the inertial feature vector are screened. According to the current environment dynamics and the characteristics of the self-motion, the feature vectors that are helpful for pose estimation are screened out, and high-noise and unreliable features are excluded to obtain the fused feature vector. The process of obtaining the fused feature vector is as follows:
[0062] Connect the visual feature vector and the inertial feature vector together to form a feature vector x, and calculate the dot product similarity of two different encoding mapping functions of the feature vector x;
[0063] Based on the dot product similarity, the flexible attention variable is obtained, and the attention vector is further obtained. The fused feature vector is obtained by adding the attention vector to the feature vector x.
[0064] In the specific implementation process, refer to Figure 2 , the process of obtaining the fused feature vector is as follows:
[0065] The visual feature vector a corresponding to two consecutive image frames V and the inertial eigenvector a I Connect together to form a feature vector x, and calculate the encoding mapping function θ(x i ) and Ф(x j )’s dot product similarity:
[0066] S(x i ,x j )=θ(x i ) T Ф(x j )
[0067] In the formula, S(x i ,x j ) represents the dot product similarity, θ(x i )=W θ x i and Ф(x j )=W Ф x j Map the feature vector x at position i and position j into two feature spaces respectively, where W θ Denotes the transformation matrix that maps the feature vector to the feature space θ, W Ф represents the transformation matrix that maps the feature vector to the feature space Φ;
[0068] The normalization factor is defined as Accumulate all features at position j and define another linear transformation function g(x j )=W g x j , where W gRepresents the transformation matrix that maps the feature vector to the feature space g;
[0069] That is, the flexible attention variable y can be obtained as:
[0070]
[0071] In the formula, the flexible attention variable y i Indicates the eigenvector x at position i i How much weight should be given when integrating;
[0072] Finally, the calculation of the flexible attention mechanism can be expressed as:
[0073]
[0074] The residual channel is introduced, and the linearly transformed attention vector is added to the feature vector x to obtain:
[0075] Fusion(x)=α(y)+x
[0076] α(y)=W α y
[0077] In the formula, Fusion(x) is the fused feature vector, α(y) is the attention vector, and W α represents the linear transformation matrix that maps the attention vector to the feature space α;
[0078] In this way, by introducing a non-local flexible attention fusion function, the visual and inertial features are adjusted to reweight each feature and obtain the fused feature vector Fusion(x).
[0079] Step 3: Perform time series modeling on the fused feature vector to obtain a fused feature vector that takes time characteristics into account. Then, the fused feature vector is mapped to a posture transformation to obtain the position and posture information of the pedestrian, specifically:
[0080] The fused feature vector x is modeled using a two-layer long short-term memory network, and the feature vector x at the current moment is associated with historical information to obtain a new feature variable. The pose estimation module based on the fully connected layer is used to map the feature to the pose transformation y. t , represents the motion transformation at the current moment, and obtains the position and posture of the pedestrian.
[0081] Example 2
[0082] like Figure 3The figure shows a pedestrian visual-inertial combined navigation device based on a flexible attention mechanism disclosed in this embodiment, which mainly includes a data acquisition module, a visual feature extraction module, an inertial feature extraction module, a multi-source navigation information fusion module, a time series modeling module and a posture estimation module. The data acquisition module constructs a data set for model training based on the visual data and inertial data generated by pedestrian movement, where the visual data is an image and the inertial data includes acceleration data and gyroscope data; the visual feature extraction module uses a convolutional neural network to extract visual feature vectors from two consecutive monocular image frames; the inertial feature extraction module uses a long short-term memory network to extract inertial feature vectors from the inertial measurement sequence; the multi-source navigation information fusion module screens the visual feature vectors and inertial feature vectors based on the flexible attention mechanism, and screens out feature vectors that are helpful for posture estimation based on the current environmental dynamics and its own motion characteristics, excludes high-noise and unreliable features, and obtains the fused feature vector; the time series modeling module performs time series modeling on the fused feature vector to obtain a fused feature vector that takes into account the time characteristics; the posture estimation module: uses a fully connected layer network to map the fused feature vector to a posture transformation to obtain the position and posture information of the pedestrian.
[0083] The pedestrian visual-inertial combined navigation device collects visual and inertial information of pedestrians during movement and produces a data set; uses a visual feature extraction module and an inertial feature extraction module to extract feature vectors from images and inertial data respectively; then re-weights the visual and inertial feature vectors through feature selection and fusion based on a flexible attention mechanism to obtain a fused feature vector; then models the time correlation through a timing modeling module, and obtains accurate position and posture estimation based on the fused features, thereby solving the problem of insufficient multi-source data fusion capability in pedestrian navigation.
[0084] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.
Claims
1. A pedestrian visual-inertial combined navigation method based on a flexible attention mechanism, characterized in that: The steps include: Step 1, obtaining visual data and inertial data generated by pedestrian movement, and obtaining a visual feature vector and an inertial feature vector based on the visual data and the inertial data respectively; Step 2: Based on the flexible attention mechanism, the visual feature vector and the inertial feature vector are screened. According to the current environment dynamics and the characteristics of the self-motion, the feature vector that is helpful for posture estimation is screened out, and the high-noise and unreliable features are excluded to obtain the fused feature vector. The process of obtaining the fused feature vector is as follows: Connect the visual feature vector and the inertial feature vector together to form a feature vector x, and calculate the dot product similarity of two different encoding mapping functions of the feature vector x; Based on the dot product similarity, the flexible attention variable is obtained, and the attention vector is further obtained. The fused feature vector is obtained by adding the attention vector to the feature vector x. The process of obtaining the fused feature vector is specifically as follows: The visual feature vector a corresponding to two consecutive image frames V and the inertial eigenvector a I Connect together to form a feature vector x, and calculate the encoding mapping function θ(x i ) and φ(x j )’s dot product similarity: S(x i ,x j )=θ(x i ) T Ф(x j ) In the formula, S(x i ,x j ) represents the dot product similarity, θ(x i )=W θ x i and Ф(x j )=W Ф x j Map the feature vector x at position i and position j into two feature spaces respectively, where W θ represents the transformation matrix that maps the feature vector to the feature space θ, W Ф represents the transformation matrix that maps the feature vector to the feature space Φ; The normalization factor is defined as Accumulate all features at position j and define another linear transformation function g(x j )=W g x j , where W g Represents the transformation matrix that maps the feature vector to the feature space g; That is, the flexible attention variable y can be obtained as: In the formula, the flexible attention variable y i Indicates the eigenvector x at position i i How much weight should be given when integrating; Finally, the calculation of the flexible attention mechanism can be expressed as: The residual channel is introduced, and the linearly transformed attention vector is added to the feature vector x to obtain: Fusion(x)=α(y)+x α(y)=W α y In the formula, Fusion(x) is the fused feature vector, α(y) is the attention vector, and W α represents the linear transformation matrix that maps the attention vector to the feature space α; In this way, by introducing a non-local flexible attention fusion function, the visual and inertial features are adjusted to reweight each feature, and the fused feature vector Fusion(x) is obtained; Step 3: Perform time series modeling on the fused feature vector to obtain a fused feature vector that takes time characteristics into account, and then map the fused feature vector to a posture transformation to obtain the position and posture information of the pedestrian.
2. The pedestrian visual-inertial integrated navigation method based on the flexible attention mechanism according to claim 1 is characterized in that: The visual data is an image, and the inertial data includes acceleration data and gyroscope data.
3. The pedestrian visual-inertial integrated navigation method based on the flexible attention mechanism according to claim 2 is characterized in that: In step 1, the visual feature vector and the inertial feature vector are both extracted by a pedestrian visual inertial navigation model based on a deep neural network; The pedestrian visual inertial navigation model based on deep neural network includes convolutional neural network and long short-term memory network, and its training process is as follows: Based on the visual data and inertial data generated by pedestrian movement, the actual position and posture of pedestrians in motion are marked as label data, which is used as a dataset for model training and testing; An end-to-end training method is adopted to train a pedestrian visual inertial navigation model based on a deep neural network on the dataset to obtain a trained pedestrian navigation model.
4. The pedestrian visual-inertial integrated navigation method based on the flexible attention mechanism according to claim 2 or 3 is characterized in that: In step 1, the process of acquiring the visual feature vector is: extracting the visual feature vector from two consecutive image frames using a convolutional neural network; The process of acquiring the inertial feature vector is as follows: using a long short-term memory network to extract the inertial feature vector from the inertial data between two consecutive image frames.
5. A pedestrian visual-inertial combined navigation device based on a flexible attention mechanism, characterized in that: The pedestrian visual-inertial combined navigation method according to claim 1, 2, 3 or 4 comprises: Data acquisition module: constructs a data set for model training based on visual data and inertial data generated by pedestrian movement, wherein the visual data is an image and the inertial data includes acceleration data and gyroscope data; Visual feature extraction module: uses a convolutional neural network to extract visual feature vectors from two consecutive monocular image frames; Inertial feature extraction module: uses long short-term memory network to extract inertial feature vectors from inertial measurement sequences; Multi-source navigation information fusion module: Based on the flexible attention mechanism, visual feature vectors and inertial feature vectors are screened, and feature vectors that are helpful for pose estimation are screened according to the current environment dynamics and the characteristics of the vehicle's own motion, and high-noise and unreliable features are excluded to obtain the fused feature vectors. Time series modeling module: performs time series modeling on the fused feature vector to obtain a fused feature vector that takes time characteristics into account; Pose estimation module: Use the fully connected layer network to map the fused feature vector to the pose transformation to obtain the position and pose information of the pedestrian.