A visual positioning system and method based on feature noise reduction

Abstract

The present invention relates to a visual positioning system and method based on feature denoising. The system comprises: a feature extraction network that extracts features from an input image to obtain image features; and a visual positioning network based on feature denoising that performs visual positioning based on the image features and outputs a pose vector sequence, including a current estimated pose and several future estimated poses. Compared to existing technologies, the present invention can provide real-time, offline motion pose predictions for the current and future based on the current scene, enabling multi-position point predictions and exhibiting significant generalization capabilities even when the motion path is not fixed.

Description

Technical Field

[0001] The present invention relates to a visual positioning system, in particular to a visual positioning system based on feature noise reduction. Background Art

[0002] Existing positioning technologies mainly include the following:

[0003] 1. Structure-based visual localization techniques: Structure-based pose regressors first extract sparse image features. They then perform inter-frame estimation and loop closure detection by matching feature points. Examples include visual SLAM based on the scale-invariant feature transform (SIFT), oriented FAST, and rotation-based BRIEF (ORB). SIFT and ORB features are widely used in visual SLAM due to their robustness, discriminative ability, and fast processing speed.

[0004] 2. The deep learning model PoseNet is used in camera relocalization: PoseNet, adapted from Google LeNet, uses a deep neural network to learn the implicit projection relationship between image poses. By inputting a pair of images into a trained PoseNet and discarding the activations of the convolutional layer, pose samples with a certain probability are generated, thus completing image localization.

[0005] 3. Visual localization technology based on structured dimensionality reduction: This paper proposes a deep learning framework based on LSTM to reconstruct scene features. This method uses a deep framework to reduce the scale of scene features, thus reducing the computational complexity by several orders of magnitude compared to SfM, thereby significantly improving localization performance.

[0006] However, the above positioning technology has the following defects:

[0007] 1. Structure-based visual localization techniques require expensive computation and storage resources and have strict and uninterrupted initial frame requirements. In contrast, deep learning-based pose regulators are more fault-tolerant and can better handle frame loss during transmission. In addition, deep learning-based pose regulators require less model storage space and are faster when performing real-time inference.

[0008] 2. Deep learning-based visual positioning technology Due to the limitations of existing dataset acquisition methods, deep learning-based pose regressors are prone to overfitting during training. These methods assume that the input images are completely unrelated and produce independent pose estimates. When applied to image sequences, these estimates will have non-negligible image noise.

[0009] 3. Positioning in actual industrial application scenarios is vulnerable to GPS attacks. Control decisions based on the fusion of GPS and inertial navigation systems: Even if attackers do not rely entirely on GPS information, they can use image attacks to influence the control signals involved in decision-making. An image attack model is currently available that identifies dirty road patches as a new domain-specific threat model, formulates the attack as an optimization problem, and solves the challenge of interdependence between attacks from pairs of consecutive camera frames. Even with inertial navigation sensors such as gyroscopes and accelerometers, this attack successfully drives the target vehicle out of the lane boundary within 1.3 seconds. The road trajectories they produce are indistinguishable from good paths (in terms of turning angles and road curvature).

[0010] 4. In visual simultaneous localization and mapping (VSLAM), to estimate the continuous pose of the camera, structure-based SLAM relies on a collected dataset to reconstruct the 3D environment and then save the environment map. This is very time-consuming and labor-intensive, and requires a high initial frame rate. It is difficult to unify different local maps into a single world coordinate system. Without GPS correction, only local positioning is possible. Summary of the Invention

[0011] The purpose of the present invention is to provide a visual positioning system based on feature noise reduction in order to overcome the defects of the above-mentioned prior art.

[0012] The purpose of the present invention can be achieved by the following technical solutions:

[0013] A visual positioning system based on feature denoising, the system comprising:

[0014] Feature extraction network: extract features from the input image to obtain image features;

[0015] Feature-based denoising visual localization network: performs visual localization based on image features and outputs a pose vector sequence, which includes the current estimated pose and several future estimated poses.

[0016] Preferably, the visual positioning network based on feature denoising includes:

[0017] Deep state module: used for one-step prediction of pose information;

[0018] Depth sensing module: used for two-step prediction of pose information.

[0019] Preferably, each pose in the pose vector sequence consists of a 3D camera position x and a direction represented by a quaternion q.

[0020] Preferably, the depth state module and the depth sensing module are respectively neural networks based on GoogLeNet, and the last fully connected layer of the GoogLeNet neural network is expanded from 7 dimensions to 28 dimensions to predict the current posture and the next three postures.

[0021] Preferably, the objective loss function of the visual positioning network training based on feature denoising is:

[0022]

[0023] Among them, loss(I) is the loss of the visual positioning network I based on feature denoising, x is the 3D camera position label value, q is the direction label value, is the predicted value of the 3D camera position, is the direction prediction value, β is the scale factor, n is the total number of images used for training, i represents the i-th image used for training, α i Represents the weight corresponding to the i-th image.

[0024] Preferably, the scaling factor β is fine-tuned by grid search.

[0025] Preferably, in an indoor scene, the scaling factor β is between 120 and 750.

[0026] Preferably, in outdoor scenes, the scaling factor β is between 250 and 2000.

[0027] A visual positioning method based on feature noise reduction, based on the system, comprising:

[0028] Input the image into the feature extraction network and perform feature extraction on the input image to obtain image features;

[0029] Visual positioning is performed based on image features, and a pose vector sequence is output. The pose vector sequence includes the current estimated pose and several future estimated poses.

[0030] Preferably, the picture is acquired using a monocular camera.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] First, we address the feature noise problem of event-based deep localization networks and propose a Kalman feature smoothing module for camera image localization. This module is embedded in an end-to-end pose estimator, and the input image directly generates the pose through the network pose inference network in an end-to-end manner.

[0033] Second, we propose a specific iterative training method for the network to improve feature noise. This method provides real-time motion pose predictions for the current and future based on the current scene, i.e., multi-position point prediction. This network architecture represents a significant improvement over event-based deep localization networks.

[0034] Third, the relocalization accuracy of the event-based deep localization network is further improved through the joint optimization of the motion inference loss function.

[0035] Fourth, experimental results demonstrate that deep neural networks can learn the motion patterns of mobile devices in a given scenario and possess the ability to reason about motion. Visual motion reasoning networks also demonstrate impressive generalization capabilities when the motion path is not fixed.

[0036] 5. Since some training sets have a limited number of images, the present invention uses transfer learning to solve the overfitting problem caused by the sparse training set. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a structural diagram of a visual positioning system based on feature noise reduction according to the present invention;

[0038] Figure 2 This is a structural block diagram of a visual positioning system based on feature noise reduction according to the present invention;

[0039] Figure 3 A theoretical model for the overall iteration of visual localization networks. DETAILED DESCRIPTION

[0040] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Note that the following embodiments are merely illustrative in nature and the present invention is not intended to limit its applicable objects or uses, and the present invention is not limited to the following embodiments.

[0041] Example

[0042] The present invention mainly introduces a visual positioning system based on feature noise reduction. Generally, a moving target takes a series of pictures of the surrounding environment from its own perspective during the movement process. The system can predict the posture of the moving target at the moment of shooting and the future movement path of the moving target through the pictures taken. Specifically, in our visual positioning network, by learning the data set (input pictures or short-term picture sequences), the system can provide real-time position prediction for the moving target under offline conditions. Furthermore, it only needs to input a picture to predict the future movement path of the moving object. The path prediction network can obtain real-time path planning when the conditions are met, and its prediction results can cooperate with the motion control system to make more time-saving and safer motion planning for unexpected situations along the way. Generally speaking, the scope of application of the present invention includes but is not limited to fields such as unmanned driving, drone swarm flight, etc. with specified paths or movement directions.

[0043] Deep learning-based localization methods have garnered significant attention in recent years. They align localization information with the world coordinate system and offer fast inference speed, meeting the real-time requirements of autonomous navigation. Although many studies have shown that the localization accuracy of deep learning methods cannot surpass that of image retrieval-based methods, event-based deep neural networks, through their relatively explicit model, can overcome these limitations. This is because they do not need to completely replicate a specific historical route; rather, they rely on memorizing past experience along a particular path, providing authoritative guidance for the vehicle. This experience must be robust. We hypothesize that learning multiple different sequences of the same environment and motion direction can reinforce this experience and enhance its robustness, leading to more reasonable motion plans. We propose a visual localization system based on feature denoising. Event-based camera pose regression architectures always contain characteristic noise when reconstructing environmental features, which can affect localization performance after feature dimensionality reduction. To address this issue, we propose a Kalman-like state smoothing module that enables the transformed recurrent units to perform feature smoothing in parallel with environmental feature reconstruction. The proposed feature denoising-based localization network first trains a state prediction network and a depth sensor network. All deep modules are then trained with joint state regression constraints, which reduces the training time of typical event-based localization architectures. Backpropagation is used to perform recursive optimization of all transformed recurrent units. Experimental results on the Cambridge and 7-scenes datasets demonstrate that our approach outperforms existing end-to-end architectures. The main work of this paper is as follows: 1) We propose a feature denoising-based localization network to address the feature noise interference problem of event-based deep localization networks; 2) We improve the loss function by embedding path prediction into the relocalization loss function to form a constraint. We find that deep neural networks with path memory capabilities can better perform the relocalization task; 3) We use Kalman filtering to experiment with fusion reasoning between consecutive frames, recording all historical pose data to protect against attacks from randomly inserted consecutive frames. We evaluate the proposed visual reasoning network on the Cambridge and 7-scenes datasets. Compared with existing methods, our method is more likely to learn motion patterns with complete motion paths. Even if the current motion path is different from the original one, the future pose can be inferred based on the previous motion patterns of the scene.

[0044] Based on the above, Figure 1 、 Figure 2 As shown, this embodiment provides a visual positioning system based on feature noise reduction, which includes:

[0045] Feature extraction network: extract features from the input image to obtain image features;

[0046] Visual localization network based on feature denoising: It performs visual localization based on image features and outputs a pose vector sequence, which includes the current estimated pose and several future estimated poses.

[0047] The visual localization network based on feature denoising includes:

[0048] Deep state module: used for one-step prediction of pose information;

[0049] Depth sensing module: used for two-step prediction of pose information.

[0050] Each pose in the pose vector sequence consists of a 3D camera position x and an orientation represented by a quaternion q, p = [x, q]

[0051] The pose p is defined relative to an arbitrary global reference frame. We choose quaternions as our orientation representation because any 4D value can be easily mapped to a legal rotation by normalizing it to unit length. This is a simpler process than the orthogonalization required for rotation matrices.

[0052] The depth state module and depth sensing module are both neural networks based on GoogLeNet. The last fully connected layer of the GoogLeNet neural network is expanded from 7 dimensions to 28 dimensions to predict the current posture and the next three postures.

[0053] The joint loss constraint for motion reasoning uses the Adam optimization algorithm and the following objective loss function to train the network on Euclidean loss. The objective loss function for visual localization network training based on feature denoising is:

[0054]

[0055] Among them, loss(I) is the loss of the visual positioning network I based on feature denoising, x is the 3D camera position label value, q is the direction label value, is the predicted value of the 3D camera position, is the direction prediction value, β is the scale factor, n is the total number of images used for training, i represents the i-th image used for training, α i denotes the weight corresponding to the i-th image. The set of rotations exists on the unit sphere of quaternion space, and the Euclidean loss function does not keep q on the unit sphere. However, we found that during training, q becomes close enough to q that the distinction between spherical distance and Euclidean distance becomes insignificant. For simplicity, and to avoid unnecessary constraints that hinder optimization, we choose to omit the spherical constraint.

[0056] In the loss function, a balance β must be struck between the orientation and translation losses. They are highly coupled because they are regressed from the same model weights. The optimal β is given by the ratio of the expected errors in position and orientation at the end of training, not at the beginning. Therefore, the scaling factor β is fine-tuned via grid search. For indoor scenes, the scaling factor β is ranged from 120 to 750. For outdoor scenes, the scaling factor β is ranged from 250 to 2000.

[0057] 1. Deep Learning Models for Visuomotor Reasoning

[0058] Here we describe a visual reasoning network that we train to estimate the future motion of a camera directly from monocular images. The network outputs a sequence of pose vectors consisting of the estimated current pose and several estimated future poses, where each pose consists of a 3D camera position x and an orientation represented by a quaternion q.

[0059] The orientation is defined relative to an arbitrary global reference system. We choose quaternions as our orientation representation because any quaternion value can be easily mapped to a valid rotation by normalizing it to unit length. This simplifies the orthogonalization required for the rotation matrix.

[0060] Figure 3 Theoretical model of the overall iteration of the visual localization network. Through backpropagation through the recurrent units, the network probability distribution gets closer and closer to the true distribution (black, dashed line). I is the grayscale value corresponding to each pixel location (x; y; z). Below the horizontal line is the sampling area of ​​I, which is uniform in this case. The upward arrow shows how the mapping x = G(I) imposes a non-uniform distribution on the transformed samples. G shrinks in high-density areas and expands in low-density areas. After several rounds of iterations, the predicted value of visual localization reaches the optimal value, at which point G(I) is approximately equal to Px.

[0061] A visual positioning method based on feature denoising is provided, the method being based on a system and comprising:

[0062] Input the image into the feature extraction network and perform feature extraction on the input image to obtain image features;

[0063] Visual positioning is performed based on image features, and a pose vector sequence is output. The pose vector sequence includes the current estimated pose and several future estimated poses.

[0064] The images are acquired using a monocular camera.

[0065] As shown in the accompanying drawings, this embodiment provides a visual positioning system based on feature denoising. The visual reasoning network uses a convolutional network for real-time six-degree-of-freedom camera relocalization as the basis of the deep architecture of the motion reasoning network, as follows:

[0066] 1. Build a positioning network based on feature noise reduction to solve the feature noise interference problem of event-based deep networks: The specific network connection method is as follows: Figure 2 As shown in Figure 1, the network consists of three parts: feature conversion module, synchronous feature smoothing module and deep differentiation module. The synchronous feature smoothing module contains two deep neural networks: deep state module and deep sensing module.

[0067] 2. First, use the VGG dataset to pre-train the feature extractor in the feature conversion module 300 times. This step is called texture feature transfer.

[0068] 3. Input the image sequence into the pre-trained feature extractor, obtain the extracted features f(x) according to the implicit mapping relationship of the deep neural network, arrange them in chronological order, reorganize them into a 64×64 feature matrix, and save it in the state extractor as the input of the synchronous feature smoothing module (gray box part).

[0069] 4. The synchronous feature smoothing module is iteratively trained 800 times. The depth state module is first randomly initialized. The output of the depth state module is used as the input of the depth sensing module for a round of correction. The output is a parameter matrix consisting of the Kalman pose and Kalman parameters at all levels, including parameters such as the P matrix and the F matrix. The parameter values ​​are learned through the flattened nonlinear activation unit.

[0070] 5. Overall recursive optimization, such as Figure 1 After two stages of training, the deep differential module is substituted into the training with the first two networks, and the output is the pose of the current image. Specifically, the parameter matrix obtained in the fourth step is used to initialize a cyclic prediction system, making full use of the latest measurement information y k+1 y k+1 and w k+1 Make reasonable corrections. Let c represent the current time block, f represent the most recent time block, and l represent the potential feature. o represents the output. N v represents the number of features v, N x represents the number of states x. Nu is the number of inputs u in a block, N l is the number of latent features l, N lh Indicates the number of hidden units of TRM. As the current time block, output weight By weight The output of the network is the pose of the current image.

[0071] 6. Dataset Collection and Evaluation

[0072] The method is evaluated on the Cambridge dataset and the 7-scenes dataset to verify the pose inference network's performance in outdoor and indoor environments, respectively.

[0073] 1) Validation on the Cambridge Dataset. Images were collected from five different outdoor scenes around the University of Cambridge. These images were captured with a smartphone at a resolution of 1920×1080 pixels while walking along different trajectories, and pose labels were computed using the SfM method. The dataset contains a large amount of clutter caused by pedestrians, cyclists, and moving vehicles, making urban relocalization challenging. Since the dataset consists of several sequences captured by researchers using handheld devices from different directions of travel, to ensure directional consistency in motion path prediction, we processed the Cambridge dataset by setting the shooting direction of the first sequence to the forward direction and reversing the order of the subsequent reverse sequences. The data from all sequences were then aggregated to form a new dataset file. We used a Python file to copy the last three poses to the same row as the current pose. This process was repeated for each row to form a new training set file. For each new training set file, each row includes the current image path, the pose corresponding to the current image, and the poses corresponding to the next three images. Table 1 shows the control accuracy comparison of the proposed network and other end-to-end pose regressors on the Cambridge dataset.

[0074] Table 1 Comparison of control accuracy of the proposed network and other end-to-end pose regressors on the Cambridge dataset

[0075]

[0076] 2) Validation on the 7-Scenes dataset. This dataset consists of RGB-D images collected from seven different scenes in an indoor office environment. The images were collected using a handheld Kinect RGB-D camera, and the ground truth pose was extracted using KinectFusion. The images are captured at a resolution of 640×480 pixels, and each scene contains multiple sequences recorded in a single room. Each sequence is recorded with different camera motions in the presence of motion blur, perceptual aliasing, and textureless features in the room, making it a popular dataset for relocalization and tracking. Table 2 compares the control accuracy of the proposed network with other end-to-end pose regressors on the 7-Scenes dataset.

[0077] Table 2 Comparison of control accuracy of the proposed network and other end-to-end pose regressors on the 7-secenes dataset

[0078]

[0079]

[0080] The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments can be implemented in various other ways, and various omissions, replacements, and changes can be made without departing from the technical concept of the present invention.

Claims

1. A visual positioning system based on feature denoising, characterized in that: The system is used for camera image positioning and includes: Feature extraction network: extract features from the input image to obtain image features; Feature-based denoising visual localization network: performs visual localization based on image features and outputs a pose vector sequence, which includes the current estimated pose and several future estimated poses. The visual positioning network based on feature denoising includes: Deep state module: used for one-step prediction of pose information; Depth sensing module: used for two-step prediction of pose information; The depth state module and depth sensing module are respectively based on the GoogLeNet neural network. The last fully connected layer of the GoogLeNet neural network is expanded from 7 dimensions to 28 dimensions to predict the current posture and the next three postures. The one-step prediction process specifically includes: the depth state module inputs a feature matrix obtained by reorganization based on the image features, outputs a posture before correction, and the posture before correction is used as an input of the depth sensing module for a round of correction; The two-step prediction process specifically includes: the depth sensing module inputs the posture before correction, and outputs a parameter matrix consisting of the Kalman posture and Kalman parameters at all levels; Each pose in the pose vector sequence consists of a 3D camera position x and a direction represented by a quaternion q.

2. A visual positioning system based on feature noise reduction according to claim 1, characterized in that: The objective loss function of the visual positioning network training based on feature denoising is: Among them, loss(I) is the loss of the visual positioning network I based on feature denoising, x is the 3D camera position label value, q is the direction label value, is the predicted value of the 3D camera position, is the direction prediction value, β is the scale factor, n is the total number of images used for training, i represents the i-th image used for training, α i Represents the weight corresponding to the i-th image.

3. A visual positioning system based on feature noise reduction according to claim 2, characterized in that: The scaling factor β is fine-tuned via grid search.

4. A visual positioning system based on feature noise reduction according to claim 3, characterized in that: In indoor scenes, the scaling factor β ranges from 120 to 750.

5. The visual positioning system based on feature noise reduction according to claim 3, characterized in that: In outdoor scenes, the scaling factor β ranges from 250 to 2000.

6. A visual positioning method based on feature denoising, characterized in that: The method is based on the system according to any one of claims 1 to 5, comprising: Input the image into the feature extraction network and perform feature extraction on the input image to obtain image features; Visual positioning is performed based on image features, and a pose vector sequence is output. The pose vector sequence includes the current estimated pose and several future estimated poses.

7. The visual positioning method based on feature denoising according to claim 6, characterized in that: The pictures are obtained using a monocular camera.

Images