A visual positioning method and system for a smart city scenario

Abstract

The application relates to a visual positioning method and system for a smart city scene, which comprises the following steps: S1, a mobile intelligent agent acquires an RGB image under a current scene through a monocular camera and carries out pretreatment; S2, an image sequence obtained through pretreatment is respectively input into a PWC-Net and a Monodepth2 network, and optical flow information and depth information are extracted; S3, the optical flow information is input into a GNN to obtain refined corresponding points; S4, camera poses are solved based on the corresponding points and a camera intrinsic parameter matrix; S5, the depth information and the camera poses are used to synthesize optical flow and an image view corresponding to the input image sequence; S6, a comprehensive loss function is constructed, and a monocular camera pose estimation model constructed in steps S2-S5 is trained; and S7, the trained monocular camera pose estimation model is used for camera pose estimation, and positioning of the mobile intelligent agent in an environment is completed. Compared with the prior art, the application improves the camera pose estimation capability by introducing a GNN to refine the optical flow.

Description

Technical Field

[0001] This invention relates to the field of visual positioning technology for mobile intelligent agents, and in particular to a visual positioning method and system for smart city scenarios. Background Technology

[0002] With the rapid development of deep learning and computer vision technologies, deep learning has demonstrated powerful capabilities in both computer vision and robotics. Compared to traditional hand-designed features, learning-based methods can automatically extract effective features from input data, exhibiting better adaptability to the environment and being better suited for mobile intelligent agents. Enabling robots to perform various tasks has always been a hot topic and a challenge in scientific research, especially for mobile robots. A prerequisite for performing other tasks is determining their own position in the environment. The accuracy of localization determines whether the robot can navigate safely. Localization is a key component of SLAM (Simultaneous Localization and Awareness Management) technology, enabling mobile robots to achieve self-localization in unknown environments based on data acquired by their onboard sensors.

[0003] Based on whether neural network training requires ground truth values, deep learning-based localization can be divided into self-supervised learning methods and unsupervised learning methods. Self-supervised learning methods do not require ground truth values. Supervised learning methods require a large amount of ground truth data, which is difficult and costly to obtain in the real world, limiting the performance and generalization ability of supervised learning methods. Unsupervised learning methods, on the other hand, do not require ground truth values. Based on camera-based visual localization, data can be easily obtained at a low cost. Unsupervised learning methods can estimate camera pose using unlabeled video sequence data. Current researchers tend to favor unsupervised or self-supervised methods, and unsupervised learning is currently a hot research topic.

[0004] Depending on the sensor used, vision-based methods mainly include monocular, binocular, and RGBD cameras. Binocular cameras suffer from high computational costs and localization failures in textureless environments; RGBD cameras are expensive and difficult to deploy; while monocular cameras are inexpensive and easy to deploy, but lose depth information.

[0005] Existing mainstream unsupervised monocular camera pose estimation methods use a PoseNet network to obtain inter-frame poses. The accuracy of the PoseNet network's predictions directly affects the performance of the entire system, limiting the performance improvement of unsupervised learning networks. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a visual positioning method and system for smart city scenarios. By introducing GNN to refine optical flow, the ability of camera pose estimation is improved.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] According to a first aspect of the present invention, a visual positioning method for smart city scenarios is provided, the method comprising the following steps:

[0009] Step S1: The mobile intelligent agent acquires RGB images of the current scene through a monocular camera and performs preprocessing to obtain an image sequence;

[0010] Step S2: Input the preprocessed image sequence into the optical flow learning network PWC-Net and the depth estimation self-supervised Monodepth2 network respectively to extract optical flow information and depth information;

[0011] Step S3: Input the optical flow information into the graph neural network (GNN) to obtain refined correspondence points;

[0012] Step S4: Solve for the camera pose based on the corresponding points and the camera intrinsic parameter matrix;

[0013] Step S5: Using the depth information obtained in step S2 and the camera pose information obtained in step S4, we obtain the synthetic optical flow and the synthetic view corresponding to the input image sequence.

[0014] Step S6: Construct a comprehensive loss function that includes optical flow loss, photometric consistency loss, structural similarity loss and depth smoothing loss, and train the monocular camera pose estimation model constructed in steps S2 to S5.

[0015] Step S7: Use the trained monocular camera pose estimation model to perform camera pose estimation and complete the localization of the mobile intelligent agent in the environment.

[0016] Preferably, the current scene in step S1 includes an indoor scene and an outdoor scene.

[0017] Preferably, the preprocessing in step S1 specifically involves: normalizing the RGB image obtained in the current scene and converting it into the required data format.

[0018] Preferably, the optical flow information in step S2 includes forward optical flow information and backward optical flow information.

[0019] Preferably, step S4 specifically involves: based on the corresponding points and the camera intrinsic parameter matrix, using epipolar geometry, the eight-point method, and singular value decomposition to obtain the fundamental matrix or essential matrix, and solving for the camera pose T = [R, t]; where R is the rotation matrix and t is the translation vector;

[0020] The camera pose includes the relative pose between the next frame and the current frame, as well as the relative pose between the current frame and the next frame.

[0021] Preferably, step S5 specifically comprises:

[0022] Image sequences of the current and next frames obtained based on depth estimation using the self-supervised Monodepth2 network. t ,I t+1 Depth information D t D t+1 And the relative pose T between the current frame and the next frame obtained in step S4. t→t+1 The relative pose T between the next frame and the current frame t+1→t The current frame image sequence I is synthesized respectively. t pixel p in t In the next frame image sequence I t+1 The corresponding point and the next frame image sequence I t+1 pixel p in t+1 In the current frame image sequence I t The corresponding point, specifically expressed as:

[0023]

[0024]

[0025] Then, the expressions for the synthesized forward optical flow and backward optical flow are respectively:

[0026]

[0027]

[0028] In the formula, K represents the camera intrinsic parameter matrix.

[0029] Preferably, the comprehensive loss function in step S6 is the superposition of optical flow loss, photometric consistency loss, structural similarity loss, and depth smoothing loss.

[0030] Preferably, the optical flow loss in step S6 includes the absolute value of the difference between the forward optical flow and the synthesized forward optical flow, and the absolute value of the difference between the backward optical flow and the synthesized backward optical flow.

[0031] According to a second aspect of the present invention, a visual positioning system for smart city scenarios is provided, employing any one of the methods described above, the system comprising:

[0032] A mobile intelligent agent equipped with a monocular camera is used to acquire RGB images of the current scene;

[0033] The pose estimation module is used to preprocess the acquired RGB image and perform pose estimation using an end-to-end monocular vision localization model based on self-supervised learning.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] 1) Based on deep learning, this invention proposes a self-supervised learning end-to-end monocular vision localization method, which breaks away from the limitations of the mainstream PoseNet network on the entire network framework, introduces a novel graph neural network to refine optical flow, and improves the camera pose estimation capability.

[0036] 2) The visual positioning method proposed in this invention effectively solves the visual positioning problem of mobile intelligent agents in different scenarios such as outdoor and indoor environments;

[0037] 3) This invention proposes a monocular vision localization network framework that uses the same framework in both the training and testing phases without requiring different treatments. In particular, during the testing phase, only the optical flow branch of the entire framework is needed to complete the pose estimation of the mobile agent. Attached Figure Description

[0038] Figure 1 This is a flowchart of the visual positioning method of the present invention;

[0039] Figure 2 This is a flowchart of the pose estimation process of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] Example

[0042] Combined with appendix Figure 1 and 2This paper presents a method embodiment of the present invention, a visual positioning method for smart city scenarios. This method achieves end-to-end monocular camera pose estimation in a self-supervised manner, without requiring the ground truth of the camera pose, only requiring the RGB color image sequence acquired by the monocular camera as the network input, and the accuracy of pose estimation is not limited by the PoseNet network; specifically, it includes the following steps:

[0043] Step S1: The mobile intelligent agent acquires RGB images of the current scene through a monocular camera and performs preprocessing to obtain an image sequence;

[0044] Step S2: Input the preprocessed image sequence into the optical flow learning network PWC-Net and the depth estimation self-supervised Monodepth2 network respectively to extract optical flow information and depth information, specifically:

[0045] From the same original adjacent image I t ,I t+1 Obtain forward optical flow and backward optical flow and the corresponding depth D t D t+1 .

[0046] Step S3: To obtain more robust optical flow correspondences, the forward and backward optical flows are input into a graph neural network (GNN) to obtain refined correspondences.

[0047] Step S4: Based on the corresponding points and the camera intrinsic parameter matrix, solve for the camera pose, specifically as follows:

[0048] After obtaining the feature correspondence points between adjacent images, the essential matrix and the fundamental matrix are obtained according to the following epipolar geometry relation and the eight-point method:

[0049]

[0050] E = t ∧ R

[0051] F = K -T EK -1

[0052] Among them, t ∧ Let R be the antisymmetric matrix for translation t, R be the rotation matrix, K be the camera intrinsic parameter matrix, and p1 and p2 be the images I and II, respectively. t ,I t+1 The pixels, where E represents the essential matrix and F represents the fundamental matrix;

[0053] Then, an eight-point method and an essential matrix are used to construct a system of linear equations. The coefficient matrix is ​​composed of the positions of the corresponding characteristic points. The elements of the essential matrix E are obtained by solving the equations constructed above. Then, the rotation matrix R and translation vector t are obtained by singular value decomposition (SVD), and the pose T = [R, t].

[0054] Step S5: Using the self-supervised Monodepth2 network for depth estimation, the depth information and the camera pose information obtained in step S4 are used to synthesize the optical flow and the synthesized view corresponding to the input image sequence. Specifically:

[0055] Input image I t ,I t+1 The predicted depth D is obtained through the Monodepth2 network. t D t+1 Simultaneously, using the pose T obtained in S4 t→t+1 T t+1→t p is synthesized using the following formula t in I t+1 The corresponding point in the view:

[0056]

[0057] Similarly:

[0058]

[0059] At this point, the synthesized forward and backward optical flows are calculated using the following formulas:

[0060]

[0061]

[0062] Step S6: Construct a comprehensive loss function that includes optical flow loss, photometric consistency loss, structural similarity loss and depth smoothing loss, and train the monocular camera pose estimation model constructed in steps S2 to S5.

[0063] The forward optical flow loss and the backward optical flow loss are respectively:

[0064]

[0065]

[0066] The photometric consistency loss and structural similarity loss (SSIM) are:

[0067]

[0068] Wherein, α and β are parameters used to balance the loss of photometric uniformity. In this embodiment, the parameters are set to α = 0.15 and β = 0.85, respectively.

[0069] The depth smoothing loss is:

[0070]

[0071] in, It represents the first derivative.

[0072] The loss function of the entire network is the sum of the above loss functions, which is used to constrain the entire network.

[0073] Step S7: Using the trained monocular camera pose estimation model, input the color video sequence acquired by the monocular camera, pass through the optical flow network and graph neural network to obtain refined feature correspondence points, and finally use the method in S4 to calculate the camera pose to complete the localization.

[0074] During the training phase, the mobile agent acquires RGB images of the current scene through a monocular camera. To facilitate network learning, the acquired images are normalized and preprocessed, and converted into the tensor data format required by the network. At the same time, pre-trained models of PWC-Net and Monodepth2 networks can be loaded to accelerate network convergence, or the network can be retrained. This invention uses pre-trained models to accelerate the convergence speed of the entire network.

[0075] During the testing phase, such as Figure 2 As shown, when a mobile agent is in an unknown environment, it captures environmental images through a camera, normalizes the input images, and loads a trained model to estimate the camera's pose, thus completing the mobile agent's localization in the environment.

[0076] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visual positioning method for a smart city scenario, characterized in that, The method comprises the following steps: Step S1, the mobile agent obtains an RGB image under a current scene through a monocular camera and performs preprocessing to obtain an image sequence; Step S2, the image sequence obtained through preprocessing is respectively input into a light flow learning network PWC-Net and a depth estimation self-supervised Monodepth2 network to extract light flow information and depth information; Step S3, the light flow information is input into a graph neural network GNN to obtain refined corresponding points; Step S4, camera poses are solved based on the corresponding points and an intrinsic matrix of the camera; Step S5, the depth information obtained in step S2 and the camera pose information solved in step S4 are used to synthesize light flows and image views corresponding to the input image sequence; Image sequences of the current and next frames are obtained based on depth estimation using the Monodepth2 self-supervised network. depth information And the relative pose between the current frame and the next frame obtained in step S4. The relative pose between the next frame and the current frame Synthesize the current frame image sequence respectively pixels in In the next frame image sequence Corresponding points and the next frame image sequence pixels in In the current frame image sequence The corresponding point, specifically expressed as: Then, the expressions of the synthesized forward light flow and backward light flow are as follows: In the formula, denotes a camera intrinsic matrix, is a translation vector; Step S6, a comprehensive loss function comprising a light flow loss, a photometric consistency loss, a structural similarity loss and a depth smoothness loss is constructed, and a monocular camera pose estimation model constructed in steps S2-S5 is trained; Step S7, the trained monocular camera pose estimation model is used for camera pose estimation to complete positioning of the mobile agent in the environment. 2.The visual positioning method for smart city scenarios according to claim 1, wherein, The current scene in step S1 comprises an indoor scene and an outdoor scene. 3.The visual positioning method for smart city scenarios according to claim 1, wherein, The preprocessing in step S1 specifically comprises normalizing the obtained RGB image under the current scene and converting it into a required data format. 4.The visual positioning method for smart city scenarios according to claim 1, wherein, The light flow information in step S2 comprises forward light flow information and backward light flow information.

5. The visual positioning method for smart city scenarios according to claim 1, characterized in that, The step S4 is specifically: based on the corresponding points and the camera intrinsic parameter matrix, using epipolar geometry, eight-point method and singular value decomposition to obtain a fundamental matrix or an essential matrix, and solving the camera pose ; wherein, is a rotation matrix.

6. The visual positioning method for smart city scenarios according to claim 5, characterized in that, The camera poses comprise relative poses between a next frame and a current frame and relative poses between the current frame and the next frame.

7. The visual positioning method for smart city scenarios according to claim 1, characterized in that, The comprehensive loss function in step S6 is a superposition of the light flow loss, the photometric consistency loss, the structural similarity loss and the depth smoothness loss.

8. The visual positioning method for smart city scenarios according to claim 7, characterized in that, The light flow loss in step S6 comprises an absolute value of a difference between the forward light flow and the synthesized forward light flow and an absolute value of a difference between the backward light flow and the synthesized backward light flow. 9.A visual positioning system for smart city scenarios, characterized in that, The system comprises the method according to any one of claims 1-8. The mobile agent equipped with a monocular camera is used to obtain an RGB image under a current scene; The pose estimation module is used to preprocess the obtained RGB image and perform pose estimation by using an end-to-end monocular visual positioning model based on self-supervised learning.

Images