A visual positioning method and system based on multi-feature fusion

Abstract

The application discloses a kind of visual positioning method and system based on multi-feature fusion.The method: scene photo is collected, three-dimensional reconstruction, data annotation and picture rendering are carried out, and annotated data set is obtained;Using annotated data set carries out semantic segmentation and normal vector estimation;The semantic segmentation result of picture, normal vector estimation result is carried out multi-feature extraction and fusion, and the fusion global vector of picture is obtained;The candidate graph of query graph is obtained by searching;The confidence of query graph is estimated;Local feature matching is carried out to query graph and each candidate graph, and a plurality of 2D-3D matching pairs are obtained;2D-3D matching pair is filtered using confidence, and the matching pair with numerical value lower than confidence threshold is removed;RANSAC PnP calculation is carried out to the remaining 2D-3D matching pair, and camera pose result is obtained.The application enhances the robustness of visual positioning method, so that it can still have accurate and stable pose calculation result when facing the change of external environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of autonomous driving, robots, augmented reality, mixed reality, etc., and in particular to a visual positioning method and system based on multi-feature fusion. BACKGROUND

[0002] Positioning refers to determining the position and orientation information of a target object in space through various technologies and methods, and it is a very important and basic task that plays a key role in many fields such as simultaneous localization and mapping (SLAM), digital twinning, autonomous driving, etc. At present, there are various methods to deal with the positioning task. In satellite positioning methods, a signal receiver receives signals transmitted by multiple satellites, obtains the distance between itself and each satellite according to the signal propagation time, and determines the specific position using the principle of triangular positioning. In Bluetooth positioning methods, a target device periodically broadcasts a Bluetooth signal, a receiver receives and measures the Bluetooth signal strength, and determines the position of the target device according to a pre-established signal strength database. In radio frequency identification positioning methods, a radio frequency identification reader sends a radio frequency signal, a target object with a specific tag responds to the signal, and the reader calculates the position of the target object using a corresponding algorithm (such as a position fingerprint positioning algorithm) after receiving the response signal.

[0003] Although the above methods have been applied in many scenarios, there are still some problems, such as satellite positioning methods being susceptible to interference in indoor environments and having unstable positioning performance; Bluetooth positioning methods having low positioning accuracy and small coverage range due to signal attenuation, multipath effects, etc.; and radio frequency identification positioning methods being susceptible to environmental interference and having high system deployment costs. At the same time, there are many virtual scene-based applications in the fields of digital twinning, augmented reality, etc., which cannot be equipped with physical hardware and do not have the conditions to use the above methods. In contrast, visual positioning technology can calculate the accurate pose of the current target based on the picture information collected by the camera, and has the advantages of low cost and high accuracy. Therefore, visual positioning technology research has a wide range of application scenarios and important research significance.

[0004] The current domestic and foreign methods for visual positioning technology mainly include four kinds, which are camera pose calculation method based on two-dimensional image retrieval, camera pose calculation method based on three-dimensional scene, hierarchical camera pose calculation method and camera pose calculation method based on semantic information. Among them, the camera pose calculation method based on two-dimensional image shows good generalization, but the accuracy of the pose calculation result is not ideal due to the lack of introduction of three-dimensional geometric information; the camera pose calculation method based on three-dimensional scene mostly contains the process of using multi-view geometry technology to solve the accurate pose, so the result accuracy is usually high, but in order to realize the geometric solving, the 2D-3D matching between the two-dimensional feature points of the image and the three-dimensional point cloud of the scene needs to be carried out in advance, which is time-consuming and prone to a large number of false matches, so this method has the disadvantages of high calculation cost and poor robustness, especially when facing large scenes and large environmental differences; the hierarchical camera pose calculation method effectively combines the first two methods, and can have robustness and accuracy, and usually shows better in the calculation result, but it is still difficult to get rid of the disadvantage of being easily disturbed when facing seasonal, weather and other environmental changes; the camera pose calculation method based on semantic information integrates semantic information into the camera pose calculation process, which to some extent alleviates the performance instability problem of the conventional camera pose calculation method when facing environmental changes, but this method appeared late and is not mature, and has many shortcomings, such as large difference in positioning results caused by unstable semantic segmentation results, difficulty in obtaining semantic annotation truth value required for semantic segmentation training and the like. SUMMARY

[0005] In order to solve the above technical problems, the present application provides a visual positioning method and system based on multi-feature fusion. The technical solution of the present application is: a visual positioning method based on multi-feature fusion, which comprises two parts of image retrieval based on multi-feature fusion and pose calculation based on confidence estimation, and the image retrieval based on multi-feature fusion comprises the following steps:

[0006] S1, pre-acquiring an RGB image in a scene by using a UAV, constructing a three-dimensional model of the scene by using a three-dimensional reconstruction technology, performing semantic annotation and normal vector calculation on the three-dimensional model, and generating a labeled image dataset by using snapshot rendering;

[0007] S2, training a semantic segmentation network, a normal vector estimation network and a multi-feature extraction and fusion network by using the labeled dataset, and generating a corresponding fusion feature database;

[0008] S3, inputting a query image into the semantic segmentation network, the normal vector estimation network and the multi-feature extraction and fusion network for calculation to obtain fusion features of the query image;

[0009] S4, Euclidean distance calculation is performed on each feature in the fusion feature database and the fusion features of the query graph, the distance results are sorted by size, and the picture corresponding to the vector with the smallest distance is selected as the candidate picture;

[0010] The pose calculation step based on confidence estimation is:

[0011] S5, the labeled image dataset is used to train the confidence estimation network, and the network parameters are stored after training;

[0012] S6, confidence estimation is performed on the query graph to obtain pixel-level confidence values in the picture as a quantitative reference for judging the quality of feature points at the pixel position;

[0013] S7, two-dimensional feature matching is performed on the query graph and its corresponding candidate picture with pose, three-dimensional coordinate mapping is performed on the feature points of the candidate picture according to the depth information, and a 2D-3D matching pair between the query graph and the three-dimensional scene is established;

[0014] S8, the 2D-3D matching pair is filtered using the confidence map, and the 2D-3D matching pair with a confidence value lower than the confidence threshold is removed;

[0015] S9, the remaining 2D-3D matching pairs are subjected to random sample consensus algorithm RANSACPnP calculation, and finally the pose calculation result of the camera is obtained to realize visual positioning.

[0016] The step S2 specifically comprises the following steps:

[0017] S2.1, the DeepLabv3+ network is trained according to the semantic label annotation in the dataset, and the network parameters after training are stored;

[0018] S2.2, the DIW network is trained according to the normal vector value annotation in the dataset, and the network parameters after training are stored;

[0019] S2.3, the data set is divided into 1:9 query graphs and candidate graphs, and according to the picture pose data in the data set, the candidate graphs with a pose difference within a threshold value from the query graph are selected from the candidate graph set as a positive set of the query graph, and the remaining candidate graphs are stored as a negative set of the query graph;

[0020] S2.4, the generated query graph and its corresponding positive and negative sets are used to train the multi-feature extraction and fusion network, and the training process uses a three-tuple loss, and when selecting positive and negative samples, the corresponding global vector is calculated by the network in the current training state, then the picture corresponding to the vector with the maximum Euclidean distance from the query graph vector is selected as the positive sample, and a certain number of pictures are randomly selected from the negative set as negative samples.

[0021] S2.5, after the network training is completed, input each picture in the rendering data set into DeepLabv3+, DIW and the multi-feature extraction and fusion network, obtain the fusion features of each picture, and store all vectors as a fusion feature database.

[0022] Further, the step S3 includes the following steps:

[0023] S3.1, input the query image into DeepLabv3+ and DIW network respectively for semantic segmentation and normal vector estimation, and obtain the semantic label prediction result and the normal vector numerical estimation result of the query image;

[0024] S3.2, input the RGB image, the semantic segmentation label prediction result and the normal vector value of the query together into the multi-feature extraction and fusion network, and calculate the global fusion vector of the query image.

[0025] Further, the step S5 includes the following steps:

[0026] S5.1, according to the pose information of the labeled data set, calculate and screen out the candidate image set R corresponding to each training image q as a query image and having a pose difference within a threshold value;

[0027] S5.2, for each pixel point x in the query image q , calculate the re-projection difference value r(x i , x q ) between it and the corresponding pixel x i in the candidate image i;

[0028] S5.3, calculate the standard deviation S(x q , i) by counting the re-projection difference of all pixel points in the neighborhood of x q ;

[0029] S5.4, repeat steps S5.2-S5.3 for each candidate image in R to obtain the corresponding standard deviation S(x q , i) under each candidate image, then apply an activation function e -s to it and take the average value as the confidence true value C(x q ) of x q ;

[0030] S5.5, use the confidence true value to train the confidence estimation network, and store the network parameters.

[0031] Further, the step S7 includes the following steps:

[0032] S7.1, SIFT feature matching is performed on the query graph and each retrieved candidate graph to obtain a series of 2D-2D matching pairs between the query graph and the candidate graph;

[0033] S7.2, the depth information of each feature point on the query graph matching of each candidate graph is obtained by reading the corresponding depth map of each candidate graph, and the coordinate value of the feature point in the three-dimensional space is calculated according to the depth information and the camera pose annotation of the candidate graph, so as to realize the 2D-3D matching pair between the candidate graph and the three-dimensional model;

[0034] S7.3, the 2D-3D matching pair between the query graph and the three-dimensional model is established according to the 2D-2D matching pair of S7.1 and the 2D-3D matching pair of S7.2.

[0035] Further, the step S8 comprises the following steps:

[0036] S8.1, for each 2D-3D matching pair, the corresponding confidence value is found according to the pixel coordinates of the 2D point in the query graph, and the confidence value is obtained in S6;

[0037] S8.2, the confidence value on the 2D point pixel is compared with the confidence threshold value, if the 2D point confidence value is lower than the threshold value, it is discarded, otherwise it is reserved;

[0038] S8.3, the RANSACPnP algorithm is applied to the remaining 2D-3D pair to obtain the camera pose, and the visual positioning is realized.

[0039] Further, it further comprises visual display of data, specifically comprising:

[0040] The three-dimensional model of the scene is loaded, rendered and displayed in the window, supporting the user to use the mouse and the scroll wheel to realize the three-dimensional interaction of the view angle; meanwhile, the query RGB graph input by the user is loaded, the semantic segmentation prediction result graph, the normal vector estimation result graph and the confidence estimation graph are displayed, the image retrieval result is displayed in the form of a list, the thumbnail and the file name of the candidate graph are contained in each item in the list, the results of the pose calculation are listed in the form of space x, y, z coordinates and yaw, pitch, roll data, and the query graph is projected in the three-dimensional scene in the calculated pose view angle.

[0041] Another aspect of the application is a multi-feature fusion visual positioning system, the system comprises the following modules:

[0042] Data processing module: using three-dimensional reconstruction technology to construct a three-dimensional model of the scene, perform semantic annotation and normal vector calculation, and render a labeled picture database from the RGB picture of the scene pre-shot by the camera; input the pictures in the database into an image segmentation network, a normal vector estimation network, and a multi-feature extraction and fusion network to extract the fusion features of each picture in the database and construct a fusion feature database;

[0043] Image retrieval module: input the query RGB picture input into the system into a semantic segmentation network and a normal vector estimation network to calculate the semantic information and normal vector information of the image; input the RGB, semantic, and normal vector three kinds of data into a multi-feature extraction and fusion network to obtain the fusion features of the image; calculate the Euclidean distance between the fusion features of the query image and each feature vector in the fusion feature database, sort the distances, and select a plurality of images with the smallest relative distance as the candidate image set result retrieved by the query image;

[0044] Pose calculation module: according to the semantic information and normal vector information calculated in the image retrieval module, joint the original RGB image to perform confidence estimation, obtain the pixel-level confidence estimation result of the query image, sequentially perform local feature matching on each image in the query image and the candidate image set, and use the depth and pose information of the candidate image to map the 2D feature points in the candidate image to 3D coordinates, thereby further establishing the matching relationship between the 2D points in the query image and the 3D points in the three-dimensional model; remove the matching pairs whose confidence is lower than the threshold value on the pixel position of the 2D point in the 2D-3D, and perform RANSAC PnP calculation on the remaining matching pairs to obtain the pose calculation result of the camera and complete the visual positioning.

[0045] Data display module: load the three-dimensional model of the scene, render and display in the window, support three-dimensional interaction of the view angle using the mouse and scroll wheel, load the query RGB image input by the user, display the semantic segmentation prediction result image, normal vector estimation result image, and confidence estimation image, display the image retrieval result in the form of a list, each item in the list contains the thumbnail and file name of the candidate image, list the results of pose calculation in the form of space x, y, z coordinates and yaw, pitch, roll data, and project the query image in the three-dimensional scene at the calculated pose view angle.

[0046] Compared with the prior art, the application has the advantages that:

[0047] 1、The method of the present application extracts and fuses multiple features in the RGB, semantic segmentation and normal vector dimensions. Compared with other methods that only use a single feature, the multi-feature fusion can more comprehensively describe the image content, improving the accuracy and robustness of image retrieval. By integrating multiple features, the method can more accurately match the query image with the images in the database, providing more accurate retrieval results.

[0048] 2、The present application introduces a confidence estimation network to estimate the confidence of the matching pair by quantifying the amount of useful information that may be generated in the pose calculation process, which can filter features during pose calculation, improving the accuracy and stability of pose calculation. With accurate confidence estimation, the method can exclude unreliable matching pairs, improve the accuracy of pose calculation, and has better robustness to cope with environmental changes and interference factors.

[0049] 3、The method of the present application realizes a camera pose calculation tool in a labeled three-dimensional scene, which can automatically calculate the pose of the user input picture and visualize the appropriate results, meeting the user's needs. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 Flow chart of visual positioning method based on multi-feature fusion;

[0051] Figure 2 Flow chart of image retrieval based on multi-feature fusion;

[0052] Figure 3 Flow chart of pose calculation based on confidence estimation;

[0053] Figure 4 Structure block diagram of visual positioning system based on multi-feature fusion. DETAILED DESCRIPTION

[0054] The present application will be further described in detail below with specific implementation steps and in conjunction with the accompanying drawings. The following examples are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application. Changes, modifications and advantages that can be thought of by those skilled in the art without departing from the spirit and scope of the present application are included in the present application.

[0055] Example 1

[0056] As shown in Figure 1 , the present application provides a flow chart of visual positioning method based on multi-feature fusion. The method comprises the following steps:

[0057] Step S1: As Figure 2As shown, the RGB image of the scene captured by the camera in advance is used to establish a three-dimensional model of the scene by using three-dimensional reconstruction technology, and the three-dimensional model is semantically labeled and the normal vector is calculated. The scene picture is generated by snapshot rendering technology, and the corresponding annotation file is generated and saved to obtain the annotated image dataset. The pictures in the annotated image dataset are input into the semantic segmentation network, the normal vector estimation network and the multi-feature extraction and fusion network to extract the fusion features of each picture to form a fusion feature database.

[0058] Step S2: input the query image into the semantic segmentation network and the normal vector estimation network to obtain the semantic information and the normal vector information of the image. The RGB, semantic and normal vector data of the query image are input into the multi-feature extraction and fusion network to obtain the fusion features of the image. The fusion feature database is sorted according to the similarity of the fusion features of the query image, and the specific calculation method is to calculate the Euclidean distance between the fusion features of the query image and each feature in the fusion feature database in turn, and sort the distances, and select the multiple images with the smallest distances as the image retrieval candidate image set result of the query image.

[0059] Step S3: based on the previously calculated semantic information and normal vector information, the confidence estimation is performed combined with the original RGB image to obtain the pixel-level confidence estimation result of the query image.

[0060] Step S4: as shown, Figure 3 The local feature matching is performed on each image in the query image and the candidate image set one by one, and the 2D feature points in the candidate image are mapped to 3D coordinates by using the depth and pose information of the candidate image, and the matching relationship between the 2D points in the query image and the 3D points in the three-dimensional model is established.

[0061] Step S5: the matching pairs with a confidence lower than a threshold value in the 2D-3D matching are removed, and the remaining matching pairs are subjected to random sample consensus algorithm RANSAC PnP calculation to obtain the pose calculation result of the camera, so as to realize the pose calculation of the subject by using the computer vision method and achieve the purpose of visual positioning.

[0062] Further, the three-dimensional model of the scene is loaded and rendered, and is displayed in the window, and the user can interact with the three-dimensional view by using the mouse and the scroll wheel; the query RGB image input by the user is loaded and the semantic segmentation prediction result image, the normal vector estimation result image and the confidence estimation image are displayed. The image retrieval result is presented in the form of a list, and each item contains the thumbnail and the file name of the candidate image. The result of the pose calculation is listed in the form of spatial coordinates and attitude data, and the pose view of the query image is projected into the three-dimensional scene to provide an intuitive viewing experience for the user.

[0063] The fusion feature database generation process specifically includes the following steps:

[0064] S2.1, train the DeepLabv3+ network according to the semantic label annotation in the data set, and store the trained network parameters;

[0065] S2.2, train the DIW network according to the normal vector numerical annotation in the data set, and store the trained network parameters;

[0066] S2.3, divide the data set into 1:9 query graphs and candidate graphs, according to the annotated image pose data in the data set, filter out the candidate graphs in the candidate graph set that have a pose difference within a threshold from each query graph, and store them as the positive set of the query graph, and the remaining candidate graphs as the negative set of the query graph;

[0067] S2.4, train the multi-feature extraction and fusion network using the generated query graph and its corresponding positive and negative sets, and use the triplet loss in the training process, and when selecting positive and negative samples, first calculate the corresponding global vector using the network in the current training state, then select the picture corresponding to the vector with the maximum Euclidean distance from the query graph vector in the positive set vector as the positive sample, and randomly select a certain number of pictures in the negative set as negative samples each time;

[0068] S2.5, after the network training is completed, input each picture in the rendering data set into DeepLabv3+, DIW and the multi-feature extraction and fusion network to obtain the global fusion vector of each picture, and store all the vectors as a fusion feature database.

[0069] Further, the generated fusion feature specifically includes the following steps:

[0070] S3.1, input the query graph into DeepLabv3+ and DIW network respectively for semantic segmentation and normal vector estimation to obtain the semantic label prediction result and normal vector numerical estimation result of the query graph;

[0071] S3.2, input the RGB graph, semantic segmentation label prediction result and normal vector value of the query into the multi-feature extraction and fusion network together to calculate the global fusion vector of the query graph.

[0072] Further, the pixel-level confidence estimation result of the query image includes the following steps:

[0073] S5.1, according to the pose information of the annotated data set, calculate and filter out the candidate graph set R corresponding to each training graph q as a query graph within a threshold of the pose difference;

[0074] S5.2, for each pixel x in the query graph q calculate the reprojection difference value r(x i of the corresponding pixel x in the candidate graph iq x i );

[0075] S5.3, regarding x q The reprojection differences of all pixels within the neighborhood are statistically analyzed, and the standard deviation S(x) is calculated. q ,i);

[0076] S5.4 Repeat steps S5.2 to S5.3 for each candidate graph in R to obtain the standard deviation S(x) for each candidate graph. q ,i), and then apply the activation function e to it. -s And take the mean as x q The true confidence level C(x) q );

[0077] S5.5. Train the confidence estimation network using the true confidence values ​​and store the network parameters.

[0078] Furthermore, the establishment of 2D-3D matching pairs between the query graph and the 3D scene specifically includes the following steps:

[0079] S7.1 Perform SIFT feature matching on the query graph and each retrieved candidate graph to obtain a series of 2D-2D matching pairs between the query graph and the candidate graph;

[0080] S7.2 Read the corresponding depth map for each candidate image, obtain the depth information of each candidate image feature point that matches the query image, calculate the coordinate value of the feature point in three-dimensional space based on the depth information and the camera pose annotation of the candidate image, and realize the 2D-3D matching pair between the candidate image and the three-dimensional model.

[0081] S7.3. Based on the 2D-2D matching pairs in S7.1 and the 2D-3D matching pairs in S7.2, establish 2D-3D matching pairs between the query graph and the 3D model.

[0082] Furthermore, the calculation of the camera pose specifically includes the following steps:

[0083] S8.1 For each 2D-3D matching pair, the corresponding confidence value is found based on the pixel coordinates of the 2D point in the query graph. The confidence value is obtained in S6.

[0084] S8.2 Compare the confidence value of the 2D point pixel with the confidence threshold. If the confidence value of the 2D point is lower than the threshold, discard it; otherwise, keep it.

[0085] S8.3. Apply the RANSAC PnP algorithm to solve the remaining 2D-3D pairs to obtain the camera pose and achieve visual positioning.

[0086] Further, the visual presentation of data is also included, specifically including:

[0087] The three-dimensional model of the scene is loaded, rendered and displayed in the window, supporting the user to use the mouse and the scroll wheel for three-dimensional interaction of the view angle; meanwhile, the query RGB image input by the user is loaded, the semantic segmentation prediction result image, the normal vector estimation result image and the confidence estimation image are displayed, the image retrieval result is displayed in the form of a list, each item in the list contains the thumbnail and the file name of the candidate image, the result of the pose calculation is listed in the form of spatial x, y, z coordinates and yaw, pitch, roll data, and the query image is projected in the three-dimensional scene at the calculated pose view angle.

[0088] Embodiment two

[0089] As shown in Figure 4 , the embodiment of the application provides a visual positioning system based on multi-feature fusion, including the following modules:

[0090] The data processing module: acquires the RGB picture in the scene photographed in advance by the camera, applies the three-dimensional reconstruction technology to construct the three-dimensional scene of the scene, loads the scene and renders it, realizes the display of the three-dimensional scene, and can interact with the user in the three-dimensional view angle; each picture in the database is input into the image segmentation network, the normal vector estimation network and the multi-feature extraction and fusion network to extract and fuse the features, and a fusion feature database is constructed.

[0091] The image retrieval module: receives the query RGB picture input by the system, and calculates the semantic information and the normal vector information of the image through the semantic segmentation network and the normal vector estimation network. Meanwhile, the RGB, semantic and normal vector three kinds of data are input into the multi-feature extraction and fusion network to obtain the fusion features of the image. The Euclidean distance is calculated between the fusion features of the query image and each feature vector in the fusion feature database, and the distance is sorted, and a plurality of images with relatively minimum distance are selected as the image retrieval candidate image set result of the query image.

[0092] The pose calculation module: according to the semantic information and the normal vector information calculated by the image retrieval module, and the original RGB image, the confidence is estimated. Each image in the query image and the candidate image set is matched with the local feature one by one, and the 3D coordinates of the 2D feature points in the candidate image are mapped by using the depth and the pose information of the candidate image, and the matching relationship between the 2D points in the query image and the 3D points in the three-dimensional model is established. Then, according to the confidence threshold, the matching pairs with low confidence in the 2D-3D matching are removed, the remaining matching pairs are reserved, and the pose of the camera is calculated by using RANSACPnP.

[0093] The data display module loads and renders the three-dimensional model of the scene and displays it in the window. The user can interact with the three-dimensional view through the mouse and scroll wheel. The user-input query RGB image is loaded and the semantic segmentation prediction result image, the normal vector estimation result image, and the confidence estimation image are displayed. The image retrieval results are presented in a list format, with each item containing a thumbnail of the candidate image and the file name. The pose calculation results are listed in the form of spatial coordinates and attitude data, and the pose view of the query image is projected onto the three-dimensional scene to provide an intuitive viewing experience for the user.

[0094] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for visual positioning based on multi-feature fusion, characterized in that, The method comprises two parts of multi-feature fusion based image retrieval and confidence estimation based pose calculation, and the multi-feature fusion based image retrieval step is: S1, pre-acquire an RGB image in a scene by using a UAV, construct a three-dimensional model of the scene by using a three-dimensional reconstruction technology, perform semantic labeling and normal vector calculation on the three-dimensional model, and generate a labeled image dataset by using snapshot rendering; S2, train a semantic segmentation network, a normal vector estimation network and a multi-feature extraction and fusion network by using the labeled image dataset, and generate a corresponding fusion feature database; S3, input a query image into the semantic segmentation network, the normal vector estimation network and the multi-feature extraction and fusion network to calculate the fusion feature of the query image; S4, calculate the Euclidean distance between each feature in the fusion feature database and the fusion feature of the query image, sort the distance results by size, and select the picture corresponding to the vector with the smallest distance as the candidate picture; The confidence estimation based pose calculation step is: S5, train a confidence estimation network by using the labeled image dataset, and store the network parameters after the training is completed; S6, estimate the confidence of the query image to obtain a pixel-level confidence value, which is used as a quantitative reference for judging the quality of the feature points at the pixel positions; S7, perform two-dimensional feature matching on the query image and the candidate picture with a pose corresponding thereto one by one, map the feature points of the candidate picture to three-dimensional coordinates according to the depth information of the candidate picture, and establish a 2D-3D matching pair between the query image and the three-dimensional scene according to the mapping; S8, filter the 2D-3D matching pairs by using the confidence map, and remove the 2D-3D matching pairs with a confidence value lower than a confidence threshold; S9, perform a random sample consensus algorithm (RANSAC PnP) on the remaining 2D-3D matching pairs to obtain the pose calculation result of the camera, and realize visual positioning.

2. The multi-feature fusion based visual positioning method according to claim 1, characterized in that, The step S2 specifically comprises the following steps: S2.1, train a DeepLabv3+ network according to the semantic label annotation in the dataset, and store the network parameters after the training is completed; S2.2, train a DIW network according to the normal vector value annotation in the dataset, and store the network parameters after the training is completed; S2.3, divide the data set into a query image and a candidate image in a ratio of 1:9, filter out the candidate images with a pose difference within a threshold from the query image from the candidate image set according to the image pose data in the data set, and store the candidate images as a positive set of the query image, and the remaining candidate images as a negative set of the query image; S2.4, train a multi-feature extraction and fusion network by using the generated query image and the corresponding positive and negative sets, and use a triplet loss in the training process, select a picture corresponding to a vector with the largest Euclidean distance from the query image vector in the positive set as a positive sample, and randomly select a specific number of pictures in the negative set as negative samples each time; S2.5, after the network training is completed, input each picture in the rendering data set into DeepLabv3+, DIW and multi-feature extraction and fusion network, obtain the global fusion vector of each picture, and store all vectors as a fusion feature database.

3. The multi-feature fusion based visual positioning method of claim 1, wherein: The step S3 includes the following steps: S3.1, input the query image into DeepLabv3+ and DIW network respectively for semantic segmentation and normal vector estimation, and obtain the semantic label prediction result and normal vector numerical estimation result of the query image; S3.2, input the RGB image, semantic segmentation label prediction result and normal vector value of the query into the multi-feature extraction and fusion network together, and calculate the global fusion vector of the query image.

4. The multi-feature fusion based visual positioning method according to claim 1, characterized in that, The step S5 includes the following steps: S5.1, according to the pose information of the labeled data set, calculate and filter out each training graph as a query graph the corresponding candidate graph set with a pose difference within a threshold value ; S5.2, for each pixel point in the query graph calculating the re-projection error value of the candidate graph with the corresponding pixel in the query graph calculating the re-projection error value of the candidate graph ; S5.3, count the reprojection difference of all the pixel points in each pixel point neighborhood in the query graph, and calculate the standard deviation ; S5.4, repeat steps S5.2-S5.3 for each candidate graph in the candidate graph set to obtain the corresponding standard deviation under each candidate graph and then apply an activation function to it and take the mean as the confidence true value of ;​ S5.5, use the confidence true value to train the confidence estimation network, and store the network parameters.

5. The multi-feature fusion based visual positioning method according to claim 1, characterized in that, The step S7 includes the following steps: S7.1, perform SIFT feature matching on the query image and each retrieved candidate image, and obtain a series of 2D-2D matching pairs between the query image and the candidate image; S7.2, read the corresponding depth map of each candidate image, obtain the depth information of each feature point of the candidate image matched on the query image, calculate the coordinate value of the feature point in the three-dimensional space according to the depth information and the camera pose annotation of the candidate image, and realize the 2D-3D matching pair between the candidate image and the three-dimensional model; The step S8 includes the following steps:

6. The multi-feature fusion based visual positioning method according to claim 1, characterized in that, S8.1, for each 2D-3D matching pair, find the corresponding confidence value according to the pixel coordinates of the 2D point in the query image, and the confidence value is obtained in S6; S8.2, compare the confidence value on the 2D point pixel with the confidence threshold value, if the 2D point confidence value is lower than the threshold value, discard it, otherwise keep it; S8.3, apply RANSAC PnP algorithm to the remaining 2D-3D pairs to obtain the camera pose and realize visual positioning. It also includes visual display of data, specifically including:

7. The multi-feature fusion based visual positioning method according to claim 1, characterized in that, Load the three-dimensional model of the scene, render and display in the window, support users to use mouse and scroll wheel for three-dimensional interaction of view angle; load the query RGB image input by the user, display the semantic segmentation prediction result image, normal vector estimation result image and confidence estimation image, and display the image retrieval result in the form of a list, each item in the list contains the thumbnail and file name of the candidate image, and the results of pose calculation are listed in the form of space x, y, z coordinates and yaw, pitch, roll data, and the query image is projected in the three-dimensional scene at the calculated pose view angle. The system includes the following modules:

8. A multi-feature fusion based visual positioning system, characterized in that, ​ The data processing module uses a three-dimensional reconstruction technology to construct a three-dimensional model of a scene from a scene photo collected in advance, applies snapshot rendering to the model to generate an annotated image dataset, inputs pictures in the annotated image dataset into a semantic segmentation network, a normal vector estimation network, and a multi-feature extraction and fusion network, extracts fusion features of each picture in the database, and constructs a fusion feature database. The image retrieval module inputs a query RGB picture input into the system into the semantic segmentation network and the normal vector estimation network, calculates semantic information and normal vector information of the image, inputs RGB, semantic, and normal vector data into the multi-feature extraction and fusion network, and obtains fusion features of the image. The fusion features of the query image and each feature in the fusion feature database are subjected to Euclidean distance calculation, and distances are sorted, and a plurality of images with relatively minimum distances are selected as a candidate image set result of image retrieval of the query image. The pose calculation module performs confidence estimation on the original RGB image based on the semantic information and the normal vector information calculated in the image retrieval module, obtains a pixel-level confidence estimation result of the query image, performs local feature matching on each image in the query image and the candidate image set, maps 2D feature points in the candidate image to 3D coordinates by using depth and pose information of the candidate image, thereby establishing a matching relationship between 2D points in the query image and 3D points in the three-dimensional model, removes matching pairs in which a 2D point in 2D-3D is located at a pixel position with a confidence lower than a threshold, and performs RANSAC PnP calculation on the remaining matching pairs to obtain a pose calculation result of the camera and complete visual positioning.

9. The multi-feature fusion based visual positioning system of claim 8, wherein, The data display module loads a three-dimensional model of a scene, renders and displays the three-dimensional model in a window, supports three-dimensional interaction of a view angle by using a mouse and a scroll wheel, simultaneously loads a query RGB image input by a user, displays a semantic segmentation prediction result image, a normal vector estimation result image, and a confidence estimation image, displays image retrieval results in a list form, each item in the list contains a thumbnail and a file name of a candidate image, lists results of pose calculation in a data form of spatial x, y, and z coordinates and yaw, pitch, and roll, and projects the query image in a calculated pose view angle in a three-dimensional scene. ​

Images