A method for assisted positioning based on a CNN-based visual prior map

Abstract

The application relates to a CNN-based visual prior map aided positioning method, which comprises the following steps: acquiring image information from a monocular camera, acquiring 2D straight line features through a 2D feature extraction method, acquiring depth image data through a pre-trained image depth estimation CNN model, and constructing a visual prior map according to the 2D straight line features and the depth image data; acquiring image straight line features for the image information, realizing matching of the visual prior map and the image straight line features according to the image straight line features and the visual prior map, acquiring an offset through pose estimation, and realizing positioning in a space environment. Compared with the prior art, the application can obtain accurate camera initial pose estimation by using the CNN-based visual prior map constructed in advance in the case that IMU lacks noise and bias measurement under high-speed motion, and can reduce the cumulative error of visual inertial navigation.

Description

TECHNICAL FIELD

[0001] The present application relates to visual inertial odometry technology, and in particular to a method for auxiliary positioning based on a CNN visual prior map. BACKGROUND

[0002] Existing visual inertial odometry technology is mostly based on feature extraction and matching between two frames of images to calculate the inter-frame transformation pose, uses a monocular camera or a binocular camera to collect environmental images, and selects point features or line features from color images to associate data between two frames through feature matching. In order to improve the accuracy of pose estimation, a triangulation method is usually used to calculate the spatial position of features. The main problem of this method is that the triangulation of multiple features between frames consumes a large amount of calculation and system resources. The CNN learning method can directly obtain the environmental spatial depth corresponding to a single color image, which can avoid the calculation of the spatial position of multiple features through triangulation. However, there is a significant deviation in determining the depth value of the corresponding pixel feature directly using the estimated image depth, and the method still relies on inter-frame feature matching as the pose solving method, which can easily introduce error in subsequent optimization of pose estimation and cause the accumulation of errors.

[0003] There is also a method of using a laser radar to construct a point cloud map as a global constraint to optimize visual positioning, which uses a point cloud splicing method to splice the laser radar point cloud frames collected in the environment into a complete laser point cloud map, extracts three-dimensional line features in the laser point cloud map using plane segmentation to construct a laser prior map, and finally continuously matches the three-dimensional lines in the laser prior map with the image line features to realize camera positioning. The main problem of this method is that the cost of the laser radar is too high, and the scene generality of the laser prior map is low.

[0004] In summary, the existing visual inertial odometry technology has the following shortcomings:

[0005] (1) The method of constructing a visual odometry by calculating the inter-frame transformation pose has accumulated errors.

[0006] (2) The visual inertial odometry lacks acceleration incentives and is difficult to initialize pose estimation.

[0007] (3) The matching of laser point clouds and camera images as heterogeneous data is difficult.

[0008] (4) The cost of the laser radar is too high.

[0009] (5) The scene generality of the laser point cloud as a map constraint is extremely low. SUMMARY

[0010] The application aims to overcome the defects of the prior art and provide a CNN-based visual prior map assisted positioning method, which extracts depth information of a monocular image by using a CNN to estimate the spatial depth of high-precision 2D linear features, reduces the calculation amount of feature triangulation, and solves or partially solves the problem of long calculation time and large resource consumption caused by multi-feature triangulation.

[0011] The object of the application can be achieved by the following technical solutions:

[0012] The application provides a CNN-based visual prior map assisted positioning method, which comprises the following steps:

[0013] Image information is acquired from a monocular camera, 2D linear features are acquired by a 2D feature extraction method, depth image data are acquired by a pre-trained image depth estimation CNN model, and a visual prior map is constructed according to the 2D linear features and the depth image data.

[0014] For the image information, image linear features are acquired, matching between the visual prior map and the image linear features is realized according to the image linear features and the visual prior map, an offset is acquired by pose estimation, and positioning in a spatial environment is realized.

[0015] As a preferred technical solution, the 2D linear feature extraction method comprises the following steps:

[0016] According to the image information, linear features and point features of a current frame are acquired, and are matched with linear features and point features of a previous frame respectively, and associated linear features and associated point features are acquired.

[0017] According to the distance between each associated linear feature and associated point feature, for each associated linear feature, the number of associated point features in a preset range is acquired, the associated linear feature is retained when the number of associated point features exceeds a preset threshold, otherwise it is discarded, and finally the remaining associated linear features are used as the 2D linear features.

[0018] As a preferred technical solution, the linear features and point features are acquired by using an LSD linear feature extraction algorithm and a FAST corner detection algorithm respectively.

[0019] The acquisition of the associated linear features comprises the following steps:

[0020] According to the linear features of the previous frame and the current frame, an LBD line descriptor is constructed, the linear features in the current frame are matched with those in the previous frame, and the 2D linear features associated with the previous frame are retained as the associated linear features.

[0021] The acquisition of the associated point features includes the following steps:

[0022] Based on the point features of the previous and current frames, a BRIEF point feature descriptor is constructed. The point features of the current frame are then matched with those of the previous frame. The RANSAC algorithm is used to eliminate incorrect matching results, and the 2D point features associated with the previous frame are retained as the associated point features.

[0023] The distance D between each point feature and each line feature in the current frame is obtained using the following formula:

[0024]

[0025] In the formula, (x,y) are the pixel coordinates of the point feature, (a,b) are the pixel coordinates of the starting point in the line feature, and (c,d) are the pixel coordinates of the ending point in the line feature.

[0026] As a preferred technical solution, the acquisition of the pre-trained image depth estimation CNN model includes the following steps:

[0027] Obtain a training sample set, wherein each sample in the training sample set includes color image data and corresponding depth image data;

[0028] The image depth estimation CNN model is trained based on the training sample set. The Kitti dataset is used as ground truth information to verify the trained model. After the loss function value reaches the preset convergence condition, the pre-trained salient object detection model is obtained.

[0029] As a preferred technical solution, constructing a visual prior map based on the 2D straight line features and the depth image data specifically includes the following steps:

[0030] For the depth image data, a rectangle of a preset size is constructed with the 2D straight line feature as the center, and then divided into multiple smaller rectangles with a width of 1.

[0031] For each small rectangle, calculate the sum of the depth values ​​on both sides and determine whether the sum of the depth values ​​on both sides exceeds a preset threshold. If so, select the largest sum of the depth values ​​on both sides as the depth value of the 2D straight line feature pixel. If not, use the average depth value of all pixels in each small rectangle as the depth value of the 2D straight line feature pixel, thus completing the construction of a visual prior map with a three-dimensional straight line structure.

[0032] As a preferred technical solution, the matching of visual prior maps with the straight line features of images specifically includes the following steps:

[0033] In the VINS-Mono algorithm, the aforementioned visual prior map is added as a global map constraint. The visual prior map is transformed from the world coordinate system to the camera coordinate system and projected into the current frame. Three-dimensional straight lines with depth values ​​greater than 0 that exist in the current frame image are retained. A matching relationship between the straight line features and the projected straight lines in the current frame is established using preset matching rules. The expected matching relationship is selected through preset filtering rules to complete the matching.

[0034] As a preferred technical solution, establishing the matching relationship between straight line features and projected straight lines onto the current frame using preset matching rules includes the following steps:

[0035] Traverse the projected lines in the current frame, calculate the angle and endpoint distance between them and each line feature in the current frame. If the angle between two lines is less than the threshold δ, and the distance between the starting points and the ending points of the two lines is less than the threshold ε, then establish a matching relationship between the projected lines and the line features.

[0036] As a preferred technical solution, selecting the expected matching relationship through preset filtering rules includes the following steps:

[0037] Multiple matching relationships are filtered by the depth value of the three-dimensional line corresponding to the projected line, and the projected line with the largest corresponding depth value is retained as the final expected match with the line feature in the current frame.

[0038] As a preferred technical solution, obtaining the offset through pose estimation to achieve localization in a spatial environment includes the following steps:

[0039] Based on the image information of the current frame and historical frames, a sliding window of a preset length is constructed. Based on the matching relationship between the visual prior map and the straight line features of the image, the objective function of the sliding window is constructed.

[0040] The angle deviation between the projected line and the line feature, as well as the distance deviation between the projected line and the line feature, are inserted as constraint factors into the pose optimization to calculate the offset when the matching error is minimized.

[0041] During the operation of the VINS-Mono algorithm, the camera pose is optimized in real time using a sliding window objective function to achieve camera positioning in the spatial environment.

[0042] As a preferred technical solution, the objective function of the sliding window is as follows:

[0043]

[0044] The angular deviation between the projected line and the line feature is calculated using the following formula:

[0045]

[0046] The distance deviation between the projected line and the line feature is calculated using the following formula:

[0047]

[0048] In the formula, W represents the number of observation frames in the sliding window that observe the same spatial line, N represents the correct matching pairs of the 3D line and the line feature in the current frame, F represents the error function, and θ represents the error function. i To find the angle between the expected pair of lines, d i The direction vector of the projected line is the endpoint distance of the expected matching line pair. The direction vector of the straight line feature is

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] (1) CNN is used to extract the depth information of monocular images to estimate the spatial depth of high-precision 2D straight line features, reduce the computation of feature triangulation, and obtain a visual prior map with the same precision as the laser map as a global map constraint.

[0051] (2) When matching the visual prior map with the camera image features, the depth value of the three-dimensional line is used for secondary constraint, which can achieve accurate matching of the three-dimensional line and the 2D line features. It can perform high-precision three-dimensional spatial positioning by relying solely on a monocular camera, and can also perform initial pose estimation well in the absence of acceleration excitation.

[0052] (3) High-precision 2D straight line features are selected by relying on the method of associating point features and line features, while retaining high-dimensional geometric contour features in the environment and avoiding the introduction of messy straight lines that reduce the accuracy of the visual prior map.

[0053] (4) Compared with the existing technical solutions that use triangulated two-dimensional image features to achieve inter-frame registration for pose estimation, resulting in high computational cost, or the low versatility of laser point cloud prior maps constructed using high-cost LiDAR, this invention proposes a high-precision 2D straight line feature depth estimation based on CNN to construct a visual prior map with the same precision as the laser prior map, thus avoiding the cumulative error in pose estimation caused by using inter-frame registration.

[0054] (5) Compared with the heterogeneous information matching between the laser point cloud prior map and the monocular camera image, the visual prior map and the monocular camera image are homogeneous information, which makes it easier to achieve registration. Attached Figure Description

[0055] Figure 1 This is a flowchart of the construction process of the CNN-based visual prior map in Example 1;

[0056] Figure 2 This is a flowchart of the pose estimation process for the visual camera in Example 1;

[0057] Figure 3 This is a flowchart of the high-precision 2D straight line feature extraction process in Example 1;

[0058] Figure 4 This is a flowchart of the CNN image depth estimation training process in Example 1;

[0059] Figure 5 This is a flowchart of the high-precision 2D straight line feature depth information acquisition process in Example 1;

[0060] Figure 6 This is a schematic diagram of the sliding window in Example 1;

[0061] Figure 7 This is a schematic diagram comparing the projection of 3D spatial straight line point cloud in Example 1, where (a) is a laser straight line, (b) is a high-precision straight line, and (c) is an LSD straight line. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0063] Example 1

[0064] like Figures 1-2 The present embodiment provides a CNN-based visual prior map-assisted localization method, which includes the following steps:

[0065] Step S1 involves filtering high-precision 2D line features based on the number of points surrounding the line features in the image, thereby preserving high-dimensional contour features in the environment. The flowchart for high-precision 2D line feature extraction is shown below. Figure 3 The extraction of high-precision 2D straight line features specifically includes the following steps:

[0066] Step S101: At the image processing front end of the VINS-Mono algorithm, the LSD line feature extraction algorithm and the FAST corner detection algorithm are used to extract 2D line features and 2D point features from all input images. Then, the LBD line descriptor is constructed to match the line features in the current frame with those in the previous frame. Only the 2D line features associated with the previous frame are retained in the current frame.

[0067] Step S102: For point features in the current frame, the point features in the current frame are matched with those in the previous frame by constructing a BRIEF point feature descriptor, and the RANSAC algorithm is used to remove incorrect matching results, so that only 2D point features that are correctly associated with those in the previous frame are retained in the current frame.

[0068] Step S103: After all line and point features in the current frame have been filtered, traverse the point and line features throughout the entire current frame, and use the formula...

[0069]

[0070] Calculate the distance D between each point feature and each line feature in the current frame, where (x, y) are the pixel coordinates of the point feature, (a, b) are the pixel coordinates of the starting point of the line feature, and (c, d) are the pixel coordinates of the ending point of the line feature. When D is 0, the point feature exists on the line feature; when D is not 0, the point feature exists outside the line feature. Count the number of point features around each line feature within the threshold [0, α]. When this number exceeds a certain threshold β, the line feature is retained as a high-precision 2D line feature.

[0071] Step S2 involves training an image depth estimation CNN model based on color and depth images. The CNN model converts the input color image into a depth image and uses classification rectangle depth matching to obtain high-precision 2D straight-line feature pixel depth values ​​to construct a visual prior map. The accuracy of this visual prior map can reach that of a laser prior map. Figure 4 The above is a flowchart of the CNN image depth estimation training process. CNN-based image depth estimation includes the following steps:

[0072] Color and depth images of real-world scenes in indoor and outdoor environments under different lighting conditions and angles were collected using an RGB-D camera. These images were then used to create a corresponding training dataset for a CNN. The VGG-16 dataset was selected as the training model. In the initial stage, the feature size was not reduced, preserving the details of the image training to the maximum extent. This method clearly distinguishes the depth estimation of image texture edges. The Kitti dataset was used as ground truth for model validation. The model was considered converged when the error between the trained and ground truth pixel depth values ​​was within a threshold ω. This method yields an image depth estimation CNN model.

[0073] Step S3: Add a visual prior map as a global map constraint to the existing visual odometry that calculates pose transformation through inter-frame registration, reduce the cumulative error of visual odometry, and achieve high-precision pose estimation based on monocular camera visual odometry.

[0074] like Figure 5 The above describes the process of constructing a visual prior map based on CNN:

[0075] Step S301: In the image processing front end of the VINS-Mono algorithm, the depth map corresponding to each input color image is calculated by the constructed CNN model. A rectangle with length n and width m is constructed with the high-precision 2D straight line feature in the current frame as the center. According to the length of the high-precision 2D straight line feature, the rectangle can be divided into n smaller rectangles with length m and width 1.

[0076] Step S302: Using the pixel of the high-precision 2D straight line feature as the center, calculate the sum of the depth values ​​of the pixels on the left and right sides of the small rectangle. If the difference between the sums of the depth values ​​of the pixels on the left and right sides exceeds a threshold σ, select the pixel with the largest sum of pixel depth values ​​as the depth value of the high-precision 2D straight line feature pixel. If the difference between the sums of the depth values ​​of the pixels on the left and right sides is less than or equal to the threshold σ, calculate the average depth value of all pixels within the small rectangle as the depth value of the high-precision 2D straight line feature pixel. By traversing n small rectangles using the same method, the complete spatial depth information of the high-precision 2D straight line feature can be obtained, as shown in the attached figure. Figure 4 As shown, a visual prior map with a three-dimensional straight-line structure is constructed based on the current frame camera pose during the operation of the VINS-Mono algorithm.

[0077] The matching process between the visual prior map and the line features of the image includes the following steps:

[0078] Step S311, through formula L c =T cb *T bw *L w The coordinates of the 3D lines in the visual prior map are transformed from the world coordinate system to the camera coordinate system, where L... c and L w T represents a three-dimensional straight line in the camera coordinate system and the world coordinate system, respectively. cb and T bw These are the transformation matrices from the body coordinate system to the camera coordinate system and from the world coordinate system to the body coordinate system, respectively. The current frame camera pose estimate output by the VINS-Mono algorithm is used as the prior pose and substituted into T. bw T cb These are external calibration parameters for the system.

[0079] Step S312: Filter the 3D lines by the depth value of the 3D lines in the camera coordinate system, and save only the 3D lines with a depth value greater than 0. Then, project these lines into the current frame through the camera intrinsic parameters, and determine whether the pixel coordinates of the two ends of the projected line are in the image. Perform a second filtering to retain the 3D lines that only exist in the current frame image.

[0080] Step S313: After calculating the 3D straight line in the current frame camera view of the visual prior map, online matching is performed using the projected straight line in the current frame and the straight line features in the current frame. First, the projected straight line in the current frame is traversed, and the angle and endpoint distance between it and all straight line features in the current frame are calculated. When the angle between two straight lines is less than the threshold δ, and the distance between the starting point pair and the ending point pair is less than the threshold ε, a matching relationship is established between the projected straight line and the straight line features.

[0081] Step S314: After traversing all the projected lines in the current frame, filter the multiple sets of matching relationships by the 3D line depth value corresponding to the projected line, and retain only the projected line with the largest corresponding depth value as the final correct match with the line feature in the current frame.

[0082] The pose estimation process of a visual camera includes the following steps:

[0083] like Figure 6 The process involves constructing a sliding window with a size of 20 frames. The current frame camera pose estimate output by the VINS-Mono algorithm is used as the prior pose. The pose information of the previous 19 frames is derived using inter-frame transformation. The 3D lines in each frame that correctly match the image's line features and are projected through the camera's intrinsic parameters can all be represented by Kexp(χ+Δχ)L, where K is the camera intrinsic parameter, L is the 3D line, χ is the Lie algebra representation of the current frame camera pose, and Δχ is the Lie algebra representation of the inter-frame transformed pose. The objective function for constructing the sliding window is shown in the equation. Where W is the number of observation frames in the sliding window that observe the same spatial line, N is the number of correct matching pairs between the 3D line and the line feature in the current frame, F is the error function, and θ i To correctly match the angle between the pairs of lines, d i To correctly match the endpoint distances of line pairs, the direction vector of the projected line is defined as follows, considering the angular deviation between the projected line and the line feature. The direction vector of the straight line feature is Through The included angle deviation is inserted as a constraint factor in pose optimization to minimize the matching error. For the distance deviation between the projected line and the line feature, the formula is used... The endpoint distance combined with the straight-line distance is used as a constraint factor inserted into the pose optimization to calculate the offset when the matching error is minimized. During the VINS-Mono algorithm, a sliding window objective function is used to perform real-time error optimization of the camera pose, achieving high-precision positioning of the camera in the spatial environment.

[0084] Figure 7This is a schematic diagram comparing the projection of 3D spatial straight line point clouds, where (a) is a laser-generated straight line, (b) is a high-precision straight line, and (c) is an LSD (Low-Stakes Map) straight line. The root mean square error comparison between the visual prior map and the laser prior map is shown in Table 1.

[0085] Table 1 Comparison of root mean square errors between visual prior maps and laser prior maps

[0086]

[0087] Therefore, visual prior maps can achieve the same accuracy as laser maps.

[0088] This embodiment proposes a CNN-based visual prior map-assisted localization method. Compared to existing solutions that use triangulated 2D image features for inter-frame registration and pose estimation, resulting in high computational costs, or methods using high-cost LiDAR-constructed LiDAR point cloud prior maps with low versatility, this method proposes a CNN-based high-precision 2D straight-line feature depth estimation method to construct a visual prior map with the same accuracy as the LiDAR prior map, avoiding the cumulative error in pose estimation caused by inter-frame registration. Furthermore, compared to the heterogeneous information matching between the LiDAR point cloud prior map and monocular camera images, the visual prior map and monocular camera images are homogeneous information, making registration easier.

[0089] This method utilizes a combination of point and line features to filter out high-precision 2D line features, preserving high-dimensional geometric contour features in the environment and avoiding the introduction of cluttered lines that could reduce the accuracy of the visual prior map. A CNN is used to extract depth information from the monocular image to estimate the spatial depth of the high-precision 2D line features, reducing the computational cost of feature triangulation and obtaining a visual prior map with the same accuracy as the laser map, serving as a global map constraint. When matching the visual prior map with camera image features, the depth values ​​of the 3D lines are used for secondary constraints, enabling accurate matching between 3D and 2D line features. This method can achieve high-precision 3D spatial localization relying solely on a monocular camera and can also perform good initial pose estimation even in the absence of acceleration excitation.

[0090] Example 2

[0091] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the assisted localization method based on a CNN-based visual prior map as described in Embodiment 1.

[0092] Example 3

[0093] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing an assisted localization method based on a CNN-based visual prior map as described in Embodiment 1.

[0094] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A CNN-based method for assisted localization using visual prior maps, characterized in that, Includes the following steps: Image information is acquired from a monocular camera, 2D line features are obtained through a 2D feature extraction method, depth image data is obtained through a pre-trained image depth estimation CNN model, and a visual prior map is constructed based on the 2D line features and the depth image data. For the image information, straight line features are obtained. Based on the straight line features and the visual prior map, the visual prior map and the straight line features are matched. The offset is obtained through pose estimation to achieve localization in the spatial environment. The specific steps for obtaining 2D line features using 2D line feature extraction methods include the following: Based on the image information, obtain the line features and point features of the current frame, and match them with the line features and point features of the previous frame respectively to obtain the associated line features and associated point features. Based on the distance between each associated line feature and associated point feature, for each associated line feature, the number of associated points whose surrounding associated point feature values ​​are within a preset range is obtained. If the number of associated points exceeds a preset threshold, the associated line feature is retained; otherwise, it is discarded. The remaining associated line features are then used as the 2D line features. The line features and point features mentioned above are obtained using the LSD line feature extraction algorithm and the FAST corner detection algorithm, respectively. The acquisition of the associated line features includes the following steps: Based on the line features of the previous and current frames, an LBD line descriptor is constructed. The line features of the current frame are matched with those of the previous frame, and the 2D line features associated with the previous frame are retained as the associated line features. The acquisition of the associated point features includes the following steps: Based on the point features of the previous and current frames, a BRIEF point feature descriptor is constructed. The point features of the current frame are then matched with those of the previous frame. The RANSAC algorithm is used to eliminate incorrect matching results, and the 2D point features associated with the previous frame are retained as the associated point features. Distance between each point feature and each line feature in the current frame Obtain it using the following formula: In the formula, The pixel coordinates of the point feature. These are the pixel coordinates of the starting point in the straight line feature. These are the pixel coordinates of the endpoint in the straight line feature. Constructing a visual prior map based on the 2D line features and the depth image data specifically includes the following steps: For the depth image data, a rectangle of a preset size is constructed with the 2D straight line feature as the center, and then divided into multiple smaller rectangles with a width of 1. For each small rectangle, calculate the sum of the depth values ​​on both sides and determine whether the sum of the depth values ​​on both sides exceeds a preset threshold. If it does, select the largest sum of the depth values ​​on both sides as the depth value of the 2D line feature pixel. If not, use the average depth value of all pixels within each small rectangle as the depth value of the 2D line feature pixel, thus completing the construction of a visual prior map with a 3D line structure. The process of obtaining offset through pose estimation to achieve localization in a spatial environment includes the following steps: Based on the image information of the current frame and historical frames, a sliding window of a preset length is constructed. Based on the matching relationship between the visual prior map and the straight line features of the image, the objective function of the sliding window is constructed. The angle deviation between the projected line and the line feature, as well as the distance deviation between the projected line and the line feature, are inserted as constraint factors into the pose optimization to calculate the offset when the matching error is minimized. During the operation of the VINS-Mono algorithm, the camera pose is optimized in real time using a sliding window objective function to achieve camera positioning in the spatial environment.

2. The CNN-based visual prior map-assisted localization method according to claim 1, characterized in that, The acquisition of the pre-trained image depth estimation CNN model includes the following steps: Obtain a training sample set, wherein each sample in the training sample set includes color image data and corresponding depth image data; The image depth estimation CNN model is trained based on the training sample set. The Kitti dataset is used as ground truth information to verify the trained model. After the loss function value reaches the preset convergence condition, the pre-trained salient object detection model is obtained.

3. The CNN-based visual prior map-assisted localization method according to claim 1, characterized in that, The specific steps for matching the visual prior map with the line features of the image include the following: In the VINS-Mono algorithm, the aforementioned visual prior map is added as a global map constraint. The visual prior map is transformed from the world coordinate system to the camera coordinate system and projected into the current frame. Three-dimensional straight lines with depth values ​​greater than 0 that exist in the current frame image are retained. A matching relationship between the straight line features and the projected straight lines in the current frame is established using preset matching rules. The expected matching relationship is selected through preset filtering rules to complete the matching.

4. The CNN-based visual prior map-assisted localization method according to claim 3, characterized in that, Establishing a matching relationship between line features and projected lines in the current frame using preset matching rules includes the following steps: Traverse the projected lines in the current frame, calculate the angle and endpoint distance between each line and each line feature in the current frame, and if the angle between two lines is less than a threshold... Furthermore, the distance between the starting points and the ending points of the two lines are both less than the threshold. At that time, a matching relationship is established between the projected line and the line feature.

5. The CNN-based visual prior map-assisted localization method according to claim 3, characterized in that, Selecting the desired matching relationship using preset filtering rules includes the following steps: Multiple matching relationships are filtered by the depth value of the three-dimensional line corresponding to the projected line, and the projected line with the largest corresponding depth value is retained as the final expected match with the line feature in the current frame.

6. The CNN-based visual prior map-assisted localization method according to claim 1, characterized in that, The objective function of the sliding window is as follows: The angular deviation between the projected line and the line feature is calculated using the following formula: The distance deviation between the projected line and the line feature is calculated using the following formula: In the formula, This represents the number of observation frames that observe the same spatial line within the sliding window. N This represents the correct matching pair of 3D lines and line features in the current frame. F Let be the error function. To match the angle between the pairs of lines, The direction vector of the projected line is the endpoint distance of the expected matching line pair. The direction vector of the straight line feature is .

Images