Visual synchronous positioning and mapping method and device fusing point and line features

Abstract

The invention provides a visual synchronous localization and mapping method and device fusing point and line features, and relates to the technical field of visual synchronous localization and mapping, and the method comprises the steps: a key point extraction branch outputs a score graph used for representing the distribution of key points in a feature graph and a descriptor used for distinguishing each key point; the line feature extraction branch outputs a line segment of a code corresponding to each pixel point and the probability that each pixel point is a line segment endpoint, and determines a line segment which corresponds to each pixel point and is expressed by a target endpoint; according to the score plot and the descriptor, key points of the front and back image frames are matched, a successfully matched target key point is associated with a line segment corresponding to each pixel point, the line segments in the front and back image frames are matched, and features of a successfully matched target line segment are determined; according to the target key points and the target line segment features, the current vehicle is controlled to perform synchronous positioning and mapping, and the adverse effects of the two types of illumination defects on visual synchronous positioning and mapping can be relieved at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of visual synchronous positioning and mapping, and in particular to a visual synchronous positioning and mapping method and device integrating point and line features. Background Art

[0002] Visual Simultaneous Localization and Mapping (vSLAM) has been widely used in robot navigation, drone flight, autonomous driving and other fields. Compared with SLAM technology based on LiDAR, vSLAM can provide high-precision positioning and mapping capabilities with cameras, which are lighter and cheaper sensors. In addition, cameras can also capture rich environmental details, including color, texture, etc., to achieve more robust performance in dynamic environments.

[0003] However, vSLAM technology still faces many challenges in practical applications, especially in scenes with complex or drastically changing lighting conditions. These lighting challenges can be divided into the following two categories:

[0004] Short-term illumination challenge: This refers to the failure of feature detection and tracking due to illumination changes (such as shadows, glare, or low light) between adjacent frames, affecting the accuracy of pose estimation.

[0005] Long-term illumination challenge: This refers to the matching failure between the query frame and the previously built map due to significant changes in lighting conditions (such as day-to-night transitions, indoor-outdoor scene switching), which reduces the relocalization success rate.

[0006] In order to deal with the above problems, recent studies have attempted to improve the illumination robustness of vSLAM through methods such as image enhancement and normalization, feature detection based on deep learning, multi-feature fusion, loop detection and map reuse. Although existing technologies have solved short-term or long-term illumination challenges to a certain extent, most methods only focus on a single problem and it is difficult to deal with the above two types of illumination challenges at the same time. Summary of the invention

[0007] In view of this, an object of the present invention is to provide a method and device for visual synchronous positioning and mapping that integrates point and line features, which can simultaneously alleviate the adverse effects of the above two types of lighting defects on visual synchronous positioning and mapping.

[0008] In a first aspect, an embodiment provides a method for visual synchronous positioning and mapping by integrating point and line features, the method comprising:

[0009] Acquire a multi-scale feature map of the current image frame; wherein the multi-scale feature map includes a first scale feature map and a second scale feature map;

[0010] Inputting the first scale feature map into a key point extraction branch to obtain a score map for characterizing the distribution of key points in the first scale feature map and a descriptor for distinguishing each key point;

[0011] Inputting the second multi-scale feature map into the line feature extraction branch, obtaining the encoded line segment corresponding to each pixel in the second multi-scale feature map, and the probability that each pixel is an endpoint of the line segment, and determining the line segment expressed by the target endpoint corresponding to each pixel;

[0012] According to the score map and the descriptor, the key points of the current image frame are matched with the key points of the previous image frame to determine the target key points that are successfully matched;

[0013] Associating the target key point with the line segment expressed by the target endpoint for each pixel point, and matching the line segment expressed by the target endpoint for each pixel point of the current image frame with the line segment expressed by the target endpoint for each pixel point of the previous image frame, and determining the target line segment features that are successfully matched;

[0014] According to the target key points and the target line segment features, the current vehicle is controlled to perform synchronous positioning and mapping.

[0015] In an optional implementation, the step of inputting the second multi-scale feature map into a line feature extraction branch, obtaining a line segment corresponding to each pixel in the second multi-scale feature map, and a probability that each pixel is an endpoint of the line segment, and determining a line segment expressed as a target endpoint corresponding to each pixel, comprises:

[0016] Outputting a tensor of the second multi-scale feature map based on the processing of the line segment encoding branch in the line feature extraction branch;

[0017] Determine, according to the tensor, a line segment code for each pixel in the second multi-scale feature map;

[0018] The line segment code is composed of the vertical distance from each pixel to the line segment corresponding to each pixel, the angle between the line segment corresponding to each pixel and the global coordinate system, the angle between the line connecting the foot point of the line segment corresponding to each pixel and the first endpoint of the line segment and the x-axis of the global coordinate system, and the angle between the line connecting the foot point of the line segment corresponding to each pixel and the second endpoint of the line segment and the x-axis of the global coordinate system;

[0019] The line segment code is decoded to obtain a line segment in which each pixel point is expressed in the form of endpoint parameters.

[0020] In an optional implementation, the step of inputting the second multi-scale feature map into a line feature extraction branch, obtaining a line segment corresponding to each pixel in the second multi-scale feature map, and a probability that each pixel is an endpoint of a line segment, and determining a line segment expressed as a target endpoint corresponding to each pixel, further includes:

[0021] Outputting the probability that each pixel point in the second multi-scale feature map is an endpoint of a line segment based on the endpoint detection branch processing in the line feature extraction branch;

[0022] According to the probability that each pixel is a line segment endpoint and the Euclidean distance between the endpoint parameter and each pixel, the target endpoint of the line segment corresponding to each pixel is determined, and the line segment corresponding to each pixel is expressed in the form of the target endpoint.

[0023] In an optional embodiment, before the step of processing the endpoint detection branch in the line feature extraction branch, the method further includes:

[0024] Input the second multi-scale feature map into the receptive field enhancement structure, firstly compress the size of the feature map by each convolution layer in the downsampling channel, and then process the feature map by each deconvolution layer in the upsampling channel; wherein each convolution layer in the downsampling channel corresponds to each deconvolution layer in the upsampling channel to form a sampling pair;

[0025] The first output result of each deconvolution layer and the second output result of the corresponding convolution layer are added, and then the summed results of each sampling pair are concatenated to output a feature map result of the same scale as the second multi-scale feature map.

[0026] In an optional embodiment, the step of associating the target key point with each pixel point and the line segment expressed by the target endpoint includes:

[0027] Calculating the distance from each of the target key points to the current line segment expressed by the target endpoint;

[0028] Compare the distance with a preset distance threshold, and determine whether the coordinates of the target key point are within the coordinate axis projection range of the current line segment, and determine each of the target key points belonging to the current line segment;

[0029] The above steps are repeated until all line segments expressed by the target endpoints of each pixel point have been traversed.

[0030] In an optional implementation, the step of matching a line segment expressed by each pixel point of the current image frame with the target endpoint with a line segment expressed by each pixel point of the previous image frame with the target endpoint to determine a target line segment feature of a successful match includes:

[0031] Calculating the confidence level of successful matching between each line segment in the current image frame and each line segment in the previous image frame based on the number of target key points associated with each line segment expressed by the target endpoint in the current image frame and the number of target key points associated with each line segment expressed by the target endpoint in the previous image frame;

[0032] If the confidence exceeds the confidence threshold, the line segment successfully matched between the current image frame and the previous image frame is the target line segment feature.

[0033] In an optional implementation, the step of obtaining a multi-scale feature map of the current image frame includes:

[0034] Acquire a current image frame, and preprocess the current image frame to obtain a grayscale image;

[0035] The grayscale image is input into the backbone network, and is processed by the convolution layer and the pooling layer of the backbone network to output a multi-scale feature map.

[0036] In a second aspect, an embodiment provides a visual synchronous positioning and mapping device integrating point and line features, the device comprising:

[0037] An acquisition module is used to acquire a multi-scale feature map of the current image frame; wherein the multi-scale feature map includes a first scale feature map and a second scale feature map;

[0038] A first extraction module, inputting the first scale feature map into a key point extraction branch, to obtain a score map for characterizing the distribution of key points in the first scale feature map and a descriptor for distinguishing each key point;

[0039] A second extraction module inputs the second multi-scale feature map into a line feature extraction branch to obtain a line segment encoded corresponding to each pixel in the second multi-scale feature map, a probability that each pixel is an endpoint of the line segment, and determines a line segment expressed as a target endpoint corresponding to each pixel;

[0040] A first matching module matches the key points of the current image frame with the key points of the previous image frame according to the score map and the descriptor, and determines the target key points that are successfully matched;

[0041] A second matching module associates the target key point with a line segment expressed by the target endpoint for each pixel point, and matches the line segment expressed by the target endpoint for each pixel point of the current image frame with the line segment expressed by the target endpoint for each pixel point of the previous image frame, to determine the target line segment features that are successfully matched;

[0042] The positioning and mapping module controls the current vehicle to perform synchronous positioning and mapping according to the target key points and the target line segment features.

[0043] In a third aspect, an embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any of the aforementioned implementation methods are implemented.

[0044] In a fourth aspect, an embodiment provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method described in any one of the aforementioned implementations.

[0045] The embodiment of the present invention provides a method and device for visual synchronous positioning and mapping that integrates point and line features, and realizes more accurate key point extraction and line feature extraction based on a multi-scale feature map. For line feature extraction, a line segment is first expressed in the form of a line segment endpoint, and the pixel points that can serve as the line segment endpoint are determined based on probability, and then based on the successfully matched target key points in the previous and next image frames, the number of target key points in the line segment in each endpoint expression form is determined, and then the same line segment that is successfully matched in the previous and next image frames is judged to obtain the target line segment feature. In this relatively simple way, relatively accurate target key points and target limiting features are obtained, which can facilitate the current vehicle to perform visual synchronous positioning and mapping operations with higher reliability based on this.

[0046] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0047] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flow chart of a method for visual simultaneous positioning and mapping that integrates point and line features provided by an embodiment of the present invention;

[0050] Figure 2 A functional module diagram of a visual synchronous positioning and mapping device integrating point and line features provided by an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the hardware architecture of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The inventors have found that many existing solutions to solve the illumination robustness of visual simultaneous positioning and mapping technologies have the following common problems:

[0054] Lack of uniformity: Most current studies deal with point features or line features independently, lacking a unified detection and optimization framework. This separation leads to poor consistency in feature detection and matching, which affects the overall performance of pose estimation and map construction.

[0055] Lack of real-time performance: The current method takes too long in the process of point and line feature extraction and matching, and does not fully utilize the spatial relationship between points and lines, which affects the overall time consumption of the SLAM system.

[0056] Based on this, an embodiment of the present invention provides a method and device for visual synchronous positioning and mapping that integrates point and line features. By simultaneously extracting point and line features and simplifying the matching between line features based on the association between successfully matched key points and line features, it can overcome two types of lighting defects while ensuring the convenience and reliability of visual synchronous positioning and mapping.

[0057] To facilitate understanding of this embodiment, a visual synchronous positioning and mapping method for integrating point and line features disclosed in an embodiment of the present invention is first introduced in detail. The method is executed in an intelligent control device such as a vehicle controller.

[0058] This method is implemented using an efficient visual simultaneous localization and mapping network based on point and line features, which includes a point and line feature detection network and a visual odometer module; the point and line feature detection network is used to simultaneously extract key point features and structured line features from the input grayscale image. The visual odometer module combines key point features and structured line features to estimate visual odometers, using a learning-driven front end and a geometrically optimized back end.

[0059] Among them, the point and line feature detection network includes a backbone network and a point and line extraction network;

[0060] The backbone network is responsible for extracting multi-scale feature maps based on the input grayscale image, providing input for subsequent key point detection and line feature detection. The backbone network is based on the design of SuperPoint, including convolutional layers and pooling layers. The input grayscale image I of size (H, W) first passes through eight convolutional layers, the activation function is ReLU, and the convolution kernel size of each layer is 3x3, and then passes through three layers of maximum pooling layers, and the pooling kernel size is 2x2. The final output multi-scale feature map F is as follows: F1 = (H, W, 64);

[0061] The point and line extraction network includes a key point extraction branch and a line feature extraction branch. It should be noted that there is no order of precedence in the processing of the key point extraction branch and the line feature extraction branch, and the input feature map is processed accordingly at the same time.

[0062] The key point extraction branch includes a scoring branch and a descriptor branch; the scoring branch is used to generate the probability of whether each pixel is a key point, and then determine whether each pixel is a key point. The structure includes a 3x3 convolution layer + activation function ReLU and a normalization layer; the convolution layer has three layers, and the convolution kernel is 3x3; its output channel number is 65 (64 key point grids and 1 "no key point" category); the normalized Softmax layer normalizes each 8x8 grid and outputs an HxW key point score map S k According to the score map, we can know the probability of each pixel being a key point and the distribution of pixels that are key points.

[0063] The descriptor branch is used to generate the descriptor D of each key point k , in order to distinguish each key point for easy matching, its structure is 3x3 convolution layer + activation function ReLU, the convolution layer is three layers, the convolution kernel is 3x3, and the number of output channels is 256; through bilinear interpolation, the descriptor is converted from Upsampled to HxW×64 D k , which is consistent with the scale of the score map.

[0064] The line feature extraction branch includes a line segment encoding branch and an extraction structure; the extraction structure includes a receptive field enhancement structure and an endpoint detection branch;

[0065] The receptive field enhancement structure can be a U-Net structure. The U-Net module is divided into a downsampling part and an upsampling part; the downsampling part is composed of 3x3 convolution layers, each layer is followed by a pooling operation, which is used to gradually compress the size of the feature map, thereby extracting higher-level semantic information; the upsampling part uses 3 layers of deconvolution, each deconvolution layer generates the feature map corresponding to the layer through a jump connection to the downsampling part, and splices the feature maps generated by the jump connections of each layer in the upsampling and downsampling parts, thereby retaining more detailed information for subsequent line feature extraction operations. The scale of the final output feature map is consistent with the input feature map F3, that is,

[0066] It should be noted that the U-Net network structure generally plays the role of image segmentation in conventional applications, but in the embodiments of the present invention, the U-Net network plays the role of improving the image receptive field so that the subsequent endpoint detection branch has a better detection effect, so as to make the line feature extraction more accurate.

[0067] In the line segment coding branch, the structure of the line segment coding branch is two layers of 3x3 convolution, and the output is A tensor based on which the line segments are encoded;

[0068] In the connection point detection branch, the branch structure is a 3x3 convolution and a normalized Softmax layer, which outputs the probability S of whether each pixel is a line segment endpoint. j , whose size is consistent with the input feature map F3.

[0069] At this point, the point and line feature detection network outputs the descriptor S k , score map D k , the line segment expresses the three parts of l(p1,p2).

[0070] The visual odometry module combines the point and line features output by the point and line feature detection network to perform pose estimation. Its main operation process includes point and line feature matching and initial pose estimation. First, the LightGlue method is used to match whether the key points in the two frames are the same. Then, the line features are represented based on the successfully matched key points to achieve the association between the point and line features. Then, the line features represented by the key point features are used to match whether the lines in the two frames are the same. Secondly, the pose of the current vehicle is estimated based on the point and line feature matching results, and positioning and mapping are achieved based on the pose of the current vehicle.

[0071] Figure 1 A flow chart of a method for visual simultaneous positioning and mapping that integrates point and line features provided in an embodiment of the present invention.

[0072] like Figure 1 As shown, the method comprises the following steps:

[0073] Step S102, obtaining a multi-scale feature map of the current image frame;

[0074] Here, the image frame at the current moment is processed by the feature model to output a feature map, which includes multiple scales; the multi-scale feature map includes a first-scale feature map and a second-scale feature map. In the embodiment of the present invention, the first-scale feature map and the second-scale feature map have the same scale.

[0075] Step S104: input the first scale feature map into the key point extraction branch to obtain a score map for characterizing the distribution of key points in the first scale feature map and a descriptor for distinguishing each key point.

[0076] The key point score and descriptor of each pixel are generated by the following formulas:

[0077] S k =Softmax(F)

[0078] D k =Descriptor(F)

[0079] Among them, S k is the score map, D k is the key point descriptor, and F is the first scale feature map of the input.

[0080] It should be noted that the output of the score map generation formula is 65 grids, including 64 key point grids and 1 "no key point" category. Then each 8x8 grid is normalized to output the HxWx64 key point score map S k The output of the key point descriptor formula is 256 channels, and then the descriptor is interpolated from Descriptor D upsampled to HxWx64 k .

[0081] Step S106: input the second multi-scale feature map into the line feature extraction branch to obtain the encoded line segment corresponding to each pixel in the second multi-scale feature map, as well as the probability that each pixel is an endpoint of the line segment, and determine the line segment expressed by the target endpoint corresponding to each pixel.

[0082] Here, the line feature extraction branch encodes the line segment corresponding to each pixel point in the feature map, and determines the pixel points belonging to the end points of the line segment based on probability; the embodiment of the present invention expresses the line features in the form of pixel points.

[0083] It should be noted that there is no order of precedence between step S104 and step S106, and it can be understood that corresponding processing is performed on the input feature map at the same time.

[0084] Step S108, matching the key points of the current image frame with the key points of the previous image frame according to the score map and the descriptor, and determining the successfully matched target key points.

[0085] The LightGlue method, score map and descriptor are used to match key points, that is, to determine whether the previous and next adjacent image frames are the same key point.

[0086] Step S110, associate the target key point with the line segment expressed by the target endpoint corresponding to each pixel point, and match the line segment expressed by the target endpoint corresponding to each pixel point of the current image frame with the line segment expressed by the target endpoint corresponding to each pixel point of the previous image frame to determine the target line segment features that are successfully matched.

[0087] Here, by establishing the association between the successfully matched target key points and the line segments, and then through the relationship between the target key points associated with the line segments, fast, simple and accurate matching between the line segments is achieved, and then the target line segment features are determined.

[0088] Step S112, according to the target key points and target line segment features, control the current vehicle to perform synchronous positioning and mapping.

[0089] Here, the initial position of the current vehicle at the current moment is estimated based on the target key points and target line segment features; then, based on the estimation results, the current vehicle is synchronously positioned and mapped.

[0090] In a preferred embodiment of practical application, more accurate key point extraction and line feature extraction are simultaneously achieved based on multi-scale feature maps; for line feature extraction, the line segment is first expressed in the form of line segment endpoints, and the pixel points that can serve as line segment endpoints are determined based on probability, and then based on the successfully matched target key points in the previous and next image frames, the number of target key points in the line segment of each endpoint expression form is determined, and then the same line segment successfully matched in the previous and next image frames is judged to obtain the target line segment feature; in this relatively simple way, relatively accurate target key points and target limiting features are obtained, which can facilitate the current vehicle to perform highly reliable visual synchronous positioning and mapping operations based on this.

[0091] In some embodiments, the image collected by the vehicle collection device may be pre-processed and then processed by a specific model to obtain a feature map; illustratively, step S102 may include:

[0092] Step 1.1), obtain the current image frame, and preprocess the current image frame to obtain a grayscale image.

[0093] The current image frame can be acquired through an acquisition device such as a camera, and a corresponding grayscale image can be obtained.

[0094] In step 1.2), the grayscale image is input into the backbone network, and the multi-scale feature map is output after being processed by the convolutional layer and pooling layer of the backbone network.

[0095] The backbone network is a SuperPoint architecture. It takes a grayscale image I as input and generates a feature map through multiple layers of convolution and pooling:

[0096] F=f backbone (I)

[0097] Among them, F is a multi-scale feature map.

[0098] In some embodiments, step S106 expresses line segments based on pixel points, and then extracts line features, which can be achieved by the following steps, specifically including:

[0099] Step 2.1), based on the processing of the line segment encoding branch in the line feature extraction branch, output a tensor of the second multi-scale feature map.

[0100] After processing by the segment coding branch above, the output is tensor; where d represents the vertical distance from the pixel point p in the attraction area of ​​the line segment l corresponding to each pixel point to the line segment l, θ is the line segment direction angle, that is, the angle between the line segment and the global coordinate system, and the two endpoints x1 and x2 of the line segment respectively form the angles θ1 and θ2 between the direction of the line connecting the pixel point p in the attraction area to the foot of the line segment l and the x-axis of the global coordinate system.

[0101] The region of attraction is a spatial region defined around a line segment, which is used to describe which pixel points p can be considered as points associated with the line segment. Its definition depends on the geometric information of the line segment (such as direction, length, and endpoint positions).

[0102] Step 2.2), determining the line segment code of each pixel in the second multi-scale feature map according to the tensor;

[0103] The line segments are encoded by the above tensor. Each line segment l = (x1, x2) is represented as:

[0104] p(l)=(d,θ,θ1,θ2)

[0105] Among them, the line segment code is composed of the vertical distance from each pixel to each pixel corresponding to the line segment, the angle between each pixel corresponding to the line segment and the global coordinate system, the angle between the foot point of the perpendicular generated by each pixel perpendicular to the line segment corresponding to each pixel and the first endpoint of the line segment and the x-axis of the global coordinate system, and the angle between the foot point of the perpendicular generated by each pixel perpendicular to the line segment corresponding to each pixel and the second endpoint of the line segment and the x-axis of the global coordinate system.

[0106] Step 2.3), decode the line segment code to obtain a line segment in which each pixel point is expressed in the form of endpoint parameters.

[0107] Here, the line segment after decoding the above line segment coding is expressed as follows:

[0108]

[0109] Among them, p is a pixel point. After matrix product calculation, the expression of l is (p1, p2).

[0110] After obtaining the two endpoints p1 and p2 of the line segment l in the above embodiment, the probability S of each pixel being an endpoint of the line segment is j Determine the two endpoints of the line segment, and then obtain the final representation of the line segment; step S106 also includes:

[0111] Step 3.1), based on the endpoint detection branch processing in the line feature extraction branch, output the probability that each pixel point in the second multi-scale feature map is a line segment endpoint;

[0112] Here, according to the processing of the endpoint detection branch, the probability S that each pixel point is a line segment endpoint is j .

[0113] Step 3.2), based on the probability that each pixel is the endpoint of the line segment, and the Euclidean distance between the endpoint parameters and each pixel, determine the target endpoint of the line segment corresponding to each pixel, and express the line segment corresponding to each pixel in the form of a target endpoint.

[0114] First, calculate the Euclidean distance between the endpoint parameters and each pixel as follows:

[0115]

[0116] The coordinates of the endpoint parameters p1 / p2 are (x j ,y j ), the coordinates of the target endpoint are (x line ,y line ), the Euclidean distance is d euclidean .

[0117] Secondly, based on the probability that each pixel is the endpoint of a line segment, as well as the Euclidean distance between the endpoint parameters and each pixel, a comprehensive scoring function S is established for each pixel. combine :

[0118]

[0119] Among them, α and β are weight parameters. For example, adjusting the weight parameters to α = 0.5, β = 0.5 represents the weight distribution of the Euclidean distance and the probability of the line segment endpoints; It is inversely proportional to the Euclidean distance, that is, the closer the Euclidean distance between the endpoint parameter and the pixel point, the higher the score.

[0120] The embodiment of the present invention is based on the comprehensive scoring function S combine The two points with the highest scores are used as the target endpoints of each line segment, and the line segment is expressed with the target endpoints.

[0121] As an optional embodiment, by effectively capturing the global information set of line features and improving the receptive field of global information, the accuracy of endpoint detection and line feature extraction is further improved. Before step 3.1), the method further includes:

[0122] In step 4.1), the second multi-scale feature map is input into the receptive field enhancement structure, first compressed in size by each convolution layer in the downsampling channel, and then processed by each deconvolution layer in the upsampling channel.

[0123] The downsampling channel and upsampling channel processing are performed in sequence according to the U-shaped channel style.

[0124] Step 4.2), add the first output result of each deconvolution layer and the second output result of the corresponding convolution layer, then concatenate the summed results of each sampling pair, and output a feature map result with the same scale as the second multi-scale feature map.

[0125] Among them, each convolution layer in the downsampling channel corresponds to each deconvolution layer in the upsampling channel to form a sampling pair; the downsampling channel includes q convolution layers, and the convolution layer a outputs the feature map b. The upsampling channel includes q deconvolution layers, and the deconvolution layer a outputs the feature map c. The feature map characteristics of the sampling pair corresponding to the convolution layer a after jump connection are b+c. The feature map features of the sampling pairs corresponding to the q convolution layers are spliced ​​to obtain a feature map result with the same scale as the second multi-scale feature map.

[0126] The inventor has found that in practical applications, for line features, most of the current visual odometers use the Line Band Descriptor (LBD) algorithm or the method of tracking sample points for matching. However, the LBD algorithm extracts descriptors from the local area of ​​the line segment. Therefore, when dealing with illumination changes or perspective changes, the line segment detection may be unstable, thereby affecting the matching effect. The method of tracking sample points can match line segments with different lengths in two frames, but the current SLAM system usually uses optical flow to track sample points. When the lighting conditions change rapidly or drastically, the performance of this method is poor. Some learning-based line feature descriptors have also been proposed, but due to the high computational cost, they are rarely used in actual SLAM systems. Therefore, in order to solve the effectiveness and efficiency problems, the embodiment of the present invention designs a fast and highly robust line feature matching method for coping with illumination challenges. First, the method establishes an association between key points and line segments through the distance between points and lines. The specific step S110 associates the target key points with the line segments expressed by the target endpoints corresponding to each pixel point, including:

[0127] Step 5.1), calculate the distance from each target key point to the current line segment expressed by the target endpoint;

[0128] If M target key points and N line segments are detected in the feature image, each target key point is represented by p i =(x i ,y i ), each line segment is represented by l j =(A j ,B j ,C j ,x j,1 ,y j,1 ,x j,2 ,y j,2 ), where (A j ,B j ,C j ) is line segment A j x+B j y+C j = 0, (x j,1 ,y j,1 ) and (x j,2 ,y j,2 ) are the two endpoints of the line segment.

[0129] The target key point p is calculated by the following formula i To line segment l j The distance d i,j :

[0130]

[0131] Step 5.2), compare the distance with the preset distance threshold ε, and determine whether the coordinates of the target key point are within the coordinate axis projection range of the current line segment, and determine each target key point belonging to the current line segment.

[0132] Here, if d i,j <ε point p i On line segment l j Within the coordinate axis projection range (i.e. satisfying min(x j,1 ,x j,2 )≤x i ≤max(x j,1 ,x j,2 ) or min(y j,1 ,y j,2 )≤y i ≤max(y j,1 ,y j,2 )), then the target key point p i is considered to belong to line segment l j .

[0133] Step 5.3), repeat the above steps 5.1)-step 5.2) until all the line segments corresponding to each pixel expressed by the target endpoints are traversed, that is, at this time, each line segment has determined the target key point belonging to the line segment.

[0134] On the basis of the above-mentioned embodiment, the line segments in the two frames of images are matched based on the matching results of the target key points associated with each line segment. k,m and line segment l in image frame k-1 k-1,n , this method calculates a score S mn To indicate the confidence that they are the same line segment: Exemplarily, step S110 further includes:

[0135] Step 6.1), based on the number of target key points associated with each line segment expressed by the target endpoint in the current image frame and the number of target key points associated with each line segment expressed by the target endpoint in the previous image frame, calculate the confidence level of successful matching between each line segment in the current image frame and each line segment in the previous image frame;

[0136]

[0137] Among them, N pm It belongs to l k,m and l k-1,n The sum of the number of successfully matched target key points, N k,m and N k-1,n They belong to l k,m and l k-1,nThe number of successfully matched target key points.

[0138] Step 6.2), if the confidence exceeds the confidence threshold, the line segment successfully matched between the current image frame and the previous image frame is the target line segment feature.

[0139] If S mn >δ S And N pm >δ N (where δ S and δ N are preset thresholds respectively), then it is considered that l k,m and l k+1,n This matching method based on point-line fusion not only improves the robustness of key point matching under illumination changes, but also avoids relying on the time-consuming line matching network.

[0140] The method of the embodiment of the present invention provides a solution for extracting point and line features at the same time, unifying the two and improving the overall performance of the method. Not only does it improve the efficiency and accuracy of line segment matching, but it also obtains point and line features through deep learning, and then uses a matching method based on point and line fusion to improve the robustness of key point matching under illumination changes and avoid relying on a line segment matching network with a high time cost.

[0141] In some embodiments, Figure 2 As shown, the embodiment of the present invention further provides a visual synchronous positioning and mapping device 200 integrating point and line features, the device comprising:

[0142] An acquisition module 201 is configured to acquire a multi-scale feature map of a current image frame; wherein the multi-scale feature map includes a first scale feature map and a second scale feature map;

[0143] A first extraction module 202 inputs the first scale feature map into a key point extraction branch to obtain a score map for characterizing the distribution of key points in the first scale feature map and a descriptor for distinguishing each key point;

[0144] The second extraction module 203 inputs the second multi-scale feature map into the line feature extraction branch, obtains the encoded line segment corresponding to each pixel point in the second multi-scale feature map, and the probability that each pixel point is an endpoint of the line segment, and determines the line segment expressed by the target endpoint corresponding to each pixel point;

[0145] A first matching module 204 matches the key points of the current image frame with the key points of the previous image frame according to the score map and the descriptor, and determines a successfully matched target key point;

[0146] The second matching module 205 associates the target key point with the line segment expressed by the target endpoint for each pixel point, and matches the line segment expressed by the target endpoint for each pixel point of the current image frame with the line segment expressed by the target endpoint for each pixel point of the previous image frame, and determines the target line segment features that are successfully matched;

[0147] The positioning and mapping module 206 controls the current vehicle to perform synchronous positioning and mapping according to the target key points and the target line segment features.

[0148] Further, the second extraction module 203 is specifically used to output a tensor of the second multi-scale feature map based on the processing of the line segment coding branch in the line feature extraction branch; determine the line segment coding of each pixel in the second multi-scale feature map according to the tensor; wherein the line segment coding is composed of the vertical distance from each pixel to the line segment corresponding to each pixel, the angle between the line segment corresponding to each pixel and the global coordinate system, the angle between the foot point of the perpendicular generated by each pixel perpendicular to the line segment corresponding to each pixel and the first endpoint of the line segment and the x-axis of the global coordinate system, and the angle between the foot point of the perpendicular generated by each pixel perpendicular to the line segment corresponding to each pixel and the second endpoint of the line segment and the x-axis of the global coordinate system; the line segment coding is decoded to obtain the line segment of each pixel expressed in the form of endpoint parameters.

[0149] Furthermore, the second extraction module 203 is specifically used to output the probability that each pixel point in the second multi-scale feature map is a line segment endpoint based on the endpoint detection branch processing in the line feature extraction branch; determine the target endpoint of the line segment corresponding to each pixel point according to the probability that each pixel point is a line segment endpoint and the Euclidean distance between the endpoint parameter and each pixel point, and express the line segment corresponding to each pixel point in the form of the target endpoint.

[0150] In some embodiments, before the step of inputting the second multi-scale feature map into the line feature extraction branch, the device is also used to input the second multi-scale feature map into the receptive field improvement structure, first compress the size of each convolution layer in the downsampling channel, and then process it through each deconvolution layer in the upsampling channel; wherein each convolution layer in the downsampling channel corresponds to each deconvolution layer in the upsampling channel to form a sampling pair; the first output result of each deconvolution layer and the second output result of the corresponding convolution layer are added, and then the summed results of each sampling pair are spliced ​​to output a feature map result of the same scale as the second multi-scale feature map.

[0151] Furthermore, the second matching module 205 is specifically used to calculate the distance from each of the target key points to the current line segment expressed by the target endpoint; compare the distance with a preset distance threshold, and determine whether the coordinates of the target key point are within the projection range of the coordinate axis of the current line segment, and determine each of the target key points belonging to the current line segment; repeat the above steps until the line segment expressed by the target endpoint for each pixel point has been traversed.

[0152] Furthermore, the second matching module 205 is specifically used to calculate the confidence of the successful match between each line segment in the current image frame and each line segment in the previous image frame based on the number of target key points associated with each line segment expressed by the target endpoint in the current image frame and the number of target key points associated with each line segment expressed by the target endpoint in the previous image frame; if the confidence exceeds the confidence threshold, the line segment that successfully matches the current image frame and the previous image frame is the target line segment feature.

[0153] Furthermore, the acquisition module 201 is specifically used to obtain a current image frame, preprocess the current image frame to obtain a grayscale image; input the grayscale image into the backbone network, and output a multi-scale feature map through the convolution layer and pooling layer of the backbone network.

[0154] Figure 3 Schematic diagram of the hardware architecture of the electronic device 300 provided in an embodiment of the present invention. Figure 3 As shown, the electronic device 300 includes: a machine-readable storage medium 301 and a processor 302, and may also include a non-volatile storage medium 303, a communication interface 304 and a bus 305; wherein the machine-readable storage medium 301, the processor 302, the non-volatile storage medium 303 and the communication interface 304 complete mutual communication through the bus 305. The processor 302 can execute the method for visual synchronous positioning and mapping of fused point and line features described in the above embodiment by reading and executing the machine executable instructions for visual synchronous positioning and mapping of fused point and line features in the machine-readable storage medium 301.

[0155] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0156] The non-volatile medium may be a non-volatile memory, a flash memory, a storage drive (such as a hard drive), any type of storage disk (such as a CD, DVD, etc.), or a similar non-volatile storage medium, or a combination thereof.

[0157] It can be understood that the specific operation methods of each functional module in this embodiment can refer to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0158] The computer-readable storage medium provided in the embodiment of the present invention stores a computer program. When the computer program code is executed, the visual synchronous positioning and mapping method for integrating point and line features described in any of the above embodiments can be implemented. For specific implementation, please refer to the method embodiment, which will not be repeated here.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0160] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0161] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0162] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the aforementioned embodiments, those of ordinary skill in the art should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed by the present invention, or can easily conceive of changes, or make equivalent replacements for some of the technical features therein. Such modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention.

Claims

1. A visual synchronous positioning and mapping method integrating point and line features, characterized in that: The method comprises: Acquire a multi-scale feature map of the current image frame; wherein the multi-scale feature map includes a first scale feature map and a second scale feature map; Inputting the first scale feature map into a key point extraction branch to obtain a score map for characterizing the distribution of key points in the first scale feature map and a descriptor for distinguishing each key point; Inputting the second multi-scale feature map into the line feature extraction branch, obtaining the encoded line segment corresponding to each pixel in the second multi-scale feature map, and the probability that each pixel is an endpoint of the line segment, and determining the line segment expressed by the target endpoint corresponding to each pixel; According to the score map and the descriptor, the key points of the current image frame are matched with the key points of the previous image frame to determine the target key points that are successfully matched; Associating the target key point with the line segment expressed by the target endpoint for each pixel point, and matching the line segment expressed by the target endpoint for each pixel point of the current image frame with the line segment expressed by the target endpoint for each pixel point of the previous image frame, and determining the target line segment features that are successfully matched; According to the target key points and the target line segment features, the current vehicle is controlled to perform synchronous positioning and mapping.

2. The method according to claim 1, characterized in that The step of inputting the second multi-scale feature map into the line feature extraction branch, obtaining the encoded line segment corresponding to each pixel in the second multi-scale feature map, and the probability that each pixel is an endpoint of the line segment, and determining the line segment expressed as the target endpoint corresponding to each pixel, includes: Outputting a tensor of the second multi-scale feature map based on the processing of the line segment encoding branch in the line feature extraction branch; Determine, according to the tensor, a line segment code for each pixel in the second multi-scale feature map; The line segment code is composed of the vertical distance from each pixel to the line segment corresponding to each pixel, the angle between the line segment corresponding to each pixel and the global coordinate system, the angle between the line connecting the foot point of the line segment corresponding to each pixel and the first endpoint of the line segment and the x-axis of the global coordinate system, and the angle between the line connecting the foot point of the line segment corresponding to each pixel and the second endpoint of the line segment and the x-axis of the global coordinate system; The line segment code is decoded to obtain a line segment in which each pixel point is expressed in the form of endpoint parameters.

3. The method according to claim 2, characterized in that The step of inputting the second multi-scale feature map into the line feature extraction branch, obtaining the encoded line segment corresponding to each pixel in the second multi-scale feature map, and the probability that each pixel is an endpoint of the line segment, and determining the line segment expressed as the target endpoint corresponding to each pixel, further includes: Outputting the probability that each pixel point in the second multi-scale feature map is an endpoint of a line segment based on the endpoint detection branch in the line feature extraction branch; According to the probability that each pixel is a line segment endpoint and the Euclidean distance between the endpoint parameter and each pixel, the target endpoint of the line segment corresponding to each pixel is determined, and the line segment corresponding to each pixel is expressed in the form of the target endpoint.

4. The method according to claim 3, characterized in that Before the step of processing the endpoint detection branch in the line feature extraction branch, the method further includes: Input the second multi-scale feature map into the receptive field enhancement structure, firstly compress the size of the feature map by each convolution layer in the downsampling channel, and then process the feature map by each deconvolution layer in the upsampling channel; wherein each convolution layer in the downsampling channel corresponds to each deconvolution layer in the upsampling channel to form a sampling pair; The first output result of each deconvolution layer and the second output result of the corresponding convolution layer are added, and then the summed results of each sampling pair are concatenated to output a feature map result of the same scale as the second multi-scale feature map.

5. The method according to claim 1, characterized in that The step of associating the target key point with each pixel point to express a line segment with the target endpoint comprises: Calculating the distance from each of the target key points to the current line segment expressed by the target endpoint; Compare the distance with a preset distance threshold, and determine whether the coordinates of the target key point are within the coordinate axis projection range of the current line segment, and determine each of the target key points belonging to the current line segment; The above steps are repeated until all line segments expressed by the target endpoints of each pixel point have been traversed.

6. The method according to claim 5, characterized in that The step of matching a line segment expressed by each pixel point of the current image frame with the target endpoint with a line segment expressed by each pixel point of the previous image frame with the target endpoint, and determining the features of the successfully matched target line segment comprises: Calculating the confidence level of successful matching between each line segment in the current image frame and each line segment in the previous image frame based on the number of target key points associated with each line segment expressed by the target endpoint in the current image frame and the number of target key points associated with each line segment expressed by the target endpoint in the previous image frame; If the confidence exceeds the confidence threshold, the line segment successfully matched between the current image frame and the previous image frame is the target line segment feature.

7. The method according to claim 1, characterized in that The steps of obtaining a multi-scale feature map of the current image frame include: Acquire a current image frame, and preprocess the current image frame to obtain a grayscale image; The grayscale image is input into the backbone network, and is processed by the convolution layer and the pooling layer of the backbone network to output a multi-scale feature map.

8. A visual synchronous positioning and mapping device integrating point and line features, characterized in that: The device comprises: An acquisition module is used to acquire a multi-scale feature map of the current image frame; wherein the multi-scale feature map includes a first scale feature map and a second scale feature map; A first extraction module, inputting the first scale feature map into a key point extraction branch, to obtain a score map for characterizing the distribution of key points in the first scale feature map and a descriptor for distinguishing each key point; A second extraction module inputs the second multi-scale feature map into a line feature extraction branch to obtain a line segment encoded corresponding to each pixel in the second multi-scale feature map, a probability that each pixel is an endpoint of the line segment, and determines a line segment expressed as a target endpoint corresponding to each pixel; A first matching module matches the key points of the current image frame with the key points of the previous image frame according to the score map and the descriptor, and determines the target key points that are successfully matched; A second matching module associates the target key point with a line segment expressed by the target endpoint for each pixel point, and matches the line segment expressed by the target endpoint for each pixel point of the current image frame with the line segment expressed by the target endpoint for each pixel point of the previous image frame, to determine the target line segment features that are successfully matched; The positioning and mapping module controls the current vehicle to perform synchronous positioning and mapping according to the target key points and the target line segment features.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method described in any one of claims 1 to 7.

Images