A visual feature point extraction and matching optimization method based on FCN and BP neural network

Through FCN and BP neural network optimization of feature point extraction and matching in SLAM algorithm, the problems of uneven distribution and mismatch of image feature extraction are solved, the calculation efficiency and positioning accuracy are improved, and more uniform and reliable feature point distribution is achieved.

CN114723964BActive Publication Date: 2024-12-03SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210443753.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-26
Publication Date
2024-12-03
Estimated Expiration
2042-04-26

AI Technical Summary

Technical Problem

Existing SLAM algorithms are prone to uneven distribution, over-extraction and mismatch problems when extracting image features, resulting in low computational efficiency and inability to ensure real-time performance.

Method used

The visual feature point extraction and matching optimization method based on FCN and BP neural networks is adopted. The FCN neural network is constructed and trained for semantic segmentation and pixel culling. The BP neural network is constructed in combination with image grayscale symbiosis matrix analysis to fit the number and distribution of feature points The nonlinear relationship of the feature pose offset stability optimization matching results can be used to reduce unnecessary feature point extraction and descriptor calculation.

Benefits of technology

The calculation efficiency of feature point extraction and matching is improved, the problem of excessive concentration and high repetition of feature point extraction is avoided, the stability and accuracy of the matching process is enhanced, and more accurate positioning and mapping data are provided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114723964B_ABST
    Figure CN114723964B_ABST
Patent Text Reader

Abstract

The invention discloses a visual feature point extraction and matching optimization method based on FCN and BP neural network, comprising step S1: constructing and training FCN neural network, performing semantic segmentation and pixel culling on dynamic objects in image frames; step S2: analyzing the image grayscale co-occurrence matrix, and constructing training set data of BP neural network in combination with the contrast threshold in ORB feature extraction algorithm, so as to fit the number of feature points, the image grayscale co-occurrence matrix, and the nonlinear relationship between the image contrast and the contrast threshold, so as to control the number and distribution of image feature points; step S3: optimizing the feature point matching result by utilizing the feature posture offset stability, and controlling the pixel position range of the matching feature points in the image; and step S4: transplanting the trained FCN and BP neural network into the algorithm framework, accelerating the feature point extraction and matching process, reducing unnecessary feature point extraction and descriptor calculation in the feature extraction process, and realizing the improvement of the calculation efficiency of ORB-SLAM algorithm.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of visual SLAM based on a feature point method, and in particular relates to a visual feature point extraction and matching optimization method based on FCN and BP neural network. Background Art

[0002] Feature extraction and matching is a key technology in the SLAM system based on the feature point method. The features of an image refer to some representative pixels in the image, which describe the information of the pixels around the pixel in a certain way. Feature points are composed of key points and descriptors. In the history of visual feature development, SIFT (Scale-invariant feature transform) and SURF (Speed ​​up robust feature) are both commonly used features. In recent years, ORB feature points have occupied a place in the ORB-SLAM series of classic SLAM algorithms due to their advantages in scale invariance, rotation invariance, insensitivity to light changes, and simple calculation. At present, the ORB-SLAM algorithm has achieved the level of real-time operation in conventional configuration computers, but the main computer computing power is used for the extraction and matching of image ORB feature points, and the functions realized are only the establishment and positioning of sparse feature point maps, which are prone to uneven distribution of image feature extraction, over-extraction, or mismatching in SLAM feature matching. The ORB-SLAM algorithm has low computational efficiency; and if other functions are to be added, the real-time operation of the SLAM algorithm cannot be guaranteed. Summary of the invention

[0003] The present invention aims to solve the problems of uneven distribution, over-extraction and mismatch in SLAM feature matching in the prior art, and provides a visual feature point extraction and matching optimization method based on FCN and BP neural network, which comprehensively considers the influence of image contrast, image texture complexity and image grayscale co-occurrence matrix, and solves the problem that when extracting ORB feature points from image frames in a SLAM (Simultaneous Localization and Mapping) system (ORBSLAM for short) which takes ORB feature points as the main algorithm processing object, the extracted feature points are dense but the utilization rate is low. The method comprises the steps of: constructing and training an FCN neural network, performing semantic segmentation and pixel culling on dynamic objects in an image frame, and using only the pixel range from which the dynamic objects have been culled as the operation area of ​​a feature detection algorithm; step S2: analyzing the image grayscale co-occurrence matrix, and constructing training set data of a BP neural network in combination with a contrast threshold in an ORB feature extraction algorithm, so as to fit the number of feature points, the image grayscale co-occurrence matrix, and the nonlinear relationship between the image contrast and the contrast threshold, thereby realizing control over the number and distribution of image feature points; step S3: optimizing the feature point matching result by utilizing the stability of feature posture offset, and controlling the pixel position range of the matching feature points in the image; and step S4: transplanting the trained FCN and BP neural networks into an algorithm framework, accelerating the extraction and matching process of feature points, reducing unnecessary feature point extraction and descriptor calculation in the feature extraction process, and realizing improvement in the calculation efficiency of the ORB-SLAM algorithm.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is: a visual feature point extraction and matching optimization method based on FCN and BP neural network, comprising the following steps:

[0005] S1, construct and train an FCN neural network: the FCN neural network comprises five convolutional layers, five pooling layers, three fully connected layers and three upsampling layers, and uses the output results of the second and third pooling layers as inputs for upsampling. For the input image sequence, the dynamic objects in the image frames are semantically segmented and pixel culled, and only the pixel range where the dynamic objects are culled is used as the operation area of ​​the feature detection algorithm.

[0006] S2, control the number and distribution of image feature points: analyze the image grayscale co-occurrence matrix, and construct the training set data of the BP neural network in combination with the contrast threshold in the ORB feature extraction algorithm to fit the nonlinear relationship between the number of feature points, the image grayscale co-occurrence matrix, the image contrast and the contrast threshold, and realize the control of the number and distribution of image feature points. The specific definition is as follows:

[0007] Th ζ =f(ψ,ζ,s)

[0008] Where: Th ζ represents the contrast threshold; ψ represents the image complexity; ζ represents the image contrast; s represents the given range of the number of feature points;

[0009] The image contrast ζ is:

[0010]

[0011] Where: ζ represents the contrast value of a frame of image; w represents the total number of pixels in the image; x i Represents the gray value of the i-th pixel; Represents the average gray value of image pixels;

[0012] S3, result optimization: optimize the feature point matching results by using the feature pose offset stability, retain the matching points that satisfy the following equations at the same time, and control the pixel position range of the matching feature points in the image;

[0013]

[0014] Where: M is the set of matching points of two frames of images, with a total of n pairs; Th r Indicates the constraint range; (x i ,y i ) and (x′ i ,y′ i ) represent the pixel coordinates of a pair of matching points in the two frames of images; l and m represent the number of rows and columns of image pixels respectively;

[0015] S4, execution: adding the constructed FCN and BP neural networks to the SLAM algorithm tracking module with ORB feature points as the main tracking object for execution.

[0016] As an improvement of the present invention, in step S2: the image complexity ψ is determined by three indicators: information entropy ξ, correlation χ and energy ε of the image, specifically:

[0017]

[0018] Where k represents the number of gray levels of the image, n l refers to the total number of the lth gray level, N represents the total number of pixels in the image; p(i,j) represents the (i,j)th value in the gray level co-occurrence matrix;

[0019] in,

[0020]

[0021]

[0022]

[0023] As an improvement of the present invention, in step S2, the hidden layer of the BP neural network is set with 9 nodes in total, the output is set with 9 nodes, the contrast threshold is represented by binary (0, 1), the training error accuracy is set to 0.251, and all inputs are normalized.

[0024] As another improvement of the present invention, in step S3, the constraint range Th r Set it according to the image pixel size, taking 10% of the image diagonal pixel length.

[0025] Compared with the prior art, the advantages of the present invention are:

[0026] (1) The present invention uses a trained FCN neural network to optimize the pixel range of feature point extraction, thereby avoiding the problems of over-concentration, high repetitiveness and uneven distribution of ORB feature extraction in traditional ORB extraction methods;

[0027] (2) The present invention adopts a trained BP neural network, combines the complexity of the image (using the gray-level co-occurrence matrix of the image to analyze the information entropy, correlation and energy of the image), contrast and demand to form an adaptive module in the feature extraction algorithm. According to the information of the image and the application requirements, the contrast threshold is adjusted adaptively to make the detection result meet the expectations, thereby improving the stability of the feature points in the matching process;

[0028] (3) The feature point optimization algorithm based on the stability of posture offset adopted by the present invention can effectively eliminate mismatches, improve the accuracy of the results, provide accurate raw data for the positioning and mapping process of SLAM, and improve the positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a comparison chart of the ORB feature extraction of the same frame image in the Opencv open source built-in ORB feature extraction algorithm and ORBSLAM in the stable tracking process;

[0030] Figure 2 It is a schematic diagram of the specific process of adding the algorithm in step S4 of the present invention;

[0031] Figure 3 It is a schematic diagram of the FCN neural network structure;

[0032] Figure 4 It is a schematic diagram of the BP neural network structure;

[0033] Figure 5 This is the schematic diagram of epipolar geometry;

[0034] Figure 6 This is the schematic diagram of camera pose offset stability. DETAILED DESCRIPTION

[0035] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0036] Example 1

[0037] A visual feature point extraction and matching optimization method based on FCN and BP neural network includes the following steps:

[0038] S1, construct and train FCN neural network: for the input image sequence, the FCN neural network is constructed and trained with the goal of improving the stability of ORB feature points during tracking. Its main function is to perform semantic segmentation and pixel culling on dynamic objects in the image frame, and only use the pixel range where dynamic objects are culled as the operation area of ​​the feature detection algorithm, reducing the impact of dynamic objects on the SLAM tracking process.

[0039] The image sequence input is the visual sensor data input when the SLAM system is running; the designed FCN network is as follows Figure 3 As shown in the figure, it contains five convolutional layers, five pooling layers, three fully connected layers and three upsampling layers. The output results of the second and third pooling layers are used as the input of the upsampling process to make up for the pixel position information lost in the continuous downsampling process and improve the segmentation accuracy. The function of the finally trained FCN neural network is to perform semantic segmentation on dynamic objects in the image and provide the algorithm framework with the ORB feature extraction pixel range in the image (other areas except dynamic objects);

[0040] S2, control the number and distribution of image feature points: analyze the image grayscale co-occurrence matrix, and construct the training set data of the BP neural network in combination with the contrast threshold in the ORB feature extraction algorithm to fit the nonlinear relationship between the number of feature points, the image grayscale co-occurrence matrix and the image contrast threshold, so as to achieve adaptive control of the distribution and number of feature points taken for each frame of the image;

[0041] The specific construction and training process is as follows:

[0042] The contrast threshold is a key parameter to improve the stability of feature points, and different image contrasts will have a great impact on the number of feature points. In order to quantitatively analyze the image contrast, the root mean square is used for calculation, as shown in the following formula:

[0043]

[0044] Among them: ζ quantitatively represents the contrast of a frame of image; n represents the number of pixels; x i Represents the gray value of the i-th pixel; Represents the average gray value of image pixels;

[0045] Pixel complexity is a key factor affecting the number of feature points. It is a description of the inherent complexity of the image and the aggregation of all image information. In order to obtain quantitative image complexity, the gray-level co-occurrence matrix is ​​used to analyze important information such as information entropy (ξ), correlation degree (χ), and energy (ε). The specific formula is as follows:

[0046]

[0047] Where: k represents the number of gray levels of the image; n l Refers to the total number of the lth gray level; N represents the total number of pixels in the image; p(i,j) represents the (i,j)th value in the gray level co-occurrence matrix;

[0048]

[0049]

[0050]

[0051]

[0052] The three indicators of information entropy, correlation and energy of the comprehensive image represent the complexity of a frame of image, which is represented by the symbol ψ;

[0053] A mathematical model between the number of feature points, image complexity, image contrast and contrast threshold is constructed to achieve adaptive control of the distribution and number of feature points taken for each frame of image. The definition is as follows:

[0054] Th ζ =f(ψ,ζ,n)

[0055] Where: Th ζ represents the contrast threshold; ψ represents the image complexity; ζ represents the image contrast; n represents the range of the number of feature points;

[0056] For a given range of feature point numbers, the model automatically matches the appropriate contrast threshold to make the actual number of feature points consistent with the requirements;

[0057] The BP neural network is used to fit the model described above, such as Figure 4As shown in the figure, the BP neural network is a multi-layer forward neural network trained according to the error back propagation algorithm. By calculating the error between the expected output and the actual output, the parameters and thresholds of the nodes are adjusted layer by layer from the output to the input, and the target accuracy is finally achieved after repeated iterations. For each image in the input image sequence, an analysis result with complexity, contrast and number of feature points as output is constructed, and the contrast threshold is obtained through repeated linear iterations. The complexity, contrast and number of required feature points are used as the input of the BP neural network, and the contrast threshold is used as the output;

[0058] A BP neural network is constructed to fit the mathematical model. The hidden layer of the neural network has 9 nodes in total to improve the generalization ability. The output has 9 nodes, and binary (0,1) is used to represent the contrast threshold. To ensure the accuracy of the neural network, the training error accuracy is set to 0.251, and all inputs are normalized to eliminate the dimensional effect. After iteration, the neural network reaches the target accuracy and the training is completed.

[0059] The trained BP neural network is used as an adaptive module in the algorithm framework to adaptively adjust the contrast threshold according to the image information and application requirements, so that the final ORB feature point detection result meets expectations.

[0060] S3, result optimization: Based on the optimization of feature point matching with stable posture offset, suppose there are n pairs of matching points in total. On the basis of the optimization of the traditional KNN (K Nearest Neighbors) algorithm, the matching points that simultaneously satisfy the following formula are retained, that is, the positions of the corresponding matching points on the second frame image should be constrained within a certain range with the positions of the matching points on the first frame image. In this way, the accurate matching points can be obtained, and the posture transformation R, t between the two frames of images can be solved according to the epipolar geometry principle;

[0061]

[0062] Where: M is the set of matching points of two frames of images; Th r The constraint range is mainly set according to the image pixel size, usually 10% of the image diagonal pixel length; (x i ,y i ) and (x′ i ,y′ i ) represent the pixel coordinates of a pair of matching points in the two frames of images; n and m represent the number of rows and columns of image pixels respectively;

[0063] like Figure 5As shown, epipolar geometry describes the intrinsic projective relationship between two frames of images, which is independent of the external scene and only depends on the camera's internal parameters and the relative posture between the two frames of images. For a point P in three-dimensional space, the camera obtains images I1 and I2 at the optical center positions O1 and O2 respectively. The pixels of P in the two frames of images are p1 and p2 respectively. The optical center line O1O2 is called the baseline, and its intersection points e1 and e2 with the pixel plane are called poles. The lines l1 and l2 connecting p1, p2 and e1, e2 are called epipolar lines, and the position of O2 is obtained by spatial transformation R, t from O1.

[0064] The principle of posture offset stability is as follows Figure 6 As shown in Figure 1, due to the principle of feature point triangulation, the matching of two image frames requires a certain displacement, and the displacement should not be too small. However, too large a displacement will cause the disappearance of the corresponding matching features or the change of the scene appearance. Figure 6 In, Δ e1 and Δ e2 Respectively represent the same matching error angle Δ θ , the error between the restored feature point 3D position result P′ and the true result P under different camera displacements t1 and t2, O1 and O2 represent the camera optical center, combined with Figure 5 From the intrinsic projective relationship between the two frames of images in , it can be seen that within the appropriate inter-frame displacement range, the pixel positions of the corresponding matching feature points should not change too much.

[0065] S4: Add the constructed FCN and BP neural networks to the SLAM algorithm tracking module with ORB feature points as the main tracking object.

[0066] Specific joining method reference Figure 2 The algorithm framework diagram is as follows:

[0067] Step 1: Read the image sequence information released by the camera;

[0068] Step 2: Complete the image contrast analysis and gray level co-occurrence matrix analysis;

[0069] Step 3: Use the image analysis results of step 2 and the number of feature points (manually set) required for tracking by the SLAM system as input to the adaptive control module based on the BP neural network;

[0070] Step 4: adjusting the dynamic contrast threshold of the image according to the output of the adaptive control module in step 3;

[0071] Step 5: Step 5 is performed synchronously with steps 2, 3, and 4. The image sequence is input into the trained FCN neural network to obtain the main target pixel range of the image after dynamic objects are removed. Its function is to limit the area of ​​ORB feature point detection to avoid feature mismatching introduced by dynamic objects.

[0072] Step 6: Detect and extract feature points, using the main target pixel range output in step 5 and the dynamic contrast threshold output in step 4 as a reference for the feature point detection and extraction process;

[0073] Step 7: Determine whether the number of feature points extracted in step 6 meets the minimum requirement for SLAM tracking. If not, feedback is given to the adaptive control module in step 3 to adjust the dynamic contrast threshold. If yes, continue to proceed.

[0074] Step 8: Perform feature point matching algorithm based on KNN;

[0075] Step 9: Based on step 8, a matching optimization method based on posture offset stability is used to eliminate incorrect matches;

[0076] Step 10: Based on the feature matching results of step 9, the epipolar geometry calculation is performed to solve the pose transformation between the two image frames before and after. Due to the existence of errors, the feature matching results will not be able to obtain a unique solution. Graph optimization is used to construct the least squares problem between the pixel coordinates of the feature points and the camera pose. The pose transformation between the image frame coordinate systems corresponding to the existing feature point matching results is solved through nonlinear optimization, and finally the pose of the current image frame is obtained.

[0077] The method of the present invention is compared with the execution of the KNN algorithm. Five image feature point matching implementations are carried out for the two methods. The time consumption results are compared as shown in the following table:

[0078]

[0079] It can be seen from the above table that although the time consumption of the method proposed in the present invention is increased by 5.3% compared with the KNN algorithm, the main reason is that the process of feature point pixel position calculation is added, but because the added process is linear calculation, the overall time consumption is still basically the same as the KNN algorithm.

[0080] like Figure 1 As shown in the figure, the ORB feature extraction algorithm of the same frame image in the Opencv open source and ORBSLAM in the stable tracking process are compared. Figure 1The black circle in the upper left picture shows the OpenCV extraction result. The feature points are extracted densely and repeatedly. The square mark in the upper right picture shows the distribution of feature points that can be stably tracked by ORB. It can be seen from the figure that it is evenly distributed and highly recognizable. The figure below is the original image used to extract feature points.

[0081] In summary, this case provides a visual feature point extraction and matching optimization method based on FCN and BP neural network, which comprehensively considers the influence of image contrast, image texture complexity, and image grayscale co-occurrence matrix, and solves the problem that the SLAM (Simultaneous Localization and Mapping) system (ORBSLAM for short) which uses ORB (Oriented Fast and Rotated Brief) feature points as the main algorithm processing object extracts dense feature points but has low utilization rate when extracting ORB feature points from image frames. It accelerates the feature point extraction and matching process, reduces unnecessary feature point extraction and descriptor calculation in the feature extraction process, and improves the computational efficiency of the ORB-SLAM algorithm.

[0082] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A visual feature point extraction and matching optimization method based on FCN and BP neural network, characterized in that: It includes the following steps: S1, construct and train an FCN neural network: the FCN neural network comprises five convolutional layers, five pooling layers, three fully connected layers and three upsampling layers, and uses the output results of the second and third pooling layers as inputs for upsampling. For the input image sequence, the dynamic objects in the image frames are semantically segmented and pixel culled, and only the pixel range where the dynamic objects are culled is used as the operation area of ​​the feature detection algorithm. S2, control the number and distribution of image feature points: analyze the image grayscale co-occurrence matrix, and construct the training set data of the BP neural network in combination with the contrast threshold in the ORB feature extraction algorithm to fit the nonlinear relationship between the number of feature points, the image grayscale co-occurrence matrix, the image contrast and the contrast threshold, and realize the control of the number and distribution of image feature points. The specific definition is as follows: Th ζ =f(ψ,ζ,s) Where: Th ζ represents the contrast threshold; ψ represents the image complexity; ζ represents the image contrast; s represents the given range of the number of feature points; The image contrast ζ is: Where: ζ represents the contrast value of a frame of image; w represents the total number of image pixels; x i Represents the gray value of the i-th pixel; Represents the average gray value of image pixels; S3, result optimization: optimize the feature point matching results by using the feature pose offset stability, retain the matching points that satisfy the following equations at the same time, and control the pixel position range of the matching feature points in the image; Where: M is the set of matching points of two frames of images, there are n pairs in total; Th r Indicates the constraint range; (x i ,y i ) and (x′ i ,y′ i ) represent the pixel coordinates of a pair of matching points in the two frames of images; l and m represent the number of rows and columns of image pixels respectively; S4, execution: adding the constructed FCN and BP neural network to the SLAM algorithm tracking module with ORB feature points as the tracking object for execution.

2. The method for extracting and matching visual feature points based on FCN and BP neural network as claimed in claim 1, characterized in that: In step S2, the image complexity ψ is determined by three indicators: information entropy ξ, correlation χ and energy ε of the image, specifically: Where k represents the number of gray levels of the image, n l refers to the total number of the lth gray level, N represents the total number of pixels in the image; p(i,j) represents the (i,j)th value in the gray level co-occurrence matrix; in, 3. A method for extracting and matching visual feature points based on FCN and BP neural network as claimed in claim 1 or 2, characterized in that: In step S2, the hidden layer of the BP neural network is set with 9 nodes in total, the output is set with 9 nodes, the contrast threshold is represented by binary (0, 1), the training error accuracy is set to 0.251, and all inputs are normalized.

4. The method for extracting and matching visual feature points based on FCN and BP neural network as claimed in claim 3, characterized in that: In step S3, the constraint range Th r Set it according to the image pixel size, taking 10% of the image diagonal pixel length.