A method for ORB feature extraction and matching based on polarization information

Abstract

The present invention provides an ORB feature extraction and matching method based on polarization information, comprising: using a polarization image sensor to collect original image frame data, and calculating a grayscale image, a polarization degree image, and a polarization direction image respectively; extracting FAST key points in five modes, with the largest number of them being the extraction result; calculating the polarization degree value and polarization direction value of the key points; calculating the FAST key point direction in three modes; and calculating the FAST key point descriptor in two modes to obtain six modes of ORB feature points. A polarization degree threshold is defined to assist in ORB feature point matching, with the largest number of matches in the six modes being the matching result. The present invention utilizes a polarization image sensor to obtain scene polarization information, and by optimizing parameter combinations, improves the number of ORB feature points extracted and matched, the matching accuracy, the SLAM accuracy, and the environmental adaptability in multiple scenarios, which is of great significance to the fields of autonomous driving, smart transportation, and the Internet of Things.

Description

Technical Field

[0001] The present invention relates to the field of feature extraction and matching in a visual SLAM system, and in particular to an ORB (Oriented FAST and Rotated BRIEF) feature extraction and matching method based on polarization information. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology is primarily used to solve the problem of positioning and mapping robots when moving in unknown environments. SLAM refers to a mobile robot equipped with specific sensors that, without prior information about the environment, builds a model of the environment while in motion and simultaneously estimates the movement of the mobile robot. Camera sensors have the advantages of being small, low cost, easy to deploy, low power, and able to provide rich information. However, due to their limited field of view, susceptibility to lighting, and inability to directly obtain depth information, vision-based positioning and mapping methods are more complex than those using other sensors. Therefore, visual SLAM has always been an important research direction in the field of SLAM.

[0003] Visual SLAM algorithms can be divided into feature-based and direct methods based on the image information they utilize. Feature-based methods rely on scene feature information, such as point features, line features, and surface features, with point features being the primary approach. Compared to more stable, precise, and robust image feature points such as SIFT (Scale-invariant Feature Transform) and SURF (Speeded Up Robust Features), ORB (OrientedFAST and RotatedBRIEF) feature points require less computation and are very fast, meeting the requirements of real-time calculation and matching. They offer a good compromise between quality and performance, with ORB-SLAM being a representative example. Classic visual SLAM frameworks are already capable of real-time positioning and mapping in normal working environments. However, their SLAM performance is often limited in scenes with low texture, reflective surfaces, and complex lighting conditions.

[0004] Polarized light, formed by atmospheric scattering of sunlight, is a natural property of Earth's atmosphere. In 1871, British physicist Rayleigh established the Rayleigh scattering theory, which has guided the study of polarization distribution in the sky. Leveraging the stability and symmetry of atmospheric polarization patterns, polarized light navigation is not only applicable to high-latitude regions where magnetic compasses are difficult to use for orientation, but also serves as a valuable supplement for autonomous robot navigation and positioning in unfamiliar environments, such as those without satellite navigation signals. It also offers the advantage of stable error accumulation over time. Currently, polarization-based SLAM or navigation methods focus on studying atmospheric polarization patterns, but underutilize scene polarization information. Polarization information, as a higher-dimensional form than flat images, can characterize the physical and chemical properties of targets. In 1981, Walraven studied the interpretation of polarization information from natural and artificial objects, demonstrating that polarization information can represent new information about the object relative to the intensity image.

[0005] The performance of visual SLAM based on ORB features is mainly related to the feature extraction and matching effects, and depends on the number of feature points extracted, the number of matches, and the matching accuracy. In low-texture, weak-texture, and sparse conditions, the number of feature points extracted is small, and the number of matches and the matching accuracy are also low. There are a large number of mismatches, which seriously affect the SLAM effect. Therefore, another dimension of information relative to the scene grayscale image information is introduced - scene polarization information, to assist in the extraction and matching of feature points, increase the number and accuracy of matches, and thereby improve the accuracy of visual SLAM and its adaptability to various scenes.

[0006] In an invention patent application published on October 19, 2016, with publication number CN106033599A, a visible light image enhancement method based on polarization imaging is disclosed. The method first uses polarization imaging to acquire polarization information from the image, then clusters it to generate a polarization feature image. Finally, the polarization feature image is fused with the intensity image to create the final enhanced image. This method highlights the polarization characteristics of the target, enabling rapid and efficient detection and identification of targets in the visible light band. However, this method suffers from poor real-time performance and high computational complexity. Furthermore, the enhanced image alters the information of feature points, introducing artifacts that affect their repeatability and distinguishability, and impacts the number and accuracy of feature point matches.

[0007] In an invention patent application published on June 15, 2018, with publication number CN108171734A, a method and apparatus for ORB feature extraction and matching is disclosed. The method first extracts corner points, calculates the centroid, and Gaussian blurs a grayscale image. The descriptor information corresponding to the feature points is then calculated. The descriptor information of the two frames is then feature-matched to obtain an ORB feature matching result. This solves the existing problem of consuming large amounts of memory and CPU resources to identify the same target image. However, ORB features are relatively scarce in low-texture, weak-texture, and sparse scenes, limiting the number of feature points extracted and failing to guarantee the number of matches and matching accuracy.

[0008] In an invention patent application published on October 8, 2018, with publication number CN108387236A, a polarization SLAM method based on an extended Kalman filter is disclosed. By establishing a state model of the drone and a measurement model based on a lidar sensor and a polarization sensor, the method uses a distributed extended Kalman filter (EKF) algorithm to determine the drone's position and construct a map of the surrounding environment. This method improves the accuracy of the drone's simultaneous positioning and mapping by leveraging the complementary matching of polarization information and lidar information, which are not subject to other external interference. However, it only utilizes atmospheric polarization information, and its applicable scenarios are limited. It also underutilizes the scene's polarization information, limiting its effectiveness in low-texture, weak-texture, and sparse scenes.

[0009] In the invention patent application published on November 3, 2020, with publication number CN111882594A, a method and device for rapid polarization image registration based on ORB feature points is disclosed. By obtaining at least two intensity images of the target in different polarization directions, determining the reference image and the image to be registered, and then using the ORB algorithm to extract feature points and match them, the image to be registered is projected and transformed to obtain the image registration result. This can effectively solve the mismatch problem of polarization images of the target, especially dynamic targets, collected at different times by the time-sharing polarization imaging system. However, it relies on the ORB feature points in the image. In low-texture conditions, the ORB features are missing and there is a problem of easy mismatching. Summary of the Invention

[0010] To address the problems of insufficient number of feature points extracted and matched, low matching accuracy, and poor pose estimation accuracy and easy tracking loss in ORB feature-based visual SLAM methods in low-texture, weak-texture, and sparse scenes, the present invention provides an ORB feature extraction and matching method based on polarization information. The original polarization information collected by a polarization image sensor is used for ORB feature extraction and matching, and the ORB feature extraction and matching effect is optimized by introducing scene polarization information. The present invention uses a polarization image sensor to obtain the polarization information of the scene, calculates a single-channel grayscale image, a single-channel polarization degree image, and a single-channel polarization direction image, and uses the scene polarization information in ORB feature extraction and matching. This can effectively improve the number of ORB feature points extracted, the number of matches, and the matching accuracy, and effectively improve SLAM performance in low-texture, weak-texture, and sparse scenes. The set parameters for feature point extraction and matching, as well as the polarization threshold, can be optimized through feature point extraction and matching effects in different scenarios and data sets, as well as the absolute trajectory error adjustment after running ORB-SLAM3, thereby improving adaptability in multiple scenarios such as low-texture, weak-texture, and sparse scenes. This is of great significance to navigation, positioning, and mapping in fields such as autonomous driving, smart transportation, and the Internet of Things.

[0011] To achieve the above object, the present invention adopts the following technical solutions:

[0012] A method for extracting and matching ORB features based on polarization information includes the following steps:

[0013] Step 1: The polarization image sensor collects raw image frame data, and uses the sliding window method to obtain the single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image corresponding to each frame of the raw image. The single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image are synthesized into a three-channel image in RGB image format as the input image of ORB-SLAM3;

[0014] Step 2: Set the FAST key point extraction mode parameters. After reading the image, separate the channels to obtain a single-channel grayscale image, a single-channel polarization degree image, and a single-channel polarization direction image. Extract FAST key points in five modes: grayscale image extraction mode alone, polarization degree image extraction mode alone, grayscale image first and then polarization degree image extraction mode, polarization degree image first and then grayscale image extraction mode, and grayscale image and polarization degree image simultaneous extraction mode. Compare the number of FAST key points in the five modes and select the one with the largest number of extracted FAST key points as the final FAST key point extraction result.

[0015] Step 3: Calculate the polarization degree and polarization direction values ​​of FAST key points;

[0016] Step 4: Set the FAST key point direction calculation mode parameters, and calculate the direction of the FAST key point for the single-channel grayscale image, single-channel polarization image, and single-channel polarization direction image respectively;

[0017] Step 5: Set the FAST key point descriptor calculation mode parameters, and calculate the descriptors of the FAST key points in the single-channel grayscale image and the single-channel polarization image according to the directions of the three FAST key points obtained in step 4, and obtain six modes of ORB feature points;

[0018] Step 6: Preliminary matching of ORB feature points within the search window is performed using the minimum distance criterion of the descriptor. Matching point pairs that do not meet the minimum distance threshold or the optimal and suboptimal distance ratio are eliminated to obtain candidate matching point pairs. A polarization degree threshold is defined and candidate matching point pairs are screened based on the polarization degree threshold. The candidate matching point pairs with the largest number of matches in the six modes are selected as the final matching result.

[0019] Step 7: After completing the ORB feature point matching, complete the complete SLAM process based on ORB-SLAM3. Adjust and optimize the parameters according to the absolute trajectory error to achieve combined mode switching in different scenarios, thereby improving the accuracy of pose estimation and enhancing adaptability to various scenarios.

[0020] Preferably, in step 1, the single-channel grayscale image is obtained according to the following method: a new pixel is calculated for each unit of the original image, and the grayscale value g of this pixel is calculated as follows:

[0021] g=g0+g2=g1+g3 g∈[0,255]

[0022] Where g is an integer, g0, g1, g2, and g3 represent the grayscale values ​​generated by light passing through polarizers in 0°, 45°, 90°, and 135° directions, respectively, which are used to obtain the image to be processed. The grayscale value of each pixel ranges from 0 to 255. In order to reduce the influence of noise, the image is further processed using the sliding window method. The upper left corner pixel of the image to be processed is used as the starting point, the step size is one pixel, and the sliding window does not exceed the image boundary. A new grayscale value is calculated for each window, which is the average grayscale value of the pixels in the window. Finally, the image size is reduced using the bicubic interpolation algorithm to obtain a single-channel grayscale image.

[0023] Preferably, the single-channel polarization degree map described in step 1 is obtained according to the following method: a new pixel is calculated for each unit of the original image, and the polarization degree of this pixel is calculated as follows:

[0024]

[0025] Where I represents the light intensity, Q and U components represent the direction and intensity of linear polarization, and V represents the circular polarization component. The circular polarization component V in natural light is very low and is ignored here and recorded as 0. P is a number between 0 and 1, and P is nonlinearly mapped to an integer between 0 and 255:

[0026] g P =-255P 2 +510P

[0027] The grayscale value of each pixel is mapped by the polarization degree g P The image to be processed is then processed using the sliding window method, and finally the image size is reduced using the bicubic interpolation algorithm to obtain a single-channel polarization degree map.

[0028] Preferably, the single-channel polarization pattern described in step 1 is obtained according to the following method:

[0029] Each unit of the original image calculates a new pixel, and the polarization direction of this pixel is calculated as follows:

[0030]

[0031] Where θ represents radians, and θ is linearly mapped to integers between 0 and 255:

[0032]

[0033] The grayscale value of each pixel is mapped by the polarization direction g θ The image to be processed is then processed using the sliding window method, and finally the image size is reduced using the bicubic interpolation algorithm to obtain a single-channel polarization direction map.

[0034] Preferably, in the step 1, the single-channel grayscale image, the single-channel polarization degree image, and the single-channel polarization direction image are synthesized into a three-channel image in RGB image format: after calculating the single-channel grayscale image, the single-channel polarization degree image, and the single-channel polarization direction image based on an original image collected by the polarization image sensor, these three types of image information need to be used as the input of ORB-SLAM3. In order to reduce the number of data readings and increase reading efficiency, the three single-channel images are synthesized into an RGB format image, the R channel information of this image is the single-channel grayscale image information, the G channel information is the single-channel polarization degree image information, and the B channel information is the single-channel polarization direction image information. The image is named using a timestamp, and the names of all images and their storage paths are written into a text file in chronological order as an index file; each frame of image obtained by the polarization image sensor is converted into an RGB image as the input image of ORB-SLAM3.

[0035] Preferably, in step 2, ORB-SLAM3 reads the image in sequence according to the index file, separates the three channels after reading, and obtains the single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image under the current timestamp respectively, sets the FAST key point extraction mode parameters, extracts FAST key points in five modes, and selects the one with the largest number of FAST key points as the extraction result:

[0036] ① Mode 1: Single grayscale image extraction mode, the image is divided into several grids, each grid is first extracted with the default feature point response value threshold on the single-channel grayscale image FAST key points, if no FAST key points are extracted, then the FAST key points are extracted with a lower feature point response value threshold;

[0037] ② Mode 2: Single polarization map extraction mode, the image is divided into several grids, and FAST key points are first extracted on the polarization map of each grid using the default feature point response value threshold. If no FAST key points are extracted, FAST key points are extracted using a lower feature point response value threshold;

[0038] ③ Mode 3: Grayscale image first, then polarization degree image extraction mode. The single-channel grayscale image and single-channel polarization degree image are divided into several grids. In each grid, FAST key points are first extracted on the single-channel grayscale image with the default feature point response value threshold. If no FAST key points are extracted, FAST key points are extracted on the single-channel polarization degree image with a lower feature point response value threshold.

[0039] ④ Mode 4: Extract the polarization map first and then the grayscale map. The single-channel grayscale map and the single-channel polarization map are divided into several grids. In each grid, FAST keypoints are first extracted on the single-channel polarization map using the default feature point response value threshold. If no FAST keypoints are extracted, the FAST keypoint extraction is switched to the single-channel grayscale map using a lower feature point response value threshold.

[0040] ⑤ Mode 5: Simultaneous extraction of grayscale and polarization images. The single-channel grayscale image and single-channel polarization image are divided into several grids. Two feature point response value thresholds are set. For each grid, FAST key points are first extracted from the single-channel grayscale image using one feature point response value threshold, and then from the single-channel polarization image using another feature point response value threshold.

[0041] Preferably, in step 3, the polarization degree value and polarization direction value calculation method of the FAST key point are as follows: the polarization degree value of the FAST key point is obtained from a single-channel polarization degree diagram, and a mean is calculated using the grayscale values ​​of surrounding pixels according to the coordinates of the FAST key point on the single-channel polarization degree diagram, as the polarization degree value of this FAST key point;

[0042] The polarization direction value of the FAST key point is obtained from the single-channel polarization direction map. On the single-channel polarization direction map, according to the coordinates of the FAST key point, the grayscale values ​​of the surrounding pixels are used to calculate an average value as the polarization direction value of this FAST key point.

[0043] Preferably, three FAST key point direction calculation mode parameters are set in step 4, and the directions of the FAST key points are calculated for the single-channel grayscale image, the single-channel polarization degree image, and the single-channel polarization direction image respectively:

[0044] ① Mode 1: Grayscale image FAST key point direction calculation mode, which uses the grayscale centroid method to calculate the direction of the FAST key point on a single-channel grayscale image;

[0045] ② Mode 2: Polarization degree map FAST key point direction calculation mode, using the grayscale centroid method to calculate the direction of the FAST key point on a single-channel polarization degree map;

[0046] ③ Mode 3: Polarization direction value replacement mode, directly using the polarization direction value of the FAST key point to replace the FAST key point direction.

[0047] Preferably, two FAST key point descriptor calculation mode parameters are set in step 5, and according to the direction of the FAST key point obtained in step 4, the descriptors of the FAST key point are calculated for the single-channel grayscale image and the single-channel polarization degree image, respectively, to obtain ORB feature points of six combination modes:

[0048] ① Mode 1: Grayscale image descriptor calculation mode, calculating the Steer BRIEF descriptor on a single-channel grayscale image;

[0049] ② Mode 2: Polarization degree map descriptor calculation mode, calculates the Steer BRIEF descriptor on a single-channel polarization degree map.

[0050] Preferably, a polarization degree threshold is defined in step 6, and candidate matching point pairs are assisted in screening based on the threshold: the minimum distance criterion of the descriptor is used to preliminarily match ORB feature points in the search window, and matching point pairs that do not meet the minimum distance threshold and do not meet the optimal and suboptimal distance ratio are eliminated to obtain candidate matching point pairs. At the same time, the minimum distance threshold is appropriately increased to obtain more candidate matching point pairs; a polarization degree threshold is defined, and its value range is 0-100%. The threshold is adjusted according to the number of matches and the accuracy rate. This threshold is used to assist in screening matching point pairs, and the one with the largest number of matches in the six modes is used as the final matching result;

[0051] Both ORB feature points in a candidate matching point pair have a polarization degree value, which is an integer between 0 and 255. When both polarization degrees are small, the match is considered successful. When the polarization degrees are large, the larger polarization value is used as a reference. If the difference between the two is less than or equal to the polarization degree threshold, the match is considered successful. If the difference between the two is greater than the polarization degree threshold, the matching point pair is discarded.

[0052] Preferably, in step 7, after completing the ORB feature point matching, the complete SLAM process is completed based on ORB-SLAM3, the map is initialized first, tracking is performed after completing the local mapping, the camera pose is optimized, and closed-loop detection is performed at the same time. After the closed loop is detected, global BA (Bundle Adjustment) is performed to reduce the cumulative error, and finally a globally consistent camera pose and map are obtained. The estimated camera pose is compared with the real camera pose to obtain the absolute trajectory error. According to the number of ORB feature points extracted, the number of successful matches, the matching accuracy and the absolute trajectory error after running ORB-SLAM3, the FAST key point extraction mode parameters, the FAST key point direction calculation mode parameters, the FAST key point descriptor calculation mode parameters and the polarization threshold are adjusted and optimized to achieve combined mode switching under different scenes, thereby improving the pose estimation accuracy and enhancing the adaptability to various low-texture, weak-texture and sparse scenes.

[0053] Beneficial effects:

[0054] The present invention applies scene polarization information to ORB feature extraction and matching. The original image frame data of the scene is collected by a polarization image sensor, and the original image is processed to obtain a single-channel grayscale image, a single-channel polarization degree image, and a single-channel polarization direction image corresponding to each frame of the original image. In order to reduce the influence of noise when extracting and matching ORB feature points, a sliding window method is used when calculating the image. A variety of modes are used when extracting and matching ORB feature points, which not only utilize the scene grayscale information but also utilize the scene polarization information. The two complement each other and effectively improve the feature point extraction and matching effect. Conventional methods have mismatched points when matching feature points, and some correct matching point pairs are eliminated because they do not meet the minimum distance threshold. Therefore, the present invention sets a polarization value threshold to assist in the matching of feature points, appropriately increases the minimum distance threshold during matching, thereby generating more candidate matching point pairs, and further screening through the polarization value threshold, effectively improving the number of feature point extraction and matching, and matching accuracy.

[0055] This invention incorporates scene polarization information into ORB feature extraction and matching, effectively increasing the number of ORB feature points extracted, the number of matches, and the matching accuracy, and effectively improving SLAM performance in low-texture, weak-texture, and sparse scenes. The set feature point extraction and matching parameters and polarization threshold can be optimized by adjusting the feature point extraction and matching results in different scenes and data sets, as well as the absolute trajectory error after running ORB-SLAM3. This improves environmental adaptability in multiple scenarios, such as low-texture, weak-texture, and sparse scenes, and has important implications for navigation, positioning, and mapping in fields such as autonomous driving, smart transportation, and the Internet of Things. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of an ORB feature extraction and matching method based on polarization information of the present invention;

[0057] Figure 2 Schematic diagram of pixel distribution of the polarization image sensor used in the present invention;

[0058] Figure 3 is a schematic diagram of an original image obtained by a polarization image sensor of the present invention;

[0059] Figure 4 Schematic diagram of a single-channel grayscale image obtained by the present invention;

[0060] Figure 5 Schematic diagram of a single-channel degree of polarization obtained by the present invention;

[0061] Figure 6 Schematic diagram of a single-channel polarization pattern obtained by the present invention;

[0062] Figure 7 It is a schematic diagram of a three-channel graph synthesized by the present invention;

[0063] Figure 8 is a schematic diagram of the matching results;

[0064] Figure 9 This is a schematic diagram of ORB feature point tracking when running ORB-SLAM3;

[0065] Figure 10 This is a schematic diagram of the camera pose and point cloud map after running ORB-SLAM3. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below in conjunction with the accompanying drawings, which serve as a part of this specification and illustrate the principles of the present invention through implementation. Other aspects, features and advantages of the present invention will become apparent from this detailed description.

[0067] The invention relates to an ORB feature extraction and matching method based on polarization information. The method comprises the following steps: collecting original image frame data by a polarization image sensor, obtaining a single-channel grayscale image, a single-channel polarization degree image, and a single-channel polarization direction image corresponding to each frame of the original image by using a sliding window method; extracting FAST (Features From Accelerated Segment Test) key points in five extraction modes, namely, a grayscale image extraction mode, a polarization degree image extraction mode, a grayscale image extraction mode followed by a polarization degree image extraction mode, a polarization degree image extraction mode followed by a grayscale image extraction mode, and a grayscale image and polarization degree image simultaneous extraction mode; selecting the FAST key point with the largest number as the FAST key point extraction result; calculating polarization degree values ​​and polarization direction values ​​of the extracted FAST key points; calculating the directions of the FAST key points on the single-channel grayscale image, the single-channel polarization degree image, and the single-channel polarization direction image; and calculating descriptors of the FAST key points on the single-channel grayscale image and the single-channel polarization degree image according to the three directions, thereby obtaining ORB feature points in six modes. The minimum distance criterion of the descriptor is used to initially match ORB feature points to obtain candidate matching point pairs. A polarization threshold is defined and then filtered based on the polarization threshold. The pair with the largest number of matches across the six modes is selected as the final match result. After ORB feature point matching is completed, the full SLAM process is completed using ORB-SLAM3. The parameters are adjusted and optimized based on the absolute trajectory error, enabling combined mode switching in different scenarios. This improves pose estimation accuracy and enhances adaptability to various scenarios, including low-texture, weak-texture, and sparse scenes.

[0068] like Figure 1 As shown, the ORB feature extraction and matching based on polarization information of the present invention includes the following steps:

[0069] Step 1: Collect the original image frame data through the polarization image sensor, use the sliding window method to obtain the single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image corresponding to each frame of the original image, and use the RGB image format to synthesize the single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image into a three-channel image as the input image of ORB-SLAM3;

[0070] Step 2: Set the FAST keypoint extraction mode parameters. After reading the image, separate the channels to obtain a single-channel grayscale image, a single-channel polarization map, and a single-channel polarization direction map. Extract FAST keypoints using five modes: grayscale image extraction alone, polarization map extraction alone, grayscale image first followed by polarization map extraction, polarization map first followed by grayscale image extraction, and simultaneous grayscale and polarization map extraction. Compare the number of FAST keypoints in each mode and select the one with the largest number of extracted FAST keypoints as the final FAST keypoint extraction result.

[0071] Step 3: Calculate the polarization degree and polarization direction values ​​of FAST key points;

[0072] Step 4: Set the FAST key point direction calculation mode parameters, and calculate the direction of the FAST key point for the single-channel grayscale image, single-channel polarization image, and single-channel polarization direction image respectively;

[0073] Step 5: Set the FAST key point descriptor calculation mode parameters, and calculate the FAST key point descriptors in the single-channel grayscale image and single-channel polarization image according to the three FAST key point directions obtained in step 4, and obtain six modes of ORB feature points;

[0074] Step 6: Preliminary matching of ORB feature points within the search window is performed using the minimum distance criterion of the descriptor. Matching point pairs that do not meet the minimum distance threshold or the optimal and suboptimal distance ratio are eliminated to obtain candidate matching point pairs. A polarization degree threshold is defined and candidate matching point pairs are screened based on the polarization degree threshold. The pair with the largest number of matches under the six modes is taken as the final matching result.

[0075] Step 7: After completing the ORB feature point matching, complete the complete SLAM process based on ORB-SLAM3. Adjust and optimize the parameters according to the absolute trajectory error to achieve combined mode switching in different scenarios, thereby improving the accuracy of pose estimation and enhancing adaptability to various scenarios.

[0076] Preferably, in step 1, the original image is obtained by using a polarization image sensor. Taking the polarization image sensor PHX050S-PC as an example, the original image is a single-channel grayscale image with a size of 2448*2048, a frequency of 20Hz, and a grayscale accuracy of 8 bits. The pixel distribution is as follows: Figure 2 As shown, the original image example is as follows Figure 3 As shown in the figure, four pixels are used as a unit, which contains 1224*1024 uniformly distributed units. Each unit contains four pixels covered by linear polarizers with angles of 90°, 45°, 135°, and 0° respectively.

[0077] The Stokes vector uses four parameters to describe the intensity and polarization state of light waves:

[0078]

[0079] Among them, the I component represents the light intensity, the Q and U components represent the direction and intensity of linear polarization, and V represents the circular polarization component. Taking four pixels as a group, its Stokes vector can be expressed as:

[0080]

[0081] Where I is the total light intensity, I0, I1, I2, and I3 represent the linear polarized light intensity in the four-way polarizer in the polarization image sensor when the transmission axis is 0°, 45°, 90°, and 135° to the reference direction, respectively. r and I l represents the intensity of right-handed and left-handed circularly polarized light. The circular polarization component V in natural light is very low and is ignored here and recorded as 0. The polarization degree P and polarization direction θ can be expressed as follows:

[0082]

[0083]

[0084] Preferably, a single-channel grayscale image is calculated using the original image, and a new pixel is calculated for each unit of the original image. The grayscale value g of this pixel is calculated as follows:

[0085] g=g0+g2=g1+g3 g∈[0,255]

[0086] Among them, g is an integer, g0, g1, g2, g3 represent the grayscale values ​​generated by light passing through the polarizer in the 0°, 45°, 90°, and 135° directions respectively, and the image to be processed can be obtained. This image has a total of 1224*1024 pixels, and the grayscale value of each pixel ranges from 0 to 255. In order to reduce the influence of noise, the image is further processed using the sliding window method. The sliding window size is 2*2 pixels. The upper left corner pixel of the image to be processed is used as the starting point, the step size is one pixel, and the sliding window does not exceed the image boundary. A new grayscale value is calculated for each window. This value is the average grayscale value of the four pixels in the window. There are a total of 1223*1023 windows, and the calculated image size is 1223*1023 pixels. Finally, the image size is reduced to 600*500 pixels using the bicubic interpolation algorithm. The resulting single-channel grayscale image is as follows Figure 4 shown.

[0087] Preferably, in step 1, the single-channel polarization degree map is calculated using the original image, and a new pixel is calculated for each unit of the original image. The polarization degree of this pixel is calculated as follows:

[0088]

[0089] P is a number between 0 and 1, and P is nonlinearly mapped to an integer between 0 and 255:

[0090] g P =-255P 2 +510P

[0091] There are 1224*1024 pixels in total, and the grayscale value of each pixel is g after polarization mapping. P The image is replaced by the value, and a single-channel polarization degree map with a size of 1224*1024 pixels is generated. Then, the sliding window method is used to obtain a polarization degree map with a size of 1223*1023 pixels. Finally, the bicubic interpolation algorithm is used to reduce the image size to 600*500 pixels. The polarization degree map obtained is as follows Figure 5 shown.

[0092] Preferably, in step 1, the single-channel polarization direction map is calculated using the original image, and a new pixel is calculated for each unit of the original image. The polarization direction of this pixel is calculated as follows:

[0093]

[0094] Where θ represents radians, and θ is linearly mapped to integers between 0 and 255:

[0095]

[0096] There are 1224*1024 pixels in total, and the grayscale value of each pixel is mapped by the polarization direction g θ The value is replaced, and a single-channel polarization direction map with a size of 1224*1024 pixels is generated. Then, a single-channel polarization direction map with a size of 1223*1023 pixels is obtained using the sliding window method. Finally, the image size is reduced to 600*500 pixels using the bicubic image interpolation algorithm. The obtained single-channel polarization direction map is as follows Figure 6 shown.

[0097] Preferably, in step 1, the obtained single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image are synthesized into a three-channel image in RGB image format: after calculating the single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image based on an original image collected by a polarization image sensor, these three types of image information need to be used as input for ORB-SLAM3. In order to reduce the number of data readings and increase reading efficiency, these three single-channel images are synthesized into an image in RGB format. The R channel information of this image is single-channel grayscale image information, the G channel information is single-channel polarization degree image information, and the B channel information is single-channel polarization direction image information. The image is named using a timestamp in seconds, accurate to six decimal places. The names of all images and their storage paths are written into a text file in chronological order as an index file. Each frame of image obtained by the polarization image sensor is converted into an RGB image of size 600*500, and this image is used as the input image for ORB-SLAM3. The synthesized three-channel image is as follows: Figure 7shown.

[0098] Preferably, in step 2, the FAST key point extraction mode parameter extractor_mode is set and the image is read: ORB-SLAM3 reads the image in sequence according to the index file, separates the three channels after reading, and obtains the single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image under the current timestamp respectively, sets the FAST key point extraction mode parameter extractor_mode, extracts the FAST key points respectively in five modes, and selects the one with the largest number of FAST key points as the extraction result:

[0099] ① Mode 1: Single grayscale image extraction mode, the image is divided into 35*35 pixel grids, and each grid is first extracted from the single-channel grayscale image with the default feature point response value threshold. If no FAST key point is extracted, a lower feature point response value threshold is used to extract the FAST key point;

[0100] ② Mode 2: Single polarization map extraction mode: The image is divided into 35*35 pixel grids. FAST keypoints are first extracted on the single-channel polarization map using the default feature point response value threshold for each grid. If no FAST keypoints are extracted, FAST keypoints are extracted using a lower feature point response value threshold.

[0101] ③ Mode 3: Grayscale image first, then polarization degree image extraction mode. The single-channel grayscale image and single-channel polarization degree image are divided into 35*35 pixel grids. FAST keypoints are first extracted from the single-channel grayscale image using the default feature point response value threshold for each grid. If no FAST keypoints are extracted, FAST keypoint extraction is switched to the single-channel polarization degree image using a lower feature point response value threshold.

[0102] ④ Mode 4: Extract the polarization image first and then the grayscale image. The single-channel grayscale image and the single-channel polarization image are divided into 35*35 pixel grids. FAST keypoints are first extracted on the single-channel polarization image with the default feature point response value threshold for each grid. If no FAST keypoints are extracted, the FAST keypoint extraction is switched to the single-channel grayscale image with a lower feature point response value threshold.

[0103] Mode 5: Simultaneous extraction of grayscale and polarization images. The single-channel grayscale and polarization images are divided into 35x35 pixel grids. Two feature point response value thresholds are set. For each grid, FAST keypoints are first extracted from the single-channel grayscale image using one feature point response value threshold, and then from the single-channel polarization image using another feature point response value threshold.

[0104] Preferably, in described step 3, the polarization degree value of the FAST key point is obtained by the single-channel polarization degree diagram, and on the single-channel polarization degree diagram, according to the coordinates at the FAST key point, the grayscale values ​​of the surrounding 3*3 pixel blocks are utilized to calculate a mean, as the polarization degree value of this FAST key point, using a 3*3 Gaussian kernel:

[0105]

[0106] The polarization direction value of the FAST key point is obtained from the single-channel polarization direction map. On the single-channel polarization direction map, according to the coordinates of the FAST key point, the grayscale values ​​of the surrounding 3*3 pixel blocks are used to calculate an average value as the polarization direction value of this FAST key point, using the above 3*3 Gaussian kernel.

[0107] Preferably, in step 4, three FAST key point direction calculation modes are set to calculate the direction of the FAST key point for a single-channel grayscale image, a single-channel polarization degree image, and a single-channel polarization direction image, respectively:

[0108] ① Mode 1: Grayscale image FAST key point direction calculation mode, which uses the grayscale centroid method to calculate the direction of the FAST key point on a single-channel grayscale image;

[0109] ② Mode 2: Polarization degree map FAST key point direction calculation mode, using the grayscale centroid method to calculate the direction of the FAST key point on a single-channel polarization degree map;

[0110] ③ Mode 3: Polarization direction value replacement mode, directly using the polarization direction value of the FAST key point to replace the FAST key point direction.

[0111] Preferably, in step 5, the FAST key point descriptor calculation mode parameter descriptors_mode is set, and two FAST key point descriptor calculation modes are set. The descriptors of the FAST key points are calculated for the single-channel grayscale image and the single-channel polarization degree image, respectively. Combined with the previous step, six ORB feature points of the combination mode are obtained:

[0112] ① Mode 1: Grayscale image descriptor calculation mode, calculating the Steer BRIEF descriptor on a single-channel grayscale image;

[0113] ② Mode 2: Polarization degree map descriptor calculation mode, calculates the Steer BRIEF descriptor on a single-channel polarization degree map.

[0114] In a preferred embodiment, a polarization degree threshold, Polar_dist, is defined and used to filter candidate matching point pairs based on the polarization degree threshold. The ORB feature points are initially matched within the search window using the minimum distance criterion of the descriptor. Matching point pairs that do not meet the minimum distance threshold or the optimal to suboptimal distance ratio are eliminated to obtain candidate matching point pairs. The minimum distance threshold is also appropriately increased to obtain more candidate matching point pairs. A polarization degree threshold is defined within a range of 0-100%, and the threshold is adjusted based on the number of matches and accuracy. This threshold is used to assist in filtering matching point pairs, and the matching result with the largest number of matches across the six modes is selected as the final match result.

[0115] The two ORB feature points in the candidate matching point pair have a polarization value, which is an integer between 0 and 255. When both polarization values ​​are less than or equal to 10, the match is considered successful. When the two polarization values ​​are other than the reference value, the larger polarization value is used as the reference value. If the difference between the two is less than or equal to 30% of the reference value (i.e., the polarization threshold, which is adjusted according to the matching effect), the match is considered successful. If the difference between the two is greater than 30% of the reference value, the matching point pair is discarded. The matching result diagram is shown as follows: Figure 8 shown.

[0116] Preferably, adjust and optimize the parameters: After completing the ORB feature point matching, complete the complete SLAM process based on ORB-SLAM3, initialize the map first, and then track after completing the local map (the ORB feature point tracking diagram is shown in the figure). Figure 9 ), optimize the camera pose, perform closed-loop detection at the same time, perform global BA (Bundle Adjustment) after detecting the closed loop, reduce the cumulative error, and finally obtain a globally consistent camera pose and map (the camera pose and point cloud map obtained by running ORB-SLAM3 is shown in the figure). Figure 10 As shown in the figure, the estimated camera pose is compared with the real camera pose to obtain the absolute trajectory error. According to the number of ORB feature points extracted, the number of successful matches, the matching accuracy and the absolute trajectory error after running ORB-SLAM3, the FAST key point extraction mode parameters extractor_mode, FAST key point direction calculation mode parameters orientations_mode, FAST key point descriptor calculation mode parameters descriptors_mode and polarization threshold Polar_dist are adjusted and optimized to achieve combined mode switching in different scenarios, thereby improving the pose estimation accuracy and the environmental adaptability in multiple scenarios such as low texture, weak texture and sparse scenes. It is of great significance to navigation, positioning and mapping in the fields of autonomous driving, smart transportation, Internet of Things and so on.

[0117] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for ORB feature extraction and matching based on polarization information, characterized in that: The following steps are involved: Step 1: The polarization image sensor collects raw image frame data, and uses the sliding window method to obtain the single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image corresponding to each frame of the raw image. The single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image are synthesized into a three-channel image in RGB image format as the input image of ORB-SLAM3; Step 2: Set the FAST key point extraction mode parameters. After reading the image, separate the channels to obtain a single-channel grayscale image, a single-channel polarization degree image, and a single-channel polarization direction image. Extract FAST key points in five modes: grayscale image extraction mode alone, polarization degree image extraction mode alone, grayscale image first and then polarization degree image extraction mode, polarization degree image first and then grayscale image extraction mode, and grayscale image and polarization degree image simultaneous extraction mode. Compare the number of FAST key points in the five modes and select the one with the largest number of extracted FAST key points as the final FAST key point extraction result. Step 3: Calculate the polarization degree and polarization direction values ​​of FAST key points; Step 4: Set the FAST key point direction calculation mode parameters, and calculate the direction of the FAST key point for the single-channel grayscale image, single-channel polarization image, and single-channel polarization direction image respectively; Step 5: Set the FAST key point descriptor calculation mode parameters, and calculate the descriptors of the FAST key points in the single-channel grayscale image and the single-channel polarization image according to the directions of the three FAST key points obtained in step 4, and obtain six modes of ORB feature points; Step 6: Preliminary matching of ORB feature points within the search window is performed using the minimum distance criterion of the descriptor. Matching point pairs that do not meet the minimum distance threshold or the optimal and suboptimal distance ratio are eliminated to obtain candidate matching point pairs. A polarization degree threshold is defined and candidate matching point pairs are screened based on the polarization degree threshold. The candidate matching point pairs with the largest number of matches in the six modes are selected as the final matching result. Step 7: After completing the ORB feature point matching, complete the complete SLAM process based on ORB-SLAM3. Adjust and optimize the parameters according to the absolute trajectory error to achieve combined mode switching in different scenarios, thereby improving the accuracy of pose estimation and enhancing adaptability to various scenarios.

2. The ORB feature extraction and matching method based on polarization information according to claim 1, characterized in that: In step 1, the single-channel grayscale image is obtained by the following method: a new pixel is calculated for each unit of the original image, and the grayscale value g of this pixel is calculated as follows: g=g0+g2=g1+g3 g∈[0,255] Where g is an integer, g0, g1, g2, and g3 represent the grayscale values ​​generated by light passing through polarizers in 0°, 45°, 90°, and 135° directions, respectively, which are used to obtain the image to be processed. The grayscale value of each pixel ranges from 0 to 255. In order to reduce the influence of noise, the image is further processed using the sliding window method. The upper left corner pixel of the image to be processed is used as the starting point, the step size is one pixel, and the sliding window does not exceed the image boundary. A new grayscale value is calculated for each window, which is the average grayscale value of the pixels in the window. Finally, the image size is reduced using the bicubic interpolation algorithm to obtain a single-channel grayscale image.

3. The ORB feature extraction and matching method based on polarization information according to claim 1, characterized in that: The single-channel polarization degree diagram described in step 1 is obtained by the following method: Each unit of the original image calculates a new pixel, and the polarization degree of this pixel is calculated as follows: Where I represents the light intensity, Q and U components represent the direction and intensity of linear polarization, and V represents the circular polarization component. The circular polarization component V in natural light is very low and is ignored here and recorded as 0. P is a number between 0 and 1, and P is nonlinearly mapped to an integer between 0 and 255: g P =-255P 2 +510P The grayscale value of each pixel is mapped by the polarization degree g P The image to be processed is then processed using the sliding window method, and finally the image size is reduced using the bicubic interpolation algorithm to obtain a single-channel polarization degree map.

4. The ORB feature extraction and matching method based on polarization information according to claim 1, characterized in that: The single-channel polarization pattern described in step 1 is obtained by the following method: Each unit of the original image calculates a new pixel, and the polarization direction of this pixel is calculated as follows: Where θ represents radians, and θ is linearly mapped to integers between 0 and 255: The grayscale value of each pixel is mapped by the polarization direction g θ The image to be processed is then processed using the sliding window method, and finally the image size is reduced using the bicubic interpolation algorithm to obtain a single-channel polarization direction map.

5. The ORB feature extraction and matching method based on polarization information according to claim 1, characterized in that: In the step 1, a single-channel grayscale image, a single-channel polarization degree image, and a single-channel polarization direction image are synthesized into a three-channel image in RGB image format: after calculating a single-channel grayscale image, a single-channel polarization degree image, and a single-channel polarization direction image based on an original image collected by a polarization image sensor, these three types of image information are used as input of ORB-SLAM3. In order to reduce the number of data readings and increase reading efficiency, the three single-channel images are synthesized into an RGB format image, the R channel information of this image is single-channel grayscale image information, the G channel information is single-channel polarization degree image information, and the B channel information is single-channel polarization direction image information. The image is named using a timestamp, and the names and storage paths of all images are written into a text file in chronological order as an index file; each frame of image obtained by the polarization image sensor is converted into an RGB image as the input image of ORB-SLAM3.

6. The ORB feature extraction and matching method based on polarization information according to claim 5, characterized in that: In step 2, ORB-SLAM3 reads the image in sequence according to the index file, separates the three channels after reading, and obtains the single-channel grayscale image, single-channel polarization degree image, and single-channel polarization direction image at the current timestamp respectively. The FAST key point extraction mode parameters are set, and the FAST key points are extracted in five modes respectively. The one with the largest number of FAST key points is selected as the extraction result: ① Mode 1: Single grayscale image extraction mode, the image is divided into several grids, each grid is first extracted with the default feature point response value threshold on the single-channel grayscale image FAST key points, if no FAST key points are extracted, then the FAST key points are extracted with a lower feature point response value threshold; ② Mode 2: Single polarization map extraction mode, the image is divided into several grids, and FAST keypoints are first extracted on the single-channel polarization map of each grid using the default feature point response value threshold. If no FAST keypoints are extracted, FAST keypoints are extracted using a lower feature point response value threshold; ③ Mode 3: Grayscale image first, then polarization degree image extraction mode. The single-channel grayscale image and single-channel polarization degree image are divided into several grids. In each grid, FAST key points are first extracted on the single-channel grayscale image with the default feature point response value threshold. If no FAST key points are extracted, FAST key points are extracted on the single-channel polarization degree image with a lower feature point response value threshold. ④ Mode 4: Extract the polarization map first and then the grayscale map. The single-channel grayscale map and the single-channel polarization map are divided into several grids. In each grid, FAST keypoints are first extracted on the single-channel polarization map using the default feature point response value threshold. If no FAST keypoints are extracted, the FAST keypoint extraction is switched to the single-channel grayscale map using a lower feature point response value threshold. ⑤ Mode 5: Simultaneous extraction of grayscale and polarization images. The single-channel grayscale image and single-channel polarization image are divided into several grids. Two feature point response value thresholds are set. For each grid, FAST key points are first extracted from the single-channel grayscale image using one feature point response value threshold, and then from the single-channel polarization image using another feature point response value threshold.

7. The ORB feature extraction and matching method based on polarization information according to claim 6, characterized in that: In described step 3, the polarization degree value and polarization direction value calculation method of the FAST key point are as follows: the polarization degree value of the FAST key point is obtained by a single-channel polarization degree diagram, and on the single-channel polarization degree diagram, according to the coordinates of the FAST key point, a mean is calculated using the grayscale values ​​of the surrounding pixels as the polarization degree value of this FAST key point; The polarization direction value of the FAST key point is obtained from the single-channel polarization direction map. On the single-channel polarization direction map, according to the coordinates of the FAST key point, the grayscale values ​​of the surrounding pixels are used to calculate an average value as the polarization direction value of this FAST key point.

8. The ORB feature extraction and matching method based on polarization information according to claim 7, characterized in that: In step 4, three FAST key point direction calculation mode parameters are set to calculate the direction of the FAST key point for the single-channel grayscale image, the single-channel polarization degree image, and the single-channel polarization direction image respectively: ① Mode 1: Grayscale image FAST key point direction calculation mode, which uses the grayscale centroid method to calculate the direction of the FAST key point on a single-channel grayscale image; ② Mode 2: Polarization degree map FAST key point direction calculation mode, using the grayscale centroid method to calculate the direction of the FAST key point on a single-channel polarization degree map; ③ Mode 3: Polarization direction value replacement mode, directly using the polarization direction value of the FAST key point to replace the FAST key point direction.

9. The ORB feature extraction and matching method based on polarization information according to claim 8, characterized in that: In step 5, two FAST key point descriptor calculation mode parameters are set. According to the direction of the FAST key point obtained in step 4, the descriptors of the FAST key point are calculated for the single-channel grayscale image and the single-channel polarization degree image, respectively, to obtain ORB feature points of six combination modes: ① Mode 1: Grayscale image descriptor calculation mode, calculating the Steer BRIEF descriptor on a single-channel grayscale image; ② Mode 2: Polarization degree map descriptor calculation mode, calculates the Steer BRIEF descriptor on a single-channel polarization degree map.

10. The ORB feature extraction and matching method based on polarization information according to claim 9, characterized in that: In step 6, a polarization degree threshold is defined, and candidate matching point pairs are assisted in screening based on the threshold: the minimum distance criterion of the descriptor is used to preliminarily match ORB feature points within the search window, and matching point pairs that do not meet the minimum distance threshold and do not meet the optimal and suboptimal distance ratio are eliminated to obtain candidate matching point pairs. At the same time, the minimum distance threshold is appropriately increased to obtain more candidate matching point pairs; a polarization degree threshold is defined, and its value range is 0-100%. The threshold is adjusted according to the number of matches and accuracy. This threshold is used to assist in screening matching point pairs, and the one with the largest number of matches in the six modes is used as the final matching result; Both ORB feature points in a candidate matching point pair have a polarization degree value, which is an integer between 0 and 255. When both polarization degrees are small, the match is considered successful. When the polarization degrees are large, the larger polarization value is used as a reference. If the difference between the two is less than or equal to the polarization degree threshold, the match is considered successful. If the difference between the two is greater than the polarization degree threshold, the matching point pair is discarded.

11. The ORB feature extraction and matching method based on polarization information according to claim 10, characterized in that: In the step 7, after completing the ORB feature point matching, the complete SLAM process is completed based on ORB-SLAM3. The map is initialized first, and tracking is performed after completing the local mapping. The camera pose is optimized and closed-loop detection is performed at the same time. After the closed loop is detected, global BA is performed to reduce the cumulative error. Finally, a globally consistent camera pose and map are obtained. The estimated camera pose is compared with the real camera pose to obtain the absolute trajectory error. According to the number of ORB feature points extracted, the number of successful matches, the matching accuracy and the absolute trajectory error after running ORB-SLAM3, the FAST key point extraction mode parameters, the FAST key point direction calculation mode parameters, the FAST key point descriptor calculation mode parameters and the polarization threshold are adjusted and optimized to achieve combined mode switching under different scenarios, thereby improving the pose estimation accuracy and enhancing the adaptability to various low-texture, weak-texture and sparse scenarios.

Images