Visual SLAM image feature matching method, terminal and storage medium

Abstract

The invention provides a visual SLAM image feature matching method, a terminal and a storage medium, and relates to the field of visual navigation. The method comprises the following steps: determining a quality score of each pixel point in an original image through a Harris corner detection algorithm; according to the Harris mass fraction of each pixel point, determining a coarse tracking level neighborhood window size of each pixel point; performing coarse tracking level optical flow tracking according to the coarse tracking level neighborhood window size of each pixel point, and determining a coarse tracking optical flow; according to the coarse tracking optical flow and a preset fine tracking level neighborhood window size, fine tracking level optical flow tracking is carried out, and a fine tracking optical flow is determined; determining feature points according to the fine tracking optical flow and carrying out feature matching; wherein the size of the coarse tracking level neighborhood window is larger than that of the fine tracking level neighborhood window. According to the invention, the problem of poor feature matching precision of a visual SLAM system can be solved.

Description

Technical Field

[0001] This invention relates to the field of visual navigation, and more particularly to a visual SLAM image feature matching method, apparatus, device, and storage medium. Background Technology

[0002] Computer vision and inertial navigation technologies have driven the practical application of Simultaneous Localization and Mapping (SLAM) systems, enabling real-time localization and 3D map construction. Based on the output of visual SLAM, intelligent terminals such as mobile robots and drones can achieve spatial perception and autonomous navigation.

[0003] The ORB_SLAM3 framework is one of the mainstream solutions in the field of visual SLAM. It can integrate multiple sensors and adopt a feature extraction and multi-threaded architecture to achieve its core functions. However, existing ORB_SLAM3 methods typically use corner detection to extract feature points during implementation. In complex scenes, feature matching is prone to errors and cannot simultaneously handle feature matching for both large and small displacements.

[0004] Therefore, there is an urgent need for a visual SLAM image feature matching method that can ensure the stability of feature matching and the quality of feature points in complex scenarios such as rapid camera movement, so as to improve the feature matching accuracy and system robustness. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a visual SLAM image feature matching method, apparatus, device and storage medium to solve the problem that existing visual SLAM systems are unable to guarantee feature matching stability and feature localization accuracy in complex scenes.

[0006] In a first aspect, embodiments of the present invention provide a visual SLAM image feature matching method, comprising: The original image acquired by visual SLAM is obtained, and the Harris quality score of each pixel in the original image is determined by the Harris corner detection algorithm. For the first type of image frame in the original image, feature points are detected and feature matching is performed using the Oriented Fast and Rotated BRIEF (ORB) algorithm. For the second type of image frame in the original image, the coarse tracking level neighborhood window size of each pixel is determined based on the Harris quality score of each pixel; coarse tracking level optical flow is performed based on the coarse tracking level neighborhood window size of each pixel to determine the coarse tracking optical flow; fine tracking level optical flow is performed based on the coarse tracking optical flow and a preset fine tracking level neighborhood window size to determine the fine tracking optical flow; feature points are determined based on the fine tracking optical flow and feature matching is performed; wherein, the coarse tracking level neighborhood window size is larger than the fine tracking level neighborhood window size.

[0007] In one possible implementation, determining the coarse tracking level neighborhood window size for each pixel based on its Harris quality score includes: For pixels whose Harris quality score is greater than or equal to a preset threshold, the size of the coarse tracking level neighborhood window is determined to be a first preset value; For pixels whose Harris quality score is less than a preset threshold, the size of the coarse tracking level neighborhood window is determined to be a second preset value; The first preset value is less than the second preset value.

[0008] In one possible implementation, determining feature points and performing feature matching based on the fine-tracking optical flow includes: Based on the preset verification-level neighborhood window size, optical flow verification is performed on the prediction points of the current frame in the fine tracking optical flow to determine the verification optical flow; Calculate the forward and backward errors between the verification optical flow and the fine tracking optical flow, and determine multiple initial screening feature points from the prediction points based on the forward and backward errors; Based on the Harris mass fraction and photometric error of the initial screening feature points, the confidence value of the initial screening feature points is determined. Based on the confidence value of each initial screening feature point, valid feature points are screened, and feature matching is performed based on the valid feature points.

[0009] In one possible implementation, the step of performing optical flow verification on the predicted points of the current frame in the fine-tracking optical flow according to a preset verification-level neighborhood window size, and determining the verification optical flow, includes: The Lucas-Kanade (LK) optical flow algorithm, which employs a two-layer image pyramid structure, uses the current frame as the image before motion and the previous frame as the image after motion. It performs layer-by-layer verification-level optical flow tracking on the prediction points of the current frame in the fine-tracking optical flow to obtain the verification optical flow.

[0010] In one possible implementation, calculating the forward and backward errors between the verification optical flow and the fine tracking optical flow, and determining multiple initial screening feature points based on the forward and backward errors, includes: The forward and backward errors between the predicted points of the previous frame in the verification optical flow and the original points of the previous frame in the fine tracking optical flow are calculated using the Euclidean norm. Based on a preset forward and backward error threshold, multiple preliminary feature points are obtained.

[0011] In one possible implementation, determining the confidence value of the initial screening feature points based on their Harris mass fraction and photometric error, and then filtering valid feature points based on their confidence values, includes: Calculate the photometric error of the initial screening feature points between the previous frame and the current frame; Based on the preset mass fraction weight and photometric error weight, the weighted sum of the Harris mass fraction and photometric error of the initial screening feature points is calculated to obtain the confidence value of the initial screening feature points. Based on the confidence level, the effective feature points are selected from the plurality of initial screening feature points.

[0012] In one possible implementation, the step of performing coarse-level optical flow tracking and determining the coarse-level optical flow based on the size of the coarse-level neighborhood window of each pixel includes: The LK optical flow algorithm, which employs a three-layer image pyramid structure, performs layer-by-layer coarse tracking optical flow tracking based on the size of the coarse tracking level neighborhood window of each pixel to obtain the coarse tracking optical flow.

[0013] In one possible implementation, the step of performing fine-tracking optical flow tracking based on the coarse tracking optical flow and a preset fine-tracking level neighborhood window size, and determining the fine-tracking optical flow, includes: The coarse tracking optical flow is determined as the initial estimated optical flow for the fine tracking-level optical flow tracking; The LK optical flow algorithm, which employs a two-layer image pyramid structure, performs layer-by-layer fine-tracking optical flow tracking based on a preset fine-tracking level neighborhood window size to obtain the fine-tracking optical flow.

[0014] In a second aspect, embodiments of the present invention provide a terminal, including a memory and a processor, wherein the memory stores a computer program and the processor executes the steps of the method as described in the first aspect or any implementation thereof.

[0015] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any implementation thereof.

[0016] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: In this embodiment of the invention, the Harris quality score of each pixel in the original image is determined by the Harris corner detection algorithm, which can quickly distinguish the trackability level of feature points, provide a basis for differentiated tracking, and reduce the interference of low-quality feature points on feature matching results. Based on the Harris quality score of each pixel, the size of the coarse tracking-level neighborhood window of each pixel is determined, which can adapt to the tracking needs of pixels of different quality and reduce the probability of feature point loss. Based on the size of the coarse tracking-level neighborhood window of each pixel, coarse tracking-level optical flow is performed to determine the coarse tracking optical flow, which can capture large displacement motion of feature points and output reliable large displacement feature matching results. Based on the coarse tracking optical flow and the preset fine tracking-level neighborhood window size, fine tracking-level optical flow is performed to determine the fine tracking optical flow, which can refine the matching accuracy based on the large displacement feature matching results and correct minor errors in the coarse tracking stage. Based on the fine tracking optical flow, feature points are determined and feature matching is performed, which can take into account both feature matching coverage and accuracy, and provide reliable feature matching data for the visual SLAM system. The embodiments of the present invention can achieve stable and continuous matching of feature points in image frames, which can effectively improve the stability and accuracy of feature matching in complex scenes and significantly enhance the robustness of visual SLAM systems. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the implementation process of a visual SLAM image feature matching method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0018] The present application will be described more clearly below with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the function of the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0019] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0020] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0021] In the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0023] Furthermore, the term "multiple" mentioned in the embodiments of this application should be interpreted as two or more.

[0024] In scenarios such as mobile robot navigation and drone operations, existing visual SLAM systems have significant limitations. They cannot adapt to varying camera speeds, and are prone to issues such as missing points or insufficient matching accuracy. Furthermore, mismatched features can directly lead to pose deviations. These problems, stemming from the difficulty in balancing tracking adaptability and result reliability, have become bottlenecks in improving the accuracy of visual SLAM methods. Therefore, developing a visual SLAM image feature matching method that can guarantee the stability of feature matching and the accuracy of feature localization in complex scenes has significant practical implications and application value.

[0025] See Figure 1 This invention provides a visual SLAM image feature matching method, detailed below: Step S101: Acquire the original image acquired by visual SLAM, and determine the Harris quality score of each pixel in the original image using the Harris corner detection algorithm.

[0026] Here, the original image acquired by visual SLAM includes multiple frames. For each frame, the Harris quality score of each pixel in the original image can be determined using the Harris corner detection algorithm. This quantitatively evaluates the characteristic of each pixel's surrounding grayscale values ​​changing significantly in multiple directions and exhibiting structural stability, thus clarifying the trackability of each pixel, distinguishing high-quality feature points, and improving the stability of the final feature matching. The principle of Harris corner detection is: when a local window moves in any direction on the image, the grayscale values ​​of the pixels within the window change significantly. The Harris quality score (response value R) is a quantitative indicator of the "degree of grayscale change of a pixel within a local window," and a corresponding R value can be calculated for each pixel.

[0027] In one possible implementation, after acquiring the original image from visual SLAM, the image frames of the original image can be preprocessed. For example, the original image can first be converted into a grayscale image, then Gaussian filtering can be used for image denoising, and finally histogram equalization algorithm can be used to enhance the contrast of the image.

[0028] Step S102: Determine the coarse tracking level neighborhood window size for each pixel based on the Harris quality score of each pixel.

[0029] In one possible implementation, the size of the coarse tracking level neighborhood window for each pixel is determined based on the Harris quality score of each pixel, including: For pixels with a Harris quality score greater than or equal to a preset threshold, the size of the coarse tracking level neighborhood window is determined to be a first preset value. For pixels with a Harris quality score less than a preset threshold, the size of the coarse tracking level neighborhood window is determined to be a second preset value; The first preset value is less than the second preset value.

[0030] In this embodiment of the invention, based on the Harris quality score of a pixel, a differentiated window can be adapted for pixels with different trackability. For pixels with lower quality scores, a larger neighborhood window is used for coarse tracking, balancing tracking efficiency and coverage, reducing the loss of low-scoring feature points, and improving the reliability of coarse tracking.

[0031] Step S103: Perform coarse tracking optical flow tracking based on the size of the coarse tracking level neighborhood window of each pixel to determine the coarse tracking optical flow.

[0032] In one possible implementation, coarse-level optical flow tracking is performed based on the size of the coarse-level neighborhood window for each pixel to determine the coarse-level optical flow, including: The LK optical flow algorithm with a three-layer image pyramid structure is adopted. Based on the size of the coarse tracking level neighborhood window of each pixel, coarse tracking level optical flow is performed layer by layer to obtain coarse tracking optical flow.

[0033] Step S104: Based on the coarse tracking optical flow and the preset fine tracking level neighborhood window size, perform fine tracking level optical flow tracking to determine the fine tracking optical flow; In one possible implementation, fine-tracking optical flow is performed based on the coarse tracking optical flow and a preset fine-tracking level neighborhood window size to determine the fine-tracking optical flow, including: The coarse tracking optical flow is determined as the initial estimated optical flow for fine tracking-level optical flow tracing; The LK optical flow algorithm, which employs a two-layer image pyramid structure, performs layer-by-layer fine-tracking optical flow tracking based on the preset fine-tracking level neighborhood window size to obtain the fine-tracking optical flow.

[0034] The embodiments of the present invention, through hierarchical optical flow tracking and initial estimation optimization, take into account both large displacement acquisition and positioning accuracy, providing a stable and accurate tracking foundation for feature matching.

[0035] Step S105: Determine feature points and perform feature matching based on fine tracking optical flow; wherein, the size of the coarse tracking level neighborhood window is larger than the size of the fine tracking level neighborhood window.

[0036] In some embodiments, when a visual SLAM system operates in scenarios with complex camera motion or significant environmental interference, the fine-tracking optical flow may suffer from tracking deviations or the inclusion of low-quality prediction points. To avoid these problems leading to a decrease in feature matching accuracy, further improvements and optimizations to the fine-tracking optical flow tracking are needed. Therefore, determining feature points and performing feature matching based on the fine-tracking optical flow may include: Based on the preset verification-level neighborhood window size, optical flow verification is performed on the prediction points of the current frame in the fine tracking optical flow to determine the verification optical flow; Calculate the forward and backward errors of the verification optical flow and the fine tracking optical flow, and determine multiple initial screening feature points from the prediction points based on the forward and backward errors; Based on the Harris mass fraction and photometric error of the initial screening feature points, the confidence values ​​of the initial screening feature points are determined. Based on the confidence values ​​of each initial screening feature point, valid feature points are selected, and feature matching is performed based on the valid feature points.

[0037] In this embodiment of the invention, the predicted points obtained by fine tracking are first verified, the forward and backward errors are calculated to initially screen feature points, and then the confidence level is determined by combining the Harris quality score and photometric error of the initially screened feature points. After screening effective feature points, matching is completed. Through multi-layer verification and quality assessment, mismatches and low-quality feature points are eliminated, thereby improving the accuracy and reliability of feature matching and avoiding the cumulative error of pose estimation caused by inter-frame feature mismatches, interference from low-quality feature points, etc., which could lead to the problem of positioning results deviating from the true position and map construction distortion.

[0038] In this embodiment of the invention, the Harris quality score of each pixel in the original image is determined by the Harris corner detection algorithm, which can quickly distinguish the trackability level of feature points, provide a basis for differentiated tracking, and reduce the interference of low-quality feature points on feature matching results. Based on the Harris quality score of each pixel, the size of the coarse tracking-level neighborhood window of each pixel is determined, which can adapt to the tracking needs of pixels of different quality and reduce the probability of feature point loss. Based on the size of the coarse tracking-level neighborhood window of each pixel, coarse tracking-level optical flow is performed to determine the coarse tracking optical flow, which can capture large displacement motion of feature points and output reliable large displacement feature matching results. Based on the coarse tracking optical flow and the preset fine tracking-level neighborhood window size, fine tracking-level optical flow is performed to determine the fine tracking optical flow, which can refine the matching accuracy based on the large displacement feature matching results and correct minor errors in the coarse tracking stage. Based on the fine tracking optical flow, feature points are determined and feature matching is performed, which can take into account both feature matching coverage and accuracy, and provide reliable feature matching data for the visual SLAM system. The embodiments of the present invention can achieve stable and continuous matching of feature points in image frames, which can effectively improve the stability and accuracy of feature matching in complex scenes and significantly enhance the robustness of visual SLAM systems.

[0039] In some embodiments, the ORB algorithm can be performed on keyframes in the original image to detect feature points and perform feature matching; the above-described visual SLAM image feature matching method can be performed on non-keyframes in the original image.

[0040] By leveraging the rotation and scale invariance of the ORB algorithm, the accuracy of keyframe feature matching is ensured, identifying the correlations of core features and supporting pose estimation and map construction. Full-frame feature matching is achieved by fusing feature matching results from keyframes and non-keyframes, covering diverse scenario requirements and effectively improving the stability and accuracy of feature matching in complex scenarios. This provides coherent and reliable feature association data for system localization and mapping, significantly enhancing the robustness of the visual SLAM system.

[0041] In one possible implementation, keyframes can be set based on the pixel tracking results after visual SLAM feature matching: when the pixel movement speed is less than the average value, the next original image frame is a keyframe to ensure the accuracy of map updates and positioning references; when the pixel movement speed is greater than or equal to the average value, the next original image frame is a non-keyframe to maintain the continuity of feature matching and adapt to the pose calculation and map building requirements in different scenarios.

[0042] In one possible implementation, keyframes can be set according to a preset threshold for the proportion of newly added feature points: if among the feature points that are successfully matched in the current frame, there are newly added feature points that were not successfully matched in the previous frame, and the proportion of newly added feature points in the current frame among all successfully matched feature points in the current frame is greater than the preset threshold, then the next original image frame is a keyframe; otherwise, it is a non-keyframe. This ensures timely feature matching based on new environmental information while controlling the number of keyframes, balancing feature richness and system operating efficiency.

[0043] In one possible implementation, keyframes can be set according to a preset keyframe interval: whenever the number of acquired raw image frames reaches the preset keyframe interval, the next acquired raw image frame is a keyframe, otherwise it is a non-keyframe, in order to prevent the problem of tracking loss caused by long-term lack of keyframes due to no significant changes in the scene.

[0044] In one possible implementation, optical flow verification is performed on the predicted points of the current frame in the fine-tracking optical flow according to a preset verification-level neighborhood window size to determine the verification optical flow, including: The LK optical flow algorithm, which employs a two-layer image pyramid structure, uses the current frame as the image before motion and the previous frame as the image after motion. It performs layer-by-layer verification-level optical flow tracking on the prediction points of the current frame in the fine tracking optical flow to obtain the verification optical flow.

[0045] In this embodiment of the invention, verification-level optical flow tracking is performed layer by layer on the current frame prediction point of the fine-tracking optical flow. The LK optical flow algorithm with a two-layer image pyramid structure is used to adapt to pixel displacements of different scales. The tracking deviation is corrected by reverse tracking verification, thereby improving the accuracy of prediction point positioning.

[0046] In some embodiments, the verification optical flow can reversely deduce the predicted position of a predicted point in the current frame in the previous frame. This may differ from the original point in the fine tracking optical flow of the previous frame. Therefore, calculating the forward and backward errors between the verification optical flow and the fine tracking optical flow, and determining multiple initial screening feature points based on these forward and backward errors, may include: The forward and backward errors between the predicted points in the previous frame of the optical flow and the original points in the previous frame of the fine-tracking optical flow are verified by Euclidean norm calculation. Based on the preset forward and backward error threshold, multiple preliminary feature points are obtained.

[0047] In this embodiment of the invention, the positional deviation between the optical flow and the fine-tracking optical flow is verified by Euclidean distance quantization. Initially screened feature points with acceptable errors are selected, and fine-tracking results with large tracking deviations are eliminated to ensure the accuracy of feature matching.

[0048] In one possible implementation, the confidence value of the initial screening feature points is determined based on their Harris quality fraction and photometric error. Then, based on the confidence values ​​of each initial screening feature point, valid feature points are selected, including: Calculate the photometric error of the initial screening feature points between the previous frame and the current frame; Based on the preset mass fraction weight and photometric error weight, the weighted sum of Harris mass fraction and photometric error of the initial screening feature points is calculated to obtain the confidence value of the initial screening feature points. Based on the confidence level, valid feature points are selected from multiple initial screening feature points.

[0049] This invention improves matching accuracy and reliability by using a weighted fusion of feature point quality and photometric consistency indices to screen effective feature points.

[0050] Based on the above, the visual SLAM image feature matching method will be described in detail in the following specific embodiment: (1) Acquisition and preprocessing of raw images The original images acquired by visual SLAM are preprocessed. First, the images are converted to grayscale. Then, Gaussian filtering is used for image denoising. Finally, histogram equalization is used to enhance image contrast, according to the following transformation formula:

[0051] In the formula, It is the number of pixels with the current grayscale value. It is the sum of the number of pixels with all grayscale values. It is the number of grayscale values ​​present in the image. It is the cumulative probability, then... and Multiplying them together will give you a new grayscale value.

[0052] (2) Pixel quality assessment based on Harris corner detection After image preprocessing, Harris corner detection is introduced to evaluate the quality of all pixels, determining a Harris quality score for each feature point. This score reflects the strength of corner characteristics in the region surrounding the feature point. A higher quality score indicates that the feature point is located in a corner region with rich texture and significant gradient changes, resulting in good trackability. Conversely, a lower quality score suggests that the feature point may be located on an edge or in a flat area, leading to relatively poor tracking stability.

[0053] Harris corner detection essentially involves moving a local window across an image to calculate the change in grayscale values ​​within that window, thereby determining whether a corner exists in that region. For an image... , with point Centered on, and in and Move in different directions , The change in grayscale value caused by window movement The formula is as follows:

[0054] In the formula, For window functions, and Points and points The grayscale value. For Performing a Taylor expansion, we get:

[0055] In the formula, and They are respectively exist and The derivative in the direction represents the change in grayscale value. It can be represented as:

[0056]

[0057] In the formula, For point The autocorrelation matrix at a given location can be obtained through the matrix... To define the Harris quality score of a pixel :

[0058] In the formula, For matrix The determinant, For matrix traces, These are constants preset based on experience.

[0059] (3) Coarse tracking-level optical flow tracking determines the initial estimate of pixel displacement. The keyframes are still handled using the same method as in ORB_SLAM3, which is a feature-based matching method and camera motion is calculated by minimizing the reprojection error.

[0060] For non-key frames, this embodiment of the invention employs the Pyramid LK optical flow algorithm. The LK optical flow algorithm is based on the assumptions of constant brightness and small motion. It calculates the motion vectors of pixels between consecutive frames by solving the pixel grayscale change equation, thus achieving feature point matching and pixel displacement calculation. The Pyramid LK optical flow algorithm, on the other hand, constructs a multi-scale image pyramid for layer-by-layer tracking based on the LK optical flow algorithm. Optical flow tracking can be performed by controlling the image size, thereby improving the robustness and applicability of optical flow tracking.

[0061] The embodiments of the present invention construct a multi-level pyramid LK optical flow to process non-key frames, which is divided into coarse tracking level and fine tracking level.

[0062] The coarse tracking stage employs the LK optical flow algorithm based on a three-layer image pyramid structure, using a multi-scale image processing method from coarse to fine. This approach can handle large inter-frame displacements caused by rapid camera movement. A 31*31 pixel neighborhood window captures the motion of feature points over a larger spatial range, improving feature point matching accuracy and providing reliable initial estimates for the fine tracking stage. Based on the Harris quality evaluation results, in the coarse tracking stage, this embodiment uses adaptive optical flow tracking parameters. When the quality score of a feature point is below 0.05, the neighborhood window is expanded from 31*31 pixels to 41*41 pixels, providing a larger search range for low-quality pixels.

[0063] (4) Preliminary determination of feature points by fine-tracking optical flow tracing The fine tracking stage receives the coarse tracking optical flow from the coarse tracking stage as the initial estimated optical flow for fine tracking. Based on this, it uses the LK optical flow algorithm with two image pyramids to calculate the optical flow on higher resolution images, which can handle small-amplitude inter-frame displacements caused by camera motion. It uses a small 15*15 neighborhood window for iterative calculation, which can refine the motion of feature points within a smaller spatial range, thereby obtaining more accurate tracking precision.

[0064] The validation layer performs forward and backward consistency verification on the tracking results output by the fine tracking stage. It tracks the predicted points in the fine tracking optical flow from the current frame image back to the previous frame image, compares the original point positions with the positions returned from backward tracking, calculates the forward and backward position error of the point using the L2 norm, and sets a validation threshold to remove points with large errors, thus obtaining the initial screened feature points. During backward tracking, a two-layer pyramid LK optical flow algorithm is used, with a 15*15 pixel window for tracking, to ensure that the forward and backward optical flow tracking uses the same accuracy standard as the fine tracking process, guaranteeing the accuracy of the validation.

[0065] (5) Assess confidence level and screen effective feature points In the confidence assessment, the Harris quality score and photometric error of the initial screening feature points are calculated. The Harris quality score is pre-weighted at 70%, and the photometric error at 30%.

[0066] Harris quality score reflects the trackability of feature points, is directly related to image texture and gradient, and is rotation-invariant, so it can be determined before tracking. Photometric error, on the other hand, is obtained after optical flow tracking, is affected by the image quality and motion speed of the current frame, and may contain random noise. Therefore, for confidence scores, the above-mentioned weighting of these two factors can be set appropriately according to the needs of feature point selection, either to avoid misjudgments due to temporary disturbances when image quality is poor, or to ensure the consistency of feature points when image quality is good.

[0067] When the confidence level is below 0.3, the corresponding initial screening feature points are marked as failed matches. By filtering based on confidence level, at least 10 feature points are retained to ensure the robustness of pose estimation in visual SLAM.

[0068] In this embodiment of the invention, adaptive neighborhood windows and hierarchical optical flow tracking are used to adapt to changes in image quality caused by camera movement and the matching difficulties brought about by feature points of different quality, thereby improving the stability of image frame tracking and the ability to capture large displacements. Through forward and backward consistency verification and confidence-weighted screening, mismatches and low-quality feature points are effectively eliminated, significantly improving the accuracy of feature matching. The quality of image frame feature association is optimized to reduce tracking drift and enhance the robustness of feature matching in the visual SLAM system.

[0069] The specific operation of the pyramid LK optical flow algorithm used in this embodiment of the invention is as follows: An image pyramid is built for the previous frame. The top image has the lowest resolution, and the resolution of the images increases from the top down. The original image is at the bottom.

[0070] Starting from the top level, by minimizing the matching error within the neighborhood of each point and The optical flow at each point in the image can be calculated:

[0071] In the formula, It is the displacement of a pixel between two consecutive frames of an image, i.e., the optical flow of the pixel; and These are two integers, representing a range of size 1 around the pixel. The neighborhood window; It is the position of the pixel in the previous frame of the image; and These represent the grayscale values ​​of pixels in the previous and next frames of the image, respectively.

[0072] Each image layer is scaled to half the size of the previous image. In the coarse tracking stage, there are a total of 3 scaling layers, with layer 0 being the original image. Therefore, the position of each pixel in each image layer is:

[0073] In the formula, The position of the pixel in the original image. For the pixel at the th The location in the layer image.

[0074] The optical flow calculation results from the top layer can be fed back to the next layer as the initial estimated optical flow for that layer. :

[0075] In the formula, For the first Estimated optical flow of the layer For the first The remaining light flow of the layer. The top layer of the pyramid. Layer, estimated optical flow initialized to 0, i.e.:

[0076] As you descend the pyramid, the optical flow calculation results from the previous section are repeatedly fed back to the lower levels. At the bottom of the pyramid, you will find:

[0077] In the formula, and For the underlying estimated optical flow and residual optical flow, This is the final optical flow result. In each layer, the optical flow calculation at each point is based on minimizing the sum of matching errors of all points in the neighborhood:

[0078] The optical flow calculation for each layer utilizes the least squares method, calculating the derivative of the sum of matching errors in the neighborhood. The sum of matching errors is minimized when the derivative is zero. The derivative is:

[0079] In the formula, express To each and Differentiation of coordinates.

[0080] See Figure 2 The diagram illustrates a terminal 2 provided in an embodiment of the present invention, which is described in detail below: like Figure 2As shown, the terminal 2 in this embodiment includes a processor 20 and a memory 21. The memory 21 stores a computer program 22. When the processor 20 executes the computer program 22, it implements the steps in the various method embodiments described above.

[0081] For example, computer program 22 may be divided into one or more units, which are stored in memory 21 and executed by processor 20 to perform the present invention. These one or more units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 22 in terminal 2.

[0082] Terminal 2 may include, but is not limited to, processor 20 and memory 21. Those skilled in the art will understand that... Figure 2 This is merely an example of terminal 2 and does not constitute a limitation on terminal 2. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal 2 may also include input / output devices, network access devices, buses, etc.

[0083] The processor 20 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0084] The memory 21 can be an internal storage unit of the terminal 2, such as the hard disk or memory of the terminal 2. The memory 21 can also be an external storage device of the terminal 2, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal 2. Furthermore, the memory 21 can include both internal storage units and external storage devices of the terminal 2. The memory 21 is used to store the computer program 22 and other programs and data required by the terminal 2. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0085] For the sake of simplicity and clarity, only the above functional unit divisions are used as examples. In practical applications, the above functions can be assigned to different functional units as needed. The above units can be implemented in hardware, software, or a combination of both.

[0086] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0087] This invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0088] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0089] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0090] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A visual SLAM image feature matching method, characterized in that, include: The original image acquired by visual SLAM is obtained, and the Harris quality score of each pixel in the original image is determined by the Harris corner detection algorithm. The size of the coarse tracking level neighborhood window for each pixel is determined based on the Harris quality score of each pixel. Based on the size of the coarse tracking level neighborhood window of each pixel, coarse tracking level optical flow is performed to determine the coarse tracking optical flow; Based on the coarse tracking optical flow and the preset fine tracking level neighborhood window size, fine tracking level optical flow tracking is performed to determine the fine tracking optical flow; Feature points are determined and feature matching is performed based on the fine tracking optical flow; wherein the size of the coarse tracking level neighborhood window is larger than the size of the fine tracking level neighborhood window.

2. The visual SLAM image feature matching method according to claim 1, characterized in that, The step of determining the coarse tracking level neighborhood window size for each pixel based on its Harris quality score includes: For pixels whose Harris quality score is greater than or equal to a preset threshold, the size of the coarse tracking level neighborhood window is determined to be a first preset value; For pixels whose Harris quality score is less than a preset threshold, the size of the coarse tracking level neighborhood window is determined to be a second preset value; The first preset value is less than the second preset value.

3. The visual SLAM image feature matching method according to claim 1, characterized in that, The step of determining feature points and performing feature matching based on the fine-tracking optical flow includes: Based on the preset verification-level neighborhood window size, optical flow verification is performed on the prediction points of the current frame in the fine tracking optical flow to determine the verification optical flow; Calculate the forward and backward errors between the verification optical flow and the fine tracking optical flow, and determine multiple initial screening feature points from the prediction points based on the forward and backward errors; Based on the Harris mass fraction and photometric error of the initial screening feature points, the confidence value of the initial screening feature points is determined. Based on the confidence value of each initial screening feature point, valid feature points are screened, and feature matching is performed based on the valid feature points.

4. The visual SLAM image feature matching method according to claim 3, characterized in that, The step of verifying the optical flow of the predicted points in the current frame of the fine-tracking optical flow according to the preset verification-level neighborhood window size, and determining the verification optical flow, includes: The LK optical flow algorithm, which employs a two-layer image pyramid structure, uses the current frame as the image before motion and the previous frame as the image after motion. It performs layer-by-layer verification-level optical flow tracking on the prediction points of the current frame in the fine tracking optical flow to obtain the verification optical flow.

5. The visual SLAM image feature matching method according to claim 3, characterized in that, The calculation of the forward and backward errors between the verification optical flow and the fine tracking optical flow, and the determination of multiple initial screening feature points based on the forward and backward errors, includes: The forward and backward errors between the predicted points of the previous frame in the verification optical flow and the original points of the previous frame in the fine tracking optical flow are calculated using the Euclidean norm. Based on a preset forward and backward error threshold, multiple preliminary feature points are obtained.

6. The visual SLAM image feature matching method according to claim 3, characterized in that, The step of determining the confidence value of the initial screening feature points based on the Harris mass fraction and photometric error, and then selecting valid feature points based on the confidence values ​​of each initial screening feature point, includes: Calculate the photometric error of the initial screening feature points between the previous frame and the current frame; Based on the preset mass fraction weight and photometric error weight, the weighted sum of the Harris mass fraction and photometric error of the initial screening feature points is calculated to obtain the confidence value of the initial screening feature points. Based on the confidence level, the effective feature points are selected from the plurality of initial screening feature points.

7. The visual SLAM image feature matching method according to any one of claims 1-6, characterized in that, The step of performing coarse-level optical flow tracking based on the coarse-level neighborhood window size of each pixel to determine the coarse-level optical flow includes: The LK optical flow algorithm, which employs a three-layer image pyramid structure, performs layer-by-layer coarse tracking optical flow tracking based on the size of the coarse tracking level neighborhood window of each pixel to obtain the coarse tracking optical flow.

8. The visual SLAM image feature matching method according to any one of claims 1-6, characterized in that, The step of performing fine-tracking optical flow tracking based on the coarse tracking optical flow and the preset fine-tracking level neighborhood window size, and determining the fine-tracking optical flow, includes: The coarse tracking optical flow is determined as the initial estimated optical flow for the fine tracking-level optical flow tracking; The LK optical flow algorithm, which employs a two-layer image pyramid structure, performs layer-by-layer fine-tracking optical flow tracking based on a preset fine-tracking level neighborhood window size to obtain the fine-tracking optical flow.

9. A terminal, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images