An image matching control method

Abstract

The application discloses an image matching control method, and an execution subject of the image matching control method is a robot fixedly equipped with a camera and an inertial sensor; in the image matching control method, the robot collects a current frame image through the camera and obtains inertial data through the inertial sensor; on the basis of the inertial data, a first feature point pair is selected from feature points of the current frame image and feature points of all reference frame images in a sliding window by using an epipolar constraint error value; then, on the basis of the inertial data, a second feature point pair is selected from the first feature point pair by using a depth value of the feature points; further, a third feature point pair can be selected from the second feature point pair according to a similarity of descriptors corresponding to the second feature point pair; and finally, a residual error formula is introduced between the feature point pairs that have been selected, a least square method is used to fit an inertial compensation value from the residual error formula, and the inertial compensation value is used to correct the inertial data.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to an image matching control method. Background Technology

[0002] Visual-inertial odometry (VIO), sometimes called visual-inertial system (VINS), is an algorithm that integrates camera and inertial measurement unit (IMU) sensor data to achieve SLAM (Simultaneous Localization and Mapping). Traditional VIO schemes begin with a pure visual SFM (Structure from Motion) using feature points in the initialization phase. This structure is then loosely coupled and aligned with IMU pre-integrated measurements to recover the metric scale, velocity, gravitational acceleration direction, and IMU bias. SLAM refers to a robot moving from an unknown location in an unknown environment, performing self-localization based on position estimation and a map during movement, and simultaneously building an incremental map based on its self-localization to achieve autonomous localization and navigation. Visual-inertial odometry, as a crucial component of visual SLAM methods, largely determines the accuracy and speed of visual SLAM. Generally, the encoder disks mounted on the robot's wheels provide distance information, and the IMU provides angle information. When the robot calculates the pose of each frame of image by setting a sliding window, it needs to repeatedly search and match all feature points in each frame of image within the sliding window. The image matching efficiency is low, which affects the computational efficiency of the robot's localization and navigation. Summary of the Invention

[0003] To address the aforementioned technical deficiencies, this invention discloses an image matching control method, the specific technical solution of which is as follows:

[0004] An image matching control method is disclosed, wherein the executing entity of the image matching control method is a robot with a fixedly mounted camera and an inertial sensor; the image matching control method includes: step S1, the robot acquires the current frame image through the camera and obtains inertial data through the inertial sensor; step S2, based on the inertial data, using the epipolar constraint error value, a first feature point pair is selected from the feature points of the current frame image and the feature points of all reference frame images within a sliding window; wherein the sliding window is set to be filled with at least one pre-acquired frame image; step S3, in the inertial data... Based on this, using the depth values ​​of the feature points, a second feature point pair is selected from the first feature point pair; in step S4, based on the similarity of the descriptors corresponding to the second feature point pair, a third feature point pair is selected from the second feature point pair; in step S5, a residual formula is introduced between the third feature point pairs, and then the least squares method is used to fit the inertial compensation value from the residual formula, and then the inertial compensation value is used to correct the inertial data; then the corrected inertial data is updated to the inertial data described in step S2, and the feature points of the third feature point pair described in step S4 in the current frame image are updated to the current feature points described in step S2. The feature points of the frame image are updated, and the feature points of the third feature point pair described in step S4 are updated to the feature points of all reference frame images within the sliding window described in step S2; Step S6, steps S2 and S3 are repeated until the number of repetitions reaches the preset number of iterations, and then matching frame images are selected from the reference frame images within the sliding window based on the number of feature points of the second feature point pair in each reference frame image; wherein, each time steps S2 and S3 are repeated, the robot introduces residual features between the second feature point pairs selected in the most recently executed step S3. The inertial compensation value is then fitted from the residual formula using the least squares method. The inertial compensation value is then used to correct the inertial data. The corrected inertial data is then updated to the inertial data described in step S2. The feature points included in the second feature point pair selected in the latest step S3 are updated to the feature points of the current frame image and the feature points of all reference frame images within the sliding window as described in step S2. In step S7, based on the epipolar constraint error value and the number of feature points of the second feature point pair in each matching frame image, the optimal matching frame image is selected from all matching frame images.

[0005] Further, in step S2, the method for selecting the first feature point pair from the feature points of the current frame image and all reference frame images within the sliding window based on the inertial data and using the epipolar constraint error value includes: the robot calculating the epipolar constraint error value of the feature point pair; when the epipolar constraint error value of the feature point pair is greater than or equal to a preset pixel distance threshold, the feature point pair is marked as an incorrectly matched point pair; when the epipolar constraint error value of the feature point pair is less than the preset pixel distance threshold, the feature point pair is marked as the first feature point pair and the first feature point pair is selected; wherein each feature point pair is configured to consist of a feature point of the current frame image and a feature point of the reference frame image, and each feature point of the current frame image forms a feature point pair with each feature point in each reference frame image within the sliding window.

[0006] Further, in step S2, when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix; then, the robot controls the first rotation matrix to transform the normalized vector of the feature points of the current frame image to the coordinate system of the reference frame image, obtaining the first one vector; then, the robot controls the first translation vector to cross-multiply the first one vector, obtaining the first two vectors; then, the robot controls the normalized vector of the feature points in the reference frame image within the sliding window to dot-multiply the first two vectors, and then sets the result of the dot-multiply as the epipolar constraint error value of the corresponding feature point pair; or, in step S2, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as the first one vector and the rotation matrix of the reference frame image relative to the current frame image as the first two vectors. When performing the matrix operation, the robot marks the translation vector of the reference frame image relative to the current frame image as the second translation vector, and the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix. Then, the robot controls the second rotation matrix to transform the normalized planar vector of the feature points of the reference frame image within the sliding window to the coordinate system of the current frame image, obtaining the second one vector. Then, the robot controls the second translation vector to cross-multiply the second one vector, obtaining the second two vector. Then, the robot controls the normalized planar vector of the feature points in the current frame image to dot-multiply the second two vector, and then sets the result of the dot-multiply as the epipolar constraint error value of the corresponding feature point pair. Here, the normalized vector of the feature points of the current frame image is the vector formed by the normalized planar coordinates of the feature points of the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the reference frame image is the vector formed by the normalized planar coordinates of the feature points of the reference frame image relative to the origin of the coordinate system of the reference frame image.

[0007] Further, in step S3, based on the inertial data, the method for selecting a second feature point pair from the first feature point pair using the depth values ​​of the feature points includes: the robot calculating the ratio of the depth values ​​of the feature points of the first feature point pair selected in step S2 in the current frame image to the depth values ​​of the feature points of the first feature point pair in the reference frame image; when the ratio of the depth values ​​of the feature points of the first feature point pair in the current frame image to the depth values ​​of the feature points of the first feature point pair in the reference frame image is within a preset ratio threshold range, the first feature point pair is marked as a second feature point pair and the second feature point pair is selected; when the ratio of the depth values ​​of the feature points of the first feature point pair in the current frame image to the depth values ​​of the feature points of the first feature point pair in the reference frame image is not within the preset ratio threshold range, the first feature point pair is marked as an incorrectly matched point pair.

[0008] Further, in step S3, when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix. The robot controls the first rotation matrix to transform the normalized vector of the feature points of the first feature point pair in the current frame image to the coordinate system of the reference frame image, obtaining the first vector; then, the robot controls the normalized vector of the feature points of the first feature point pair in the reference frame image to cross-multiply the first vector, obtaining... The first two vectors are used; simultaneously, the normalized vector of the first feature point in the reference frame image is cross-multiplied by the first translation vector, and then the cross-multiplication result is inverted to obtain the first three vectors; then the product of the first three vectors and the inverse of the first two vectors is set as the depth value of the feature point of the first feature point in the current frame image, and marked as the first depth value, representing the distance between the 3D point detected by the camera and the optical center when the camera captures the current frame image; then the sum of the product of the first three vectors and the first depth value and the first translation vector is marked as the first four vectors; then the first four vectors are combined with the normalized vector of the feature point of the first feature point in the reference frame image. The product of the inverse vectors is set as the depth value of the first feature point pair in the reference frame image, and marked as the second depth value, representing the distance between the same 3D point and the optical center when the camera captures the reference frame image; or, in step S3, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot records the translation vector of the reference frame image relative to the current frame image as the second translation vector, and the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix. The robot controls the second rotation matrix to store the first feature point pair in the reference frame image... The normalized vector of the feature point is transformed into the coordinate system of the current frame image to obtain the second vector; then, the normalized vector of the first feature point in the current frame image is cross-multiplied by the second vector to obtain the second vector; simultaneously, the normalized vector of the first feature point in the current frame image is cross-multiplied by the second translation vector, and the cross-multiplication result is inverted to obtain the second vector; then, the product of the second vector and the inverse of the second vector is set as the depth value of the first feature point in the reference frame image, and marked as the second depth value, which represents the distance between the 3D point detected by the camera and the optical center when the camera acquires the reference frame image;Then, the sum of the product of the second vector and the second depth value and the second translation vector is labeled as the second quad vector. The product of this second quad vector and the inverse of the normalized vector of the first feature point pair in the current frame image is then set as the depth value of the feature point of the first feature point pair in the current frame image, and labeled as the first depth value, representing the distance between the same 3D point and the optical center when the camera captures the current frame image. Here, the normalized vector of the first feature point pair in the current frame image is the vector formed by the normalized planar coordinates of the first feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the first feature point pair in the reference frame image is the vector formed by the normalized planar coordinates of the first feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image.

[0009] Further, in step S4, the method for selecting a third feature point pair from the second feature point pairs based on the similarity of the descriptors corresponding to the second feature point pairs includes: for the current frame image and each reference frame image within the sliding window, the robot calculates the similarity between the descriptors of the feature points of the second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image; when the similarity between the descriptors of the feature points of the second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image is greater than the similarity between the descriptor of the current frame image and the descriptor of the reference frame image where the feature points of the second feature point pair are located... When the minimum value is reached, the second feature point pair is marked as the third feature point pair and the third feature point pair is selected. The descriptor of the reference frame image containing the feature points of the second feature point pair is the descriptor of all feature points within the reference frame image that make up the second feature point pair. The descriptor of the current frame image is the descriptor of the feature points within the current frame image that, together with the feature points in the reference frame image containing the feature points of the second feature point pair, form the second feature point pair. The similarity between the descriptors corresponding to the second feature point pair is represented by the Euclidean distance or Hamming distance between the descriptors of the feature points in the current frame image and the descriptors of the feature points in the corresponding reference frame image within the sliding window.

[0010] Furthermore, step S4 also includes: whenever the robot searches for all feature points that form a second feature point pair between the current frame image and a reference frame image within the sliding window, if the number of third feature point pairs counted by the robot within the reference frame image is less than or equal to a first preset point threshold, then it is determined that the current frame image and the reference frame image have failed to match, and the reference frame image is set as a mismatched reference frame image; if the number of third feature point pairs counted by the robot within the reference frame image is greater than the first preset point threshold, then it is determined that the current frame image and the reference frame image have successfully matched; wherein, when the robot determines that the current frame image and all reference frame images within the sliding window have failed to match, it is determined that the robot's tracking using the window matching method has failed, and then the robot clears the images within the sliding window.

[0011] Furthermore, the robot marks the line connecting the optical center of the current frame image captured by the camera with the feature points of the same preset feature point pair within the current frame image as the first observation line, and the line connecting the optical center of the reference frame image captured by the camera with the feature points of the same preset feature point pair within the reference frame image as the second observation line. The intersection of the first and second observation lines is then marked as the target detection point. The preset feature point pair, the optical center of the current frame image captured by the camera, and the optical center of the reference frame image captured by the camera are all located in the same plane; or, the optical center of the current frame image captured by the camera, the optical center of the reference frame image captured by the camera, and the target detection point are all located in the same plane; this same plane is the polar plane; the robot records the intersection of the polar plane and the current frame image as the line in the imaging plane of the current frame image. The epipolar line is defined as the line of intersection between the epipolar plane and the reference frame image, which is denoted as the epipolar line of the imaging plane of the reference frame image. Within the same preset feature point pair, a feature point from the current frame image is transformed into a first projection point in the reference frame image, and its coordinates are defined as the first coordinate. The distance from the first projection point to the epipolar line in the imaging plane of the reference frame image is represented as the first residual value. Within the same preset feature point pair, a feature point from the reference frame image is transformed into a second projection point in the current frame image, and its coordinates are defined as the second coordinate. The distance from the second projection point to the epipolar line in the imaging plane of the current frame image is represented as the second residual value. In step S5, the preset feature point pair is the third feature point pair. In step S6, each time steps S2 and S3 are repeated, the preset feature point pair is the second feature point pair selected in the most recently executed step S3.

[0012] Further, in step S5 or step S6, when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot denotes the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix. The robot controls the first rotation matrix to transform the normalized vector of the feature points of the preset feature point pair in the current frame image to the coordinate system of the reference frame image, obtaining the first one vector; then, the robot controls the first translation vector to cross-multiply the first one vector to obtain the first two vectors, and forms the reference frame. The epipolar line in the imaging plane of the image; then, the square root of the sum of the squares of the horizontal and vertical coordinates of the first two vectors is obtained to obtain the magnitude of the epipolar line; simultaneously, the normalized vector of the feature point pair in the reference frame image is multiplied by the first two vectors, and the result of the multiplication is set as the epipolar constraint error value of the preset feature point pair; then, the ratio of the epipolar constraint error value of the preset feature point pair to the magnitude of the epipolar line is set as the first residual value; or, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot will relative to the reference frame image. The translation vector of the current frame image is denoted as the second translation vector, and the rotation matrix of the reference frame image relative to the current frame image is denoted as the second rotation matrix. The robot controls the second rotation matrix to transform the normalized vector of the feature points of the preset feature point pair in the reference frame image to the coordinate system of the current frame image, obtaining the second first vector. Then, the robot controls the second translation vector to cross-multiply the second first vector to obtain the second second vector, which forms the epipolar line in the imaging plane of the current frame image. Then, the robot takes the square root of the sum of the squares of the horizontal and vertical coordinates of the second second vector to obtain the magnitude of the epipolar line. At the same time, the robot controls the normalization of the feature points of the preset feature point pair in the current frame image. The normalized vector is multiplied by the second binary vector, and the result of the multiplication is set as the epipolar constraint error value of the preset feature point pair. Then, the ratio of the epipolar constraint error value of the preset feature point pair to the magnitude of the epipolar line is set as the second residual value. Here, the normalized vector of the feature points of the preset feature point pair in the current frame image is the vector formed by the normalized planar coordinates of the feature points of the preset feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the preset feature point pair in the reference frame image is the vector formed by the normalized planar coordinates of the feature points of the preset feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image.

[0013] Further, in step S5 or step S6, the method of introducing a residual formula between preset feature point pairs, fitting the inertial compensation value from the residual formula using the least squares method, and then using the inertial compensation value to correct the inertial data includes: when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the expression for multiplying the first rotation matrix with the normalized vector of the feature points of the preset feature point pair in the current frame image as the first transformation expression; then marks the expression for cross-multiplying the first translation vector with the first transformation expression as the first transformation expression; then marks the expression for dot-matrix multiplication of the normalized vector of the feature points of the preset feature point pair in the reference frame image with the first transformation expression as the first transformation expression. This is denoted as the first and third transformation formulas; the calculation result of the first and second transformation formulas is set to 0, forming a linear equation. The sum of squares of the coefficients of this linear equation in the horizontal and vertical coordinate dimensions is then calculated, and the square root of the sum is obtained to get the first square root. The reciprocal of the first square root is multiplied by the first and third transformation formulas to form the first and fourth transformation formulas. The calculation result of the first and fourth transformation formulas is then set as the first residual value, forming the first residual derivation formula, and the residual formula is introduced between the preset feature point pairs. The first residual derivation formula is then used to calculate the partial derivatives with respect to the first translation vector and the first rotation matrix to obtain the Jacobian matrix. The product of the inverse of the Jacobian matrix and the first residual value is then set as the inertia compensation value. Then the machine... Humans use inertial compensation values ​​to correct inertial data. The first residual derivation is the function model equation to be fitted using the least squares method, corresponding to the residual formula introduced between preset feature point pairs. The inertial compensation value is the fitting result obtained under the least squares method. Alternatively, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot marks the expression for multiplying the second rotation matrix by the normalized vector of the feature points of the preset feature point pair in the reference frame image as the second-first transformation expression; then, it marks the expression for cross-multiplying the second translation vector by the second-first transformation expression as the second-second transformation expression; and finally, it performs a dot-matrix multiplication of the normalized vector of the feature points of the preset feature point pair in the current frame image with the second-second transformation expression. The formula is marked as the second and third transformation formula; the calculation result of the second and second transformation formula is set to 0 to form a linear equation. Then, the sum of squares of the coefficients of the linear equation in the horizontal and vertical coordinate dimensions is calculated, and the square root of the sum of squares is calculated to obtain the second square root. The formula for multiplying the reciprocal of the second square root with the second and third transformation formula is set as the second and fourth transformation formula. Then, the calculation result of the second and fourth transformation formula is set as the second residual value to form the second residual derivation formula, and the residual formula is determined to be introduced between the preset feature point pairs. Then, the second residual derivation formula is controlled to take partial derivatives with respect to the second translation vector and the second rotation matrix to obtain the Jacobian matrix. Then, the product of the inverse of the Jacobian matrix and the second residual value is set as the inertia compensation value.The robot then corrects the inertial data using inertial compensation values. The second residual derivation is the function model equation to be fitted using the least squares method; the inertial compensation value is the fitting result obtained using the least squares method.

[0014] Further, regarding step S6, after the robot completes step S5, when step S2 is repeated for the first time, the robot calculates the epipolar constraint error value of each third feature point pair other than the mismatched reference frame image, wherein the epipolar constraint error value of each third feature point pair is determined by the inertial data corrected in step S5; when the epipolar constraint error value of the third feature point pair is less than a preset pixel distance threshold, the third feature point pair is updated to a first feature point pair, and a new first feature point pair is selected from the third feature point pairs; when step S2 is repeated for the Nth time, the robot calculates the epipolar constraint error value of each second feature point pair selected in the latest executed step S3; when the epipolar constraint error value of the second feature point pair is less than a preset pixel distance threshold, the second feature point pair is updated to a first feature point pair, and a new first feature point pair is selected from all the second feature point pairs selected in step S3; wherein N is set to be greater than 1 and less than or equal to the preset number of iterations.

[0015] Further, in step S6, the method of selecting matching frame images from the reference frame images within the sliding window based on the number of feature points of the second feature point pair in each reference frame image includes: the robot counting the number of feature points of the second feature point pair in each reference frame image within the sliding window; if the number of second feature point pairs matched by the robot in one reference frame image is less than or equal to a second preset point threshold, then it is determined that the one reference frame image fails to match the current frame image; if the number of second feature point pairs matched by the robot in one reference frame image is greater than the second preset point threshold, then it is determined that the one reference frame image successfully matches the current frame image, and the one reference frame image is set as the matching frame image; if the number of second feature point pairs matched by the robot in each reference frame image is less than or equal to the second preset point threshold, then it is determined that each reference frame image within the sliding window fails to match the current frame image, and the robot's tracking using the window matching method fails.

[0016] Further, in step S7, the method for selecting the optimal matching frame image from all matching frame images based on the epipolar constraint error value and the number of feature points of the second feature point pair in each matching frame image includes: within each matching frame image, calculating the sum of the epipolar constraint error values ​​of the second feature point pair to which the feature points belong, as the accumulated epipolar constraint error value of the matching frame image; within each matching frame image, counting the number of feature points that make up the second feature point pair, as the feature point matching number of the matching frame image; and then setting the matching frame image with the smallest accumulated epipolar constraint error value and the largest feature point matching number as the optimal matching frame image.

[0017] In summary, the robot selects all first feature point pairs from the feature points of the current frame image and all feature points of the reference frame images within the sliding window. This completes the matching of each feature point in the current frame with all feature points in the reference frames, obtaining initially filtered feature point pairs and removing interference from feature point pairs that do not meet the corresponding epipolar constraint error values. Based on this, depth information is combined to obtain the matching of feature points between the two frames in another scale dimension. Second feature point pairs are then selected from the first feature point pairs, instead of repeatedly searching and matching all feature points in both frames. This improves the robustness and accuracy of feature point pair matching and image tracking, making robot localization more reliable. A residual formula is then introduced between the selected feature point pairs, and the least squares method is used to fit the inertial compensation value from the residual formula. This inertial compensation value is then used to correct the inertial data. After the original inertial data undergoes iterative matching of the ratio of visual feature point pairs with respect to the epipolar constraint error value and the depth value, corrective partial derivative information is obtained, allowing the inertial data to be optimized among the filtered feature points in the two frames, thus improving the robot's localization accuracy. It saves a lot of computation, improves image matching efficiency, and increases the speed of localization and map building. Attached Figure Description

[0018] Figure 1 This is a flowchart of an image matching control method incorporating inertial data, as disclosed in one embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. To further illustrate the embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, mainly used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention. The flowchart depicts a process or method. Although the flowchart describes the steps as sequential processes, many of the steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0020] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0022] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0023] As one embodiment, an image matching control method is disclosed, wherein the executing entity of the image matching control method is a robot fixedly equipped with a camera and an inertial sensor, wherein the robot is an autonomous mobile robot. Figure 1 As shown, the image matching control method includes the following steps:

[0024] In step S101, the robot acquires the current frame image through the camera and obtains inertial data through the inertial sensor; then the robot executes step S102. The camera is mounted on the outside of the robot, with its lens pointing in the robot's forward direction, used to acquire image information in front of the robot. Based on frame count, the camera acquires the current frame image. The robot obtains feature points from the current frame image. Feature points refer to environmental elements existing in the form of points in the environment where the camera is located, facilitating matching with previously acquired images to achieve tracking of the current frame image. The inertial sensor is generally installed inside the robot's body; for example, an encoder is installed in the robot's drive wheel, used to acquire displacement information generated during the robot's movement. The inertial measurement unit (such as a gyroscope) is used to acquire angle information generated during the robot's movement. The encoder and the inertial measurement unit form an inertial system used to acquire inertial data, thereby determining the target position. The robot can use the camera pose transformation relationship between two frames of images to perform feature point matching and transformation. The initial state quantities of the rotation matrix and translation vector involved in the camera pose transformation relationship between the two frames are preset. Based on these initial state quantities, the robot integrates the displacement changes sensed by the encoder between the two frames acquired by the camera and the angle changes sensed by the gyroscope between the two frames. This integration can be done using Euler integrals to integrate the displacement and angle changes separately, yielding the robot's pose change between any two frames (including multiple frames acquired pre-captured). This allows for the acquisition of the latest rotation matrix and translation vector. Therefore, the inertial data disclosed in this embodiment can represent the transformation relationship between the coordinate system of the current frame image and the coordinate system of the reference frame image, including translation transformation, rotation transformation, displacement difference, and angle increment. The reference frame image is the image within a fixed-size sliding window disclosed in the aforementioned embodiment.

[0025] Step S102: Based on the inertial data, the robot uses the epipolar constraint error value to select the first feature point pair from the feature points of the current frame image and all the feature points of the reference frame images within the sliding window, thereby filtering out feature point pairs with excessively large epipolar constraint error values, and achieving the filtering of mismatched feature points between the current frame image and each reference frame image; then, step S103 is executed. In this embodiment, based on the transformation relationship between the coordinate system of the current frame image and the coordinate system of the reference frame image involved in the inertial data, the robot applies epipolar constraints to the feature points of the current frame image and all the feature points of the reference frame images within the sliding window to obtain the epipolar constraint error value of each feature point pair. The epipolar constraint error value is the error value calculated by the feature points of the current frame image and the feature points of the reference frame images within the sliding window according to the geometric imaging relationship under epipolar constraints. Epipolar constraints are used to represent the geometric relationship of the pixel points corresponding to a three-dimensional point in space on different imaging planes, and also represent the projective relationship (or the geometric relationship of each matching point) of each pixel in two frames of images acquired successively by the camera following the robot's movement. The feature points of the first feature point pair can be located in the current frame image and each reference frame image respectively, or they can be located in the current frame image and some reference frame images respectively; the sliding window is set to be filled with at least one pre-acquired frame image so that the current frame can be matched with each frame in the sliding window in the subsequent implementation; the feature point is the pixel of the image, and the feature point is an environmental element in the form of a point in the environment where the camera is located, which describes the environmental features that the robot needs to track.

[0026] Step S103: Based on the inertial data, the robot uses the depth values ​​of the feature points to select a second feature point pair from the first feature point pair; then step S104 is executed. In step S103, based on the first feature point pair filtered out in step S102, the robot calculates the depth information of the feature points of the first feature point pair in the current frame image and the depth information of the feature points of the first feature point pair in the reference frame image, within the range of pixel noise influence. Specifically, it uses the displacement information of the robot (or camera) between the current frame image and the reference frame image in the inertial data, and the normalized planar coordinates of each feature point of the first feature point pair to calculate the depth value of the corresponding feature point. In some embodiments, the feature point of the first feature point pair in the current frame image is marked as P1, and the optical center when the camera captures the current frame image is marked as O1. The feature point of the first feature point pair in the reference frame image is marked as P2, and the optical center when the camera captures the reference frame image is marked as O2. Then, without considering the influence of pixel noise, the straight line O1P1 and the straight line O2P2 intersect at point P3. The length of the line segment O1P3 is the depth value of feature point P1, and the length of the line segment O2P3 is the depth value of feature point P2. Then, if the ratio of the depth value of feature point P1 to the depth value of feature point P2 meets a certain ratio range, feature point P1 and feature point P2 are combined into a second feature point pair; otherwise, feature point P1 and feature point P2 are combined into an incorrect matching point pair, thereby selecting the second feature point pair from the first feature point pair.

[0027] Step S104: Based on the similarity of the descriptors corresponding to the second feature point pairs, select the third feature point pair from the second feature point pairs; then execute step S105. Step S104 specifically includes: for the current frame image and each reference frame image within the sliding window, the robot calculates the similarity between the descriptor of each feature point in the reference frame image and the descriptor of the feature point in the current frame image; when the similarity calculated by the robot between the descriptor of a feature point in the reference frame image and the descriptor of the feature point in the current frame image is the minimum among the similarities between the descriptor of the current frame image and the descriptor of the reference frame image containing the feature point of the second feature point pair, the second feature point pair is marked as the third feature point pair and the third feature point pair is selected; thereby narrowing the search range of feature points. The descriptor of the reference frame image containing the feature point of the second feature point pair is the descriptor of all feature points constituting the second feature point pair within the reference frame image containing the feature point of the second feature point pair, and can be represented using frame descriptors. The descriptor of the current frame image is the descriptor of the feature points in the reference frame image where the feature points of the second feature point pair are located. The similarity between the descriptors corresponding to the second feature point pair is represented by the Euclidean distance or Hamming distance between the descriptors of the feature points in the current frame image and the descriptors of the feature points in the corresponding reference frame image within the sliding window. This allows the tracking of the robot's motion to be achieved by using the pixel similarity between two frames.

[0028] Specifically, in step S104, the robot records the reference frame image where the feature points of each second feature point pair are located as the reference frame image to be matched. Generally, the number of reference frame images to be matched is equal to the number of reference frame images. The robot also marks the second feature point pairs that exist between the current frame image and the reference frame image to be matched as the second feature point pairs to be matched. The feature points of the second feature point pairs to be matched in the current frame image are recorded as the second first feature points, and the feature points of the second feature point pairs to be matched in the reference frame image are recorded as the second second feature points. The second second feature points are located in the reference frame image to be matched. The robot needs to calculate the similarity between the descriptors of all second second feature points in the reference frame image and the descriptors of their corresponding second first feature points. Then, when the similarity between the descriptor of a second feature point pair to be matched in the reference frame image and the descriptor of the feature point pair to be matched in the current frame image is the minimum among the similarities between the descriptors of all second feature points in the reference frame image and their corresponding second feature points, the second feature point pair to be matched is marked as a third feature point pair and the third feature point pair is selected. Multiple third feature point pairs can be selected between each reference frame image and the current frame image. The similarity between the descriptor of a second feature point pair to be matched in the reference frame image and the descriptor of the feature point pair to be matched in the current frame image is the similarity between the descriptor of the second feature point and the descriptor of the second feature point. As a similarity measure of the two descriptors, it is specifically expressed as the square root of the sum of the squares of the Euclidean distance or Hamming distance between the second feature point and the second feature point in multiple dimensions. Each dimension can represent a binary encoding form of the feature point.

[0029] Based on the above embodiments, whenever the robot has searched all feature points that form a second feature point pair between the current frame image and a reference frame image within the sliding window, that is, after the robot has calculated the similarity between the descriptors of all second feature points and the corresponding descriptors of the second feature points within a reference frame image, if the number of third feature point pairs counted by the robot within the reference frame image is greater than a first preset point threshold, then it is determined that the current frame image and the reference frame image are successfully matched, and then the robot continues to search for all feature points that form a second feature point pair between the current frame image and the next reference frame image within the sliding window; if the number of third feature point pairs counted by the robot within the reference frame image is less than or equal to the first preset point threshold, then it is determined that the current frame image and the reference frame image are not matched, and the reference frame image is set as a mismatched reference frame image, and then the robot continues to search for all feature points that form a second feature point pair between the current frame image and the next reference frame image within the sliding window; in some embodiments, the feature points of the reference frame image are not subsequently used to match the feature points of the current frame image, preferably, the first preset point threshold is set to 20. In some embodiments, feature points of the reference frame image are marked as mismatched feature points and are no longer used to form the first feature point pair, the second feature point pair, or the third feature point pair with feature points in the current frame image. If the number of third feature point pairs counted by the robot in the reference frame image is greater than a first preset point count threshold, it is determined that the current frame image and the reference frame image are successfully matched. When the robot determines that the current frame image and all reference frame images in the sliding window fail to match, it is determined that the robot's tracking using the window matching method has failed, and then the robot clears the images in the sliding window.

[0030] Step S105: The robot introduces a residual formula between the third feature point pairs, then uses the least squares method to fit the inertial compensation value from the residual formula, and then uses the inertial compensation value to correct the inertial data. That is, the inertial compensation value compensates the inertial data so that the corrected inertial data is the inertial data with the smallest error (or sum of squares of error) with the expected target data under the least multiplication condition. In some embodiments, the feature point of the third feature point pair in the current frame image e1 is marked as point P4, and the optical center when the camera captures the current frame image e1 is marked as point O1. The feature point of the third feature point pair in the reference frame image e2 is marked as point P5, and the optical center when the camera captures the reference frame image e2 is marked as point O1. The optical center of image e2 is marked as point O2. The lines O1P4 and O2P5 intersect at point P6. Points O1, O2, and P6 form a polar plane. After the line O1P4 is transformed into the reference frame image e2, it becomes the epipolar line L. Without considering the error, the intersection of the polar plane and the reference frame image e2 coincides with the epipolar line L. This intersection line passes through point P5, but in reality, due to the existence of the error, they do not coincide. In this embodiment, point P5 is set as an observation point on the epipolar line L, and then the distance from point P5 to the epipolar line L is used to represent this error. This error is then set as the residual, and the residual formula is used to describe or perform function pattern fitting.Therefore, in this embodiment, obtaining the residual requires constructing a derivation equivalent to calculating the distance from a point to a line. First, when the corresponding rotation matrix and translation vector are set as known state variables, the residual value can be calculated using this derivation, serving as the numerical result of the residual. Then, the corresponding rotation matrix and translation vector are set as unknown state variables, and the derivation (equivalent to an equation) is controlled to take partial derivatives with respect to the translation vector and rotation matrix respectively, obtaining the Jacobian matrix. This achieves the differentiation of the residual formula with respect to pose, thereby storing the differentiation result of the inertial data in matrix form. Then, combining the properties of derivatives, it is known that the robot sets the product of the inverse of the Jacobian matrix and the residual value as the inertial compensation value, obtaining the compensation amount for the displacement integral and the compensation amount for the angle integral, serving as the inertial compensation value, realizing: projecting a feature point of an image frame onto... After another frame image, the deviation distance between the projected feature points and the epipolar line in the other frame image (the projected feature points should be located on the epipolar line in the other frame image without considering errors, but there is actually a deviation) is set as the error between the data to be fitted and the target data. The least squares method is used to process the optimal compensation value (including constructing the aforementioned derivation formula (equivalent to the expression of the minimum error sum of squares), specifying the parameters (rotation matrix and translation vector), taking the partial derivatives of the rotation matrix and translation vector (to obtain the best estimated parameters under the minimum error), and obtaining the optimal compensation value of the rotation matrix and translation vector in matrix form). Then, the optimal compensation value (the inertia compensation value) is used to correct the inertia data. The specific correction methods include, but are not limited to, adding, subtracting, multiplying, or dividing with the original inertia data, or even directly updating and replacing the original inertia data. Then the robot updates the corrected inertial data to the inertial data, updates the feature points of the third feature point pair in the current frame image as described in step S104 to the feature points of the current frame image, and updates the feature points of the third feature point pair in the reference frame image as described in step S104 to the feature points of all reference frame images within the sliding window, narrowing the search range of subsequent feature points, completing a feature point filtering, and also completing an initialization of the feature points of the current frame image and each reference frame image within the sliding window; then, step S106 is executed.

[0031] Step S106: Determine whether the execution counts of steps S107 and S108 have both reached the preset iteration matching count. If yes, execute step S10; otherwise, execute step S107. In some embodiments, step S106 can also be understood as determining whether the execution counts of steps S107, S108, and S109 have all reached the preset iteration matching count. If yes, execute step S10; otherwise, execute step S107, so that the number of corrections to the inertial data reaches the sum of the preset iteration matching count and the value 1. The preset iteration matching count is preferably 2 or 3. The robot repeatedly executes steps S107 and S108 to filter out the second feature point pairs and eliminate more erroneous matching point pairs, reducing the search range and the computational load of the inertial compensation value.

[0032] Step S107: Based on the inertial data, the robot uses the epipolar constraint error value to select the first feature point pair from the feature points of the current frame image and the feature points of all reference frame images within the sliding window. This is equivalent to repeating step S102, thereby filtering out feature point pairs with excessively large epipolar constraint error values, and filtering out mismatched feature points between the current frame image and each reference frame image; then step S108 is executed.

[0033] Step S108: Based on the inertial data, the robot uses the depth values ​​of the feature points to select a second feature point pair from the first feature point pair (selected in the most recently executed step S107). This is equivalent to repeating step S103, using the displacement and angle information of the robot (or camera) between the current frame image and the reference frame image in the inertial data, as well as the normalized planar coordinates of each feature point in the first feature point pair, to calculate the triangular geometric relationship for matching the depth values ​​of the corresponding feature points, calculate the depth values ​​of each feature point in the first feature point pair, and then select the second feature point pair by comparing the depth values ​​of the feature points of the first feature point pair in the current frame image with the depth values ​​of the feature points of the first feature point pair in the reference frame image. Then, step S109 is executed.

[0034] In step S109, the robot introduces a residual formula between the second feature point pairs selected in the latest executed step S108, then uses the least squares method to fit the inertial compensation value from the residual formula, and then uses the inertial compensation value to correct the latest obtained inertial data. The corrected inertial data is then updated to the inertial data described in step S107, and the feature points included in the second feature point pairs selected in the latest executed step S108 are updated to the feature points of the current frame image and all reference frame images within the sliding window as described in step S107. This is equivalent to repeating step S105, but step S103 skips the selection of the third feature point pairs in step S104 and directly executes step S105. Then, step S106 is executed to determine whether the execution counts of step S107 and step S108 have both reached the preset iteration matching count, thereby repeatedly correcting the inertial data in step S109, reducing the residual value, optimizing the reference frame images subsequently filled into the sliding window, and improving the accuracy of robot positioning.

[0035] Understandably, after executing steps S102 and S103, or whenever steps S107 and S108 are repeated (beginning step S109), the robot constructs a residual formula between the newly selected second feature point pairs, and then uses the least squares method to fit the residual formula to obtain the inertial compensation value. Specifically, this becomes the result of differentiating the newly obtained inertial data to minimize the relevant error or sum of squares error. The inertial compensation value is then used to correct the newly obtained inertial data, and the corrected inertial data is updated to the aforementioned inertial data. The feature points included in the newly selected second feature point pairs are updated to the feature points of the current frame image and the feature points of all reference frame images within the sliding window, thereby narrowing the search range of feature points, further saving a large amount of matching calculations, and improving the robot's localization and map building speed.

[0036] It should be noted that, regarding step S107, after the robot has completed step S105, during the first execution of step S107, the robot calculates the epipolar constraint error value for each third feature point pair. The epipolar constraint error value for each third feature point pair is determined by the inertial data corrected in step S105. The specific epipolar constraint method for the third feature point pair is the same as in step S102, and the calculation method for the epipolar constraint error value is also the same as in step S102, but the labeling type and number of feature point pairs subject to epipolar constraint are different. When the epipolar constraint error value of a third feature point pair calculated by the robot is less than a preset pixel distance threshold, the third feature point pair is updated to the first feature point pair, and a new first feature point pair is selected from the third feature point pairs selected in step S104, excluding mismatched reference frame images.

[0037] It should be noted that, for step S107, when step S107 is repeated for the Nth time, the robot calculates the epipolar constraint error value of each second feature point pair selected in the latest executed step S108, wherein the epipolar constraint error value of each second feature point pair is determined by the inertial data corrected in the previous executed step S109; when the epipolar constraint error value of a second feature point pair calculated by the robot is less than the preset pixel distance threshold, the second feature point pair is marked as the first feature point pair to update the first feature point pair, and a new first feature point pair is selected from all the second feature point pairs selected in step S3; wherein, N is set to be greater than 1 and less than or equal to the preset number of iterations.

[0038] Step S110: Based on the number of feature points of the second feature point pair in each reference frame image, a matching frame image is selected from the reference frame images in the sliding window; then step S111 is executed. Therefore, after the robot completes the iterative matching of each feature point in the current frame image with all feature points in each reference frame image in the sliding window, it counts the number of feature points of the second feature point pair in each reference frame image. Then, based on whether the number of the second feature points counted in each reference frame image meets the threshold condition, a reference frame image matching the current frame image is selected in the sliding window, thereby forming a matching frame image pair between the current frame image and the corresponding reference frame image. The feature points of the second feature point pair in the current frame image are denoted as the second first feature point, and the feature points of the second feature point pair in the reference frame image are denoted as the second second feature point.

[0039] Specifically, the method for selecting matching frame images from reference frame images within a sliding window based on the number of feature points of the second feature point pair in each reference frame image includes: after the number of times the inertial data has been repeatedly corrected reaches a preset number of iterations, the robot counts the number of feature points of the second feature point pair in each reference frame image within the sliding window, which is taken as the number of second feature point pairs matched in the corresponding reference frame image. If the feature points of the second feature point pair in the reference frame image are marked as second feature points, then the number of second feature point pairs matched in that reference frame image is equal to the number of second feature points in that reference frame image. If the robot matches a second feature point in one of the reference frames... If the number of second feature point pairs matched by the robot in one of the reference frame images is less than or equal to the second preset point threshold, then it is determined that the match between the reference frame image and the current frame image is failed, and the reference frame image can be set as a mismatched reference frame image. If the number of second feature point pairs matched by the robot in one of the reference frame images is greater than the second preset point threshold, then it is determined that the reference frame image and the current frame image are successfully matched, and the reference frame image is set as a matched frame image. Further, if the number of second feature point pairs matched by the robot in each reference frame image is less than or equal to the second preset point threshold, it is determined that the match between each reference frame image in the sliding window and the current frame image is failed, and the robot's tracking using the window matching method is determined to be failed. Preferably, the second preset point threshold is set to 15, which is less than the first preset point threshold. When the preset number of iterations is increased, the number of incorrect matching point pairs excluded in the sliding window becomes more or remains unchanged. Therefore, the number of second feature point pairs matched in all reference frame images will decrease or remain unchanged. Thus, the number of second feature points in each reference frame image or the total number of second feature points in all reference frame images will decrease or remain unchanged.

[0040] In step S111, based on the epipolar constraint error value and the number of feature points of the second feature point pair in each matching frame image, the optimal matching frame image is selected from all matching frame images, and it is determined that the robot has successfully tracked using the window matching method. Then, the robot removes the earliest reference frame image filled into the sliding window to free up memory space, and then fills the current frame image into the sliding window to update it as a new reference frame image. In step S111, among all the selected matching frame images, the smaller the sum of the epipolar constraint errors corresponding to the second feature points in a single matching frame image, the better the matching degree between that matching frame image and the current frame image, and the lower the matching error; among all the selected matching frame images, the more second feature points in a single matching frame image, the better the matching degree between that matching frame image and the current frame image, and the more matching points.

[0041] Therefore, in step S111, the method of selecting the optimal matching frame image from all matching frame images based on the epipolar constraint error value and the number of feature points of the second feature point pair in each matching frame image specifically includes: in each matching frame image, calculating the sum of the epipolar constraint error values ​​of the second feature point pair to which the feature point belongs in that matching frame image, and using it as the accumulated epipolar constraint error value of that matching frame image, so that each matching frame image is configured with an accumulated epipolar constraint error value; wherein, the feature point of the second feature point pair in the matching frame image is the second second feature point, and the robot accumulates the epipolar constraint error values ​​of the second feature point pair to which each newly marked second second feature point belongs in a matching frame image to obtain the sum of the epipolar constraint error values ​​of the second feature point pair to which the feature point belongs in that matching frame image. Within each matching frame image, the number of feature points forming the second feature point pair is counted and used as the feature point matching quantity for that matching frame image, ensuring that each matching frame image has a configured feature point matching quantity. When the feature point of the matching frame image is the second feature point, the number of feature points forming the second feature point pair within that matching frame image is the total number of second feature points present in that matching frame image. Then, the robot sets the matching frame image with the largest accumulated extreme constraint error value (for all matching frame images) and the largest number of feature point matching quantities (for all matching frame images) as the optimal matching frame image.

[0042] In summary, the robot selects all first feature point pairs from the feature points of the current frame image and all feature points of the reference frame images within the sliding window. This completes the matching of each feature point in the current frame with all feature points in the reference frames, obtaining initially filtered feature point pairs and removing interference from feature point pairs that do not meet the corresponding epipolar constraint error values. Based on this, depth information is combined to obtain the matching status of feature points between the two frames in another scale dimension. Second feature point pairs are then selected from the first feature point pairs, instead of repeatedly searching and matching all feature points in both frames. This improves the robustness and accuracy of feature point pair matching and image tracking, making robot localization more reliable. Then, a residual formula is introduced between the selected feature point pairs. The least squares method is then used to fit the inertial compensation value from the residual formula. This inertial compensation value is then used to correct the inertial data. The calculated inertial compensation value allows the original inertial data to undergo iterative matching of the ratio of the epipolar constraint error value and the depth value of the visual feature point pairs, resulting in correctable partial derivative information. This optimizes the inertial data within the filtered feature points of two image frames, improving the robot's positioning accuracy. This approach saves significant computational resources, improves image matching efficiency, and increases the speed of localization and map building.

[0043] As one embodiment, for step S102 or step S107, the method of selecting the first feature point pair from the feature points of the current frame image and all reference frame images within the sliding window based on the inertial data and using the epipolar constraint error value includes: the robot calculates the epipolar constraint error value of each feature point pair; when the epipolar constraint error value of a feature point pair calculated by the robot is greater than or equal to a preset pixel distance threshold, the feature point pair is marked as an incorrect matching point pair, and the corresponding pair of feature points cannot be used as the matching object in subsequent steps; wherein, the robot sets the preset pixel distance threshold to a distance of 3 pixel spans, such as 3 adjacent pixels (with one pixel as the center, the pixels in the left and right neighboring areas of the center). The distance formed by the point and the three adjacent pixels at the center position can be equivalent to the two-pixel spacing formed by three pixels in the same row or column. When the epipolar constraint error value of a feature point pair calculated by the robot is less than the preset pixel distance threshold, the feature point pair is marked as the first feature point pair and the first feature point pair is selected. The robot then selects the first feature point pair from the feature points of the current frame image and the feature points of all reference frame images in the sliding window. It should be noted that the smaller the epipolar constraint error value of a feature point pair, that is, the smaller the epipolar constraint error value generated by a feature point in the current frame image and a feature point in the reference frame image in the sliding window under epipolar constraint, the smaller the matching error between this pair of feature points.

[0044] In this embodiment, each feature point pair is configured to consist of a feature point (any feature point) in the current frame image and a feature point (any feature point) in the reference frame image. It cannot consist of a pair of feature points in the same reference frame image, feature points between different reference frame images, or a pair of feature points in the current frame image. Each feature point in the current frame image forms a feature point pair with each feature point in each reference frame image within the sliding window, thereby achieving brute-force matching between the current frame image and all reference frame images within the sliding window. The robot controls each feature point in the current frame image to calculate the epipolar constraint error value of the corresponding feature point pair by sequentially comparing the normalized planar coordinates of each feature point in each reference frame image within the sliding window with the normalized planar coordinates of each feature point. Then, whenever the calculated epipolar constraint error value of a feature point pair is greater than or equal to a preset pixel distance threshold, the feature point pair is filtered out; otherwise, the feature point pair is marked as the first feature point pair. After traversing all the feature point pairs, the robot selects all the first feature point pairs from the feature points of the current frame image and all the feature points of all the reference frame images within the sliding window, completing the matching of each feature point in the current frame with all the feature points in the reference frames, obtaining the initially filtered feature point pairs, and removing interference from some feature point pairs that do not meet the error value.

[0045] It should be noted that the rigid body motion of the camera is consistent with the motion of the robot. Two consecutively acquired image frames will have two coordinate system representations: the current frame relative to the reference frame, and the reference frame relative to the current frame. A certain geometric relationship exists between points in the two consecutively acquired image frames, which can be described using epipolar geometry. Epipolar geometry describes the projective relationship (or the geometric relationship between matching points) of pixels in the two image frames. In some embodiments, it is independent of the external scene itself and only depends on the camera's intrinsic parameters and the shooting positions of the two images. Ideally, the epipolar constraint error value is equal to 0. However, due to noise, the epipolar constraint error value will inevitably not be 0. This non-zero number can be used to measure the magnitude of the matching error between feature points in the reference frame and feature points in the current frame.

[0046] In some embodiments, R is denoted as the rotation matrix from coordinate system C1 to coordinate system C0, which can represent the rotation from frame k to frame (k+1). Vector C0-C1 is the translation of optical center C1 relative to optical center C0, denoted as T. R and T can represent the robot's motion between two frames, provided by the inertial sensor, and can be included in the inertial data. There will be two coordinate system representations, including the current frame relative to the reference frame and the reference frame relative to the current frame. C0 and C1 are the optical centers of the camera in the two motion positions of the robot, which are the pinholes in the pinhole camera model. Q is a three-dimensional point in space, and Q0 and Q1 are the pixel points corresponding to point Q on different imaging planes. Q0 and Q1 are both two-dimensional points on the image. In this embodiment, they are both treated as three-dimensional direction vectors. Assuming a normalized image plane with focal length f=1, it can be defined in a coordinate system with optical center C0 as the origin. Q0 is in the reference coordinate system with optical center C0 as the origin, and Q1 is in the reference coordinate system with optical center C1 as the origin. Therefore, coordinate system transformation is also required. Here, the coordinates of all points are transformed to a coordinate system with C0 as the origin. Since the direction vector is independent of the vector's starting position, the coordinate system transformation for Q0 and Q1 only needs to consider rotation. The normalized image plane here is the polar plane composed of C0-C1-Q0-Q1.

[0047] As one embodiment, in step S102 or S107, when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot denotes the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix. The first translation vector represents the translation vector from the coordinate system of the current frame image to the coordinate system of the reference frame image, and the first rotation matrix represents the rotation matrix from the coordinate system of the current frame image to the coordinate system of the reference frame image, so that the inertial data represents displacement and angle information in the coordinate system of the reference frame image. Based on this, the robot uses the first rotation matrix to revert a feature point (which can be extended to any feature point) of the current frame image. The normalized planar coordinates of the feature point are transformed into the coordinate system of the reference frame image to obtain the first coordinate. The normalized planar coordinates of this feature point are represented by a direction vector in the coordinate system of the current frame image. Only the direction of the direction vector is considered, not its starting or ending point, and it forms a column vector with an inverse vector. Therefore, the coordinate system transformation of all feature points in the current frame image only requires rotation. The first coordinate can be represented by a direction vector in the coordinate system of the reference frame image. Thus, in this embodiment, the normalized vector of the feature point in the current frame image is set as the vector formed by the normalized planar coordinates of the feature point in the current frame image relative to the origin of the coordinate system of the current frame image; and the normalized vector of the feature point in the reference frame image is set as the vector formed by the normalized planar coordinates of the feature point in the reference frame image relative to the origin of the coordinate system of the reference frame image. The robot controls the first rotation matrix to transform the normalized vector of the feature points in the current frame image to the coordinate system of the reference frame image, obtaining the first vector; then it controls the first translation vector to cross-multiply the first vector, obtaining the first vector, where the first vector is perpendicular to both the first translation vector and the first vector; then it controls the normalized vector of the feature points in the reference frame image within the sliding window to dot-multiply the first vector, and then sets the result of the dot-multiply (representing the cosine of the angle between the normalized vector of the feature points in the reference frame image and the first vector) as the epipolar constraint error value of the corresponding feature point pair; specifically, the robot controls the normalized vector of each feature point in each frame of the reference frame image within the sliding window to sequentially perform a dot-multiply with the first vector, and then sets the result of each dot-multiply as the epipolar constraint error value of the corresponding feature point pair.

[0048] It should be noted that the normalized vector of the feature points in the current frame image is the vector formed by the normalized planar coordinates (the endpoint of the vector) of the feature points in the current frame image relative to the origin (the starting point of the vector) of the coordinate system of the current frame image; the normalized vector of the feature points in the reference frame image is the vector formed by the normalized planar coordinates (the endpoint of the vector) of the feature points in the reference frame image relative to the origin (the starting point of the vector) of the coordinate system of the reference frame image.

[0049] As one embodiment, in step S102 or S107, when the inertial data includes a translation vector of the reference frame image relative to the current frame image and a rotation matrix of the reference frame image relative to the current frame image, the robot marks the translation vector of the reference frame image relative to the current frame image as a second translation vector and the rotation matrix of the reference frame image relative to the current frame image as a second rotation matrix. The second translation vector represents the translation vector from the coordinate system of the reference frame image to the coordinate system of the current frame image, and the second rotation matrix represents the rotation matrix from the coordinate system of the reference frame image to the coordinate system of the current frame image, such that the inertial data represents displacement and angle information in the coordinate system of the current frame image. Then, the robot controls the second rotation matrix. The normalized vectors of feature points in the reference frame image within the sliding window are transformed to the coordinate system of the current frame image to obtain a second vector. Then, the second translation vector is cross-multiplied by the second vector to obtain a second vector perpendicular to both the second translation vector and the second vector. Next, the normalized vector of the feature point in the current frame image is multiplied by the second vector, and the result of this multiplication (representing the cosine of the angle between the normalized vector of the feature point in the current frame image and the second vector) is set as the epipolar constraint error value for the corresponding feature point pair. Specifically, the robot controls the normalized vector of each feature point in the current frame image to be multiplied by the first vector in turn, and the result of each multiplication is set as the epipolar constraint error value for the corresponding feature point pair. This allows the epipolar constraint error value to geometrically describe the feature point matching error information between image frames acquired by the camera from different viewpoints.

[0050] As one embodiment, in step S103 or step S108, based on the inertial data, the method of selecting a second feature point pair from the first feature point pair using the depth value of the feature points includes: the robot calculating the ratio of the depth value of the feature points of the selected first feature point pair (which may be selected in step S102 or step S107) in the current frame image to the depth value of the feature points of the first feature point pair in the reference frame image; if each first feature point pair consists of a first feature point in the current frame image and a first feature point in the reference frame image, the ratio of the depth value of the first feature point in the current frame image to the depth value of the corresponding first feature point in the reference frame image is recorded and used for threshold comparison to filter out first feature point pairs containing first feature points with mismatched ratios. When the ratio of the depth value of the feature point in the current frame image to the depth value of the feature point in the reference frame image calculated by the robot is within a preset ratio threshold range, the first feature point pair is marked as the second feature point pair and the second feature point pair is selected. Preferably, the preset ratio threshold range is set to be greater than 0.5 and less than 1.5. When the ratio of the depth value of the feature point in the current frame image to the depth value of the feature point in the reference frame image calculated by the robot is not within the preset ratio threshold range, the first feature point pair is marked as an incorrect matching point pair, thereby excluding the incorrect matching point pair from the first feature point pairs selected in steps S102 and S107. Filtering the first feature point pairs can narrow the search range of feature point pairs during subsequent feature point matching.

[0051] As one embodiment, in step S103 or step S108, the method for the robot to calculate the depth value of the feature points includes: when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot denotes the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix, wherein the first translation vector represents the translation vector from the coordinate system of the current frame image to the coordinate system of the reference frame image, and the first rotation matrix represents the rotation matrix from the coordinate system of the current frame image to the coordinate system of the reference frame image; then, the robot controls the first rotation matrix to transform the normalized vector of the first feature point pair in the current frame image to the coordinate system of the reference frame image, obtaining the first one vector; then, the robot controls the normalized vector of the first feature point pair in the reference frame image to cross-multiply the first one vector, obtaining the first two vectors; simultaneously, the robot controls the normalized vector of the first feature point pair in the reference frame image to cross-multiply the first translation vector, and then inverts the cross-multiplication result to obtain the first three vectors. The quantity, wherein the cross product of the normalized vector of the first feature point pair in the reference frame image and the first translation vector is a vector that is perpendicular to both the normalized vector and the first translation vector, then the opposite vector of this vector is the first third vector; then the product of the first third vector and the inverse of the first second vector is set as the depth value of the feature point of the first feature point pair in the current frame image, and marked as the first depth value, representing the distance between the three-dimensional point detected by the camera and the optical center (the origin of the coordinate system of the current frame image) when the camera acquires the current frame image; then the first The sum of the product of a vector and a first depth value and a first translation vector is labeled as the first quad vector. Then, the product of this first quad vector and the inverse of the normalized vector of the first feature point pair in the reference frame image is set as the depth value of the feature point in the reference frame image. This is equivalent to setting the product of the first quad vector and the inverse of the normalized vector as the depth value of the feature point in the reference frame image, and labeling it as the second depth value. This second depth value represents the distance between the same 3D point and the optical center (the origin of the coordinate system of the reference frame image) when the camera captures the reference frame image. Thus, based on the pose transformation information of the camera between two frames, the depth information of a pair of feature points is calculated using triangulation.

[0052] As one embodiment, in step S103 or step S108, the method for the robot to calculate the depth value of the feature points includes: when the inertial data includes a translation vector of the reference frame image relative to the current frame image and a rotation matrix of the reference frame image relative to the current frame image, the robot denotes the translation vector of the reference frame image relative to the current frame image as a second translation vector and the rotation matrix of the reference frame image relative to the current frame image as a second rotation matrix, wherein the second translation vector represents the translation vector from the coordinate system of the reference frame image to the coordinate system of the current frame image, and the second rotation matrix represents the rotation matrix from the coordinate system of the reference frame image to the coordinate system of the current frame image; then, the robot controls the second rotation matrix to transform the normalized vector of the first feature point pair in the reference frame image to the coordinate system of the current frame image, obtaining a second vector; then, the robot controls the normalized vector of the first feature point pair in the current frame image to transform the normalized vector of the feature points in the reference frame image to the coordinate system of the current frame image. The normalized vector is cross-multiplied by the second vector to obtain the second vector. Simultaneously, the normalized vector of the first feature point in the current frame image is cross-multiplied by the second translation vector, and the cross-multiplication result is inverted to obtain the second vector. The product of the second vector and the inverse of the second vector is then set as the depth value of the first feature point in the reference frame image, and marked as the second depth value, representing the distance between the 3D point detected by the camera and the optical center when the camera captures the reference frame image. The sum of the product of the second vector and the second depth value and the second translation vector is then marked as the second vector. The product of the second vector and the inverse of the normalized vector of the first feature point in the current frame image is then set as the depth value of the first feature point in the current frame image, and marked as the first depth value, representing the distance between the same 3D point and the optical center when the camera captures the current frame image.

[0053] By combining steps S103 or S108, the aforementioned embodiment for calculating depth values ​​is based on the geometric relationship formed by the projection points of the same point on two frames of images from different perspectives and the corresponding optical centers in each frame. It also combines depth information to obtain the matching situation of feature points in two frames of images in another dimension of scale information, thereby improving the robustness and accuracy of feature points for matching and image tracking, making robot localization more reliable.

[0054] It should be noted that the normalized vector of the feature points of the first feature point pair in the current frame image is the vector formed by the normalized planar coordinates of the feature points of the first feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the first feature point pair in the reference frame image is the vector formed by the normalized planar coordinates of the feature points of the first feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image. In some embodiments, if the number of first feature point pairs selected in step S102 or step S107 is large, it is necessary to obtain a batch of first feature point pairs with a high degree of matching through the least squares method before solving for the depth value of the feature points; since step S103 or step S108 is a preliminary screening and the accuracy requirement is not high, the use of the least squares method is not necessary.

[0055] In some embodiments, the feature points of the first feature point pair in the current frame image are marked as P1, the optical center when the camera captures the current frame image is marked as O1, the feature points of the first feature point pair in the reference frame image are marked as P2, the optical center when the camera captures the reference frame image is marked as O2, O1-O2-P1-P2 form the polar plane, the intersection of the polar plane and the current frame image becomes the polar line of the imaging plane of the current frame image and passes through P1, and the intersection of the polar plane and the reference frame image becomes the polar line of the imaging plane of the reference frame image and passes through P2. Without considering the noise effect of pixels, lines O1P1 and O2P2 intersect at point P3. The length of line segment O1P3 is the depth value of feature point P1, and the length of line segment O2P3 is the depth value of feature point P2. When considering the noise effect of pixels, the intersection of lines O1P1 and O2P2 is point P0, which is not point P3. Therefore, the positional deviation between point P0 and point P3 can be used to measure the matching error. Thus, it is necessary to set the preset ratio threshold range to compare the ratio of depth values ​​between feature point pairs.

[0056] As one embodiment, in step S104, the method of selecting a third feature point pair from the second feature point pairs based on the similarity of the descriptors corresponding to the second feature point pairs specifically includes: for the current frame image and the corresponding reference frame image in the sliding window, it can be considered that for the second feature point pairs marked between the current frame image and each reference frame image in the sliding window, the robot calculates the similarity between the descriptors of the feature points of the second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image. This can be understood as calculating the similarity between the descriptors of the feature points of each second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image, which is also equivalent to the similarity between the frame descriptors of each reference frame image in the sliding window and the frame descriptors of the current frame image. Then, when the similarity between the descriptor of a second feature point pair in the reference frame image and the descriptor of the second feature point pair in the current frame image is the minimum among the similarities between the descriptor of the current frame image and the descriptor of the reference frame image containing the feature points of the second feature point pair, the second feature point pair is marked as the third feature point pair and the third feature point pair is selected. Here, the descriptor of the reference frame image containing the feature points of the second feature point pair is the descriptor of all feature points constituting the second feature point pair within the reference frame image containing the feature points of the second feature point pair; that is, within the same reference frame image, there are multiple descriptors related to the second feature point pair. The descriptor of the two feature point pairs; the descriptor of the current frame image is the descriptor of the feature points of the reference frame image where the feature points of the second feature point pair are located, which together form the feature points of the second feature point pair. That is, there are multiple descriptors of the second feature point pair within the same current frame image. Preferably, the similarity of the descriptors corresponding to the second feature point pair is represented by the Euclidean distance or Hamming distance between the descriptors of the feature points in the current frame image and the descriptors of the feature points in the corresponding reference frame image within the sliding window. Thus, the similarity between the descriptors of the matching points can be calculated using the sum of the squares of the Euclidean distances or the distances of the feature points in multiple dimensions, and the one with the smallest distance is taken as the more accurate point to be matched.

[0057] Specifically, in step S104, the robot records the feature points of the second feature point pair in the current frame image as the second first feature point, and the feature points of the second feature point pair in the reference frame image as the second second feature point; the robot needs to calculate the similarity between the descriptors of all second second feature points in the reference frame image and the descriptors of their corresponding second first feature points. Then, when the similarity between the descriptor of a second feature point pair in the reference frame image and the descriptor of the feature point in the current frame image is the minimum among the similarities between the descriptors of all second feature points in the reference frame image and their corresponding second feature points, the second feature point pair is marked as a third feature point pair and the third feature point pair is selected. Multiple third feature point pairs can be selected between each reference frame image and the current frame image. The similarity between the descriptor of a second feature point pair in the reference frame image and the descriptor of the feature point in the current frame image is the similarity between the descriptor of the second feature point and the descriptor of the second feature point. As a similarity measure between the two descriptors, the specific calculation method can be expressed as the square root of the sum of the squares of the Euclidean distance or Hamming distance between the second feature point and the second feature point in multiple dimensions. Each dimension can represent a binary encoding form of the feature point.

[0058] As one embodiment, for step S105 or step S109, the robot marks the line connecting the optical center of the current frame image captured by the camera and the feature points of the same preset feature point pair within the current frame image as the first observation line, and marks the line connecting the optical center of the reference frame image captured by the camera and the feature points of the same preset feature point pair within the reference frame image as the second observation line. Then, without considering errors, the intersection of the first observation line and the second observation line is marked as the target detection point. The optical center of the current frame image captured by the camera, the optical center of the reference frame image captured by the camera, and the target detection point are all on the same plane, i.e., forming a three-point coplanar state, and then the same plane is set as the polar plane; or, the preset feature point pair, the optical center of the current frame image captured by the camera, and the optical center of the reference frame image captured by the camera are all on the same plane, i.e., forming a four-point coplanar state. The robot records the intersection of the polar plane with the current frame image as the polar line in the imaging plane of the current frame image (which can be regarded as the coordinate system of the current frame image in some embodiments), and records the intersection of the polar plane with the reference frame image as the polar line in the imaging plane of the reference frame image (which can be regarded as the coordinate system of the reference frame image in some embodiments). Specifically, within the same preset feature point pair, a feature point from the current frame image, after being transformed to the reference frame image, becomes a first projection point with coordinates designated as the first coordinate. The distance from the first projection point to the epipolar line in the imaging plane of the reference frame image (in the coordinate system of the reference frame image) is represented as the first residual value. It should be noted that, without considering pixel noise, the first projection point is located on the epipolar line in the imaging plane of the reference frame image (in the coordinate system of the current frame image). That is, the line segment actually observed from the current frame image from the perspective of the reference frame image can coincide with the epipolar line in the imaging plane of the reference frame image after coordinate transformation. Within the same preset feature point pair, a feature point from the reference frame image, after being transformed to the current frame image, becomes a second projection point with coordinates designated as the second coordinate. The distance from the second projection point to the epipolar line in the imaging plane of the current frame image is represented as the second residual value. The smaller the first or second residual value, the smaller the deviation of the corresponding projection point from the epipolar line in the imaging plane it is transformed to, and the higher the matching degree of the corresponding preset feature point pair.

[0059] It should be noted that in step S105, the preset feature point pair is the third feature point pair, which comes from the feature point pair selected in step S104; in step S109, after each repetition of steps S107 and S108, the preset feature point pair is the second feature point pair selected in the most recently executed step S108. The first, second, and third feature point pairs are all pairs of feature points consisting of a feature point located in the current frame image and a feature point located in the reference frame image. The normalized vector of the feature point in the current frame image of the preset feature point pair is the vector formed by the normalized planar coordinates of the feature point in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature point in the reference frame image of the preset feature point pair is the vector formed by the normalized planar coordinates of the feature point in the reference frame image relative to the origin of the coordinate system of the reference frame image. The normalized planar coordinates can belong to the coordinates in the polar plane, so that the coordinates of the feature points of the preset feature point pair in the current frame image and the coordinates of the feature points of the preset feature point pair in the reference frame image are both normalized to the polar plane. Of course, corresponding coordinate normalization processing can also be performed for other types of feature point pairs.

[0060] As one embodiment, in step S105 or step S109, when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot records the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix; in step S105, the preset feature point pair is the third feature point pair, which comes from the feature point pair selected in step S104; in step S109, whenever steps S107 and S108 are repeated, the preset feature point pair is the second feature point pair selected in the most recently executed step S108. The robot controls a first rotation matrix to transform the normalized vectors of the feature points of the preset feature point pair in the current frame image to the coordinate system of the reference frame image, obtaining a first vector. In this embodiment, the normalized vectors of the feature points of the preset feature point pair in the current frame image are represented by direction vectors. Only the direction of the direction vector is considered, not the starting or ending vector, and it forms a column vector with an inverse vector. Therefore, the coordinate system transformation of all feature points in the current frame image only requires rotation, and the first vector in the coordinate system of the reference frame image can be represented by a direction vector. The robot then controls a first translation vector to cross-multiply the first vector to obtain a first second vector, which forms the epipolar line in the imaging plane of the reference frame image. The first second vector serves as a three-dimensional direction vector, and its direction is parallel to the epipolar line. The epipolar line is the intersection line of the epipolar plane formed by the preset feature point pair, the optical center corresponding to the current frame image, and the optical center corresponding to the reference frame image, and the imaging plane of the reference frame image. Then, the square root of the sum of the squares of the horizontal and vertical coordinates in the first two vectors is taken to obtain the epipolar length, which can be regarded as the length of the epipolar line in some embodiments. At the same time, the robot controls the normalized vector of the feature points in the reference frame image of the preset feature point pair to be multiplied by the first two vectors, and then the result of the multiplication is set as the epipolar constraint error value of the preset feature point pair. Then, the ratio of the epipolar constraint error value of the preset feature point pair to the epipolar length is set as the first residual value, and the result value of introducing residuals between the preset feature point pairs is determined.

[0061] Based on the above embodiments, the method of introducing a residual formula between preset feature point pairs, fitting an inertial compensation value from the residual formula using the least squares method, and then using the inertial compensation value to correct the inertial data specifically includes: when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the formula for multiplying the first rotation matrix with the normalized planar coordinates of the feature points of the preset feature point pair in the current frame image as the first transformation formula; then marks the formula for cross-multiplying the first translation vector with the first transformation formula as the first second transformation formula; then marks the formula for dot-multiplying the normalized planar coordinates of the feature points of the preset feature point pair in the reference frame image with the first second transformation formula as the first third transformation formula; and sets the calculation result of the first second transformation formula to 0, forming a linear equation, and then calculates the coefficients and ordinates of this linear equation in the horizontal axis and vertical axis dimensions. The coefficients of the dimension are summed squared, and the square root of the sum is calculated to obtain the first square root, which in some embodiments is equivalent to the projection length of the line represented by the linear equation in the imaging plane of the reference frame image. The reciprocal of the first square root is multiplied by the first three transformation formula to form the first four transformation formula. The calculation result of the first four transformation formula is then set as the first residual value to form the first residual derivation formula, and it is determined that a residual is introduced between the preset feature point pairs. Then, the first residual derivation formula is controlled to calculate the partial derivatives with respect to the first translation vector and the first rotation matrix respectively (at this time, the first translation vector and the first rotation matrix are set as unknowns in the process of calculating the partial derivatives) to obtain the first residual partial derivative equation and the Jacobian matrix. The Jacobian matrix is ​​a combination of the partial derivative of the residual with respect to the first translation vector and the partial derivative of the error value with respect to the first rotation matrix, so as to achieve the effect of correcting the influence of changes in inertial data.Then, solving the first residual partial derivative equation is equivalent to setting the product of the inverse of the Jacobian matrix and the first residual value as the inertia compensation value. This ensures that the inertia compensation value is the fitting result with the minimum error to the target data under the least squares method. In this embodiment, the first residual derivation is the function model equation to be fitted by the least squares method, and the inertia compensation value is the fitting result obtained under the least squares method. Specifically, the first residual derivation is equivalent to setting a fitting function model to fit the compensation value of the inertia data. The residuals in this model belong to error information, such as the minimum sum of squared errors under the least squares condition, making the linear equation the solution with the minimum error. The expression for the sum of squares is given; then, partial derivatives of the first residual derivation are taken with respect to the parameters of the first translation vector and the first rotation matrix, respectively. The resulting formula can be summarized as follows: the result of multiplying the Jacobian matrix by the compensation value of the fitted inertial data (inertial compensation value) is set to be equal to the first residual value. Then, the robot sets the product of the inverse of the Jacobian matrix and the first residual value as the inertial compensation value, thereby completing the least squares problem to find the optimal inertial compensation value. Then, the robot uses the inertial compensation value to correct the inertial data. The specific correction includes addition, subtraction, multiplication, and division operations on the original inertial data, which can be simple coefficient multiplication and division, or matrix-vector multiplication.

[0062] As one embodiment, in step S105 or step S109, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot records the translation vector of the reference frame image relative to the current frame image as the second translation vector and the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix; in step S105, the preset feature point pair is the third feature point pair, which comes from the feature point pair selected in step S104; in step S109, whenever steps S107 and S108 are repeated, the preset feature point pair is the second feature point pair selected in the most recently executed step S108. The robot controls the second rotation matrix to transform the normalized vectors of the feature points of the preset feature point pair in the reference frame image to the coordinate system of the current frame image, obtaining the second vector. In this embodiment, the normalized vectors of the feature points of the preset feature point pair in the reference frame image are represented by direction vectors. Only the direction of the direction vector is considered, not the starting or ending vectors of the direction vector, and it forms a column vector with an inverse vector. Therefore, the coordinate system transformation of all feature points in the current frame image only requires rotation. The first vector in the coordinate system of the current frame image can be represented by a direction vector. Moreover, the preset feature point pair is for step S105, and can also be updated to the second feature point pair in step S109. Then, the robot controls the second translation vector to cross-multiply the second vector to obtain the second vector, and forms the epipolar line in the imaging plane of the current frame image. The second vector serves as a three-dimensional direction vector, and its direction is parallel to the epipolar line. The epipolar line is the intersection line of the epipolar plane formed by the preset feature point pair, the optical center corresponding to the current frame image, and the optical center corresponding to the reference frame image, and the imaging plane of the current frame image. Then, the square root of the sum of the squares of the horizontal and vertical coordinates of the second binary vector is taken to obtain the epipolar length, which in some embodiments can be regarded as the projection length of the straight line pointed to by the second binary vector in the imaging plane of the current frame image. At the same time, the robot controls the normalized vector of the feature point of the preset feature point pair in the current frame image to perform a dot product with the second binary vector, and then sets the result of the dot product as the epipolar constraint error value of the preset feature point pair. Then, the ratio of the epipolar constraint error value of the preset feature point pair to the epipolar length is set as the second residual value, and the result value of introducing residuals between the preset feature point pairs is determined.

[0063] Based on the above embodiments, the method of introducing residuals between preset feature point pairs, calculating inertial compensation values ​​by combining the residuals and their derivatives with respect to inertial data, and then using the inertial compensation values ​​to correct the inertial data specifically includes: when the inertial data includes a translation vector of the reference frame image relative to the current frame image and a rotation matrix of the reference frame image relative to the current frame image, the robot marks the expression for multiplying the second rotation matrix with the normalized planar coordinates of the feature points of the preset feature point pair in the reference frame image as the second first transformation expression; then, the expression for cross-multiplying the second translation vector with the second first transformation expression is marked as the second second transformation expression; then, the expression for dot-multiplying the normalized planar coordinates of the feature points of the preset feature point pair in the current frame image with the second second transformation expression is marked as the second third transformation expression; then, the calculation result of the second second transformation expression is set to 0, forming a linear equation, and then the linear equation is plotted on the horizontal axis. The coefficients of the degree and the coefficients of the vertical axis coordinate dimension are summed and squared. The square root of the sum of squares is then calculated to obtain the second square root, which in some embodiments is equivalent to the projection length of the line represented by the linear equation in the imaging plane of the current frame image. The reciprocal of the second square root is then multiplied by the second and third transformation formulas to form the second and fourth transformation formulas. The calculation result of the second and fourth transformation formulas is then set as the second residual value to form the second residual derivation formula, and it is determined that residuals are introduced between the preset feature point pairs. Then, the second residual derivation formulas are controlled to take partial derivatives with respect to the second translation vector and the second rotation matrix (at this time, the second translation vector and the second rotation matrix are set as unknowns during the process of taking partial derivatives) to obtain the Jacobian matrix. The Jacobian matrix here is a combination of the partial derivative of the residual with respect to the second translation vector and the partial derivative of the error value with respect to the second rotation matrix, so as to achieve the effect of correcting the influence of inertial data changes.Then, solving the second residual partial derivative equation is equivalent to setting the product of the inverse of the Jacobian matrix and the second residual value as the inertia compensation value. This ensures that the inertia compensation value becomes the fitting result with the minimum error to the target data under the least squares method. In this embodiment, the second residual derivation is the function model equation to be fitted by the least squares method, and the inertia compensation value is the fitting result obtained under the least squares method. Specifically, the first residual derivation is equivalent to setting a fitting function model to fit the compensation value of the inertia data. The residuals in this model belong to error information, such as the minimum sum of squared errors under the least squares condition, making the linear equation the solution with the minimum error. The expression for the sum of squares is given; then, partial derivatives of the first residual derivation are taken with respect to the two parameters of the second translation vector and the second rotation matrix. The resulting formula can be rearranged as follows: the result of multiplying the Jacobian matrix by the compensation value of the fitted inertial data (inertial compensation value) is set to be equal to the second residual value. Then, the robot sets the product of the inverse of the Jacobian matrix and the second residual value as the inertial compensation value, thereby completing the least squares problem to find the optimal inertial compensation value. Then, the robot uses the inertial compensation value to correct the inertial data. The specific correction includes adding, subtracting, multiplying, and dividing the original inertial data. This can be simple coefficient multiplication and division, or matrix-vector multiplication.

[0064] In summary, after undergoing iterative matching of visual feature point pairs, the original inertial data yields partial derivative information that can be corrected, allowing the inertial data to be optimized and improving the robot's positioning accuracy.

[0065] It should be noted that the similarity between the descriptors is calculated using Euclidean distance or Hamming distance, which are used under standard matching conditions between descriptors. A descriptor for a feature point is a binary description vector, consisting of many 0s and 1s. These 0s and 1s encode the magnitude relationship between the brightness of two pixels near the feature point (e.g., m and n). If m is smaller than n, a 1 is used; otherwise, a 0 is used.

[0066] Step A1, the source of the descriptor specifically includes: selecting a square neighborhood centered on a feature point, and setting the square neighborhood as the region of the descriptor;

[0067] Step A2: Then, the neighborhood of the square can be denoised. Gaussian kernel convolution can be used to eliminate pixel noise because the descriptor has strong randomness and is more sensitive to noise.

[0068] Step A3: Generate point pairs using a certain randomization algorithm.<m,n> If the brightness of pixel m is less than the brightness of pixel n, it is encoded as a value of 1; otherwise, it is encoded as a value of 0.

[0069] Step A4: Repeat step 233 several times (e.g., 128 times) to obtain a 128-bit binary code, which is the descriptor of the feature point.

[0070] Preferably, the method for selecting feature points includes: selecting a pixel r in the image, assuming its brightness is Ir; then setting a threshold T0 (e.g., 20% of Ir); and then selecting 16 pixels on a circle with a radius of 3 pixels, centered on pixel r. If the brightness of 9 consecutive points on the selected circle is greater than (Ir+T0) or less than (Ir-T0), then pixel r can be considered a feature point.

[0071] It should be noted that, in the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0072] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0073] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0074] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image matching control method, characterized in that, The image matching control method is executed by a robot with a fixedly mounted camera and inertial sensor. The image matching control method includes: Step S1: The robot acquires the current frame image through the camera and obtains inertial data through the inertial sensor; Step S2: Based on the inertial data, using the epipolar constraint error value, select the first feature point pair from the feature points of the current frame image and the feature points of all reference frame images within the sliding window; wherein, the sliding window is set to be filled with at least one pre-acquired frame image. Step S3: Based on the inertial data, the second feature point pair is selected from the first feature point pair using the depth value of the feature points; Step S4: Based on the similarity of the descriptors corresponding to the second feature point pairs, select the third feature point pairs from the second feature point pairs; Step S5: Introduce a residual formula between the third feature point pairs, then use the least squares method to fit the inertia compensation value from the residual formula, and then use the inertia compensation value to correct the inertia data; then update the corrected inertia data to the inertia data described in step S2, and update the feature points of the third feature point pairs described in step S4 in the current frame image to the feature points of the current frame image described in step S2, and update the feature points of the third feature point pairs described in step S4 in the reference frame image to the feature points of all reference frame images within the sliding window described in step S2; Step S6: Repeat steps S2 and S3 until the number of repetitions reaches the preset number of iterations. Then, based on the number of feature points of the second feature point pair in each reference frame image, select matching frame images from the reference frame images in the sliding window. Each time steps S2 and S3 are repeated, the robot introduces a residual formula between the second feature point pairs selected in the latest step S3, then uses the least squares method to fit the inertial compensation value from the residual formula, then uses the inertial compensation value to correct the inertial data, then updates the corrected inertial data to the inertial data described in step S2, and updates the feature points included in the second feature point pair selected in the latest step S3 to the feature points of the current frame image and the feature points of all reference frame images in the sliding window described in step S2. Step S7: Based on the epipolar constraint error value and the number of feature points of the second feature point pair in each matching frame image, select the optimal matching frame image from all matching frame images.

2. The image matching control method according to claim 1, characterized in that, In step S2, the method for selecting the first feature point pair from the feature points of the current frame image and all reference frame images within the sliding window, based on the inertial data and utilizing the epipolar constraint error value, includes: The robot calculates the epipolar constraint error value of the feature point pair; when the epipolar constraint error value of the feature point pair is greater than or equal to the preset pixel distance threshold, the feature point pair is marked as an incorrect matching point pair; when the epipolar constraint error value of the feature point pair is less than the preset pixel distance threshold, the feature point pair is marked as the first feature point pair and the first feature point pair is selected. Each feature point pair is configured to consist of a feature point from the current frame image and a feature point from a reference frame image. Furthermore, each feature point from the current frame image forms a feature point pair with each feature point in each reference frame image within the sliding window.

3. The image matching control method according to claim 2, characterized in that, In step S2, when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix. Then, the robot controls the first rotation matrix to transform the normalized vector of the feature points of the current frame image to the coordinate system of the reference frame image, obtaining the first one vector. Then, the robot controls the first translation vector to cross-multiply the first one vector to obtain the first two vectors. Then, the robot controls the normalized vector of the feature points in the reference frame image within the sliding window to dot-multiply the first two vectors, and then sets the result of the dot-multiply as the epipolar constraint error value of the corresponding feature point pair. Alternatively, in step S2, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot marks the translation vector of the reference frame image relative to the current frame image as the second translation vector and the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix. Then, the robot controls the second rotation matrix to transform the normalized plane vector of the feature points of the reference frame image within the sliding window to the coordinate system of the current frame image, obtaining the second one vector. Then, the robot controls the second translation vector to cross-multiply the second one vector to obtain the second two vector. Then, the robot controls the normalized plane vector of the feature points in the current frame image to dot-multiply the second two vector, and then sets the result of the dot-multiply as the epipolar constraint error value of the corresponding feature point pair. Wherein, the normalized vector of the feature points of the current frame image is the vector formed by the normalized planar coordinates of the feature points of the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the reference frame image is the vector formed by the normalized planar coordinates of the feature points of the reference frame image relative to the origin of the coordinate system of the reference frame image.

4. The image matching control method according to claim 1, characterized in that, In step S3, the method for selecting a second pair of feature points from the first pair of feature points based on the inertial data and using the depth values ​​of the feature points includes: The robot calculates the ratio of the depth value of the feature points of the first feature point pair selected in step S2 in the current frame image to the depth value of the feature points of the first feature point pair in the reference frame image; When the ratio of the depth value of the feature point of the first feature point pair in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image is within a preset ratio threshold range, the first feature point pair is marked as the second feature point pair and the second feature point pair is selected. When the ratio of the depth value of the feature point of the first feature point pair in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image is not within the preset ratio threshold range, the first feature point pair is marked as an incorrect matching point pair.

5. The image matching control method according to claim 4, characterized in that, In step S3, when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix. The robot controls the first rotation matrix to transform the normalized vector of the first feature point pair in the current frame image to the coordinate system of the reference frame image, obtaining the first one vector; then, the robot controls the normalized vector of the first feature point pair in the reference frame image to cross-multiply the first one vector, obtaining the first two vectors; simultaneously, the robot controls the normalized vector of the first feature point pair in the reference frame image to cross-multiply the first one vector. The vector is shifted, and the cross product result is inverted to obtain the first third vector. Then, the product of the first third vector and the inverse of the first second vector is set as the depth value of the feature point of the first feature point pair in the current frame image, and is marked as the first depth value, representing the distance between the 3D point detected by the camera and the optical center when the camera captures the current frame image. Then, the sum of the product of the first first vector and the first depth value and the first shift vector is marked as the first fourth vector. Then, the product of the first fourth vector and the inverse of the normalized vector of the feature point of the first feature point pair in the reference frame image is set as the depth value of the feature point of the first feature point pair in the reference frame image, and is marked as the second depth value, representing the distance between the same 3D point and the optical center when the camera captures the reference frame image. Alternatively, in step S3, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot denotes the translation vector of the reference frame image relative to the current frame image as the second translation vector and the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix. The robot controls the second rotation matrix to transform the normalized vector of the first feature point pair in the reference frame image to the coordinate system of the current frame image, obtaining the second one vector; then, the robot controls the normalized vector of the first feature point pair in the current frame image to cross-multiply the second one vector, obtaining the second two vector; simultaneously, the robot controls the normalized vector of the first feature point pair in the current frame image to cross-multiply the second one vector. The translation vector is then inverted, and the cross product result is used to obtain the second third vector. The product of the second third vector and the inverse of the second second vector is then set as the depth value of the feature point of the first feature point pair in the reference frame image, and labeled as the second depth value, representing the distance between the 3D point detected by the camera and the optical center when the camera captures the reference frame image. The sum of the product of the second first vector and the second depth value, and the second translation vector, is then labeled as the second fourth vector. The product of the second fourth vector and the inverse of the normalized vector of the feature point of the first feature point pair in the current frame image is then set as the depth value of the feature point of the first feature point pair in the current frame image, and labeled as the first depth value, representing the distance between the same 3D point and the optical center when the camera captures the current frame image. Wherein, the normalized vector of the feature points of the first feature point pair in the current frame image is the vector formed by the normalized planar coordinates of the feature points of the first feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the first feature point pair in the reference frame image is the vector formed by the normalized planar coordinates of the feature points of the first feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image.

6. The image matching control method according to claim 1, characterized in that, In step S4, the method for selecting a third feature point pair from the second feature point pair based on the similarity of the descriptors corresponding to the second feature point pair includes: For the current frame image and each reference frame image within the sliding window, the robot calculates the similarity between the descriptor of the feature points of the second feature point pair in the reference frame image and the descriptor of the feature points of the second feature point pair in the current frame image. When the similarity between the descriptor of the feature point in the reference frame image of the second feature point pair and the descriptor of the feature point in the current frame image is the minimum value among the similarity between the descriptor of the current frame image and the descriptor of the reference frame image where the feature point of the second feature point pair is located, the second feature point pair is marked as the third feature point pair and the third feature point pair is selected. Wherein, the descriptor of the reference frame image where the feature points of the second feature point pair are located is the descriptor of all feature points that make up the second feature point pair within the reference frame image where the feature points of the second feature point pair are located; the descriptor of the current frame image is the descriptor of the feature points that, together with the feature points of the reference frame image where the feature points of the second feature point pair are located, form the second feature point pair within the current frame image. The similarity between the descriptors corresponding to the second feature point pairs is represented by the Euclidean distance or Hamming distance between the descriptors of the feature points in the current frame image and the descriptors of the feature points in the corresponding reference frame image within the sliding window.

7. The image matching control method according to claim 6, characterized in that, Step S4 further includes: whenever the robot searches all feature points that form a second feature point pair between the current frame image and a reference frame image within the sliding window, if the number of third feature point pairs counted by the robot in the reference frame image is less than or equal to a first preset point threshold, then it is determined that the current frame image and the reference frame image have failed to match, and the reference frame image is set as a mismatched reference frame image; if the number of third feature point pairs counted by the robot in the reference frame image is greater than the first preset point threshold, then it is determined that the current frame image and the reference frame image have matched successfully. Specifically, when the robot determines that the current frame image and all frame reference frames within the sliding window fail to match, it determines that the robot's tracking using the window matching method has failed, and then the robot clears the images within the sliding window.

8. The image matching control method according to claim 1, characterized in that, The robot marks the line connecting the optical center of the current frame image captured by the camera with the feature points of the same preset feature point pair in the current frame image as the first observation line, and the line connecting the optical center of the reference frame image captured by the camera with the feature points of the same preset feature point pair in the reference frame image as the second observation line. Then, the intersection of the first observation line and the second observation line is marked as the target detection point. In this configuration, the preset feature point pair, the optical center when the camera captures the current frame image, and the optical center when the camera captures the reference frame image are all located in the same plane; or, the optical center when the camera captures the current frame image, the optical center when the camera captures the reference frame image, and the target detection point are all located in the same plane; the same plane is a polar plane; The robot records the intersection of the polar plane with the current frame image as the epipolar line in the imaging plane of the current frame image, and records the intersection of the polar plane with the reference frame image as the epipolar line in the imaging plane of the reference frame image. In the same preset feature point pair, after the feature point from the current frame image is transformed to the reference frame image, it becomes the first projection point, and its coordinates are the first coordinates; the distance from the first projection point to the epipolar line in the imaging plane of the reference frame image is represented as the first residual value; in the same preset feature point pair, after the feature point from the reference frame image is transformed to the current frame image, it becomes the second projection point, and its coordinates are the second coordinates; the distance from the second projection point to the epipolar line in the imaging plane of the current frame image is represented as the second residual value; In step S5, the preset feature point pair is the third feature point pair; In step S6, each time step S2 and step S3 are executed repeatedly, the preset feature point pair is the second feature point pair selected by the most recently executed step S3.

9. The image matching control method according to claim 8, characterized in that, In step S5 or step S6, when the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot denotes the translation vector of the current frame image relative to the reference frame image as the first translation vector and the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix. The robot controls the first rotation matrix to transform the normalized vector of the feature points of the preset feature point pair in the current frame image to the coordinate system of the reference frame image, obtaining the first one vector; then, the robot controls the first translation vector to cross-multiply the first one vector to obtain the first two vectors, forming the epipolar line in the imaging plane of the reference frame image; then, the robot takes the square root of the sum of the squares of the horizontal axis coordinates and the vertical axis coordinates of the first two vectors to obtain the magnitude of the epipolar line; simultaneously, the robot controls the normalized vector of the feature points of the preset feature point pair in the reference frame image to dot-multiply the first two vectors, and then sets the result of the dot-multiply as the epipolar constraint error value of the preset feature point pair; then, the ratio of the epipolar constraint error value of the preset feature point pair to the magnitude of the epipolar line is set as the first residual value. Alternatively, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot denotes the translation vector of the reference frame image relative to the current frame image as the second translation vector and the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix. The robot controls the second rotation matrix to transform the normalized vector of the feature points of the preset feature point pair in the reference frame image to the coordinate system of the current frame image, obtaining the second one vector; then, the robot controls the second translation vector to cross-multiply the second one vector to obtain the second two vector, which forms the epipolar line in the imaging plane of the current frame image; then, the robot takes the square root of the sum of the squares of the horizontal axis coordinates and the vertical axis coordinates of the second two vectors to obtain the magnitude of the epipolar line; simultaneously, the robot controls the normalized vector of the feature points of the preset feature point pair in the current frame image to dot-multiply the second two vector, and then sets the result of the dot-multiply as the epipolar constraint error value of the preset feature point pair; then, the ratio of the epipolar constraint error value of the preset feature point pair to the magnitude of the epipolar line is set as the second residual value. Wherein, the normalized vector of the feature points of the preset feature point pair in the current frame image is the vector formed by the normalized planar coordinates of the feature points of the preset feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the preset feature point pair in the reference frame image is the vector formed by the normalized planar coordinates of the feature points of the preset feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image.

10. The image matching control method according to claim 9, characterized in that, In step S5 or step S6, the method of introducing a residual formula between preset feature point pairs, fitting an inertial compensation value from the residual formula using the least squares method, and then using the inertial compensation value to correct the inertial data includes: When the inertial data includes the translation vector of the current frame image relative to the reference frame image and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the expression for multiplying the first rotation matrix with the normalized vector of the feature points of the preset feature point pair in the current frame image as the first transformation expression; then it marks the expression for cross-multiplying the first translation vector with the first transformation expression as the first transformation expression; then it marks the expression for dot-multiplying the expression for dot-multiplying the normalized vector of the feature points of the preset feature point pair in the reference frame image with the first transformation expression as the first transformation expression; and then it sets the calculation result of the first transformation expression to 0, forming a linear equation, and then calculates the sum of squares of the coefficients of the linear equation in the horizontal axis and the vertical axis, and then calculates the square of the obtained sum of squares. The first square root is obtained, and the reciprocal of the first square root is multiplied by the first three transformation formula to form the first four transformation formula. The result of the first four transformation formula is then set as the first residual value, forming the first residual derivation formula, and the residual formula is determined to be introduced between the preset feature point pairs. Then, the first residual derivation formula is controlled to take partial derivatives with respect to the first translation vector and the first rotation matrix to obtain the Jacobian matrix. The product of the inverse of the Jacobian matrix and the first residual value is then set as the inertial compensation value. The robot then uses the inertial compensation value to correct the inertial data. Here, the first residual derivation formula is the function model equation that needs to be fitted by the least squares method, corresponding to the residual formula introduced between the preset feature point pairs; the inertial compensation value is the fitting result obtained by the least squares method. Alternatively, when the inertial data includes the translation vector of the reference frame image relative to the current frame image and the rotation matrix of the reference frame image relative to the current frame image, the robot marks the expression for multiplying the second rotation matrix by the normalized vector of the feature points of the preset feature point pair in the reference frame image as the second-first transformation expression; then, it marks the expression for cross-multiplying the second translation vector by the second-first transformation expression as the second-second transformation expression; then, it marks the expression for dot-multiplying the expression for dot-multiplying the normalized vector of the feature points of the preset feature point pair in the current frame image by the second-second transformation expression as the second-third transformation expression; and sets the calculation result of the second-second transformation expression to 0, forming a linear equation, then summing the squares of the coefficients of this linear equation in the horizontal and vertical coordinate dimensions, and then... The square root of the sum of squares is calculated to obtain the second square root. The reciprocal of the second square root is then multiplied by the second and third transformation formulas to form the second and fourth transformation formulas. The result of the second and fourth transformation formulas is then set as the second residual value, forming the second residual derivation formula. The residual formula is then introduced between the preset feature point pairs. The second residual derivation formula is then used to calculate the partial derivatives with respect to the second translation vector and the second rotation matrix to obtain the Jacobian matrix. The product of the inverse of the Jacobian matrix and the second residual value is then set as the inertial compensation value. The robot then uses the inertial compensation value to correct the inertial data. Here, the second residual derivation formula is the function model equation to be fitted by the least squares method, and the inertial compensation value is the fitting result obtained by the least squares method.

11. The image matching control method according to claim 3, characterized in that, For step S6, after the robot has completed step S5, when step S2 is repeated for the first time, the robot calculates the epipolar constraint error value of each third feature point pair except for the mismatched reference frame image. The epipolar constraint error value of each third feature point pair is determined by the inertial data corrected in step S5. When the epipolar constraint error value of the third feature point pair is less than the preset pixel distance threshold, the third feature point pair is updated to the first feature point pair, and a new first feature point pair is selected from the third feature point pairs. When step S2 is repeated for the Nth time, the robot calculates the epipolar constraint error value of each second feature point pair selected in the latest step S3; when the epipolar constraint error value of the second feature point pair is less than the preset pixel distance threshold, the second feature point pair is updated to the first feature point pair, and a new first feature point pair is selected from all the second feature point pairs selected in step S3; where N is set to be greater than 1 and less than or equal to the preset number of iterations.

12. The image matching control method according to claim 4, characterized in that, In step S6, the method for selecting matching frame images from the reference frame images within the sliding window based on the number of feature points of the second feature point pair within each reference frame image includes: The robot counts the number of feature points of the second feature point pair in each reference frame image within the sliding window; If the number of second feature point pairs matched by the robot in one of the reference frame images is less than or equal to the second preset point threshold, then it is determined that the one of the reference frame images has failed to match the current frame image; if the number of second feature point pairs matched by the robot in one of the reference frame images is greater than the second preset point threshold, then it is determined that the one of the reference frame images has successfully matched the current frame image, and the one of the reference frame images is set as the matching frame image; if the number of second feature point pairs matched by the robot in each reference frame image is less than or equal to the second preset point threshold, then it is determined that each reference frame image in the sliding window has failed to match the current frame image, and the robot's tracking using the window matching method has failed.

13. The image matching control method according to claim 1, characterized in that, In step S7, the method for selecting the optimal matching frame image from all matching frame images based on the epipolar constraint error value and the number of feature points of the second feature point pair in each matching frame image includes: Within each matching frame image, the sum of the epipolar constraint error values ​​of the second feature point pair to which the feature point belongs within the matching frame image is calculated, and used as the cumulative epipolar constraint error value of the matching frame image. Within each matching frame image, the number of feature points that form the second feature point pair is counted, and this number is taken as the feature point matching count of the matching frame image. Then, the matching frame image with the smallest cumulative extreme constraint error value and the largest number of matching feature points is set as the optimal matching frame image.

Images