A robot visual tracking method

Abstract

The invention discloses a robot visual tracking method. The execution subject of the robot visual tracking method is a robot equipped with a camera and an inertial sensor. The robot visual tracking method comprises the following steps: the robot uses a window matching method to perform image tracking. When the robot successfully performs image tracking using the window matching method, the robot stops using the window matching method to perform image tracking, and then the robot uses a projection matching method to perform image tracking. Then, when the robot fails to perform image tracking using the projection matching method, the robot stops using the projection matching method to perform image tracking, and then the robot uses the window matching method to perform image tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a robot vision tracking method. Background Art

[0002] Visual-inertial odometry (VIO), sometimes also called visual-inertial system (VINS), is an algorithm that integrates camera and inertial measurement unit (IMU) sensor data to achieve SLAM. The traditional classic VIO solution starts with the initialization phase of pure visual SFM (Structure from Motion) using feature points, and then restores the measurement scale, speed, gravity acceleration direction and IMU zero bias by loosely coupling and aligning the structure with the IMU pre-integrated measurement value. SLAM (simultaneous localization and mapping) refers to the robot moving from an unknown position in an unknown environment, positioning itself according to the position estimate and map during the movement, and building an incremental map based on its own positioning to achieve autonomous positioning and navigation of the robot.

[0003] At present, the point feature-based SLAM algorithm uses feature points with projection relationships as the basis, and can perform feature tracking, mapping, and closed-loop detection in real time, completing the entire process of simultaneous positioning and mapping. However, feature tracking (feature extraction and matching) consumes a lot of computing power, reducing the real-time performance of robot positioning and navigation. Summary of the invention

[0004] In order to solve the above technical defects, the present invention discloses a robot visual tracking method, and the specific technical solution is as follows:

[0005] A robot visual tracking method, wherein the execution subject of the robot visual tracking method is a robot fixedly equipped with a camera and an inertial sensor; the robot visual tracking method comprises: the robot uses a window matching method to track an image, when the robot successfully tracks using the window matching method, the robot stops using the window matching method to track the image, and then the robot uses a projection matching method to track the image; then, when the robot fails to track using the projection matching method, the robot stops using the projection matching method to track the image, and then the robot uses the window matching method to track the image.

[0006] Furthermore, the robot visual tracking method also includes: when the robot fails to track using the window matching method, the robot stops using the window matching method for image tracking, the robot clears the sliding window, and then uses the window matching method for image tracking; wherein, image tracking is used to represent the match between the feature points of the previously captured image and the feature points of the current frame image; wherein, after the robot successfully tracks using the window matching method, the current frame image is filled into the sliding window to facilitate tracking of the image captured by the robot in real time; during the process of the robot using the projection matching method for image tracking, if it is detected that the time interval between the current frame image and the previous frame image exceeds a preset time threshold, the robot stops using the projection matching method for image tracking and instead uses the window matching method for image tracking; wherein, the feature point is a pixel point belonging to the image, and the feature point is an environmental element that exists in the form of a point in the environment where the camera is located.

[0007] Furthermore, the method for the robot to perform image tracking using a window matching method includes: step S11, the robot collects the current frame image through a camera and obtains inertial data through an inertial sensor; step S12, based on the inertial data, using the epipolar constraint error value, screening out a first feature point pair from the feature points of the current frame image and the feature points of all reference frame images in the sliding window; wherein the sliding window is set to fill in at least one frame of image collected in advance; the feature point is the pixel point of the image, and the feature point is an environmental element that exists in the form of a point in the environment where the camera is located; step S13, based on the inertial data, using the depth value of the feature point, from the first A second feature point pair is selected from the first feature point pair; in step S14, a third feature point pair is selected from the second feature point pair according to the similarity of the descriptors corresponding to the second feature point pair; in step S15, a residual is introduced between the third feature point pair, and an inertia compensation value is calculated by combining the residual and its derivative result with respect to the inertia data, and the inertia compensation value is used to correct the inertia data; then the corrected inertia data is updated to the inertia data described in step S12, and the feature points of the third feature point pair described in step S14 in the current frame image are updated to the feature points of the current frame image described in step S12, and the feature points of the third feature point pair described in step S14 in the reference frame image are updated to Step S16, repeating steps S12 and S13 until the number of repetitions reaches a preset number of iterative matching times, and then filtering out matching frame images from the reference frame images in the sliding window based on the number of feature points of the second feature point pairs in each frame of the reference frame image; wherein, each time after repeating steps S12 and S13, the robot introduces a residual between the second feature point pairs filtered out by the most recently executed step S13, and then combines the residual and the result of its derivation with respect to the most recently acquired inertial data to calculate the inertial compensation value, and then uses the inertial compensation value to correct the most recently acquired inertial data, and then Then, the corrected inertial data is updated to the inertial data described in step S12, and the feature points included in the second feature point pair screened out by the most recently executed step S13 are updated to the feature points of the current frame image described in step S12 and the feature points of all reference frame images in the sliding window; step S17, based on the epipolar constraint error value and the number of feature points of the second feature point pair in each frame matching frame image, the optimal matching frame image is selected from all matching frame images, and it is determined that the robot has successfully tracked using the window matching method; wherein, the combination of step S12, step S13, step S14, step S15, step S16 and step S17 is the window matching method.

[0008] Further, in the step S12, based on the inertial data, using the epipolar constraint error value, the method of selecting the first feature point pair from the feature points of the current frame image and the feature points of all the reference frame images in the sliding window includes: the robot calculates the epipolar constraint error value of the feature point pair; when the epipolar constraint error value of the feature point pair is greater than or equal to the preset pixel distance threshold, the feature point pair is marked as an incorrectly matched point pair; when the epipolar constraint error value of the feature point pair is less than the preset pixel distance threshold, the feature point pair is marked as a first feature point pair and the first feature point pair is determined to be selected; wherein each feature point pair is configured to consist of a feature point of the current frame image and a feature point of the reference frame image, and each feature point of the current frame image forms a feature point pair with each feature point in each reference frame image in the sliding window.

[0009] Further, in the step S12, when the inertial data includes the translation vector of the current frame image relative to the reference frame image, and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as the first translation vector, and marks the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix; then, the robot controls the first rotation matrix to transform the normalized vector of the feature point of the current frame image into the coordinate system of the reference frame image to obtain a first vector; then, the robot controls the first translation vector to cross-multiply the first vector to obtain a first vector; then, the robot controls the normalized vector of the feature point in the reference frame image within the sliding window to dot-multiply the first vector, and then sets the result of the dot-multiplication as the epipolar constraint error value of the corresponding feature point pair; or, in the step S12, when the inertial data includes the translation vector of the reference frame image relative to the current frame image, and the rotation matrix of the reference frame image relative to the current frame image When converting the matrix, the robot marks the translation vector of the reference frame image relative to the current frame image as the second translation vector, and marks the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix; then, the robot controls the second rotation matrix to convert the normalized plane vectors of the feature points of the reference frame image in the sliding window to the coordinate system of the current frame image to obtain a second first vector; then, the robot controls the second translation vector to cross-multiply the second first vector to obtain a second second vector; then, the robot controls the normalized plane vectors of the feature points in the current frame image to be dot-multiplied with the second second vector, and then sets the result of the dot multiplication as the epipolar constraint error value of the corresponding feature point pair; wherein, the normalized vectors of the feature points of the current frame image are the vectors formed by the normalized plane coordinates of the feature points of the current frame image relative to the origin of the coordinate system of the current frame image; and the normalized vectors of the feature points of the reference frame image are the vectors formed by the normalized plane coordinates of the feature points of the reference frame image relative to the origin of the coordinate system of the reference frame image.

[0010] Further, in the step S13, based on the inertial data, the method of selecting the second feature point pair from the first feature point pair using the depth value of the feature point includes: the robot calculates the ratio of the depth value of the feature point of the first feature point pair selected in the step S12 in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image; when the ratio of the depth value of the feature point of the first feature point pair in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image is within a preset ratio threshold range, marking the first feature point pair as a second feature point pair and determining to select the second feature point pair; when the ratio of the depth value of the feature point of the first feature point pair in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image is not within the preset ratio threshold range, marking the first feature point pair as an incorrectly matched point pair.

[0011] Further, in step S13, when the inertial data includes a translation vector of the current frame image relative to the reference frame image, and a rotation matrix of the current frame image relative to the reference frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as a first translation vector, and marks the rotation matrix of the current frame image relative to the reference frame image as a first rotation matrix, and the robot controls the first rotation matrix to convert the normalized vector of the feature point of the first feature point pair in the current frame image to the coordinate system of the reference frame image to obtain a first vector; and then controls the normalized vector of the feature point of the first feature point pair in the reference frame image to cross-multiply the first vector to obtain A first two-vector; at the same time, the normalized vector of the feature point of the first feature point pair in the reference frame image is controlled to be cross-multiplied by the first translation vector, and then the cross-multiplication result is inverted to obtain a first three-vector; then the product of the first three-vector and the inverse vector of the first two-vector is set as the depth value of the feature point of the first feature point pair in the current frame image, and is marked as the first depth value, indicating the distance between the three-dimensional point detected by the camera and the optical center when the camera collects the current frame image; then the sum of the product of the first one-vector and the first depth value and the first translation vector is marked as a first four-vector, and then the first four-vector and the normalized vector of the feature point of the first feature point pair in the reference frame image are obtained. The product of the inverse vector of is set as the depth value of the feature point of the first feature point pair in the reference frame image, and is marked as the second depth value, which represents the distance between the same three-dimensional point and the optical center when the camera collects the reference frame image; or, in step S13, when the inertial data includes the translation vector of the reference frame image relative to the current frame image, and the rotation matrix of the reference frame image relative to the current frame image, the robot records the translation vector of the reference frame image relative to the current frame image as the second translation vector, and records the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix, and the robot controls the second rotation matrix to convert the first feature point pair in the reference frame image The normalized vector of the feature point is converted to the coordinate system of the current frame image to obtain a second one-vector; then the normalized vector of the feature point of the first feature point pair in the current frame image is controlled to cross-multiply the second one-vector to obtain a second two-vector; at the same time, the normalized vector of the feature point of the first feature point pair in the current frame image is controlled to cross-multiply the second translation vector, and then the cross-multiplication result is negated to obtain a second three-vector; then the product of the second three-vector and the inverse vector of the second two-vector is set as the depth value of the feature point of the first feature point pair in the reference frame image, and marked as the second depth value, indicating the distance between the three-dimensional point detected by the camera and the optical center when the camera collects the reference frame image;Then the product of the second vector and the second depth value and the sum of the second translation vector are marked as the second four-vector, and then the product of the second four-vector and the inverse vector of the normalized vector of the feature point of the first feature point pair in the current frame image is set as the depth value of the feature point of the first feature point pair in the current frame image, and marked as the first depth value, indicating the distance between the same three-dimensional point and the optical center when the camera collects the current frame image; wherein the normalized vector of the feature point of the first feature point pair in the current frame image is the vector formed by the normalized plane coordinates of the feature point of the first feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature point of the first feature point pair in the reference frame image is the vector formed by the normalized plane coordinates of the feature point of the first feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image. ;

[0012] Further, in the step S14, the method for selecting a third feature point pair from the second feature point pair according to the similarity of the descriptors corresponding to the second feature point pair includes: for the current frame image and each reference frame image in the sliding window, the robot calculates the similarity between the descriptors of the feature points of the second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image; when the similarity between the descriptors of the feature points of the second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image is within the similarity between the descriptors of the current frame image and the descriptors of the reference frame image where the feature points of the second feature point pair are located When the minimum value of is reached, the second feature point pair is marked as the third feature point pair and the third feature point pair is determined to be screened out; wherein the descriptor of the reference frame image where the feature point of the second feature point pair is located is the descriptor of all feature points constituting the second feature point pair in the reference frame image where the feature point of the second feature point pair is located; the descriptor of the current frame image is the descriptor of the feature points in the current frame image that constitute the second feature point pair together with the feature points of the reference frame image where the feature point of the second feature point pair is located; wherein the similarity of the descriptors corresponding to the second feature point pair is represented by the Euclidean distance or the Hamming distance between the descriptors of the feature points in the current frame image and the descriptors of the feature points in the corresponding reference frame image in the sliding window.

[0013] Furthermore, the step S14 also includes: whenever the robot searches for all feature points constituting the second feature point pairs between the current frame image and a frame reference frame image in the sliding window, if the number of third feature point pairs counted by the robot in the frame reference frame image is less than or equal to the first preset point count threshold, it is determined that the current frame image and the frame reference frame image have failed to match, and the frame reference frame image is set as a mismatched reference frame image; if the number of third feature point pairs counted by the robot in the frame reference frame image is greater than the first preset point count threshold, it is determined that the current frame image and the frame reference frame image have successfully matched; wherein, when the robot determines that the current frame image and all frame reference frame images in the sliding window have failed to match, it is determined that the robot has failed to track using the window matching method, and then the robot clears the image in the sliding window.

[0014] Furthermore, the robot marks the line connecting the optical center when the camera collects the current frame image and the feature points of the preset feature point pair in the current frame image as the first observation line, and marks the line connecting the optical center when the camera collects the reference frame image and the feature points of the same preset feature point pair in the reference frame image as the second observation line, and then marks the intersection of the first observation line and the second observation line as the target detection point; wherein the preset feature point pair, the optical center when the camera collects the current frame image, and the optical center when the camera collects the reference frame image are all in the same plane; or, the optical center when the camera collects the current frame image, the optical center when the camera collects the reference frame image, and the target detection point are all in the same plane; the same plane is the polar plane; the robot records the intersection of the polar plane and the current frame image as the polar line in the imaging plane of the current frame image , and the intersection line of the polar plane and the reference frame image is recorded as the polar line of the imaging plane of the reference frame image; in the same preset feature point pair, after the feature point of the current frame image is converted to the reference frame image, it becomes the first projection point, and its coordinates are the first coordinates; the distance from the first projection point to the polar line in the imaging plane of the reference frame image is represented as the first residual value; in the same preset feature point pair, after the feature point of the reference frame image is converted to the current frame image, it becomes the second projection point, and its coordinates are the second coordinates; the distance from the second projection point to the polar line in the imaging plane of the current frame image is represented as the second residual value; in step S15, the preset feature point pair is the third feature point pair; in step S16, each time after repeating step S12 and step S13, the preset feature point pair is the second feature point pair selected by the most recently executed step S13.

[0015] Further, in step S15 or step S16, the method of introducing the residual includes: when the inertial data includes a translation vector of the current frame image relative to the reference frame image, and a rotation matrix of the current frame image relative to the reference frame image, the robot records the translation vector of the current frame image relative to the reference frame image as a first translation vector, and records the rotation matrix of the current frame image relative to the reference frame image as a first rotation matrix, the robot controls the first rotation matrix to convert the normalized vector of the feature point of the preset feature point pair in the current frame image to the coordinate system of the reference frame image to obtain a first vector; and then controls the first translation vector to cross-multiply the first vector to obtain a first rotation matrix. vector, and form the epipolar line in the imaging plane of the reference frame image; then take the square root of the sum of the squares of the horizontal axis coordinate in the first two vectors and the vertical axis coordinate in the first two vectors to obtain the modulus of the epipolar line; at the same time, control the normalized vector of the feature point of the preset feature point pair in the reference frame image to be dot-multiplied with the first two vectors, and then set the result of the dot-multiplication as the epipolar constraint error value of the preset feature point pair; then set the ratio of the epipolar constraint error value of the preset feature point pair to the modulus of the epipolar line as the first residual value; or, when the inertial data includes the translation vector of the reference frame image relative to the current frame image, and the rotation matrix of the reference frame image relative to the current frame image, the robot sets the reference The translation vector of the reference frame image relative to the current frame image is recorded as the second translation vector, and the rotation matrix of the reference frame image relative to the current frame image is recorded as the second rotation matrix. The robot controls the second rotation matrix to convert the normalized vector of the feature point of the preset feature point pair in the reference frame image to the coordinate system of the current frame image to obtain the second first vector; then the second translation vector is controlled to cross-multiply the second first vector to obtain the second second vector, and form the polar line in the imaging plane of the current frame image; then the square root of the sum of the squares of the horizontal axis coordinate in the second second vector and the vertical axis coordinate in the second second vector is calculated to obtain the modulus of the polar line; at the same time, the feature point of the preset feature point pair in the current frame image is controlled to be The normalized vector of the point is dot-multiplied with the second two vectors, and the result of the dot multiplication is set as the epipolar constraint error value of the preset feature point pair; then the ratio of the epipolar constraint error value of the preset feature point pair to the modulus of the epipolar line is set as the second residual value; wherein, the normalized vector of the feature points of the preset feature point pair in the current frame image is the vector formed by the normalized plane coordinates of the feature points of the preset feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the preset feature point pair in the reference frame image is the vector formed by the normalized plane coordinates of the feature points of the preset feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image.

[0016] Further, in step S15 or step S16, a residual is introduced between the preset feature point pairs, and then the inertia compensation value is calculated by combining the residual and the result of derivation of the latest inertial data, and then the inertia compensation value is used to correct the latest inertial data. The method includes: when the inertial data includes a translation vector of the current frame image relative to the reference frame image, and a rotation matrix of the current frame image relative to the reference frame image, the robot marks the formula of multiplying the first rotation matrix and the normalized vector of the feature point of the preset feature point pair in the current frame image as a first-to-first conversion formula; then marks the formula of cross-multiplying the first translation vector and the first-to-first conversion formula as a first-to-second conversion formula; then marks the formula of cross-multiplying the preset feature point pair in the reference frame image as a first-to-second conversion formula. The formula for multiplying the normalized vector of the feature point with the first two-transformation formula is marked as the first three-transformation formula; the calculation result of the first two-transformation formula is set to the value 0 to form a straight line equation, and then the square sum of the coefficients of the straight line equation in the horizontal axis coordinate dimension and the coefficients of the vertical axis coordinate dimension is calculated, and then the square root of the obtained square sum is calculated to obtain the first square root, and then the formula for multiplying the reciprocal of the first square root with the first three-transformation formula is set as the first four-transformation formula; then the calculation result of the first four-transformation formula is set as the first residual value to form a first residual derivation formula, and it is determined to introduce a residual between the preset feature point pairs; then the first residual derivation formula is controlled to respectively calculate the partial derivatives of the first translation vector and the first rotation matrix to obtain the Jacobian matrix; then The product of the inverse matrix of the Jacobian matrix and the first residual value is set as the inertia compensation value; then the robot uses the inertia compensation value to correct the inertia data; or, when the inertia data includes the translation vector of the reference frame image relative to the current frame image, and the rotation matrix of the reference frame image relative to the current frame image, the robot marks the formula of multiplying the second rotation matrix with the normalized vector of the feature point of the preset feature point pair in the reference frame image as the second-first transformation formula; then marks the formula of cross-multiplying the second translation vector with the second-first transformation formula as the second-second transformation formula; then marks the formula of dot-multiplying the normalized vector of the feature point of the preset feature point pair in the current frame image with the second-second transformation formula as the second-third transformation formula; and then sets the calculation result of the second-second transformation formula to The value is 0 to form a straight line equation, and then the square sum of the coefficients of the straight line equation in the horizontal axis coordinate dimension and the coefficients of the vertical axis coordinate dimension is calculated, and then the square root of the obtained square sum is calculated to obtain the second square root, and then the formula of multiplying the reciprocal of the second square root and the second third conversion formula is set to the second fourth conversion formula; then the calculation result of the second fourth conversion formula is set to the second residual value to form a second residual derivation formula, and it is determined to introduce a residual between the preset feature point pairs; then the second residual derivation formula is controlled to respectively calculate the partial derivative of the second translation vector and the second rotation matrix to obtain the Jacobian matrix; then the product of the inverse matrix of the Jacobian matrix and the second residual value is set as the inertia compensation value; then the robot uses the inertia compensation value to correct the inertia data.

[0017] Further, for step S16, after the robot executes step S15, when step S12 is repeated for the first time, the robot calculates the epipolar constraint error value of each third feature point pair except the mismatched reference frame image, wherein the epipolar constraint error value of each third feature point pair is determined by the inertial data corrected in step S15; when the epipolar constraint error value of the third feature point pair is less than the preset pixel distance threshold, the third feature point pair is updated to the first feature point pair, and a new first feature point pair is determined to be selected from the third feature point pairs; when step S12 is repeated for the Nth time, the robot calculates the epipolar constraint error value of each second feature point pair selected in the most recently executed step S13; when the epipolar constraint error value of the second feature point pair is less than the preset pixel distance threshold, the second feature point pair is updated to the first feature point pair, and a new first feature point pair is determined to be selected from all the second feature point pairs selected in step S13; wherein N is set to be greater than 1 and less than or equal to the preset number of iterative matching times.

[0018] Further, in step S16, the method for selecting matching frame images from the reference frame images in the sliding window based on the number of feature points of the second feature point pairs in each frame of the reference frame image includes: the robot counts the number of feature points of the second feature point pairs in each frame of the reference frame image in the sliding window; if the number of second feature point pairs matched by the robot in one of the reference frame images is less than or equal to the second preset point threshold, it is determined that the one of the reference frame images fails to match the current frame image; if the number of second feature point pairs matched by the robot in one of the reference frame images is greater than the second preset point threshold, it is determined that the one of the reference frame images successfully matches the current frame image, and the one of the reference frame images is set as the matching frame image; if the number of second feature point pairs matched by the robot in each frame of the reference frame image is less than or equal to the second preset point threshold, it is determined that each frame of the reference frame image in the sliding window fails to match the current frame image, and it is determined that the robot fails to track using the window matching method.

[0019] Further, in the step S17, the method for selecting the best matching frame image from all matching frame images based on the epipolar constraint error value and the number of feature points of the second feature point pair in each frame matching frame image includes: in each frame matching frame image, calculating the sum of the epipolar constraint error values ​​of the second feature point pairs to which the feature points in the frame matching frame image belong, as the accumulated value of the epipolar constraint error values ​​of the frame matching frame image; in each frame matching frame image, counting the number of feature points constituting the second feature point pairs in the frame matching frame image, as the number of feature point matches of the frame matching frame image; and then setting the matching frame image with the smallest accumulated value of the epipolar constraint error value and the largest number of feature point matches as the best matching frame image.

[0020] Furthermore, the method for the robot to perform image tracking using projection matching includes: step S21, the robot collects images through a camera and obtains inertial data through an inertial sensor; wherein the image collected by the camera includes a previous frame image and a current frame image; step S22, the robot uses the inertial data to project the feature points of the previous frame image into the current frame image to obtain projection points, wherein the inertial data includes a rotation matrix of the previous frame image relative to the current frame image, and a translation vector of the previous frame image relative to the current frame image; step S23, the robot searches for the preset points at each projection point according to the standard distance between the descriptors. Search for the point to be matched in the neighborhood; then the robot calculates the vector between the projection point and each searched point to be matched, and marks the vector pointing from the projection point to the searched point to be matched as the vector to be matched; wherein the point to be matched is a feature point from the current frame image, and the point to be matched does not belong to the projection point; each vector to be matched corresponds to a projection point and a point to be matched; step S24, the robot counts the number of mutually parallel vectors to be matched, and when the number of mutually parallel vectors to be matched is greater than or equal to the preset matching number, it is determined that the robot has successfully tracked using the projection matching method, and that the robot has successfully tracked the current frame image.

[0021] Furthermore, the robot sets each mutually parallel vector to be matched as a target matching vector, and marks the vector to be matched that is not parallel to the target matching vector as an incorrect matching vector within a preset search neighborhood of all projection points, and then sets a projection point corresponding to the incorrect matching vector and a point to be matched corresponding to the incorrect matching vector as a pair of incorrect matching points, and sets a projection point corresponding to the target matching vector and a point to be matched corresponding to the target matching vector as a pair of target matching points; wherein the directions of each mutually parallel vector to be matched are the same or opposite, and the target matching vector remains parallel to a preset mapping direction, and the preset mapping direction is associated with the inertial data.

[0022] Furthermore, step S24 also includes: when the number of mutually parallel vectors to be matched is less than the preset matching number, the robot expands the coverage of the preset search neighborhood of each projection point according to the preset expansion step to obtain the expanded preset search neighborhood, and updates the expanded preset search neighborhood to the preset search neighborhood described in step S23, and then executes step S23 until the robot repeats step S23 for a preset number of expansions, and then stops expanding the coverage of the preset search neighborhood of each projection point, and determines that the robot has failed to track using the projection matching method; wherein, the combination of step S22, step S23 and step S24 is the projection matching method.

[0023] Furthermore, the method in which the robot searches for a point to be matched in a preset search neighborhood of each projection point according to a standard distance between descriptors includes: the robot sets a circular neighborhood with each projection point as the center point, and sets the circular neighborhood as the preset search neighborhood of the projection point, wherein the inertial data includes the amount of change in the camera's posture between the previous frame image and the current frame image; the greater the amount of change in the camera's posture between the previous frame image and the current frame image, the larger the radius of the preset search neighborhood is set; the smaller the amount of change in the camera's posture between the previous frame image and the current frame image, the smaller the radius of the preset search neighborhood is set; the robot searches for feature points other than the projection point in the preset search neighborhood of each projection point, starting from the center point of the preset search neighborhood of the projection point; when the descriptor of the feature point in the current frame image searched is closest to the standard distance of the descriptor of the center point of the preset search neighborhood, the feature point in the current frame image searched is set as the point to be matched in the preset search neighborhood; wherein the standard distance is represented by Euclidean distance or Hamming distance.

[0024] In summary, the present invention combines the window matching method and the projection matching method for image tracking. Specifically, under different tracking results, an appropriate matching method is used for image tracking, so as to realize adaptive tracking of the current frame image collected by the robot in real time in stages. Under different matching methods, the predetermined image is tracked and collected in real time to the current frame image with different feature point search ranges and conversion methods, so as to complete efficient and reasonable matching between two adjacent frames of images, and solve the problems of low running frame rate and poor real-time navigation and positioning of the simple feature method visual odometer in the robot platform with limited computing power, greatly reduce the average tracking time of the current frame image, improve the running frame rate of the visual odometer composed of the camera and the inertial sensor, and well realize the real-time positioning of the robot.

[0025] When performing image tracking in the window matching mode, the robot selects all the first feature point pairs from the feature points of the current frame image and the feature points of all the reference frame images in the sliding window, completes the matching of each feature point of the current frame with all the feature points of the reference frame, obtains the initially filtered feature point pairs, and removes the interference of some feature point pairs that do not meet the corresponding epipolar constraint error value. On this basis, the matching of the feature points of the two frames of images in another scale information dimension is obtained in combination with the depth information, thereby improving the robustness and accuracy of feature point pair matching and image tracking, making the robot positioning more reliable. Then, the residual is introduced between the preset feature point pairs, and the residual and its derivative results of the inertial data are combined to calculate the inertial compensation value so that the original inertial data is subjected to the iterative matching test of the ratio of the epipolar constraint error value and the depth value of the visual feature point pair, and the partial derivative information that can be corrected is obtained, so that the inertial data is optimized and processed, and the positioning accuracy of the robot is improved. A large amount of calculation is saved, the calculation efficiency is improved, and the positioning and map construction speed is improved.

[0026] When image tracking is performed using the projection matching method, the robot's posture change between two adjacent frames of images and the projection transformation relationship of the feature points are combined to identify the vectors to be matched with the same direction in the current frame image to be tracked and count their number to determine whether the robot has successfully completed image tracking using the projection matching method, thereby reducing the feature mismatch rate and the difficulty of calculation. The window matching method can also be switched to after tracking fails to further narrow the search range of feature point matching and improve the accuracy and efficiency of the robot's visual positioning. In addition, the present invention only uses a single camera for positioning, and the equipment is simple and low in cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 The present invention discloses a flowchart of a robot visual tracking method according to an embodiment of the present invention.

[0028] Figure 2 It is a flow chart of a method for a robot to perform image tracking using a window matching method, disclosed in another embodiment of the present invention. DETAILED DESCRIPTION

[0029] The technical scheme in the embodiment of the present invention will be described in detail below in conjunction with the drawings in the embodiment of the present invention. To further illustrate each embodiment, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiment, and can be used in conjunction with the relevant description of the specification to explain the operating principle of the embodiment. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementation methods and the advantages of the present invention. Processing or method depicted as a flow chart. Although the flow chart describes each step as a sequential process, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of each step can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0030] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatuses.

[0031] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0032] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0033] As an embodiment, a robot visual tracking method is disclosed, wherein the execution subject of the robot visual tracking method is a robot fixedly equipped with a camera and an inertial sensor, wherein the robot is an autonomous mobile robot. Figure 1As shown, the robot visual tracking method includes: the robot uses the window matching method to track the image, and when the robot successfully tracks the image using the window matching method, the robot stops using the window matching method to track the image, and then the robot uses the projection matching method to track the image. In this embodiment, during the process of the robot using the window matching method to track the image, the robot uses the window matching method to match all reference frame images in the sliding window with the same current frame image (specifically involving the matching of feature points), so as to track the current frame image through the reference frame image frame; when the robot successfully tracks the current frame image using the window matching method, the robot has completed the matching of all reference frame images in the sliding window with the same current frame image, and obtained the reference frame image with the best matching degree from all reference frame images, then the robot can stop using the window matching method to track the image, and then use the projection matching method to match the previous frame image with the current frame image, and start tracking the current frame image through the previous frame image.

[0034] Then, when the robot fails to track using the projection matching method, the robot stops using the projection matching method for image tracking, and then the robot uses the window matching method for image tracking. When the robot fails to track an image using the projection matching method, it is determined that the match between the previous frame image and the current frame image fails, and then the projection matching method is not continued for image tracking, and then the window matching method is used again to track the newly acquired image. Generally, when the robot fails to track using the window matching method or the projection matching method, it is because the multiple frames of images acquired previously cannot be successfully matched with the current frame image, and then the current frame image cannot be tracked, which is understood as losing the current frame image.

[0035] In summary, this embodiment uses the window matching method and the projection matching method for image tracking in turn. Specifically, under different tracking results, an appropriate matching method is used for image tracking, so as to realize adaptive tracking of the current frame image collected by the robot in real time in stages. Under different matching methods, the predetermined image is tracked and collected in real time to the current frame image with different feature point search ranges and conversion methods, so as to complete efficient and reasonable matching between two adjacent frames of images, and solves the problems of low running frame rate and poor real-time navigation and positioning of the simple feature method visual odometer in a robot platform with limited computing power, greatly reduces the average tracking time of the current frame image, improves the running frame rate of the visual odometer composed of a camera and an inertial sensor, and well realizes the real-time positioning of the robot.

[0036] As an embodiment, the robot visual tracking method further includes: when the robot fails to track using the window matching method, the robot stops using the window matching method for image tracking, the robot clears the sliding window, including clearing all frame images in the sliding window, so as to fill in the newly collected image, and then uses the window matching method to track the image, and performs image tracking based on the updated reference frame image, so as to establish a matching relationship between the new reference frame and the current frame image; when the robot successfully tracks using the window matching method, the current frame image is filled into the sliding window, so as to track the image collected by the robot in real time, and then the next frame image collected by the camera will be updated as the current frame image, and the robot can match the current frame image newly filled in the sliding window with the next frame image to track the next frame image. Thus, the interference of the image frame with the wrong matching is eliminated, and the real-time and accuracy of the tracking is guaranteed by introducing the newly collected image frame.

[0037] In addition, when the robot uses the projection matching method to track images, if it detects that the time interval between the current frame image and the previous frame image exceeds the preset time threshold, the robot stops using the projection matching method to track the image and instead uses the window matching method to track the image; preferably, the preset time threshold is set to 1 second, and the detection method here is timing detection; when the time interval between two adjacent frames of images continuously acquired by the robot exceeds 1 second, the robot stops using the projection matching method to track the image and instead uses the window matching method to track the image, because an excessively large acquisition time interval will accumulate a larger posture error in the projection conversion between two adjacent frames of images.

[0038] It should be noted that image tracking is used to indicate the match between the feature points of the previously acquired image and the feature points of the current frame image; in the process of the robot using the window matching method to perform image tracking, the robot has filled at least one frame of the previously acquired image into the sliding window, so that the feature points of the image in the sliding window can be matched with the feature points of the current frame image. The number of frames of all images filled in the sliding window required by the window matching method is a fixed value, that is, the size of the sliding window; the images filled in the sliding window are marked as reference frame images as a group of candidate matching key frames; the reference frame image mentioned in this embodiment can be referred to as a reference frame, the current frame image can be referred to as a current frame, a frame image can be referred to as an image frame, and two adjacent frames of images can be referred to as adjacent two frames, or two adjacent image frames. The feature point is a pixel point belonging to the image, and the feature point is an environmental element that exists in the form of a point in the environment where the camera is located.

[0039] As an example, Figure 2 As shown, the method for the robot to perform image tracking using the window matching method includes the following steps:

[0040] Step S101, the robot collects the current frame image through the camera, and obtains inertial data through the inertial sensor; then the robot executes step S102. The camera is installed on the outside of the robot, and the lens of the camera points to the forward direction of the robot, which is used to collect image information in front of the robot. According to the frame count, the camera collects the current frame image, and the robot obtains feature points from the current frame image, wherein the feature point refers to the environmental element in the form of a point in the environment where the camera device is located, so as to facilitate matching with the previously collected image, so as to achieve tracking of the current frame image; the inertial sensor is generally installed inside the robot body, such as the code disk installed in the driving wheel of the robot, which is used to obtain the displacement information generated during the movement of the robot; the inertial measurement unit (such as a gyroscope) is used to obtain the angle information generated during the movement of the robot, wherein the code disk and the inertial measurement unit constitute an inertial system, which is used to obtain inertial data, so as to determine any The camera posture transformation relationship between the two frames of images is used, and the robot can use this posture transformation relationship to perform matching transformation between feature points, wherein the initial state quantity of the rotation matrix and the initial state quantity of the translation vector involved in the posture transformation relationship of the camera between the two frames of images are pre-set; based on these initial state quantities, the robot relies on the displacement change amount sensed by the encoder between the two frames of images successively collected by the camera, and the angle change amount sensed by the gyroscope between the two frames of images successively collected by the camera, and performs integral processing, which can be using Euler integration to integrate the displacement change amount and the angle change amount respectively, and obtain the robot's posture change amount between any two frames of images (including from multiple frames of images collected in advance), and then the latest rotation matrix and the latest translation vector can be obtained. Therefore, the inertial data disclosed in this embodiment can represent the conversion relationship between the coordinate system of the current frame image and the coordinate system of the reference frame image, including the translation conversion relationship, the rotation conversion relationship, the displacement difference, the angle increment, etc., wherein the reference frame image is the image in the sliding window with a fixed size disclosed in the above embodiment.

[0041] Step S102, based on the inertial data, the robot uses the epipolar constraint error value to select the first feature point pair from the feature points of the current frame image and the feature points of all the reference frame images in the sliding window, thereby filtering out the feature point pairs with too large epipolar constraint error values, and filtering out the unmatched feature points between the current frame image and each reference frame image; then step S103 is executed. In this embodiment, the robot performs epipolar constraints on the feature points of the current frame image and the feature points of all the reference frame images in the sliding window based on the conversion relationship between the coordinate system of the current frame image involved in the inertial data and the coordinate system of the reference frame image, and obtains the epipolar constraint error value of each feature point pair, wherein the epipolar constraint error value is the error value calculated according to the geometric imaging relationship between the feature points of the current frame image and the feature points of the reference frame image in the sliding window under the epipolar constraint, and the epipolar constraint is used to represent the geometric relationship between the pixel points corresponding to a three-dimensional point in space on different imaging planes, and also represents the projective relationship between each pixel in two frames of images successively captured by the camera following the movement of the robot (or the geometric relationship between each matching point). The feature points of the first feature point pair can be located respectively in the current frame image and each frame reference frame image, or respectively in the current frame image and some frame reference frame images; the sliding window is set to fill in at least one frame image captured in advance, so as to facilitate the subsequent matching of the current frame with each frame in the sliding window; the feature point is the pixel point of the image, and the feature point is an environmental element that exists in the form of a point in the environment where the camera is located, and describes the environmental features that the robot needs to track.

[0042] Step S103, based on the inertial data, the robot uses the depth value of the feature point to select the second feature point pair from the first feature point pair; then execute step S104. In step S103, the robot calculates the depth information of the feature points of the first feature point pair in the current frame image and the depth information of the feature points of the first feature point pair in the reference frame image within the range of the noise influence of the pixel points based on the first feature point pair filtered out in step S102, specifically, the depth value of the corresponding feature point is calculated using the displacement information of the robot (or camera) between the current frame image and the reference frame image in the inertial data and the normalized plane coordinates of each feature point of the first feature point pair; in some embodiments, the feature point of the first feature point pair in the current frame image is marked as P1, and the optical center when the camera collects the current frame image is marked as O1, the feature point of the first feature point pair in the reference frame image is marked as P2, and the optical center when the camera collects the reference frame image is marked as O2, then when the noise influence of the pixel points is not considered, the straight line O1P1 intersects the straight line O2P2 at point P3, then the length of the line segment O1P3 is the depth value of the feature point P1, and the length of the line segment O2P3 is the depth value of the feature point P2. Then, when it is determined that the ratio of the depth value of the feature point P1 to the depth value of the feature point P2 satisfies a certain ratio range, the feature point P1 and the feature point P2 are combined into a second feature point pair, otherwise the feature point P1 and the feature point P2 form an incorrect matching point pair, thereby screening out the second feature point pair from the first feature point pair.

[0043] Step S104, according to the similarity of the descriptors corresponding to the second feature point pairs, a third feature point pair is selected from the second feature point pairs; and then step S105 is executed. Step S104 specifically includes: for the current frame image and each reference frame image in the sliding window, the robot calculates the similarity between the descriptors of the feature points of each second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image; when the similarity between the descriptors of the feature points of a second feature point pair in the reference frame image calculated by the robot and the descriptors of the feature points of the second feature point pair in the current frame image is the minimum value among the similarities between the descriptors of the current frame image and the descriptors of the reference frame image where the feature points of the second feature point pair are located, the second feature point pair is marked as a third feature point pair and the third feature point pair is selected; thereby narrowing the search range of the feature points. Among them, the descriptor of the reference frame image where the feature points of the second feature point pair are located is the descriptor of all the feature points constituting the second feature point pair in the reference frame image where the feature points of the second feature point pair are located, and can be represented by a frame descriptor. The descriptor of the current frame image is the descriptor of the feature point in the current frame image that forms the feature point pair of the second feature point with the feature point of the reference frame image where the feature point of the second feature point pair is located; the similarity of the descriptors corresponding to the second feature point pair is represented by the Euclidean distance or Hamming distance between the descriptor of the feature point in the current frame image and the descriptor of the feature point in the corresponding reference frame image in the sliding window. Thus, the current frame image is tracked by using the similarity of the pixels between the two frames to track the movement of the robot.

[0044] Specifically, in step S104, the robot records the reference frame image where the feature points of each second feature point pair are located as the reference frame image to be matched. Generally, the number of reference frame images to be matched is equal to the number of reference frame images. The robot also marks the second feature point pairs between the current frame image and the reference frame image to be matched as second feature point pairs to be matched, wherein the feature points of the second feature point pair to be matched in the current frame image are recorded as second-first feature points, and the feature points of the second feature point pair to be matched in the reference frame image are recorded as second-second feature points, and the second-second feature points are located in the reference frame image to be matched. The robot needs to calculate the similarity between the descriptors of all second-second feature points in the reference frame image and the descriptors of their corresponding second-first feature points. Then, when the similarity between the descriptor of the feature point of a second feature point pair to be matched in the reference frame image calculated by the robot and the descriptor of the feature point of the second feature point pair to be matched in the current frame image is the minimum value among the similarities between the descriptors of all the second-second feature points in the reference frame image and the descriptors of their corresponding second-first feature points, the second feature point pair to be matched is marked as a third feature point pair and the third feature point pair is determined to be screened out, wherein multiple pairs of third feature point pairs can be screened out between each reference frame image and the current frame image; the similarity between the descriptor of the feature point of a second feature point pair to be matched in the reference frame image and the descriptor of the feature point of the second feature point pair to be matched in the current frame image is the similarity between the descriptor of the second-second feature point and the descriptor of the second-first feature point, which is specifically expressed as the square root of the sum of the squares of the Euclidean distance or Hamming distance between the second-second feature point and the second-first feature point in multiple dimensions as a similarity measure of the two descriptors, wherein each dimension can represent a binary encoding form of the feature point.

[0045] On the basis of the above embodiment, whenever the robot searches for all feature points constituting the second feature point pair between the current frame image and a frame of reference frame image in the sliding window, that is, after the robot calculates the similarity between the descriptors of all second-first feature points and the descriptors of the corresponding second-first feature points in a frame of reference frame image, if the number of third feature point pairs counted by the robot in the frame of reference frame image is greater than the first preset point number threshold, it is determined that the current frame image and the frame of reference frame image are matched successfully, and then the robot continues to search for all feature points constituting the second feature point pair between the current frame image and the next frame of reference frame image in the sliding window; if the number of third feature point pairs counted by the robot in the frame of reference frame image is less than or equal to the first preset point number threshold, it is determined that the current frame image and the frame of reference frame image fail to match, and the frame of reference frame image is set as a mismatched reference frame image, and then the robot continues to search for all feature points constituting the second feature point pair between the current frame image and the next frame of reference frame image in the sliding window; in some embodiments, the feature points of the frame of reference frame image are not used to match the feature points of the current frame image subsequently, and preferably, the first preset point number threshold is set to 20. In some embodiments, the feature points of the frame reference frame image are marked as incorrectly matched feature points, and no longer form the first feature point pair, the second feature point pair, or the third feature point pair with the feature points in the current frame image; if the number of third feature point pairs counted by the robot in the frame reference frame image is greater than the first preset point count threshold, it is determined that the current frame image and the frame reference frame image are matched successfully; wherein, when the robot determines that the current frame image and all frame reference frame images in the sliding window have failed to match, it is determined that the robot has failed to track using the window matching method, and then the robot clears the image in the sliding window.

[0046] Step S105, the robot introduces a residual between the third feature point pair, and then calculates an inertia compensation value by combining the residual and its derivative result with respect to the inertia data, and then uses the inertia compensation value to correct the inertia data; in some embodiments, the feature point of the third feature point pair in the current frame image e1 is marked as point P4, and the optical center when the camera collects the current frame image e1 is marked as point O1, and the feature point of the third feature point pair in the reference frame image e2 is marked as point P5, and the optical center when the camera collects the reference frame image e2 is marked as point O2, then Line O1P4 intersects with straight line O2P5 at point P6. Points O1, O2 and P6 form an epipolar plane. Straight line O1P4 becomes epipolar line L after being converted to reference frame image e2. Without considering the error, the intersection of the epipolar plane and the reference frame image e2 coincides with epipolar line L. The intersection passes through point P5, but in fact does not coincide due to the existence of the error. In this embodiment, point P5 is set as an observation point on epipolar line L, and then the distance from point P5 to epipolar line L is used to represent the error, and then the error is set as the residual.

[0047] Furthermore, in this embodiment, obtaining the residual requires constructing a derivation formula equivalent to obtaining the distance from the point to the line. When the corresponding rotation matrix and translation vector are first set as known state quantities, the residual value can be calculated by this derivation formula as the numerical result of the residual; then the corresponding rotation matrix and translation vector are set as unknown state quantities, and then the derivation formula (equivalent to an equation) is controlled to respectively obtain partial derivatives of the translation vector and the rotation matrix to obtain the Jacobian matrix, and the derivative of the residual to the posture is realized, so as to save the derivative result of the inertial data in the form of a matrix; then, combined with the properties of the derivative, it can be known that the robot sets the product of the inverse matrix of the Jacobian matrix and the residual value as the inertia compensation value, obtains the compensation amount for the displacement integral and the compensation amount for the angle integral, as the inertia compensation value, so as to realize the use of the least squares method to solve the derivation formula corresponding to the residual to obtain the inertia compensation value; then the optimal compensation value is used to correct the inertia data, and the specific correction method includes but is not limited to adding, subtracting, multiplying or dividing the original inertia data. Then the robot updates the corrected inertial data to the inertial data, and updates the feature points of the third feature point pair described in step S104 in the current frame image to the feature points of the current frame image, and updates the feature points of the third feature point pair described in step S104 in the reference frame image to the feature points of all reference frame images in the sliding window, thereby narrowing the search range of subsequent feature points, completing a feature point filtering, and completing an initialization of the feature points of the current frame image and each reference frame image in the sliding window; then executing step S106.

[0048] Step S106, judging whether the execution times of step S107 and step S108 have reached the preset iterative matching times, if yes, then executing step S110, otherwise executing step S107. In some embodiments, step S106 can also be understood as judging whether the execution times of step S107, step S108, and step S109 have reached the preset iterative matching times, if yes, then executing step S110, otherwise executing step S107, so that the correction times of inertial data reach the sum of the preset iterative matching times and the value 1. The preset iterative matching times are preferably 2 or 3. The robot repeatedly executes steps S107 and S108 to screen out the second feature point pairs, exclude more wrong matching point pairs, reduce the search range, and reduce the calculation amount of the inertia compensation value.

[0049] Step S107, based on the inertial data, the robot uses the epipolar constraint error value to filter out the first feature point pair from the feature points of the current frame image and the feature points of all the reference frame images in the sliding window, which is equivalent to repeatedly executing step S102, thereby filtering out the feature point pairs with excessively large epipolar constraint error values, thereby filtering out unmatched feature points between the current frame image and each reference frame image; then execute step S108.

[0050] Step S108, based on the inertial data, the robot uses the depth value of the feature point to select the second feature point pair from the first feature point pair (selected in the most recently executed step S107); equivalent to repeatedly executing step S103, using the displacement information and angle information of the robot (or camera) between the current frame image and the reference frame image in the inertial data, and the normalized plane coordinates of each feature point of the first feature point pair to calculate the triangular geometric relationship of the depth value matching of the corresponding feature points, calculate the depth value of each feature point of the first feature point pair, and then compare the depth value of the feature point of the first feature point pair in the current frame image with the depth value of the feature point of the first feature point pair in the reference frame image to select the second feature point pair. Then execute step S109.

[0051] In step S109, the robot introduces a residual between the second feature point pairs selected in the most recently executed step S108, and then calculates the inertia compensation value by combining the residual and its derivative result with respect to the most recently acquired inertial data (the most recently executed step S105 or the inertial data corrected in the last executed step S109), and then uses the inertia compensation value to correct the most recently acquired inertial data, and then updates the corrected inertial data to the inertial data described in step S107, and updates the feature points included in the second feature point pairs selected in the most recently executed step S108 to the feature points of the current frame image described in step S107 and the feature points of all reference frame images in the sliding window; this is equivalent to repeatedly executing step S105, but step S103 skips the screening of the third feature point pair in step S104, and directly executes step S105. Then execute step S106 to determine whether the execution times of step S107 and step S108 have reached the preset iteration matching times, thereby repeatedly correcting the inertial data in step S109, reducing the residual, optimizing the reference frame image subsequently filled in the sliding window, and improving the accuracy of robot positioning.

[0052] It can be understood that after executing step S102 and step S103, or each time after repeating step S107 and step S108 (starting to execute step S109), the robot introduces residuals between the latest filtered second feature point pairs, and then calculates the inertia compensation value based on the result of derivation of the latest inertial data obtained with the residuals, and then uses the inertia compensation value to correct the latest inertial data, and then updates the corrected inertial data to the inertial data, and updates the feature points included in the latest filtered second feature point pairs to the feature points of the current frame image and the feature points of all reference frame images in the sliding window, so as to narrow the search range of the feature points, further save a lot of matching calculations, and improve the robot positioning and map construction speed.

[0053] It should be noted that, for step S107, after the robot executes step S105, when executing step S107 for the first time, the robot calculates the epipolar constraint error value of each third feature point pair, wherein the epipolar constraint error value of each third feature point pair is determined by the inertial data corrected in step S105, and the specific epipolar constraint method for the third feature point pair is the same as that of step S102, and the calculation method of the epipolar constraint error value of the third feature point pair is the same as that of step S102, but the marking type and number of the feature point pairs for epipolar constraint are different. When the epipolar constraint error value of a third feature point pair calculated by the robot is less than the preset pixel distance threshold, the third feature point pair is updated to the first feature point pair, and it is determined that a new first feature point pair is selected from the third feature point pairs selected in step S104, except for the mismatched reference frame image.

[0054] It should be noted that, for the step S107, when the step S107 is repeated for the Nth time, the robot calculates the epipolar constraint error value of each second feature point pair screened out in the most recently executed step S108, wherein the epipolar constraint error value of each second feature point pair is determined by the inertial data corrected by the last executed step S109; when the epipolar constraint error value of a second feature point pair calculated by the robot is less than a preset pixel distance threshold, the second feature point pair is marked as the first feature point pair to update the first feature point pair, and a new first feature point pair is determined to be screened out from all the second feature point pairs screened out in the step S13; wherein N is set to be greater than 1 and less than or equal to the preset number of iterative matching times.

[0055] Step S110, based on the number of feature points of the second feature point pair in each reference frame image, select a matching frame image from the reference frame image in the sliding window; then execute step S111. Therefore, after completing the iterative matching of each feature point in the current frame image with all feature points of each reference frame image in the sliding window, the robot selects to count the number of feature points of the second feature point pair in each reference frame image, and then based on whether the number of the second feature points counted in each reference frame meets the threshold condition, selects the reference frame image matching the current frame image in the sliding window, thereby forming a matching frame image pair with the corresponding reference frame image, wherein the feature points of the second feature point pair in the current frame image are recorded as the second first feature points, and the feature points of the second feature point pair in the reference frame image are recorded as the second second feature points.

[0056] Specifically, the method for selecting matching frame images from reference frame images in a sliding window based on the number of feature points of the second feature point pair in each frame of the reference frame image includes: after the number of times the inertial data is repeatedly corrected reaches a preset number of iterative matching times, the robot counts the number of feature points of the second feature point pair in each frame of the reference frame image in the sliding window as the number of second feature point pairs matched in the corresponding reference frame image; if the feature points of the second feature point pair in the reference frame image are marked as second-second feature points, then the number of second feature point pairs matched in the reference frame image is equal to the number of second-second feature points in the reference frame image; if the robot matches the second feature points in one of the reference frame images If the number of pairs of second feature points matched by the robot in one of the reference frame images is less than or equal to the second preset point threshold, it is determined that one of the reference frame images fails to match the current frame image, and the one of the reference frame images can be set as a mismatched reference frame image; if the number of second feature point pairs matched by the robot in one of the reference frame images is greater than the second preset point threshold, it is determined that one of the reference frame images successfully matches the current frame image, and the one of the reference frame images is set as a matching frame image; further, if the number of second feature point pairs matched by the robot in each reference frame image is less than or equal to the second preset point threshold, it is determined that each reference frame image in the sliding window fails to match the current frame image, and it is determined that the robot fails to track using the window matching method. Preferably, the second preset point threshold is set to 15, which is less than the first preset point threshold. When the preset number of iterative matching times is set to increase, the excluded erroneous matching point pairs in the sliding window become more or remain unchanged, so the number of second feature point pairs matched in all reference frame images will become smaller or remain unchanged, and the number of second feature points in each frame reference frame image or the total number of second feature points in all frame reference frame images will become smaller or remain unchanged.

[0057] Step S111, based on the epipolar constraint error value and the number of feature points of the second feature point pair in each frame matching frame image, the best matching frame image is selected from all matching frame images, and it is determined that the robot successfully tracks using the window matching method, and then the robot removes the reference frame image that was first filled in the sliding window to free up memory space, and then fills the current frame image into the sliding window to update it to a new reference frame image. In step S111, among all the selected matching frame images, the smaller the sum of the epipolar constraint errors corresponding to the second feature points in a single matching frame image, the better the matching degree between the matching frame image and the current frame image, and the lower the matching error; among all the selected matching frame images, the more the number of second feature points in a single matching frame image, the better the matching degree between the matching frame image and the current frame image, and the more matching points.

[0058] Therefore, in the step S111, the method for selecting the optimal matching frame image from all matching frame images based on the epipolar constraint error value and the number of feature points of the second feature point pair in each frame of the matching frame image specifically includes: in each frame of the matching frame image, calculating the sum of the epipolar constraint error values ​​of the second feature point pairs to which the feature points in the frame of the matching frame image belong, as the epipolar constraint error value accumulation value of the frame of the matching frame image, so that each frame of the matching frame image is configured with an epipolar constraint error value accumulation value; wherein, the feature point of the second feature point pair in the matching frame image is the second-second feature point, and the robot accumulates the epipolar constraint error values ​​of the second feature point pairs where each second-second feature point newly marked in a frame of the matching frame image is located, to obtain the sum of the epipolar constraint error values ​​of the second feature point pairs to which the feature points in the frame of the matching frame image belong. In each matching frame image, the number of feature points constituting the second feature point pair in the matching frame image is counted as the feature point matching number of the matching frame image, so that each matching frame image is configured with a feature point matching number; when the feature point of the second feature point pair in the matching frame image is the second second feature point, the number of feature points constituting the second feature point pair in the matching frame image is the number of the second second feature points existing in the matching frame image. Then the robot sets the matching frame image with the maximum accumulated value of the extreme constraint error value (for all matching frame images) and the maximum number of feature point matches (for all matching frame images) as the optimal matching frame image. In summary, the combination of steps S102 to S111 is the window matching method, and the process of the robot executing steps S102 to S111 is the process of the robot using the window matching method to perform image tracking.

[0059] As an embodiment, for step S102 or step S107, based on inertial data, using the epipolar constraint error value, a method for selecting a first feature point pair from the feature points of the current frame image and the feature points of all reference frame images in the sliding window includes: the robot calculates the epipolar constraint error value of each feature point pair; when the epipolar constraint error value of a feature point pair calculated by the robot is greater than or equal to a preset pixel distance threshold, the feature point pair is marked as an incorrect matching point pair, and the corresponding pair of feature points cannot be used as a matching object in subsequent steps; wherein the robot sets the preset pixel distance threshold to the distance of a span of 3 pixels, for example, 3 adjacent pixels (one pixel is the center, and the pixels in the left and right neighborhoods of the center are point and the three pixel points adjacent to the center component position) can be equivalent to the 2-pixel spacing formed by the three pixel points in the same row or the same column; when the epipolar constraint error value of a feature point pair calculated by the robot is less than the preset pixel distance threshold, the feature point pair is marked as the first feature point pair and the first feature point pair is determined to be screened out, and the robot screens out the first feature point pair from the feature points of the current frame image and the feature points of all the reference frame images in the sliding window; it should be noted that the smaller the epipolar constraint error value of a feature point pair, that is, the smaller the epipolar constraint error value generated by a feature point of the current frame image and a feature point of the reference frame image in the sliding window under the epipolar constraint, the smaller the matching error between the pair of feature points.

[0060] In this embodiment, each feature point pair is configured to be composed of a feature point (any feature point) of the current frame image and a feature point (any feature point) of the reference frame image, and cannot be a pair of feature points in the same frame reference frame image, feature points between different frames of reference frame images, or a pair of feature points in the current frame image; each feature point of the current frame image forms a feature point pair with each feature point in each reference frame image in the sliding window, thereby achieving brute force matching between the current frame image and all reference frame images in the sliding window. The robot controls the normalized plane coordinates of each feature point in the current frame image and the normalized plane coordinates of each feature point in each reference frame image in the sliding window to calculate the epipolar constraint error value of the corresponding feature point pair in turn, and then whenever the epipolar constraint error value of a feature point pair calculated is greater than or equal to a preset pixel distance threshold, the feature point pair is filtered, otherwise the feature point pair is marked as the first feature point pair; after traversing all the feature point pairs, the robot selects all the first feature point pairs from the feature points of the current frame image and the feature points of all the reference frame images in the sliding window, completes the matching of each feature point of the current frame with all the feature points of the reference frame, obtains the feature point pairs that are initially filtered out, and removes the interference of some feature point pairs that do not meet the error value.

[0061] It should be noted that the rigid body motion of the camera is consistent with the motion of the robot, and the two frames of images collected successively will be expressed in two coordinate systems, including the current frame image relative to the reference frame image, and the reference frame image relative to the current frame image; there is a certain geometric relationship between the points in the two frames of images collected successively by the camera, and this relationship can be described by epipolar geometry. Epipolar geometry describes the projective relationship between each pixel in the two frames of images (or the geometric relationship between each matching point). In some embodiments, it has nothing to do with the external scene itself, but is only related to the intrinsic parameters of the camera and the shooting positions of the two images. Ideally, the epipolar constraint error value is equal to the value 0, but due to the presence of noise, the epipolar constraint error value must not be the value 0. This non-zero number can be used to measure the size of the matching error between the feature points of the reference frame image and the feature points of the current frame image.

[0062] In some embodiments, R is the rotation matrix from the C1 coordinate system to the C0 coordinate system, which can represent the rotation from the kth frame image to the k+1th frame image; the vector C0-C1 is the translation of the optical center C1 relative to the optical center C0, which is denoted as T. R and T can represent the movement of the robot between two frames of images, which are given by the inertial sensor and can be included in the inertial data. There will be two forms of expression in the coordinate system, including the current frame relative to the reference frame and the reference frame relative to the current frame. C0 and C1 are respectively the optical centers of the camera in the two moving positions of the robot, that is, the pinhole in the pinhole camera model; Q is a three-dimensional point in space, and Q0 and Q1 are respectively the pixel points corresponding to the Q point on different imaging planes; Q0 and Q1 are both two-dimensional points on the image, and this embodiment sets them all as three-dimensional direction vectors for processing; assuming a normalized image plane, the focal length f=1 on the plane, so it can be defined in a coordinate system with the optical center C0 as the origin, Q0 is in the reference coordinate system with the optical center C0 as the origin, and Q1 is in the reference coordinate system with the optical center C1 as the origin, so the coordinate system needs to be converted. Here, the coordinates of all points are converted to a coordinate system with C0 as the origin. Since the direction vector is independent of the starting position of the vector, the coordinate system transformation of Q0 and Q1 only considers rotation. The normalized image plane here is the polar plane composed of C0-C1-Q0-Q1.

[0063] As an embodiment, in step S102 or step S107, when the inertial data includes a translation vector of the current frame image relative to the reference frame image, and a rotation matrix of the current frame image relative to the reference frame image, the robot records the translation vector of the current frame image relative to the reference frame image as a first translation vector, and records the rotation matrix of the current frame image relative to the reference frame image as a first rotation matrix, wherein the first translation vector represents the translation vector from the coordinate system of the current frame image to the coordinate system of the reference frame image, and the first rotation matrix represents the rotation matrix from the coordinate system of the current frame image to the coordinate system of the reference frame image, so that the inertial data selects to represent the displacement information and the angle information in the coordinate system of the reference frame image; on this basis, the robot uses the first rotation matrix to convert a feature point (which can be extended to any feature point) of the current frame image into a rotation matrix. The normalized plane coordinates are converted to the coordinate system of the reference frame image to obtain the first coordinates, wherein the normalized plane coordinates of the feature point are represented by a direction vector in the coordinate system of the current frame image, only the direction of the direction vector is considered, and the vector of the starting point or the end point of the direction vector is not considered, and a column vector is formed, and an inverse vector exists; then the coordinate system conversion of all feature points of the current frame image only needs to be rotated, and the first coordinates can be represented by the direction vector in the coordinate system of the reference frame image; therefore, this embodiment sets the normalized vectors of the feature points of the current frame image to the vectors formed by the normalized plane coordinates of the feature points of the current frame image relative to the origin of the coordinate system of the current frame image; and sets the normalized vectors of the feature points of the reference frame image to the vectors formed by the normalized plane coordinates of the feature points of the reference frame image relative to the origin of the coordinate system of the reference frame image. The robot controls the first rotation matrix to convert the normalized vector of the feature point of the current frame image into the coordinate system of the reference frame image to obtain the first vector; then controls the first translation vector to cross-multiply the first vector to obtain the first vector, wherein the first vector is perpendicular to the first translation vector and the first vector at the same time; then controls the normalized vector of the feature point in the reference frame image in the sliding window to be dot-multiplied with the first vector, and then sets the result of the dot multiplication (representing the cosine value of the angle between the normalized vector of the feature point in the reference frame image and the first vector) as the epipolar constraint error value of the corresponding feature point pair; specifically, the robot controls the normalized vector of each feature point in each frame of the reference frame image in the sliding window to be dot-multiplied with the first vector in turn, and then sets the result of each dot multiplication as the epipolar constraint error value of the corresponding feature point pair.

[0064] It should be noted that the normalized vector of the feature point of the current frame image is the vector formed by the normalized plane coordinates of the feature point of the current frame image (the end point of the vector) relative to the origin of the coordinate system of the current frame image (the starting point of the vector); the normalized vector of the feature point of the reference frame image is the vector formed by the normalized plane coordinates of the feature point of the reference frame image (the end point of the vector) relative to the origin of the coordinate system of the reference frame image (the starting point of the vector).

[0065] As an embodiment, in step S102 or step S107, when the inertial data includes a translation vector of the reference frame image relative to the current frame image, and a rotation matrix of the reference frame image relative to the current frame image, the robot marks the translation vector of the reference frame image relative to the current frame image as a second translation vector, and marks the rotation matrix of the reference frame image relative to the current frame image as a second rotation matrix, wherein the second translation vector represents the translation vector from the coordinate system of the reference frame image to the coordinate system of the current frame image, and the second rotation matrix represents the rotation matrix from the coordinate system of the reference frame image to the coordinate system of the current frame image, so that the inertial data selects to represent the displacement information and the angle information in the coordinate system of the current frame image; then, the robot controls the second rotation matrix The normalized vector of the feature point of the reference frame image in the sliding window is converted to the coordinate system of the current frame image to obtain the second first vector; then the second translation vector is controlled to cross-multiply the second first vector to obtain the second second vector, wherein the second second vector is perpendicular to both the second translation vector and the second first vector; then the normalized vector of the feature point in the current frame image is controlled to be dot-multiplied with the second second vector, and the result of the dot multiplication (representing the cosine value of the angle between the normalized vector of the feature point in the current frame image and the second second vector) is set as the epipolar constraint error value of the corresponding feature point pair; specifically, the robot will control the normalized vector of each feature point in the current frame image to be dot-multiplied with the first second vector in turn, and then the result of each dot multiplication is set as the epipolar constraint error value of the corresponding feature point pair. This allows the epipolar constraint error value to describe the feature point matching error information between image frames captured by the camera at different viewing angles from a geometric dimension.

[0066] As an embodiment, in step S103 or step S108, based on the inertial data, a method for selecting a second feature point pair from a first feature point pair using the depth value of the feature point includes: the robot calculates the ratio of the depth value of the feature point of the selected first feature point pair (which may be the one selected in step S102 or step S107) in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image; if each first feature point pair is composed of the first feature point in the current frame image and the first feature point in the reference frame image, then the ratio of the depth value of the first feature point in the current frame image to the depth value of the corresponding first feature point in the reference frame image is recorded and used for threshold comparison to filter out the first feature point pairs where some first feature points with mismatched ratios are located. When the ratio of the depth value of the feature point of the first feature point pair calculated by the robot in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image is within the preset ratio threshold range, the first feature point pair is marked as the second feature point pair and the second feature point pair is determined to be screened out; preferably, the preset ratio threshold range is set to be greater than 0.5 and less than 1.5. When the ratio of the depth value of the feature point of the first feature point pair calculated by the robot in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image is not within the preset ratio threshold range, the first feature point pair is marked as an incorrect matching point pair, thereby excluding the incorrect matching point pair from the first feature point pairs screened out in step S102 and step S107, and filtering the first feature point pair, which can narrow the search range of the feature point pair in subsequent feature point matching.

[0067] As an embodiment, in step S103 or step S108, the method for the robot to calculate the depth value of the feature point includes: when the inertial data includes a translation vector of the current frame image relative to the reference frame image, and a rotation matrix of the current frame image relative to the reference frame image, the robot records the translation vector of the current frame image relative to the reference frame image as a first translation vector, and records the rotation matrix of the current frame image relative to the reference frame image as a first rotation matrix, wherein the first translation vector represents the translation vector from the coordinate system of the current frame image to the coordinate system of the reference frame image, and the first rotation matrix represents the rotation matrix from the coordinate system of the current frame image to the coordinate system of the reference frame image; then, the robot controls the first rotation matrix to convert the normalized vector of the feature point of the first feature point pair in the current frame image to the coordinate system of the reference frame image to obtain a first one-vector; then, the robot controls the normalized vector of the feature point of the first feature point pair in the reference frame image to cross-multiply the first one-vector to obtain a first two-vector; at the same time, the normalized vector of the feature point of the first feature point pair in the reference frame image is controlled to cross-multiply the first translation vector, and then the cross-multiplication result is inverted to obtain a first three-dimensional vector. , wherein the cross product result of the normalized vector of the feature point of the first feature point pair in the reference frame image and the first translation vector is a vector that is perpendicular to both the normalized vector and the first translation vector, and the opposite vector of the vector is the first three-vector; then the product of the first three-vector and the inverse vector of the first two-vector is set to the depth value of the feature point of the first feature point pair in the current frame image, and marked as the first depth value, indicating the distance between the three-dimensional point detected by the camera and the optical center (the origin of the coordinate system of the current frame image) when the camera captures the current frame image; then the first The product of a vector and the first depth value and the sum of the first translation vector are marked as the first four-vector, and then the product of the first four-vector and the inverse vector of the normalized vector of the feature point of the first feature point pair in the reference frame image is set as the depth value of the feature point of the first feature point pair in the reference frame image, which is equivalent to setting the product of the first four-vector and the inverse vector of the normalized vector as the depth value of the feature point in the reference frame image and marking it as the second depth value, indicating the distance between the same three-dimensional point and the optical center (the origin of the coordinate system of the reference frame image) when the camera captures the reference frame image. Therefore, based on the posture transformation information of the camera between the two frames of images, the depth information of a pair of feature points is triangulated and calculated.

[0068] As an embodiment, in step S103 or step S108, the method for the robot to calculate the depth value of the feature point includes: when the inertial data includes a translation vector of the reference frame image relative to the current frame image, and a rotation matrix of the reference frame image relative to the current frame image, the robot records the translation vector of the reference frame image relative to the current frame image as a second translation vector, and records the rotation matrix of the reference frame image relative to the current frame image as a second rotation matrix, wherein the second translation vector represents the translation vector from the coordinate system of the reference frame image to the coordinate system of the current frame image, and the second rotation matrix represents the rotation matrix from the coordinate system of the reference frame image to the coordinate system of the current frame image; then, the robot controls the second rotation matrix to convert the normalized vector of the feature point of the first feature point pair in the reference frame image to the coordinate system of the current frame image to obtain a second vector; and then controls the normalized vector of the feature point of the first feature point pair in the current frame image to obtain a second vector. The normalized vector is cross-multiplied with the second vector to obtain a second vector; at the same time, the normalized vector of the feature point of the first feature point pair in the current frame image is cross-multiplied with the second translation vector, and the cross-multiplication result is inverted to obtain a second vector; then the product of the second vector and the inverse vector of the second vector is set as the depth value of the feature point of the first feature point pair in the reference frame image, and is marked as the second depth value, indicating the distance between the three-dimensional point detected by the camera and the optical center when the camera collects the reference frame image; then the product of the second vector and the second depth value and the sum of the second translation vector are marked as a second vector, and then the product of the second vector and the inverse vector of the normalized vector of the feature point of the first feature point pair in the current frame image is set as the depth value of the feature point of the first feature point pair in the current frame image, and is marked as the first depth value, indicating the distance between the same three-dimensional point and the optical center when the camera collects the current frame image.

[0069] Integrating step S103 or step S108, the aforementioned embodiment of calculating the depth value is based on the geometric relationship formed by the projection point of the point in each frame image and the corresponding optical center when the same point is projected onto two frames of images with different perspectives, and combining the depth information to obtain the matching of the feature points of the two frames of images in another scale information dimension, thereby improving the robustness and accuracy of feature point pair matching and image tracking, making the robot positioning more reliable.

[0070] It should be noted that the normalized vector of the feature points of the first feature point pair in the current frame image is the vector formed by the normalized plane coordinates of the feature points of the first feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the first feature point pair in the reference frame image is the vector formed by the normalized plane coordinates of the feature points of the first feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image. In some embodiments, if the number of first feature point pairs screened out in step S102 or step S107 is large, it is necessary to obtain a batch of first feature point pairs with a high degree of matching by the least squares method, and then solve the depth value of the feature point; since step S103 or step S108 is a preliminary screening, the accuracy requirement is not high, so the use of the least squares method is not necessary.

[0071] In some embodiments, the feature points of the first feature point pair in the current frame image are marked as P1, and the optical center when the camera captures the current frame image is marked as O1. The feature points of the first feature point pair in the reference frame image are marked as P2, and the optical center when the camera captures the reference frame image is marked as O2. O1-O2-P1-P2 constitute an epipolar plane, and the intersection line of the epipolar plane and the current frame image becomes the epipolar line of the imaging plane of the current frame image and passes through P1. The intersection line of the epipolar plane and the reference frame image becomes the epipolar line of the imaging plane of the reference frame image and passes through P2. When the noise influence of the pixel points is not considered, the straight line O1P1 and the straight line O2P2 intersect at point P3, the length of the line segment O1P3 is the depth value of the feature point P1, and the length of the line segment O2P3 is the depth value of the feature point P2; when the noise influence range of the pixel points is considered, the intersection of the straight line O1P1 and the straight line O2P2 is point P0, and point P0 is not point P3, then the position deviation between point P0 and point P3 can be used to measure the matching error, therefore, it is necessary to set the preset ratio threshold range to compare the ratio of the depth values ​​between the feature point pairs.

[0072] As an embodiment, in step S104, the method of screening out a third feature point pair from the second feature point pairs based on the similarity of the descriptors corresponding to the second feature point pairs specifically includes: for the current frame image and the corresponding reference frame image in the sliding window, it can be considered that for the second feature point pairs marked between the current frame image and each reference frame image in the sliding window, the robot calculates the similarity between the descriptors of the feature points of the second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image, which can be understood as calculating the similarity between the descriptors of the feature points of each second feature point pair in the reference frame image and the descriptors of the feature points of the second feature point pair in the current frame image, which is also equivalent to the similarity between the frame descriptors of each reference frame image in the sliding window and the frame descriptors of the current frame image. Then, when the similarity between the descriptor of the feature point of a second feature point pair in the reference frame image calculated by the robot and the descriptor of the feature point of the second feature point pair in the current frame image is the minimum value among the similarities between the descriptor of the current frame image and the descriptor of the reference frame image where the feature point of the second feature point pair is located, the second feature point pair is marked as a third feature point pair and the third feature point pair is determined to be screened out; wherein the descriptor of the reference frame image where the feature point of the second feature point pair is located is the descriptor of all the feature points constituting the second feature point pair in the reference frame image where the feature point of the second feature point pair is located, that is, there are multiple descriptors about the third feature point pair in the same frame of the reference frame image. descriptors of two feature point pairs; the descriptor of the current frame image is the descriptor of the feature points in the current frame image that constitute the second feature point pair together with the feature points of the reference frame image where the feature points of the second feature point pair are located, that is, there are multiple descriptors about the second feature point pair in the same current frame image; preferably, the similarity of the descriptors corresponding to the second feature point pair is represented by the Euclidean distance or Hamming distance between the descriptor of the feature point in the current frame image and the descriptor of the feature point in the corresponding reference frame image in the sliding window, so that the similarity between the descriptors of the matching points can be calculated using the Euclidean distance or the sum of the squares of multiple dimensions, and then the point with the smallest distance is taken as a more accurate point to be matched.

[0073] Specifically, in step S104, the robot records the feature points of the second feature point pair in the current frame image as second-first feature points, and records the feature points of the second feature point pair in the reference frame image as second-second feature points; the robot needs to calculate the similarity between the descriptors of all second-second feature points in the reference frame image and the descriptors of their corresponding second-first feature points. Then, when there is a second feature point pair to be matched whose descriptor of the feature point in the reference frame image and the descriptor of the feature point in the current frame image is the minimum value among the similarities between the descriptors of all the second feature points in the reference frame image and the descriptors of their corresponding second feature points, the second feature point pair is marked as a third feature point pair and the third feature point pair is determined to be screened out, wherein multiple pairs of third feature point pairs can be screened out between each reference frame image and the current frame image; the similarity between the descriptor of the feature point of a second feature point pair in the reference frame image and the descriptor of the feature point of the second feature point pair in the current frame image is the similarity between the descriptor of the second feature point and the descriptor of the second feature point. As a similarity measure of the two descriptors, the specific calculation method can be expressed as: the square root of the sum of the squares of the Euclidean distance or Hamming distance between the second feature point and the second feature point in multiple dimensions, wherein each dimension can represent a binary encoding form of the feature point.

[0074] As an embodiment, for step S105 or step S109, the robot marks the line connecting the optical center when the camera collects the current frame image and the feature points of the preset feature point pair in the current frame image as the first observation line, and marks the line connecting the optical center when the camera collects the reference frame image and the feature points of the same preset feature point pair in the reference frame image as the second observation line, and then, under environmental conditions where errors are not considered, marks the intersection of the first observation line and the second observation line as the target detection point; wherein, the optical center when the camera collects the current frame image, the optical center when the camera collects the reference frame image, and the target detection point are all in the same plane, that is, forming a three-point coplanar state, and then the same plane is set as the polar plane; or, the preset feature point pair, the optical center when the camera collects the current frame image, and the optical center when the camera collects the reference frame image are all in the same plane, that is, forming a four-point coplanar state. The robot records the intersection line of the polar plane and the current frame image as the polar line in the imaging plane of the current frame image (which can be regarded as the coordinate system of the current frame image in some embodiments), and records the intersection line of the polar plane and the reference frame image as the polar line in the imaging plane of the reference frame image (which can be regarded as the coordinate system of the reference frame image in some embodiments). Specifically, in the same preset feature point pair, after the feature point of the current frame image is converted to the reference frame image, it becomes the first projection point, whose coordinates are the first coordinates; the distance from the first projection point to the epipolar line in the imaging plane of the reference frame image (in the coordinate system of the reference frame image) is represented as the first residual value. It should be noted that, without considering pixel noise, the first projection point is located on the epipolar line in the imaging plane of the reference frame image (in the coordinate system of the current frame image), that is, the line segment from the current frame image actually observed from the perspective of the reference frame image can coincide with the epipolar line in the imaging plane of the reference frame image after the coordinate system conversion; in the same preset feature point pair, after the feature point of the reference frame image is converted to the current frame image, it becomes the second projection point, whose coordinates are the second coordinates; the distance from the second projection point to the epipolar line in the imaging plane of the current frame image is represented as the second residual value. When the first residual value or the second residual value is smaller, the corresponding projection point relative to the epipolar line deviation in the imaging plane to which it is converted is smaller, and the corresponding preset feature point pair has a higher matching degree.

[0075] It should be noted that in step S105, the preset feature point pair is the third feature point pair, which is the feature point pair selected from step S104; in step S109, each time after step S107 and step S108 are repeatedly executed, the preset feature point pair is the second feature point pair selected by the most recently executed step S108. The first feature point pair, the second feature point pair and the third feature point pair are all a pair of feature points consisting of a feature point located in the current frame image and a feature point located in the reference frame image. The normalized vector of the feature point of the preset feature point pair in the current frame image is the vector formed by the normalized plane coordinates of the feature point of the preset feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature point of the preset feature point pair in the reference frame image is the vector formed by the normalized plane coordinates of the feature point of the preset feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image. The normalized plane coordinates may belong to the coordinates in the polar plane, so that the coordinates of the feature points of the preset feature point pair in the current frame image and the coordinates of the feature points of the preset feature point pair in the reference frame image are normalized to the polar plane. Of course, corresponding coordinate normalization processing can also be performed for other types of feature point pairs.

[0076] As an embodiment, in step S105 or step S109, the method of introducing residuals into the pre-screened feature point pairs specifically includes: when the inertial data includes the translation vector of the current frame image relative to the reference frame image, and the rotation matrix of the current frame image relative to the reference frame image, the robot records the translation vector of the current frame image relative to the reference frame image as the first translation vector, and records the rotation matrix of the current frame image relative to the reference frame image as the first rotation matrix; in step S105, the preset feature point pair is the third feature point pair, which comes from the feature point pair screened out in step S104; in step S109, each time step S107 and step S108 are repeatedly executed, the preset feature point pair is the second feature point pair screened out by the most recently executed step S108. The robot controls the first rotation matrix to convert the normalized vector of the feature point of the preset feature point pair in the current frame image to the coordinate system of the reference frame image to obtain the first vector; in this embodiment, the normalized vector of the feature point of the preset feature point pair in the current frame image is represented by a direction vector, only the direction of the direction vector is considered, and the vector of the starting point or the end point of the direction vector is not considered, and a column vector is formed, and there is an inverse vector; then the coordinate system conversion of all feature points of the current frame image only needs to be rotated, and the first vector can be represented by the direction vector in the coordinate system of the reference frame image. The robot then controls the first translation vector to cross-multiply the first vector to obtain the first vector, and form the polar line in the imaging plane of the reference frame image. The first vector is used as a three-dimensional direction vector, and the direction of the first vector is parallel to the polar line. The polar line is the intersection of the polar plane formed by a preset feature point pair, the optical center corresponding to the current frame image, and the optical center corresponding to the reference frame image and the imaging plane of the reference frame image. Then, the square root of the sum of the squares of the horizontal axis coordinate in the first two vectors and the vertical axis coordinate in the first two vectors is calculated to obtain the modulus of the polar line, which can be regarded as the length of the polar line in some embodiments; at the same time, the robot controls the normalized vector of the feature point of the preset feature point pair in the reference frame image to multiply by the first two vectors, and then sets the result of the multiplication as the epipolar constraint error value of the preset feature point pair; then, the ratio of the epipolar constraint error value of the preset feature point pair to the modulus of the polar line is set as the first residual value, and the result value of the residual introduced between the preset feature point pair is determined.

[0077] On the basis of the above embodiment, the method of introducing residuals between preset feature point pairs, combining the residuals and the derivative results of the residuals with respect to the inertial data to calculate the inertial compensation value, and then using the inertial compensation value to correct the inertial data specifically includes: when the inertial data includes the translation vector of the current frame image relative to the reference frame image, and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the formula of multiplying the first rotation matrix and the normalized plane coordinates of the feature points of the preset feature point pair in the current frame image as a first-first transformation formula; then marks the formula of cross-multiplying the first translation vector and the first-first transformation formula as a first-second transformation formula; then marks the formula of multiplying the normalized plane coordinates of the feature points of the preset feature point pair in the reference frame image with the first-second transformation formula as a first-third transformation formula; then sets the calculation result of the first-second transformation formula to a value of 0 to form a straight line. A line equation is obtained, and then the square sum of the coefficients of the line equation in the horizontal axis coordinate dimension and the coefficients of the vertical axis coordinate dimension is calculated, and then the square root of the obtained square sum is calculated to obtain a first square root, which is equivalent to the projection length of the line represented by the line equation in the imaging plane under the reference frame image in some embodiments; then the formula of multiplying the reciprocal of the first square root and the first three-transformation formula is set as the first four-transformation formula; then the calculation result of the first four-transformation formula is set as the first residual value to form a first residual derivation formula, and it is determined to introduce a residual between the preset feature point pairs; then the first residual derivation formula is controlled to respectively calculate the partial derivatives of the first translation vector and the first rotation matrix to obtain a Jacobian matrix, where the Jacobian matrix is ​​a combination of the partial derivative results of the residual with respect to the first translation vector and the partial derivative results of the error value with respect to the first rotation matrix, so as to achieve the effect of correcting the influence of the change of inertial data. Then, the product of the inverse matrix of the Jacobian matrix and the first residual value is set as the inertia compensation value, so as to realize the construction of the least squares problem to find the optimal inertia compensation value. In this embodiment, the first residual derivation formula is equivalent to: a fitting function model of the point-to-line deviation error value set in order to fit the compensation value of the inertia data, wherein the residual belongs to the error information, such as the minimum sum of squares of errors under the least squares method, so that the first residual derivation formula or the straight line equation becomes an expression for solving the minimum sum of squares of errors; then, the partial derivative of the first residual derivation formula is obtained for the two parameters of the first translation vector and the first rotation matrix, respectively, and the obtained formula can be summarized as: the result of multiplying the Jacobian matrix and the compensation value (inertia compensation value) of the fitted inertia data is set to be equal to the first residual value, and the robot sets the product of the inverse matrix of the Jacobian matrix and the first residual value as the inertia compensation value, thereby completing the construction of the least squares problem to find the optimal inertia compensation value. The robot then uses the inertia compensation value to correct the inertia data. The specific correction includes addition, subtraction, multiplication and division operations on the original inertia data. It can be a simple coefficient multiplication and division or a matrix vector multiplication.

[0078] As an embodiment, in step S105 or step S109, the method of introducing residuals into the pre-screened feature point pairs specifically includes: when the inertial data includes a translation vector of the reference frame image relative to the current frame image, and a rotation matrix of the reference frame image relative to the current frame image, the robot records the translation vector of the reference frame image relative to the current frame image as the second translation vector, and records the rotation matrix of the reference frame image relative to the current frame image as the second rotation matrix; in step S105, the preset feature point pair is the third feature point pair, which comes from the feature point pair screened out in step S104; in step S109, each time step S107 and step S108 are repeatedly executed, the preset feature point pair is the second feature point pair screened out by the most recently executed step S108. The robot controls the second rotation matrix to convert the normalized vector of the feature points of the preset feature point pair in the reference frame image to the coordinate system of the current frame image to obtain the second vector. In this embodiment, the normalized vector of the feature points of the preset feature point pair in the reference frame image is represented by a direction vector, only the direction of the direction vector is considered, and the vector of the starting point or the end point of the direction vector is not considered, and a column vector is formed, and there is an inverse vector; then the coordinate system conversion of all feature points of the current frame image only needs to be rotated, and the first vector can be represented by the direction vector in the coordinate system of the current frame image; and the preset feature point pair is for the step S105, and can also be updated to the second feature point pair in the step S109. Then the robot controls the second translation vector to cross-multiply the second vector to obtain the second vector, and form the polar line in the imaging plane of the current frame image. The second vector is used as a three-dimensional direction vector, and the direction of the second vector is parallel to the polar line. The polar line is the intersection of the polar plane formed by a preset feature point pair, the optical center corresponding to the current frame image, and the optical center corresponding to the reference frame image and the imaging plane of the current frame image. Then, the square root of the sum of the squares of the horizontal axis coordinate in the second two vector and the vertical axis coordinate in the second two vector is calculated to obtain the modulus of the polar line, which in some embodiments can be regarded as the projection length of the straight line pointed to by the second two vector in the imaging plane of the current frame image; at the same time, the robot controls the normalized vector of the feature point of the preset feature point pair in the current frame image to multiply by the second two vector, and then sets the result of the dot product to the epipolar constraint error value of the preset feature point pair; then, the ratio of the epipolar constraint error value of the preset feature point pair to the modulus of the polar line is set as the second residual value, and the result value of the residual introduced between the preset feature point pair is determined.

[0079] On the basis of the above embodiment, the method of introducing residuals between preset feature point pairs, combining the residuals and the derivative results of the residuals with respect to the inertial data to calculate the inertial compensation value, and then using the inertial compensation value to correct the inertial data specifically includes: when the inertial data includes the translation vector of the reference frame image relative to the current frame image, and the rotation matrix of the reference frame image relative to the current frame image, the robot marks the formula of multiplying the second rotation matrix with the normalized plane coordinates of the feature points of the preset feature point pair in the reference frame image as the second-first transformation formula; then marks the formula of cross-multiplying the second translation vector with the second-first transformation formula as the second-second transformation formula; then marks the formula of multiplying the normalized plane coordinates of the feature points of the preset feature point pair in the current frame image with the second-second transformation formula as the second-third transformation formula; then sets the calculation result of the second-second transformation formula to the value 0 to form a straight line. A line equation is obtained, and then the square sum of the coefficients of the line equation in the horizontal axis coordinate dimension and the coefficients of the vertical axis coordinate dimension is calculated, and then the square root of the obtained square sum is calculated to obtain a second square root, which is equivalent to the projection length of the line represented by the line equation in the imaging plane under the current frame image in some embodiments; then the formula of multiplying the reciprocal of the second square root and the second three-transformation formula is set to a second four-transformation formula; then the calculation result of the second four-transformation formula is set to a second residual value to form a second residual derivation formula, and it is determined to introduce a residual between the preset feature point pairs; then the second residual derivation formula is controlled to respectively calculate the partial derivatives of the second translation vector and the second rotation matrix to obtain a Jacobian matrix, where the Jacobian matrix is ​​a combination of the partial derivative results of the residual with respect to the second translation vector and the partial derivative results of the error value with respect to the second rotation matrix, so as to achieve the effect of correcting the influence of the change of inertial data. Then, the product of the inverse matrix of the Jacobian matrix and the second residual value is set as the inertia compensation value, thereby realizing the construction of a least squares problem to find the optimal inertia compensation value. In this embodiment, the second residual derivation formula is equivalent to: a fitting function model set in order to fit the compensation value of the inertia data, wherein the residual belongs to error information, such as the minimum sum of squares of errors under the least squares method, so that the straight line equation or the second residual derivation formula becomes an expression for solving the minimum sum of squares of errors; then, the first residual derivation formula is partially derived for two parameters, the first translation vector and the first rotation matrix, respectively, and the obtained formula can be summarized as: the result of multiplying the Jacobian matrix and the compensation value (inertia compensation value) of the fitted inertia data is set to be equal to the first residual value, and the robot sets the product of the inverse matrix of the Jacobian matrix and the first residual value as the inertia compensation value, thereby completing the construction of a least squares problem to find the optimal inertia compensation value. Then the robot uses the inertial compensation value to correct the inertial data. The specific correction includes adding, subtracting, multiplying and dividing the original inertial data, which can be simple coefficient multiplication and division, or matrix vector multiplication. After the original inertial data is tested by iterative matching of visual feature point pairs, partial derivative information that can be corrected is obtained, so that the inertial data can be optimized and the positioning accuracy of the robot can be improved.

[0080] As an embodiment, a method for image tracking by a robot using projection matching includes: step 21, the robot collects images through a camera and obtains inertial data through an inertial sensor; wherein the images collected by the camera are the previous frame image and the current frame image in sequence, and are marked as two adjacent frames of images. Then execute step 22. After the camera collects each frame of image, it will obtain feature points from each frame of image, wherein the feature points refer to environmental elements in the form of points in the environment where the robot is located, so as to facilitate matching with the feature points of the previously collected image, so that the previous frame image can track the current frame image. It should be noted that in the posture transformation relationship of the camera between two adjacent frames (two consecutive frames of images), the initial state quantities of the rotation matrix and the initial state quantities of the translation vector involved are pre-set; on the basis of these initial state quantities, the robot relies on the displacement change sensed by the code disk between two frames of images successively acquired by the camera, and the angle change sensed by the gyroscope between two frames of images successively acquired by the camera, and performs integration processing. The Euler integral can be used to integrate the displacement change and the angle change respectively, to obtain the posture transformation relationship and posture change of the robot within the specified acquisition time interval, and then the latest rotation matrix and the latest translation vector can be obtained.

[0081] Step 22, the robot projects the feature points of the previous frame image into the current frame image according to the inertial data to obtain the projection point; then executes step 23. The inertial data also includes the rotation matrix of the previous frame image relative to the current frame image, and the translation vector of the previous frame image relative to the current frame image; the robot controls the rotation matrix and the translation vector to convert the feature points in the previous frame image into the coordinate system of the current frame image, and then projects them to the imaging plane of the current frame image through the intrinsic parameters of the camera to obtain the projection point. In the process of controlling the rotation matrix and the translation vector to transform the feature points in the previous frame image into the coordinate system of the current frame image, the coordinates of the feature points in the previous frame image can be directly rotated and translated to transform the coordinates of the feature points into the coordinate system of the current frame image; or: first construct a vector of the feature points in the previous frame image relative to the origin of the coordinate system of the previous frame image, recorded as the vector to be transformed; then control the rotation matrix to multiply the vector to be transformed to obtain the rotated vector; then perform a dot multiplication operation on the translation vector used and the rotated vector to obtain the transformed vector; when the starting point of the transformed vector is the origin of the coordinate system of the current frame image, the end point of the transformed vector is the projection point; wherein, the feature points in the previous frame image can be normalized plane coordinates, but the vector to be transformed, the rotated vector, or the transformed vector can be a three-dimensional vector. Thus, a posture transformation constraint relationship (polar geometric constraint) is formed between the two frames of images.

[0082] Step 23, the robot searches for the to-be-matched point in the preset search neighborhood of each projection point according to the standard distance between the descriptors, wherein the to-be-matched point is a feature point from the current frame image, and the to-be-matched point does not belong to the projection point; then the robot calculates the vector between the projection point and each to-be-matched point searched out to determine the direction of the vector between the projection point and each to-be-matched point searched out in the preset search neighborhood to which it belongs, and the robot marks the vector between a projection point and a to-be-matched point that participates in the calculation in the preset search neighborhood as the to-be-matched vector, which is the vector from the projection point to the to-be-matched point, and the direction from the projection point to the to-be-matched point is the to-be-matched vector. The direction of the matching vector; the calculation method of the vector to be matched is to use the normalized plane coordinates of the point to be matched in the current frame image, and subtract the normalized plane coordinates of the projection point obtained by projecting the previous frame image onto the current frame image, to generate the vector to be matched; in some embodiments, the projection point is the starting point of the vector to be matched, and the point to be matched is the end point of the vector to be matched; in some embodiments, the line segment connecting the projection point and the point to be matched can be used in the current frame image to represent the modulus of the vector to be matched, and the line segment connecting the projection point and the point to be matched is marked as the line segment to be matched; the modulus length of the vector is equal to the straight-line distance between the projection point and the point to be matched; then the robot executes step 24.

[0083] Specifically, the method for the robot to search for the point to be matched in the preset search neighborhood of each projection point according to the standard distance between the descriptors includes: the robot sets a circular neighborhood with each projection point as the center point, and sets the circular neighborhood as the preset search neighborhood of the projection point, wherein the inertial data includes the change in the position and posture of the camera between the previous frame image and the current frame image, which can be equivalent to the change in the position and posture of the robot within the specified acquisition time interval, and the camera is fixedly mounted on the robot; the larger the change in the position and posture of the camera between the previous frame image and the current frame image, the larger the radius of the preset search neighborhood is set; the smaller the change in the position and posture of the camera between the previous frame image and the current frame image, the smaller the radius of the preset search neighborhood is set, so that the matching range of the feature point adapts to the actual acquisition range of the camera. Wherein, each projection point is set as the center point of the circle and a corresponding preset search neighborhood is set. Then, the robot starts searching from the center point of the preset search neighborhood of each projection point within the preset search neighborhood, specifically searching for feature points other than the projection point (feature points originally belonging to the current frame image) within the preset search neighborhood; whenever the descriptor of the feature point in the current frame image searched by the robot is closest to the standard distance of the descriptor of the center point of the preset search neighborhood, the feature point in the current frame image searched is set as a to-be-matched point in the preset search neighborhood as a candidate matching point in the current frame image with a higher matching degree with the projection point, wherein the number of to-be-matched points in the preset search neighborhood may be at least one or may be 0. In the event that a to-be-matched point is not found, the radius of the preset search neighborhood needs to be expanded to continue searching within a larger circular domain.

[0084] It should be noted that the standard distance is the Euclidean distance or Hamming distance used under the standard matching conditions between descriptors. The descriptor of a feature point is a binary description vector, which consists of many 0s and 1s. The 0s and 1s here encode the magnitude relationship of the brightness of two pixels near the feature point (such as m and n). If m is smaller than n, it takes 1; otherwise, it takes 0.

[0085] Step 231, the source of the descriptor specifically includes: selecting a square neighborhood centered on a feature point, and setting the square neighborhood as the area of ​​the descriptor;

[0086] Step 232, the square neighborhood can then be denoised, and Gaussian kernel convolution can be used to eliminate pixel noise, because the descriptor is highly random and sensitive to noise.

[0087] Step 233, generate point pairs using a certain randomization algorithm<m,n> If the brightness of pixel m is less than the brightness of pixel n, it is encoded as 1, otherwise it is encoded as 0

[0088] Step 234, repeat step 233 several times (eg, 128 times) to obtain a 128-bit binary code, which is the descriptor of the feature point.

[0089] Preferably, the method for selecting feature points includes: selecting a pixel point r in the image, assuming that its brightness is Ir; then setting a threshold T0 (for example, 20% of Ir); then taking the pixel point r as the center, selecting 16 pixel points on a circle with a radius of 3 pixel points. If there are 9 consecutive points on the selected circle whose brightness is greater than (Ir+T0) or less than (Ir-T0), then the pixel point r can be considered as a feature point.

[0090] Step 24, the robot counts the number of mutually parallel vectors to be matched, specifically the preset search neighborhood of all projection points, or the current frame image or its imaging plane. After all, the vectors to be matched are initially marked only in the preset search neighborhood of the projection point, and there is no counting interference factor in the area outside the preset search neighborhood; in each mutually parallel vector to be matched, the directions (directions) of any two vectors to be matched are the same or opposite. It should be noted that between the previous frame image and the current frame image, if a projection point and a point to be matched are matched normally, the direction of the vector to be matched from the projection point to the point to be matched is parallel to a fixed preset mapping direction, and the vectors to be matched corresponding to all pairs of normally matched feature points are parallel to each other; wherein the preset mapping direction is associated with the inertial data, especially the angle feature of its pointing is determined by the rotation matrix. Then execute step 25.

[0091] Step 25, determine whether the number of mutually parallel vectors to be matched is greater than or equal to the preset matching number, if yes, execute step 26, otherwise execute step 27. The number of mutually parallel vectors to be matched is a statistical result of the robot in the preset search neighborhood of all projection points or in the current frame image.

[0092] Step 26, determine whether the robot successfully tracks using the projection matching method, and determine whether the robot successfully tracks the current frame image. Specifically, when the number of mutually parallel vectors to be matched is greater than or equal to the preset matching number, each mutually parallel vector to be matched is set as a target matching vector, which is equivalent to setting at least two mutually parallel vectors to be matched in the same direction and at least two mutually parallel vectors to be matched in the opposite direction as target matching vectors, and accordingly, set mutually parallel or overlapping line segments to be matched as target matching line segments, and set the starting point and the end point of the target matching vector as a pair of target matching points, and correspondingly set the two end points of the target matching line segment as a pair of target matching points, which are all feature points. Then set the starting point of the vector to be matched that is not parallel to the target matching vector and the end point of the vector to be matched as a pair of mismatching points, and accordingly, set the line segment to be matched that is not parallel to the target matching line segment and does not overlap with the target matching line segment as a mismatching line segment, and then set the two end points of the mismatching line segment as a pair of mismatching points. Thus, a feature point match is completed within the preset search neighborhood of each projection point set in step S23, and a to-be-matched point with the closest standard distance to the descriptor of the projection point is obtained, and each pair of mismatched points is filtered out within the preset search neighborhood.

[0093] Step 27, the robot counts the number of mutually parallel vectors to be matched in the preset search neighborhood of all projection points (or in the current frame image) to be less than the preset matching number, then determines whether the number of repeated executions of step 23 reaches the preset expansion number, if yes, then stops expanding the coverage of the preset search neighborhood of each projection point, and determines that the robot fails to track using the projection matching method; otherwise, expands the coverage of the preset search neighborhood of each projection point to obtain an expanded preset search neighborhood, and updates the expanded preset search neighborhood to the preset search neighborhood described in step 23, and then executes step 23. Preferably, the preset matching number is set to 15, and the preset expansion number is set to 2. In some embodiments, the number of points to be matched in the preset search neighborhood can be at least one or 0. If no point to be matched is found, the radius of the preset search neighborhood needs to be expanded, and then returns to step 23 to continue searching in a larger circular domain. Among them, the preset number of expansion times is associated with the size of the current frame image and the preset expansion step. In the current frame image, if the number of parallel vectors to be matched is less than the preset matching number, it is necessary to set the preset expansion step to expand the coverage of the preset search neighborhood of each projection point. However, due to the constraint of the size of the current frame image, the preset search neighborhood of each projection point can only be reasonably expanded within a limited number of times to allow the same projection point to match more reasonable (the standard distance of the descriptor is closest) points to be matched.

[0094] After each execution of step 23, step 24 will be executed, and in step 24, the vector direction consistency will be used to remove the mismatched points, that is, each mismatched vector in the preset search neighborhood is filtered out, and then when step 23 is repeated, the mismatched vector does not need to be calculated, which greatly reduces the amount of calculation. After the robot repeats step 23 for the preset extension number of times, if the number of mutually parallel vectors to be matched in the current frame image is still less than the preset matching number, the coverage of the preset search neighborhood of each projection point is stopped from being expanded, and it is determined that the robot has failed to track using the projection matching method.

[0095] In summary, the combination of step 22 to step 27 is the projection matching method, which combines the robot's posture change between two adjacent frame images and the projection transformation relationship of the feature points, identifies the vectors to be matched with the same direction in the current frame image to be tracked and counts their number, determines whether the robot has successfully completed image tracking using the projection matching method, reduces the feature mismatch rate, reduces the calculation difficulty, and can switch to the window matching method after tracking fails, further narrowing the search range of feature point matching and improving the accuracy and efficiency of the robot's visual positioning; in addition, the present invention only uses a single camera for positioning, and the equipment is simple and low in cost.

[0096] It should be noted that in the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0097] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0099] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A robot visual tracking method, characterized in that: The execution subject of the robot visual tracking method is a robot fixedly equipped with a camera and an inertial sensor; The robot vision tracking method comprises: The robot uses the window matching method to track the image. When the robot successfully tracks the image using the window matching method, the robot stops tracking the image using the window matching method, and then the robot uses the projection matching method to track the image. Then, when the robot fails to track using the projection matching method, the robot stops using the projection matching method for image tracking, and then the robot uses the window matching method for image tracking; The robot uses window matching to perform image tracking methods including: Step S11, the robot collects the current frame image through the camera and obtains inertial data through the inertial sensor; Step S12, based on the inertial data, using the epipolar constraint error value, selecting a first feature point pair from the feature points of the current frame image and the feature points of all reference frame images in the sliding window; wherein the sliding window is set to be filled with at least one frame of image collected in advance; the feature point is a pixel point of the image, and the feature point is an environmental element existing in the form of a point in the environment where the camera is located; Step S13, based on the inertial data, using the depth value of the feature point, selecting a second feature point pair from the first feature point pair; Step S14, selecting a third feature point pair from the second feature point pair according to the similarity of the descriptors corresponding to the second feature point pair; Step S15, introducing a residual between the third feature point pair, and then combining the residual and the derivative result of the residual with respect to the inertial data to calculate the inertial compensation value, and then using the inertial compensation value to correct the inertial data; then updating the corrected inertial data to the inertial data described in step S12, and updating the feature points of the third feature point pair described in step S14 in the current frame image to the feature points of the current frame image described in step S12, and updating the feature points of the third feature point pair described in step S14 in the reference frame image to the feature points of all reference frame images in the sliding window described in step S12; Step S16, repeatedly executing step S12 and step S13 until the number of repeated executions reaches a preset number of iterative matching times, and then based on the number of feature points of the second feature point pair in each frame of the reference frame image, filtering out the matching frame image from the reference frame image in the sliding window; wherein, each time after repeatedly executing step S12 and step S13, the robot introduces a residual between the second feature point pairs filtered out by the most recently executed step S13, and then calculates an inertia compensation value by combining the residual and the result of derivation of the most recently acquired inertial data, and then uses the inertia compensation value to correct the most recently acquired inertial data, and then updates the corrected inertial data to the inertial data described in step S12, and updates the feature points included in the second feature point pairs filtered out by the most recently executed step S13 to the feature points of the current frame image described in step S12 and the feature points of all the reference frame images in the sliding window; Step S17, based on the epipolar constraint error value and the number of feature points of the second feature point pair in each matching frame image, selecting the best matching frame image from all matching frame images, and determining that the robot is successfully tracked using the window matching method; Wherein, the combination of step S12, step S13, step S14, step S15, step S16 and step S17 is the window matching method; The methods for robots to use projection matching to perform image tracking include: Step S21, the robot collects images through a camera and obtains inertial data through an inertial sensor; wherein the images collected by the camera include a previous frame image and a current frame image; Step S22, the robot projects the feature points of the previous frame image into the current frame image using inertial data to obtain projection points, wherein the inertial data includes a rotation matrix of the previous frame image relative to the current frame image, and a translation vector of the previous frame image relative to the current frame image; Step S23, the robot searches for the to-be-matched points in the preset search neighborhood of each projection point according to the standard distance between the descriptors; then the robot calculates the vector between the projection point and each to-be-matched point searched out, and marks the vector pointing from the projection point to the searched to-be-matched point as the to-be-matched vector; wherein the to-be-matched point is a feature point derived from the current frame image, and the to-be-matched point does not belong to the projection point; each to-be-matched vector corresponds to a projection point and a to-be-matched point; Step S24, the robot counts the number of mutually parallel vectors to be matched. When the number of mutually parallel vectors to be matched is greater than or equal to the preset matching number, it is determined that the robot has successfully tracked using the projection matching method, and that the robot has successfully tracked the current frame image.

2. The robot visual tracking method according to claim 1, characterized in that: The robot visual tracking method further includes: when the robot fails to track using the window matching method, the robot stops using the window matching method to track the image, the robot clears the sliding window, and then uses the window matching method to track the image; Among them, image tracking is used to represent the matching between the feature points of the previously acquired image and the feature points of the current frame image; After the robot successfully tracks using the window matching method, the current frame image is filled into the sliding window to facilitate tracking of the image collected by the robot in real time; When the robot uses the projection matching method to track images, if it detects that the time interval between the current frame image and the previous frame image exceeds the preset time threshold, the robot stops using the projection matching method for image tracking and instead uses the window matching method for image tracking.

3. The robot visual tracking method according to claim 1, characterized in that: In step S12, based on the inertial data, using the epipolar constraint error value, the method of selecting the first feature point pair from the feature points of the current frame image and the feature points of all reference frame images in the sliding window includes: The robot calculates the epipolar constraint error value of the feature point pair; when the epipolar constraint error value of the feature point pair is greater than or equal to a preset pixel distance threshold, the feature point pair is marked as an incorrectly matched point pair; when the epipolar constraint error value of the feature point pair is less than the preset pixel distance threshold, the feature point pair is marked as a first feature point pair and the first feature point pair is determined to be screened out; Each feature point pair is configured to consist of a feature point of a current frame image and a feature point of a reference frame image, and each feature point of the current frame image forms a feature point pair with each feature point in each reference frame image within the sliding window.

4. The robot visual tracking method according to claim 3, characterized in that: In step S12, when the inertial data includes a translation vector of the current frame image relative to the reference frame image, and a rotation matrix of the current frame image relative to the reference frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as a first translation vector, and marks the rotation matrix of the current frame image relative to the reference frame image as a first rotation matrix; then, the robot controls the first rotation matrix to transform the normalized vector of the feature point of the current frame image into the coordinate system of the reference frame image to obtain a first vector; then, the robot controls the first translation vector to cross-multiply the first vector to obtain a first vector; then, the robot controls the normalized vector of the feature point in the reference frame image within the sliding window to be dot-multiplied with the first vector, and then sets the result of the dot-multiplication to the epipolar constraint error value of the corresponding feature point pair; Alternatively, in step S12, when the inertial data includes a translation vector of the reference frame image relative to the current frame image, and a rotation matrix of the reference frame image relative to the current frame image, the robot marks the translation vector of the reference frame image relative to the current frame image as a second translation vector, and marks the rotation matrix of the reference frame image relative to the current frame image as a second rotation matrix; then, the robot controls the second rotation matrix to transform the normalized plane vectors of the feature points of the reference frame image in the sliding window into the coordinate system of the current frame image to obtain a second first vector; then, the robot controls the second translation vector to cross-multiply the second first vector to obtain a second second vector; then, the robot controls the normalized plane vectors of the feature points in the current frame image to be dot-multiplied with the second second vector, and then sets the result of the dot-multiplication to the epipolar constraint error value of the corresponding feature point pair; Among them, the normalized vector of the feature point of the current frame image is the vector formed by the normalized plane coordinates of the feature point of the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature point of the reference frame image is the vector formed by the normalized plane coordinates of the feature point of the reference frame image relative to the origin of the coordinate system of the reference frame image.

5. The robot visual tracking method according to claim 1, characterized in that: In the step S13, based on the inertial data, the method of selecting the second feature point pair from the first feature point pair by using the depth value of the feature point includes: The robot calculates the ratio of the depth value of the feature point of the first feature point pair selected in step S12 in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image; When the ratio of the depth value of the feature point of the first feature point pair in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image is within a preset ratio threshold range, marking the first feature point pair as a second feature point pair and determining to select the second feature point pair; When the ratio of the depth value of the feature point of the first feature point pair in the current frame image to the depth value of the feature point of the first feature point pair in the reference frame image is not within a preset ratio threshold range, the first feature point pair is marked as an incorrect matching point pair.

6. The robot visual tracking method according to claim 5, characterized in that: In step S13, when the inertial data includes a translation vector of the current frame image relative to the reference frame image, and a rotation matrix of the current frame image relative to the reference frame image, the robot marks the translation vector of the current frame image relative to the reference frame image as a first translation vector, and marks the rotation matrix of the current frame image relative to the reference frame image as a first rotation matrix, and the robot controls the first rotation matrix to convert the normalized vector of the feature point of the first feature point pair in the current frame image to the coordinate system of the reference frame image to obtain a first vector; then controls the normalized vector of the feature point of the first feature point pair in the reference frame image to be cross-multiplied by the first vector to obtain a first vector; and at the same time controls the normalized vector of the feature point of the first feature point pair in the reference frame image to be cross-multiplied by the first vector to obtain a first vector. The translation vector is then inverted to obtain the first three-vector; the product of the first three-vector and the inverse vector of the first two-vector is then set as the depth value of the feature point of the first feature point pair in the current frame image, and is marked as the first depth value, indicating the distance between the three-dimensional point detected by the camera and the optical center when the camera collects the current frame image; the product of the first one-vector and the first depth value and the sum of the first translation vector are then marked as the first four-vector, and the product of the first four-vector and the inverse vector of the normalized vector of the feature point of the first feature point pair in the reference frame image is set as the depth value of the feature point of the first feature point pair in the reference frame image, and is marked as the second depth value, indicating the distance between the same three-dimensional point and the optical center when the camera collects the reference frame image; Alternatively, in step S13, when the inertial data includes a translation vector of the reference frame image relative to the current frame image, and a rotation matrix of the reference frame image relative to the current frame image, the robot records the translation vector of the reference frame image relative to the current frame image as a second translation vector, and records the rotation matrix of the reference frame image relative to the current frame image as a second rotation matrix, and the robot controls the second rotation matrix to convert the normalized vector of the feature point of the first feature point pair in the reference frame image to the coordinate system of the current frame image to obtain a second first vector; then controls the normalized vector of the feature point of the first feature point pair in the current frame image to be cross-multiplied by the second first vector to obtain a second second vector; and at the same time controls the normalized vector of the feature point of the first feature point pair in the current frame image to be cross-multiplied by the second first vector to obtain a second second vector. The translation vector is then inverted to obtain the second three-vector; the product of the second three-vector and the inverse vector of the second two-vector is then set as the depth value of the feature point of the first feature point pair in the reference frame image, and is marked as the second depth value, indicating the distance between the three-dimensional point detected by the camera and the optical center when the camera collects the reference frame image; the product of the second one-vector and the second depth value and the sum of the second translation vector are then marked as the second four-vector, and the product of the second four-vector and the inverse vector of the normalized vector of the feature point of the first feature point pair in the current frame image is set as the depth value of the feature point of the first feature point pair in the current frame image, and is marked as the first depth value, indicating the distance between the same three-dimensional point and the optical center when the camera collects the current frame image; Among them, the normalized vector of the feature points of the first feature point pair in the current frame image is the vector formed by the normalized plane coordinates of the feature points of the first feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the first feature point pair in the reference frame image is the vector formed by the normalized plane coordinates of the feature points of the first feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image.

7. The robot visual tracking method according to claim 1, characterized in that: In the step S14, the method of selecting the third feature point pair from the second feature point pair according to the similarity of the descriptors corresponding to the second feature point pair includes: For the current frame image and each reference frame image in the sliding window, the robot calculates the similarity between the descriptor of the feature point of the second feature point pair in the reference frame image and the descriptor of the feature point of the second feature point pair in the current frame image; When the similarity between the descriptor of the feature point of the second feature point pair in the reference frame image and the descriptor of the feature point of the second feature point pair in the current frame image is the minimum value among the similarities between the descriptor of the current frame image and the descriptor of the reference frame image where the feature point of the second feature point pair is located, marking the second feature point pair as a third feature point pair and determining to filter out the third feature point pair; The descriptor of the reference frame image where the feature point of the second feature point pair is located is the descriptor of all feature points constituting the second feature point pair in the reference frame image where the feature point of the second feature point pair is located; the descriptor of the current frame image is the descriptor of the feature points in the current frame image that constitute the second feature point pair together with the feature points of the reference frame image where the feature point of the second feature point pair is located; The similarity of the descriptors corresponding to the second feature point pair is represented by the Euclidean distance or the Hamming distance between the descriptors of the feature points in the current frame image and the descriptors of the feature points in the corresponding reference frame image in the sliding window.

8. The robot visual tracking method according to claim 7, characterized in that: The step S14 further includes: whenever the robot searches for all feature points constituting the second feature point pairs between the current frame image and a frame of reference frame image in the sliding window, if the number of third feature point pairs counted by the robot in the frame of reference frame image is less than or equal to the first preset point number threshold, it is determined that the current frame image and the frame of reference frame image fail to match, and the frame of reference frame image is set as a mismatched reference frame image; if the number of third feature point pairs counted by the robot in the frame of reference frame image is greater than the first preset point number threshold, it is determined that the current frame image and the frame of reference frame image successfully match; When the robot determines that the current frame image and all frame reference frame images in the sliding window fail to match, it is determined that the robot fails to track using the window matching method, and then the robot clears the image in the sliding window.

9. The robot visual tracking method according to claim 1, characterized in that: The robot marks the line connecting the optical center when the camera collects the current frame image and the feature points of the preset feature point pair in the current frame image as the first observation line, and marks the line connecting the optical center when the camera collects the reference frame image and the feature points of the same preset feature point pair in the reference frame image as the second observation line, and then marks the intersection of the first observation line and the second observation line as the target detection point; The preset feature point pair, the optical center when the camera collects the current frame image, and the optical center when the camera collects the reference frame image are all in the same plane; or, the optical center when the camera collects the current frame image, the optical center when the camera collects the reference frame image, and the target detection point are all in the same plane; the same plane is an epipolar plane; The robot records the intersection line of the polar plane and the current frame image as the polar line in the imaging plane of the current frame image, and records the intersection line of the polar plane and the reference frame image as the polar line in the imaging plane of the reference frame image; In the same preset feature point pair, after the feature point of the current frame image is converted to the reference frame image, it becomes a first projection point, and its coordinates are first coordinates; the distance from the first projection point to the polar line in the imaging plane of the reference frame image is represented as a first residual value; in the same preset feature point pair, after the feature point of the reference frame image is converted to the current frame image, it becomes a second projection point, and its coordinates are second coordinates; the distance from the second projection point to the polar line in the imaging plane of the current frame image is represented as a second residual value; In step S15, the preset feature point pair is the third feature point pair; In step S16, each time after step S12 and step S13 are repeatedly executed, the preset feature point pair is the second feature point pair selected by the most recently executed step S13.

10. The robot visual tracking method according to claim 9, characterized in that: In step S15 or step S16, the method of introducing the residual includes: When the inertial data includes a translation vector of the current frame image relative to the reference frame image, and a rotation matrix of the current frame image relative to the reference frame image, the robot records the translation vector of the current frame image relative to the reference frame image as a first translation vector, and records the rotation matrix of the current frame image relative to the reference frame image as a first rotation matrix, and the robot controls the first rotation matrix to convert the normalized vector of the feature point of the preset feature point pair in the current frame image to the coordinate system of the reference frame image to obtain a first vector; then controls the first translation vector to cross-multiply the first vector to obtain a first vector, and forms an epipolar line in the imaging plane of the reference frame image; then calculates the square root of the sum of the squares of the horizontal axis coordinates in the first vector and the vertical axis coordinates in the first vector to obtain the modulus of the epipolar line; at the same time, controls the normalized vector of the feature point of the preset feature point pair in the reference frame image to be dot-multiplied with the first vector, and then sets the result of the dot multiplication as the epipolar constraint error value of the preset feature point pair; then sets the ratio of the epipolar constraint error value of the preset feature point pair to the modulus of the epipolar line as the first residual value; Alternatively, when the inertial data includes a translation vector of the reference frame image relative to the current frame image, and a rotation matrix of the reference frame image relative to the current frame image, the robot records the translation vector of the reference frame image relative to the current frame image as a second translation vector, and records the rotation matrix of the reference frame image relative to the current frame image as a second rotation matrix, and the robot controls the second rotation matrix to convert the normalized vector of the feature point of the preset feature point pair in the reference frame image to the coordinate system of the current frame image to obtain a second first vector; then controls the second translation vector to cross-multiply the second first vector to obtain a second second vector, and forms an epipolar line in the imaging plane of the current frame image; then calculates the square root of the sum of the squares of the horizontal axis coordinate in the second second vector and the vertical axis coordinate in the second second vector to obtain the modulus of the epipolar line; at the same time, controls the normalized vector of the feature point of the preset feature point pair in the current frame image to be dot-multiplied with the second second vector, and then sets the result of the dot multiplication as the epipolar constraint error value of the preset feature point pair; then sets the ratio of the epipolar constraint error value of the preset feature point pair to the modulus of the epipolar line as the second residual value; Among them, the normalized vector of the feature points of the preset feature point pair in the current frame image is the vector formed by the normalized plane coordinates of the feature points of the preset feature point pair in the current frame image relative to the origin of the coordinate system of the current frame image; the normalized vector of the feature points of the preset feature point pair in the reference frame image is the vector formed by the normalized plane coordinates of the feature points of the preset feature point pair in the reference frame image relative to the origin of the coordinate system of the reference frame image.

11. The robot visual tracking method according to claim 10, characterized in that: In step S15 or step S16, a residual is introduced between the preset feature point pairs, and an inertia compensation value is calculated by combining the residual and a result of derivation of the latest inertia data, and then the inertia compensation value is used to correct the latest inertia data. The method includes: When the inertial data includes the translation vector of the current frame image relative to the reference frame image, and the rotation matrix of the current frame image relative to the reference frame image, the robot marks the formula of multiplying the first rotation matrix and the normalized vector of the feature point of the preset feature point pair in the current frame image as the first-first conversion formula; then marks the formula of cross-multiplying the first translation vector and the first-first conversion formula as the first-second conversion formula; then marks the formula of multiplying the normalized vector of the feature point of the preset feature point pair in the reference frame image with the first-second conversion formula as the first-third conversion formula; then sets the calculation result of the first-second conversion formula to the value 0 to form a straight line equation, and then calculates the straight line equation in the horizontal axis coordinate dimension. The square sum of the coefficients of the coefficients and the coefficients of the vertical axis coordinate dimension is calculated, and then the square root of the obtained square sum is calculated to obtain the first square root, and then the formula of multiplying the reciprocal of the first square root and the first three-conversion formula is set as the first four-conversion formula; then the calculation result of the first four-conversion formula is set as the first residual value to form a first residual derivation formula, and it is determined to introduce a residual between the preset feature point pairs; then the first residual derivation formula is controlled to respectively calculate the partial derivatives of the first translation vector and the first rotation matrix to obtain the Jacobian matrix; then the product of the inverse matrix of the Jacobian matrix and the first residual value is set as the inertia compensation value; then the robot uses the inertia compensation value to correct the inertia data; Alternatively, when the inertial data includes a translation vector of a reference frame image relative to a current frame image, and a rotation matrix of the reference frame image relative to the current frame image, the robot marks the formula for multiplying the second rotation matrix with the normalized vector of a feature point of a preset feature point pair in the reference frame image as a second-first transformation formula; then marks the formula for cross-multiplying the second translation vector with the second-first transformation formula as a second-second transformation formula; then marks the formula for multiplying the normalized vector of a feature point of a preset feature point pair in the current frame image with the second-second transformation formula as a second-third transformation formula; then sets the calculation result of the second-second transformation formula to a value of 0 to form a straight line equation, and then calculates the straight line equation at the horizontal axis coordinate The square sum of the coefficient of the dimension and the coefficient of the ordinate coordinate dimension is calculated, and then the square root of the obtained square sum is calculated to obtain a second square root, and then the formula of multiplying the reciprocal of the second square root and the second third conversion formula is set as the second fourth conversion formula; then the calculation result of the second fourth conversion formula is set as the second residual value to form a second residual derivation formula, and it is determined to introduce a residual between the preset feature point pairs; then the second residual derivation formula is controlled to respectively calculate the partial derivative of the second translation vector and the second rotation matrix to obtain the Jacobian matrix; then the product of the inverse matrix of the Jacobian matrix and the second residual value is set as the inertia compensation value; then the robot uses the inertia compensation value to correct the inertia data.

12. The robot visual tracking method according to claim 8, characterized in that: For step S16, after the robot completes step S15, when step S12 is repeated for the first time, the robot calculates the epipolar constraint error value of each third feature point pair except for the mismatched reference frame image, wherein the epipolar constraint error value of each third feature point pair is determined by the inertial data corrected in step S15; when the epipolar constraint error value of the third feature point pair is less than a preset pixel distance threshold, the third feature point pair is updated to the first feature point pair, and a new first feature point pair is determined to be selected from the third feature point pairs; When step S12 is repeated for the Nth time, the robot calculates the epipolar constraint error value of each second feature point pair screened out in the most recently executed step S13; when the epipolar constraint error value of the second feature point pair is less than the preset pixel distance threshold, the second feature point pair is updated to the first feature point pair, and a new first feature point pair is selected from all the second feature point pairs screened out in the step S13; wherein N is set to be greater than 1 and less than or equal to the preset number of iterative matching times.

13. The robot visual tracking method according to claim 5, characterized in that: In the step S16, the method of selecting matching frame images from the reference frame images in the sliding window based on the number of feature points of the second feature point pair in each reference frame image includes: The robot counts the number of feature points of the second feature point pair in each reference frame image in the sliding window; If the number of second feature point pairs matched by the robot in one of the reference frame images is less than or equal to the second preset point threshold, it is determined that one of the reference frame images fails to match the current frame image; if the number of second feature point pairs matched by the robot in one of the reference frame images is greater than the second preset point threshold, it is determined that one of the reference frame images successfully matches the current frame image, and one of the reference frame images is set as the matching frame image; if the number of second feature point pairs matched by the robot in each reference frame image is less than or equal to the second preset point threshold, it is determined that each reference frame image in the sliding window fails to match the current frame image, and it is determined that the robot fails to track using the window matching method.

14. The robot visual tracking method according to claim 4, characterized in that: In step S17, the method of selecting the best matching frame image from all matching frame images based on the epipolar constraint error value and the number of feature points of the second feature point pair in each matching frame image includes: In each matching frame image, a sum of the epipolar constraint error values ​​of the second feature point pair to which the feature points in the matching frame image belong is calculated as an accumulated value of the epipolar constraint error value of the matching frame image; In each matching frame image, the number of feature points constituting the second feature point pairs in the matching frame image is counted as the number of feature point matches in the matching frame image; Then, the matching frame image with the smallest accumulated value of the extreme constraint error value and the largest number of feature point matches is set as the optimal matching frame image.

15. The robot visual tracking method according to claim 1, characterized in that: The robot sets each mutually parallel to-be-matched vector as a target matching vector, and marks the to-be-matched vector that is not parallel to the target matching vector as an incorrect matching vector within a preset search neighborhood of all projection points, and then sets a projection point corresponding to the incorrect matching vector and a to-be-matched point corresponding to the incorrect matching vector as a pair of incorrect matching points, and sets a projection point corresponding to the target matching vector and a to-be-matched point corresponding to the target matching vector as a pair of target matching points; The directions of each mutually parallel vector to be matched are the same or opposite, and the target matching vector remains parallel to the preset mapping direction, and the preset mapping direction is associated with the inertial data.

16. The robot visual tracking method according to claim 15, characterized in that: Step S24 also includes: when the number of mutually parallel vectors to be matched is less than the preset matching number, the robot expands the coverage of the preset search neighborhood of each projection point according to the preset expansion step length to obtain the expanded preset search neighborhood, and updates the expanded preset search neighborhood to the preset search neighborhood described in step S23, and then executes step S23 until the robot repeats step S23 for a preset expansion number of times, then stops expanding the coverage of the preset search neighborhood of each projection point, and determines that the robot fails to track using the projection matching method; The combination of step S22, step S23 and step S24 is the projection matching method.

17. The robot visual tracking method according to claim 15, characterized in that: The method for the robot to search for a point to be matched in a preset search neighborhood of each projection point according to the standard distance between descriptors includes: The robot sets a circular neighborhood with each projection point as the center point, and sets the circular neighborhood as a preset search neighborhood of the projection point, wherein the inertial data includes a change in the position and posture of the camera between the previous frame image and the current frame image; the larger the change in the position and posture of the camera between the previous frame image and the current frame image, the larger the radius of the preset search neighborhood is set; the smaller the change in the position and posture of the camera between the previous frame image and the current frame image, the smaller the radius of the preset search neighborhood is set; The robot searches for feature points other than the projection point in a preset search neighborhood of each projection point, starting from the center point of the preset search neighborhood of the projection point; when the descriptor of the searched feature point in the current frame image is closest to the standard distance of the descriptor of the center point of the preset search neighborhood, the searched feature point in the current frame image is set as a to-be-matched point in the preset search neighborhood; The standard distance is represented by Euclidean distance or Hamming distance.

Images