Visual navigation mobile robot in-situ slip detection method based on optical flow

Abstract

The invention relates to a visual navigation mobile robot in-situ slip detection method based on optical flow, and the method comprises the steps: obtaining feature points of a current frame image of a mobile robot, and storing the current frame image, the feature points of the current frame image and a physical odometer value in a queue; acquiring pixel coordinate differences of corresponding feature points of the current frame image and the adjacent frame image in the queue by adopting an optical flow tracking algorithm, and acquiring a first proportion value r1 of the feature points with unobvious pixel changes in the number of all the feature points according to the number of the pixel coordinate differences; acquiring a second proportion value r2 of the current frame image and the first frame image in the queue; and when the r2 is greater than a specified numerical value, judging whether a difference value between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image is greater than a specified threshold value, and if so, determining that the current mobile robot is in a slipping state. The method ensures the reliability of slip detection.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to an on-site skidding detection method for a visual navigation mobile robot based on optical flow. Background Art

[0002] The slip problem is the most common type of anomaly that mobile robots (AGVs) face when performing autonomous positioning and navigation tasks. Usually, slipping is caused by some reason that the friction between its wheels and the ground is insufficient to maintain its normal driving trajectory, resulting in slipping or deviation from the predetermined path. Slipping is caused by many factors, including but not limited to the following:

[0003] (1) Ground conditions: A wet, oily, uneven ground or one with foreign objects may reduce the friction between the wheels and the ground, causing the AGV to slip.

[0004] (2) Wheel material and wear: The wheel material, hardness, and wear degree will affect its contact effect and friction with the ground. For example, severely worn wheels or wheel materials that are not suitable for the current ground conditions may cause slipping.

[0005] (3) Load and center of gravity: If the load carried by the AGV is too heavy or unevenly distributed, as well as the center of gravity of the vehicle is offset, it may affect the wheel grip and driving stability, thereby causing slipping.

[0006] (4) Drive system problems: Insufficient drive motor power, transmission system failure, or improper control algorithm may cause the AGV to lack power during driving, resulting in slippage.

[0007] (5) Speed ​​control: If the AGV travels too fast, especially when turning or encountering obstacles, it may also cause slipping if the speed is not properly controlled.

[0008] Among the slipping phenomena, the most serious one is the mobile robot slipping in place, that is, the wheels of the mobile robot are spinning, but the actual AGV is not moving at all; usually, the reason for the slipping in place is that the AGV collides with rigid obstacles such as walls and equipment, or the AGV driving wheel or driven wheel is stuck in the depression of the ground / equipment. After the slipping in place, the AGV wheels are constantly rubbing against the ground, which will cause the working current of the drive motor to be overloaded for a long time, and it is easy to have overheating and other faults. At the same time, the wheels and the ground will have different degrees of wear, which will also cause the mobile robot to lose its positioning. Therefore, compared with the general mobile robot slipping problem (such as slipping while walking, drifting and slipping, etc.), the detection and alarm of the slipping in place abnormality is particularly important.

[0009] Generally, the existing method to solve the problem of skidding anomaly detection is to collect wheel encoder data (physical odometer), inertial odometer data (IMU), and visual odometer data (optical flow odometer), and judge whether the robot is skidding by comparing the deviation between these odometers. The principles of these methods are essentially the same, that is, when the mobile robot is not skidding, the movement of the mobile robot recorded by the physical odometer, inertial odometer, and visual odometer should be relatively close. When skidding occurs, due to the relative sliding of the wheels relative to the ground or the idling of the wheels, the movement of the mobile robot recorded by the physical odometer, inertial odometer, and visual odometer will deviate. Therefore, the existing visual positioning skidding detection methods all use the above deviations as the basis for judging whether the robot is skidding. However, this type of solution faces a fatal problem, that is, the optical flow method has fixed defects: (1) Sensitive to changes in illumination: The optical flow method is based on the assumption of constant brightness, so when the illumination conditions change, the accuracy of the algorithm will be affected. (2) There are problems with the accuracy and robustness of the optical flow visual odometry. There is scale drift, and the tracking effect is limited. In some scenarios (such as areas with low local photometric discrimination), the tracking effect of the optical flow method may decline or even lose tracking.

[0010] When the optical flow visual odometer has a large error or fault, the above-mentioned deviation-based skidding detection method has a great hidden danger of false alarm or missed alarm of skidding. Summary of the invention

[0011] 1. Technical issues to be resolved

[0012] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method for detecting slippage of a visually navigated mobile robot based on optical flow, which improves the reliability of the optical flow method for slippage detection and avoids the errors caused by the introduction of visual odometers in the prior art, so as to effectively ensure the reliability of slippage detection.

[0013] (II) Technical solution

[0014] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0015] In a first aspect, an embodiment of the present invention provides a method for detecting slippage of a mobile robot based on visual navigation using optical flow, comprising:

[0016] A100, obtaining feature points of the current frame image of the mobile robot, and when the number of feature points of the current frame image meets the requirement, storing the current frame image and the feature points of the current frame image and the physical odometer value corresponding to the current frame image in a queue; the current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm;

[0017] A200, when there are at least two frames of images in the queue that meet the timestamp requirements, an optical flow tracking algorithm is used to obtain the difference in pixel coordinates of corresponding feature points between the current frame image and the adjacent frame image, and according to the number of pixel coordinate differences, a first ratio value r1 of feature points with insignificant pixel changes to the number of all feature points is obtained,

[0018] A300, when r1 is greater than a specified value, obtaining a second ratio value r2 between the current frame image and the first frame image in the queue;

[0019] S400. When r2 is greater than a specified value, determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold. If so, determine that the current mobile robot is in a slipping state.

[0020] Optionally, before A100, the method further includes:

[0021] Determine whether the brightness of the current frame image belongs to the image range of the reliability calculation of the optical flow tracking algorithm. If so, execute step A100, otherwise discard it;

[0022] and / or,

[0023] The A100 includes:

[0024] After acquiring the feature points of the current frame image of the mobile robot, determining whether the number of the feature points is greater than a first threshold, if so, determining that the requirement is met, otherwise, discarding;

[0025] and / or,

[0026] After A100, the method further includes:

[0027] If there is only the current frame image in the queue, end this detection;

[0028] The timestamp requirement of the queue is less than or equal to 4 to 7 seconds. That is to say, in this embodiment, it is necessary to ensure that the image frames in the queue are image frames at adjacent moments. When the difference between the timestamps of the first frame and the last frame of the queue is greater than the above value, the first frame of the queue is removed, and so on, until the timestamp of the first frame of the queue meets the above requirement.

[0029] Optionally, the A200 includes:

[0030] Select the feature point set Ω of the last frame image in the queue except the current frame image 最后一帧 , that is, Ω 最后一帧 ={c1,c2,…,c n}, n is the number of feature points in the last frame image;

[0031] For each feature point of the last frame image, the optical flow tracking algorithm is used to find the feature point with Ω in the current frame. 最后一帧 The feature points corresponding to each feature point in Φ form a set, 匹配特征点集合 ={d1,d2,…,d n};

[0032] Calculate the two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points diff i ,Right now

[0033] diff i =||c i -d i || for i=1,2,3,…,n

[0034] Among them, ||c i -d i || means for feature point c i , d i The pixel coordinates of the two images are used to find the Euclidean space distance;

[0035] Statistical tracking of the difference in pixel coordinates of feature points i The number num that is less than a given second threshold;

[0036] And, r1=num / n is calculated, where r1 represents the ratio of feature points with insignificant changes to the total number of feature points.

[0037] Optionally, after A300, the method further includes:

[0038] If r1 is less than or equal to the specified value, the queue is cleared, and the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image are stored in the cleared queue.

[0039] Optionally, after A400, the method further includes:

[0040] If r2 is less than or equal to the specified value, the first frame image in the queue is discarded and step A300 is repeated.

[0041] Optionally, judging whether the brightness of the current frame image belongs to the image range for reliability calculation of the optical flow tracking algorithm includes:

[0042] Calculate the average value of all pixels in the current frame image. When the average value is less than the pre-given sixth threshold, it is determined that the current frame image is too dark; when the average value is greater than the pre-given seventh threshold, it is determined that the current frame image is too bright, and both are not within the image range for reliability calculation of the optical flow tracking algorithm.

[0043] In a second aspect, an embodiment of the present invention further provides a detection device for detecting slippage of a mobile robot, comprising:

[0044] The feature point processing module is used to obtain the feature points of the current frame image of the mobile robot. When the number of feature points of the current frame image meets the requirement, the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image are stored in a queue; the current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm;

[0045] The first pixel change judgment module is used to use the optical flow tracking algorithm to obtain the difference in pixel coordinates of the corresponding feature points of the current frame image and the adjacent frame image when there are at least two frames of images that meet the timestamp requirements in the queue, and to obtain a first ratio value r1 of the number of feature points with insignificant pixel changes to all feature points according to the number of pixel coordinate differences.

[0046] The second pixel change judgment module is used to obtain a second ratio value r2 between the current frame image and the first frame image in the queue when r1 is greater than a specified value;

[0047] The slip detection module is used to determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold when r2 is greater than a specified value. If so, it is determined that the current mobile robot is in a slip state.

[0048] Optionally, the first pixel change judgment module is specifically used to:

[0049] Select the feature point set Ω of the last frame image in the queue except the current frame image 最后一帧 , that is, Ω 最后一帧 ={c1,c2,…,c n}, n is the number of feature points in the last frame image;

[0050] For each feature point of the last frame image, the optical flow tracking algorithm is used to find the feature point with Ω in the current frame. 最后一帧 The feature points corresponding to each feature point in Φ form a set, 匹配特征点集合 ={d1,d2,…,d n};

[0051] Calculate the two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points diff i ,Right now

[0052] diff i =||c i -d i || for i=1,2,3,…,n

[0053] Among them, ||c i -d i || means for feature point c i , d i The pixel coordinates of the two images are used to find the Euclidean space distance;

[0054] Statistical tracking of the difference in pixel coordinates of feature points i The number num that is less than a given second threshold;

[0055] And, r1=num / n is calculated, where r1 represents the ratio of feature points with insignificant changes to the total number of feature points.

[0056] In a third aspect, an embodiment of the present invention further provides a mobile robot, comprising: a detection device, a storage device and a camera, wherein the camera and the detection device are both connected to the storage device, and the detection device executes the steps of the method for detecting slippage of a mobile robot in place based on visual navigation based on optical flow as described in any one of the first aspects above.

[0057] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, wherein the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for detecting in-situ slipping of a visually navigated mobile robot based on optical flow as described in any one of the first aspects.

[0058] (III) Beneficial effects

[0059] The method of the present invention aims at the most serious skidding situation, namely skidding on the spot, and makes full use of the characteristic that the visual image will remain unchanged when skidding on the spot. The optical flow method is used to directly compare the image data, while avoiding inter-frame pose calculation based on optical flow and feature point matching. This can avoid introducing errors caused by the visual odometer, thereby ensuring the reliability of skidding detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic flow chart of a method for detecting slippage of a mobile robot using visual navigation based on optical flow according to an embodiment of the present invention;

[0061] Figure 2 A schematic flow chart of a method for detecting slippage of a mobile robot using visual navigation based on optical flow according to another embodiment of the present invention;

[0062] Figure 3 A schematic structural diagram of a detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0064] In an embodiment of the present invention, in order to improve the reliability of the optical flow method for skidding detection, the present invention proposes a method for detecting skidding in situ of a visually navigated mobile robot based on optical flow. For the most serious skidding situation such as skidding in situ, the method makes full use of the characteristic that the visual image will remain unchanged when skidding in situ, and uses the optical flow method to directly compare the image data, avoiding inter-frame pose calculation based on optical flow and feature point matching, thereby avoiding the introduction of errors caused by the visual odometer, and further ensuring the reliability of skidding detection.

[0065] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0066] Embodiment 1

[0067] like Figure 1 As shown, Figure 1 A flow chart of a method for detecting slippage in situ of a visually navigated mobile robot based on optical flow provided in one embodiment of the present invention is shown. The executor of the method of this embodiment is a detection device of a controller on a mobile robot. An image acquisition device such as a camera is installed on the mobile robot. Images acquired by the camera are stored in a storage device. The detection device can obtain images acquired by the camera in real time. Specifically, the detection device can execute the following steps.

[0068] A100. Obtain feature points of the current frame image of the mobile robot. When the number of feature points of the current frame image meets the requirement, store the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image in a queue; the current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm.

[0069] For example, before the step A100, it can be determined whether the brightness of the current frame image belongs to the image range of the reliability calculation of the optical flow tracking algorithm. If so, step A100 is executed, otherwise it is discarded. For example, the average value of all pixels in the current frame image is calculated. When the average value is less than the sixth threshold value given in advance, it is determined that the current frame image is too dark; when the average value is greater than the seventh threshold value given in advance, it is determined that the current frame image is too bright, and both do not belong to the image range of the reliability calculation of the optical flow tracking algorithm.

[0070] Furthermore, in A100, after obtaining the feature points of the current frame image of the mobile robot, it is determined whether the number of the feature points is greater than a first threshold. If so, it is determined that the requirement is met, otherwise, it is discarded.

[0071] In a specific application, after A100, if there is only the current frame image in the queue, the detection is terminated; the timestamp requirement of the queue in this embodiment is less than or equal to 4 to 7 seconds.

[0072] In this embodiment, it is necessary to ensure that the image frames in the queue are image frames at adjacent moments. When the difference between the timestamps of the first frame and the last frame of the queue is greater than the above value, the first frame of the queue is discarded, and so on, until the timestamp of the first frame of the queue meets the above requirements.

[0073] A200. When there are at least two frames of images in the queue that meet the timestamp requirements, an optical flow tracking algorithm is used to obtain the difference in pixel coordinates of corresponding feature points between the current frame image and the adjacent frame image, and based on the number of pixel coordinate differences, a first ratio value r1 of feature points whose pixel changes are not obvious to the total number of feature points is obtained.

[0074] In this embodiment, the optical flow tracking algorithm is an existing algorithm, which is used to obtain the difference in pixel coordinates of corresponding feature points of two frame images. In this step, the optical flow tracking algorithm is used to process the current frame image and the last frame image in the queue except the current frame image, and obtain the difference in pixel coordinates of corresponding feature points of the two frame images.

[0075] For example, in this embodiment, the feature point set Ω of the last frame image in the queue except the current frame image can be selected. 最后一帧 , that is, Ω 最后一帧 ={c1,c2,…,c n}, n is the number of feature points in the last frame image;

[0076] For each feature point of the last frame image, the optical flow tracking algorithm is used to find the feature point with Ω in the current frame. 最后一帧 The feature points corresponding to each feature point in Φ form a set, 匹配特征点集合 ={d1,d2,…,d n};

[0077] Calculate the two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points diff i ,Right now

[0078] diff i =||c i -d i || for i=1,2,3,…,n

[0079] Among them, ||c i -d i || means for feature point c i , b i The Euclidean space distance between the pixel coordinates of the two images is calculated; i is a natural number that is less than or equal to n.

[0080] Statistical tracking of the difference in pixel coordinates of feature points i The number num that is less than a given second threshold;

[0081] And, r1=num / n is calculated, where r1 represents the ratio of feature points with insignificant changes to the total number of feature points.

[0082] A300, when r1 is greater than a specified value, obtain a second ratio value r2 between the current frame image and the first frame image in the queue.

[0083] In this embodiment, if r1 is less than or equal to the specified value, the queue is cleared, and the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image are stored in the cleared queue.

[0084] It should be noted that the method for obtaining the second ratio value r2 in this embodiment is the same as the method for obtaining the first ratio value r1 mentioned above. Both use an optical flow tracking algorithm to obtain the difference in pixel coordinates of corresponding feature points between two frames of images (where r1 is for the current frame image and the last frame image except the current frame image, and r2 is for the current frame image and the first frame image in the queue), and then obtain the second ratio value r2 of the feature points with no obvious pixel changes to the total number of feature points based on the number of pixel coordinate differences.

[0085] S400. When r2 is greater than a specified value, determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold. If so, determine that the current mobile robot is in a slipping state.

[0086] Of course, if r2 is less than or equal to the specified value, the first frame image in the queue is removed and the step A300 is repeated.

[0087] The method of this embodiment eliminates the influence of lighting, improves the accuracy of self-detection of the detection device, overcomes the defects of scale drift in the calculation of the existing optical flow tracking algorithm and the limited tracking effect, improves the reliability of detection, and at the same time avoids the problem of false alarm or missed alarm due to slipping.

[0088] Embodiment 2

[0089] See also Figure 2 , Figure 2A schematic flow chart of a method for detecting slippage of a mobile robot using visual navigation based on optical flow according to another embodiment of the present invention is shown. The method of this embodiment includes the following steps.

[0090] S1. The image acquisition device of the mobile robot acquires and stores images and the physical odometer value at the current moment in real time; the detection device acquires the real-time acquired image and the physical odometer value corresponding to the image;

[0091] S2, the detection device determines whether the brightness of the real-time collected image is too dark or too bright, if it is too dark or too bright, then the current calculation is terminated and the next frame of image is waited for. If the image is not too dark or too bright, then the process proceeds to step S3.

[0092] Specifically, the average value of all pixels in the current image is calculated. When the average value is less than a predetermined sixth threshold, it is determined that the current image is too dark; when the average value is greater than a predetermined seventh threshold, it is defined as the current image is too bright.

[0093] In this embodiment, an image that is too dark or too bright is likely to cause failure or error in optical flow tracking, so when this situation is detected, the current calculation is terminated directly and the next frame of image is waited for. Otherwise, the process proceeds to step S3.

[0094] S3. The detection device extracts feature points of the current image.

[0095] For example, the feature points in this step may refer to corner points or interest points in the detected image. In this embodiment, feature points in the current image may be extracted by means of feature point extraction methods such as SIFT, ORB, SURF, Harris, etc.

[0096] S4, the detection device determines whether the number of feature points of the current image is greater than the first threshold value, if so, it enters step S5 of determining the queue storing the current image, otherwise, the current calculation is terminated. That is, when the number of feature points of the current image is not greater than the first threshold value, it lacks reliable texture information, and thus cannot be tracked using the optical flow method, and the current calculation is terminated.

[0097] In this embodiment, the number of feature points is judged to determine whether the optical flow method can be used for tracking information, thereby ensuring the reliability of the algorithm.

[0098] That is, in step S3, it is necessary to ensure that enough trackable feature points are extracted. When this number is small, it means that there is a lack of reliable texture information in the current image, and there is a great risk in using the optical flow method for tracking. Therefore, if this situation is detected, the calculation is terminated directly and waits for the next frame image.

[0099] S5. Determine whether the queue is empty. If the queue is empty, add the current image with enough trackable feature points to the queue and end the calculation.

[0100] When the queue is not empty, step S6 may be executed.

[0101] The queue in this embodiment is empty, which means that the system is running for the first time and no image has been used for optical flow calculation before. Therefore, the image of the current frame and image related information are added to the queue to end the current calculation and wait for the next frame of image data.

[0102] The image association information in this embodiment may include: a current frame image with a timestamp, feature points of the current frame image, and a physical odometer value corresponding to the current frame image).

[0103] The queue of this embodiment stores all image frames that are determined by the optical flow method to be not moving; the timestamp difference of all images in the queue is within 3s, 4s or 5s. If the timestamp difference is exceeded, the exceeded images can be cleared. In actual mobile robots, there is only a few seconds of slippage, and its detection is time-sensitive, so it is impossible to detect slippage characteristics with images of longer time.

[0104] S6, optical flow tracking calculation; obtaining the difference in pixel coordinates between corresponding feature points of two frames of images in the queue.

[0105] Specifically, the feature point set of the last frame of the selection queue except the current frame is selected, that is, Ω 最后一帧 ={c1,c2,…,c n}, n is the number of feature points in the last frame, and each feature point uses the optical flow method to find the feature points corresponding to each feature point in Ω in the feature points of the current frame to form a set Φ 匹配特征点集合 ={d1,d2,…,d n}.

[0106] Calculate the difference in pixel coordinates between all corresponding feature points in the two sets, that is,

[0107] diff i =||c i -d i || for i=1,2,3,…,n

[0108] Among them, ||c i -d i || means for feature point c i , d i The Euclidean space distance between the pixel coordinates of the two images is calculated.

[0109] S7, counting the number of pixel changes of the feature points on the track that are less than a given second threshold, refers to counting the above diff i The number of values ​​less than the given second threshold is denoted as num.

[0110] And, r=num / n is calculated, where r represents the ratio of feature points with insignificant changes to the total number of feature points.

[0111] S8, judging whether r is less than a given third threshold value. When r is less than a given threshold value, it is determined that a considerable number of feature points in the two frames of images have obviously moved, and it can be considered that the robot has definitely moved, so the queue is cleared;

[0112] At the same time, the current frame information is added to the queue (the information here includes the image of the current frame, feature points and the physical odometer value corresponding to the current frame), and the calculation is completed.

[0113] On the contrary, it is determined that the positions of most of the feature points on the two frames of images have no obvious change, and it can be determined that the robot has not moved, and the process goes to step S9.

[0114] S9, traverse the queue from front to back, select the feature points of the first frame in the current queue and perform optical flow tracking calculation with the current frame. The feature point set of the first frame in the current queue is recorded as Ω′={c1,c2,…,c m}, m is the number of feature points in the last frame, and the optical flow method is used to find the feature points corresponding to each feature point in Ω′ in the current frame to form a set Φ′={d1,d2,…,d m}.

[0115] Calculate the difference in pixel coordinates between all corresponding feature points in the two sets, that is,

[0116] diff i =||c i -d i || for i=1,2,3,…,m

[0117] Among them, ||c i -d i || means for feature point c i , d i The Euclidean space distance between the pixel coordinates of the two images is calculated.

[0118] S10, counting the number of pixel changes of the feature points on the track that are less than a given second threshold, refers to counting the above-mentioned diff i The number of feature points less than the given second threshold is recorded as num. Calculate r=num / n, where r represents the ratio of feature points with insignificant changes to the total number of feature points.

[0119] S11, judging whether r is less than a given third threshold value. When r is less than a given threshold value, it means that a considerable number of feature points have obviously moved from the first frame of the queue to the current frame, and it can be considered that the robot has definitely moved. Therefore, the first frame of the current queue is removed from the queue, and the process returns to step S9. Otherwise, it means that the positions of most feature points have not changed significantly from the first frame of the queue to the current frame, indicating that the robot has not moved, and the process goes to step S12.

[0120] S12. Compare whether the difference between the physical odometer corresponding to the first frame of the queue and the physical odometer of the current frame is greater than the fourth threshold. If it is greater, it means that the physical odometer believes that the robot has moved, and S11 has ensured that there is no obvious change between the first frame and the current frame of the queue, indicating that although the physical odometer of the robot has changed, the car has not moved, so the slip is detected and reported.

[0121] In the above embodiment, for the most serious slipping situation such as skidding on the spot, the characteristic that the visual image will remain unchanged when skidding on the spot is fully utilized, and the optical flow method is used to directly compare the image data, rather than performing inter-frame pose calculation based on optical flow and feature point matching. This can avoid introducing errors caused by the visual odometer, thereby ensuring the reliability of skidding detection.

[0122] Embodiment 3

[0123] like Figure 3 As shown, this embodiment also provides a detection device for skidding detection of a mobile robot, which includes: a feature point processing module, a first pixel change judgment module, a second pixel change judgment module, and a skidding detection module;

[0124] The feature point processing module is used to obtain the feature points of the current frame image of the mobile robot. When the number of feature points of the current frame image meets the requirement, the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image are stored in a queue; the current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm;

[0125] The first pixel change judgment module is used to use the optical flow tracking algorithm to obtain the difference in pixel coordinates of the corresponding feature points of the current frame image and the adjacent frame image when there are at least two frames of images that meet the timestamp requirements in the queue, and to obtain a first ratio value r1 of the number of feature points with insignificant pixel changes to all feature points according to the number of pixel coordinate differences.

[0126] The second pixel change judgment module is used to obtain a second ratio value r2 between the current frame image and the first frame image in the queue when r1 is greater than a specified value;

[0127] The slip detection module is used to determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold when r2 is greater than a specified value. If so, it is determined that the current mobile robot is in a slip state.

[0128] For example, the first pixel change determination module is specifically used to:

[0129] Select the feature point set Ω of the last frame image in the queue except the current frame image 最后一帧 , that is, Ω 最后一帧 ={c1,c2,…,c n}, n is the number of feature points in the last frame image;

[0130] For each feature point of the last frame image, the optical flow tracking algorithm is used to find the feature point with Ω in the current frame. 最后一帧 The feature points corresponding to each feature point in Φ form a set, 匹配特征点集合 ={d1,d2,…,d n};

[0131] Calculate the two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points diff i ,Right now

[0132] diff i =||c i -d i ||fori=1,2,3,…,n

[0133] Among them, ||c i -d i || means for feature point c i , d i The pixel coordinates of the two images are used to find the Euclidean space distance;

[0134] The difference in pixel coordinates of feature points on the statistical tracking is diffi i The number num that is less than a given second threshold;

[0135] And, r1=num / n is calculated, where r1 represents the ratio of feature points with insignificant changes to the total number of feature points.

[0136] The detection device of this embodiment uses the optical flow method to directly compare the image data, and avoids inter-frame pose calculation based on optical flow and feature point matching, thereby avoiding the introduction of errors caused by the visual odometer and ensuring the reliability of slip detection.

[0137] In addition, an embodiment of the present invention also provides a mobile robot, which includes: a detection device, a storage device and a camera, wherein the camera and the detection device are both connected to the storage device, and the detection device executes the steps of the method for detecting in-situ slipping of a visually navigated mobile robot based on optical flow as described in any of the above embodiments.

[0138] An embodiment of the present invention further provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for detecting in-situ slipping of a visually navigated mobile robot based on optical flow as described in any of the above embodiments are implemented.

[0139] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0141] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0142] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0143] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0144] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A method for detecting slippage of a mobile robot based on visual navigation using optical flow, characterized in that: include: A100, obtaining feature points of the current frame image of the mobile robot, and when the number of feature points of the current frame image meets the requirement, storing the current frame image and the feature points of the current frame image and the physical odometer value corresponding to the current frame image in a queue; the current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm; A200, when there are at least two frames of images in the queue that meet the timestamp requirements, an optical flow tracking algorithm is used to obtain the difference in pixel coordinates of corresponding feature points between the current frame image and the adjacent frame image, and according to the number of pixel coordinate differences, a first ratio value r1 of feature points with insignificant pixel changes to the number of all feature points is obtained, A300, when r1 is greater than a specified value, obtaining a second ratio value r2 between the current frame image and the first frame image in the queue; S400. When r2 is greater than a specified value, determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold. If so, determine that the current mobile robot is in a slipping state.

2. The method according to claim 1, characterized in that Before the A100, the method further comprises: Determine whether the brightness of the current frame image belongs to the image range of the reliability calculation of the optical flow tracking algorithm. If so, execute step A100, otherwise discard it; and / or, The A100 includes: After acquiring the feature points of the current frame image of the mobile robot, determining whether the number of the feature points is greater than a first threshold, if so, determining that the requirement is met, otherwise, discarding; and / or, After A100, the method further includes: If there is only the current frame image in the queue, end this detection; The timestamp requirement of the queue is less than or equal to 4 to 7 seconds.

3. The method according to claim 1, characterized in that The A200 includes: Select the feature point set Ω of the last frame image in the queue except the current frame image 最后一帧 , that is, Ω 最后一帧 ={c1,c2,…,c n }, n is the number of feature points in the last frame image; For each feature point of the last frame image, the optical flow tracking algorithm is used to find the feature point with Ω in the current frame. 最后一帧 The feature points corresponding to each feature point in form a set, Φ 匹配特征点集合 ={d1,d2,…,d n }; Calculate the two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points diff i ,Right now diffi i =‖c i -d i ‖ for i=1,2,3,…,n Among them, ‖c i -d i ‖ means for feature point c i , d i The Euclidean space distance between the pixel coordinates of the two images is calculated; i is a natural number less than n; Statistical tracking of the difference in pixel coordinates of feature points i The number num that is less than a given second threshold; And, r1=num / n is calculated, where r1 represents the ratio of feature points with insignificant changes to the total number of feature points.

4. The method according to claim 1, characterized in that: After A300, the method further includes: If r1 is less than or equal to the specified value, the queue is cleared, and the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image are stored in the cleared queue.

5. The method according to claim 1, characterized in that After A400, the method further includes: If r2 is less than or equal to the specified value, the first frame image in the queue is discarded and step A300 is repeated.

6. The method according to claim 2, characterized in that Determine whether the brightness of the current frame image is within the image range of the optical flow tracking algorithm reliability calculation, including: Calculate the average value of all pixels in the current frame image. When the average value is less than the pre-given sixth threshold, it is determined that the current frame image is too dark; when the average value is greater than the pre-given seventh threshold, it is determined that the current frame image is too bright, and both are not within the image range for reliability calculation of the optical flow tracking algorithm.

7. A detection device for detecting slippage of a mobile robot, characterized in that: include: A feature point processing module is used to obtain feature points of the current frame image of the mobile robot, and when the number of feature points of the current frame image meets the requirement, the current frame image and the feature points of the current frame image and the physical odometer value corresponding to the current frame image are stored in a queue; The current frame image is an image that satisfies the reliability calculation of the optical flow tracking algorithm; The first pixel change judgment module is used to use the optical flow tracking algorithm to obtain the difference in pixel coordinates of the corresponding feature points of the current frame image and the adjacent frame image when there are at least two frames of images that meet the timestamp requirements in the queue, and to obtain a first ratio value r1 of the number of feature points with insignificant pixel changes to all feature points according to the number of pixel coordinate differences. The second pixel change judgment module is used to obtain a second ratio value r2 between the current frame image and the first frame image in the queue when r1 is greater than a specified value; The slip detection module is used to determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold when r2 is greater than a specified value. If so, it is determined that the current mobile robot is in a slip state.

8. The detection device according to claim 7, characterized in that: The first pixel change judgment module is specifically used to: Select the feature point set Ω of the last frame image in the queue except the current frame image 最后一帧 , that is, Ω 最后一帧 ={c1,c2,…,c n }, n is the number of feature points in the last frame image; For each feature point of the last frame image, the optical flow tracking algorithm is used to find the feature point with Ω in the current frame. 最后一帧 The feature points corresponding to each feature point in form a set, Φ 匹配特征点集合 ={d1,d2,…,d n }; Calculate the two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points diff i ,Right now diff i =‖c i -d i ‖ for i=1,2,3,…,n Among them, ‖c i -d i ‖ means for feature point c i , d i The pixel coordinates of the two images are used to find the Euclidean space distance; Statistical tracking of the difference in pixel coordinates of feature points i The number num is smaller than the given second threshold value; i is a natural number smaller than n; And, r1=num / n is calculated, where r1 represents the ratio of feature points with insignificant changes to the total number of feature points.

9. A mobile robot, characterized in that: include: A detection device, a storage device and a camera, wherein the camera and the detection device are both connected to the storage device, and the detection device executes the steps of the method for detecting slippage in situ of a visually navigated mobile robot based on optical flow as described in any one of claims 1 to 6 above.

10. A computer storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for detecting slippage in situ of a visually navigated mobile robot based on optical flow as described in any one of claims 1 to 6 are implemented.

Images