A visual navigation mobile robot slip detection method based on optical flow

Abstract

The application relates to a visual navigation mobile robot slip detection method based on optical flow, which comprises the following steps: acquiring feature points of a current frame image of a mobile robot, storing the current frame image, the feature points of the current frame image and a physical odometer value in a queue; acquiring the difference between the pixel coordinates of the corresponding feature points of the current frame image and adjacent frame images by using an optical flow tracking algorithm in the queue, and acquiring a first proportion value r1 of the feature points with no obvious pixel change in all feature points according to the number of the pixel coordinate difference; when the r1 is greater than a specified value, acquiring a second proportion value r2 of the current frame image and a first frame image in the queue; when the r2 is greater than a specified value, judging whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image is greater than a specified threshold value, and if yes, determining that the current mobile robot is in a slip state. The above method ensures the reliability of slip detection.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method for detecting slippage in place in a visually-guided mobile robot based on optical flow. Background Technology

[0002] Slippage is one of the most common anomalies encountered by automated guided vehicles (AGVs) during autonomous localization and navigation tasks. Typically, slippage occurs when the friction between the wheels and the ground is insufficient to maintain the vehicle's normal trajectory, resulting in sliding or deviation from the intended path. Slippage can be caused by a variety of factors, including but not limited to the following:

[0003] (1) Ground conditions: Slippery, oily, uneven or foreign objects on the ground may reduce the friction between the wheels and the ground, causing the AGV to slip.

[0004] (2) Wheel material and wear: The material, hardness, and wear of the wheel will affect its contact with the ground and the friction. For example, severely worn wheels or wheels made of materials unsuitable for the current ground conditions may cause slippage.

[0005] (3) Load and center of gravity: If the load carried by the AGV is too heavy or unevenly distributed, or if the center of gravity of the vehicle is shifted, it may affect the grip of the wheels and driving stability, thus causing slippage.

[0006] (4) Drive system problems: Insufficient power of the drive motor, transmission system failure or improper control algorithm may cause the AGV to lack power during operation, which may lead to slippage.

[0007] (5) Speed ​​control: If the AGV travels too fast, especially when turning or encountering obstacles, improper speed control may also cause it to slip.

[0008] Among slippage phenomena, the most serious is stationary slippage of the mobile robot, where the robot's wheels spin freely without the AGV actually moving. Typically, this is caused by the AGV colliding with rigid obstacles such as walls or equipment, or by the AGV's drive or driven wheels getting stuck in depressions in the ground or equipment. When stationary slippage occurs, the AGV wheels continuously rub against the ground, causing prolonged overload of the drive motor's operating current, potentially leading to overheating and other malfunctions. Simultaneously, both the wheels and the ground experience varying degrees of wear, which can also cause the mobile robot to lose its positioning. Therefore, compared to general mobile robot slippage problems (such as slipping while moving or drifting), the detection and alarm for stationary slippage is particularly important.

[0009] Typically, existing methods for detecting slippage anomalies involve collecting data from wheel encoders (physical odometry), inertial odometry (IMU), and visual odometry (optical flow odometry). By comparing the deviations between these odometry methods, it is determined whether slippage has occurred. These methods are essentially based on the same principle: when the mobile robot is not slipping, the amount of movement recorded by the physical odometry, inertial odometry, and visual odometry should be relatively close. However, when slippage occurs, due to the relative sliding of the wheels with respect to the ground or the wheels spinning freely, the amount of movement recorded by the physical odometry, inertial odometry, and visual odometry will deviate. Therefore, existing visual positioning slippage detection methods all use the above-mentioned deviations as the basis for determining whether slippage has occurred. However, this type of solution faces a fatal problem: the optical flow method has a fixed defect: (1) It is sensitive to changes in illumination: the optical flow method is based on the assumption of constant brightness, so when the illumination conditions change, the accuracy of the algorithm will be affected. (2) Optical flow visual odometry has problems with accuracy and robustness, scale drift, and limited tracking performance. In some scenarios (such as areas with low local luminance resolution), the tracking performance of optical flow may decrease or even lose tracking.

[0010] When optical flow visual odometry has large errors or mistakes, the above-mentioned deviation-based slippage detection method has a great risk of false or missed slippage detection. Summary of the Invention

[0011] (a) Technical problems to be solved

[0012] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a method for detecting slippage in place of a visual navigation mobile robot based on optical flow, which improves the reliability of the optical flow method for slippage detection and avoids the error caused by the introduction of visual odometry in the prior art, so as to effectively ensure the reliability of slippage detection.

[0013] (II) Technical Solution

[0014] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0015] In a first aspect, embodiments of the present invention provide a method for detecting slippage in place in a visually-guided mobile robot based on optical flow, comprising:

[0016] A100: Obtain the feature points of the current frame image of the mobile robot. When the number of feature points in the current frame image meets the requirements, store the current frame image, the feature points of the current frame image, and the physical odometry value corresponding to the current frame image in the queue. The current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm.

[0017] A200. When there are at least two frames in the queue that meet the timestamp requirements, the optical flow tracing algorithm is used to obtain the difference in pixel coordinates of corresponding feature points between the current frame image and the adjacent frame image. Based on the number of differences in pixel coordinates, the first proportion value r1 of feature points with insignificant pixel changes among all feature points is obtained.

[0018] A300: When r1 is greater than a specified value, obtain the second ratio value r2 between the current frame image and the first frame image in the queue;

[0019] S400. When r2 is greater than a specified value, determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold. If so, determine that the current mobile robot is in a slipping state.

[0020] Optionally, prior to A100, the method further includes:

[0021] Determine whether the brightness of the current frame image falls within the image range calculated by the optical flow tracing algorithm. If so, proceed to step A100; otherwise, discard the image.

[0022] And / or,

[0023] The A100 includes:

[0024] After obtaining the feature points of the current frame image of the mobile robot, determine whether the number of the feature points is greater than a first threshold. If so, it is determined that the requirement is met; otherwise, it is discarded.

[0025] And / or,

[0026] Following A100, the method further includes:

[0027] If the queue contains only the current frame image, the current detection ends.

[0028] The timestamp requirement for the queue is less than or equal to 4 to 7 seconds. That is, in this embodiment, it is necessary to ensure that the image frames in the queue are all image frames from a recent time. If the difference between the timestamps of the first and last frames in the queue is greater than the above value, the first frame in the queue is removed, and so on, until the timestamp of the first frame in the queue meets the above requirement.

[0029] Optionally, the A200 includes:

[0030] Select the feature point set Ω from the last frame image in the queue, excluding the current frame image. 最后一帧 , i.e. Ω 最后一帧 ={c1,c2,…,c n}, where n is the number of feature points in the last frame of the image;

[0031] For each feature point in the last frame, an optical flow tracing algorithm is used to find the feature points in the current frame that correspond to Ω. 最后一帧 Each feature point in the set forms a set of feature points, Φ. 匹配特征点集合 ={d1,d2,…,d n};

[0032] Calculate two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points (diff) i ,Right now

[0033] diff i =||c i -d i || for i = 1, 2, 3, ..., n

[0034] Among them, ||c i -d i || indicates that for feature point c i d i Calculate the Euclidean distance between the pixel coordinates of the two images;

[0035] The difference in pixel coordinates of feature points in statistical tracking (diff) i The number of items less than the given second threshold, num;

[0036] Also, calculate r1 = num / n, where r1 represents the proportion of feature points that do not change significantly out of the total number of feature points.

[0037] Optionally, after A300, the method further includes:

[0038] If r1 is less than or equal to the specified value, the queue is cleared, and the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image are stored in the cleared queue.

[0039] Optionally, after A400, the method further includes:

[0040] If r2 is less than or equal to the specified value, the first frame image in the queue is removed, and the steps of A300 are repeated.

[0041] Optionally, determining whether the brightness of the current frame image falls within the image range calculated by the optical flow tracing algorithm's reliability includes:

[0042] Calculate the average value of all pixels in the current frame image. If the average value is less than a pre-given sixth threshold, the current frame image is determined to be too dark; if the average value is greater than a pre-given seventh threshold, the current frame image is determined to be too bright. Neither of these values ​​falls within the image range for the reliability calculation of the optical flow tracing algorithm.

[0043] Secondly, embodiments of the present invention also provide a detection device for detecting slippage in a mobile robot, comprising:

[0044] The feature point processing module is used to acquire the feature points of the current frame image of the mobile robot. When the number of feature points in the current frame image meets the requirements, the current frame image, the feature points of the current frame image, and the physical odometry value corresponding to the current frame image are stored in the queue. The current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm.

[0045] The first pixel change judgment module is used to obtain the difference in pixel coordinates of corresponding feature points between the current frame image and the adjacent frame image when there are at least two frames of images in the queue that meet the timestamp requirements, using an optical flow tracing algorithm. It also determines the first proportion value r1 of feature points with insignificant pixel changes to the total number of feature points based on the number of differences in pixel coordinates.

[0046] The second pixel change judgment module is used to obtain the second ratio value r2 between the current frame image and the first frame image in the queue when r1 is greater than a specified value;

[0047] The slip detection module is used to determine whether the difference between the physical odometry value corresponding to the current frame image and the physical odometry value corresponding to the first frame image in the queue is greater than a specified threshold when r2 is greater than a specified value. If so, it is determined that the current mobile robot is in a slip state.

[0048] Optionally, the first pixel change determination module is specifically used for:

[0049] Select the feature point set Ω from the last frame image in the queue, excluding the current frame image. 最后一帧 , i.e. Ω 最后一帧 ={c1,c2,…,c n}, where n is the number of feature points in the last frame of the image;

[0050] For each feature point in the last frame, an optical flow tracing algorithm is used to find the feature points in the current frame that correspond to Ω. 最后一帧 Each feature point in the set forms a set of feature points, Φ. 匹配特征点集合 ={d1,d2,…,d n};

[0051] Calculate two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points (diff) i ,Right now

[0052] diff i =||c i -d i || for i = 1, 2, 3, ..., n

[0053] Among them, ||c i -d i || indicates that for feature point c i d i Calculate the Euclidean distance between the pixel coordinates of the two images;

[0054] The difference in pixel coordinates of feature points in statistical tracking (diff) i The number of items less than the given second threshold, num;

[0055] Also, calculate r1 = num / n, where r1 represents the proportion of feature points that do not change significantly out of the total number of feature points.

[0056] Thirdly, embodiments of the present invention also provide a mobile robot, including: a detection device, a storage device, and a camera, wherein the camera and the detection device are both connected to the storage device, and the detection device performs the steps of the optical flow-based visual navigation mobile robot slipping detection method described in any of the first aspects above.

[0057] Fourthly, embodiments of the present invention also provide a computer storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the optical flow-based visual navigation mobile robot slip-in-place detection method as described in any of the first aspects.

[0058] (III) Beneficial Effects

[0059] The method of this invention, targeting the most severe slippage situation of stationary slippage, makes full use of the characteristic that the visual image will remain unchanged when slipping in place. It uses optical flow to directly compare the image data, avoiding inter-frame pose calculation based on optical flow and feature point matching. This avoids the error caused by visual odometry and thus ensures the reliability of slippage detection. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating a method for detecting slippage in place in a visually-guided mobile robot based on optical flow, according to an embodiment of the present invention.

[0061] Figure 2 A flowchart illustrating a method for detecting slippage in place in a visually guided mobile robot based on optical flow, as provided in another embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of a detection device provided in an embodiment of the present invention. Detailed Implementation

[0063] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] In this embodiment of the invention, in order to improve the reliability of optical flow method for slip detection, this invention proposes an optical flow-based visual navigation mobile robot slip detection method. For the most serious slip situation of slipping in place, this invention makes full use of the characteristic that the visual image will remain unchanged when slipping in place, and uses optical flow method to directly compare on the image data, avoiding inter-frame pose calculation based on optical flow and feature point matching. This can avoid introducing errors caused by visual odometry, thereby ensuring the reliability of slip detection.

[0065] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0066] Example 1

[0067] like Figure 1 As shown, Figure 1 The diagram shows a flowchart of a visual navigation mobile robot slip detection method based on optical flow according to an embodiment of the present invention. The main body executing the method in this embodiment is the detection device of the controller on the mobile robot. The mobile robot is equipped with an image acquisition device such as a camera, and the images acquired by the camera are stored in a storage device. The detection device can acquire the images acquired by the camera in real time. Specifically, the detection device can perform the following steps.

[0068] A100: Obtain the feature points of the current frame image of the mobile robot. When the number of feature points in the current frame image meets the requirements, store the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image in the queue. The current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm.

[0069] For example, before step A100, it can be determined whether the brightness of the current frame image falls within the image range calculated for the reliability of the optical flow tracing algorithm. If so, step A100 is executed; otherwise, it is discarded. For instance, the average value of all pixels in the current frame image is calculated. If the average value is less than a pre-defined sixth threshold, the current frame image is determined to be too dark; if the average value is greater than a pre-defined seventh threshold, the current frame image is determined to be too bright. Neither of these conditions falls within the image range calculated for the reliability of the optical flow tracing algorithm.

[0070] Furthermore, in A100, after obtaining the feature points of the current frame image of the mobile robot, it is determined whether the number of feature points is greater than a first threshold. If so, it is determined that the requirement is met; otherwise, it is discarded.

[0071] In practical applications, after A100, if the queue contains only the current frame image, the current detection ends; in this embodiment, the queue timestamp requirement is less than or equal to 4 to 7 seconds.

[0072] In this embodiment, it is necessary to ensure that all image frames in the queue are image frames from a recent time. If the difference between the timestamps of the first and last frames in the queue is greater than the above value, the first frame in the queue is removed, and so on, until the timestamp of the first frame in the queue meets the above requirements.

[0073] A200. When there are at least two frames of images in the queue that meet the timestamp requirements, the optical flow tracing algorithm is used to obtain the difference in pixel coordinates of the corresponding feature points between the current frame image and the adjacent frame image, and based on the number of differences in pixel coordinates, the first proportion value r1 of the number of feature points with insignificant pixel changes to the total number of feature points is obtained.

[0074] In this embodiment, the optical flow tracing algorithm is an existing algorithm, which is used to obtain the difference in pixel coordinates of corresponding feature points between two frames of images. In this step, the optical flow tracing algorithm is used to process the current frame image and the last frame image in the queue other than the current frame image, and obtain the difference in pixel coordinates of corresponding feature points between the two frames of images.

[0075] For example, in this embodiment, the feature point set Ω of the last frame image excluding the current frame image in the queue can be selected. 最后一帧 , i.e. Ω 最后一帧 ={c1,c2,…,c n}, where n is the number of feature points in the last frame of the image;

[0076] For each feature point in the last frame, an optical flow tracing algorithm is used to find the feature points in the current frame that correspond to Ω. 最后一帧 Each feature point in the set forms a set of feature points, Φ. 匹配特征点集合 ={d1,d2,…,d n};

[0077] Calculate two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points (diff) i ,Right now

[0078] diff i =||c i -d i || for i = 1, 2, 3, ..., n

[0079] Among them, ||c i -d i || indicates that for feature point c i b i Calculate the Euclidean distance between the pixel coordinates of the two images; i is a natural number, less than or equal to n.

[0080] The difference in pixel coordinates of feature points in statistical tracking (diff) i The number of items less than the given second threshold, num;

[0081] Also, calculate r1 = num / n, where r1 represents the proportion of feature points that do not change significantly out of the total number of feature points.

[0082] A300: When r1 is greater than a specified value, obtain the second ratio value r2 between the current frame image and the first frame image in the queue.

[0083] In this embodiment, if r1 is less than or equal to a specified value, the queue is cleared, and the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image are stored in the cleared queue.

[0084] It should be noted that the method for obtaining the second ratio value r2 in this embodiment is the same as the method for obtaining the first ratio value r1. Both use the optical flow tracing algorithm to obtain the difference in pixel coordinates of corresponding feature points between two frames of images (where r1 is for the current frame image and the last frame image excluding the current frame image, and r2 is for the current frame image and the first frame image in the queue). Then, based on the number of differences in pixel coordinates, the second ratio value r2 is obtained, which is the proportion of feature points with insignificant pixel changes to the total number of feature points.

[0085] S400. When r2 is greater than a specified value, determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold. If so, determine that the current mobile robot is in a slipping state.

[0086] Of course, if r2 is less than or equal to the specified value, the first frame image in the queue is removed, and the steps of A300 are repeated.

[0087] The method in this embodiment eliminates the influence of illumination, improves the accuracy of self-detection of the detection device, overcomes the defects of existing optical flow tracking algorithms such as scale drift and limited tracking effect, improves the reliability of detection, and avoids the problems of slippage false alarms or missed alarms.

[0088] Example 2

[0089] See Figure 2 , Figure 2A flowchart of a visual navigation mobile robot slip detection method based on optical flow provided in another embodiment of the present invention is shown. The method of this embodiment includes the following steps.

[0090] S1. The mobile robot's image acquisition device acquires and stores images and the current physical odometer value in real time; the detection device acquires the real-time acquired image and the corresponding physical odometer value.

[0091] S2. The detection device determines whether the brightness of the real-time acquired image is too dark or too bright. If it is too dark or too bright, the current calculation ends and the device waits for the next frame. If the image is neither too dark nor too bright, the process proceeds to step S3.

[0092] Specifically, the average value of all pixels in the current image is calculated. If the average value is less than a pre-defined sixth threshold, the current image is determined to be too dark; if the average value is greater than a pre-defined seventh threshold, the current image is defined as too bright.

[0093] In this embodiment, images that are too dark or too bright can easily lead to failure or errors in optical flow tracking. Therefore, if this situation is detected, the current calculation is terminated directly, and the system waits for the next frame. Otherwise, proceed to step S3.

[0094] S3. The detection device extracts feature points from the current image.

[0095] For example, the feature points in this step can refer to corner points or points of interest in the detected image. In this embodiment, feature points in the current image can be extracted using feature point extraction methods such as SIFT, ORB, SURF, and Harris.

[0096] S4. The detection device determines whether the number of feature points in the current image is greater than the first threshold. If so, it proceeds to step S5, which determines the queue for storing the current image; otherwise, the current calculation ends. That is, when the number of feature points in the current image is not greater than the first threshold, it lacks reliable texture information, making optical flow tracking impossible, thus ending the current calculation.

[0097] In this embodiment, the number of feature points is used to determine whether optical flow tracking can be used, thereby ensuring the reliability of the algorithm.

[0098] In step S3, it is necessary to ensure that enough traceable feature points are extracted. When this number is small, it indicates that there is a lack of reliable texture information in the current image, and there is a high risk in using optical flow tracking. Therefore, if this situation is detected, the current calculation is terminated directly, and the next frame image is waited for.

[0099] S5. Determine if the queue is empty. If the queue is empty, add the current image with enough traceable feature points to the queue and end the current calculation.

[0100] If the queue is not empty, step S6 can be executed.

[0101] In this embodiment, the queue is empty, which means that the system is running for the first time and there are no images for optical flow calculation before. Therefore, the image of the current frame and its associated information are added to the queue to end the current calculation and wait for the next frame of image data.

[0102] The image association information in this embodiment may include: the current frame image with a timestamp, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image.

[0103] In this embodiment, the queue stores all image frames for which the robot was determined not to have moved using the optical flow method; the timestamp difference of all images in the queue is within 3, 4, or 5 seconds. If the timestamp difference exceeds this range, the images can be deleted. In actual mobile robot slippage, there are only a few seconds, making detection time-sensitive; therefore, images with longer durations cannot detect slippage characteristics.

[0104] S6. Optical flow tracking calculation: Obtain the difference in pixel coordinates between corresponding feature points of two frames in the queue.

[0105] Specifically, the feature point set of the last frame in the selection queue, excluding the current frame, is selected, i.e., Ω. 最后一帧 ={c1,c2,…,c n}, where n is the number of feature points in the last frame. For each feature point, optical flow is used to find the corresponding feature points in the current frame that correspond to each feature point in Ω, forming a set Φ. 匹配特征点集合 ={d1,d2,…,d n}

[0106] Calculate the difference in pixel coordinates between all corresponding feature points in the two sets, i.e.

[0107] diff i =||c i -d i || for i = 1, 2, 3, ..., n

[0108] Among them, ||c i -d i || indicates that for feature point c i d i Calculate the Euclidean distance between the pixel coordinates of the two images.

[0109] S7. The number of pixel changes of feature points in the statistical tracking that are less than a given second threshold refers to the statistical diff mentioned above. i The number of values ​​less than the given second threshold is denoted as num.

[0110] And, calculate r = num / n, where r represents the proportion of feature points that do not change significantly out of the total number of feature points.

[0111] S8. Determine if r is less than a given third threshold. If r is less than a given threshold, it is determined that a considerable number of feature points in the two frames have moved significantly. Therefore, it can be assumed that the robot has definitely moved, and the queue is cleared.

[0112] Simultaneously, the current frame information is added to the queue (the information here includes the image of the current frame, feature points, and the physical odometer value corresponding to the current frame), and the calculation ends.

[0113] Conversely, if the positions of most feature points on the two frames of images do not change significantly, it can be determined that the robot has not moved, and the process proceeds to step S9.

[0114] S9. Traverse the queue from front to back, select the feature points of the first frame in the current queue, and perform optical flow tracking calculations on the current frame. The set of feature points of the first frame in the current queue is denoted as Ω′={c1,c2,…,c m}, where m is the number of feature points in the last frame. For each feature point, optical flow is used to find the corresponding feature points in the current frame to form a set Φ′={d1,d2,…,d m}

[0115] Calculate the difference in pixel coordinates between all corresponding feature points in the two sets, i.e.

[0116] diff i =||c i -d i || for i = 1, 2, 3, ..., m

[0117] Among them, ||c i -d i || indicates that for feature point c i d i Calculate the Euclidean distance between the pixel coordinates of the two images.

[0118] S10. The number of pixel changes of feature points in the statistical tracking that are less than a given second threshold refers to the statistical diff mentioned above. i The number of features less than a given second threshold is denoted as num. Calculate r = num / n, where r represents the proportion of feature points with insignificant changes to the total number of feature points.

[0119] S11. Determine whether r is less than a given third threshold. When r is less than a given threshold, it means that a considerable number of feature points have moved significantly from the first frame of the queue to the current frame. Therefore, it can be considered that the robot has definitely moved. So remove the first frame of the current queue from the queue and return to step S9. Otherwise, it means that the position of most feature points has not changed significantly from the first frame of the queue to the current frame. This means that the robot has not moved. Proceed to step S12.

[0120] S12. Compare the difference between the physical odometry of the first frame in the queue and the physical odometry of the current frame. If the difference is greater than the fourth threshold, it means that the physical odometry thinks that the robot has moved. S11 has ensured that there is no significant change between the first frame and the current frame. This means that although the robot's physical odometry has changed, the robot has not moved. Therefore, slippage is detected and reported.

[0121] In the above embodiments, for the most severe slippage situation of slipping in place, the characteristic that the visual image will remain unchanged when slipping in place is fully utilized. The optical flow method is used to directly compare the image data, avoiding inter-frame pose calculation based on optical flow and feature point matching. This can avoid introducing errors caused by visual odometry, thereby ensuring the reliability of slippage detection.

[0122] Example 3

[0123] like Figure 3 As shown, this embodiment also provides a detection device for slip detection of a mobile robot, which includes: a feature point processing module, a first pixel change judgment module, a second pixel change judgment module, and a slip detection module;

[0124] The feature point processing module is used to acquire the feature points of the current frame image of the mobile robot. When the number of feature points in the current frame image meets the requirements, the current frame image, the feature points of the current frame image, and the physical odometry value corresponding to the current frame image are stored in the queue. The current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm.

[0125] The first pixel change judgment module is used to obtain the difference in pixel coordinates of corresponding feature points between the current frame image and the adjacent frame image when there are at least two frames of images in the queue that meet the timestamp requirements, using an optical flow tracing algorithm. It also determines the first proportion value r1 of feature points with insignificant pixel changes to the total number of feature points based on the number of differences in pixel coordinates.

[0126] The second pixel change judgment module is used to obtain the second ratio value r2 between the current frame image and the first frame image in the queue when r1 is greater than a specified value;

[0127] The slip detection module is used to determine whether the difference between the physical odometry value corresponding to the current frame image and the physical odometry value corresponding to the first frame image in the queue is greater than a specified threshold when r2 is greater than a specified value. If so, it is determined that the current mobile robot is in a slip state.

[0128] For example, the first pixel change judgment module is specifically used for:

[0129] Select the feature point set Ω from the last frame image in the queue, excluding the current frame image. 最后一帧 , i.e. Ω 最后一帧 ={c1,c2,…,c n}, where n is the number of feature points in the last frame of the image;

[0130] For each feature point in the last frame, an optical flow tracing algorithm is used to find the feature points in the current frame that correspond to Ω. 最后一帧 Each feature point in the set forms a set of feature points, Φ. 匹配特征点集合 ={d1,d2,…,d n};

[0131] Calculate two sets Ω 最后一帧 and Φ 匹配特征点集合 The difference in pixel coordinates between all corresponding feature points (diff) i ,Right now

[0132] diff i =||c i -d i ||for i = 1, 2, 3, ..., n

[0133] Among them, ||c i -d i || indicates that for feature point c i d i Calculate the Euclidean distance between the pixel coordinates of the two images;

[0134] The difference in pixel coordinates of feature points in statistical tracking i The number of items less than the given second threshold, num;

[0135] Also, calculate r1 = num / n, where r1 represents the proportion of feature points that do not change significantly out of the total number of feature points.

[0136] The detection device in this embodiment uses optical flow to directly compare image data, avoiding inter-frame pose calculation based on optical flow and feature point matching. This avoids introducing errors caused by visual odometry, thereby ensuring the reliability of slip detection.

[0137] In addition, this invention also provides a mobile robot, which includes: a detection device, a storage device, and a camera. The camera and the detection device are both connected to the storage device. The detection device performs the steps of the optical flow-based visual navigation mobile robot slip-in-place detection method described in any of the above embodiments.

[0138] This invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the steps of the optical flow-based visual navigation mobile robot slip detection method as described in any of the above embodiments.

[0139] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0141] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0142] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0143] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0144] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A method for detecting slippage in place in a visually-guided mobile robot based on optical flow, characterized in that, include: A100: Obtain the feature points of the current frame image of the mobile robot. When the number of feature points in the current frame image meets the requirements, store the current frame image, the feature points of the current frame image, and the physical odometry value corresponding to the current frame image in a queue. The current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm. A200. When there are at least two frames in the queue that meet the timestamp requirements, the optical flow tracing algorithm is used to obtain the difference in pixel coordinates of corresponding feature points between the current frame image and the adjacent frame image. Based on the number of differences in pixel coordinates, the first proportion value r1 of feature points with insignificant pixel changes among all feature points is obtained. A300: When r1 is greater than a specified value, obtain the second ratio value r2 between the current frame image and the first frame image in the queue; the second ratio value r2 is the difference between the pixel coordinates of the corresponding feature points between the current frame image and the first frame image in the queue obtained by using the optical flow tracing algorithm, and obtain the proportion of feature points with insignificant pixel changes to the total number of feature points based on the number of differences in pixel coordinates. A400: When r2 is greater than a specified value, determine whether the difference between the physical odometer value corresponding to the current frame image and the physical odometer value corresponding to the first frame image in the queue is greater than a specified threshold. If so, determine that the current mobile robot is in a slipping state.

2. The method according to claim 1, characterized in that, Prior to A100, the method further includes: Determine whether the brightness of the current frame image falls within the image range calculated by the optical flow tracing algorithm. If so, proceed to step A100; otherwise, discard the image. And / or, The A100 includes: After obtaining the feature points of the current frame image of the mobile robot, determine whether the number of the feature points is greater than a first threshold. If so, it is determined that the requirement is met; otherwise, it is discarded. And / or, Following A100, the method further includes: If the queue contains only the current frame image, the current detection ends. The timestamp requirement for the queue is less than or equal to 7 seconds.

3. The method according to claim 1, characterized in that, The A200 includes: Get the feature point set of the last frame image in the queue excluding the current frame image. ,Right now n is the number of feature points in the last frame of the image; For each feature point in the last frame, an optical flow tracing algorithm is used to find the feature points in the current frame that are related to... Each feature point in the dataset forms a set of corresponding feature points. ; Calculate two sets and The difference in pixel coordinates between all corresponding feature points ,Right now ； in, This indicates that for feature points Calculate the Euclidean distance between the pixel coordinates of the two images; i is a natural number less than n; Calculate the difference in pixel coordinates of all corresponding feature points in two sets. The number of items less than the given second threshold, num; Also, calculate r1=num / n, where r1 represents the proportion of feature points with insignificant pixel changes to the total number of feature points.

4. The method according to claim 1, characterized in that, Following A300, the method further includes: If r1 is less than or equal to the specified value, the queue is cleared, and the current frame image, the feature points of the current frame image, and the physical odometer value corresponding to the current frame image are stored in the cleared queue.

5. The method according to claim 1, characterized in that, Following A400, the method further includes: If r2 is less than or equal to the specified value, the first frame image in the queue is removed, and the steps of A300 are repeated.

6. The method according to claim 2, characterized in that, Determine whether the brightness of the current frame image falls within the image range calculated for the reliability of the optical flow tracing algorithm, including: Calculate the average value of all pixels in the current frame image. If the average value is less than a pre-given sixth threshold, the current frame image is determined to be too dark; if the average value is greater than a pre-given seventh threshold, the current frame image is determined to be too bright. Neither of these values ​​falls within the image range for the reliability calculation of the optical flow tracing algorithm.

7. A detection device for detecting slippage in a mobile robot, characterized in that, include: The feature point processing module is used to acquire the feature points of the current frame image of the mobile robot. When the number of feature points in the current frame image meets the requirements, the current frame image, the feature points of the current frame image, and the physical odometry value corresponding to the current frame image are stored in a queue. The current frame image is an image that meets the reliability calculation of the optical flow tracking algorithm. The first pixel change judgment module is used to obtain the difference in pixel coordinates of corresponding feature points between the current frame image and the adjacent frame image when there are at least two frames of images in the queue that meet the timestamp requirements, using an optical flow tracing algorithm. It also determines the first proportion value r1 of feature points with insignificant pixel changes to the total number of feature points based on the number of differences in pixel coordinates. The second pixel change judgment module is used to obtain the second ratio value r2 between the current frame image and the first frame image in the queue when r1 is greater than a specified value. The second ratio value r2 is obtained by using the optical flow tracing algorithm to obtain the difference of pixel coordinates of corresponding feature points between the current frame image and the first frame image in the queue. Based on the number of differences in pixel coordinates, the proportion of feature points with insignificant pixel changes to the total number of feature points is obtained. The slip detection module is used to determine whether the difference between the physical odometry value corresponding to the current frame image and the physical odometry value corresponding to the first frame image in the queue is greater than a specified threshold when r2 is greater than a specified value. If so, it is determined that the current mobile robot is in a slip state.

8. The detection device according to claim 7, characterized in that, The first pixel change judgment module is specifically used for: Get the feature point set of the last frame image in the queue excluding the current frame image. ,Right now n is the number of feature points in the last frame of the image; For each feature point in the last frame, an optical flow tracing algorithm is used to find the feature points in the current frame that are related to... Each feature point in the dataset forms a set of corresponding feature points. ; Calculate two sets and The difference in pixel coordinates between all corresponding feature points ,Right now ； in, This indicates that for feature points Calculate the Euclidean distance between the pixel coordinates of the two images; Calculate the difference in pixel coordinates of all corresponding feature points in two sets. The number of values ​​less than the given second threshold, num; i is a natural number less than n; Also, calculate r1=num / n, where r1 represents the proportion of feature points with insignificant pixel changes to the total number of feature points.

9. A mobile robot, characterized in that, include: The system includes a detection device, a storage device, and a camera, wherein the camera and the detection device are both connected to the storage device, and the detection device performs the steps of the optical flow-based visual navigation mobile robot slippage detection method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the steps of the optical flow-based visual navigation mobile robot slip detection method as described in any one of claims 1 to 6.

Images