Method for Controlling Movement of a Mobile Robot in Case of Localization Failure

Abstract

A method for controlling the movement of a mobile robot in the event of a positioning failure is described. The method includes receiving an image from an image sensor of the mobile robot when a positioning failure occurs. The method includes dividing the received image into at least two vertical portions and determining whether one of the at least two vertical portions indicates a passable path. The method includes controlling the mobile robot to travel along the passable path indicated by one of the at least two vertical portions selected.

Description

Technical Field

[0001] The present invention particularly relates to, but is not limited to, an improved method for controlling the movement of a mobile robot in the event of a localization failure, and to a mobile robot configured to move along a preferred path in the event of a localization failure. Background Art

[0002] Simultaneous localization and mapping (SLAM) operations are of great significance in mobile robot technology. It refers to the process of gradually building a map of the local, i.e., the surrounding environment, at least from the robot sensor data, while maintaining an estimate of the current position / pose of the mobile robot at any given point in time. However, SLAM localization failures can occur for a variety of reasons. A localization failure is an event in which a mobile robot is unable to determine its current position / pose relative to the local environment map using sensor and / or image data from the mobile robot at a certain point in time and / or for a period of time. SLAM localization failures may occur due to outliers in the sensor data, incomplete map data, and / or distorted local environment maps constructed from inaccurate map data that may be caused by limitations of the SLAM algorithm. SLAM localization failures may also occur due to the presence of dynamic objects in the mobile robot's surrounding environment. Dynamic objects such as humans pose one of the greatest challenges to the operation of SLAM systems. Additionally, SLAM localization failures may further occur due to the presence of nearly identical repeating patterns in the local environment map data, such as nearly identical cubicles in an office space and nearly identical walls and doors in a corridor (both of which are common in office environments).

[0003] One way to address the SLAM localization failure problem is to control the mobile robot to stop and rotate when a localization failure occurs and use the robot sensor / image data obtained during the rotation to re-locate the robot's position relative to the local environment map. However, using this method to attempt to recover from a localization failure has a low probability of success because it is attempted at the location where the localization failure has already occurred. This is especially true in office environments where the local environment map data has repeating patterns. This solution not only has a low probability of success but also significantly reduces the navigation speed of the mobile robot in the local environment.

[0004] Another way to address the SLAM localization failure problem is to construct all or part of a three-dimensional image from the two-dimensional image data obtained from the mobile robot's camera or image sensor so that the mobile robot can travel in a safe direction while avoiding obstacles. However, this is not a simple task computationally and typically requires the use of one or more deep neural networks. In addition to expensive computing resources, this solution is also time-consuming and requires training and tuning of one or more deep neural networks.

[0005] Another method to solve the problem of SLAM positioning failure is to adopt a heuristic approach and attempt to reconstruct the entire three-dimensional scene around the mobile robot. This generally involves making a judgment on the environment around the mobile robot based on a selected number of three-dimensional geometric environmental features extracted from two-dimensional image data. This solution is most suitable for static environments with precise and regular geometric characteristics. It is difficult to function in an environment with dynamic obstacles (such as humans).

[0006] CN112767373A discloses a method for a robot to avoid obstacles in a complex indoor scene based on a monocular camera. The monocular obstacle avoidance navigation network consists of an environmental perception stage and a control decision-making stage, specifically including a depth prediction module, a semantic mask module, a depth slicing module, a feature extraction guidance module, a reinforcement learning module, and data augmentation. The network takes a monocular RGB image as input. After obtaining the semantic depth map, a dynamic min-pooling operation is performed to obtain pseudo-laser data; then the pseudo-laser data is used as the state input of reinforcement learning to generate the final decision-making action of the robot. Here, the RGB image includes an image composed of 3 data arrays, and these data arrays define the "red", "green", and "blue" color components for each image pixel.

[0007] CN106092104B discloses an indoor robot navigation system. If the robot gets lost, the system will perform laser relocalization based on laser point cloud data and visual relocalization based on visual image data. According to the relocalization results of the laser and visual data, the system will determine whether there is a familiar area. When there is no familiar area, the robot is controlled to perform obstacle avoidance movement until a familiar area is determined.

[0008] US9,984,467B discloses a vehicle navigation system. When the system determines that the vehicle is lost, a set of subsequent images will be collected. The features in the images are associated with the features of the industrial facility map to form feature pairs. The vehicle position can be calculated and relocalized based on these feature pairs.

[0009] In summary, an improved method is needed to control the movement of a mobile robot in case of positioning failure and configure the mobile robot to move along a preferred path in case of positioning failure.

[0010] Object of the Invention

[0011] An object of the present invention is to alleviate or eliminate to some extent one or more problems related to the known methods for controlling the movement of a mobile robot in case of positioning failure.

[0012] The above object is achieved by the combination of features in the main claims; the dependent claims disclose further advantageous embodiments of the present invention.

[0013] Another object of the present invention is to provide a mobile robot configured to move along a preferred path in the event of positioning failure.

[0014] Those skilled in the art will derive other objects of the present invention from the following description. Therefore, the above object statements are not exhaustive and are only illustrative of some of the many objects of the present invention. Summary of the Invention

[0015] The present invention relates to a method for recovering from positioning failure of a mobile robot, particularly an indoor mobile robot. The present invention provides a navigation recovery algorithm when a mobile robot faces intermittent positioning failure. In the event of positioning failure, the algorithm for mobile robot navigation is particularly applicable to mobile robots with a monocular vision SLAM system, but not limited thereto. The monocular vision SLAM algorithm is an algorithm for a mobile robot with a SLAM system and a monocular vision or a monocular camera or an image sensor. The monocular vision SLAM system can be referred to as a vision SLAM system, including one or a set of algorithms that use camera observations (such as images) as the main data source for positioning determination and map construction. The vision SLAM system typically operates in two stages, namely: (i) a map construction stage, in which, when the mobile robot moves in the local environment, a map of the local environment is constructed based on real-time camera images; (ii) a positioning stage, including the mobile robot determining its current position in the local environment by comparing the last captured or latest image from the camera with the map constructed in the map construction stage, i.e., positioning.

[0016] In a first main aspect, the present invention provides a method for controlling the movement of a mobile robot in the event of positioning failure. The method includes, upon occurrence of positioning failure, receiving an image from an image sensor of the mobile robot. The method includes dividing the received image into at least two vertical portions and determining whether one of the at least two vertical portions indicates a passable path. The method includes controlling the mobile robot to travel along the passable path indicated by one of the at least two vertical portions selected.

[0017] One advantage of the present invention is the recognition that if the mobile robot is controlled to continue moving forward relative to its previous direction of travel, the probability of the mobile robot recovering from positioning failure increases.

[0018] Another advantage is that the method of the present invention does not require expensive computing resources.

[0019] In a second main aspect, the present invention provides a mobile robot comprising a memory storing machine-readable instructions and a controller for executing the machine-readable instructions such that when the controller executes the machine-readable instructions, it configures the mobile robot to perform the steps of the first main aspect of the present invention.

[0020] In a third main aspect, the present invention provides a non-transitory computer-readable medium storing machine-readable instructions which, when executed by a processor, cause a mobile robot to perform the steps of the first main aspect of the present invention.

[0021] The abstract of the present invention does not necessarily disclose all features necessary to define the invention; the invention may reside in sub-combinations of the disclosed features.

[0022] The features of the present invention have been outlined generally above so that the detailed description of the present invention that follows may be better understood. Other features and advantages of the present invention will be described hereinafter, which form the subject matter of the claims of the present invention. Those skilled in the art will understand that the disclosed concepts and specific embodiments may be readily used as a basis for modifying or designing other structures for the same purpose of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and further features of the present invention will become apparent from the following description of preferred embodiments, which are given by way of example only in conjunction with the accompanying drawings, in which:

[0024] Figure 1 is a schematic block diagram of the mobile robot of the present invention;

[0025] Figure 2 is a schematic block diagram of the software components of the controller of the mobile robot of the present invention;

[0026] Figure 3 is an image taken by the camera of the mobile robot in the forward direction when the positioning fails (the original image is in color);

[0027] Figure 4 is the same Figure 3 captured image, but segmented to reduce the tonal difference of the local object image;

[0028] Figure 5 is the same Figure 3-4 captured image, after segmentation, the image is converted into a binary image;

[0029] Figure 6 is the same Figure 3-5 captured image, divided into two equal-width vertical parts;

[0030] Figure 7 compares the global path adopted by the virtual robot with the actual path adopted by the mobile robot;

[0031] Figure 8 Show the derivation of control variables for controlling the movement of a mobile robot;

[0032] Figure 9 Schematically show the first part of the method of the present invention;

[0033] Figure 10 Show further steps of the method of the present invention;

[0034] Figure 11 Schematically show that the positioning failure recovery software component includes an alignment controller and a navigation controller;

[0035] Figure 12 Schematically show the function of the alignment controller;

[0036] Figure 13 Schematically show the function of the navigation controller; and

[0037] Figure 14 Show that the navigation controller includes two proportional controllers. Detailed Description of the Invention

[0038] The following description is only for illustrative purposes to describe the preferred embodiments and does not limit the combination of essential features for implementing the present invention.

[0039] As used herein, the phrase "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic related to that embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" that appears throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. Additionally, the various features described may be shown by some embodiments but not by others. Similarly, various requirements are described, and these requirements may be requirements of some embodiments but not others.

[0040] It should be understood that the elements shown in the figures can be implemented in various forms of hardware, software, or a combination thereof. These elements can be implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, a memory, and an input / output interface.

[0041] This specification illustrates the principles of the present invention. Accordingly, it should be understood that those skilled in the art will be able to design various arrangements that, although not explicitly described or shown herein, embody the principles of the present invention and are included within its spirit and scope.

[0042] In addition, this document describes the principles, aspects, and embodiments of the present invention, as well as specific examples thereof, and is intended to cover structural and functional equivalents thereof. Such equivalents also include currently known equivalents and equivalents developed in the future, that is, any elements developed that perform the same function, regardless of their structure.

[0043] Thus, for example, those skilled in the art will understand that the block diagrams presented herein represent conceptual diagrams of systems and devices embodying the principles of the present invention.

[0044] The functions of the various elements shown in the figures can be provided by using dedicated hardware as well as hardware capable of executing software in conjunction with appropriate software. When provided by a processor, these functions can be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. In addition, the explicit use of the terms "processor" or "controller" should not be construed as referring only to hardware capable of executing software and can implicitly include, but is not limited to, digital signal processor ("DSP") hardware, read-only memory ("ROM") for storing software, random access memory ("RAM"), and non-volatile memory.

[0045] In the claims, any element expressed as a means for performing a specific function is intended to cover any means for performing that function, including, for example, a) a combination of circuit elements that perform that function or b) any form of software, and thus includes firmware, microcode, etc., in combination with appropriate circuitry for executing the software to perform the function. The invention as defined by these claims lies in the functions provided by the various recited means being combined and brought together in the manner claimed. It is thus contemplated that any means that provides these functions is equivalent to the means shown herein.

[0046] Figure 1Schematic block diagram of components of a mobile robot 10 including an embodiment of the present invention. The mobile robot 10 has a memory 12 for storing one or more control algorithms, the one or more control algorithms including machine-readable instructions. The one or more control algorithms preferably include one or more SLAM algorithms. The mobile robot 10 has a processor / controller 14 that executes the machine-readable instructions to control the mobile robot 10 of the present invention. The mobile robot 10 has at least one image sensor or camera 16. In a preferred embodiment, the mobile robot 10 has a single monocular camera 16. An odometer 18 is provided to provide distance data indicating how far the mobile robot 10 has moved from, for example, a starting position. The mobile robot 10 may have some other sensors 20, for example, including a compass that provides azimuth data for the mobile robot 10 and an inertial measurement unit (IMU) that provides specific force and angular rate data. The robot 10 includes one or more motors 22 that are connected to a drive system 24 for moving the robot 10 in accordance with control signals from the controller 14. The drive system 24 may include any suitable drive system for a mobile robot, for example, a set of wheels, a set of rollers, a set of continuous tracks, or a set of legs.

[0047] Figure 2 Schematic block diagram of software components of the controller 14 for the mobile robot 10 according to the present invention. The controller 14 includes a control scheduler 26. The control scheduler 26 may include a finite state machine (FSM) or a behavior tree. One function of the control scheduler 26 is to call a localization failure recovery software component 30 when the mobile robot 10 suffers a localization failure. The function of the localization failure recovery software component 30 according to the present invention is described below. Otherwise, the control scheduler 26 controls the mobile robot 10 through a visual SLAM software component 28. A global planner 32 is used to determine a path from the starting position of the mobile robot 10 to a desired position in a local environment map. A local planner 34 is used to avoid local obstacles in the path traveled by the mobile robot 10 and to bring the mobile robot 10 close to the determined global path. A motion controller 36 provides motion control signals to the drive system 24 to drive the mobile robot 10 along the path between the starting position and the desired position.

[0048] It can be seen that the controller 14 includes a plurality of software components or functional blocks for performing various functions of controlling the mobile robot 10, including the movement of the mobile robot 10 along a path approaching the determined global path. The plurality of software components or functional blocks constituting the controller 14 are preferably implemented by machine-readable instructions stored in the memory 12, which are executed by the controller 14 in use. The machine-readable instructions may be stored in one or more memories 12 (e.g., random access memory (RAM), read-only memory (ROM), flash memory, magnetic memory, optical memory, etc.), suitable for storing one or more instruction sets (e.g., application software, firmware, operating system, applets, etc.), data (e.g., configuration parameters, operating parameters and / or thresholds, collected data, processed data, etc.), etc. One or more memories 12 may include processor-readable memories for use with one or more processors of the controller 14 including executable machine-readable instructions. The controller 14 may include one or more dedicated processors (e.g., application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU)).

[0049] The present invention particularly relates to a mobile robot 10 equipped with a monocular camera 16 and having a positioning failure recovery software component 30, which includes one or more algorithms to enable the mobile robot 10 to continue moving when the visual SLAM software component 28 recognizes a positioning failure event. The method of the present invention may include a heuristic robot navigation method designed for an indoor environment that is physically limited in terms of passable areas. However, it can be understood that the method of the present invention can be applied to mobile robots in any environment that is physically limited in terms of passable areas. The method of the present invention is particularly suitable for ground detection based on image segmentation using "Mean Shift" segmentation or other suitable image segmentation methods. "Mean Shift" segmentation includes a local homogenization technique that can be effectively used to suppress or mask tonal differences in local object images. The method of the present invention is configured to control the mobile robot 10 to move along the most likely passable path while avoiding static and dynamic obstacles in the absence of positioning, i.e., in the case of positioning failure. "Dynamic obstacles" refer to those obstacles that change their positions by themselves or otherwise.

[0050] A preferred method of controlling the mobile robot 10 according to the present invention includes: receiving an image from the camera 16, dividing the received image into at least two vertical parts, determining whether one of the at least two vertical parts indicates a passable path, and controlling the mobile robot 10 to travel along the passable path indicated by one of the at least two vertical parts (determined to indicate a passable path). This method is preferably invoked when the mobile robot 10 loses positioning, i.e., in the case of positioning failure.

[0051] In some embodiments, the method includes continuously attempting to recover localization while controlling the mobile robot to travel along a passable path according to a preferred method.

[0052] In some embodiments, the method includes periodically attempting to recover localization while controlling the mobile robot to travel along a passable path according to a preferred method.

[0053] Preferably, once the localization is recovered, the localization failure recovery is terminated and the motion control of the mobile robot 10 is returned to the visual SLAM software component 28.

[0054] Preferably, the image received from the camera 16 is an image taken in the traveling direction of the mobile robot 10 when the localization fails.

[0055] Preferably, the received image is divided into at least two vertical portions of equal width, preferably including a left vertical image portion and a right vertical image portion. A passable path can be selected by determining which of the left vertical image portion and the right vertical image portion indicates a passable path. In the case where each of the left vertical image portion and the right vertical image portion is determined to indicate a respective passable path, the method preferably includes controlling the mobile robot 10 to travel along a preferred one of the respective passable paths.

[0056] Preferably, the received image is converted into a binary image including two colors, where one of the two colors of pixels represents the ground or surface on which the mobile robot 10 is traveling, and the other of the two colors of pixels represents the background environment of the location where the mobile robot 10 is located. The pixels representing the ground or surface on which the mobile robot 10 is traveling represent one or more possible passable paths. Preferably, at least the lower portion of the binary image is divided into at least two vertical portions. Then, a preferred passable path is determined, which includes one of the at least two vertical portions having the largest number of pixels of the color indicating the ground or surface on which the mobile robot 10 is traveling.

[0057] Reference Figure 3 , Figure 3 is a photographed image (the original image is in color) taken by the camera 16 of the mobile robot 10 in the forward direction when the localization fails. For ease of reference, the photographed image is marked with a rectangular frame to highlight a portion of the ground, i.e., the ground or surface along which the mobile robot 10 is traveling. It can also be seen that a dynamic obstacle including a person is located on the left side of the image, blocking the line of sight of a portion of the ground in the forward movement direction of the mobile robot 10.

[0058] Compared with Figure 3 Figure 4Shows the same photographic image, but segmented to attenuate the tonal differences of the local object images. The segmentation in this example uses the "mean shift" segmentation method. The "mean shift" segmentation method and other similar image segmentation methods can delimit or extract the part of the image that represents the ground or floor. The "mean shift" segmentation method was first proposed in [1] Fukunaga, Keinosuke; Larry D. Hostetler: "The Estimation of the Gradient of a Density Function, with Applications in Pattern Recognition". IEEE Transactions on Information Theory. 21(1):32–40, Jan. 1975. The modern version of the "mean shift" segmentation method is disclosed in [2] D. Comanicu, P. Meer: "Mean shift: A robust approach toward feature space analysis". IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 5, May 2002. Any one of the public materials [1] or [2] can be used in the method of the present invention. In addition, since the method of the present invention focuses on delimiting or extracting the ground or floor on which the mobile robot 10 is traveling from the captured image, this factor can be used to accelerate the "mean shift" segmentation method because the method of the present invention only needs to separate the ground / floor from non-ground / floor objects. The acceleration of this "mean shift" segmentation method appears in [3] André M. Santana, Kelson R. T. Aires, Rodrigo M. S. Veras, Adelardo A. D. Medeiros, "An Approach for 2D Visual Occupancy Grid Map Using Monocular Vision", Electronic Notes in Theoretical Computer Science, vol. 281:175-191, 2011.

[0059] Figure 5 Shows the same captured image after segmentation, which has been converted into a binary image. The preferred binary colors used are "black", representing the ground or floor, and "white", representing non-ground objects. However, it can be understood that when converting the Figure 4 segmented image to Figure 5When generating a binary image, any two contrasting colors can be used. Thus, in a preferred embodiment, the black pixels of the binary image represent the ground or surface on which the mobile robot 10 is traveling.

[0060] In Figure 6 the Figure 5 binary grayscale image is preferably divided into two equal-width vertical portions, including a left vertical portion and a right vertical portion. As can be seen from Figure 6 the right vertical portion of the binary grayscale image has more black pixels. Thus, the right vertical portion of the image represents a more passable path along the ground or floor.

[0061] If P is taken as Figure 6 the total number of pixels in the binary grayscale image, then P S can be considered as the total number of pixels in the passable area, and P S includes the total number of pixels in the black portion of the image. In this example, the total number of pixels P = 600 pixel width × 800 pixel height = 480,000 pixels. The total number of pixels P S in the passable area = 231,151. The division of the number of pixels in the passable area between the left and right portions of the image can be represented as P L and P R respectively. In this example, P L = 102,671 pixels, while P R = 128,480 pixels. Thus, P R > P L . In the method of the present invention, the right side of the image in this example represents the preferred passable path of the mobile robot 10. It is not necessary to convert the entire captured image into a binary image. It is only necessary to extract the lower part of the image that contains all the segmented ground portions of the image and then divide it into at least two equal-width vertical portions. However, in practice, considering the low computational complexity of processing the captured image, converting the entire segmented image into a binary image may be equally effective, as discussed in Figures 3 to 6 .

[0062] In a preferred embodiment, a virtual robot is provided as a reference robot for the mobile robot 10. According to the sensor input from the sensor 20 of the mobile robot 10, the virtual robot is controlled to virtually travel along a global path.

[0063] Refer to Figure 7, this figure compares the global path 40 adopted by the virtual robot with the actual path 42 adopted by the mobile robot 10. It should be understood that the actual path 42 adopted by the mobile robot 10 may deviate from the global path 40 in response to control inputs from the local planner 34, which controls the mobile robot 10 to avoid local obstacles in the actual path 42 adopted by the mobile robot 10 while attempting to bring the mobile robot 10 closer to the determined global path 40. The arrow line 44 indicates the moving direction of the virtual robot relative to the global path 40 at a certain time point and at a certain position on the global path 40 (as shown by the star device 46), and the mobile robot 10 experiences a positioning failure on the actual path 42. The angle represents the orientation of the virtual robot relative to the global path 40 at the star device 46 (i.e., when the positioning failure occurs). The arrow line 48 indicates the moving direction of the mobile robot 10 relative to the actual path 42 when the positioning failure occurs, and the angle θ represents the orientation of the mobile robot 10 relative to the actual path 42 when the positioning failure occurs.

[0064] Figure 8 Shows the derivation of the control variables ω, v for controlling the movement of the mobile robot 10, where is the desired angular velocity of the mobile robot 10, and the desired linear velocity of the mobile robot 10. The camera 16 provides image data to the controller 14. The variables s, θ, come from other sensors 20, including the IMU, the odometer 18, and the distance traveled by the virtual robot along the global path 40 and its orientation relative to the global path 40 when the positioning failure occurs.

[0065] When a positioning failure occurs, the virtual robot will be at the point on the global path 40 closest to the position of the mobile robot 10 on the actual path 42. After the positioning failure occurs, the virtual robot is controlled to travel a distance along the global path 40 equal to the distance traveled by the mobile robot 10 along a selected traversable path, which now constitutes the actual path 42.

[0066] In Figure 8 :

[0067] s represents the path length of the global path 40. When a new positioning failure phase begins, s is set to zero. The time t is also set to zero. When a new positioning failure occurs, the pose of the virtual robot is determined at the closest point on the global path 40 where s = 0. The variable s saturates at the end of the global path.

[0068] θ represents the orientation of the mobile robot 10 relative to the actual path 42 when a new positioning failure occurs.

[0069] Indicates the orientation of the virtual robot relative to the global path 40 when a new positioning failure occurs.

[0070] They are respectively the angular velocity of moving the robot 10 on the actual path 42 and the linear velocity of moving the robot 10.

[0071] Figure 9 Schematically shows the first part of the method 50 of the present invention. The method 50 starts from the starting point 52, and in the judgment step 54, it is determined whether a (new) positioning event has occurred. If not, then the method 50 continues to normally control the movement of the mobile robot 10 under visual SLAM navigation in step 56. The method 50 may include periodically returning to the starting point 52 to recheck in the judgment step 54 whether a positioning failure has occurred, and / or when a new positioning failure event is detected by the controller 14, it may trigger a return to the starting point 52. However, if it is judged to be yes in the judgment step 54, then the method 50 proceeds to step 58 to update the mobile robot sensor input, including obtaining or receiving at least one image from the camera 16. The image data from the camera 16 is processed as described above regarding Figure 3-6 the description.

[0072] Figure 10 Schematically shows the further steps of the method 50 after step 58. Once the mobile robot sensor input is updated in step 58, it is determined in the judgment step 60 whether the mobile robot 10 is aligned with the virtual robot. The alignment can be determined by the formula , that is, when a positioning failure occurs, the difference between the orientation θ of the mobile robot 10 relative to the actual path 42 and the orientation of the virtual robot 10 relative to the global path 40 is equal to or almost zero, where in this example s is the length that the mobile robot 10 has traveled since t = 0, and t is the current physical time. If not, then the method 50 proceeds to step 62 to realign the mobile robot 10 with respect to the orientation of the virtual robot. Step 62 includes the alignment controller branch of the method 50, which will be explained below. If so, then the mobile robot 10 is controlled to move along the most likely passable area or path, that is, the preferred passable path is determined from processing the image data from the camera 16. Step 64 includes the navigation controller branch of the method 50, which will be explained below. After steps 62 and 64, the method 50 returns to the starting point 52 and makes a new judgment in the judgment step 54 on whether a positioning failure has occurred or still exists.

[0073] Figure 11Schematically showing that the positioning failure recovery software component 30 preferably includes an alignment controller 30A and a navigation controller 30B. It should be understood that each of the alignment controller 30A and the navigation controller 30B is also preferably implemented as a software component of the controller 14.

[0074] Figure 12 Some steps of the method 50 are shown in more detail, and the functions of the alignment controller 30A are shown more specifically. If at the determination step 60 it is determined that the mobile robot 10 is not aligned with the virtual robot, i.e., not equal to or not close to zero, then at the determination step 66, it is determined by the control scheduler 26 of the controller 14 whether to call the alignment controller 30A to realign the mobile robot 10 with respect to the virtual robot, as in step 62( Figure 10 ), or call the navigation controller 30B to move the mobile robot 10 along the most likely passable path / area, as in step 64( Figure 10 ). If it is determined that either of the two conditions exists, the determination at step 66 will be "yes", and the two conditions include (i) determining that the speed of the mobile robot 10 is lower than the minimum predetermined threshold speed or (ii) the orientation of the mobile robot 10 with respect to the virtual robot is 90° or greater If it is determined that neither of the two conditions (i) and (ii) exists, the determination at the determination step 66 will be "no", resulting in step 64, where the control scheduler 26 accordingly calls the navigation controller 30B to move the mobile robot 10 along the most likely passable path / area. In the case where the determination at the determination step 66 is "yes", the alignment controller 30A will issue control signals v, ω to make the linear speed of the mobile robot 10 zero, and control the angular velocity of the mobile robot 10 to align the mobile robot 10 with the virtual robot. In the case where the determination at the determination step 66 is "no", the navigation controller 30B will issue control signals v, ω to control the linear speed of the mobile robot 10 and the angular velocity to move the mobile robot 10 along the most likely passable path / area.

[0075] In some embodiments, the alignment controller 30A includes a proportional controller with saturation. The saturation includes a pre-specified or predetermined angular velocity value, which can prevent the mobile robot 10 from rotating too fast. As described above, when the alignment controller 30A is called, it controls the linear speed of the mobile robot 10 i.e., the mobile robot stops in the moving direction. The alignment controller 30A controls the angular velocity of the mobile robot 10 to align the mobile robot 10 with the virtual robot, preferably in the following manner:

[0076] Where: k1 is an adjustable control parameter;

[0077] ω max is the (positive) saturation value of the angular velocity control variable;

[0078] θ is the orientation of the mobile robot 10 relative to the actual path 42;

[0079] is the orientation of the virtual robot relative to the global path 40, where the variable s saturates at the end of the global path 40; and

[0080] (ω, v) is the control output of the mobile robot 10.

[0081] The alignment controller 30A aligns the mobile robot 10 relative to the virtual robot such that when the difference between the orientation θ of the mobile robot 10 and the orientation of the virtual robot 10 is large, the angular velocity of the mobile robot 10 (rotation) is large. The proportional controller multiplies the difference between the orientation of the virtual robot and the orientation of the mobile robot 10 by a constant coefficient, preferably by a constant positive coefficient.

[0082] Thus, it can be seen that the orientation of the virtual robot on the global path 40 is used as the input to the alignment controller 30A to control the alignment of the mobile robot 10 on the selected traversable path. It can also be seen that if the linear velocity of the mobile robot is lower than a predetermined, selected, or calculated threshold velocity v min and / or when the alignment of the mobile robot 10 relative to the virtual robot is at or greater than 90°, the alignment controller 30A is invoked.

[0083] Figure 13 Some steps of the method 50 are shown in more detail, and the function of the navigation controller 30B is shown more specifically. Assuming that in the determination step 60 it is determined that the mobile robot 10 is aligned with the virtual robot, the control scheduler 26 of the controller 14 will invoke the navigation controller 30B to attempt to move the mobile robot 10 along the preferred path.

[0084] Before the alignment controller 30A has aligned the mobile robot 10 relative to the virtual robot and before invoking the navigation controller 30B, it is determined in the determination step 70 whether the linear velocity of the mobile robot 10 is equal to zero. If the determination in the determination step 70 is "yes", the timer 72 is started. Whenever the alignment controller 30A realigns the mobile robot 10 with the virtual robot, and it is determined in the determination step 70 that the linear velocity of the mobile robot 10 When it is equal to 0, the timer 72 is reset. However, if the determination in the determination step 70 is "no", the timer 72 is reset in step 74, and the navigation controller 30B is called to control the movement of the mobile robot 10 along the preferred passable path.

[0085] If the timer 72 is started after the determination in the determination step 70 is "yes", it is determined in the determination step 76 that the timer is active, and then the navigation controller 30B is called to control the movement of the mobile robot 10 along the preferred passable path. In this context, "the timer is active" means the time period elapsed since the mobile robot 10 stopped moving, that is, since the linear speed of the mobile robot 10 became zero and the elapsed time period exceeds a predetermined time period. If the timer 72 has been started after the determination in the determination step 70 is "yes" and it is determined in the determination step 76 that the timer is inactive, an error condition 78 is returned.

[0086] As Figure 14 shown, in some embodiments, the navigation controller 30B includes two proportional controllers, the first proportional controller 80 controls the linear speed of the mobile robot 10 The second proportional controller 82 controls the angular velocity or rotational speed of the mobile robot 10 The first proportional controller 80 is preferably controlled according to the following:

[0087]

[0088] The second proportional controller 82 is preferably controlled according to the following:

[0089]

[0090] where k2, k3 are adjustable control parameters;

[0091] c>0 is a constant that can be adjusted by the user;

[0092] v max is the (positive) saturation value of the angular velocity control of the mobile robot 10;

[0093] P is the total number of pixels in the received image;

[0094] P S is the total number of pixels of the (preferred) passable path / area;

[0095] P L is the number of pixels of the passable area in the left vertical part of the received image;

[0096] P Ris the number of pixels of the passable area in the right vertical part of the received image; and

[0097] ω max is the (positive) saturation value of the angular velocity control of the mobile robot 10.

[0098] P S The larger the value of P, the faster the linear velocity control v of the mobile robot 10. In fact, the larger the passable path or passable area, the larger the linear velocity control v of the mobile robot 10. The larger the passable path or passable area on the right or left, the mobile robot 10 will be controlled to turn right or left respectively. The control value of ω is determined to turn the mobile robot 10 in the direction with less restriction in the passable path or area.

[0099] Various control parameters can be adjusted to improve the motion control of the mobile robot 10. In some embodiments, the adjustment of the control parameters may include the following:

[0100] First, the shift coefficient c is determined by ensuring that the mobile robot 10 stops before hitting the wall and appropriately setting other parameter values. The fixed value of v max can be used to determine the shift coefficient c;

[0101] Second, the proportional parameter k3 of the angular velocity control of the navigation controller 30B is determined by ensuring that the mobile robot 10 can avoid dynamic obstacles, such as a person walking in an open area, and appropriately setting other parameter values. This can use v = 0.5*v max with a fixed value and ω max with a fixed value to determine;

[0102] Third, the proportional parameter k2 of the speed control of the navigation controller 30B is determined by ensuring that the mobile robot 10 can avoid dynamic obstacles, such as a person walking in a physically restricted area (which is typical in an office environment). This can use v max with a fixed value and ω max with a fixed value to determine;

[0103] Fourth, the proportional parameter k1 of the alignment controller 30A is determined, which controls the rotation speed of the mobile robot 10 when performing azimuth alignment, ensuring that it is not too fast. This can use v = 0 and ω max with a fixed value to determine.

[0104] Therefore, it can be seen that when the mobile robot 10 is aligned with the virtual robot, the navigation controller 30B controls the linear velocity and angular velocity of the mobile robot 10 along the selected or preferred passable path When it is zero, timer 72 is started and the navigation controller 30B is repeatedly called until timer 72 expires and / or it is determined that the mobile robot 10 is determined to be traveling along a preferred traversable path.

[0105] In some embodiments, the number of pixels of the image portion indicating the determined preferred traversable path (normalized by the total number of pixels in the received image) is used to at least control the initial linear velocity of the mobile robot 10 along the determined preferred traversable path.

[0106] In some embodiments, the difference between the number of pixels indicating the traversable path in the left vertical portion and the number of pixels indicating the traversable path in the right vertical portion is used to at least control the initial angular velocity of the mobile robot 10 along the determined preferred traversable path.

[0107] The present invention also provides a mobile robot, including a memory storing machine-readable instructions and a controller for executing the machine-readable instructions, such that when the controller executes the machine-readable instructions, it configures the mobile robot to implement the steps of the method of the present invention.

[0108] The present invention also provides a non-transitory computer-readable medium storing machine-readable instructions, wherein when the machine-readable instructions are executed by a processor or a controller 14, they will configure the processor or the controller 14 to implement the above method of the present invention.

[0109] The above device can be implemented at least partially in software. Those skilled in the art will understand that the above device can be implemented at least partially using a general-purpose computer device or using a customized device.

[0110] Here, aspects of the methods and apparatuses described herein can be performed on any device including a communication system. The program aspects of the technology can be considered a "product" or "article of manufacture", typically in the form of executable code and / or associated data, carried or embodied in a machine-readable medium. "Storage" type media includes any or all of the memories of a mobile station, computer, processor, or similar device, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives, etc., which can provide storage for software programming at any time. All or part of the software can sometimes be communicated via the Internet or various other telecommunications networks. For example, such communication can cause the software to be loaded from one computer or processor to another. Accordingly, another type of media that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as used over a physical interface between local devices, via wired and optical landline networks, and via various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered media that carry software. As used herein, unless restricted to tangible non-transitory "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0111] Although the invention has been illustrated and described in detail in the drawings and foregoing description, it should be regarded as illustrative and not restrictive. It should be understood that only exemplary embodiments have been shown and described and that the scope of the invention is in no way limited thereby. It will be understood that any feature described herein can be used in any embodiment. The exemplary embodiments do not exclude each other or other embodiments not described herein. Accordingly, the invention also provides embodiments that include combinations of one or more of the above-described exemplary embodiments. Modifications and variations can be made to the invention without departing from the spirit and scope thereof, and thus only the limitations as set forth in the appended claims should be imposed.

[0112] In the foregoing description of the appended claims and the present invention, unless the context requires otherwise due to express language or necessary implication, the word "comprising" or variants such as "includes" or "containing" are used in an inclusive sense, i.e., specifying the presence of the stated features, but not excluding the presence or addition of further features in the various embodiments of the present invention.

[0113] It should be understood that if any prior art publication is mentioned herein, such reference does not constitute an admission that the publication forms part of the common general knowledge in the art.

Claims

1. A method for controlling the movement of a mobile robot in case of positioning failure, the method comprising: Receiving an image from an image sensor of the mobile robot when positioning failure occurs; Dividing the received image into at least two vertical parts; Determining whether one of the at least two vertical parts indicates a passable path; And Controlling the mobile robot to travel along the determined passable path, the determined passable path being the passable path indicated by one selected from the at least two vertical parts; Wherein the received image is converted into a binary image including two colors, wherein the pixels of one of the two colors indicate the floor or surface on which the mobile robot is traveling, and the pixels of the other of the two colors indicate the background environment, and the pixels indicating the floor or surface on which the mobile robot is traveling represent one or more possible passable paths; Dividing at least the lower half of the binary image into at least two vertical parts; Determining a preferred passable path, the preferred passable path being one of the at least two vertical parts of at least the lower half of the binary image that has the largest number of pixels of the color indicating the floor or surface on which the mobile robot is traveling; Wherein the number of pixels of the image part indicating the determined preferred passable path is normalized by the total number of pixels of the received image for at least controlling the initial linear speed of the mobile robot along the determined preferred passable path.

2. The method according to claim 1, wherein, The method includes: while controlling the mobile robot to travel along the passable path, continuously or periodically attempting to resume positioning.

3. The method according to claim 1, wherein, The method includes: continuously or periodically receiving further images from the image sensor of the mobile robot and repeating the steps of claim 1 for each received image.

4. The method according to claim 1, wherein The image received from the image sensor is an image in the traveling direction of the mobile robot when the positioning failure occurs.

5. The method according to claim 1, wherein, Dividing the received image into at least two vertical parts of equal width.

6. The method according to claim 1, wherein The method includes: Dividing the received image into a left vertical image part and a right vertical image part; and Determining that one of the left vertical image part and the right vertical image part indicates a passable path.

7. The method according to claim 6, wherein If it is determined that each of the left vertical image part and the right vertical image part indicates its respective passable path, controlling the mobile robot to travel along a preferred one of the respective passable paths.

8. The method according to claim 1, wherein Before dividing at least the lower half of the binary image into the at least two vertical parts, dividing the received image to determine the lower half of the received image, the lower half indicating the floor or surface on which the mobile robot is traveling along the way.

9. The method according to claim 1, wherein Converting the received image into a binary grayscale image, wherein the two colors include black and white, and the black pixels represent the floor or surface on which the mobile robot is traveling along the way.

10. The method according to claim 1, wherein The method includes: at least dividing the lower half of the binary image into a left vertical part and a right vertical part, and using the difference between the number of pixels indicating a passable path in the left vertical part and the number of pixels indicating a passable path in the right vertical part to at least control the initial angular velocity of the mobile robot along the determined preferred passable path.

11. The method according to claim 1, wherein, Define a global path for the mobile robot and provide a virtual robot as a reference robot for the mobile robot, the virtual robot being controlled to travel along the global path such that when the positioning fails, the virtual robot is at the point on the global path closest to the position of the mobile robot, and after the positioning fails, the virtual robot is controlled to travel a distance along the global path equal to the distance traveled by the mobile robot along the selected passable path.

12. The method according to claim 11, wherein the orientation of the virtual robot on the global path is used as an input to an alignment controller to control the alignment of the mobile robot on the selected passable path.

13. The method according to claim 11, wherein, If the linear velocity of the mobile robot is lower than a predetermined, selected or calculated threshold velocity value, and / or when the alignment of the mobile robot relative to the virtual robot is greater than 90°, then the alignment controller is invoked.

14. The method according to claim 12, wherein, When the mobile robot is aligned with the virtual robot, the navigation controller controls the linear velocity and the angular velocity of the mobile robot along the selected passable path.

15. The method according to claim 14, wherein, When the linear velocity of the mobile robot is zero, a timer is started and the navigation controller is repeatedly invoked until the timer expires and / or the mobile robot is determined to be moving.

16. The method according to claim 1, wherein, The controller of the mobile robot includes a global planning module and a local planning module, the global planning module being configured to determine a global path based on a pre-constructed map of the operating environment of the mobile robot, the global path being determined between the starting position of the mobile robot and a selected or desired stop position, the local planning module being configured to assist the mobile robot in avoiding obstacles when the mobile robot travels along or close to the global path.

17. The method according to claim 2, wherein, Once positioning is restored, the steps of claim 1 are terminated.

18. A mobile robot, comprising a memory storing machine-readable instructions and a controller for executing the machine-readable instructions such that when the controller executes the machine-readable instructions, it configures the mobile robot to perform the steps of any one of claims 1-17.

Images