Mobile robot control method, computer-implemented storage medium, and mobile robot

Abstract

A mobile robot control method, a computer-implemented storage medium, and a mobile robot, the method comprising: when the robot (11) is in a desired posture, obtaining a first image (S41) taken by a camera on the robot (11); when the robot (11) is in a current posture, obtaining a second image (S42) taken by a camera on the robot (11); extracting multiple pairs of matching feature points from the first image and the second image and projecting the extracted feature points onto a virtual unit sphere to obtain multiple projection feature points (S43), the center of the virtual unit sphere coincides with the optical center of the coordinates of the camera; obtaining an image invariant feature and a rotation vector feature based on the multiple projection feature points, and controlling the robot (11) to move according to the image invariant feature and the rotation vector feature until the robot (11) is in the desired posture (S44). Since the image invariant feature and the rotation vector feature are used to determine the desired posture of the robot (11), it is not necessary to know the target model in advance, and it is not necessary to estimate the posture of the robot (11) relative to the target.

Description

Technical Field

[0001] The present invention generally relates to a mobile robot, and more particularly to an image-based control method for a mobile robot without posture measurement and a mobile robot. Background Art

[0002] Ground mobile robots are being actively developed in the fields of logistics, search and rescue, surveillance, etc. to perform various repetitive and dangerous activities. The adjustment control of many ground mobile robots is mainly carried out in a Cartesian coordinate system. That is, the output of the system is metric coordinates (along the x-axis and y-axis) and angles measured in degrees or radians (around the z-axis). This requires that the measurement information collected by the sensor must be used for control calculations. However, commonly used distance sensors, such as LiDAR, RGB-D, and stereo cameras, are expensive and bulky.

[0003] The relative pose of the robot can be reconstructed using a monocular camera. However, pose reconstruction based on a monocular camera requires prior knowledge of the target model and the pose estimation response, which is not always available. In addition, the calculations required for pose estimation are extremely complex.

[0004] Therefore, it is necessary to provide an image-based mobile robot control method without pose measurement to overcome the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a robot assistant to solve the above-mentioned existing problems.

[0006] The present invention is implemented as follows: a computer-implemented method for controlling a mobile robot executed by one or more processors, the method comprising: when the robot is in a desired posture, obtaining a first image taken by a camera on the robot; when the robot is in a current posture, obtaining a second image taken by a camera on the robot; extracting multiple pairs of matching feature points from the first image and the second image and projecting the extracted feature points onto a virtual unit sphere to obtain multiple projected feature points, wherein the center of the virtual unit sphere coincides with the optical center of the coordinates of the camera; obtaining an image invariant feature and a rotation vector feature based on the multiple projected feature points, and controlling the movement of the robot according to the image invariant feature and the rotation vector feature until the robot is in the desired posture.

[0007] Furthermore, controlling the movement of the robot includes: controlling the robot to translate according to the image invariant feature; and controlling the robot to rotate according to the rotation vector feature.

[0008] Furthermore, controlling the robot to translate according to the image invariant feature includes: calculating a first angular velocity and a first linear velocity according to the image invariant feature and a preset control model; controlling the robot to translate according to the first angular velocity and the first linear velocity; determining whether the translation error is less than a first preset threshold; if the translation error is greater than or equal to the first preset threshold, returning to the step of acquiring the second image; if the translation error is less than the first preset threshold, controlling the robot to rotate according to the rotation vector feature.

[0009] Furthermore, controlling the rotation of the robot according to the rotation vector feature includes: calculating a second angular velocity and a second linear velocity based on the rotation vector feature and a control model; controlling the rotation of the robot according to the second angular velocity and the second linear velocity; determining whether a direction error of the robot after rotation is less than a second preset threshold; if the direction error of the robot is greater than or equal to the second preset threshold, acquiring a third image through the camera; extracting multiple pairs of matching feature points from the first image and the third image, and projecting the extracted feature points onto the unit sphere to obtain multiple projection feature points; and acquiring the rotation vector feature based on the multiple projection feature points extracted from the first image and the third image, and then returning to the step of calculating the second angular velocity and the second linear velocity based on the rotation vector feature and the control model.

[0010] Further, the image invariant feature includes one or more of the reciprocal of the distance between two of the projected feature points, the image moment and the area.

[0011] Furthermore, obtaining the image invariant feature based on the multiple projected feature points includes: obtaining at least two of the distance between two of the projected feature points, the image moment and the area; and calculating the average value of at least two of the distance between two of the projected feature points, the image moment and the area, and using the average value as the image invariant feature.

[0012] Further, obtaining the rotation vector feature based on the multiple projection feature points includes: determining the acceleration direction of the robot based on the multiple projection feature points; and taking the angle between the acceleration direction and an x-axis of a robot coordinate system as the rotation vector feature.

[0013] Furthermore, extracting multiple pairs of matching feature points from the first image and the second image includes: extracting a first number of original feature points from each of the first image and the second image using a scale-invariant feature transform; and obtaining a second number of pairs of matching feature points by comparing and matching the extracted original feature points.

[0014] Furthermore, the method also includes: controlling the robot to stop at a desired position with a preset posture; and using the image of the environment in front of the robot taken by the camera as the first image, wherein the environment in front of the robot includes at least three feature points.

[0015] The present invention also provides a non-temporary computer-readable storage medium storing one or more programs to be executed in a mobile robot. When the one or more programs are executed by one or more processors of the robot, the robot performs the following processing: when the robot is in a desired posture, obtaining a first image taken by a camera on the robot; when the robot is in a current posture, obtaining a second image taken by a camera on the robot; extracting multiple pairs of matching feature points from the first image and the second image and projecting the extracted feature points onto a virtual unit sphere to obtain multiple projected feature points, wherein the center of the virtual unit sphere coincides with the optical center of the coordinates of the camera; obtaining an image invariant feature and a rotation vector feature based on the multiple projected feature points, and controlling the movement of the robot according to the image invariant feature and the rotation vector feature until the robot is in the desired posture.

[0016] Furthermore, the image invariant feature includes one or more of the reciprocal of the distance between two of the projection feature points, the image moment and the area; obtaining the rotation vector feature based on the multiple projection feature points includes: determining the acceleration direction of the robot based on the multiple projection feature points; and taking the angle between the acceleration direction and the x-axis of a robot coordinate system as the rotation vector feature.

[0017] The present invention also provides a mobile robot, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include: instructions for acquiring a first image taken by a camera on the robot when the robot is in a desired posture; instructions for acquiring a second image taken by a camera on the robot when the robot is in a current posture; instructions for extracting multiple pairs of matching feature points from the first image and the second image and projecting the extracted feature points onto a virtual unit sphere to obtain multiple projected feature points, wherein the center of the virtual unit sphere coincides with the optical center of the coordinates of the camera; instructions for acquiring an image invariant feature and a rotation vector feature based on the multiple projected feature points and controlling the movement of the robot according to the image invariant feature and the rotation vector feature until the robot is in the desired posture.

[0018] Furthermore, controlling the movement of the robot includes: controlling the robot to translate according to the image invariant feature; and controlling the robot to rotate according to the rotation vector feature.

[0019] Furthermore, controlling the robot to translate according to the image invariant feature includes: calculating a first angular velocity and a first linear velocity according to the image invariant feature and a preset control model; controlling the robot to translate according to the first angular velocity and the first linear velocity; determining whether the translation error is less than a first preset threshold; if the translation error is greater than or equal to the first preset threshold, returning to the step of acquiring the second image; if the translation error is less than the first preset threshold, controlling the robot to rotate according to the rotation vector feature. Furthermore, controlling the rotation of the robot according to the rotation vector feature includes: calculating a second angular velocity and a second linear velocity based on the rotation vector feature and a control model; controlling the rotation of the robot according to the second angular velocity and the second linear velocity; determining whether a direction error of the robot after rotation is less than a second preset threshold; if the direction error of the robot is greater than or equal to the second preset threshold, acquiring a third image through the camera; extracting multiple pairs of matching feature points from the first image and the third image, and projecting the extracted feature points onto the unit sphere to obtain multiple projection feature points; and acquiring the rotation vector feature based on the multiple projection feature points extracted from the first image and the third image, and then returning to the step of calculating the second angular velocity and the second linear velocity based on the rotation vector feature and the control model.

[0020] Further, the image invariant feature includes one or more of the reciprocal of the distance between two of the projected feature points, the image moment and the area.

[0021] Furthermore, obtaining the image invariant feature based on the multiple projected feature points includes: obtaining at least two of the distance between two of the projected feature points, the image moment and the area; and calculating the average value of at least two of the distance between two of the projected feature points, the image moment and the area, and using the average value as the image invariant feature.

[0022] Further, obtaining the rotation vector feature based on the multiple projection feature points includes: determining the acceleration direction of the robot based on the multiple projection feature points; and taking the angle between the acceleration direction and an x-axis of a robot coordinate system as the rotation vector feature.

[0023] Furthermore, extracting multiple pairs of matching feature points from the first image and the second image includes: extracting a first number of original feature points from each of the first image and the second image using a scale-invariant feature transform; and obtaining a second number of pairs of matching feature points by comparing and matching the extracted original feature points.

[0024] Furthermore, it also includes: controlling the robot to stop at a desired position with a preset posture; and using the image of the environment in front of the robot taken by the camera as the first image, wherein the environment in front of the robot includes at least three feature points.

[0025] The technical effect of the present invention relative to the prior art is that the mobile robot control method of the present invention only requires one camera, rather than relying on the use of traditional expensive and bulky sensors. On the other hand, since the desired posture of the robot is determined using image invariant features and rotation vector features, neither the target model needs to be known in advance nor the posture of the robot relative to the target needs to be estimated. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 is a schematic block diagram of a robot according to one embodiment.

[0028] Figure 2 is a schematic block diagram of a robot according to another embodiment.

[0029] Figure 3 is a schematic isometric view of a wheeled robot according to one embodiment.

[0030] Figure 4 A scene showing that the robot can be controlled to dock at a charging station.

[0031] Figure 5 is a schematic flow chart of a mobile robot control method according to an embodiment.

[0032] Figure 6 is a schematic flowchart of a mobile robot control method according to another embodiment.

[0033] Figure 7 is a schematic diagram of the virtual unit sphere onto which the feature points are projected.

[0034] Figure 8 System inputs for acceleration levels are shown according to one embodiment.

[0035] Fig. 9 An image-based switching control scheme using invariant features is shown.

[0036] Fig.10 is a schematic flow chart of a method for controlling robot translation according to one embodiment.

[0037] Fig.11 is a schematic flow chart of a method for controlling robot translation according to one embodiment. DETAILED DESCRIPTION

[0038] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0039] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0040] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0041] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0043] Figure 1 1 is a schematic block diagram of a robot 11 according to an embodiment. The robot 11 may be a mobile robot. The robot 11 may include a processor 110, a memory 111, and one or more computer programs 112 stored in the memory 111 and executable by the processor 110. When the processor 110 executes the computer program 112, the steps in the method embodiment for controlling the robot 11, such as Figure 5 Steps S41 to S44 in Figure 6Steps S51 to S56 in Fig.10 Steps S551 to S56 in Fig.11 Steps S561 to S566 in are implemented.

[0044] Exemplarily, the one or more computer programs 112 may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 111 and executed by the processor 110. The one or more modules / units may be a series of computer program instruction segments capable of executing specific functions, and the instruction segments are used to describe the execution process of the one or more computer programs 112 in the robot 11.

[0045] The processor 110 may be a central processing unit (CPU), a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate, a transistor logic device, or a discrete hardware component. The general purpose processor may be a microprocessor or any conventional processor, etc.

[0046] The memory 111 may be an internal storage unit of the robot 11, such as a hard disk or a memory. The memory 111 may also be an external storage device of the robot 11, such as a plug-in hard disk, a smart memory card (SMC) and a secure digital (SD) card, or any suitable flash memory card. In addition, the memory 111 may also include an internal storage unit and an external storage device. The memory 111 is used to store computer programs, other programs and data required by the robot. The memory 111 may also be used to temporarily store data that has been output or is to be output.

[0047] It should be noted that Figure 1 This is only an example of the robot 11 and does not limit the robot 11. The robot 11 may include a different number of components than shown in the figure, or may include some other different components. Figure 2 and Figure 3 In the illustrated embodiment, the robot 11 may further include an actuator 113, a moving mechanism 114, a camera 115, and a communication interface module 106. The robot 11 may further include input and output devices, network access devices, buses, and the like.

[0048] In one embodiment, the actuator 113 may include one or more motors and / or servos. The mobile mechanism 114 may include one or more wheels and / or tracks. The actuator 113 is electrically coupled to the mobile mechanism 114 and the processor 110, and may drive the movement of the mobile mechanism 114 according to the instructions of the processor 110. The camera 115 may be, for example, a camera mounted on the robot 11. The camera 115 is electrically connected to the processor 110 and is configured to transmit the captured image to the processor 110. The communication interface module 116 may include a wireless transmitter, a wireless receiver, and a computer program executable by the processor 110. The communication interface module 116 is electrically connected to the processor 103 for communication between the processor 110 and an external device. In one embodiment, the processor 110, the memory 111, the actuator 113, the mobile mechanism 114, the camera 115, and the communication interface module 116 may be connected to each other via a bus.

[0049] In one embodiment, the robot 11 can observe a wall socket for automatic charging, observe items on a shelf for picking, and observe furniture in a room for home services. Figure 4 An exemplary scenario is shown in which the robot 11 can automatically dock at a charging station to perform an automatic charging process. Before controlling the robot 11 to self-charge, the robot 11 is first stopped in front of the charging station and faces the charging station. The posture of the robot 11 at this time is used as the desired posture, and the image of the environment in front of the robot 11 at this time is captured as the desired image. Then, when the robot 11 needs to be charged, no matter what posture the robot 11 is in, the current posture is used as the initial posture, and the image of the environment in front of the robot 11 is captured as the current image. Then, by executing the steps in the control method provided in the following embodiments, the robot 11 is controlled to move and rotate to the desired posture based on the desired image and the feature points in the current image.

[0050] Figure 5 A flow chart of a mobile robot control method according to an embodiment is shown. The method can be used to control Figure 1-3 The robot 11 in the robot moves. The method may be executed by one or more processors of the robot or one or more processors of other control devices electrically coupled to the robot. The control device may include but is not limited to: a desktop computer, a tablet computer, a laptop computer, a multimedia player, a server, a smart mobile device (such as a smart phone, a handheld phone, etc.) and a smart wearable device (such as a smart watch, smart glasses, a smart camera, a smart bracelet, etc.) and other computing devices with computing and control capabilities. In one embodiment, the method may include steps S41 to S44.

[0051] Step S41: When the robot is in a desired posture, a first image captured by a camera on the robot is acquired.

[0052] Step S42: When the robot is in the current posture, obtain a second image taken by the camera on the robot.

[0053] Step S43: extracting a plurality of pairs of matching feature points from the first image and the second image and projecting the extracted feature points onto a virtual unit sphere to obtain a plurality of projected feature points.

[0054] The virtual unit sphere is a unified spherical model that simulates the central imaging system using a combination of a virtual sphere and perspective projection. The center of the virtual unit sphere coincides with the optical center of the camera coordinates. Multiple pairs of matching feature points extracted from the first image and the second image are projected onto the virtual unit sphere, and the corresponding projection points on the unit sphere are the projected feature points.

[0055] Step S44: acquiring an image invariant feature and a rotation vector feature based on the multiple projection feature points, and controlling the movement of the robot according to the image invariant feature and the rotation vector feature until the robot is in the desired posture.

[0056] In one embodiment, the image invariant feature may include one or more of the inverse of the distance between two of the projected feature points, the image moment and the area. The rotation vector feature may be an angle vector or a direction vector. Controlling the robot movement includes controlling the robot translation and controlling the robot rotation.

[0057] Figure 6 A flow chart of a mobile robot control method according to another embodiment is shown. The method can be used to control Figure 1-3 The method may be performed by one or more processors of the robot or one or more processors of other control devices electrically coupled to the robot. The control device may include, but is not limited to: a desktop computer, a tablet computer, a laptop computer, a multimedia player, a server, a smart mobile device (such as a smart phone, a handheld phone, etc.) and a smart wearable device (such as a smart watch, smart glasses, a smart camera, a smart bracelet, etc.) and other computing devices with computing and control capabilities. In one embodiment, the method may include steps S51 to S56.

[0058] Step S51: When the robot is in a desired posture, a first image captured by a camera on the robot is acquired.

[0059] In one embodiment, the desired posture may include the desired position and desired orientation of the robot. The robot is expected to move to the desired position and be in the desired orientation. The robot can be controlled (e.g., by a user) to stop at the desired position and desired orientation so that an image of the environment in front of the robot (e.g., a wall with a socket) can be captured by a camera mounted on the robot as a first image (i.e., the desired image). When the robot stops at the desired position and desired orientation, the environment in the robot's field of view is required to include at least three feature points. The robot's camera can be a fisheye pinhole camera or a non-pinhole camera, such as a catadioptric camera.

[0060] Step S52: When the robot is in the current posture, a second image taken by a camera on the robot is acquired.

[0061] When the robot needs to be controlled to move to a desired position and in a desired direction, the current image of the environment in front of the robot is captured by a camera mounted on the robot as the second image (i.e., the current image). When capturing the second image, the environment within the robot's field of view is required to contain at least three feature points:

[0062] Step S53: extract multiple pairs of matching feature points from the first image and the second image and project the extracted feature points onto a virtual unit sphere to obtain multiple projected feature points. The center of the virtual unit sphere coincides with the optical center of the coordinates of the camera.

[0063] In one embodiment, extracting multiple pairs of matching feature points from the first image and the second image may be achieved in the following manner.

[0064] First, a first number of original feature points are extracted from each of the first image and the second image using a scale-invariant feature transform (SIFT) descriptor, and then a second pair of matching feature points is obtained by comparing and matching the extracted original feature points.

[0065] For example, the SIFT descriptor can be used to extract 200 SIFT feature points from any position in the first image and 200 SIFT feature points from any position in the second image. Then, the closest Euclidean distance algorithm can be used to compare and match the extracted 400 (200 pairs) SIFT feature points to obtain at least 3 pairs of matching feature points. Optionally, during the comparison and matching process, a balanced binary search tree, such as KDTree, can be used to speed up the search process.

[0066] It should be noted that there is no restriction on the positions of the above-mentioned feature points, that is, these feature points may be coplanar or non-coplanar, and the plane where these feature points are located may be perpendicular to or non-perpendicular to the motion plane.

[0067] An object of the present disclosure is to adjust the robot's posture based on visual feedback of at least three non-collinear static feature points in a first image, and to make it independent of the robot's posture requirements relative to an inertial reference frame or a visual target. To achieve this goal, multiple pairs of matching feature points extracted from the first image and the second image need to be projected onto a unit sphere.

[0068] refer to Figure 7 In one embodiment, the camera mounted on the robot is a calibrated forward-facing onboard camera with a coordinate system F, and the coordinates of the robot and the camera are the same. Figure 7 The x-axis and y-axis in the coordinate system F define the plane of motion, where the positive direction of the x-axis is the direction of travel of the robot, and the y-axis coincides with the axis around which the robot's wheels rotate. Point P in the coordinate system F i is projected onto the imaginary unit sphere and denoted by h i The virtual unit sphere is a unified spherical model that uses a virtual sphere and perspective projection to simulate the central imaging system.

[0069] Step S54: acquiring image invariant features and rotation vector features based on multiple projection feature points.

[0070] Image invariant features are a type of image features that are invariant to camera rotation. As the system output to represent the translational motion of the robot, Represents a two-dimensional real coordinate space. The dynamic model of the image invariant features needs to satisfy the following formula: in, is the interaction matrix, υ=[υ x , y ] T represents the linear velocity of the robot in the coordinate system F, which is not restricted by the non-holonomic constraints. The acceleration of the robot coordinate system satisfies It should be noted that the formula The robot's angular velocity ω is represented by the z-axis, which is independent of the coordinate system F.

[0071] In one embodiment, the invariant image feature may include one or more of an inverse of a distance between two of the projected feature points, an image moment, and an area.

[0072] In such Figure 7 In the example shown, the projected feature point hi and h j Distance d ij The inverse of is used as the image invariant feature because it has the best linearization property between the task space and the image space. i and h j represents the projection of feature points extracted from the same image onto the unit sphere, e.g. Figure 7 The feature point P i and P j The corresponding projected feature point. That is, point h i and h j It can represent the projection of the feature points extracted from the second image (i.e., the current image) on the unit sphere. Point h i and h j It can represent the projection of the feature points extracted from the first image (i.e., the desired image) onto the unit sphere. It should be noted that two projected feature points can be connected to each other to form a small line segment. The closer the robot is to the desired position, the longer the line segment. The goal of translation control is to make the selected line segment equal to the desired value. For example, assuming there are 3 projected feature points, i=0, 1, 2, j=0, 1, 2, and i≠j, there are 3 combinations (0, 1) (0, 2) (1, 2), and two of them can be selected.

[0073] Assume a static feature point, where y is measured in pixels i =[y i1 ,y i2 ] T represents the homogeneous coordinates of the feature points extracted from the image plane, and i represents a positive integer greater than 0. Construct an imaginary image plane that can be called the retina, which is associated with an imaginary perspective camera. The corresponding coordinates in the retina can be obtained according to the following formula: i =[r i1 , r i2 , 1] T =A -1 y i , where A represents the generalized camera projection matrix associated with the mirror and camera intrinsic parameters. The projected feature points projected onto the unit sphere can be obtained according to the following formula: i =γ[r i1 , r i2 , 1-ξ / γ] T ;γ=(ξ+μ(r i )) / ||r i || 2 ;and where ξ represents the type of mirror or lens used. When ξ is equal to 0, the unified model corresponds to a perspective camera and A represents the intrinsic matrix of the camera.

[0074] Based on the above principle, at least three pairs of feature points extracted from the expected image and the real environment represented by the current image are projected onto i The corresponding image invariant features can be obtained according to the following formula: S = [1 / d 01 , 1 / d 02 ] T ,in, d 01 Represents the projection feature point h 0 With h 1 Distance on the unit sphere, d 02 Represents the projection feature point h 0 With h 2 The distance on the unit sphere. At this time, the corresponding interaction matrix is in, G j =(Ih j h j T )I 12 , j = 0, 1, 2, I is the identity matrix, I 12 Represents the first two columns of the identity matrix. The projected feature point P on the unit sphere j The distances to the origin are generally unknown. In this embodiment, they can be replaced by constant values ​​close to the expected values. Experiments have shown that the closed-loop system is stable under this approximation.

[0075] In one embodiment, if the image moment is used as the image invariant feature, the image invariant feature can be obtained according to the following formula: in, It represents the i+j+k order central moment of the image.

[0076] In one embodiment, if the area is used as the image invariant feature, the image invariant feature can be obtained according to the following formula: Among them, h i =(x i ,y i , z i ), which represents the three-dimensional coordinates of the i-th projection feature point on the unit sphere.

[0077] In one embodiment, the mobile robot is a unicycle-like vehicle under nonholonomic constraints, and its linear velocity in the camera coordinate system F is υ = [υ x ,0] T . Combined Figure 8 , the system input of the acceleration level is defined as and Among them, x and x d denote the actual x-axis and the desired x-axis respectively. The translational dynamics is obtained according to the following formula: Where x = [1, 0] T , which represents the x-axis in the coordinate system F. The image error of the invariant feature can be obtained as follows: S =ss d , where S d represents the expected value calculated from the expected image.

[0078] In one embodiment, when used for motion control adjustment, The error kinematics can be expressed as Multiply both sides of the above equation by J -1 And taking its time derivative we get Then replace the result in the formula The α in the image gives the following translational dynamics of the camera-robot system:

[0079] The robustness can be improved by calculating the rotation vector features using all the projected feature points obtained by projection. Since the calculated rotation vector features provide a direct mapping between the rotation speed and the rotation vector rate, better characteristics can be obtained.

[0080] It should be noted that no matter which of the image invariant features is distance, image moment, or area, the rotation vector feature remains unchanged and can be calculated according to the following formula: δ = -arctan(a dy / a dx ), a dx ≠0. The calculation principle is: calculate the acceleration direction of the robot according to the following formula: Then calculate the angle between the acceleration direction and the x-axis of the robot coordinate system F. The angle is the rotation vector characteristic in the following formula: δ = -arctan(a dy / a dx ), a dx ≠0. In addition, the directional dynamics can be obtained according to the following formula: Here, δ represents the direction difference from the desired direction to the x-axis, and ω represents the angular velocity.

[0081] Step S55: Control the robot to translate according to the invariant features of the image.

[0082] Step S56: Control the robot to rotate according to the rotation vector feature.

[0083] It should be noted that the robot's motion is a rigid body motion, which can be decomposed into rotational motion and translational motion. Fig. 9In this embodiment, the adjustment of the robot's posture needs to be switched between two steps. The first step is to use the image invariant features to control the robot to translate. The goal is to eliminate the translation error (also called the invariant feature error), that is, the goal is to make the translation error equal to 0. The second step is to use the rotation vector feature to control the robot to rotate after eliminating the translation error. The goal is to eliminate the direction error, that is, the direction error is equal to 0.

[0084] In the first step above, image invariant features are used to represent the translation state, so even if the desired pose cannot be obtained, the robot can still be controlled to translate. By using image invariant features, the direction is decoupled from the position control. In the position control, by reducing e s To generate the desired acceleration α d =[α dx , α dy ] T .

[0085] In the second step, the direction control is to align the x-axis with the α d For example, the desired x-axis satisfies x d =α d / ||α d ||. In e s After convergence, the robot is controlled to rotate to align the centroid of the projected feature points on the unit sphere with the desired value.

[0086] refer to Fig.10 In one embodiment, the first step (ie, step S55) may include the following steps.

[0087] Step S551: Calculate a first angular velocity and a first linear velocity according to the image invariant features and a preset control model.

[0088] Step S552: Control the robot to translate according to the first angular velocity and the first linear velocity.

[0089] Step S553: ​​Determine whether the translation error is less than a first preset threshold.

[0090] If the translation error is less than the first preset threshold, the process enters step S56: controlling the robot to rotate according to the rotation vector feature.

[0091] If the translation error is greater than or equal to the first preset threshold, it means that the translation error has not been eliminated, the robot has not reached the desired position, and the robot needs to continue to translate to the desired position. Then the process returns to step S52 of acquiring the second image. Steps S551, S552, S553 and S52 are repeated until the translation error is less than the first preset threshold, and then the second step of controlling the direction of the robot is started.

[0092] The goal of the first step above is to ensure thats In the first step, based on the image invariant features and the preset control model, the above steps S551 to S553 can be calculated according to formula e S =ss d And the following formula: u 2 =-k δ δ-k ω ω; and δ = -arctan(a dy / a dx ), a dx ≠0, where k s , k v , k δ and k ω represents the positive control gain. Assume that the singularity of J will not be reached. When ||α d ||≠0, δ is well defined. This is because ||α is only valid if the translation error is 0 at the end of the first step. d || = 0. δ≠±π / 2 is one of the sufficient conditions for the stability of the camera-robot system.

[0093] The first angular velocity used to control the translation of the robot can be calculated according to the formula u 2 =-k δ δ-k ω The first linear velocity can be obtained according to the formula The formula e S =ss d Can be used to determine if the translation error is zero.

[0094] According to the above formula, it can be found that in the position control process of the first step, the trajectory of the robot is not a straight line, but a curve. That is to say, in the process of controlling the translation of the robot through the first step, not only the amplitude of the robot movement should be adjusted, but also the direction of the robot should be adjusted accordingly to make it move to the desired position.

[0095] When according to the formula δ = -arctan(a dy / a dx ), a dx ≠0 When calculating δ, if α dx <0, α d The orientation of the camera will be flipped over 180 degrees. This is designed to ensure that the robot can move backwards and keep the visual target in front if the target is behind. It helps to keep the visual target in the field of view (FOV) of the camera if the FOV is limited.

[0096] refer to Fig.11 In one embodiment, the second step (ie, step S56) may include the following steps.

[0097] Step S561: Calculate the second angular velocity and the second linear velocity based on the rotation vector characteristics and the control model.

[0098] Step S562: Control the robot to rotate according to the second angular velocity and the second linear velocity.

[0099] Step S563: Determine whether the direction error of the robot after rotation is less than a second preset threshold.

[0100] Step S564: If the direction error of the robot is greater than or equal to the second preset threshold, this means that the direction error has not been eliminated and the robot is not in the desired direction. In this case, a third image is acquired by the camera of the robot.

[0101] Step S565: extracting a plurality of pairs of matching feature points from the first image and the third image, and projecting the extracted feature points onto the unit sphere to obtain a plurality of projected feature points.

[0102] Step S566: Acquire the rotation vector feature based on the multiple projected feature points extracted from the first image and the third image, and then the process returns to step S561 of calculating the second angular velocity and the second linear velocity based on the rotation vector feature and the control model. Repeat steps S551, S552, S553 and S56 until the direction error is less than the second preset threshold.

[0103] Step S567: If the direction error of the robot is less than the second preset threshold, it means that the direction error has been eliminated, the robot is in the desired direction, and the process ends.

[0104] In one embodiment, steps S561 to S567 can be implemented according to the following formula based on the rotation vector characteristics and the control model: 1 =0;u 2 =-k δ δ-k ω ω; and δ = sgn(e ρ ×ρ·e 3 )||e ρ ||, where e 3 =[0,0,1] T , which represents the z-axis, It represents the centroid of n feature points.

[0105] The second angular velocity ω used to control the rotation of the robot can be calculated according to the formula u 2 =-k δ δ-k ω The second linear velocity can be obtained according to the formula u 1=0, the linear velocity in this stage is equal to 0. The direction error δ can be obtained according to the formula δ=sgn(e ρ ×ρ·e 3 )||e ρ ||Calculate.

[0106] In one embodiment, the image invariant feature can be determined based on a combination of two or all of the distance, image moment and area. Specifically, the average value of at least two of the distance between two of the projected feature points, the image moment and the area is used as the image invariant feature. The image invariant feature is then substituted into the above formula to achieve the position control of the robot. In this way, multiple parameters are combined to determine the invariant characteristics of the final application, thereby improving the accuracy of the control result.

[0107] On the one hand, image features in the image coordinate system are used to represent the kinematics / dynamics of the mobile robot instead of the distances and angles defined in the Cartesian coordinate system. Only one camera is needed instead of relying on the use of traditional expensive and bulky sensors (such as lidar, sonar, wheel odometer, etc.) for robot movement control, which can reduce the manufacturing cost of the robot and enable the robot to have a reduced size.

[0108] On the other hand, since the image invariant features and rotation vector features are used to determine the desired posture of the robot, neither the target model needs to be known in advance nor the posture of the robot relative to the target needs to be estimated. In other words, there is no need to calculate and decompose the homography matrix or the essential matrix, so the computational complexity can be reduced and the computational speed can be improved.

[0109] On the other hand, since the feature points in the expected image and the current image can be arbitrarily selected, there is no requirement for the physical position of the points in the environment. For example, the feature points in the environment can be coplanar or non-coplanar. Therefore, the method can be applied to more scenarios, which can improve the versatility of the method.

[0110] In one embodiment, it can be similar to Figure 1 The robot configuration mobile robot control device. That is, the mobile robot control device may include a processor and a memory, the memory being electrically coupled to the processor and storing therein one or more computer programs executable by the processor. When the processor executes the computer program, the steps in the method embodiment for controlling the robot 11, such as Figure 5 Steps S41 to S44 in Figure 6 Steps S551 to S56 in Fig.10 Steps S561 to S56 in Fig.11Steps S561 to S566 in are implemented. The processor, memory, and computer program of the mobile robot control device can be the same as the processor 110, memory 111, and computer program 112 described above, and will not be repeated here. The robot mobile control device can be various computer system devices with data interaction functions, including but not limited to mobile phones, smart phones, other wireless communication devices, personal digital assistants, audio players, other media players, music recorders, etc., video recorders, cameras, other media recorders, radios, vehicle transportation equipment, programmable remote controls, laptop computers, desktop computers, printers, netbook computers, portable gaming devices, portable Internet devices, data storage devices, smart wearable devices (e.g., head-mounted devices (HMDs), such as smart glasses, smart clothes, smart bracelets, smart necklaces, or smart watches) and combinations thereof.

[0111] Exemplarily, one or more computer programs may be divided into one or more modules / units, and one or more modules / units may be stored in a memory and executed by a processor. One or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of one or more computer programs. For example, one or more computer programs may be divided into a first acquisition module, a second acquisition module, a feature extraction module, a calculation module, and a control module.

[0112] The first acquisition module is configured to acquire a first image taken by a camera on the robot when the robot is in a desired posture.

[0113] The second acquisition module is configured to acquire a second image taken by a camera on the robot when the robot is in a current posture;

[0114] The feature extraction module is configured to extract a plurality of pairs of matching feature points from the first image and the second image and project the extracted feature points onto a virtual unit sphere to obtain a plurality of projected feature points, wherein the center of the virtual unit sphere coincides with the optical center of the coordinates of the camera;

[0115] The calculation module is configured to obtain an image invariant feature and a rotation vector feature based on the multiple projection feature points.

[0116] The control module is configured to control the robot to move according to the image invariant feature and the rotation vector feature until the robot is in the desired posture.

[0117] It should be noted that the mobile robot control device may include a different number of components than those described above, or may be combined with some other different components. For example, the mobile robot control device may also include input and output devices, network access devices, buses, etc.

[0118] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present disclosure. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be described here.

[0119] In one embodiment, a non-transitory computer-readable storage medium may be configured in the robot 11 or the mobile robot control device as described above. The non-transitory computer-readable storage medium may be a storage unit configured in the main control chip and the data acquisition chip in the above embodiment. One or more computer programs are stored on the non-transitory computer-readable storage medium, and when the computer program is executed by one or more processors, the robot control method described in the above embodiment is implemented.

[0120] In the above embodiments, the description of each embodiment has its own emphasis. For a part that is not described or recorded in detail in one embodiment, reference can be made to the relevant description of other embodiments.

[0121] It will be appreciated by those skilled in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0122] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may also be an indirect coupling or communication connection through some interfaces, devices or units, which may be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] Each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0125] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0126] For the purpose of explanation, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the principles of the present disclosure and its practical application, so that other technical personnel in the art can best utilize the present disclosure and various embodiments, and make various modifications thereto to suit the specific purpose contemplated.

Claims

1. A computer-implemented method for controlling a mobile robot executed by one or more processors, the method comprising: When the robot is in a desired posture, acquiring a first image captured by a camera on the robot; When the robot is in the current posture, acquiring a second image taken by a camera on the robot; Extracting multiple pairs of matching feature points from the first image and the second image and projecting the extracted feature points onto a virtual unit sphere to obtain multiple projected feature points with three-dimensional coordinates based on the virtual unit sphere, wherein the virtual unit sphere is a unified spherical model, and the unified spherical model uses a combination of a virtual sphere and perspective projection to simulate a central imaging system, and the center of the virtual unit sphere coincides with the optical center of the coordinates of the camera; Image invariant features and rotation vector features are acquired based on the multiple projected feature points, and the robot is controlled to move according to the image invariant features and the rotation vector features until the robot is in the desired posture, wherein the robot is controlled to translate and to rotate to align the center of mass of the projected feature points on the virtual unit sphere with the desired value.

2. The method according to claim 1, characterized in that Controlling the robot movement includes: Controlling the robot to translate according to the invariant features of the image; and The robot is controlled to rotate according to the rotation vector feature.

3. The method according to claim 2, characterized in that Controlling the robot to translate according to the invariant features of the image includes: Calculate a first angular velocity and a first linear velocity according to the image invariant feature and a preset control model; Controlling the robot to translate according to the first angular velocity and the first linear velocity; Determining whether the translation error is less than a first preset threshold; If the translation error is greater than or equal to the first preset threshold, returning to the step of acquiring the second image; If the translation error is less than the first preset threshold, the robot is controlled to rotate according to the rotation vector feature.

4. The method according to claim 2, characterized in that Controlling the robot to rotate according to the rotation vector feature includes: Calculate a second angular velocity and a second linear velocity based on the rotation vector characteristic and the control model; Controlling the robot to rotate according to the second angular velocity and the second linear velocity; Determine whether the direction error of the robot after rotation is less than a second preset threshold; If the direction error of the robot is greater than or equal to the second preset threshold, acquiring a third image through the camera; Extracting a plurality of pairs of matching feature points from the first image and the third image, and projecting the extracted feature points onto the unit sphere to obtain a plurality of projected feature points; and The rotation vector feature is acquired based on the plurality of projected feature points extracted from the first image and the third image, and then the step of calculating the second angular velocity and the second linear velocity based on the rotation vector feature and the control model is returned to.

5. The method according to claim 1, characterized in that The image invariant feature comprises one or more of the reciprocal of the distance between two of the projected feature points, the image moment and the area.

6. The method according to claim 1, characterized in that Acquiring the image invariant feature based on the multiple projection feature points includes: acquiring at least two of a distance between two of the projected feature points, an image moment, and an area; and An average value of at least two of a distance between two of the projected feature points, an image moment, and an area is calculated, and the average value is used as an image invariant feature.

7. The method according to claim 1, characterized in that Acquiring the rotation vector feature based on the multiple projection feature points includes: Determining the acceleration direction of the robot based on the multiple projection feature points; The angle between the acceleration direction and the x-axis of the robot coordinate system is taken as the rotation vector feature.

8. The method according to claim 1, characterized in that Extracting a plurality of pairs of matching feature points from the first image and the second image comprises: extracting a first number of raw feature points from each of the first image and the second image using a scale-invariant feature transform; and A second number of pairs of matching feature points are obtained by comparing and matching the extracted original feature points.

9. The method according to claim 1, characterized in that Also includes: Control the robot to stop at the desired position with a preset posture; and An image of the environment in front of the robot taken by the camera is used as the first image, wherein the environment in front of the robot includes at least three feature points.

10. A non-transitory computer-readable storage medium storing one or more programs to be executed in a mobile robot, wherein the one or more programs, when executed by one or more processors of the robot, cause the robot to perform a process comprising: When the robot is in a desired posture, acquiring a first image captured by a camera on the robot; When the robot is in the current posture, acquiring a second image taken by a camera on the robot; Extracting multiple pairs of matching feature points from the first image and the second image and projecting the extracted feature points onto a virtual unit sphere to obtain multiple projected feature points with three-dimensional coordinates based on the virtual unit sphere, wherein the virtual unit sphere is a unified spherical model, and the unified spherical model uses a combination of a virtual sphere and perspective projection to simulate a central imaging system, and the center of the virtual unit sphere coincides with the optical center of the coordinates of the camera; Image invariant features and rotation vector features are acquired based on the multiple projected feature points, and the robot is controlled to move according to the image invariant features and the rotation vector features until the robot is in the desired posture, wherein the robot is controlled to translate and to rotate to align the center of mass of the projected feature points on the virtual unit sphere with the desired value.

11. The non-transitory computer-readable storage medium of claim 10, wherein: The image invariant feature comprises one or more of the reciprocal of the distance between two of the projected feature points, the image moment and the area; Acquiring the rotation vector feature based on the multiple projection feature points includes: Determining the acceleration direction of the robot based on the multiple projection feature points; The angle between the acceleration direction and the x-axis of the robot coordinate system is taken as the rotation vector feature.

12. A mobile robot comprising: one or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: instructions for acquiring a first image captured by a camera on the robot when the robot is in a desired posture; instructions for acquiring a second image captured by a camera on the robot when the robot is in the current pose; An instruction for extracting a plurality of pairs of matching feature points from the first image and the second image and projecting the extracted feature points onto a virtual unit sphere to obtain a plurality of projection feature points with three-dimensional coordinates based on the virtual unit sphere, wherein the virtual unit sphere is a unified spherical model, and the unified spherical model simulates a central imaging system by combining a virtual sphere and perspective projection, and the center of the virtual unit sphere coincides with the optical center of the coordinates of the camera; An instruction for acquiring an image invariant feature and a rotation vector feature based on the multiple projected feature points and controlling the movement of the robot according to the image invariant feature and the rotation vector feature until the robot is in the desired posture, wherein the robot is controlled to translate and is controlled to rotate to align the center of mass of the projected feature points on the virtual unit sphere with the desired value.

13. The robot according to claim 12, characterized in that: Controlling the robot movement includes: Controlling the robot to translate according to the invariant features of the image; and The robot is controlled to rotate according to the rotation vector feature.

14. The robot according to claim 13, characterized in that: Controlling the robot to translate according to the invariant features of the image includes: Calculate a first angular velocity and a first linear velocity according to the image invariant feature and a preset control model; Controlling the robot to translate according to the first angular velocity and the first linear velocity; Determining whether the translation error is less than a first preset threshold; If the translation error is greater than or equal to the first preset threshold, returning to the step of acquiring the second image; If the translation error is less than the first preset threshold, the robot is controlled to rotate according to the rotation vector feature.

15. The robot according to claim 13, characterized in that: Controlling the robot to rotate according to the rotation vector feature includes: Calculate a second angular velocity and a second linear velocity based on the rotation vector characteristic and the control model; Controlling the robot to rotate according to the second angular velocity and the second linear velocity; Determine whether the direction error of the robot after rotation is less than a second preset threshold; If the direction error of the robot is greater than or equal to the second preset threshold, acquiring a third image through the camera; Extracting a plurality of pairs of matching feature points from the first image and the third image, and projecting the extracted feature points onto the unit sphere to obtain a plurality of projected feature points; and The rotation vector feature is acquired based on the plurality of projected feature points extracted from the first image and the third image, and then the step of calculating the second angular velocity and the second linear velocity based on the rotation vector feature and the control model is returned to.

16. The robot according to claim 12, characterized in that: The image invariant feature comprises one or more of the reciprocal of the distance between two of the projected feature points, the image moment and the area.

17. The robot according to claim 12, characterized in that: Acquiring the image invariant feature based on the multiple projection feature points includes: acquiring at least two of a distance between two of the projected feature points, an image moment, and an area; and An average value of at least two of a distance between two of the projected feature points, an image moment, and an area is calculated, and the average value is used as an image invariant feature.

18. The robot according to claim 12, characterized in that: Acquiring the rotation vector feature based on the multiple projection feature points includes: Determining the acceleration direction of the robot based on the multiple projection feature points; The angle between the acceleration direction and the x-axis of the robot coordinate system is taken as the rotation vector feature.

19. The robot according to claim 12, characterized in that: Extracting a plurality of pairs of matching feature points from the first image and the second image comprises: extracting a first number of raw feature points from each of the first image and the second image using a scale-invariant feature transform; and A second number of pairs of matching feature points are obtained by comparing and matching the extracted original feature points.

20. The robot according to claim 12, characterized in that: Also includes: Control the robot to stop at the desired position with a preset posture; and An image of the environment in front of the robot taken by the camera is used as the first image, wherein the environment in front of the robot includes at least three feature points.

Images