Robot following method, device, robot and computer readable storage medium

Abstract

The present invention relates to the field of artificial intelligence, and discloses a robot following method, device, robot and storage medium, wherein the method comprises: constructing an initial three-dimensional point cloud map of a target environment; when receiving a robot following instruction, recording the target environment to obtain a motion video; calculating the motion trajectory of the robot within any two adjacent frames of images according to the motion video; determining the spatial position information of a UWB module in the robot according to the motion trajectory; calculating the distance between the UWB module in the robot and the UWB module in the user handle by using a TOF ranging method; calculating the spatial position information of the UWB module in the user handle by using a triangulation positioning method according to the distance; generating a following path of the robot according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle. The present invention can improve the accuracy of robot following.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a robot following method, a device, a robot and a readable storage medium. Background Art

[0002] Robot following refers to the process in which a following robot remains within a certain range of the user through a positioning system, for example, a dog-walking robot.

[0003] Currently, the common robot following methods are GPS positioning navigation, Bluetooth guided navigation, laser guided navigation and ultrasonic guided navigation. Among them, GPS positioning navigation is only suitable for outdoor use due to signal limitations, and the indoor accuracy is not high; Bluetooth guided navigation is in the 2.4G band, which is occupied by a lot of people, and is easily interfered by common wireless devices such as mobile phone signals, WiFi, wireless mouse, etc., resulting in a large measured distance and low accuracy; laser guided navigation and ultrasonic guided navigation are difficult to achieve single target following in crowded places, resulting in low accuracy. Summary of the invention

[0004] The present invention provides a robot following method, a device, a robot and a readable storage medium, the purpose of which is to improve the accuracy of the robot in the following process.

[0005] To achieve the above object, the present invention provides a robot following method, the method comprising:

[0006] Obtain image information of the target environment and initial position information of the UWB module in the robot, calculate corner points in the image information using a corner point algorithm, and construct an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology based on the image information and the corner points;

[0007] When receiving a robot follow instruction, a camera module in the robot is used to record the target environment to obtain a motion video;

[0008] According to the image information of each frame of the motion video, a target corner point is selected from the corner points, and according to the target corner points in any two adjacent frames of the motion video, a motion trajectory of the robot within the time of any two adjacent frames of the motion video is calculated;

[0009] Determine the spatial position information of each frame time node of the UWB module in the robot during the video recording process according to the motion trajectory and the initial position information;

[0010] Acquire spatial position information of the UWB module in the robot of all adjacent frames of the motion video at preset time intervals within a preset time period to form a first spatial position information set, and acquire signal reception time and signal response time of the UWB module in the robot and the UWB module in the user handle, and calculate the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time;

[0011] Calculate the spatial position information of the UWB module in the user handle using a triangulation method according to the spatial position information set and the distance;

[0012] A following path of the robot is generated according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle.

[0013] Optionally, the step of calculating the motion trajectory of the robot within the time period of any two adjacent frames of images according to the target corner points in any two adjacent frames of images in the motion video includes:

[0014] Matching target corner points in any two adjacent frames of the motion video to obtain a matching result;

[0015] According to the matching result, the rotation vector and the displacement vector between any two adjacent frames of images are calculated using the epipolar geometry formula;

[0016] Using the least square method, nonlinearly optimizing the rotation vector and the displacement vector to obtain a target rotation vector and a target displacement vector;

[0017] According to the target rotation vector and the target displacement vector, any two adjacent frames of images are spliced ​​to obtain the motion trajectory of the robot.

[0018] Optionally, the calculating the corner points in the image information by using a corner point algorithm includes:

[0019] Acquire pixel information in the image information to obtain a pixel image;

[0020] Select any pixel point in the pixel image as the center of the circle and draw a circle with a preset length as the radius to obtain a target circle;

[0021] Determine whether there are a first preset number of continuous comparison pixel points on the target circle;

[0022] When there are no first preset number of continuous comparison pixel points on the target circle, determining that the pixel point is not a corner point;

[0023] When there are a first preset number of continuous comparison pixel points on the target circle, calculating the difference between the brightness value of the comparison pixel point and the brightness value of the pixel point;

[0024] Determine whether there is a second preset number of differences among the differences that are greater than or less than a preset threshold;

[0025] When there is no second preset number of differences in the differences that are greater than or less than a preset threshold, it is determined that the pixel point is not a corner point;

[0026] When a second preset number of differences among the differences is greater than or less than a preset threshold, it is determined that the pixel point is a corner point.

[0027] Optionally, constructing an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology according to the image information and the corner points includes:

[0028] Performing filtering processing on the image information to obtain denoised image information;

[0029] Performing a dedistortion process on the denoised image information to obtain target image information;

[0030] According to the corner points, performing corner point matching on the target image information to obtain a matching result;

[0031] According to the matching results, the target image information is spliced ​​to obtain an initial three-dimensional point cloud map.

[0032] Optionally, determining the spatial position information of each frame time node of the UWB module in the robot during the recording process according to the motion trajectory and the initial position information includes:

[0033] Obtaining the movement speed of the robot and the time interval between two adjacent frames;

[0034] Calculate the movement distance of the robot within the time interval of two adjacent frames according to the movement trajectory, the movement speed and the time interval;

[0035] According to the initial position information of the UWB module in the robot and the movement distance of the robot within the time interval of two adjacent frames of images, the spatial position information of the robot at each frame time node is calculated in chronological order.

[0036] Optionally, calculating the spatial position information of the UWB module in the user handle by using a triangulation method according to the spatial position information set and the distance includes:

[0037] Taking the spatial position information of any UWB module in the robot in any frame in the spatial position information set as the origin, constructing a spatial coordinate system and obtaining the origin coordinates of the origin;

[0038] Acquire the residual spatial position information of the remaining UWB modules in the robot in any frame in the spatial coordinate system, and the adjacent spatial position information of all UWB modules in the robot in adjacent frames of the any frame;

[0039] Calculate the reference coordinates of the remaining UWB modules in the robot in any frame and all UWB modules in the robot in frames adjacent to any frame in the spatial coordinate system according to the spatial correspondence between the residual spatial position information and the adjacent spatial position information and the origin in the spatial coordinate system;

[0040] Construct a spherical surface according to the origin coordinates, the reference coordinates, and the distance between the UWB module corresponding to each coordinate and the UWB module in the user handle, and use the coordinates of the points through which all spherical surfaces pass as the handle coordinates of the UWB module in the user handle;

[0041] The spatial position information of the UWB module in the user handle is calculated according to the corresponding relationship between the handle coordinates and the origin coordinates.

[0042] Optionally, calculating the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time includes:

[0043] The distance Z between the UWB module in the robot and the UWB module in the user handle is calculated using the following distance measurement formula:

[0044] Z=X*(AB)

[0045] Wherein, X is the speed of the electromagnetic wave signal emitted by the robot or the user handle; A is the signal receiving time of the UWB module in the robot or the UWB module in the user handle; B is the signal response time of the UWB module in the robot or the UWB module in the user handle.

[0046] In order to solve the above problems, the present invention further provides a robot following device, the device comprising:

[0047] A three-dimensional map construction module is used to obtain image information of the target environment and initial position information of the UWB module in the robot, calculate corner points in the image information using a corner point algorithm, and construct an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology based on the image information and the corner points;

[0048] A user handle positioning module is used to, when receiving a robot follow instruction, use the camera module in the robot to record the target environment to obtain a motion video, select the target corner point from the corner point according to the image information of each frame image in the motion video, and calculate the motion trajectory of the robot within the time of any two adjacent frames of the image according to the target corner point in any two adjacent frames of the motion video, determine the spatial position information of each frame time node of the UWB module in the robot during the recording process according to the motion trajectory and the initial position information, obtain the spatial position information of the UWB module in the robot in all adjacent frames of the motion video at a preset time interval within a preset time period, form a first spatial position information set, and obtain the UWB module in the robot;

[0049] A following path generation module is used to generate a following path for the robot according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle.

[0050] In order to solve the above problems, the present invention further provides a robot, comprising:

[0051] a memory storing at least one computer program; and

[0052] The processor executes the computer program stored in the memory to implement the robot following method described above.

[0053] In order to solve the above problem, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in the robot to implement the robot following method described above.

[0054] The embodiment of the present invention obtains image information of the target environment and initial position information of the UWB module in the robot, calculates the corner points in the image information by using a corner point algorithm, and constructs an initial three-dimensional point cloud map of the target environment based on the image information and the corner points by using visual real-time positioning and mapping technology, thereby providing a complete target environment map for the robot, thereby helping the robot to perform obstacle avoidance operations during navigation and travel, and ensuring the safety of the robot when following. Secondly, when a robot following instruction is received, the target environment is recorded by using a camera module in the robot to obtain a motion video, and the motion trajectory of the robot is calculated according to the motion video, and the spatial position information of each frame time node of the UWB module in the robot during the recording process is determined according to the motion trajectory, thereby improving the accuracy of robot positioning, and ensuring that the robot can accurately avoid obstacles and accurately position itself during the following task clock. Furthermore, the spatial position information of the UWB module in the robot of all adjacent frames of the motion video is obtained at preset time intervals within a preset time period to form a first spatial position information set, and the UWB module in the robot is obtained. The signal receiving time and signal response time of the UWB module in the robot and the UWB module in the user handle are calculated according to the signal receiving time and the signal response time, and the distance between the UWB module in the robot and the UWB module in the user handle is determined to determine whether the robot is within the restricted range of the robot following task, thereby ensuring that the robot can successfully complete the robot following task. In addition, by calculating the distance between the UWB module in the robot and the UWB module in the user handle, the accuracy of the robot following is also improved. Finally, according to the spatial position information set and the distance, the spatial position information of the UWB module in the user handle is calculated using the triangulation positioning method, thereby improving the accuracy of the distance between the UWB module in the robot and the UWB module in the user handle, so as to facilitate the robot to search for the position of the user handle, thereby performing the following task in the target environment. Furthermore, according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle, the following path of the robot is generated, thereby completing the robot following task, ensuring that the robot can achieve obstacle avoidance and precise positioning, thereby improving the accuracy of the robot following. Therefore, the robot following method, device, equipment and storage medium provided by the present invention can improve the accuracy of robot following. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic diagram of a flow chart of a robot following method provided by an embodiment of the present invention;

[0056] Figure 2 and Figure 3 A detailed implementation flow chart of one of the steps in the robot following method provided by one embodiment of the present invention;

[0057] Figure 4 A schematic diagram of a module of a robot following device provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of the internal structure of a robot for implementing a robot following method provided by an embodiment of the present invention;

[0059] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0061] The embodiment of the present invention provides a robot following method. The robot following method is mainly executed by an intelligent robot. In other words, the robot following method can be executed by software or hardware installed in the intelligent robot.

[0062] Reference Figure 1 FIG. 1 is a flow chart of a robot following method provided by an embodiment of the present invention. In the embodiment of the present invention, the robot following method comprises the following steps:

[0063] S1. Obtain image information of the target environment and the initial position information of the UWB module in the robot, calculate the corner points in the image information using a corner point algorithm, and construct an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology based on the image information and the corner points.

[0064] In an embodiment of the present invention, the target environment may be the working environment of the robot, that is, the environment in which the robot follows the user's handle to navigate. The image information may be a photo of the target environment taken by the robot, or a photo taken in advance and stored in a preset database. The UWB module may be a module that uses a wireless carrier to complete communication. The initial position information may be the starting position of the UWB module in the robot in the target environment. The definition of the corner point may be that a certain pixel point and enough pixel points in its surrounding area are in different areas, then this pixel point is called a corner point.

[0065] In an optional embodiment of the present invention, since the image taken by a monocular camera is two-dimensional and the depth information of the target environment cannot be obtained, the target environment can be photographed by a binocular camera provided by the robot to obtain image information of the target environment. Furthermore, the images obtained during the photographing process are fused to directly calculate the depth value of the object, thereby obtaining the three-dimensional information of the target environment and the initial position information of the UWB module in the robot, thereby improving the accuracy of target environment acquisition.

[0066] The embodiment of the present invention uses a corner point algorithm to calculate the corner points in the image information, thereby calculating the robot's motion trajectory and the position of each corner point in the target environment, ensuring the smooth construction of the three-dimensional point cloud map and improving the accuracy of the three-dimensional point cloud map.

[0067] Further, as an optional embodiment of the present invention, refer to Figure 2 As shown, the corner point algorithm is used to calculate the corner points in the image information, including:

[0068] S11, acquiring pixel information in the image information to obtain a pixel image;

[0069] S12, selecting any pixel point in the pixel image as the center of the circle and drawing a circle with a preset length as the radius to obtain a target circle;

[0070] S13, determining whether there are a first preset number of continuous comparison pixel points on the target circle;

[0071] When there are no first preset number of continuous comparison pixel points on the target circle, S14 determines that the pixel point is not a corner point;

[0072] S15. When there are a first preset number of continuous comparison pixel points on the target circle, calculate the difference between the brightness value of the comparison pixel point and the brightness value of the pixel point;

[0073] S16, determining whether there is a second preset number of differences among the differences that are greater than or less than a preset threshold;

[0074] When there is no second preset number of differences in the differences that are greater than or less than the preset threshold, executing S14 to determine that the pixel point is not a corner point;

[0075] S17: When a second preset number of differences among the differences are greater than or less than a preset threshold, determine that the pixel point is a corner point.

[0076] In the embodiment of the present invention, the comparison pixel points are pixel points covered by the edge of the target circle.

[0077] In the embodiment of the present invention, the pixel information includes the pixel value and brightness value of each pixel in the image. The preset length is generally the length of two pixels spaced from the pixel at the center of the circle.

[0078] In an optional embodiment of the present invention, first, a pixel point in the image information is arbitrarily selected, and it is determined whether a target circle can be formed with the pixel point as the center and a preset length as the radius. When the target circle exists, the difference between the brightness value of each pixel point on the target circle and the brightness value of the pixel point is calculated. Further, it is determined whether there are a preset number of differences in the differences that are greater than or less than a preset threshold. For example, if there are 16 pixel points on the target circle, and the difference between the brightness value of 12 pixel points and the brightness value of the center pixel point is greater than or less than the preset threshold, then the pixel point is determined to be a corner point.

[0079] The embodiment of the present invention constructs an initial three-dimensional point cloud map of the target environment based on the image information and the corner points using visual real-time positioning and mapping technology, and provides the robot with a complete target environment map, thereby helping the robot to perform obstacle avoidance operations during navigation and ensuring the safety of the robot when following the movement of the handle.

[0080] Further, as an optional embodiment of the present invention, constructing an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology according to the image information and the corner points includes:

[0081] Performing filtering processing on the image information to obtain denoised image information;

[0082] Performing a dedistortion process on the denoised image information to obtain target image information;

[0083] According to the corner points, performing corner point matching on the target image information to obtain a matching result;

[0084] According to the matching results, the target image information is spliced ​​to obtain an initial three-dimensional point cloud map.

[0085] In the embodiment of the present invention, the filtering process may be a filtering technique such as median filtering or Gaussian filtering. The de-distortion process may be to remove pincushion distortion or barrel distortion in the de-noised image information.

[0086] In an optional embodiment of the present invention, when performing corner point matching on the target image information, matching errors are prone to occur, resulting in image splicing errors and causing the initial three-dimensional point cloud map to be inaccurate. Therefore, in the corner point matching process, the Hamming distance of the corner point descriptor can be calculated to remove the corner points with larger gaps and obtain a matching result.

[0087] Furthermore, in another optional embodiment of the present invention, based on the matching results, the rotation matrix and displacement vector between any two adjacent target image information and the bit depth of each pixel in the target image information are calculated using geometric calculation formulas, and then PointCloud is constructed. Furthermore, the target image information is spliced ​​into an initial three-dimensional point cloud map through a preset PCL point cloud library and the PointCloud.

[0088] S2. When receiving a robot follow instruction, the target environment is recorded using a camera module in the robot to obtain a motion video.

[0089] In the embodiment of the present invention, the robot following instruction can be a program instruction input manually or a pre-set instruction triggered under certain conditions. The camera module includes a binocular camera and a sensor provided by the robot.

[0090] In an optional embodiment of the present invention, the robot will receive a robot following instruction in the following two situations: one is when the distance between the robot and the user's handle exceeds the maximum limit distance, the robot automatically starts the robot following instruction to achieve robot following; the other is when the user transmits a following command to the robot, the robot receives the robot following instruction to achieve robot following.

[0091] Furthermore, as another optional embodiment of the present invention, when the robot receives a robot follow instruction, it will automatically start the binocular camera to record the target environment, and input the recording result into the sensor to form motion video data, so that the visual odometer module in the robot can read the motion video and calculate the motion trajectory of the robot based on the motion video.

[0092] S3. Filter out target corner points from the corner points according to the image information of each frame in the motion video, and calculate the motion trajectory of the robot within the time of any two adjacent frames according to the target corner points in any two adjacent frames in the motion video.

[0093] In the embodiment of the present invention, the motion video may be composed of multiple frames of images. Any two adjacent frames of images may be the first frame of image and the second frame of image in the motion video, or the second frame of image and the third frame of image, and so on.

[0094] In an optional embodiment of the present invention, the image information of each frame image in the motion video is compared with the image information of the initial three-dimensional point cloud map to determine the corner points contained in each frame image in the motion video, and then the target corner points that match the image information of each frame image in the motion video are filtered out from the corner point parameters stored in the process of constructing the initial three-dimensional point cloud map.

[0095] The embodiment of the present invention calculates the motion trajectory of the robot within any two adjacent frames of images based on the target corner points in any two adjacent frames of images in the motion video, ensuring that the robot is in normal motion during the process of completing the robot following instruction, thereby ensuring the smooth completion of the robot following task.

[0096] Further, as an optional embodiment of the present invention, refer to Figure 3 As shown, the method of calculating the motion trajectory of the robot within the time period of any two adjacent frames of images according to the target corner points in any two adjacent frames of images in the motion video includes:

[0097] S31, matching target corner points in any two adjacent frames of the motion video to obtain a matching result;

[0098] S32, calculating the rotation vector and the displacement vector between any two adjacent frames of images using an epipolar geometry formula according to the matching result;

[0099] S33, using the least square method, performing nonlinear optimization on the rotation vector and the displacement vector to obtain a target rotation vector and a target displacement vector;

[0100] S34. According to the target rotation vector and the target displacement vector, any two adjacent frames of images are spliced ​​to obtain the motion trajectory of the robot.

[0101] In the embodiment of the present invention, the least square method may be any one of the calculation methods such as the Gauss-Newton method and the Levenberg-Marquardt method.

[0102] Furthermore, in an optional embodiment of the present invention, the principle of generating the motion trajectory is similar to the principle of constructing the initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology, so it is not repeated here.

[0103] S4. Determine the spatial position information of each frame time node of the UWB module in the robot during the recording process according to the motion trajectory and the initial position information.

[0104] In the embodiment of the present invention, the time node of each frame may be a timestamp of each frame of image.

[0105] The embodiment of the present invention determines the spatial position information of each frame time node of the UWB module in the robot during the recording process according to the motion trajectory and the initial position information, and subdivides the motion trajectory of the robot as a whole, thereby improving the accuracy of the robot positioning and ensuring that the robot can accurately avoid obstacles and accurately position itself when following the task clock.

[0106] Further, as an optional embodiment of the present invention, determining the spatial position information of each frame time node of the UWB module in the robot during the recording process according to the motion trajectory and the initial position information includes:

[0107] Obtaining the movement speed of the robot and the time interval between two adjacent frames;

[0108] Calculate the movement distance of the robot within the time interval of two adjacent frames according to the movement trajectory, the movement speed and the time interval;

[0109] According to the initial position information of the UWB module in the robot and the movement distance of the robot within the time interval of two adjacent frames of images, the spatial position information of the robot at each frame time node is calculated in chronological order.

[0110] In an optional embodiment of the present invention, the movement speed of the robot can be measured by a speed measuring tool. Furthermore, by checking the timestamp information of each frame image of the motion video, the time interval between two adjacent frames in the motion video can be determined. Furthermore, based on the motion trajectory, the movement speed and the time interval, the distance calculation formula is used to calculate the movement distance of the robot in the time interval between two adjacent frames, thereby determining the displacement distance of the robot. Finally, based on the initial position information of the UWB module in the robot, the spatial position information of the robot at each frame time node is calculated, thereby improving the accuracy of robot positioning.

[0111] S5. Within a preset time period and at a preset time interval, obtain the spatial position information of the UWB module in the robot of all adjacent frames of the motion video to form a first spatial position information set, and obtain the signal reception time and signal response time of the UWB module in the robot and the UWB module in the user handle, and calculate the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time.

[0112] In the embodiment of the present invention, the preset time period may be a time period set manually, generally the entire video recording process. The preset time interval may be a time period set manually, generally the time interval between two adjacent frames in the motion video recording for the convenience of calculation.

[0113] In the embodiment of the present invention, the robot and the user handle generally include two or more UWB modules to ensure that the spatial position information of the user handle in the three-dimensional space can be accurately located.

[0114] The embodiment of the present invention obtains the signal reception time and signal response time of the UWB module in the robot and the UWB module in the user handle, calculates the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time, and determines whether the robot is within the restricted range of the robot following task, thereby ensuring that the robot can successfully complete the robot following task. In addition, by calculating the distance between the UWB module in the robot and the UWB module in the user handle, the accuracy of the robot following is also improved.

[0115] Further, as an optional embodiment of the present invention, the calculating the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time includes:

[0116] The distance Z between the UWB module in the robot and the UWB module in the user handle is calculated using the following distance measurement formula:

[0117] Z=X*(AB)

[0118] Where X is the speed of the electromagnetic wave signal emitted by the robot or the user handle, which is generally 3×10 8 m / s; A is the signal receiving time of the UWB module in the robot or the UWB module in the user handle; B is the signal response time of the UWB module in the robot or the UWB module in the user handle.

[0119] An optional embodiment of the present invention can use the TOF ranging method to calculate the distance between the UWB module in the robot and the UWB module in the user's handle at each frame time node in the motion video. Furthermore, in order to reduce the time error, the bilateral two-way ranging method in the TOF ranging method can be used.

[0120] S6. Calculate the spatial position information of the UWB module in the user handle using a triangulation method according to the spatial position information set and the distance.

[0121] In an embodiment of the present invention, the triangulation positioning method may be a mathematical method that uses two or more detectors to detect the target orientation at different positions, and then uses triangulation geometry principles to determine the position and distance of the target.

[0122] The embodiment of the present invention calculates the spatial position information of the UWB module in the user handle using a triangulation positioning method based on the spatial position information set and the distance, thereby improving the accuracy of the distance between the UWB module in the robot and the UWB module in the user handle, making it easier for the robot to search for the position of the user handle and perform following tasks in the target environment.

[0123] Further, as an optional embodiment of the present invention, the calculating the spatial position information of the UWB module in the user handle by using a triangulation positioning method according to the spatial position information set and the distance includes:

[0124] Taking the spatial position information of any UWB module in the robot in any frame in the spatial position information set as the origin, constructing a spatial coordinate system and obtaining the origin coordinates of the origin;

[0125] Acquire the residual spatial position information of the remaining UWB modules in the robot in any frame in the spatial coordinate system, and the adjacent spatial position information of all UWB modules in the robot in adjacent frames of the any frame;

[0126] Calculate the reference coordinates of the remaining UWB modules in the robot in any frame and all UWB modules in the robot in frames adjacent to any frame in the spatial coordinate system according to the spatial correspondence between the residual spatial position information and the adjacent spatial position information and the origin in the spatial coordinate system;

[0127] Construct a spherical surface according to the origin coordinates, the reference coordinates, and the distance between the UWB module corresponding to each coordinate and the UWB module in the user handle, and use the coordinates of the points through which all spherical surfaces pass as the handle coordinates of the UWB module in the user handle;

[0128] The spatial position information of the UWB module in the user handle is calculated according to the corresponding relationship between the handle coordinates and the origin coordinates.

[0129] In an optional embodiment of the present invention, since positioning in three-dimensional space using the triangulation method requires four or more spherical surfaces, the robot is provided with at least two UWB modules, thereby improving the accuracy of the spatial position information of the UWB module in the user's handle and improving the accuracy of the robot's following.

[0130] S7. Generate a following path for the robot according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle.

[0131] In an optional embodiment of the present invention, in order to provide the robot with an obstacle avoidance function, when performing a robot following task, it is necessary to plan a following path in the initial three-dimensional point cloud map, so as to ensure the safety of the robot and the accuracy of positioning during following.

[0132] Furthermore, in an optional embodiment of the present invention, the user handle may be movable. When the spatial position information of the user handle changes, the robot will generate a new following path based on the initial three-dimensional point cloud map and the latest spatial position information of the UWB module in the user handle, thereby ensuring that the robot can successfully complete the robot following task.

[0133] The embodiment of the present invention obtains image information of the target environment and initial position information of the UWB module in the robot, calculates the corner points in the image information by using a corner point algorithm, and constructs an initial three-dimensional point cloud map of the target environment based on the image information and the corner points by using visual real-time positioning and mapping technology, thereby providing a complete target environment map for the robot, thereby helping the robot to perform obstacle avoidance operations during navigation and travel, and ensuring the safety of the robot when following. Secondly, when a robot following instruction is received, the target environment is recorded by using a camera module in the robot to obtain a motion video, and the motion trajectory of the robot is calculated according to the motion video, and the spatial position information of each frame time node of the UWB module in the robot during the recording process is determined according to the motion trajectory, thereby improving the accuracy of robot positioning, and ensuring that the robot can accurately avoid obstacles and accurately position itself during the following task clock. Furthermore, the spatial position information of the UWB module in the robot of all adjacent frames of the motion video is obtained at preset time intervals within a preset time period to form a first spatial position information set, and the UWB module in the robot is obtained. The signal receiving time and signal response time of the UWB module in the robot and the UWB module in the user handle are calculated according to the signal receiving time and the signal response time, and the distance between the UWB module in the robot and the UWB module in the user handle is determined to determine whether the robot is within the restricted range of the robot following task, thereby ensuring that the robot can successfully complete the robot following task. In addition, by calculating the distance between the UWB module in the robot and the UWB module in the user handle, the accuracy of the robot following is also improved. Finally, according to the spatial position information set and the distance, the spatial position information of the UWB module in the user handle is calculated using the triangulation positioning method, thereby improving the accuracy of the distance between the UWB module in the robot and the UWB module in the user handle, facilitating the robot to search for the position of the user handle, thereby performing the following task in the target environment. Further, according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle, the following path of the robot is generated, thereby completing the robot following task, ensuring that the robot can achieve obstacle avoidance and precise positioning, thereby improving the accuracy of the robot following. Therefore, the robot following method provided by the present invention can improve the accuracy of the robot following.

[0134] like Figure 4 The figure shows a functional module diagram of the robot following device of the present invention.

[0135] The robot following device 100 of the present invention can be installed in a robot. According to the functions to be implemented, the robot following device 100 can include a three-dimensional map construction module 101, a user handle positioning module 102 and a following path generation module 103. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a robot processor and can complete fixed functions, which are stored in the memory of the robot.

[0136] In this embodiment, the functions of each module / unit are as follows:

[0137] The three-dimensional map construction module 101 is used to obtain image information of the target environment and the initial position information of the UWB module in the robot, calculate the corner points in the image information using a corner point algorithm, and construct an initial three-dimensional point cloud map of the target environment based on the image information and the corner points using visual real-time positioning and mapping technology.

[0138] The user handle positioning module 102 is used to, when receiving a robot follow instruction, use the camera module in the robot to record the target environment to obtain a motion video, select the target corner point from the corner points according to the image information of each frame image in the motion video, and calculate the motion trajectory of the robot within any two adjacent frames of the image according to the target corner point in any two adjacent frames of the motion video, determine the spatial position information of each frame time node of the UWB module in the robot during the recording process according to the motion trajectory and the initial position information, obtain the spatial position information of the UWB module in the robot of all adjacent frames of the motion video at a preset time interval within a preset time period to form a first spatial position information set, and obtain the signal reception time and signal response time of the UWB module in the robot and the UWB module in the user handle, calculate the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time, and calculate the spatial position information of the UWB module in the user handle according to the spatial position information set and the distance.

[0139] The following path generating module 103 is used to generate a following path of the robot according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle.

[0140] In detail, each module described in the robot following device 100 in the embodiment of the present invention is used in the same manner as described above. Figures 1 to 3 The robot following method described in the invention has the same technical means and can produce the same technical effects, so I will not go into details here.

[0141] like Figure 5, which is a schematic diagram of the structure of a robot for implementing the robot following method of the present invention.

[0142] The robot may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a robot following program.

[0143] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 11 may be an internal storage unit of the robot in some embodiments, such as a mobile hard disk of the robot. The memory 11 may also be an external storage device of the robot in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. equipped on the robot. Further, the memory 11 may also include both an internal storage unit and an external storage device of the robot. The memory 11 may not only be used to store application software and various types of data installed in the robot, such as the code of the robot following the program, but also be used to temporarily store data that has been output or is to be output.

[0144] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the robot, and uses various interfaces and lines to connect various components of the entire robot, and executes various functions of the robot and processes data by running or executing programs or modules (such as robot following programs, etc.) stored in the memory 11, and calling data stored in the memory 11.

[0145] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The communication bus 12 is configured to realize connection and communication between the memory 11 and at least one processor 10, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0146] Figure 5 Only a robot with components is shown, and those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the robot, and may include fewer or more components than shown, or combine certain components, or arrange the components differently.

[0147] For example, although not shown, the robot may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The robot may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0148] Optionally, the communication interface 13 may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the robot and other robots.

[0149] Optionally, the communication interface 13 may also include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the robot and to display a visual user interface.

[0150] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0151] The robot following program stored in the memory 11 in the robot is a combination of multiple computer programs. When running in the processor 10, it can achieve:

[0152] Obtain image information of the target environment and initial position information of the UWB module in the robot, calculate corner points in the image information using a corner point algorithm, and construct an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology based on the image information and the corner points;

[0153] When receiving a robot follow instruction, a camera module in the robot is used to record the target environment to obtain a motion video;

[0154] According to the image information of each frame of the motion video, a target corner point is selected from the corner points, and according to the target corner points in any two adjacent frames of the motion video, a motion trajectory of the robot within the time of any two adjacent frames of the motion video is calculated;

[0155] Determine the spatial position information of each frame time node of the UWB module in the robot during the video recording process according to the motion trajectory and the initial position information;

[0156] Acquire spatial position information of the UWB module in the robot of all adjacent frames of the motion video at preset time intervals within a preset time period to form a first spatial position information set, and acquire signal reception time and signal response time of the UWB module in the robot and the UWB module in the user handle, and calculate the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time;

[0157] Calculate the spatial position information of the UWB module in the user handle using a triangulation method according to the spatial position information set and the distance;

[0158] A following path of the robot is generated according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle.

[0159] Specifically, the specific implementation method of the processor 10 for the above computer program can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0160] Furthermore, if the module / unit integrated in the robot is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0161] The embodiment of the present invention may also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of a robot, the computer program may implement:

[0162] Obtain image information of the target environment and initial position information of the UWB module in the robot, calculate corner points in the image information using a corner point algorithm, and construct an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology based on the image information and the corner points;

[0163] When receiving a robot follow instruction, a camera module in the robot is used to record the target environment to obtain a motion video;

[0164] According to the image information of each frame of the motion video, a target corner point is selected from the corner points, and according to the target corner points in any two adjacent frames of the motion video, a motion trajectory of the robot within the time of any two adjacent frames of the motion video is calculated;

[0165] Determine the spatial position information of each frame time node of the UWB module in the robot during the video recording process according to the motion trajectory and the initial position information;

[0166] Acquire spatial position information of the UWB module in the robot of all adjacent frames of the motion video at preset time intervals within a preset time period to form a first spatial position information set, and acquire signal reception time and signal response time of the UWB module in the robot and the UWB module in the user handle, and calculate the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time;

[0167] Calculate the spatial position information of the UWB module in the user handle using a triangulation method according to the spatial position information set and the distance;

[0168] A following path of the robot is generated according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle.

[0169] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0170] In the several embodiments provided by the present invention, it should be understood that the disclosed robots, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0171] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0173] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0174] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0175] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A robot following method, characterized in that: The method comprises: Obtain image information of the target environment and initial position information of the UWB module in the robot, calculate corner points in the image information using a corner point algorithm, and construct an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology based on the image information and the corner points; When receiving a robot follow instruction, a camera module in the robot is used to record the target environment to obtain a motion video; According to the image information of each frame of the motion video, a target corner point is selected from the corner points, and according to the target corner points in any two adjacent frames of the motion video, a motion trajectory of the robot within the time of any two adjacent frames of the motion video is calculated; Determine the spatial position information of each frame time node of the UWB module in the robot during the video recording process according to the motion trajectory and the initial position information; Acquire spatial position information of the UWB module in the robot of all adjacent frames of the motion video at preset time intervals within a preset time period to form a first spatial position information set, and acquire signal reception time and signal response time of the UWB module in the robot and the UWB module in the user handle, and calculate the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time; Calculate the spatial position information of the UWB module in the user handle using a triangulation method according to the spatial position information set and the distance; A following path of the robot is generated according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle.

2. The robot following method according to claim 1, characterized in that: The step of calculating the motion trajectory of the robot within the time period of any two adjacent frames of images according to the target corner points in any two adjacent frames of images in the motion video includes: Matching target corner points in any two adjacent frames of the motion video to obtain a matching result; According to the matching result, the rotation vector and the displacement vector between any two adjacent frames of images are calculated using the epipolar geometry formula; Using the least square method, nonlinearly optimizing the rotation vector and the displacement vector to obtain a target rotation vector and a target displacement vector; According to the target rotation vector and the target displacement vector, any two adjacent frames of images are spliced ​​to obtain the motion trajectory of the robot.

3. The robot following method according to claim 1, characterized in that: The step of calculating the corner points in the image information by using a corner point algorithm includes: Acquire pixel information in the image information to obtain a pixel image; Select any pixel point in the pixel image as the center of the circle and draw a circle with a preset length as the radius to obtain a target circle; Determine whether there are a first preset number of continuous comparison pixel points on the target circle; When there are no first preset number of continuous comparison pixel points on the target circle, determining that the pixel point is not a corner point; When there are a first preset number of continuous comparison pixel points on the target circle, calculating the difference between the brightness value of the comparison pixel point and the brightness value of the pixel point; Determine whether there is a second preset number of differences among the differences that are greater than or less than a preset threshold; When there is no second preset number of differences in the differences that are greater than or less than a preset threshold, it is determined that the pixel point is not a corner point; When a second preset number of differences among the differences is greater than or less than a preset threshold, it is determined that the pixel point is a corner point.

4. The robot following method according to claim 1, characterized in that: The initial three-dimensional point cloud map of the target environment is constructed using visual real-time positioning and mapping technology according to the image information and the corner points, including: Performing filtering processing on the image information to obtain denoised image information; Performing a dedistortion process on the denoised image information to obtain target image information; According to the corner points, performing corner point matching on the target image information to obtain a matching result; According to the matching results, the target image information is spliced ​​to obtain an initial three-dimensional point cloud map.

5. The robot following method according to claim 1, characterized in that: Determining the spatial position information of each frame time node of the UWB module in the robot during the video recording process according to the motion trajectory and the initial position information includes: Obtaining the movement speed of the robot and the time interval between two adjacent frames; Calculate the movement distance of the robot within the time interval of two adjacent frames according to the movement trajectory, the movement speed and the time interval; According to the initial position information of the UWB module in the robot and the movement distance of the robot within the time interval of two adjacent frames of images, the spatial position information of the robot at each frame time node is calculated in chronological order.

6. The robot following method according to claim 1, characterized in that: The calculating the spatial position information of the UWB module in the user handle by using a triangulation method according to the spatial position information set and the distance includes: Taking the spatial position information of any UWB module in the robot in any frame in the spatial position information set as the origin, constructing a spatial coordinate system and obtaining the origin coordinates of the origin; Acquire the residual spatial position information of the remaining UWB modules in the robot in any frame in the spatial coordinate system, and the adjacent spatial position information of all UWB modules in the robot in adjacent frames of the any frame; Calculate the reference coordinates of the remaining UWB modules in the robot in any frame and all UWB modules in the robot in frames adjacent to any frame in the spatial coordinate system according to the spatial correspondence between the residual spatial position information and the adjacent spatial position information and the origin in the spatial coordinate system; According to the origin coordinates, the reference coordinates and the distance between the UWB module in the robot and the UWB module in the user handle corresponding to each coordinate, a spherical surface is constructed, and the coordinates of the points through which all spherical surfaces pass are used as the handle coordinates of the UWB module in the user handle; The spatial position information of the UWB module in the user handle is calculated according to the corresponding relationship between the handle coordinates and the origin coordinates.

7. The robot following method according to claim 1, characterized in that: Calculating the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time includes: The distance Z between the UWB module in the robot and the UWB module in the user handle is calculated using the following distance measurement formula: Z=X*(AB) Wherein, X is the speed of the electromagnetic wave signal emitted by the robot or the user handle; A is the signal receiving time of the UWB module in the robot or the UWB module in the user handle; B is the signal response time of the UWB module in the robot or the UWB module in the user handle.

8. A robot following device, characterized in that: The device comprises: A three-dimensional map construction module is used to obtain image information of the target environment and initial position information of the UWB module in the robot, calculate corner points in the image information using a corner point algorithm, and construct an initial three-dimensional point cloud map of the target environment using visual real-time positioning and mapping technology based on the image information and the corner points; A user handle positioning module, which is used to, when receiving a robot follow instruction, use the camera module in the robot to record the target environment to obtain a motion video, select the target corner point from the corner point according to the image information of each frame image in the motion video, and calculate the motion trajectory of the robot within any two adjacent frames of the image according to the target corner point in any two adjacent frames of the image in the motion video, determine the spatial position information of each frame time node of the UWB module in the robot during the recording process according to the motion trajectory and the initial position information, obtain the spatial position information of the UWB module in the robot of all adjacent frames of the motion video at a preset time interval within a preset time period to form a first spatial position information set, and obtain the signal reception time and signal response time of the UWB module in the robot and the UWB module in the user handle, calculate the distance between the UWB module in the robot and the UWB module in the user handle according to the signal reception time and the signal response time, and calculate the spatial position information of the UWB module in the user handle according to the spatial position information set and the distance by using a triangulation positioning method; A following path generation module is used to generate a following path for the robot according to the initial three-dimensional point cloud map and the spatial position information of the UWB module in the user handle.

9. A robot, characterized in that: The robot comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions executable by the at least one processor, wherein the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the robot following method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot following method according to any one of claims 1 to 7 is implemented.

Images