Robot, following method and apparatus for robot, and storage medium
By combining the target image, depth map and radar information, relocking and planning the motion path, the problem of missing targets in the robot following the task is solved, and the robot accurately follows the target.
Patent Information
- Application Number
- PCT/CN2024/084455
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2024-03-28
- Publication Date
- 2024-12-05
AI Technical Summary
Existing robot following schemes are prone to losing the following target object, resulting in the inability to continue to perform follow-up tasks.
By combining the target image, depth map and radar information, the object information is determined and matched with the historical information. If the target is lost, it will be re-locked, joined to the target queue, plan the motion path and sample the trajectory point to determine the motion speed, and achieve the goal follow.
It effectively reduces the probability of losing the following target due to occlusion, and can automatically restore the following task, ensuring that the robot accurately and effectively follows the target object.
Smart Images

Figure CN2024084455_05122024_PF_FP_ABST
Abstract
Description
Robot, robot following method, device and storage medium
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on May 31, 2023, with application number 2023106379307 and application name “Robot, Robot Following Method, Device and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field
[0003] The present application relates to the field of robotics, and in particular to a robot, a robot following method, a device, and a storage medium. Background Art
[0004] With the continuous development of robotics technology, robots are widely used in various application scenarios, such as food delivery and home services. In some practical applications, robots are required to follow a target. Existing robot following solutions mainly rely on vision-based automatic following. For example, the robot's camera captures image information of the target object, analyzes this image information to determine the target object's motion trajectory, and then predicts the robot's motion path for following.
[0005] However, in existing robot following solutions, it is easy to lose the following target object, making it impossible to continue the following task.
[0006] Summary of the Invention
[0007] According to various embodiments of the present application, a robot, a robot following method, an apparatus, and a storage medium are provided.
[0008] A robot comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the robot performs the following steps:
[0009] Determine object information of multiple objects based on a first target image, a first depth map, and first radar information; match the object information with historical object information to obtain a matching result; the historical object information is information corresponding to a target object locked historically; if the matching result indicates that the multiple objects do not include the target object, re-lock the target object; if the matching result indicates that the multiple objects include the target object, add the object information of the target object to a target queue to obtain a motion trajectory of the target object; plan the motion path of the robot according to the current state of the robot and the motion trajectory; sample the motion path, and determine the movement speed of the robot at each trajectory point based on the distance between each trajectory point obtained by sampling, and follow the target object to move at the movement speed.
[0010] A robot following method, the method comprising:
[0011] Determine object information of multiple objects based on a first target image, a first depth map, and first radar information; match the object information with historical object information to obtain a matching result; the historical object information is information corresponding to a target object locked historically; if the matching result indicates that the multiple objects do not include the target object, re-lock the target object; if the matching result indicates that the multiple objects include the target object, add the object information of the target object to a target queue to obtain a motion trajectory of the target object; plan the motion path of the robot according to the current state of the robot and the motion trajectory; sample the motion path, and determine the movement speed of the robot at each trajectory point based on the distance between each trajectory point obtained by sampling, and follow the target object to move at the movement speed.
[0012] A robot following device, comprising:
[0013] a first determining module, configured to determine object information of a plurality of objects based on the first target image, the first depth map, and the first radar information;
[0014] A matching module, configured to match the object information with historical object information to obtain a matching result; the historical object information is information corresponding to the target object locked in the past;
[0015] a locking module, configured to re-lock the target object if the matching result indicates that the multiple objects do not include the target object;
[0016] an adding module, configured to add the object information of the target object to a target queue if the matching result indicates that the multiple objects include the target object, and obtain a motion trajectory of the target object;
[0017] A planning module, configured to plan a motion path of the robot according to the current state of the robot and the motion trajectory;
[0018] The following module is used to sample the motion path, and determine the movement speed of the robot at each trajectory point based on the distance between each trajectory point obtained by sampling, and follow the target object to move at the movement speed.
[0019] A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the robot following method when executed by a processor.
[0020] A computer program product includes a computer program, which implements the steps of the robot following method when executed by a processor.
[0021] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, without paying any creative work, they can also obtain drawings of other embodiments based on these drawings.
[0023] FIG1 is a diagram illustrating an application environment of a robot following method according to an embodiment;
[0024] FIG2 is a schematic flow chart of a robot following method according to an embodiment;
[0025] FIG3 is a schematic diagram of a motion trajectory of a target object in one embodiment;
[0026] FIG4 is a schematic diagram of a robot's following trajectory in one embodiment;
[0027] FIG5 is a schematic flow chart of a robot following method in another embodiment;
[0028] FIG6 is a schematic diagram of a process of data fusion and target tracking in one embodiment;
[0029] FIG7 is a block diagram of a robot following device according to an embodiment;
[0030] FIG8 is a structural block diagram of a robot following device in another embodiment;
[0031] FIG9 is a diagram showing the internal structure of a robot in one embodiment. DETAILED DESCRIPTION
[0032] To facilitate understanding of the present application, a more comprehensive description of the present application will be provided below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present application.
[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the invention pertains. The terms used in the specification of the invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0034] It should be noted that in the following description, the terms "first", "second" and "third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first", "second" and "third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application can be implemented in an order other than that shown or described.
[0035] The robot following methods provided in various embodiments of the present application can be applied in the application environment shown in FIG1 . Here, robot 102 communicates with server 104 via a network. A data storage system can store data that server 104 needs to process. The data storage system can be integrated with server 104 or placed in the cloud or on another network server. Server 104 can store new functional plug-ins or programs for robot 102, as well as various system versions. When the latest system version is available, it can be distributed to robot 102 for system updates.
[0036] The robot 102 includes but is not limited to a food delivery robot, a guest welcoming robot, a cleaning robot, and other types of robots, etc. The cleaning robot may be a sweeping robot, a sweeping and mopping robot, or other types of robots with cleaning functions.
[0037] Server 104 can be a standalone physical server or a service node in a blockchain system. Each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP). Furthermore, server 104 can be a server cluster consisting of multiple physical servers. It can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0038] The robot 102 and the server 104 can be connected via Bluetooth, Universal Serial Bus (USB) or network communication connection methods, which are not limited in this application.
[0039] In one embodiment, as shown in FIG2 , a robot following method is provided, which is described by taking the method applied to the robot in FIG1 as an example, and includes the following steps:
[0040] S202 : Determine object information of a plurality of objects based on a first target image, a first depth map, and first radar information.
[0041] The first target image may be an image captured by a conventional camera during the robot's motion, such as a red, green, and blue (RGB) image. The first depth map may be an image containing depth information captured by a depth camera during the robot's motion, such as an RGB-D image. The first radar information may be radar information captured by a lidar during the robot's motion.
[0042] The object represents a human object within the robot's field of view, such as a pedestrian. The object information can be attribute information of the object, such as the object's feature value and position information (i.e., the object's position and posture).
[0043] In one embodiment, the robot first receives the voice command issued by the user through the microphone array, and then recognizes the voice command. When the recognition is successful, the robot is awakened to the following task. The microphone array will return the sound source angle information, and the following task will be entered at this time.
[0044] In one embodiment, the robot can also recognize the follow gesture made by the user through an image recognition sensor such as a camera, and then match and recognize the gesture instruction. When the recognition is successful, the robot's follow task is awakened, the camera will return the user's location information, and then enter the follow task.
[0045] In one embodiment, the user can also input follow-up task control instructions through the robot's touch screen, physical buttons, remote control, and APP on the terminal to wake up the robot's follow-up task.
[0046] Specifically, step S202 includes:
[0047] The robot obtains a first target image, a first depth map, and first radar information collected during movement; performs object detection on each object in the first target image to obtain first object initial information;
[0048] Performing obstacle detection on the first depth map and the first radar information to obtain first obstacle coordinates corresponding to the first depth map and the first radar information respectively;
[0049] The first object initial information and the coordinates of each first obstacle are fused to obtain first fused data, and object information of each object is determined based on the first fused data.
[0050] The first obstacle coordinates are the coordinates of each obstacle within the robot's field of view. The target object refers to the object that the robot is to follow, that is, the user or operator who issues the robot a follow-up task control command (voice command, gesture command, remote control command, etc.).
[0051] In one embodiment, before performing data fusion, the robot may also first perform coordinate conversion on the object coordinates of each object in the first object initial information to obtain the first world coordinates of each object in the robot navigation system; and perform coordinate conversion on the first obstacle coordinates corresponding to each of the first depth map and the first radar information to obtain the second world coordinates and third world coordinates of each obstacle in the robot navigation system. The robot then fuses the feature values, first world coordinates, second world coordinates, and third world coordinates in the first object initial information.
[0052] For example, the robot uses a deep learning method to capture RGB images through an RGB camera, then extracts all pedestrians from the RGB images, and then converts the corresponding coordinates of these pedestrians into world coordinates under the robot navigation system, obtaining the coordinates of pedestrians within the field of view of the robot navigation system and the corresponding eigenvalues. An RGB-D depth camera is used to capture a depth map, and the coordinates of obstacles within the field of view are obtained based on the depth map, and then converted into world coordinates under the robot navigation system. A single-line laser radar is used to obtain the coordinates of obstacles within the field of view, and then converted into obstacle coordinates under the robot navigation system. Then, the world coordinates of pedestrians within the field of view, the corresponding eigenvalues, and the world coordinates of obstacles within the field of view are fused to obtain the object information of multiple pedestrians within the field of view in the world coordinate system. At this time, S204 is then executed, such as matching the object information with the previously locked target object to find the previously locked target object. After finding the locked target object, the object information of the target object is updated and added to the Kalman filter.
[0053] During the target locking stage, the robot obtains the sound source angle of each object when making a speech, as well as the second target image, second depth map and second radar information collected during the movement; performs object detection on each object in the second target image to obtain the second object initial information; performs coordinate conversion on the object coordinates of each object in the second object initial information to obtain the fourth world coordinate of each object under the robot navigation system; performs obstacle detection on the second depth map and second radar information to obtain the second obstacle coordinates corresponding to the second depth map and the second radar information; performs coordinate conversion on the second obstacle coordinates corresponding to the second depth map and the second radar information respectively to obtain the fifth world coordinate and the sixth world coordinate of each obstacle under the robot navigation system; converts the sound source angle into a sound source direction vector under the robot navigation system, and determines the seventh world coordinate of each object under the robot navigation system based on the sound source direction vector; fuses the eigenvalue, fourth world coordinate, fifth world coordinate, sixth world coordinate and seventh world coordinate in the second object initial information to obtain second fused data; locks the target object based on the second fused data to obtain historical object information of the target object; the historical object information includes the eigenvalue and posture information of the target object, and the posture information includes the world coordinate and posture under the robot navigation system.
[0054] The second target image, second depth map, and second radar information are acquired before the first target image, first depth map, and first radar information, respectively. Similarly, the second target image can be an image acquired by a conventional camera during the robot's motion, such as an RGB image. The second depth map can be an image containing depth information acquired by a depth camera during the robot's motion, such as an RGB-D image. The second radar information can be radar information acquired by a lidar during the robot's motion.
[0055] The second obstacle coordinates are the coordinates of each obstacle within the robot's field of view.
[0056] For example, the robot obtains the sound source angle in the microphone coordinate system returned by the microphone array and then converts this sound source angle into the direction vector of the sound source for the robot's navigation system. Using deep learning methods, the robot captures an RGB image with an RGB camera. It then extracts all pedestrians from the RGB image, obtains pedestrian information and its eigenvalues, and constructs a right triangle based on the RGB camera's intrinsic and extrinsic parameters, the camera-to-ground distance, and the imaging epipolar line. This is used to obtain the coordinates of pedestrians within the field of view in the RGB camera coordinate system, which are then converted into the world coordinates of the pedestrian within the robot's navigation system. An RGB-D depth camera is used to obtain the coordinates of obstacles observed within the field of view and convert them into their world coordinates within the robot's navigation system. A single-line laser radar is used to obtain the coordinates of obstacles observed within the field of view and then convert them into their world coordinates within the robot's navigation system. Based on the information obtained by these four sensors, the robot integrates the sound source direction vector, pedestrian information detected by the RGB camera, the world coordinates of obstacles detected by the RGB-D camera, and the world coordinates of obstacles detected by the single-line laser radar. Ultimately, the robot locates a target object to be followed and obtains the coordinates of the target object in the world coordinate system and its eigenvalues. If the locking fails, the robot will wait for user instructions again through voice prompts to activate the robot's following task.
[0057] S204: Match the object information with the historical object information to obtain a matching result.
[0058] The historical object information is information corresponding to the target object locked historically.
[0059] Specifically, the robot calculates the similarity between each object information and the historical object information, and determines the matching result between each object information and the historical object information according to the similarity.
[0060] S206: If the matching result indicates that the multiple objects do not include the target object, re-lock the target object.
[0061] Among them, the matching result indicates that multiple objects do not contain the target object, indicating that the target following fails.
[0062] Specifically, if the matching result indicates that the multiple objects do not include the target object, the robot issues a prompt message; the prompt message is used to prompt the user object to reissue the follow-up task control instruction, and the robot re-locks the target object based on the control instruction.
[0063] S208: If the matching result indicates that the target object is included in the plurality of objects, the object information of the target object is added to the target queue to obtain the motion trajectory of the target object.
[0064] Specifically, if each matching result indicates that multiple objects include the target object, the world coordinates in the object information of the target object obtained each time are added to the target queue, and the motion trajectory of the target object can be obtained according to the world coordinates in the target queue.
[0065] S210 , planning the motion path of the robot according to the current state and motion trajectory of the robot.
[0066] The motion path represents the robot's intended route at the next moment. The current state includes the robot's motion direction, current position, linear velocity, angular velocity, linear acceleration, and angular acceleration, or a motion state calculated based on the robot's motion direction, current position, linear velocity, angular velocity, linear acceleration, and angular acceleration.
[0067] In one embodiment, the robot can determine its current state in the following manner: if the matching result indicates that multiple objects include a target object, the robot inputs the target object's object information into a target filter, so that the target filter predicts the target object's position based on the object information, obtaining a predicted position of the target object at the next moment; the robot determines its motion direction based on the predicted position and the target object's current position; the current position is the first world coordinate of the target object in the robot's navigation system; and the robot's current state is determined based on the motion direction, velocity information, and current position. The target filter can be, but is not limited to, a Kalman filter.
[0068] S212: Sampling the motion path, and determining the motion speed of the robot at each trajectory point based on the distance between each trajectory point obtained by sampling, and following the target object to move according to the motion speed.
[0069] For planning the robot's motion path, the specific steps include: the robot samples the motion trajectory and fits the sampled trajectory points to obtain a following trajectory; according to the robot's current state and the following trajectory, the robot's motion path is planned.
[0070] Among them, the robot's motion path planning is divided into two parts: the first part samples the historical motion trajectory of the target object to obtain the following trajectory; the second part obtains the robot's motion path based on the robot's current state and the following trajectory.
[0071] For the sampling process of the motion trajectory, the specific steps include: the robot samples the motion trajectory according to a first sampling frequency to obtain at least two initial sampling points; when the distance between adjacent initial sampling points in the at least two initial sampling points is greater than a first preset distance, the adjacent initial sampling points are added to a sampling queue.
[0072] In addition, when the distance between adjacent initial sampling points among at least two initial sampling points is less than or equal to the first preset distance, the first sub-motion trajectory is sampled according to the second sampling frequency to obtain a first sampling point; wherein the second sampling frequency is greater than the first sampling frequency, and the first sub-motion trajectory is the motion trajectory between adjacent initial sampling points; when the distance between the first sampling point and the at least two initial sampling points is greater than the second preset distance, the at least two initial sampling points and the first sampling point are added to the sampling queue; when the distance between the first sampling point and the at least two initial sampling points is less than or equal to the second preset distance, the adjacent initial sampling point is added to the sampling queue, and the second sub-motion trajectory is sampled according to the third sampling frequency to obtain a second sampling point; when the distance between the second sampling point and the first sampling point and the first initial sampling point is greater than the third preset distance, the second sampling point is added to the sampling queue; wherein the third sampling frequency is greater than the second sampling frequency, and the second sub-motion trajectory is the motion trajectory between the first sampling point and the first initial sampling point among the adjacent initial sampling points, as shown in FIG. 3 .
[0073] For example, set the resolution to K meters / second and the maximum sampling frequency to J times / second; select the motion trajectory within each second in the motion trajectory of the following target, and then select the sampling frequency to be 1 time / second, then the corresponding sampling results are Ta and Tb, the points sampled at the two sampling times (i.e., sampling points) are A and B, assuming that the Euclidean distance between points AB is greater than K meters, then Ta and Tb are added to the sampling queue, otherwise the sampling frequency is doubled, at this time the sampling frequency is 2 times / second, the sampling results are Ta, Tc, and Tb. The sampling points at the three times are A, C, and B, and the Euclidean distances of AC and CB are calculated respectively. If the Euclidean distance is greater than K / 2, then Add the three sampling points A, C, and B to the sampling queue, otherwise the sampling frequency is doubled; assuming that the Euclidean distance of AC is greater than K / 2 and the Euclidean distance of CB is less than K / 2, then Ta and Tc are added to the sampling queue, and Ta and Tc added to the sampling queue are not sampled; for Tc and Tb, the sampling frequency is doubled to 4 times / second, and Tc and Tb are sampled. The sampling results are Tc, Td, and Tb. The sampling points at these three times are C, D, and B. Calculate the Euclidean distance of CD and DB respectively, determine the size with K / 4, and repeat the above operation until the sampling frequency reaches the maximum sampling frequency, then stop the above sampling process.
[0074] Then, the sampling points in the sampling queue are fitted into a straight line using the least squares method. If the fitted straight line is opposite to the final direction and is short, the straight line is discarded and the points are added to the next time period for further fitting. Finally, the following trajectory generated after sampling is obtained, as shown in Figure 4, where the black dots on the line are sampling points.
[0075] In the above embodiment, object information of multiple objects is determined based on a first target image, a first depth map, and first radar information; the object information is matched with the historical object information of a historically locked target object to obtain a matching result. Using multiple sensors for target tracking can effectively reduce the probability of losing the tracking target due to occlusion in a certain direction. Furthermore, if the matching result indicates that the multiple objects do not include the target object, the target object is re-locked, so that even when the tracking target is lost, the following task can be automatically resumed, avoiding the problem of being unable to continue the following task due to occlusion. Furthermore, if the matching result indicates that the multiple objects include the target object, the object information of the target object is added to the target queue to obtain the motion trajectory of the target object; the robot's motion path is planned based on the robot's current state and motion trajectory; the motion path is sampled, and the robot's motion speed at each trajectory point is determined based on the distance between each trajectory point obtained by the sampling, and the target object is followed according to the motion speed, thereby accurately and effectively following the target object and performing the following task.
[0076] In order to more clearly illustrate the technical solution of the present application, a specific embodiment is described in conjunction with FIG5 and FIG6 , as follows:
[0077] (1) Voice wake-up of the robot
[0078] The microphone array recognizes the voice command. When the recognition is successful, the microphone array will return the sound source angle. At this time, the robot is awakened and enters the following task.
[0079] (2) Target locking on objects within the field of view
[0080] According to the sound source angle returned by the microphone array (the coordinate in the microphone coordinate system), it is converted into the world coordinate in the robot navigation system to obtain the direction vector of the sound source in the world coordinate system.
[0081] Using deep learning methods, an RGB camera is used to detect objects within the field of view, obtain object information and its eigenvalues, and construct a right triangle based on the intrinsic and extrinsic parameters of the RGB camera, the distance from the camera to the ground, and the imaging epipolar line. This is used to obtain the coordinates of the objects within the field of view in the RGB camera coordinate system, which are then converted into world coordinates under the robot navigation system to obtain the coordinates of the objects within the field of view in the world coordinate system.
[0082] Use an RGB-D depth camera to obtain the coordinates of obstacles observed within the field of view, convert them into world coordinates for the robot navigation system, and obtain the coordinates of obstacles within the field of view in the world coordinate system. Similarly, use a single-line lidar to obtain the coordinates of obstacles observed within the field of view, convert them into world coordinates for the robot navigation system, and obtain the coordinates of obstacles within the field of view in the world coordinate system.
[0083] The four processed information are fused together to obtain fused data: the sound source direction vector, the object coordinates detected by the RGB camera, the obstacle coordinates detected by the RGB-D camera, and the obstacle coordinates detected by the single-line lidar. The coordinates of the target object in the world coordinate system and its eigenvalues are obtained based on the fused data.
[0084] If the lock fails, return to the above step (1) and wait for wake-up again through the robot's voice prompt.
[0085] (3) Follow the target object
[0086] As shown in FIG6 , the specific process of following the target object is as follows:
[0087] Using deep learning methods, all objects in the RGB image captured by the RGB camera are extracted and converted into world coordinates under the robot navigation system, obtaining the object coordinates and corresponding feature values within the field of view in the world coordinate system;
[0088] Use an RGB-D depth camera to obtain the coordinates of obstacles within the field of view in the world coordinate system;
[0089] Use a single-line laser radar to obtain the coordinates of obstacles within the field of view in the world coordinate system.
[0090] By fusing the above three data information, the target information of multiple objects within the field of view in the world coordinate system can be obtained. Then, it is matched with the target object locked in step (2). According to the corresponding eigenvalues and posture information, the locked target object is found, and the information of the follow-up target object (i.e., target information) is updated and added to the Kalman filter.
[0091] (4) Generate motion trajectory for the target object to follow
[0092] All target information obtained by following in step (3) is added to the queue to obtain the motion trajectory of the target object.
[0093] (5) Predict the position and movement direction of the target object
[0094] Use the Kalman filter in step (3) to obtain the possible position of the target object at the next moment, and then obtain the motion direction vector at the next time by combining the predicted position and the current position.
[0095] (6) Generate motion path for the robot
[0096] The robot's motion path is divided into two parts:
[0097] In the first part, the target object's trajectory is sampled to obtain a tracking trajectory. Specifically, the target object's trajectory is sampled based on the sampling time. The principle is that the greater the distance moved within the time, the faster the speed, the lower the sampling frequency, and vice versa. Using the binary approach, further sampling is performed for intervals with smaller distances.
[0098] Set the resolution to K meters / second and the maximum sampling frequency to J times / second; select historical information within one second and the sampling frequency to be 1 time / second, then the sampling results are Ta and Tb. For the sampling points A and B at the two sampling times, if the Euclidean distance between AB is greater than K meters, then Ta and Tb are added to the sampling queue. Otherwise, the sampling frequency is doubled. At this time, the sampling frequency is 2 times / second, and the sampling results are Ta, Tc, and Tb. For the sampling points A, C, and B at the three times, calculate the Euclidean distance between AC and CB respectively. If the Euclidean distance is greater than K / 2, then add Sampling queue, otherwise the sampling frequency is doubled; assuming that the Euclidean distance of AC is greater than K / 2 and the Euclidean distance of CB is less than K / 2, then Ta and Tc are added to the sampling queue, the sampling frequency is doubled, and the sampling frequency is 4 times / second. Ta and Tc added to the sampling queue are not sampled, and Tc and Tb are sampled. The sampling results are Tc, Td, and Tb. The Euclidean distances of CD and DB at the three sampling points C, D, and B are calculated respectively to determine the size with K / 4. Repeat the above operation until the sampling frequency reaches the maximum sampling frequency.
[0099] The sampling points in the sampling queue are fitted into a straight line using the least squares method. If the fitted straight line is opposite to the final direction and is short, the straight line is discarded and the sampling points are added to the next time period for further fitting, ultimately obtaining the tracking trajectory generated after sampling.
[0100] The second part is to obtain the robot's motion path based on its current state and the trajectory it follows. Specifically, this method uses sampling to calculate all possible paths the robot can move within a second based on its current linear velocity, angular velocity, linear acceleration, and angular acceleration. Then, using obstacle constraints and trajectory constraints, it calculates a trajectory that best matches the trajectory it follows.
[0101] (7) Control the robot's motion
[0102] The above trajectory is sampled into N trajectory points according to time t, and the linear velocity and angular velocity are calculated based on the distance between the trajectory points as the instantaneous sending speed; the distance to the target object is continuously calculated. If the distance between the target object and the robot is too close, the speed is sent to zero to avoid collision with the target object.
[0103] (8) Re-lock the target object
[0104] When the target is lost or switched, the robot will issue a prompt and the user can reset the robot's status through voice commands and re-enter the target locking process to improve the fault tolerance rate.
[0105] The technical solution of this application has the following technical effects:
[0106] 1) Using voice signal wakeup as a trigger and reset mechanism for the following task can improve the practicality and fault tolerance of the solution. In some unavoidable scenarios, such as when an obstacle suddenly appears in the middle of the following target and causes the target to be lost, it can serve as a more natural recovery method.
[0107] 2) Compared with the general single-target tracking solution, the multi-target tracking solution has a lower probability of false detection leading to incorrect tracking targets;
[0108] 3) Based on the fusion of multi-sensor data, the robot can automatically follow the control scheme according to the fusion results, which is more accurate;
[0109] 4) By optimizing the sampling and processing method of the motion trajectory, the motion trajectory of the target can be restored to a large extent.
[0110] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.
[0111] Based on the same inventive concept, embodiments of the present application also provide a robot following device for implementing the aforementioned robot following method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more robot following device embodiments provided below can be found in the above-described limitations of the robot following method and will not be further elaborated here.
[0112] In one embodiment, as shown in FIG7 , a robot following device is provided, comprising: a first determining module 702 , a matching module 704 , a locking module 706 , a joining module 708 , a planning module 710 , and a following module 712 , wherein:
[0113] A first determining module 702 is configured to determine object information of a plurality of objects based on the first target image, the first depth map, and the first radar information;
[0114] Matching module 704 is used to match the object information with the historical object information to obtain a matching result; the historical object information is the information corresponding to the target object locked in the past;
[0115] A locking module 706 is configured to relock the target object if the matching result indicates that the multiple objects do not include the target object;
[0116] An adding module 708 is configured to add the target object's object information to a target queue to obtain a motion trajectory of the target object if the matching result indicates that the multiple objects include the target object;
[0117] Planning module 710, for planning the robot's motion path based on the robot's current state and motion trajectory;
[0118] The following module 712 is used to sample the motion path, and determine the motion speed of the robot at each trajectory point based on the distance between each trajectory point obtained by sampling, and follow the target object according to the motion speed.
[0119] In one embodiment, the first determination module 702 is further used to obtain a first target image, a first depth map, and first radar information collected during the movement process; perform object detection on each object in the first target image to obtain first object initial information; perform obstacle detection on the first depth map and the first radar information to obtain first obstacle coordinates corresponding to the first depth map and the first radar information; the first obstacle coordinates are the coordinates of each obstacle within the robot's field of view; perform data fusion on the first object initial information and each first obstacle coordinate to obtain first fused data; and determine the object information of each object based on the first fused data.
[0120] In one embodiment, as shown in FIG8 , the apparatus further comprises:
[0121] The first conversion module 714 is configured to perform coordinate conversion on the object coordinates of each object in the first object initial information to obtain a first world coordinate of each object in the robot navigation system; and to perform coordinate conversion on the first obstacle coordinates corresponding to each of the first depth map and the first radar information to obtain a second world coordinate and a third world coordinate of each obstacle in the robot navigation system.
[0122] The first determining module 702 is further configured to fuse the feature value, the first world coordinate, the second world coordinate, and the third world coordinate in the initial information of the first object.
[0123] In one embodiment, as shown in FIG8 , the apparatus further comprises:
[0124] An acquisition module 716 is configured to acquire the sound source angle of each object when it utters a sound, as well as a second target image, a second depth map, and second radar information collected during the movement; the second target image, the second depth map, and the second radar information are collected before the first target image, the first depth map, and the first radar information, respectively;
[0125] A second conversion module 718 is configured to convert the sound source angle into a sound source direction vector in the robot navigation system, and determine the seventh world coordinate of each object in the robot navigation system according to the sound source direction vector;
[0126] a detection module 720 configured to perform object detection on each object in the second target image to obtain initial information of the second object;
[0127] The second conversion module 718 is further configured to perform coordinate conversion on the object coordinates of each object in the second object initial information to obtain the fourth world coordinates of each object under the robot navigation system;
[0128] The detection module 720 is further configured to perform obstacle detection on the second depth map and the second radar information to obtain second obstacle coordinates corresponding to the second depth map and the second radar information respectively;
[0129] The second conversion module 718 is further configured to perform coordinate conversion on the second obstacle coordinates corresponding to the second depth map and the second radar information, respectively, to obtain fifth world coordinates and sixth world coordinates of each obstacle in the robot navigation system; wherein the second obstacle coordinates are the coordinates of each obstacle within the robot's field of view;
[0130] a fusion module 722 for fusing the feature value, the fourth world coordinate, the fifth world coordinate, the sixth world coordinate, and the seventh world coordinate in the initial information of the second object to obtain second fused data;
[0131] The locking module 706 is further configured to lock the target object based on the second fused data and obtain historical object information of the target object; the historical object information includes feature values of the target object and world coordinates in the robot navigation system.
[0132] In one embodiment, the first determination module 702 is further used to input the object information of the target object into the target filter if the matching result indicates that multiple objects include the target object, so that the target filter performs position prediction based on the object information to obtain the predicted position of the target object at the next moment; determine the movement direction based on the predicted position and the current position of the target object; the current position is the first world coordinate of the target object under the robot navigation system; and determine the current state of the robot based on the movement direction, speed information and current position.
[0133] In one embodiment, the planning module 710 is further configured to sample the motion trajectory and fit the sampled trajectory points to obtain a following trajectory; and plan the motion path of the robot according to the current state of the robot and the following trajectory.
[0134] In one embodiment, the following module 712 is further configured to sample the motion trajectory according to a first sampling frequency to obtain at least two initial sampling points; when a distance between adjacent initial sampling points among the at least two initial sampling points is greater than a first preset distance, the adjacent initial sampling points are added to a sampling queue.
[0135] In one embodiment, the following module 712 is further configured to, when a distance between adjacent initial sampling points among at least two initial sampling points is less than or equal to a first preset distance, sample the first sub-motion trajectory according to a second sampling frequency to obtain a first sampling point; wherein the second sampling frequency is greater than the first sampling frequency, and the first sub-motion trajectory is a motion trajectory between adjacent initial sampling points; when the distance between the first sampling point and the at least two initial sampling points is greater than the second preset distance, add the at least two initial sampling points and the first sampling point to a sampling queue; when the distance between the first sampling point and the at least two initial sampling points is less than or equal to the second preset distance, add the adjacent initial sampling point to the sampling queue, and sample the second sub-motion trajectory according to a third sampling frequency to obtain a second sampling point; when the distance between the second sampling point and the first sampling point and the first initial sampling point is greater than the third preset distance, add the second sampling point to the sampling queue; wherein the third sampling frequency is greater than the second sampling frequency, and the second sub-motion trajectory is a motion trajectory between the first sampling point and the first initial sampling point among the adjacent initial sampling points.
[0136] In one embodiment, the locking module 706 is further configured to issue a prompt message if the matching result indicates that multiple objects do not include the target object; the prompt message is configured to prompt the user object to issue a follow-up task control instruction and re-lock the target object based on the follow-up task control instruction.
[0137] In the above embodiment, object information of multiple objects is determined based on a first target image, a first depth map, and first radar information; the object information is matched with the historical object information of a historically locked target object to obtain a matching result. Using multiple sensors for target tracking can effectively reduce the probability of losing the tracking target due to occlusion in a certain direction. Furthermore, if the matching result indicates that the multiple objects do not include the target object, the target object is re-locked, so that even when the tracking target is lost, the following task can be automatically resumed, avoiding the problem of being unable to continue the following task due to occlusion. Furthermore, if the matching result indicates that the multiple objects include the target object, the object information of the target object is added to the target queue to obtain the motion trajectory of the target object; the robot's motion path is planned based on the robot's current state and motion trajectory; the motion path is sampled, and the robot's motion speed at each trajectory point is determined based on the distance between each trajectory point obtained by the sampling, and the target object is followed according to the motion speed, thereby accurately and effectively following the target object and performing the following task.
[0138] Each module in the aforementioned robot following device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the robot's processor in hardware form, or stored in the robot's memory in software form, so that the processor can call and execute the corresponding operations of each module.
[0139] In one embodiment, a robot is provided, the internal structure of which is shown in Figure 9. The robot includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The robot's processor is used to provide computing and control capabilities. The robot's memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The robot's input / output interface is used to exchange information between the processor and external devices. The robot's communication interface is used to communicate with the outside world via wired or wireless communication, with the wireless communication being achieved via Wi-Fi, a mobile cellular network, NFC, or other technologies. When executed by the processor, the computer program implements a robot following method. The display unit of the robot is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the robot can be a touch layer covering the display screen, or a button, trackball or touchpad set on the robot shell, or an external keyboard, touchpad or mouse, etc.
[0140] Those skilled in the art should understand that the structure shown in Figure 9 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the robot to which the scheme of the present application is applied. The specific robot may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0141] In one embodiment, a robot is provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: determining object information of multiple objects based on a first target image, a first depth map, and first radar information; matching the object information with historical object information to obtain a matching result; the historical object information is information corresponding to a target object locked historically; if the matching result indicates that the multiple objects do not include the target object, re-locking the target object; if the matching result indicates that the multiple objects include the target object, adding the object information of the target object to a target queue to obtain a motion trajectory of the target object; planning the motion path of the robot according to the current state and motion trajectory of the robot; sampling the motion path, and determining the motion speed of the robot at each trajectory point based on the distance between each trajectory point obtained by sampling, and following the target object according to the motion speed.
[0142] In one embodiment, when the processor determines the object information of multiple objects based on the first target image, the first depth map and the first radar information, the processor specifically performs the following steps: obtaining the first target image, the first depth map and the first radar information collected during the movement process; performing object detection on each object in the first target image to obtain first object initial information; performing obstacle detection on the first depth map and the first radar information to obtain first obstacle coordinates corresponding to the first depth map and the first radar information; the first obstacle coordinates are the coordinates of each obstacle within the field of view of the robot; performing data fusion on the first object initial information and the first obstacle coordinates to obtain first fused data; and determining the object information of each object based on the first fused data.
[0143] In one embodiment, when the processor executes the computer program, it also performs the following steps: performing coordinate conversion on the object coordinates of each object in the initial information of the first object to obtain the first world coordinates of each object under the robot navigation system; performing coordinate conversion on the first obstacle coordinates corresponding to the first depth map and the first radar information respectively to obtain the second world coordinates and third world coordinates of each obstacle under the robot navigation system; and fusing the eigenvalues, first world coordinates, second world coordinates and third world coordinates in the initial information of the first object.
[0144] In one embodiment, when the processor executes the computer program, it also performs the following steps: obtaining the sound source angle when each object makes a speech, and the second target image, the second depth map and the second radar information collected during the movement; the second target image, the second depth map and the second radar information are respectively collected before the first target image, the first depth map and the first radar information; performing object detection on each object in the second target image to obtain the second object initial information; performing coordinate conversion on the object coordinates of each object in the second object initial information to obtain the fourth world coordinates of each object under the robot navigation system; performing obstacle detection on the second depth map and the second radar information to obtain the second obstacle coordinates corresponding to the second depth map and the second radar information; performing obstacle detection on the second depth map and the second radar information to obtain the second obstacle coordinates corresponding to the second depth map and the second radar information respectively .... The second obstacle coordinates corresponding to the two depth maps and the second radar information are converted into coordinates to obtain the fifth world coordinates and sixth world coordinates of each obstacle under the robot navigation system; wherein the second obstacle coordinates are the coordinates of each obstacle within the robot's field of view; the sound source angle is converted into a sound source direction vector under the robot navigation system, and the seventh world coordinate of each object under the robot navigation system is determined according to the sound source direction vector; the eigenvalue, fourth world coordinate, fifth world coordinate, sixth world coordinate and seventh world coordinate in the initial information of the second object are fused to obtain second fused data; the target object is locked based on the second fused data to obtain historical object information of the target object; the historical object information includes the eigenvalue and world coordinate of the target object under the robot navigation system.
[0145] In one embodiment, when the processor executes the computer program, it also performs the following steps: if the matching result indicates that multiple objects include a target object, the object information of the target object is input into the target filter, so that the target filter performs position prediction based on the object information to obtain the predicted position of the target object at the next moment; the movement direction is determined based on the predicted position and the current position of the target object; the current position is the first world coordinate of the target object under the robot navigation system; and the current state of the robot is determined based on the movement direction, speed information and current position.
[0146] In one embodiment, when the processor plans the motion path of the robot according to the current state and motion trajectory of the robot, the processor specifically performs the following steps: sampling the motion trajectory and fitting the trajectory points obtained by the sampling to obtain a following trajectory; planning the motion path of the robot according to the current state and the following trajectory of the robot.
[0147] In one embodiment, the processor executes sampling of the motion trajectory, specifically performing the following steps: sampling the motion trajectory according to a first sampling frequency to obtain at least two initial sampling points; when a distance between adjacent initial sampling points in the at least two initial sampling points is greater than a first preset distance, adding the adjacent initial sampling points to a sampling queue.
[0148] In one embodiment, the processor executes sampling of the motion trajectory, specifically performing the following steps: when the distance between adjacent initial sampling points among at least two initial sampling points is less than or equal to a first preset distance, sampling the first sub-motion trajectory according to a second sampling frequency to obtain a first sampling point; wherein the second sampling frequency is greater than the first sampling frequency, and the first sub-motion trajectory is the motion trajectory between the adjacent initial sampling points; when the distance between the first sampling point and the at least two initial sampling points is greater than the second preset distance, adding the at least two initial sampling points and the first sampling point to a sampling queue; when the distance between the first sampling point and the at least two initial sampling points is less than or equal to the second preset distance, adding the adjacent initial sampling point to the sampling queue, and sampling the second sub-motion trajectory according to a third sampling frequency to obtain a second sampling point; when the distance between the second sampling point and the first sampling point and the first initial sampling point is greater than the third preset distance, adding the second sampling point to the sampling queue; wherein the third sampling frequency is greater than the second sampling frequency, and the second sub-motion trajectory is the motion trajectory between the first sampling point and the first initial sampling point among the adjacent initial sampling points.
[0149] In one embodiment, when the processor executes if the matching result indicates that multiple objects do not include the target object, and re-locks the target object, the processor specifically performs the following steps: if the matching result indicates that multiple objects do not include the target object, a prompt message is issued; the prompt message is used to prompt the user object to issue a follow-up task voice control instruction; and according to the follow-up task voice control instruction, the target object is re-locked.
[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.
[0151] It should be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but are not limited to these.
[0152] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0153] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.
Claims
1. A robot comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed: determining object information of a plurality of objects based on the first target image, the first depth map, and the first radar information; Matching the object information with historical object information to obtain a matching result; the historical object information is information corresponding to the target object locked historically; If the matching result indicates that the multiple objects do not include the target object, re-locking the target object; If the matching result indicates that the multiple objects include the target object, adding the object information of the target object to the target queue to obtain the motion trajectory of the target object; Planning the motion path of the robot according to the current state of the robot and the motion trajectory; and The motion path is sampled, and the motion speed of the robot at each of the trajectory points is determined based on the distance between the sampled trajectory points, and the robot moves along the target object at the motion speed.
2. The robot according to claim 1, characterized in that: When the processor executes the step of determining object information of multiple objects based on the first target image, the first depth map, and the first radar information, the following steps are specifically performed: Acquire a first target image, a first depth map, and first radar information collected during the movement; Performing object detection on each object in the first target image to obtain first object initial information; Performing obstacle detection on the first depth map and the first radar information to obtain first obstacle coordinates corresponding to the first depth map and the first radar information respectively; the first obstacle coordinates are coordinates of each obstacle within the field of view of the robot; Performing data fusion on the first object initial information and the first obstacle coordinates to obtain first fused data; and Object information of each of the plurality of objects is determined based on the first fused data.
3. The robot according to claim 2, characterized in that: When the processor executes the computer program, the processor further performs the following steps: Performing coordinate conversion on the object coordinates of each of the objects in the first object initial information to obtain the first world coordinates of each of the objects under the robot navigation system; and Performing coordinate conversion on first obstacle coordinates corresponding to the first depth map and the first radar information respectively, to obtain second world coordinates and third world coordinates of each obstacle under the robot navigation system; The data fusion of the first object initial information and the first obstacle coordinates includes: The feature value in the first object initial information, the first world coordinate, the second world coordinate and the third world coordinate are merged.
4. The robot according to claim 1, characterized in that: When the processor executes the computer program, the processor further performs the following steps: Acquire the sound source angle when each of the objects makes a speech, and a second target image, a second depth map, and a second radar information collected during the movement; the second target image, the second depth map, and the second radar information are collected before the first target image, the first depth map, and the first radar information, respectively; Performing object detection on each object in the second target image to obtain second object initial information; Performing coordinate conversion on the object coordinates of each of the objects in the second object initial information to obtain a fourth world coordinate of each of the objects under the robot navigation system; Perform obstacle detection on the second depth map and the second radar information to obtain second obstacle coordinates corresponding to the second depth map and the second radar information respectively; and respectively detect the second obstacle coordinates corresponding to the second depth map and the second radar information respectively. The coordinates are converted by the robot navigation system to obtain the fifth world coordinates and the sixth world coordinates of each obstacle in the robot navigation system; wherein the second obstacle coordinates are the coordinates of each obstacle within the robot's field of view; Converting the sound source angle into a sound source direction vector under the robot navigation system, and determining the seventh world coordinate of each object under the robot navigation system according to the sound source direction vector; fusing the feature value in the second object initial information, the fourth world coordinate, the fifth world coordinate, the sixth world coordinate and the seventh world coordinate to obtain second fused data; and The target object is locked based on the second fused data to obtain historical object information of the target object; the historical object information includes a feature value of the target object and a world coordinate under the robot navigation system.
5. The robot according to claim 1, characterized in that: When the processor executes the computer program, the processor further performs the following steps: If the matching result indicates that the multiple objects include the target object, inputting the object information of the target object into the target filter, so that the target filter performs position prediction based on the object information to obtain the predicted position of the target object at the next moment; Determine the direction of movement based on the predicted position and the current position of the target object; the current position is the first world coordinate of the target object under the robot navigation system; and The current state of the robot is determined according to the movement direction, speed information and the current position.
6. The robot according to claim 1, characterized in that: When the processor plans the motion path of the robot according to the current state of the robot and the motion trajectory, the processor specifically performs the following steps: Sampling the motion trajectory, and fitting the trajectory points obtained by sampling to obtain a following trajectory; and The motion path of the robot is planned according to the current state of the robot and the following trajectory.
7. The robot according to claim 6, characterized in that: The processor performs sampling of the motion trajectory, specifically performing the following steps: Sampling the motion trajectory according to a first sampling frequency to obtain at least two initial sampling points; and When the distance between adjacent initial sampling points among the at least two initial sampling points is greater than a first preset distance, the adjacent initial sampling points are added to a sampling queue.
8. The robot according to claim 7, characterized in that: The processor performs sampling of the motion trajectory, specifically performing the following steps: When the distance between adjacent initial sampling points among the at least two initial sampling points is less than or equal to the first preset distance, sampling the first sub-motion trajectory according to the second sampling frequency to obtain a first sampling point; wherein the second sampling frequency is greater than the first sampling frequency, and the first sub-motion trajectory is the motion trajectory between the adjacent initial sampling points; and When the distance between the first sampling point and the at least two initial sampling points is greater than a second preset distance, adding the at least two initial sampling points and the first sampling point to the sampling queue; When the distance between the first sampling point and the at least two initial sampling points is less than or equal to a second preset distance, the adjacent initial sampling point is added to the sampling queue, and the second sub-motion trajectory is sampled according to a third sampling frequency to obtain a second sampling point; when the distance between the second sampling point and the first sampling point and the first initial sampling point is greater than a third preset distance, the second sampling point is added to the sampling queue; wherein the third sampling frequency is greater than the second sampling frequency, and the second sub-motion trajectory is a motion trajectory between the first sampling point and the first initial sampling point among the adjacent initial sampling points.
9. The robot according to any one of claims 1 to 8, characterized in that: When the processor executes the step of re-locking the target object if the matching result indicates that the multiple objects do not include the target object, the processor specifically performs the following steps: If the matching result indicates that the multiple objects do not include the target object, a prompt message is issued; the prompt message is used to prompt The user issues voice control instructions for the follow-up task; and The target object is re-locked according to the voice control instruction of the following task.
10. A robot following method, the method comprising: determining object information of a plurality of objects based on the first target image, the first depth map, and the first radar information; Matching the object information with historical object information to obtain a matching result; the historical object information is information corresponding to the target object locked historically; If the matching result indicates that the multiple objects do not include the target object, re-locking the target object; If the matching result indicates that the multiple objects include the target object, adding the object information of the target object to the target queue to obtain the motion trajectory of the target object; Planning the motion path of the robot according to the current state of the robot and the motion trajectory; and The motion path is sampled, and the motion speed of the robot at each of the trajectory points is determined based on the distance between the sampled trajectory points, and the robot moves along the target object at the motion speed.
11. The method according to claim 10, characterized in that The determining of the object information of the plurality of objects based on the first target image, the first depth map and the first radar information comprises: Acquire a first target image, a first depth map, and first radar information collected during the movement; Performing object detection on each object in the first target image to obtain first object initial information; Performing obstacle detection on the first depth map and the first radar information to obtain first obstacle coordinates corresponding to the first depth map and the first radar information respectively; the first obstacle coordinates are coordinates of each obstacle within the field of view of the robot; Fusing the first object initial information and the first obstacle coordinates to obtain first fused data; and Object information of each of the plurality of objects is determined based on the first fused data.
12. The method according to claim 11, characterized in that The method further comprises: Performing coordinate conversion on the object coordinates of each of the objects in the first object initial information to obtain the first world coordinates of each of the objects under the robot navigation system; and Performing coordinate conversion on first obstacle coordinates corresponding to the first depth map and the first radar information respectively, to obtain second world coordinates and third world coordinates of each obstacle under the robot navigation system; The fusing the first object initial information and the first obstacle coordinates includes: The feature value in the first object initial information, the first world coordinate, the second world coordinate and the third world coordinate are merged.
13. The method according to claim 10, characterized in that The method further comprises: Acquire the sound source angle when each of the objects makes a speech, and a second target image, a second depth map, and a second radar information collected during the movement; the second target image, the second depth map, and the second radar information are collected before the first target image, the first depth map, and the first radar information, respectively; Performing object detection on each object in the second target image to obtain second object initial information; performing coordinate conversion on the object coordinates of each object in the second object initial information to obtain fourth world coordinates of each object under the robot navigation system; Perform obstacle detection on the second depth map and the second radar information to obtain second obstacle coordinates corresponding to the second depth map and the second radar information respectively; perform coordinate conversion on the second obstacle coordinates corresponding to the second depth map and the second radar information respectively to obtain fifth world coordinates and sixth world coordinates of each obstacle under the robot navigation system; wherein the second obstacle coordinates are the coordinates of each obstacle within the field of view of the robot; Converting the sound source angle into a sound source direction vector under the robot navigation system, and determining the seventh world coordinate of each object under the robot navigation system according to the sound source direction vector; fusing the feature value in the second object initial information, the fourth world coordinate, the fifth world coordinate, the sixth world coordinate and the seventh world coordinate to obtain second fused data; and The target object is locked based on the second fused data to obtain historical object information of the target object; the historical object information includes a feature value of the target object and a world coordinate under the robot navigation system.
14. The method according to claim 10, characterized in that The method further comprises: If the matching result indicates that the multiple objects include the target object, inputting the object information of the target object into the target filter, so that the target filter performs position prediction based on the object information to obtain the predicted position of the target object at the next moment; Determine the direction of movement based on the predicted position and the current position of the target object; the current position is the first world coordinate of the target object under the robot navigation system; and The current state of the robot is determined according to the movement direction, speed information and the current position.
15. The method according to claim 10, characterized in that The step of planning the motion path of the robot according to the current state of the robot and the motion trajectory comprises: Sampling the motion trajectory, and fitting the trajectory points obtained by sampling to obtain a following trajectory; and The motion path of the robot is planned according to the current state of the robot and the following trajectory.
16. The method according to claim 15, characterized in that The sampling of the motion trajectory comprises: Sampling the motion trajectory according to a first sampling frequency to obtain at least two initial sampling points; and When the distance between adjacent initial sampling points among the at least two initial sampling points is greater than a first preset distance, the adjacent initial sampling points are added to a sampling queue.
17. The method according to claim 16, characterized in that The sampling of the motion trajectory comprises: When the distance between adjacent initial sampling points among the at least two initial sampling points is less than or equal to the first preset distance, sampling the first sub-motion trajectory according to the second sampling frequency to obtain a first sampling point; wherein the second sampling frequency is greater than the first sampling frequency, and the first sub-motion trajectory is the motion trajectory between the adjacent initial sampling points; and When the distance between the first sampling point and the at least two initial sampling points is greater than a second preset distance, adding the at least two initial sampling points and the first sampling point to the sampling queue; When the distance between the first sampling point and the at least two initial sampling points is less than or equal to a second preset distance, the adjacent initial sampling point is added to the sampling queue, and the second sub-motion trajectory is sampled according to a third sampling frequency to obtain a second sampling point; when the distance between the second sampling point and the first sampling point and the first initial sampling point is greater than a third preset distance, the second sampling point is added to the sampling queue; wherein the third sampling frequency is greater than the second sampling frequency, and the second sub-motion trajectory is a motion trajectory between the first sampling point and the first initial sampling point among the adjacent initial sampling points.
18. The method according to any one of claims 10 to 17, characterized in that If the matching result indicates that the multiple objects do not include the target object, re-locking the target object includes: If the matching result indicates that the multiple objects do not include the target object, a prompt message is issued; the prompt message is used to prompt the user to issue a voice control instruction for the follow-up task; and The target object is re-locked according to the voice control instruction of the following task.
19. A robot following device, comprising: A first determination module, configured to determine object information of a plurality of objects based on a first target image, a first depth map, and first radar information; The matching module is used to match the object information with the historical object information to obtain a matching result; the historical object information The information is the information corresponding to the target object locked historically; A locking module, configured to re-lock the target object if the matching result indicates that the multiple objects do not include the target object; an adding module, configured to add the object information of the target object to a target queue to obtain a motion trajectory of the target object if the matching result indicates that the multiple objects include the target object; A planning module, used for planning the motion path of the robot according to the current state of the robot and the motion trajectory; The following module is used to sample the motion path, and determine the motion speed of the robot at each trajectory point based on the distance between each trajectory point obtained by sampling, and follow the target object to move at the motion speed.
20. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 10 to 18 are implemented.
Citation Information
Patent Citations
Sound source tracking system and method for vision robot and cleaning robot
CN109318243A
Motion control method and device, robot and storage medium
CN113156962A
Self-propelled vehicle following system and self-propelled vehicle following method
CN114326695A
Robot, obstacle avoidance method and device and storage medium
CN115565058A
Robot, robot following method and device and storage medium
CN116661505A
Cited By
Sampling robot, control method thereof and medium
CN119871416A
Sampling robot and control method thereof, and medium
CN119871416B
