Object following robot system and method based on multi-sensor fusion

Abstract

The invention discloses an object following robot system and method based on multi-sensor fusion, and the system comprises an object detection module which comprises a plurality of sensors, and the plurality of sensors are used for obtaining the position information of a following object; the sensor fusion module is used for processing and fusing the position information from various sensors to obtain target position data; the path planning module is used for generating a real-time robot advancing track through a dynamic path planning algorithm according to the target position data; and the motion control module is used for controlling the robot to move according to the advancing track of the robot. According to the object following robot system and method based on multi-sensor fusion, multiple sensors are adopted to jointly sense the environment, and the target detection and positioning accuracy is improved; and in combination with a dynamic path planning algorithm, precise path planning and tracking of the following object are realized.

Description

Technical Field

[0001] The present application relates to the technical field of object following robots, and in particular to an object following robot system and method based on multi-sensor fusion. Background Art

[0002] With the development of mobile robotics, object following tasks have become increasingly common across a variety of fields. Existing following robotic systems typically rely on single sensors, such as vision sensors, lidar, and ultra-wideband (UWB) sensors, for environmental perception. However, single sensors are often affected by factors such as lighting variations, occlusions, and sensor accuracy limitations, leading to problems such as low tracking accuracy and target loss.

[0003] For example, vision-based sensors are often affected by lighting variations and occlusions during object tracking, resulting in unstable tracking of the object. LiDAR sensors, on the other hand, have a limited field of view and are easily interfered with by other objects in the environment, resulting in reduced detection accuracy. While UWB sensors can provide highly accurate positioning information, they can only provide valid data within a limited range. Therefore, combining the strengths of these sensors to overcome the limitations of each single sensor is a key issue in achieving a stable and reliable object tracking system. Summary of the Invention

[0004] In order to address the shortcomings of the existing technology, the purpose of this application is to provide an object following robot system and method based on multi-sensor fusion, which adopts multiple sensors for joint perception and combines with dynamic path planning algorithm to achieve accurate tracking and path planning of the specified target.

[0005] To achieve the above objectives, the present application provides an object following robot system based on multi-sensor fusion, comprising:

[0006] An object detection module, the object detection module including a plurality of sensors, the plurality of sensors being used to obtain position information of a follow-up object;

[0007] The sensor fusion module is used to process and fuse the position information from various sensors to obtain the target position data;

[0008] A path planning module is used to generate a real-time robot trajectory through a dynamic path planning algorithm based on the target position data;

[0009] The motion control module is used to control the motion of the robot according to the robot's travel trajectory.

[0010] Furthermore, the object detection module includes: at least one visual sensor, at least one infrared sensor and / or at least one lidar sensor, and / or at least one UWB sensor.

[0011] Furthermore, the object detection module uses a ZED camera to acquire two-dimensional image data of the following object;

[0012] The three-dimensional point cloud data is acquired by using a laser radar, and the three-dimensional point cloud data is combined with the two-dimensional image data through coordinate transformation to obtain the three-dimensional position data of the followed object.

[0013] Furthermore, the sensor fusion module is used to receive UWB data and process the UWB data using an AOA algorithm to obtain positioning information of the followed object;

[0014] It is used to receive two-dimensional image data, perform target detection on the image based on the YOLO algorithm, and use bounding boxes to identify the positions of all followable objects in the image;

[0015] Used to receive the three-dimensional position data of the following object, perform UWB data filtering on the three-dimensional position data of the following object according to the positioning information; compare the coordinate values ​​in the filtered point cloud data with the bounding box to obtain the target position data.

[0016] Furthermore, the sensor fusion module assigns different weight coefficients to the distance between the followed object and the first UWB anchor point and the UWB coordinate value of the followed object, and performs bandpass filtering to obtain filtered point cloud coordinate values; the point cloud coordinate values ​​are compared with the bounding box to obtain target position data.

[0017] Furthermore, the dynamic path planning algorithm is a dynamic window algorithm or an adaptive prediction dynamic window algorithm.

[0018] Furthermore, the adaptive prediction dynamic window algorithm includes:

[0019] The speed space of the robot path is obtained according to the speed space determined by the robot kinematic constraints and the speed space determined by the robot acceleration limit;

[0020] Generate multiple predicted poses of the robot at future moments based on the different linear velocities and angular velocities in the velocity space where the robot path is located;

[0021] All planned paths are evaluated using the evaluation function.

[0022] Furthermore, it also includes a feedback module; the feedback module is used to monitor the relative position of the robot and the followed object in real time. When the followed object changes rapidly or an obstacle appears, the feedback module will adjust the robot's motion parameters according to a preset strategy to ensure that the robot always maintains within a preset distance and stably follows the object.

[0023] Furthermore, the motion control module includes a safety control subsystem; the safety control subsystem is used to control the robot movement by adopting obstacle avoidance or deceleration strategies when potential obstacles or dangers are detected.

[0024] To achieve the above objectives, the present application further provides an object following method based on multi-sensor fusion, which uses the object following robot system based on multi-sensor fusion as described above, and includes the following steps:

[0025] acquiring position data of a tracking object by using at least two sensors;

[0026] Processing and fusing the position data from the at least two sensors to obtain target position data;

[0027] Generate a real-time robot trajectory through a dynamic path planning algorithm based on the target position data;

[0028] The robot movement is controlled according to the robot travel trajectory.

[0029] The object-following robot system based on multi-sensor fusion provided in this application improves the accuracy of target detection and positioning by adopting multiple sensors to jointly perceive the environment; combined with a dynamic path planning algorithm, it realizes accurate path planning and tracking of the followed object.

[0030] Other features and advantages of the present application will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings:

[0032] Figure 1 This is a block diagram of an object following robot system based on multi-sensor fusion according to the first embodiment of the present application;

[0033] Figure 2 Schematic diagram of the structure of an object following robot system based on multi-sensor fusion according to the first embodiment of the present application;

[0034] Figure 3Schematic diagram of the positions of the gimbal camera and the object being followed according to the first embodiment of the present application;

[0035] Figure 4 Schematic diagram of the AOA algorithm principle according to the first embodiment of the present application;

[0036] Figure 5 Schematic diagram of adaptive parameter trajectory planning according to an embodiment of the present application;

[0037] Figure 6 Schematic diagram of the structure of an object following robot system based on multi-sensor fusion according to the second embodiment of the present application;

[0038] Figure 7 4 is a flow chart of an object following method based on multi-sensor fusion according to an embodiment of the present application.

[0039] Reference numerals:

[0040] 101 - object detection module; 102 - sensor fusion module; 103 - path planning module; 104 - motion control module; 105 - feedback module. DETAILED DESCRIPTION

[0041] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application.

[0042] The following describes embodiments of the present application in more detail with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of protection of the present application.

[0043] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0044] It should be noted that the concepts of "first" and "second" may be mentioned in this application only to distinguish different devices, components or parts, and are not used to limit the order or interdependence of the functions performed by these devices, components or parts.

[0045] It should be noted that the modifications of "one" and "multiple" that may be mentioned in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more". "Multiple" should be understood as two or more. The object following robot system based on multi-sensor fusion of the present application includes: an object detection module, the object detection module includes a plurality of sensors, and the plurality of sensors are used to obtain the position information of the followed object; a sensor fusion module, which is used to process and fuse the position information from various sensors to obtain target position data; a path planning module, which is used to generate a real-time robot travel trajectory through a dynamic path planning algorithm according to the target position data; and a motion control module, which is used to control the robot motion according to the robot travel trajectory.

[0046] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0047] In the description of this application, the following objects of the robot include animals and objects that can move. It should be understood that the following object refers to the target that the mobile robot needs to follow.

[0048] Example 1 Figure 1 This is a block diagram of an object following robot system based on multi-sensor fusion according to the first embodiment of the present application. Figure 2 This is a schematic diagram of the structure of an object following robot system based on multi-sensor fusion according to the first embodiment of the present application. Figure 1 and Figure 2 , the object following robot system based on multi-sensor fusion of the present application is described in detail.

[0049] The object following robot system based on multi-sensor fusion of the present application includes an object detection module 101 , a sensor fusion module 102 , a path planning module 103 , a motion control module 104 and a feedback module 105 .

[0050] The object detection module 101 uses a visual sensor (such as a ZED camera) to acquire 2D image data of the object being followed. It then uses a lidar to acquire 3D point cloud data. This data is then combined with the 2D image data through coordinate transformation to obtain the 3D position data of the object being followed. Combining the 3D data provided by the lidar with UWB positioning data allows for more accurate positioning of the object.

[0051] In an embodiment of the present application, the object detection module 101 includes at least one visual sensor, at least one infrared sensor and / or at least one lidar sensor, and at least one UWB sensor.

[0052] The object detection module 101 is used for target detection and positioning and is one of the core modules of the entire system. This embodiment uses a combination of ZED cameras, laser radars, and UWB sensors to improve the accuracy of target positioning through multi-sensor fusion.

[0053] The ZED camera utilizes binocular stereo vision technology, simulating the parallax of the human eye through dual lenses to achieve high-precision depth perception and environmental modeling, acquiring depth information. The ZED camera provides two-dimensional image information, which is then processed using the YOLOX algorithm to determine the position of each tracked object within the image. Because the YOLOX algorithm is a deep learning-based object detection method, it can efficiently detect people in crowded environments and extract the bounding box of each person.

[0054] In the embodiment of the present application, the binocular camera is equipped with a pan-tilt system, such as Figure 2 As shown, the pan-tilt system can control the binocular camera based on target position data. The system also includes a pan-tilt head, a mechanical support device used in photographic equipment to stabilize and control the movement of a camera or video camera. The pan-tilt head can be remotely or manually operated to perform various movements, such as translation, rotation, and tilt. The pan-tilt head is designed to maintain camera or video camera stability, reducing vibration and jitter, thereby producing clearer, more stable images. Figure 3 Schematic diagram of the positions of the gimbal camera and the following object.

[0055] LiDAR provides three-dimensional point cloud information. To combine this 3D point cloud data with visual image data, the present invention employs coordinate conversion technology. First, the LiDAR 3D point cloud data is converted to the LiDAR coordinate system, then gradually converted to the vehicle coordinate system, the camera coordinate system, and finally to the image coordinate system. These steps ensure that the LiDAR data and visual data are accurately matched within the same coordinate system. Background information in the point cloud data is removed through point cloud filtering technology, ensuring that only point cloud data relevant to the object being tracked is retained.

[0056] UWB sensors use radio waves to measure the distance between a target and a base station and calculate their relative position using an AOA (angle of arrival) algorithm. UWB sensors offer highly accurate positioning capabilities, providing real-time location data within a certain range. To ensure accurate location data, this paper uses sliding window filtering and low-pass filtering techniques to smooth the data acquired by the UWB sensor, thereby eliminating the effects of noise.

[0057] By comprehensively utilizing visual data, point cloud data, and UWB sensor data, the present invention can accurately calculate the three-dimensional position of the followed object and provide accurate data support for subsequent path planning and motion control.

[0058] Specifically, the sensor fusion module 102 is used to fuse data from different sensors (such as image data, point cloud data, UWB positioning data, and infrared data) in real time, eliminating errors from individual sensors and improving the positioning accuracy and robustness of the overall system. Through advanced multi-sensor data fusion algorithms, the system can achieve precise target detection and tracking in complex environments.

[0059] In an embodiment of the present application, the sensor fusion module 102 receives UWB data, processes the UWB data using the AOA algorithm, and obtains positioning information of the following object; the sensor fusion module 102 receives two-dimensional image data, performs target detection on the image based on the YOLOX algorithm, and uses a bounding box to identify the positions of all following objects in the image; the sensor fusion module 102 receives three-dimensional position data of the following object, performs UWB data filtering on the three-dimensional position data of the following object according to the positioning information, compares the coordinate values ​​in the filtered point cloud data with the bounding box, and obtains the final target position data.

[0060] In the embodiment of the present application, after obtaining the sensor data, the first step is to process the UWB data. The UWB sensor used in this embodiment includes a UWB tag and two UWB anchors. The UWB processing unit of this embodiment uses the AOA algorithm to achieve target positioning. The principle is as follows: Figure 4 As shown, point T represents the follower object, point , Represent two UWB anchors respectively. We can calculate T and and The distance between them is recorded as , , then the coordinates of T can be calculated as: , In the formula is the angle of arrival; is the wavelength of the signal; is the phase difference of the signals received by the two receivers; The target point T is on the plane Coordinate values ​​on T.

[0061] In the second step, we use YOLOv5 as the recognition algorithm, taking the current camera image as input and using bounding boxes to identify the locations of all objects in the field of view. To distinguish each target, this example uses DeepSORT to assign a unique ID to each target. DeepSORT is a deep learning-based multi-object tracking (MOT) algorithm that significantly improves tracking accuracy and robustness in complex scenarios by combining Kalman filtering and appearance feature matching.

[0062] In the third step, we need to filter the binocular camera point cloud based on the data obtained in the first and second steps. First, perform UWB data filtering on the point cloud. In order to prevent the distance between the follower and the UWB anchors from affecting the UWB positioning accuracy, this embodiment will and Different weight coefficients are assigned and used for bandpass filtering. The point cloud coordinates are then compared with the bounding boxes identified by a YOLO algorithm (such as YOLOv5 or YOLOX), and the point cloud is further filtered to obtain more accurate target locations. This filtering and optimization of sensor data significantly reduces the impact of environmental noise.

[0063] Binocular camera point cloud refers to three-dimensional point cloud data generated by a binocular camera system, where the binocular camera system consists of two cameras, simulates human binocular vision, and calculates the depth information of objects by capturing images of the same scene from two different perspectives.

[0064] In an embodiment of the present application, the sensor fusion module 102 adopts multi-sensor fusion technology, and fuses data from visual sensors, lidars, and UWB sensors in real time through a weighted fusion method, optimizes data accuracy and reliability, and effectively copes with uncertainties in dynamic environments. During the fusion process, the combination of visual data and lidar data is mainly achieved through the coordinate transformation of point cloud data and the matching of image detection bounding boxes. By filtering the lidar point cloud, background noise can be removed and valid point cloud data related to the followed object can be extracted. Combined with the target bounding box detected by the YOLO algorithm, the part of the point cloud data corresponding to the followed object is filtered out to further improve the accuracy of the data.

[0065] The UWB sensor data provides the relative position of the target and the base station, so it needs to be fused with data from other sensors to obtain a more accurate target location. Through multi-sensor fusion, the position information provided by the UWB sensor is combined with data from vision sensors and lidar to obtain more accurate 3D coordinate information of the tracking object.

[0066] The path planning module 103 is used to perform dynamic optimization based on the target motion trajectory and environmental data using an artificial intelligence-based learning algorithm to provide an optimal following path.

[0067] In this embodiment, the path planning module 103 uses a dynamic path planning algorithm to obtain the robot's trajectory according to the target position data.

[0068] Path planning is another challenge in object-following robotic systems. Traditional path planning algorithms often rely on prior environmental maps and are unable to adapt to dynamically changing environments. In object-following tasks, the robot must dynamically adjust its path based on the position of the object being followed. This requires the path planning algorithm to be not only real-time but also capable of handling the dynamic relationship between the robot and the object, ensuring safety and stability during the following process.

[0069] Since the traditional dynamic window approach (DWA) algorithm has many fixed parameters, it performs poorly in solving the object following problem. To address this shortcoming, this application proposes an adaptive parameter APDWA algorithm. Figure 5 This is a schematic diagram of adaptive parameter trajectory planning according to an embodiment of the present application. APDWA (Adaptive Predictive Dynamic Window Approach) is an improved path planning algorithm proposed in this embodiment. It optimizes the traditional DWA algorithm to improve the real-time performance and path smoothness of the robot in object following tasks. The APDWA algorithm consists of three steps: 1) searching the velocity space; 2) trajectory planning; and 3) trajectory evaluation. The following is a detailed explanation of APDWA:

[0070] In step 1), you need to first confirm the speed space where the robot path is located , according to the actual situation, It consists of two parts:

[0071] Part 1: Velocity space determined by robot kinematic constraints : , In the formula is the maximum speed.

[0072] Part 2: Speed ​​space determined by the robot's acceleration limit : , in is the linear acceleration, is the angular acceleration, It's the robot in time The linear and angular velocities, is the time step. Therefore, the required It is given by the following formula: .

[0073] In step 2), according to the velocity space The different linear angular velocities in the robot generate multiple predicted poses at future moments. The process is as follows: Figure 5 The robot's predicted pose is calculated as follows: , Where (x,y, ) represents the robot's posture, k and k+1 represent the sampling time and prediction time respectively; the prediction step of the algorithm is t each time, and the robot path in the time period t is given by t / discrete postures.

[0074] For the currently widely used DWA, the algorithm’s t and are all fixed values, which causes the planned path length to be too long or too short when the moving speed of the follower changes, affecting the results of subsequent path evaluation. It will also increase the number of search paths during the following process, increasing unnecessary computing power consumption. Therefore, the t and The solution is as follows: , in is the scale parameter, is a constant, ||P t ||2 2 Indicates P t In this embodiment, t is determined by the following object is determined by the Euclidean distance between the robot and The function representing the motion state of the following object is This function ensures that in different following scenarios, such as turning, straight driving, dynamic obstacle avoidance, static obstacle avoidance, etc. The value of improves the overall planning speed.

[0075] In step 3), we need to evaluate all planned paths. The evaluation function plays a crucial role in the dynamic path planning algorithm. It not only affects the path planning results, but is also one of the key factors in the efficiency and effectiveness of the algorithm. In this embodiment, the evaluation function is as follows: Min heading(v, )+ speed(v, )+ obstacle(v, )+ distance(v, )+ acceleration(v, ) Where heading( ), speed( ), obstacle( ), distance( ), acceleration( ) are respectively the speed of the path(v, ) function item, to are the weight coefficients of each item, and Min means taking the minimum value of J as the score of the path. ) controls the angle of the robot corresponding to different trajectories. This term controls the robot's movement towards the object being followed. It is defined as follows: heading(v, )= , In the formula is the robot heading angle at the end of the trajectory.

[0076] speed(v, Including the limitation of linear speed and angular speed, requiring the robot's linear and angular speed to be kept as close as possible. , , are the desired linear and angular velocities. They are defined as follows: speed(v, )= , Where V is the velocity corresponding to the current trajectory, is the expected speed, which corresponds to the moving speed of the following object in the following task.

[0077] Compared with the currently widely used DWA, APDWA adds evaluation items that are suitable for object following tasks to the evaluation function and improves the problem of fixed coefficients in the original evaluation items.

[0078] In obstacle(v, ) function, the classic DWA algorithm uses the same weight for each obstacle. This method will incorrectly evaluate the impact of different obstacles on the planned path. To this end, this application adds the obstacle weight coefficient μ to the evaluation function, and the function is defined as follows: obstacle(v, )=

[0079] ( , )and( , ) represent the positions of the path points and obstacles respectively. The expression of μ in the formula is as follows: , In the formula, m and n are weight coefficients. is the location of the obstacle, To follow the target’s position, μ adaptively changes according to the Euclidean distance between the obstacle and the waypoint and the motion state of the followed object.

[0080] To ensure that the robot can reach the position of the object being followed, this embodiment adds the distance between the robot and the object being followed (v, ) item. In order to reduce the energy consumption in the following process and ensure the smoothness of the following path, this embodiment also adds acceleration (v, ) limits the robot's linear acceleration and angular acceleration. The definitions of these two parts are as follows: , acceleration(v, )= , In the formula The end point of the path.

[0081] Path planning is a crucial component of object-following robot systems. To ensure the robot can smoothly follow objects in complex environments, this embodiment employs the Adaptive Predictive Dynamic Window Algorithm (APDWA) for path planning. The APDWA algorithm is an adaptive path planning algorithm based on a dynamic window. It dynamically generates the robot's motion trajectory by considering the robot's kinematic constraints, the position of the object being followed, and the distribution of obstacles. During this process, path planning not only considers the robot's static constraints (such as velocity and acceleration) but also calculates the motion state of the object being followed in real time, ensuring that path planning effectively adapts to changes in the object being followed.

[0082] Unlike traditional path planning algorithms, the APDWA algorithm does not rely on a priori maps of the environment. Instead, it obtains environmental information through real-time sensor data and dynamically plans the path. The introduction of the APDWA algorithm enables the path planning module 103 to not only effectively plan paths on flat ground, but also adjust the robot's trajectory in real time based on factors such as obstacles and terrain undulations. This enables the object-following robot system based on multi-sensor fusion provided in this application to effectively perform following tasks in complex, dynamically changing environments.

[0083] The motion control module 104 uses precise motor control technology to control the robot's speed and direction based on the instructions output by the path planning module. In the embodiments of this application, a closed-loop control mechanism is employed to ensure that the robot can smoothly follow an object. Real-time feedback is used to adjust the robot's speed and direction to prevent the robot from deviating from the intended path.

[0084] In this embodiment, the motion control module 104 further includes a safety control subsystem, which is configured to automatically adopt an obstacle avoidance or deceleration strategy when a potential obstacle or danger is detected, to ensure the safety of the followed object and the robot.

[0085] Feedback module 105 monitors the relative position of the robot and the target in real time and dynamically adjusts the robot's following mode based on the target's speed and direction of motion. When the target undergoes rapid changes or an obstacle appears, feedback module 105 adjusts the robot's motion parameters (such as speed and steering angle) according to a pre-set strategy to ensure the robot consistently follows the target within a predetermined, reasonable distance. This real-time feedback mechanism effectively addresses various changes in a dynamic environment and ensures a stable relative position between the robot and the target.

[0086] The object following robot system based on multi-sensor fusion in this application has high precision, high stability and strong environmental adaptability. It can be widely used in smart home, medical care, service robots and other fields, and has important practical value and promotion prospects.

[0087] Example 2 The difference between this embodiment and embodiment 1 is that the object detection module includes at least one visual sensor and at least one laser radar.

[0088] Figure 6 FIG. 1 is a schematic diagram of the structure of an object following robot system based on multi-sensor fusion according to the second embodiment of the present application, as shown in FIG. Figure 6As shown, in this embodiment, the object detection module 101 includes a camera and a 2D laser radar, obtains image data through the camera, obtains point cloud data through the 2D laser radar, and transmits the image data and point cloud data to the sensor fusion module 102. After the three stages of visual detection, point cloud filtering (the visual detection and point cloud filtering methods refer to the above embodiment 1), and obstacle fitting based on RANSAC (random sample consensus algorithm) are processed in sequence, the target position data is obtained and transmitted to the path planning module 103. The path planning module 103 gives the robot's travel trajectory (including movement direction, speed, etc.) based on the APDWA algorithm.

[0089] Example 3 In an embodiment of the present application, a multi-sensor fusion-based object-following method is also provided, which utilizes the multi-sensor fusion-based object-following robot system described above. The method comprises: acquiring position data of a followable object using at least two sensors; processing and fusing the position data from the at least two sensors to obtain target position data; generating a real-time robot trajectory based on the target position data using a dynamic path planning algorithm; and controlling the robot's motion based on the robot trajectory.

[0090] Figure 7 The following is a flow chart of the object following method based on multi-sensor fusion according to an embodiment of the present application. Figure 7 The object following method based on multi-sensor fusion of the present application is described in detail.

[0091] First, in step 701, position data of a following object is obtained. This step includes: obtaining position information of the following object through at least two sensors; and obtaining three-dimensional position data of the following object through multi-sensor data fusion.

[0092] In some embodiments, a visual sensor (such as a ZED camera) acquires 2D image data of the object being followed, and a lidar (lidar) is used to acquire 3D point cloud data of the object being followed. Coordinate transformation is used to combine the 3D point cloud data with the 2D image data to obtain 3D position data of the object being followed.

[0093] In other embodiments, the position data of the followed object is obtained by using a ZED camera, a lidar, and a UWB sensor respectively, and the three types of data are combined to perform target detection and positioning, thereby further improving the accuracy of the target position.

[0094] In step 702, data fusion is performed to obtain target position data.

[0095] In an embodiment of the present application, step 702 includes: using the AOA algorithm to process the UWB data to obtain positioning information of the following object; performing target detection on the two-dimensional image data based on the YOLOX algorithm, and using a bounding box to identify the positions of all following objects in the image; performing UWB data filtering on the three-dimensional position data of the following object based on the positioning information obtained from the UWB data, and comparing the coordinate values ​​in the filtered point cloud data with the bounding box to obtain the final target position data.

[0096] In step 703, a real-time robot trajectory is generated by a dynamic path planning algorithm, which is a dynamic window algorithm or the adaptive prediction dynamic window algorithm described above.

[0097] In step 704, the robot is controlled to move according to the robot's trajectory. The path planning module outputs corresponding speed and direction instructions based on the robot's trajectory, and controls the robot's movement according to the instructions through precise motor control technology.

[0098] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. An object following robot system based on multi-sensor fusion, characterized in that: include: An object detection module, the object detection module including a plurality of sensors, the plurality of sensors being used to obtain position information of a follow-up object; The sensor fusion module is used to process and fuse the position information from various sensors to obtain the target position data; A path planning module is used to generate a real-time robot trajectory through a dynamic path planning algorithm based on the target position data; The motion control module is used to control the motion of the robot according to the robot's travel trajectory.

2. The object following robot system based on multi-sensor fusion according to claim 1, characterized in that: The object detection module includes: at least one visual sensor, at least one infrared sensor and / or at least one lidar sensor, and / or at least one UWB sensor.

3. The object following robot system based on multi-sensor fusion according to claim 1, characterized in that: The object detection module uses a ZED camera to acquire two-dimensional image data of the following object; The three-dimensional point cloud data is acquired by using a laser radar, and the three-dimensional point cloud data is combined with the two-dimensional image data through coordinate transformation to obtain the three-dimensional position data of the followed object.

4. The object following robot system based on multi-sensor fusion according to claim 3, characterized in that: The sensor fusion module is used to receive UWB data and process the UWB data using the AOA algorithm to obtain positioning information of the followed object; It is used to receive two-dimensional image data, perform target detection on the image based on the YOLO algorithm, and use bounding boxes to identify the positions of all followable objects in the image; Used to receive the three-dimensional position data of the following object, perform UWB data filtering on the three-dimensional position data of the following object according to the positioning information; compare the coordinate values ​​in the filtered point cloud data with the bounding box to obtain the target position data.

5. The object following robot system based on multi-sensor fusion according to claim 4, characterized in that: The sensor fusion module assigns different weight coefficients to the distance between the followed object and the first UWB anchor point and the UWB coordinate value of the followed object, and performs bandpass filtering on the weight coefficients to obtain filtered point cloud coordinate values; the point cloud coordinate values ​​are compared with the bounding box to obtain target position data.

6. The object following robot system based on multi-sensor fusion according to claim 1, characterized in that: The dynamic path planning algorithm is a dynamic window algorithm or an adaptive prediction dynamic window algorithm.

7. The object following robot system based on multi-sensor fusion according to claim 6, characterized in that: The adaptive prediction dynamic window algorithm includes: The speed space of the robot path is obtained according to the speed space determined by the robot kinematic constraints and the speed space determined by the robot acceleration limit; Generate multiple predicted poses of the robot at future moments based on the different linear velocities and angular velocities in the velocity space where the robot path is located; All planned paths are evaluated using the evaluation function.

8. The object following robot system based on multi-sensor fusion according to claim 1, characterized in that: It also includes a feedback module; the feedback module is used to monitor the relative position of the robot and the object being followed in real time. When the object being followed changes rapidly or an obstacle appears, the feedback module will adjust the robot's motion parameters according to a preset strategy to ensure that the robot always maintains within a preset distance and stably follows the object.

9. The object following robot system based on multi-sensor fusion according to claim 1, characterized in that: The motion control module includes a safety control subsystem; the safety control subsystem is used to adopt obstacle avoidance or deceleration strategies to control the robot motion when potential obstacles or dangers are detected.

10. An object following method based on multi-sensor fusion, using the object following robot system based on multi-sensor fusion according to any one of claims 1 to 9, characterized in that: The following steps are involved: acquiring position data of a tracking object by using at least two sensors; Processing and fusing the position data from the at least two sensors to obtain target position data; Generate a real-time robot trajectory through a dynamic path planning algorithm based on the target position data; The robot movement is controlled according to the robot travel trajectory.

Images