Multi-sensor fusion obstacle tracking method, device, equipment and medium

Abstract

Embodiments of this specification disclose a multi-sensor fusion obstacle tracking method, apparatus, device, and medium. The method includes: obtaining radar sensor data and image sensor data of an outdoor robot, determining three-dimensional obstacle data of the current scene based on the radar sensor data, and determining two-dimensional obstacle data of the current scene based on the image sensor data; fusing and judging the two-dimensional obstacle data with the three-dimensional obstacle data to obtain a lidar obstacle list and a camera obstacle list that was not successfully fused; inputting the lidar obstacle list into a tracking node of the outdoor robot to obtain a first obstacle tracking list; deduplicating obstacle lists detected by multiple cameras and the camera obstacle list to obtain a camera target obstacle list, tracking and detecting the camera target obstacle list to obtain a second obstacle tracking list; and merging the first obstacle tracking list with the second obstacle tracking list to obtain a tracking list for the outdoor robot.

Description

Technical Field

[0001] This specification relates to the field of robotics, and in particular to a multi-sensor fusion obstacle tracking method, device, equipment, and medium. Background Art

[0002] Driven by market demand and market trends, robotics has flourished across various application areas. For outdoor robots, such as delivery robots and inspection robots, obstacle detection is a crucial task due to the uncertainty of the outdoor environment. Object detection involves finding objects of interest in an image or video and simultaneously determining their position and size. Unlike simple image classification tasks, object detection requires not only classification but also localization, making it a multi-task problem. Therefore, achieving object detection and tracking for outdoor robots is a crucial issue in the robotics field.

[0003] Existing technologies rely solely on the robot's camera for visual control analysis, or solely on LiDAR for 3D data analysis. This single-sensor approach to target detection is susceptible to lighting conditions, while radar has limitations in detection range. Therefore, tracking based solely on a single sensor can easily lead to obstacles in the detection process, making it difficult for outdoor robots to track and identify obstacles. This can raise questions about the reliability of the tracking and identification results.

[0004] Therefore, there is a need for an obstacle tracking method that can improve the reliability of target detection tasks. Summary of the Invention

[0005] One or more embodiments of this specification provide a multi-sensor for solving the following technical problem: how to provide an obstacle tracking method that can improve the reliability of target detection tasks.

[0006] One or more embodiments of this specification adopt the following technical solutions:

[0007] One or more embodiments of this specification provide a multi-sensor fusion obstacle tracking method, the method comprising:

[0008] Acquire radar sensor data and image sensor data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and determine two-dimensional obstacle data of the current scene based on the image sensor data;

[0009] Performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that was not successfully fused;

[0010] Inputting obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list;

[0011] Deduplicating and fusing obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and inputting obstacle information in the camera target obstacle list into a tracking node of the outdoor robot to obtain a second obstacle tracking list;

[0012] The first obstacle tracking list and the second obstacle tracking list are merged to obtain a tracking list of the outdoor robot.

[0013] In one or more embodiments of this specification, determining three-dimensional obstacle data of the current scene based on the radar sensor data, and determining two-dimensional obstacle data of the current scene based on the image sensor data, specifically includes:

[0014] The two-dimensional image data is detected by a preset target detection model Yolox to obtain two-dimensional obstacle data;

[0015] Downsampling the three-dimensional point cloud data to obtain three-dimensional point cloud data to be analyzed;

[0016] Acquire any one of the three-dimensional point cloud data to be analyzed, and obtain the distance between the any one of the three-dimensional point cloud data to be analyzed and other three-dimensional point cloud data to be analyzed based on a preset balanced binary tree algorithm;

[0017] The other three-dimensional point cloud data to be analyzed that are less than the preset distance threshold are divided into a preset first set, and any one of the three-dimensional point cloud data to be analyzed in the preset first set is obtained for iterative analysis to implement cluster analysis of the three-dimensional point cloud data and obtain the three-dimensional obstacle data in the current scene.

[0018] In one or more embodiments of this specification, before performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data, the method further includes:

[0019] Calibrate a camera pre-installed on the outdoor robot to obtain internal parameters of the camera;

[0020] The coordinates of the three-dimensional obstacle data are used as preset reference coordinates, and the three-dimensional obstacle data is combined with the camera coordinate system of the two-dimensional obstacle data through a camera pose estimation algorithm to obtain the external parameter matrix of the camera and lidar preset on the outdoor robot;

[0021] Based on the joint calculation of the external parameter matrix and the internal parameter matrix, a transformation relationship between the laser radar coordinate system and the camera coordinate system is obtained;

[0022] The three-dimensional obstacle data is projected onto the plane of the camera coordinate system based on the conversion relationship to obtain a two-dimensional detection frame of the three-dimensional obstacle data on the plane where the two-dimensional obstacle data is located.

[0023] In one or more embodiments of this specification, the fusing and judging the two-dimensional obstacle data and the three-dimensional obstacle data specifically includes:

[0024] Inputting the two-dimensional obstacle data and the three-dimensional obstacle data into the same ROS node;

[0025] At the ROS node, a Yolox detection frame is obtained by detecting the two-dimensional image data using a preset target detection model Yolox;

[0026] Obtaining an intersection area between the two-dimensional detection frame and the Yolox detection frame and a deduplicated area between the two-dimensional detection frame and the Yolox detection frame;

[0027] The IOU area is obtained by using the intersection area for the deduplication area, so as to determine whether the two-dimensional obstacle data and the three-dimensional obstacle data are fused based on a threshold between the IOU area and a preset IOU area.

[0028] In one or more embodiments of this specification, inputting obstacle information in the laser radar obstacle list into a tracking node of the outdoor robot to obtain a first obstacle tracking list specifically includes:

[0029] Obtaining a preset tracking trajectory of the outdoor robot, and obtaining a current tracking obstacle in the laser radar obstacle list based on a motion estimation of the preset tracking trajectory based on a Kalman filter;

[0030] According to the obstacle information in the laser radar list, the IOU distance between the tracked obstacle and each obstacle in the laser radar obstacle list is calculated to obtain the tracking cost matrix of the outdoor robot;

[0031] The tracking cost matrix is ​​input into a preset Hungarian algorithm to match each obstacle corresponding to the lidar obstacle list to obtain a first obstacle tracking list.

[0032] In one or more embodiments of this specification, deduplicating and fusing the obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list specifically includes:

[0033] transforming the obstacle lists detected by the plurality of cameras into the same coordinate system based on the camera coordinate system;

[0034] Obtaining an overlapping area of ​​fields of view of the multiple cameras in the same coordinate system;

[0035] Obtaining overlapping obstacles in the overlapping field of view area according to the obstacle list detected by the multiple cameras, the identifier of each obstacle in the camera obstacle list, and the position coordinates of each obstacle;

[0036] Based on the overlapping obstacles, the obstacle lists detected by the multiple cameras are deduplicated and fused with the camera obstacle list to obtain a successfully fused camera target obstacle list.

[0037] In one or more embodiments of this specification, inputting the obstacle information of the camera target obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list specifically includes:

[0038] Obtaining a preset tracking trajectory of the outdoor robot, estimating the motion of the preset tracking trajectory based on a Kalman filter, and obtaining a current tracking obstacle in the camera target obstacle list;

[0039] Calculate the IOU distance between the tracked obstacle and each obstacle in the camera target obstacle list according to the obstacle information in the camera target obstacle list to obtain the tracking cost matrix of the camera;

[0040] The tracking cost matrix is ​​input into a preset Hungarian algorithm to match each obstacle corresponding to the camera target obstacle list to obtain a second obstacle tracking list.

[0041] One or more embodiments of this specification provide a multi-sensor fusion obstacle tracking device, characterized in that the device includes:

[0042] an acquisition unit, configured to acquire radar sensing data and image sensing data of the outdoor robot, to determine three-dimensional obstacle data of the current scene based on the radar sensing data, and to determine two-dimensional obstacle data of the current scene based on the image sensing data;

[0043] a first fusion unit, configured to fuse and determine the two-dimensional obstacle data and the three-dimensional obstacle data, and obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the unfused two-dimensional obstacle data;

[0044] An input unit, configured to input obstacle information in the laser radar obstacle list into a tracking node of the outdoor robot to obtain a first obstacle tracking list;

[0045] a second fusion unit, configured to de-duplicate and fuse the obstacle lists detected by the multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and input the obstacle information in the camera target obstacle list into the tracking node of the outdoor robot to obtain a second obstacle tracking list;

[0046] A merging unit is configured to merge the first obstacle tracking list and the second obstacle tracking list to obtain a tracking list of the outdoor robot.

[0047] One or more embodiments of this specification provide a multi-sensor fusion obstacle tracking device, the device comprising:

[0048] at least one processor; and,

[0049] a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0051] Acquire radar sensor data and image sensor data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and determine two-dimensional obstacle data of the current scene based on the image sensor data;

[0052] Performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that was not successfully fused;

[0053] Inputting obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list;

[0054] Deduplicating and fusing obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and inputting obstacle information in the camera target obstacle list into a tracking node of the outdoor robot to obtain a second obstacle tracking list;

[0055] The first obstacle tracking list and the second obstacle tracking list are merged to obtain a tracking list of the outdoor robot.

[0056] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to:

[0057] Acquire radar sensor data and image sensor data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and determine two-dimensional obstacle data of the current scene based on the image sensor data;

[0058] Performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that was not successfully fused;

[0059] Inputting obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list;

[0060] Deduplicating and fusing obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and inputting obstacle information in the camera target obstacle list into a tracking node of the outdoor robot to obtain a second obstacle tracking list;

[0061] The first obstacle tracking list and the second obstacle tracking list are merged to obtain a tracking list of the outdoor robot.

[0062] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0063] First, based on the outdoor robot's radar and image sensor data, the system determines the current scene's three-dimensional and two-dimensional obstacle data. This data is then fused and judged, resolving the low reliability of target tracking caused by single-sensor target tracking methods based solely on cameras or lidar. Furthermore, the system removes duplicates from the camera obstacle list corresponding to unfused two-dimensional obstacle data by combining the obstacle lists of multiple cameras' overlapping fields of view with the camera obstacle list to obtain a camera target obstacle list. This addresses inaccurate camera recognition caused by external factors such as lighting. The first and second obstacle tracking lists are then merged, integrating the obstacles detected by lidar and cameras, improving the accuracy and reliability of the outdoor robot's obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0065] Figure 1 A schematic diagram of a method flow of a multi-sensor fusion obstacle tracking method provided in an embodiment of this specification;

[0066] Figure 2 A schematic diagram of an obstacle tracking process using multi-sensor fusion in an application scenario provided by an embodiment of this specification;

[0067] Figure 3 A schematic diagram of the internal structure of a multi-sensor fusion obstacle tracking device provided in an embodiment of this specification;

[0068] Figure 4 A schematic diagram of the internal structure of a multi-sensor fusion obstacle tracking device provided in an embodiment of this specification.

[0069] Figure 5 A schematic diagram of the internal structure of a non-volatile storage medium provided in an embodiment of this specification. DETAILED DESCRIPTION

[0070] The embodiments of this specification provide a multi-sensor fusion obstacle tracking method, device, equipment, and medium.

[0071] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this specification without creative work should fall within the scope of protection of this specification.

[0072] like Figure 1 As shown in FIG, the embodiment of this specification provides a method flow chart of a multi-sensor fusion obstacle tracking method. Figure 1 It can be seen that the method includes the following steps:

[0073] S101: Acquire radar sensing data and image sensing data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensing data, and determine two-dimensional obstacle data of the current scene based on the image sensing data.

[0074] The outdoor robot is equipped with multiple sensors, including lidar and cameras. The lidar collects radar data, while the camera collects image data. The robot then determines three-dimensional obstacle data for the current scene based on the radar data, while simultaneously determining two-dimensional obstacle data based on the image data.

[0075] To improve detection speed and accuracy, in one or more embodiments of this specification, determining three-dimensional obstacle data of the current scene based on radar sensor data and determining two-dimensional obstacle data of the current scene based on image sensor data specifically includes the following process:

[0076] First, the preset target detection model, Yolox, is used to detect 2D image data and obtain 2D obstacle data. It should be noted that Yolox-based detection of 2D image data is superior to other Yolo models because it utilizes state-of-the-art detection technologies, including a decoupled head, anchor-free detection, and a more advanced label assignment strategy. Compared to other Yolo models of all model sizes, Yolox achieves a good balance between speed and accuracy, significantly improving detection accuracy. This balance allows for improved speed and accuracy in identifying obstacles in image sensor data, resulting in improved detection precision.

[0077] To conserve computing resources and improve analysis speed, embodiments of this specification downsample the 3D point cloud data acquired by the lidar, obtaining a portion of the data within the 3D point cloud data as the 3D point cloud data to be analyzed. To determine the obstacle information captured by the 3D point cloud data, cluster analysis is performed on the downsampled 3D point cloud data to be analyzed. Specifically, for any 3D point cloud data to be analyzed, the distance between any 3D point cloud data to be analyzed and other 3D point cloud data to be analyzed is determined based on a preset balanced binary tree algorithm. Other 3D point cloud data to be analyzed that are less than a preset distance threshold are grouped into a preset first set. Any 3D point cloud data to be analyzed within the preset first set is then iteratively analyzed to perform cluster analysis on the 3D point cloud data, thereby obtaining 3D obstacle data in the current scene. It should be noted that the preset balanced binary tree algorithm is a K-dimension tree (KD-Tree), a data structure that partitions data points in k-dimensional space. To effectively find the nearest neighbors, the KD-Tree employs a divide-and-conquer approach, dividing the entire space into several smaller parts. The application of KD-Tree algorithm can be divided into two aspects. One is the algorithm for establishing the data structure of KD-Tree itself, and the other is the algorithm for performing nearest neighbor search on the established KD-Tree.

[0078] S102: Perform fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that has not been successfully fused.

[0079] After obtaining the 2D and 3D obstacle data in step S101, in order to address the issue of inaccurate detection results due to the limitations of cameras and lidar when performing target detection and tracking solely based on cameras or lidar, in the embodiments of this specification, the 2D and 3D obstacle data are fused and judged to obtain a lidar obstacle list corresponding to the 3D obstacle data and a camera obstacle list corresponding to the 2D obstacle data that failed to be fused. It should be noted that the lidar obstacle list includes both successfully fused 3D obstacle data and unfused obstacle data.

[0080] Furthermore, in one or more embodiments of this specification, before performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data, the method further includes:

[0081] The camera pre-installed on the outdoor robot is calibrated to obtain the camera's internal parameter matrix. It should be noted that the camera calibration methods include: Zhang Zhengyou calibration, Tsai two-step calibration, layered step-by-step calibration and other calibration methods, which are not specifically limited here. The coordinates of the three-dimensional obstacle data are used as the preset reference coordinates, and the camera coordinate system of the three-dimensional obstacle data and the two-dimensional obstacle data are combined through the camera pose estimation algorithm to obtain the external parameter matrix of the camera and lidar pre-installed on the outdoor robot. According to the joint calculation of the external parameter matrix and the internal parameter matrix, the conversion relationship between the lidar coordinate system and the camera coordinate system is obtained, and the three-dimensional obstacle data is projected onto the plane of the camera coordinate system based on the conversion relationship to obtain a two-dimensional detection frame of the three-dimensional obstacle data on the plane where the two-dimensional obstacle data is located.

[0082] Specifically, in one or more embodiments of this specification, the fusion judgment of the two-dimensional obstacle data and the three-dimensional obstacle data includes the following process:

[0083] The two-dimensional obstacle data and the three-dimensional obstacle data obtained in the above steps are input into the same ROS node. It should be noted that ROS (Robot Operating System) provides a series of program libraries and tools to help software developers create robot application software. It provides many functions such as hardware abstraction, device driver, library function, visualization, message passing and software package management. The fusion judgment in the embodiment of this specification is based on a node in the robot operating system. In this ROS node, a Yolox detection frame is obtained for detecting the two-dimensional image data through the preset target detection model Yolox. The above-mentioned three-dimensional obstacle data is mapped to the two-dimensional detection frame of the plane where the two-dimensional obstacle data is located, and the intersection area of ​​the Yolox detection frame, as well as the deduplication area of ​​the two-dimensional detection frame and the Yolox detection frame are obtained. Thus, the IOU area is obtained based on the intersection area and the deduplication area, so as to judge whether the two-dimensional obstacle data and the three-dimensional obstacle data are fused based on the threshold value of the IOU area and the preset IOU area. If the IOU area is greater than or equal to a preset threshold, the 2D obstacle data and 3D obstacle data corresponding to the 2D detection frame and the Yolox detection frame are determined to be the same obstacle, and the fusion is successful. If the IOU area is less than the preset threshold, the 2D obstacle data and 3D obstacle data corresponding to the 2D detection frame and the Yolox detection frame are determined to be different obstacles, and the fusion of the 2D obstacle data and the 3D obstacle data fails.

[0084] S103: Input obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list.

[0085] After the laser radar obstacle list is obtained according to the above step S102, the obstacle information in the laser radar obstacle list is input into the tracking node of the outdoor robot to obtain the first obstacle tracking list of the laser radar.

[0086] Specifically, in order to obtain optimal estimation and optimal allocation, in one or more embodiments of this specification, obstacle information in the lidar obstacle list is input into the tracking node of the outdoor robot to obtain a first obstacle tracking list, which specifically includes the following process:

[0087] First, the preset tracking trajectory of the outdoor robot is obtained. Using the Kalman filter's motion estimation of the preset tracking trajectory, the currently tracked obstacles in the LiDAR obstacle list are retrieved. It should be noted that the Kalman filter is an algorithm that uses the linear system state equation and observation data from the system's input and output to optimally estimate the system state. Because the observation data includes the effects of noise and interference in the system, the optimal estimation can also be considered a filtering process. The position, velocity, and acceleration measurements of the tracked target are often noisy at all times. The Kalman filter utilizes the target's dynamic information to eliminate the effects of noise and obtain a good estimate of the target's position. This estimate can be of the current target position, future position, or past position.

[0088] Based on the obstacle information in the LiDAR obstacle list, the IOU distance between the tracked obstacle and each obstacle in the LiDAR obstacle list is calculated to obtain the tracking cost matrix for the outdoor robot. The tracking cost matrix is ​​input into a preset Hungarian algorithm to match each obstacle corresponding to the LiDAR obstacle list to obtain the first obstacle tracking list.

[0089] S104: De-duplication and fuse the obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and input the obstacle information in the camera target obstacle list into the tracking node of the outdoor robot to obtain a second obstacle tracking list.

[0090] After obtaining the camera obstacle list corresponding to the unfused two-dimensional obstacle data based on step S102, in order to improve the accuracy of camera tracking detection, the obstacle lists detected by multiple cameras and the camera obstacle list are deduplicated and fused to obtain a successfully fused camera target obstacle list. The obstacle information in the camera target obstacle list is then input into the tracking node of the outdoor robot to obtain a tracking list for the second obstacle.

[0091] In one or more embodiments of this specification, the obstacle lists detected by multiple cameras are deduplicated and fused with the camera obstacle list to obtain a successfully fused camera target obstacle list, which specifically includes the following process:

[0092] Based on the camera coordinate system obtained through the camera calibration process, the obstacle lists detected by multiple cameras are converted to the same coordinate system. The overlapping area of ​​view for the multiple cameras in this same coordinate system is then obtained. Based on the identifiers and position coordinates of each obstacle in the obstacle lists detected by the multiple cameras and the camera obstacle list, the overlapping obstacles in the overlapping area of ​​view are obtained. Based on the determined overlapping obstacles, the obstacle data in the obstacle lists detected by the multiple cameras and the camera obstacle list are deduplicated and fused to obtain a successfully fused camera target obstacle list.

[0093] Specifically, to achieve optimal obstacle target detection, in one or more embodiments of this specification, obstacle information in the camera target obstacle list is input into the tracking node of the outdoor robot to obtain a second obstacle tracking list, which specifically includes the following process:

[0094] First, the preset tracking trajectory of the outdoor robot is obtained. The Kalman filter is used to estimate the motion of the preset tracking trajectory and obtain the current tracking obstacle in the camera's target obstacle list. Then, based on the obstacle information in the camera's target obstacle list, the IoU distance between the tracking obstacle and each obstacle in the camera's target obstacle list is calculated to obtain the camera's tracking cost matrix. This tracking cost matrix is ​​then input into the preset Hungarian algorithm to match each obstacle corresponding to the camera's target obstacle list, obtaining a second obstacle tracking list.

[0095] S105: Merge the first obstacle tracking list and the second obstacle tracking list to obtain a tracking list of the outdoor robot.

[0096] After obtaining the first obstacle tracking list of the lidar and the second obstacle tracking list of the camera based on the above process, in order to make the lidar and the camera complement each other and thus improve the accuracy and reliability of detection, in the embodiment of this specification, the first obstacle tracking list and the second obstacle tracking list are merged and deduplicated to obtain the tracking list of the outdoor robot.

[0097] like Figure 2 As shown, one or more embodiments of this specification provide a schematic diagram of the obstacle tracking process of multi-sensor fusion in an application scenario. Figure 2As can be seen, Yolox target detection is performed on camera images to obtain 2D obstacle data. Simultaneously, 3D obstacle data is obtained from the LiDAR scan point cloud through downsampling and Euclidean distance. Both 2D and 3D obstacle data are input into the same ROS node. The 3D obstacle data is mapped onto the camera plane and compared and fused with the Yolox detection box. A 3D obstacle list and a list of unsuccessfully fused 2D obstacles are published. The 3D obstacle list is input into the tracking node, which outputs the first 3D obstacle tracking list. The unsuccessfully fused 2D obstacle tracking list is then fused and deduplicated across multiple cameras. The fused list is then input into the tracking node, which outputs the second obstacle tracking list. The first and second obstacle lists are then merged to produce a unified obstacle tracking list.

[0098] The fusion and judgment of two-dimensional obstacle data and three-dimensional obstacle data solves the low target tracking reliability problem caused by single-sensor target tracking methods based solely on cameras or lidar. At the same time, the camera obstacle list corresponding to the unfused two-dimensional obstacle data is deduplicated by fusing the obstacle lists of the overlapping field of view of multiple cameras with the camera obstacle list to obtain the camera target obstacle list. This solves the problem of inaccurate camera recognition caused by external factors such as lighting. The first obstacle tracking list and the second obstacle tracking list are merged, integrating the obstacles detected by lidar and camera, improving the accuracy and reliability of obstacle detection for outdoor robots.

[0099] like Figure 3 As shown, one or more embodiments of this specification provide a schematic diagram of the internal structure of a multi-sensor fusion obstacle tracking device, which consists of Figure 3 It can be seen that the device includes:

[0100] An acquisition unit 301 is configured to acquire radar sensor data and image sensor data of the outdoor robot, to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and to determine two-dimensional obstacle data of the current scene based on the image sensor data;

[0101] A first fusion unit 302 is configured to fuse the two-dimensional obstacle data with the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the unfused two-dimensional obstacle data;

[0102] An input unit 303 is configured to input obstacle information in the laser radar obstacle list into a tracking node of the outdoor robot to obtain a first obstacle tracking list;

[0103] A second fusion unit 304 is configured to perform deduplication fusion on the obstacle lists detected by multiple cameras and the camera obstacle list to obtain a successfully fused camera target obstacle list, and input the obstacle information in the camera target obstacle list into the tracking node of the outdoor robot to obtain a second obstacle tracking list;

[0104] The merging unit 305 is configured to merge the first obstacle tracking list and the second obstacle tracking list to obtain a tracking list of the outdoor robot.

[0105] Furthermore, in one or more embodiments of this specification, the acquisition unit specifically includes:

[0106] A detection unit, configured to detect the two-dimensional image data using a preset target detection model Yolox to obtain two-dimensional obstacle data;

[0107] a sampling unit, configured to downsample the three-dimensional point cloud data to obtain three-dimensional point cloud data to be analyzed;

[0108] a distance acquisition module, configured to acquire any one of the three-dimensional point cloud data to be analyzed and obtain the distance between the any one of the three-dimensional point cloud data to be analyzed and the other three-dimensional point cloud data to be analyzed based on a preset balanced binary tree algorithm;

[0109] The clustering unit is used to divide other three-dimensional point cloud data to be analyzed that is less than a preset distance threshold into a preset first set, obtain any one of the three-dimensional point cloud data to be analyzed in the preset first set for iterative analysis, implement cluster analysis of the three-dimensional point cloud data, and obtain three-dimensional obstacle data in the current scene.

[0110] Furthermore, in one or more embodiments of this specification, the device further includes:

[0111] a calibration unit, configured to calibrate a camera pre-installed on the outdoor robot to obtain internal parameters of the camera;

[0112] a combining unit, configured to use the coordinates of the three-dimensional obstacle data as preset reference coordinates, and combine the three-dimensional obstacle data with the camera coordinate system of the two-dimensional obstacle data through a camera pose estimation algorithm to obtain an external parameter matrix of a camera and a lidar preset on the outdoor robot;

[0113] A calculation unit, configured to obtain a transformation relationship between a laser radar coordinate system and a camera coordinate system based on a joint calculation of the external parameter matrix and the internal parameter matrix;

[0114] A projection unit is configured to project the three-dimensional obstacle data onto the plane of the camera coordinate system based on the conversion relationship to obtain a two-dimensional detection frame of the three-dimensional obstacle data on the plane where the two-dimensional obstacle data is located.

[0115] Furthermore, in one or more embodiments of this specification, the first fusion unit specifically includes:

[0116] A first input unit, configured to input the two-dimensional obstacle data and the three-dimensional obstacle data into the same ROS node;

[0117] A detection frame acquisition unit is used to acquire, on the ROS node, a Yolox detection frame detected by a preset target detection model Yolox on the two-dimensional image data;

[0118] an area acquisition unit, configured to acquire an intersection area of ​​the two-dimensional detection frame and the Yolox detection frame and a deduplicated area of ​​the two-dimensional detection frame and the Yolox detection frame;

[0119] A judging unit is configured to obtain an IOU area according to the intersection area and the deduplication area, and to judge whether the two-dimensional obstacle data and the three-dimensional obstacle data are fused based on a threshold between the IOU area and a preset IOU area.

[0120] Furthermore, in one or more embodiments of this specification, the input unit specifically includes:

[0121] a first estimation unit, configured to obtain a preset tracking trajectory of the outdoor robot, estimate the motion of the preset tracking trajectory based on a Kalman filter, and obtain a current tracking obstacle in the laser radar obstacle list;

[0122] A first matrix calculation unit is configured to calculate the IOU distance between the tracked obstacle and each obstacle in the laser radar obstacle list according to the obstacle information in the laser radar obstacle list, so as to obtain a tracking cost matrix of the outdoor robot;

[0123] The first matching unit is configured to input the tracking cost matrix into a preset Hungarian algorithm to match each obstacle corresponding to the laser radar obstacle list to obtain a first obstacle tracking list.

[0124] Furthermore, in one or more embodiments of this specification, the second fusion unit specifically includes:

[0125] a conversion unit, configured to convert the obstacle lists detected by the plurality of cameras into a same coordinate system based on the camera coordinate system;

[0126] An area acquisition unit, configured to acquire an overlapping area of ​​fields of view of the plurality of cameras in the same coordinate system;

[0127] a comparing unit, configured to obtain overlapping obstacles in the overlapping field of view area according to the obstacle list detected by the multiple cameras, the identifier of each obstacle in the camera obstacle list, and the position coordinates of each obstacle;

[0128] A deduplication unit is configured to perform deduplication fusion on the obstacle lists detected by the multiple cameras and the camera obstacle list based on the overlapping obstacles to obtain a successfully fused camera target obstacle list.

[0129] Furthermore, in one or more embodiments of this specification, the second fusion unit further includes:

[0130] a second estimation unit, configured to obtain a preset tracking trajectory of the outdoor robot, estimate the motion of the preset tracking trajectory based on a Kalman filter, and obtain a current tracking obstacle in the camera target obstacle list;

[0131] A second matrix calculation unit is used to calculate the IOU distance between the tracked obstacle and each obstacle in the camera target obstacle list according to the obstacle information in the camera target obstacle list, and obtain the tracking cost matrix of the camera;

[0132] The second matching unit is configured to input the tracking cost matrix into a preset Hungarian algorithm to match each obstacle corresponding to the camera target obstacle list to obtain a second obstacle tracking list.

[0133] like Figure 4 As shown, in one or more embodiments of this specification, a schematic diagram of the internal structure of a multi-sensor fusion obstacle tracking device is provided. Figure 4 It can be seen that the equipment includes:

[0134] at least one processor 401; and,

[0135] A memory 402 in communication with the at least one processor 401; wherein,

[0136] The memory 402 stores instructions that can be executed by the at least one processor 401. The instructions are executed by the at least one processor to enable the at least one processor to:

[0137] Acquire radar sensor data and image sensor data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and determine two-dimensional obstacle data of the current scene based on the image sensor data;

[0138] Performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that was not successfully fused;

[0139] Inputting obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list;

[0140] Deduplicating and fusing obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and inputting obstacle information in the camera target obstacle list into a tracking node of the outdoor robot to obtain a second obstacle tracking list;

[0141] The first obstacle tracking list and the second obstacle tracking list are merged to obtain a tracking list of the outdoor robot.

[0142] like Figure 5 As shown in FIG, the embodiment of this specification provides a schematic diagram of the internal structure of a non-volatile storage medium. Figure 5 As can be seen, a non-volatile storage medium stores computer executable instructions 501, and the computer executable instructions 501 include:

[0143] Acquire radar sensor data and image sensor data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and determine two-dimensional obstacle data of the current scene based on the image sensor data;

[0144] Performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that was not successfully fused;

[0145] Inputting obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list;

[0146] Deduplicating and fusing obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and inputting obstacle information in the camera target obstacle list into a tracking node of the outdoor robot to obtain a second obstacle tracking list;

[0147] The first obstacle tracking list and the second obstacle tracking list are merged to obtain a tracking list of the outdoor robot.

[0148] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0149] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0150] The foregoing description is merely one or more embodiments of this specification and is not intended to limit this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of one or more embodiments of this specification are intended to be within the scope of the claims of this specification.

Claims

1. A multi-sensor fusion obstacle tracking method, characterized in that: The method comprises: Acquire radar sensor data and image sensor data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and determine two-dimensional obstacle data of the current scene based on the image sensor data; Performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that was not successfully fused; Inputting obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list; Deduplicating and fusing obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and inputting obstacle information in the camera target obstacle list into a tracking node of the outdoor robot to obtain a second obstacle tracking list; Merging the first obstacle tracking list and the second obstacle tracking list to obtain a tracking list of the outdoor robot; The fusing and judging the two-dimensional obstacle data and the three-dimensional obstacle data specifically includes: Inputting the two-dimensional obstacle data and the three-dimensional obstacle data into the same ROS node; At the ROS node, a Yolox detection frame is obtained by detecting the two-dimensional image data using a preset target detection model Yolox; Obtaining an intersection area between a two-dimensional detection frame and the Yolox detection frame and a deduplicated area between the two-dimensional detection frame and the Yolox detection frame; Obtaining an IOU area according to the intersection area and the deduplication area, and determining whether the two-dimensional obstacle data and the three-dimensional obstacle data are fused based on a threshold between the IOU area and a preset IOU area; The step of deduplicating and fusing the obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list specifically includes: Converting the obstacle lists detected by the multiple cameras into the same coordinate system based on the camera coordinate system; Obtaining an overlapping area of ​​fields of view of the multiple cameras in the same coordinate system; Obtaining overlapping obstacles in the overlapping field of view area according to the obstacle list detected by the multiple cameras, the identifier of each obstacle in the camera obstacle list, and the position coordinates of each obstacle; Based on the overlapping obstacles, the obstacle lists detected by the multiple cameras are deduplicated and fused with the camera obstacle list to obtain a successfully fused camera target obstacle list.

2. The obstacle tracking method based on multi-sensor fusion according to claim 1, characterized in that: Determining three-dimensional obstacle data of the current scene based on the radar sensor data, and determining two-dimensional obstacle data of the current scene based on the image sensor data, specifically includes: The two-dimensional image data is detected by a preset target detection model Yolox to obtain two-dimensional obstacle data; Downsampling the three-dimensional point cloud data to obtain the three-dimensional point cloud data to be analyzed; Acquire any one of the three-dimensional point cloud data to be analyzed, and obtain the distance between the any one of the three-dimensional point cloud data to be analyzed and other three-dimensional point cloud data to be analyzed based on a preset balanced binary tree algorithm; The other three-dimensional point cloud data to be analyzed that are less than the preset distance threshold are divided into a preset first set, and any one of the three-dimensional point cloud data to be analyzed in the preset first set is obtained for iterative analysis to implement cluster analysis of the three-dimensional point cloud data and obtain the three-dimensional obstacle data in the current scene.

3. The obstacle tracking method based on multi-sensor fusion according to claim 2, characterized in that: Before performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data, the method further includes: Calibrate a camera pre-installed on the outdoor robot to obtain an internal parameter matrix of the camera; The coordinates of the three-dimensional obstacle data are used as preset reference coordinates, and the three-dimensional obstacle data is combined with the camera coordinate system of the two-dimensional obstacle data through a camera pose estimation algorithm to obtain the external parameter matrix of the camera and lidar preset on the outdoor robot; Based on the joint calculation of the external parameter matrix and the internal parameter matrix, a transformation relationship between the laser radar coordinate system and the camera coordinate system is obtained; The three-dimensional obstacle data is projected onto the plane of the camera coordinate system based on the conversion relationship to obtain a two-dimensional detection frame of the three-dimensional obstacle data on the plane where the two-dimensional obstacle data is located.

4. The obstacle tracking method based on multi-sensor fusion according to claim 1, characterized in that: Inputting the obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list specifically includes: Obtaining a preset tracking trajectory of the outdoor robot, and obtaining a current tracking obstacle in the laser radar obstacle list based on a motion estimation of the preset tracking trajectory based on a Kalman filter; According to the obstacle information in the laser radar obstacle list, the IOU distance between the tracked obstacle and each obstacle in the laser radar obstacle list is calculated to obtain the tracking cost matrix of the outdoor robot; The tracking cost matrix is ​​input into a preset Hungarian algorithm to match each obstacle corresponding to the lidar obstacle list to obtain a first obstacle tracking list.

5. The obstacle tracking method based on multi-sensor fusion according to claim 1, characterized in that: Inputting the obstacle information of the camera target obstacle list into the tracking node of the outdoor robot to obtain a second obstacle tracking list specifically includes: Obtaining a preset tracking trajectory of the outdoor robot, estimating the motion of the preset tracking trajectory based on a Kalman filter, and obtaining a current tracking obstacle in the camera target obstacle list; Calculate the IOU distance between the tracked obstacle and each obstacle in the camera target obstacle list according to the obstacle information in the camera target obstacle list to obtain the tracking cost matrix of the camera; The tracking cost matrix is ​​input into a preset Hungarian algorithm to match each obstacle corresponding to the camera target obstacle list to obtain a second obstacle tracking list.

6. A multi-sensor fusion obstacle tracking device, characterized in that: The device comprises: an acquisition unit, configured to acquire radar sensing data and image sensing data of the outdoor robot, to determine three-dimensional obstacle data of the current scene based on the radar sensing data, and to determine two-dimensional obstacle data of the current scene based on the image sensing data; a first fusion unit, configured to fuse and determine the two-dimensional obstacle data and the three-dimensional obstacle data, and obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the unfused two-dimensional obstacle data; An input unit, configured to input obstacle information in the laser radar obstacle list into a tracking node of the outdoor robot to obtain a first obstacle tracking list; a second fusion unit, configured to de-duplicate and fuse the obstacle lists detected by the multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and input the obstacle information in the camera target obstacle list into the tracking node of the outdoor robot to obtain a second obstacle tracking list; a merging unit, configured to merge the first obstacle tracking list and the second obstacle tracking list to obtain a tracking list of the outdoor robot; The fusing and judging the two-dimensional obstacle data and the three-dimensional obstacle data specifically includes: Inputting the two-dimensional obstacle data and the three-dimensional obstacle data into the same ROS node; At the ROS node, a Yolox detection frame is obtained by detecting the two-dimensional image data using a preset target detection model Yolox; Obtaining an intersection area between a two-dimensional detection frame and the Yolox detection frame and a deduplicated area between the two-dimensional detection frame and the Yolox detection frame; Obtaining an IOU area according to the intersection area and the deduplication area, and determining whether the two-dimensional obstacle data and the three-dimensional obstacle data are fused based on a threshold between the IOU area and a preset IOU area; The step of deduplicating and fusing the obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list specifically includes: Converting the obstacle lists detected by the multiple cameras into the same coordinate system based on the camera coordinate system; Obtaining an overlapping area of ​​fields of view of the multiple cameras in the same coordinate system; Obtaining overlapping obstacles in the overlapping field of view area according to the obstacle list detected by the multiple cameras, the identifier of each obstacle in the camera obstacle list, and the position coordinates of each obstacle; Based on the overlapping obstacles, the obstacle lists detected by the multiple cameras are deduplicated and fused with the camera obstacle list to obtain a successfully fused camera target obstacle list.

7. A multi-sensor fusion obstacle tracking device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire radar sensor data and image sensor data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and determine two-dimensional obstacle data of the current scene based on the image sensor data; Performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that was not successfully fused; Inputting obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list; Deduplicating and fusing obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and inputting obstacle information in the camera target obstacle list into a tracking node of the outdoor robot to obtain a second obstacle tracking list; Merging the first obstacle tracking list and the second obstacle tracking list to obtain a tracking list of the outdoor robot; The fusing and judging the two-dimensional obstacle data and the three-dimensional obstacle data specifically includes: Inputting the two-dimensional obstacle data and the three-dimensional obstacle data into the same ROS node; At the ROS node, a Yolox detection frame is obtained by detecting the two-dimensional image data using a preset target detection model Yolox; Obtaining an intersection area between a two-dimensional detection frame and the Yolox detection frame and a deduplicated area between the two-dimensional detection frame and the Yolox detection frame; Obtaining an IOU area according to the intersection area and the deduplication area, and determining whether the two-dimensional obstacle data and the three-dimensional obstacle data are fused based on a threshold between the IOU area and a preset IOU area; The step of deduplicating and fusing the obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list specifically includes: Converting the obstacle lists detected by the multiple cameras into the same coordinate system based on the camera coordinate system; Obtaining an overlapping area of ​​fields of view of the multiple cameras in the same coordinate system; Obtaining overlapping obstacles in the overlapping field of view area according to the obstacle list detected by the multiple cameras, the identifier of each obstacle in the camera obstacle list, and the position coordinates of each obstacle; Based on the overlapping obstacles, the obstacle lists detected by the multiple cameras are deduplicated and fused with the camera obstacle list to obtain a successfully fused camera target obstacle list.

8. A non-volatile storage medium storing computer-executable instructions, characterized in that: The computer-executable instructions include: Acquire radar sensor data and image sensor data of the outdoor robot to determine three-dimensional obstacle data of the current scene based on the radar sensor data, and determine two-dimensional obstacle data of the current scene based on the image sensor data; Performing fusion judgment on the two-dimensional obstacle data and the three-dimensional obstacle data to obtain a lidar obstacle list corresponding to the three-dimensional obstacle data and a camera obstacle list corresponding to the two-dimensional obstacle data that was not successfully fused; Inputting obstacle information in the laser radar obstacle list into the tracking node of the outdoor robot to obtain a first obstacle tracking list; Deduplicating and fusing obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list, and inputting obstacle information in the camera target obstacle list into a tracking node of the outdoor robot to obtain a second obstacle tracking list; Merging the first obstacle tracking list and the second obstacle tracking list to obtain a tracking list of the outdoor robot; The fusing and judging the two-dimensional obstacle data and the three-dimensional obstacle data specifically includes: Inputting the two-dimensional obstacle data and the three-dimensional obstacle data into the same ROS node; At the ROS node, a Yolox detection frame is obtained by detecting the two-dimensional image data using a preset target detection model Yolox; Obtaining an intersection area between a two-dimensional detection frame and the Yolox detection frame and a deduplicated area between the two-dimensional detection frame and the Yolox detection frame; Obtaining an IOU area according to the intersection area and the deduplication area, and determining whether the two-dimensional obstacle data and the three-dimensional obstacle data are fused based on a threshold between the IOU area and a preset IOU area; The step of deduplicating and fusing the obstacle lists detected by multiple cameras with the camera obstacle list to obtain a successfully fused camera target obstacle list specifically includes: Converting the obstacle lists detected by the multiple cameras into the same coordinate system based on the camera coordinate system; Obtaining an overlapping area of ​​fields of view of the multiple cameras in the same coordinate system; Obtaining overlapping obstacles in the overlapping field of view area according to the obstacle list detected by the multiple cameras, the identifier of each obstacle in the camera obstacle list, and the position coordinates of each obstacle; Based on the overlapping obstacles, the obstacle lists detected by the multiple cameras are deduplicated and fused with the camera obstacle list to obtain a successfully fused camera target obstacle list.

Images