Target tracking method and device, computer device and storage medium

Abstract

The application relates to a target tracking method and device, computer equipment and a storage medium. The method comprises the following steps: determining the three-dimensional detection position of each target in a target pedestrian image at a current acquisition time according to a point cloud and the target pedestrian image; matching the three-dimensional detection position of any target in the target pedestrian image with the three-dimensional prediction position of each target in a previous target pedestrian image to determine a first matching result; judging whether any target is successfully tracked according to the first matching result; if any target is successfully tracked, adjusting the three-dimensional detection position of a first target frame by using a preset algorithm model to obtain an updated three-dimensional position of the first target frame. By decoupling the Kalman filter of 2D and 3D, the 3D information of a target pedestrian is acquired by using laser radar and visual sensor data, the 2D detection frame information of the target pedestrian is acquired by using visual sensor data, the data is fully utilized to track the target pedestrian, and the target tracking accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of information and communication technology, and in particular to a target tracking method, apparatus, computer device, and storage medium. Background Technology

[0002] In robot environmental perception, the most important aspect is 3D perception of pedestrians, which includes both position and velocity. Unlike 2D vision detection, 3D object detection can be used for spatial path planning, robot navigation, and obstacle avoidance. However, 3D object detection presents greater challenges, requiring more output parameters to specify the bounding box facing the target in 3D space. Existing 3D object detection algorithms can be broadly categorized into three types based on the sensor used: vision, laser point cloud, and multimodal fusion. Vision methods are widely used in object detection due to their low cost and rich texture features. Laser point cloud data, with its accurate depth information and distinct 3D spatial features, is also widely applied in 3D object detection. However, the input data resolution of methods such as LiDAR is typically lower than that of images, resulting in lower accuracy for detecting targets at greater distances. Summary of the Invention

[0003] Therefore, it is necessary to provide a robot to address the above-mentioned technical problems. The robot is equipped with a lidar and a vision sensor. The robot includes a memory and a processor. The memory stores executable program code, and the processor can achieve stable and accurate target tracking when executing the executable program code.

[0004] In a first aspect, the processor on the robot provided in this application executes the executable program code to perform the following steps:

[0005] The robot acquires target point cloud data in real time using LiDAR and captures images of pedestrians in real time using visual sensors.

[0006] Using a pre-defined pedestrian detection model, pedestrian detection is performed on the current pedestrian image to obtain at least one first target box in the current pedestrian image;

[0007] Each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image, and a second target box matching each first target box is obtained from the at least one second target box, wherein the previous pedestrian image is an image acquired by the visual sensor before acquiring the current pedestrian image;

[0008] Based on each first target bounding box and target point cloud data in the current pedestrian image, the 3D detection position of each first target bounding box is determined. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image.

[0009] When the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked. If the first target box is successfully tracked, the 3D detection position of the first target box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target box.

[0010] In one embodiment, target point cloud data of the robot's location space is acquired in real time using lidar, including:

[0011] The robot acquires multiple primary data points in real time from its location using LiDAR.

[0012] Perform coordinate transformation on each of the current multiple first data points to obtain the first target point corresponding to each first data point after coordinate transformation;

[0013] A line detection algorithm model is used to filter and detect lines from multiple first target points to obtain multiple second target points;

[0014] Clustering multiple second target points yields multiple first target classes;

[0015] Determine whether the number of points in each of the multiple first target classes is within a preset range, and determine the first target class whose number of points is within the preset range as the second target class;

[0016] The set of all points in the second target class is defined as the target point cloud data.

[0017] In one embodiment, matching each first target box in the current pedestrian image with at least one second target box in the previous pedestrian image, and obtaining a second target box that matches each first target box from the at least one second target box, includes:

[0018] For each second target box in at least one second target box in the previous row of people image, predict the predicted target box corresponding to each second target box in at least one second target box in the previous row of people image;

[0019] Match each first target box in the current pedestrian image with the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and determine the predicted target box corresponding to each first target box in the current pedestrian image from the multiple predicted target boxes corresponding to at least one second target box in the previous pedestrian image;

[0020] Based on the predicted target box corresponding to each first target box in the current pedestrian image, a second target box that matches each first target box is obtained from at least one second target box.

[0021] In one embodiment, matching each first target box in the current pedestrian image with the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and determining the predicted target box corresponding to each first target box in the current pedestrian image from the plurality of predicted target boxes corresponding to at least one second target box in the previous pedestrian image, includes:

[0022] Calculate the cross-union ratio (CUP) between each first target box in the current pedestrian image and the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and obtain multiple CUP values ​​for each first target box in the current pedestrian image;

[0023] The predicted target box corresponding to the smallest cross-union ratio among the multiple cross-union ratios of each first target box in the current pedestrian image is taken as the predicted target box corresponding to each first target box in the current pedestrian image.

[0024] In one embodiment, there are several frames of pedestrian images between the previous pedestrian image and the current target pedestrian image; accordingly, each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image, and a second target box matching each first target box is obtained from the at least one second target box, including:

[0025] The previous pedestrian image, the current target pedestrian image, and several frames of pedestrian images are used as a set of matching images;

[0026] Match the target boxes in each pair of adjacent pedestrian images in a set of matching images to obtain the matching target boxes in each pair of adjacent pedestrian images in a set of matching images.

[0027] Based on the matching bounding boxes in each pair of adjacent pedestrian images in a set of matching images, determine at least one second bounding box that matches each first bounding box.

[0028] In one embodiment, the target bounding boxes in every two adjacent frames of pedestrian images in a set of matching images are matched to obtain the matching target bounding boxes in every two adjacent frames of pedestrian images in a set of matching images, including:

[0029] The earliest captured image among any two adjacent frames in a set of matching images is used as the first matching image, and the matching image in a set of matching images that is an adjacent frame to the first matching image and was captured at a later time than the first matching image is used as the second matching image.

[0030] The first algorithm model is used to predict the position of any target box in the first matching image in the second matching image, so as to obtain the predicted target box of any target box in the first matching image in the second matching image;

[0031] The second preset algorithm is used to match any target box in the second matching image with any target box in the first matching image and the predicted target box in the second matching image, so as to obtain the predicted target box in the first matching image that matches any target box in the second matching image;

[0032] Based on the correspondence between the predicted bounding box in the first matching image and the bounding box in the second matching image, the matching bounding boxes in the first and second matching images are obtained.

[0033] In one embodiment, determining the 3D detection position of each first target bounding box based on the target point cloud data in the current pedestrian image includes:

[0034] Project each point in the target point cloud data onto the current pedestrian image;

[0035] Based on the position of each first target box in the current pedestrian image and the position of each point in the target point cloud data projected onto the target pedestrian image, the target points in the target point cloud data belonging to each first target box in the current target pedestrian image are obtained;

[0036] Based on the 3D coordinates of the points in the target point cloud data that belong to each first target box in the current pedestrian image, the 3D detection position of each first target box in the current target pedestrian image is obtained.

[0037] Secondly, this application provides a target tracking method, the method comprising:

[0038] The robot acquires target point cloud data in real time using LiDAR and captures images of pedestrians in real time using visual sensors.

[0039] Using a pre-defined pedestrian detection model, pedestrian detection is performed on the current pedestrian image to obtain at least one first target box in the current pedestrian image;

[0040] Each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image, and a second target box matching each first target box is obtained from the at least one second target box, wherein the previous pedestrian image is an image acquired by the visual sensor before acquiring the current pedestrian image;

[0041] Based on each first target bounding box and target point cloud data in the current pedestrian image, the 3D detection position of each first target bounding box is determined. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image.

[0042] When the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D position of the second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked. If the first target box is successfully tracked, the 3D detection position of the first target box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target box.

[0043] In one embodiment, target point cloud data of the robot's location space is acquired in real time using lidar, including:

[0044] The robot acquires multiple primary data points in real time from its location using LiDAR.

[0045] Perform coordinate transformation on each of the current multiple first data points to obtain the first target point corresponding to each first data point after coordinate transformation;

[0046] A line detection algorithm model is used to filter and detect lines from multiple first target points to obtain multiple second target points;

[0047] Clustering multiple second target points yields multiple first target classes;

[0048] Determine whether the number of points in each of the multiple first target classes is within a preset range, and determine the first target class whose number of points is within the preset range as the second target class;

[0049] The set of all points in the second target class is defined as the target point cloud data.

[0050] Thirdly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0051] The robot acquires target point cloud data in real time using LiDAR and captures images of pedestrians in real time using visual sensors.

[0052] Using a pre-defined pedestrian detection model, pedestrian detection is performed on the current pedestrian image to obtain at least one first target box in the current pedestrian image;

[0053] Each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image, and a second target box matching each first target box is obtained from the at least one second target box, wherein the previous pedestrian image is an image acquired by the visual sensor before acquiring the current pedestrian image;

[0054] Based on each first target bounding box and target point cloud data in the current pedestrian image, the 3D detection position of each first target bounding box is determined. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image.

[0055] When the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked. If the first target box is successfully tracked, the 3D detection position of the first target box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target box.

[0056] The aforementioned robot, target tracking method, computer equipment, and storage medium acquire target point cloud data of the robot's location in real time via LiDAR and obtain current pedestrian images of the robot's location in real time via a vision sensor. A preset pedestrian detection model is used to detect pedestrians in the current pedestrian images, obtaining at least one first target bounding box in the current pedestrian images. Each first target bounding box in the current pedestrian images is matched with at least one second target bounding box in the previous pedestrian image, obtaining a second target bounding box matching each first target bounding box from the at least one second target bounding box. The previous pedestrian image is obtained by the vision sensor when it acquires data... Before acquiring the current pedestrian image, an image is obtained. Based on each first target bounding box and target point cloud data in the current pedestrian image, the 3D detection position of each first target bounding box is determined. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image. When the distance between the 3D detection position of the first target bounding box in the current pedestrian image and the 3D detection position of the corresponding second target bounding box meets a preset threshold, it indicates that the first target bounding box has been successfully tracked. If the first target bounding box is successfully tracked, the 3D detection position of the first target bounding box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target bounding box. By decoupling 2D and 3D Kalman filters, 3D information of the target pedestrian is acquired using LiDAR and visual sensor data, and 2D detection bounding box information of the target pedestrian is acquired using visual sensor data. By making full use of all data to track the target pedestrian, the accuracy of the detection results is improved. Attached Figure Description

[0057] Figure 1 This is a diagram illustrating the application environment of a target tracking method in one embodiment.

[0058] Figure 2 This is a flowchart illustrating a target tracking method in one embodiment;

[0059] Figure 3 This is a schematic diagram illustrating the time of data acquisition in a target tracking method in one embodiment;

[0060] Figure 4 This is a schematic diagram of a pedestrian image detection box in one embodiment;

[0061] Figure 5 Here is a top view of point cloud filtering in one embodiment;

[0062] Figure 6 This is a flowchart illustrating the target tracking method in another embodiment;

[0063] Figure 7 This is a flowchart illustrating the target tracking method in yet another embodiment;

[0064] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] The target tracking method provided in this application can be applied to real-time video surveillance scenarios as well as video post-processing scenarios; this application does not specifically limit its application. For ease of understanding, this application takes a real-time video surveillance scenario as an example, demonstrating how the method provided can enable a robot to track and detect target pedestrians. The method provided in this application can be applied to scenarios such as… Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. Specifically, it transmits captured video streams and point cloud data acquired by LiDAR to server 104, which then tracks the target based on the video streams and point cloud data. A data storage system can store the video streams and point cloud data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers.

[0067] Terminal 102 is a device capable of acquiring video streams, which may include, but is not limited to, robots with visual sensors, various personal computers, laptops, smartphones, tablets, and logistics network devices. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0068] In some embodiments, the terminal is a robot, such as... Figure 2 As shown in the illustration, this application provides a method performed by a robot equipped with a lidar and a vision sensor. The robot includes a memory and a processor. The memory stores executable program code, and the processor executes the executable program code to perform the following steps:

[0069] Step 202: Real-time acquisition of target point cloud data in the robot's space using LiDAR, and real-time capture of pedestrian images in the robot's space using a visual sensor.

[0070] Point clouds are collections of sampled points with spatial coordinates acquired by LiDAR (LiDAR radar). Essentially, a point cloud is a set of vectors in a three-dimensional coordinate system, containing the detected geometric position information. Specifically, when a robot is equipped with a LiDAR to detect and track pedestrians, the LiDAR emits lasers at a preset frequency. The reflection of these lasers acquires point cloud data corresponding to "obstacles" in the surrounding environment. These "obstacles" could be pedestrians, buildings, vegetation, or other objects in the environment. Since point clouds are vectors with spatial coordinates, the 3D position of a target pedestrian can be determined by finding the point cloud corresponding to that pedestrian.

[0071] It's important to note that there's a certain degree of coupling between the LiDAR and the vision sensor. While the LiDAR is detecting pedestrians, the vision sensor is also acquiring pedestrian images. However, the frequency at which the vision sensor takes pictures and the frequency at which the LiDAR emits lasers are typically different. For example, an RGB camera might have a frame rate of 30 FPS, while the LiDAR's frame rate is only 10 FPS. This frame rate difference means that the RGB image and the LiDAR point cloud cannot match all frames. Therefore, each time the LiDAR acquires point cloud data, it needs to identify a target pedestrian image from the pedestrian images acquired by the vision sensor. For instance, if we consider the moment the LiDAR acquires point cloud data as the current acquisition time, the target pedestrian image should satisfy the condition that its capture time is before and closest to the current acquisition time. This ensures that the error is minimized when the point cloud data and the pedestrian image are fused. See [link to relevant documentation]. Figure 3If the target point cloud data is acquired at time T, then the corresponding target pedestrian image should be the pedestrian image acquired at time K. It should be added that the robot's current spatial location is captured in real-time using a visual sensor; if a pedestrian is present in the current image, then that image is identified as the current pedestrian image.

[0072] Step 204: Use a preset pedestrian detection model to perform pedestrian detection on the current pedestrian image to obtain at least one first target box in the current pedestrian image;

[0073] Pedestrian detection models aim to obtain the coordinates of the 2D detection box of a pedestrian in an image. General object detection models can achieve this well. Common models include Faster R-CNN, CenterNet, and the YOLO series. Among them, lightweight object detection models can easily achieve real-time detection on GPUs (Graphics Processing Units) or DSPs (Digital Signal Processing).

[0074] The pedestrian detection model takes an RGB image as input and outputs the pedestrian's position within the image, such as... Figure 4 The white box in the image shows the detection result of the pedestrian in the current frame. The position of the detection box can be represented by an array (x, y, w, h), where x and y are the coordinates of the top-left corner of the detection box in the image coordinate system, and w and h represent the pixel width and pixel height of the detection box, respectively. At least one first target box is defined in the current pedestrian image. This first target box can be one, two, or more first target boxes. Generally, when detecting pedestrian images, multiple pedestrians can be detected, meaning multiple target boxes will be detected. In this embodiment, one first target box represents one pedestrian in the pedestrian image. A motion model is established for the target to calculate its state in the next frame of the pedestrian image. In this embodiment, the target motion between adjacent frames is approximated as constant velocity motion.

[0075] Step 206: Match each first target box in the current pedestrian image with at least one second target box in the previous pedestrian image, and obtain a second target box that matches each first target box from the at least one second target box, wherein the previous pedestrian image is an image acquired by the visual sensor before acquiring the current pedestrian image;

[0076] It is understandable that when using a pedestrian detection model to detect each pedestrian image, the corresponding target box for each pedestrian can be obtained. However, the correspondence between target boxes in two pedestrian images cannot be determined. Therefore, an algorithm is needed to match the two. At least one second target box can be one second target box, or two or more second target boxes. This application does not specifically limit the method of matching the first target box and the second target box, including but not limited to using Kalman filtering or particle filtering algorithms to predict the second target box, and then matching it according to the matching algorithm.

[0077] In addition, in this embodiment of the application, it is necessary to obtain the three-dimensional position information of the tracking target. According to the principle of obtaining three-dimensional position information, for any target, the pedestrian image and the point cloud data must be synchronized in time. That is, the previous pedestrian image was obtained before the current pedestrian image, and it is the pedestrian image closest to the timestamp of the previous point cloud data acquisition.

[0078] Specifically, if the acquisition time of the current pedestrian image is defined as time K and the acquisition time of the previous pedestrian image is defined as time K-1, then Kalman filtering can be used to obtain the predicted state of at least one second target box in the previous pedestrian image at time K. Then, the state of at least one first target box at time K is matched with the predicted state of at least one second target box at time K to determine the second target box that successfully matches the first target box. An important parameter in this algorithm is the time difference from time K-1 to time K, which can be determined according to the operating frequency of the video sensor or directly by calculating the time difference based on the acquisition time.

[0079] During matching, the Hungarian matching algorithm can be used to associate the predicted results with the detection results of the current frame. For example, if the previous frame outputs m tracked objects and the current frame detects n target objects, the Hungarian algorithm constructs an association matrix to link the target objects in the two frames. The core of the association matrix is ​​to measure the distance between two objects using the intersection-over-union (IoU) ratio. Because the Kalman filter updates its parameters very frequently and the changes in pedestrians between two frames are small, the IoU ratio is large, resulting in a high matching success rate and thus more stable tracking results. Figure 4 For example, the intersection-union ratio of the white box and the gray box is very large, which easily meets the threshold condition and thus the matching is successful.

[0080] Furthermore, in the Hungarian matching algorithm, the type of matching index parameter can be determined based on the types of data included in the motion state and the types of data included in the features. For example, if the motion state includes both position and velocity, the predicted value of the tracking target's motion state in the next frame can be regarded as a point composed of position and velocity, and the detected value of the detected target's motion state in the next frame can also be regarded as a point.

[0081] Therefore, motion state can include two types of data, and the matching index parameter can accordingly include Euclidean distance, which is the Euclidean distance between the predicted value of the tracked target's motion state in the next frame and the detected value of the detected target's motion state in the next frame. Furthermore, Euclidean distance, as a reference for measuring the difference between two samples, may fail to measure due to the different metrics used for position and velocity, given its calculation process. Therefore, in practical implementation, the matching index parameter may not include Euclidean distance but instead include Mahalanobis distance. By using Mahalanobis distance, the measurement process is not affected by different metrics, and Mahalanobis distance can also eliminate interference from correlations between variables (such as between position and velocity).

[0082] Step 208: Determine the 3D detection position of each first target box based on each first target box and the target point cloud data in the current pedestrian image. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image.

[0083] As explained above, the current pedestrian image and the target point cloud data are synchronized in time. For any target pedestrian, both the current pedestrian image and the point cloud data contain the target pedestrian's location information. By converting between two-dimensional and three-dimensional information, the three-dimensional detection location of the target pedestrian can be obtained.

[0084] Specifically, given the known installation parameters of the video sensor and LiDAR on the robot, including their position and angle, data can be transferred from the sensor coordinate system (LiDAR and video sensor) to the robot coordinate system. Therefore, data acquired by the LiDAR and video sensor can be transformed using the robot coordinate system. This allows the point cloud data to be projected onto the pedestrian image captured by the video sensor. By determining whether the point cloud is projected into a detection box in the pedestrian image, the position of each detection box can be determined. This position is determined by the point cloud projected into the detection box and is a three-dimensional location. Furthermore, the three-dimensional detection position of the target box can also be determined by projecting the pedestrian image onto the three-dimensional space containing the point cloud data.

[0085] Step 210: When the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked. If the first target box is successfully tracked, the 3D detection position of the first target box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target box.

[0086] Given the continuity of the tracked target across different frames, and the fact that video frames in the same video stream are typically of the same size and correspond to the same viewpoint, if the tracked target appears in a certain region of a video frame, and also appears in the next frame, then the position of the tracked target in the next frame is usually also near that region. Based on this, in actual implementation, the distance between the three-dimensional positions of the two target boxes can be used to further determine whether the tracked target has been successfully tracked.

[0087] Since the robot's perception of pedestrians is 3D, its 3D information includes position and velocity. Therefore, it is necessary to fuse information based on the 3D detected position to calculate the velocity. This application approximates the target motion between two adjacent frames as constant velocity motion, independent of the motion of other targets and the camera motion. The state of each target is represented by a 6-dimensional vector (x, y, z, v). xt v yt v zt To describe it completely, (x, y, z) represents the pedestrian's coordinates in the world coordinate system, and (v... xt v yt v zt The numbers () represent velocities in different directions. Then, a 3D Kalman filter can be used to predict the current position coordinates and velocity of the pedestrian being tracked.

[0088] Specifically, the velocity of any target at the current acquisition time is determined based on its 3D detection position at the current acquisition time and its 3D position information at the previous acquisition time. When calculating the velocity using position information and time intervals, the calculation is performed separately along the three axes of the 3D coordinate system, i.e., using the 3D detection position (x...). t-1 y t-1 , z t-1 ) and (x t y t , z t The calculated velocity is (v) xt ,v yt ,v zt ).

[0089] When using a 3D Kalman filter to predict pedestrian positions, for any target, it is possible to determine the 3D detection position (x) of any target at the current acquisition time. t y t , z t The three-dimensional predicted position (x′) at the current acquisition time corresponding to the successfully matched second target box. t y′ t , z′ tBased on these two positions, the actual position of any target at the current acquisition time can be determined. The three-dimensional detection position (x, y) can be directly used as the reference point. t y t , z t ) or three-dimensional predicted location (x′ t y′ t , z′ t The position of any target at the current acquisition time can be determined by fusing the two positions using a preset algorithm. Therefore, the three-dimensional position information of any target at the current acquisition time includes position and velocity, and can be represented as (x, y, z, v). xt v yt v zt ).

[0090] In addition, when matching each first target box in the current pedestrian image with at least one second target box in the previous pedestrian image, and failing to obtain a second target box that matches the first target box from at least one second target box, it indicates that the first target box has not been tracked successfully; or when the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the second target box that matches the corresponding first target box does not meet a preset threshold, it indicates that the first target box has not been tracked successfully; when the first target box has not been tracked successfully, the 3D detection position of the untracked first target box is determined directly by combining the untracked first target box with the target point cloud data, and the 3D detection position of the untracked first target box is directly output.

[0091] In the method provided in the above embodiments, target point cloud data of the robot's space is acquired in real time by LiDAR, and the current pedestrian image of the robot's space is captured in real time by a vision sensor; pedestrian detection is performed on the current pedestrian image using a preset pedestrian detection model to obtain at least one first target box in the current pedestrian image; each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image to obtain a second target box that matches each first target box from the at least one second target box, wherein the previous pedestrian image is an image acquired by the vision sensor before acquiring the current pedestrian image; the three-dimensional detection position of each first target box is determined based on each first target box in the current pedestrian image and the target point cloud data, wherein the target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image; when the distance between the three-dimensional detection position of the first target box in the current pedestrian image and the three-dimensional detection position of the corresponding second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked; if the first target box is successfully tracked, the three-dimensional detection position of the first target box is adjusted using a preset algorithm model to obtain the updated three-dimensional position of the first target box. By decoupling 2D and 3D Kalman filters, 3D information of the target pedestrian is obtained using LiDAR and visual sensor data, while 2D detection box information of the target pedestrian is obtained using visual sensor data. This fully utilizes all data to track the target pedestrian, improving data utilization and target tracking accuracy.

[0092] In one embodiment, target point cloud data of the robot's location space is acquired in real time using lidar, including:

[0093] The robot acquires multiple primary data points in real time from its location using LiDAR.

[0094] Perform coordinate transformation on each of the current multiple first data points to obtain the first target point corresponding to each first data point after coordinate transformation;

[0095] A line detection algorithm model is used to filter and detect lines from multiple first target points to obtain multiple second target points;

[0096] Clustering multiple second target points yields multiple first target classes;

[0097] Determine whether the number of points in each of the multiple first target classes is within a preset range, and determine the first target class whose number of points is within the preset range as the second target class;

[0098] The set of all points in the second target class is defined as the target point cloud data.

[0099] The raw point cloud data is filtered using a spatial top-down view. For example, by acquiring a 3D spatial view of the point cloud, multiple combined shapes formed by the point cloud in the spatial view are determined. Among these combined shapes, those meeting preset conditions are identified, and point clouds forming these preset shapes are removed. The spatial view includes a top view, a front view, and a side view, and the 3D space and all point cloud data are in the same 3D coordinate system. Combined shapes are defined as the shapes presented by a portion of the point cloud projected into the spatial view. The preset conditions are set based on unwanted point cloud shapes, such as straight lines.

[0100] Specifically, first project all point cloud data into a top view, see [link to top view]. Figure 5 A bird's-eye view Figure 1 Then, based on the shape characteristics of the human foot, some points such as those on walls and counters are filtered out through line detection. (See [link]). Figure 5 A bird's-eye view Figure 2 Finally, the filtered point cloud is clustered, for example using a common density-based clustering algorithm. The clustering results can be found in [link to relevant documentation]. Figure 5 A bird's-eye view Figure 3 The remaining point cloud in the image is clustered into two classes. It should be noted that in actual operation, to improve accuracy, the specific target shape can be determined based on human body characteristics, and the points in the top view can be filtered accordingly.

[0101] In the method provided in the above embodiments, the original point cloud data is filtered by a preset algorithm and a spatial view of the point cloud to remove point cloud data that cannot represent pedestrians, thereby improving the accuracy of pedestrian three-dimensional position determination.

[0102] In one embodiment, see Figure 6 Matching each first bounding box in the current pedestrian image with at least one second bounding box in the previous pedestrian image, and obtaining a second bounding box that matches each first bounding box from the at least one second bounding box, including:

[0103] Step 602: Predict each second target box in at least one second target box in the previous row of people image to obtain the predicted target box corresponding to each second target box in at least one second target box in the previous row of people image;

[0104] Step 604: Match each first target box in the current pedestrian image with the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and determine the predicted target box corresponding to each first target box in the current pedestrian image from the multiple predicted target boxes corresponding to at least one second target box in the previous pedestrian image.

[0105] Step 606: Based on the predicted target box corresponding to each first target box in the current pedestrian image, obtain a second target box that matches each first target box from at least one second target box.

[0106] In this embodiment, the Kalman filter algorithm is used as an example to explain the matching process between the first and second target boxes. The acquisition time of the previous pedestrian image is defined as time K-1, and the acquisition time of the current pedestrian image is time K. The state of the second target box at time K is predicted based on the state at time K-1. Then, the state of the first target box detected at time K is matched with the predicted state of the second target box at time K to determine the second target box that successfully matches the first target box. The second target box represents the position of the tracking target at time K-1, and the first target box represents the position of the tracking target at time K. The time parameter in the Kalman filter prediction is determined based on time K and time K-1.

[0107] The method provided in the above embodiments predicts the position of the target box at the next moment, and then matches the prediction result with the detection result at the next moment to determine whether each target has been successfully tracked. This method can effectively track targets in the image and provides accurate data for the determination of three-dimensional position, making the entire target tracking and position determination model highly robust and stable.

[0108] In one embodiment, matching each first target box in the current pedestrian image with the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and determining the predicted target box corresponding to each first target box in the current pedestrian image from the plurality of predicted target boxes corresponding to at least one second target box in the previous pedestrian image, includes:

[0109] Calculate the cross-union ratio (CUP) between each first target box in the current pedestrian image and the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and obtain multiple CUP values ​​for each first target box in the current pedestrian image;

[0110] The predicted target box corresponding to the smallest cross-union ratio among the multiple cross-union ratios of each first target box in the current pedestrian image is taken as the predicted target box corresponding to each first target box in the current pedestrian image.

[0111] In one embodiment, there are several frames of pedestrian images between the previous pedestrian image and the current target pedestrian image; correspondingly, see [link to relevant documentation]. Figure 7 Matching each first bounding box in the current pedestrian image with at least one second bounding box in the previous pedestrian image, and obtaining a second bounding box that matches each first bounding box from the at least one second bounding box, including:

[0112] Step 702: Combine the previous pedestrian image, the current target pedestrian image, and several frames of pedestrian images into a set of matching images;

[0113] Step 704: Match the target boxes in each pair of adjacent pedestrian images in a set of matching images to obtain the matching target boxes in each pair of adjacent pedestrian images in a set of matching images.

[0114] Step 706: Based on the matching target boxes in each pair of adjacent pedestrian images in a set of matching images, determine at least one second target box that matches each first target box.

[0115] In practical applications, the data acquisition frequency of video sensors is generally higher than that of LiDAR. (See [link / reference]) Figure 3 As can be seen from the above explanation of the embodiments, the current point cloud data acquisition time is time T, and the corresponding previous pedestrian image acquisition time is time K-3; the previous point cloud data acquisition time is time T-1, and the corresponding current pedestrian image acquisition time is time K; therefore, several frames of pedestrian images acquired by the video sensor are not utilized, but in order to improve the accuracy of target tracking, in this embodiment, all data acquired by the video sensor is utilized.

[0116] Specifically, for video frames acquired by the video sensor, see [link to relevant documentation]. Figure 3 In the process of matching the first target box with the second target box, the time interval between the previous pedestrian image (time K-3) and the current pedestrian image (time K) can be used directly for prediction matching. Alternatively, the time parameter of the Kalman filter algorithm can be changed, and the matching relationship between the second target box of the previous pedestrian image (time K-3) and the first target box of the current pedestrian image (time K) can be indirectly determined through the relationship between time K-3, K-2, K-1 and K.

[0117] In the method provided in the above embodiments, the previous pedestrian image, the current target pedestrian image, and several frames of pedestrian images are taken as a set of matching images; the target boxes in every two adjacent frames of pedestrian images in the set of matching images are matched to obtain matching target boxes in every two adjacent frames of pedestrian images in the set of matching images; based on the matching target boxes in every two adjacent frames of pedestrian images in the set of matching images, at least one second target box is determined to match each first target box. By decoupling 2D and 3D Kalman filtering, both 2D and 3D information are accurately predicted simultaneously, making full use of all data and improving system stability and algorithm robustness.

[0118] In one embodiment, the target bounding boxes in every two adjacent frames of pedestrian images in a set of matching images are matched to obtain the matching target bounding boxes in every two adjacent frames of pedestrian images in a set of matching images, including:

[0119] The earliest captured image among any two adjacent frames in a set of matching images is used as the first matching image, and the matching image in a set of matching images that is an adjacent frame to the first matching image and was captured at a later time than the first matching image is used as the second matching image.

[0120] The first algorithm model is used to predict the position of any target box in the first matching image in the second matching image, so as to obtain the predicted target box of any target box in the first matching image in the second matching image;

[0121] The second preset algorithm is used to match any target box in the second matching image with any target box in the first matching image and the predicted target box in the second matching image, so as to obtain the predicted target box in the first matching image that matches any target box in the second matching image;

[0122] Based on the correspondence between the predicted bounding box in the first matching image and the bounding box in the second matching image, the matching bounding boxes in the first and second matching images are obtained.

[0123] In one embodiment, determining the 3D detection position of each first target bounding box based on the target point cloud data in the current pedestrian image includes:

[0124] Project each point in the target point cloud data onto the current pedestrian image;

[0125] Based on the position of each first target box in the current pedestrian image and the position of each point in the target point cloud data projected onto the target pedestrian image, the target points in the target point cloud data belonging to each first target box in the current target pedestrian image are obtained;

[0126] Based on the 3D coordinates of the points in the target point cloud data that belong to each first target box in the current pedestrian image, the 3D detection position of each first target box in the current target pedestrian image is obtained.

[0127] It should be noted that the points in the target point cloud can be clustered to obtain multiple first point cloud clusters. Each first point cloud cluster is projected onto the target pedestrian image. For any detection box, the ratio between the number of points in each first point cloud cluster projected onto any detection box and the number of points in each first point cloud cluster is calculated. If there is a ratio among all ratios that is greater than a first preset threshold, the points in the first point cloud cluster corresponding to the largest ratio among all ratios that are greater than the first preset threshold are determined as the points of the corresponding detection box. The centroid of all points corresponding to each detection box is calculated and determined as the three-dimensional detection position of the target corresponding to the detection box in the first target pedestrian image.

[0128] As is understandable, LiDAR determines the position and outline of detected "obstacles" by emitting and receiving reflected laser light, using vectors. Therefore, for a single "obstacle," there are generally multiple vectors (point clouds). Density-based clustering algorithms are used to cluster all point clouds, resulting in multiple point cloud clusters, with each cluster representing a single "obstacle."

[0129] It's important to note that projecting point cloud clusters onto the target pedestrian image, or projecting point clouds onto the target pedestrian image, is accomplished through coordinate system transformations. Point cloud clusters are introduced because each cluster can represent an "obstacle," and the proportion of points within a cluster falling into any detection bounding box determines whether the point cloud data and the detection box match successfully. In essence, if a majority of points in a cluster fall into any detection box, it indicates the presence of a target within that box, and the target's 3D position can be calculated from the points falling into that box.

[0130] The method provided in the above embodiments projects three-dimensional point cloud data onto two-dimensional pedestrian images through the relationship between sensor coordinate systems, performs data matching and fusion, and then determines the three-dimensional position of the pedestrian. This improves the accuracy of pedestrian position determination and the accuracy of pedestrian tracking models, and solves problems such as poor algorithm robustness and unstable tracking.

[0131] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0132] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores point cloud data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a target tracking method.

[0133] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0135] The robot acquires target point cloud data in real time using LiDAR and captures images of pedestrians in real time using visual sensors.

[0136] Using a pre-defined pedestrian detection model, pedestrian detection is performed on the current pedestrian image to obtain at least one first target box in the current pedestrian image;

[0137] Each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image, and a second target box matching each first target box is obtained from the at least one second target box, wherein the previous pedestrian image is an image acquired by the visual sensor before acquiring the current pedestrian image;

[0138] Based on each first target bounding box and target point cloud data in the current pedestrian image, the 3D detection position of each first target bounding box is determined. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image.

[0139] When the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked. If the first target box is successfully tracked, the 3D detection position of the first target box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target box.

[0140] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0141] The robot acquires multiple primary data points in real time from its location using LiDAR.

[0142] Perform coordinate transformation on each of the current multiple first data points to obtain the first target point corresponding to each first data point after coordinate transformation;

[0143] A line detection algorithm model is used to filter and detect lines from multiple first target points to obtain multiple second target points;

[0144] Clustering multiple second target points yields multiple first target classes;

[0145] Determine whether the number of points in each of the multiple first target classes is within a preset range, and determine the first target class whose number of points is within the preset range as the second target class;

[0146] The set of all points in the second target class is defined as the target point cloud data.

[0147] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0148] For each second target box in at least one second target box in the previous row of people image, predict the predicted target box corresponding to each second target box in at least one second target box in the previous row of people image;

[0149] Match each first target box in the current pedestrian image with the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and determine the predicted target box corresponding to each first target box in the current pedestrian image from the multiple predicted target boxes corresponding to at least one second target box in the previous pedestrian image;

[0150] Based on the predicted target box corresponding to each first target box in the current pedestrian image, a second target box that matches each first target box is obtained from at least one second target box.

[0151] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0152] Calculate the cross-union ratio (CUP) between each first target box in the current pedestrian image and the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and obtain multiple CUP values ​​for each first target box in the current pedestrian image;

[0153] The predicted target box corresponding to the smallest cross-union ratio among the multiple cross-union ratios of each first target box in the current pedestrian image is taken as the predicted target box corresponding to each first target box in the current pedestrian image.

[0154] In one embodiment, there are several frames of pedestrian images between the previous pedestrian image and the current target pedestrian image; correspondingly, when the processor executes the computer program, it also performs the following steps:

[0155] The previous pedestrian image, the current target pedestrian image, and several frames of pedestrian images are used as a set of matching images;

[0156] Match the target boxes in each pair of adjacent pedestrian images in a set of matching images to obtain the matching target boxes in each pair of adjacent pedestrian images in a set of matching images.

[0157] Based on the matching bounding boxes in each pair of adjacent pedestrian images in a set of matching images, determine at least one second bounding box that matches each first bounding box.

[0158] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0159] The earliest captured image among any two adjacent frames in a set of matching images is used as the first matching image, and the matching image in a set of matching images that is an adjacent frame to the first matching image and was captured at a later time than the first matching image is used as the second matching image.

[0160] The first algorithm model is used to predict the position of any target box in the first matching image in the second matching image, so as to obtain the predicted target box of any target box in the first matching image in the second matching image;

[0161] The second preset algorithm is used to match any target box in the second matching image with any target box in the first matching image and the predicted target box in the second matching image, so as to obtain the predicted target box in the first matching image that matches any target box in the second matching image;

[0162] Based on the correspondence between the predicted bounding box in the first matching image and the bounding box in the second matching image, the matching bounding boxes in the first and second matching images are obtained.

[0163] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0164] Project each point in the target point cloud data onto the current pedestrian image;

[0165] Based on the position of each first target box in the current pedestrian image and the position of each point in the target point cloud data projected onto the target pedestrian image, the target points in the target point cloud data belonging to each first target box in the current target pedestrian image are obtained;

[0166] Based on the 3D coordinates of the points in the target point cloud data that belong to each first target box in the current pedestrian image, the 3D detection position of each first target box in the current target pedestrian image is obtained.

[0167] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0168] The robot acquires target point cloud data in real time using LiDAR and captures images of pedestrians in real time using visual sensors.

[0169] Using a pre-defined pedestrian detection model, pedestrian detection is performed on the current pedestrian image to obtain at least one first target box in the current pedestrian image;

[0170] Each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image, and a second target box matching each first target box is obtained from the at least one second target box, wherein the previous pedestrian image is an image acquired by the visual sensor before acquiring the current pedestrian image;

[0171] Based on each first target bounding box and target point cloud data in the current pedestrian image, the 3D detection position of each first target bounding box is determined. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image.

[0172] When the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked. If the first target box is successfully tracked, the 3D detection position of the first target box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target box.

[0173] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0174] The robot acquires multiple primary data points in real time from its location using LiDAR.

[0175] Perform coordinate transformation on each of the current multiple first data points to obtain the first target point corresponding to each first data point after coordinate transformation;

[0176] A line detection algorithm model is used to filter and detect lines from multiple first target points to obtain multiple second target points;

[0177] Clustering multiple second target points yields multiple first target classes;

[0178] Determine whether the number of points in each of the multiple first target classes is within a preset range, and determine the first target class whose number of points is within the preset range as the second target class;

[0179] The set of all points in the second target class is defined as the target point cloud data.

[0180] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0181] For each second target box in at least one second target box in the previous row of people image, predict the predicted target box corresponding to each second target box in at least one second target box in the previous row of people image;

[0182] Match each first target box in the current pedestrian image with the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and determine the predicted target box corresponding to each first target box in the current pedestrian image from the multiple predicted target boxes corresponding to at least one second target box in the previous pedestrian image;

[0183] Based on the predicted target box corresponding to each first target box in the current pedestrian image, a second target box that matches each first target box is obtained from at least one second target box.

[0184] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0185] Calculate the cross-union ratio (CUP) between each first target box in the current pedestrian image and the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and obtain multiple CUP values ​​for each first target box in the current pedestrian image;

[0186] The predicted target box corresponding to the smallest cross-union ratio among the multiple cross-union ratios of each first target box in the current pedestrian image is taken as the predicted target box corresponding to each first target box in the current pedestrian image.

[0187] In one embodiment, there are several frames of pedestrian images between the previous pedestrian image and the current target pedestrian image; correspondingly, when the computer program is executed by the processor, it also performs the following steps:

[0188] The previous pedestrian image, the current target pedestrian image, and several frames of pedestrian images are used as a set of matching images;

[0189] Match the target boxes in each pair of adjacent pedestrian images in a set of matching images to obtain the matching target boxes in each pair of adjacent pedestrian images in a set of matching images.

[0190] Based on the matching bounding boxes in each pair of adjacent pedestrian images in a set of matching images, determine at least one second bounding box that matches each first bounding box.

[0191] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0192] The earliest captured image among any two adjacent frames in a set of matching images is used as the first matching image, and the matching image in a set of matching images that is an adjacent frame to the first matching image and was captured at a later time than the first matching image is used as the second matching image.

[0193] The first algorithm model is used to predict the position of any target box in the first matching image in the second matching image, so as to obtain the predicted target box of any target box in the first matching image in the second matching image;

[0194] The second preset algorithm is used to match any target box in the second matching image with any target box in the first matching image and the predicted target box in the second matching image, so as to obtain the predicted target box in the first matching image that matches any target box in the second matching image;

[0195] Based on the correspondence between the predicted bounding box in the first matching image and the bounding box in the second matching image, the matching bounding boxes in the first and second matching images are obtained.

[0196] In one embodiment, when the computer program is executed by a processor, it further performs the following steps:

[0197] Project each point in the target point cloud data onto the current pedestrian image;

[0198] Based on the position of each first target box in the current pedestrian image and the position of each point in the target point cloud data projected onto the target pedestrian image, the target points in the target point cloud data belonging to each first target box in the current target pedestrian image are obtained;

[0199] Based on the 3D coordinates of the points in the target point cloud data that belong to each first target box in the current pedestrian image, the 3D detection position of each first target box in the current target pedestrian image is obtained.

[0200] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0201] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above embodiments merely illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application's patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot, characterized in that, The robot is equipped with a lidar and a vision sensor. The robot includes a memory and a processor. The memory stores executable program code, and the processor executes the executable program code to perform the following steps: The laser radar acquires target point cloud data of the space where the robot is located in real time, and the vision sensor captures images of pedestrians in the space where the robot is located in real time. The current pedestrian image is subjected to pedestrian detection using a preset pedestrian detection model to obtain at least one first target box in the current pedestrian image; Each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image, and a second target box matching each first target box is obtained from the at least one second target box, wherein the previous pedestrian image is an image acquired by the visual sensor before acquiring the current pedestrian image; Based on each first target bounding box in the current pedestrian image and the target point cloud data, the three-dimensional detection position of each first target bounding box is determined. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image. When the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked. If the first target box is successfully tracked, the 3D detection position of the first target box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target box. If a second target box that matches the first target box cannot be obtained from at least one second target box, it indicates that the first target box has not been successfully tracked; or if the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the matching second target box does not meet a preset threshold, it indicates that the first target box has not been successfully tracked; if the first target box has not been successfully tracked, the 3D detection position of the first target box that has not been successfully tracked is directly output.

2. The robot according to claim 1, characterized in that, The step of acquiring target point cloud data of the robot's location in real time via the lidar includes: The laser radar acquires multiple current first data points of the space where the robot is located in real time. Perform coordinate transformation on each of the current plurality of first data to obtain the first target point corresponding to each of the current plurality of first data after coordinate transformation; A line detection algorithm model is used to perform line filtering detection on multiple first target points to obtain multiple second target points; Clustering the multiple second target points yields multiple first target classes; Determine whether the number of points in each of the plurality of first target classes is within a preset range, and determine the first target class whose number of points is within the preset range as the second target class; The set of all points in the second target class is defined as the target point cloud data.

3. The robot according to claim 1, characterized in that, The step of matching each first target box in the current pedestrian image with at least one second target box in the previous pedestrian image, and obtaining a second target box that matches each first target box from the at least one second target box, includes: For each second target box in at least one second target box in the previous row of people image, predict the predicted target box corresponding to each second target box in at least one second target box in the previous row of people image; Match each first target box in the current pedestrian image with the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and determine the predicted target box corresponding to each first target box in the current pedestrian image from the multiple predicted target boxes corresponding to at least one second target box in the previous pedestrian image; Based on the predicted target box corresponding to each first target box in the current pedestrian image, a second target box matching each first target box is obtained from the at least one second target box.

4. The robot according to claim 3, characterized in that, The step of matching each first target box in the current pedestrian image with the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and determining the predicted target box corresponding to each first target box in the current pedestrian image from the multiple predicted target boxes corresponding to at least one second target box in the previous pedestrian image, includes: Calculate the intersection-union ratio (IUU) between each first target box in the current pedestrian image and the predicted target box corresponding to each second target box in at least one second target box in the previous pedestrian image, and obtain multiple IUU values ​​for each first target box in the current pedestrian image; The predicted target box corresponding to the smallest cross-union ratio among the multiple cross-union ratios of each first target box in the current pedestrian image is taken as the predicted target box corresponding to each first target box in the current pedestrian image.

5. The robot according to claim 1, characterized in that, There are several frames of pedestrian images between the previous pedestrian image and the current pedestrian image; correspondingly, the step of matching each first target box in the current pedestrian image with at least one second target box in the previous pedestrian image, and obtaining a second target box matching each first target box from the at least one second target box, includes: The previous pedestrian image, the current pedestrian image, and the several frames of pedestrian images are used as a set of matching images; Match the target boxes in each pair of adjacent pedestrian images in the set of matching images to obtain the matching target boxes in each pair of adjacent pedestrian images in the set of matching images. Based on the matching target boxes in each pair of adjacent pedestrian images in the set of matching images, determine the second target box in the at least one second target box that matches each first target box.

6. The robot according to claim 5, characterized in that, The step of matching the target bounding boxes in every two adjacent frames of pedestrian images in the set of matching images to obtain the matching target bounding boxes in every two adjacent frames of pedestrian images in the set of matching images includes: The earliest captured image among any two adjacent frames in the set of matching images is taken as the first matching image, and the matching image in the set of matching images that is an adjacent frame to the first matching image and was captured at a later time than the first matching image is taken as the second matching image. The first algorithm model is used to predict the position of any target box in the first matching image in the second matching image, so as to obtain the predicted target box of any target box in the first matching image in the second matching image; The second preset algorithm is used to match any target box in the second matching image with any target box in the first matching image and the predicted target box in the second matching image, so as to obtain the predicted target box in the first matching image that matches any target box in the second matching image; Based on the correspondence between the predicted bounding box in the first matching image and the bounding box in the second matching image, the matching bounding boxes in the first and second matching images are obtained.

7. The robot according to claim 1, characterized in that, The step of determining the 3D detection position of each first target bounding box based on each first target bounding box in the current pedestrian image and the target point cloud data includes: Project each point in the target point cloud data onto the current pedestrian image; Based on the position of each first target box in the current pedestrian image and the position of each point in the target point cloud data projected onto the current pedestrian image, the target points in the target point cloud data belonging to each first target box in the current pedestrian image are obtained; Based on the three-dimensional coordinates of the points in the target point cloud data that belong to each first target box in the current pedestrian image, the three-dimensional detection position of each first target box in the current pedestrian image is obtained.

8. A target tracking method, characterized in that, The method includes: The robot acquires target point cloud data in real time using LiDAR and captures images of pedestrians in real time using visual sensors. The current pedestrian image is subjected to pedestrian detection using a preset pedestrian detection model to obtain at least one first target box in the current pedestrian image; Each first target box in the current pedestrian image is matched with at least one second target box in the previous pedestrian image, and a second target box matching each first target box is obtained from the at least one second target box, wherein the previous pedestrian image is an image acquired by the visual sensor before acquiring the current pedestrian image; Based on each first target bounding box in the current pedestrian image and the target point cloud data, the three-dimensional detection position of each first target bounding box is determined. The target point cloud data is point cloud data acquired synchronously with the acquisition time of the current pedestrian image. When the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the second target box that matches the first target box meets a preset threshold, it indicates that the first target box is successfully tracked. If the first target box is successfully tracked, the 3D detection position of the first target box is adjusted using a preset algorithm model to obtain the updated 3D position of the first target box. If a second target box that matches the first target box cannot be obtained from at least one second target box, it indicates that the first target box has not been successfully tracked; or if the distance between the 3D detection position of the first target box in the current pedestrian image and the 3D detection position of the matching second target box does not meet a preset threshold, it indicates that the first target box has not been successfully tracked; if the first target box has not been successfully tracked, the 3D detection position of the first target box that has not been successfully tracked is directly output.

9. The method according to claim 8, characterized in that, The step of acquiring target point cloud data of the robot's location in real time via the lidar includes: The laser radar acquires multiple current first data points of the space where the robot is located in real time. Perform coordinate transformation on each of the current plurality of first data to obtain the first target point corresponding to each of the current plurality of first data after coordinate transformation; A line detection algorithm model is used to perform line filtering detection on multiple first target points to obtain multiple second target points; Clustering the multiple second target points yields multiple first target classes; Determine whether the number of points in each of the plurality of first target classes is within a preset range, and determine the first target class whose number of points is within the preset range as the second target class; The set of all points in the second target class is defined as the target point cloud data.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 8 or 9.

Images