An unmanned vehicle guide person identification and tracking method based on a multi-modal sensor

Abstract

The present application relates to a kind of unmanned vehicle guide personnel identification tracking method based on multi-modal sensor, belong to unmanned vehicle technical field, solve the existing tracking robustness insufficient, target is easy to lose or mis-switching problem under the shielding scene.The method comprises: obtaining the continuous frame image and continuous frame laser radar point cloud of the timestamp alignment of unmanned vehicle;Multi-target detection and tracking are carried out to continuous frame image, and the image tracking ID of each personnel target and corresponding image detection frame in each frame image are obtained;Wherein, a personnel target is specified as guide personnel, and the corresponding image tracking ID is as lock ID;Extract personnel class clustering cluster in each frame laser radar point cloud;Personnel class clustering cluster is projected to image plane frame by frame, and is matched with image detection frame in image;Based on the three-dimensional state information of personnel class clustering cluster in the laser radar point cloud that image detection frame corresponding with lock ID is successfully matched, the position information of the guide personnel of unmanned vehicle is obtained.

Description

Technical Field

[0001] This invention relates to the field of unmanned vehicle technology, and in particular to a method for guiding and tracking people in unmanned vehicles based on multimodal sensors. Background Technology

[0002] Personnel following is a key function for autonomous vehicles to achieve autonomous collaborative operations. Its core requirement is that the autonomous vehicle can accurately lock onto and track a specific person in front of it in real time, and maintain a stable following state in complex and dynamic environments. Personnel identification and tracking technology aims to capture the position, three-dimensional dimensions, and motion state of a specific person in real time using sensors, enabling continuous tracking of the target in complex scenarios and providing reliable input for subsequent autonomous vehicle path planning, decision-making, and control.

[0003] Current methods for identifying and tracking guides primarily revolve around sensor selection and tracking algorithm design. In terms of sensor selection, mainstream methods use cameras and LiDAR as core sensors, achieving 3D target detection and tracking through single-sensor or multi-sensor fusion. Tracking methods are mainly divided into two categories: one is based on appearance feature matching, which uses the appearance features of a specific target from the previous frame to match the current frame data to find the target; the other is based on target motion characteristics, which constructs the target's motion paradigm, uses the target's motion trend to predict the target's position, and matches it with data-detected targets to obtain the tracked target. Appearance feature matching methods heavily rely on the integrity and stability of target point clouds or image features. In real-world scenarios such as streets and warehouses, guides are easily partially or completely obscured by dense crowds, roadside obstacles, or other moving objects, resulting in a significant loss of their feature information. Tracking methods based on target motion characteristics typically assume that the target follows a fixed or smooth motion model over a period of time. However, the guide's walking path, speed, and direction may change frequently and suddenly depending on the task at hand, causing motion extrapolation predictions based on historical trajectories to quickly diverge and fail. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide a method for guiding and tracking unmanned vehicles based on multimodal sensors, in order to solve the problems of insufficient tracking robustness, easy target loss or erroneous switching in existing methods under occluded scenarios.

[0005] The objective of this invention is mainly achieved through the following technical solutions:

[0006] On the one hand, the present invention provides a method for guiding and tracking personnel in unmanned vehicles based on multimodal sensors, comprising the following steps: Acquire continuous frame images and continuous frame LiDAR point clouds captured by the autonomous vehicle based on timestamp alignment; Multi-target detection and tracking are performed on consecutive frame images to obtain the image tracking ID of each person target and the corresponding image detection box in each frame image; wherein, a person target is designated as the guide in the initial frame image, and its corresponding image tracking ID is used as the lock ID. Based on the point cloud of each frame of LiDAR, extract the clusters of personnel categories in the environment; The clusters of personnel categories identified in the LiDAR point cloud are projected onto the image plane frame by frame and matched with the image detection boxes of each personnel target in the corresponding frame image; Once the image detection box corresponding to the locked ID begins to match the personnel category cluster of the LiDAR point cloud, the tracking initialization phase begins. When the image detection box corresponding to the locked ID successfully matches the personnel category cluster of the LiDAR point cloud for a preset number of consecutive times, the tracking phase begins. For each frame in the tracking phase, the three-dimensional state information of the personnel category clusters in the LiDAR point cloud is extracted, and the position information of the personnel guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters that are successfully matched with the image detection box corresponding to the locked ID.

[0007] Furthermore, the step of projecting the clusters of personnel categories identified in the lidar point cloud onto the image plane and matching them with the image detection boxes of each personnel target in the corresponding frame image includes: The personnel category clusters in the lidar point cloud of this frame are projected onto the two-dimensional space of the image to obtain the detection boxes of each personnel category cluster; The similarity measurement is performed between the cluster detection boxes of each of the aforementioned personnel categories and the image detection boxes of each personnel target in the corresponding frame image; wherein, the similarity measurement includes spatial overlap measurement and geometric similarity measurement; A match is considered successful when the spatial overlap metric is greater than a preset threshold and the geometric similarity metric is within a preset range; otherwise, a match is considered unsuccessful.

[0008] Furthermore, the spatial overlap metric is the intersection-union ratio of the personnel category cluster detection box and the image detection box; The geometric similarity metric is the aspect ratio of the personnel category cluster detection box and the image detection box.

[0009] Furthermore, the step of extracting personnel category clusters in the environment based on each frame of LiDAR point cloud includes: performing the following processing on each frame of LiDAR point cloud: The lidar point cloud in the current frame is preprocessed, and the preprocessed obstacle points are clustered to obtain multiple independent obstacle clusters; Feature extraction and classification are performed on each of the independent obstacle clusters to obtain several initial personnel category clusters; During the initialization of tracking, all initial personnel category clusters within a first preset spatial range relative to the current location of the unmanned vehicle are retained as the identified personnel category clusters; During the tracking phase, the initial personnel category cluster that is closest to the predicted location information of the autonomous vehicle guide within a second preset spatial range is retained as the identified personnel category cluster; wherein, the first preset spatial range is larger than the second preset spatial range.

[0010] Furthermore, the three-dimensional state information of the personnel category clusters is extracted, including: Extract the minimum bounding cube of the personnel category cluster point cloud in three-dimensional space; Based on the minimum circumscribed cuboid, the three-dimensional coordinates, length, width, and height of its center point are extracted as the three-dimensional state information of the personnel category cluster.

[0011] Furthermore, in the initial frame at the start of the tracking phase, an image occlusion set is initialized, and in each frame of the tracking phase, based on the image detection box corresponding to the lock ID, it is detected whether there is an occlusion; when there is an occlusion, the occlusion flag of the current frame is set to occlusion, and the image recognition IDs of all occluded image detection boxes are recorded.

[0012] Furthermore, during the tracking phase, when the personnel category cluster in the current frame's LiDAR point cloud fails to match the image detection box corresponding to the lock ID in the current frame image, the following operation is performed: Determine whether the image detection box corresponding to the lock ID exists in the current frame image; If an image detection box corresponding to the locked ID exists in the current frame image, determine whether there is a personnel category cluster in the current frame LiDAR point cloud; If there is no personnel category cluster in the current frame of the LiDAR point cloud, the LiDAR point cloud is re-clustered based on the image detection box to obtain the three-dimensional state information of the re-clustered personnel category cluster. If there is a cluster of people categories in the current frame of the lidar point cloud, determine whether the occlusion flag of the current frame is occlusion; When the occlusion flag in the current frame is occlusion, the location information of the unmanned vehicle guiding the personnel is obtained based on the three-dimensional state information of the personnel category clusters in the current frame's lidar point cloud. When the occlusion flag in the current frame is not occluded, image tracking ID rematching is performed based on the personnel category clusters of the current frame's lidar point cloud. If the image detection box corresponding to the lock ID does not exist in the current frame image, determine whether the occlusion flag of the previous frame is an occlusion; When the occlusion flag of the current frame is occlusion, the image recognition IDs of all image detection boxes that recorded occlusion in the previous frame are stored in the image occlusion set, and it is determined whether an image tracking ID that does not exist in the image occlusion set appears in the target tracking result of the current frame. When this occurs, image tracking ID rematching is performed based on the personnel category clusters in the current frame's LiDAR point cloud; When the person does not appear, the location information of the person being guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame of the lidar point cloud. When the occlusion marker in the current frame is not occluded, the location information of the personnel guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame's lidar point cloud.

[0013] Furthermore, the image tracking ID re-matching based on the personnel category clusters of the current frame's lidar point cloud includes: If a cluster of people categories is identified in K consecutive frames of the LiDAR point cloud, the cluster of people categories in the current frame of the LiDAR point cloud is rematched with the image detection box of the current frame image. When the image tracking ID of the rematched image detection box does not exist in the image occlusion set, the matched image tracking ID is set as the new locking ID, and the image occlusion set is cleared. Based on the three-dimensional state information of the personnel category cluster of the current frame LiDAR point cloud, the position information of the unmanned vehicle guiding the personnel is obtained. When the image tracking ID of the rematched image detection box exists in the image occlusion set, the locked ID remains unchanged. Based on the three-dimensional state information of the personnel category cluster of the current frame LiDAR point cloud, the position information of the unmanned vehicle guiding the personnel is obtained. If the matching fails again, the location information of the people guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame of the LiDAR point cloud. If no consecutive K frames identify a cluster of personnel categories in the LiDAR point cloud, the location information of the personnel guided by the autonomous vehicle can be obtained based on the three-dimensional state information of the personnel category clusters in the current frame of the LiDAR point cloud.

[0014] Furthermore, the step of detecting whether there is occlusion based on the image detection box corresponding to the locked ID includes: Based on the image detection box corresponding to the locked ID and all the image detection boxes in the current frame image, perform intersection-union calculation; When the intersection-union ratio is greater than the preset occlusion threshold, all image detection boxes that occlude the image detection box corresponding to the locked ID are obtained.

[0015] Furthermore, the re-clustering of LiDAR point clouds based on the image detection bounding box includes: Based on the calibration parameters of the camera and LiDAR, the image detection box is mapped to the LiDAR coordinate system, and a subset of the LiDAR point cloud in the image detection box is extracted. Clustering the lidar point cloud subset yields at least one candidate cluster; For each of the candidate clusters, feature extraction and classification are performed to obtain the personnel category clusters; The three-dimensional state information of the personnel category clusters is extracted and used as the result of re-clustering.

[0016] (Further connections are used in accordance with the claims) Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. This invention adopts a multimodal post-fusion architecture, which first performs independent target detection and tracking processing on the data from LiDAR and image sensors to ensure that each mode outputs valid detection and tracking results first. Then, the results from each sensor are accurately matched and fused through a cross-modal fusion algorithm. At the same time, the invention has a targeted fault-tolerant strategy for abnormal situations such as LiDAR detection failure and image tracking anomalies, so that the tracking system can still maintain continuous and stable target tracking under complex abnormal conditions.

[0017] 2. In the initial tracking phase, this invention searches for targets within a large spatial range centered on the unmanned vehicle to ensure that the guiding personnel can be captured. After completing the initialization and entering the stable tracking phase, the state estimator is used to predict the target position in the next frame and perform a high-precision search within a small range around the predicted position. This not only significantly improves the real-time performance of tracking but also filters out irrelevant and interfering targets in the distance, reducing the probability of mismatch and misfollowing in densely populated or cross-movement scenarios.

[0018] 3. When the present invention detects that a target is occluded, it records the other image tracking IDs that caused the occlusion to a cache list. The system does not immediately switch the tracking target, but continuously monitors the continuity of the LiDAR detection. Only when the LiDAR maintains stable detection for multiple consecutive frames and subsequently matches a new image tracking ID that does not exist in the cache list, does the system determine that the new ID is the correct target reproduction and safely update the tracking target. This ensures that target switching is only performed under extremely high confidence, effectively preventing the system from mistakenly tracking the previous occluder or other interfering targets.

[0019] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0021] Figure 1 This is a flowchart illustrating a method for guiding and tracking personnel in an unmanned vehicle based on multimodal sensors, as described in an embodiment of the present invention. Detailed Implementation

[0022] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0023] A specific embodiment of the present invention discloses a method for guiding and tracking personnel in unmanned vehicles based on multimodal sensors, such as... Figure 1 As shown, it includes the following steps S1-S5: Step S1: Acquire continuous frame images and continuous frame LiDAR point clouds captured by the unmanned vehicle based on timestamp alignment.

[0024] Specifically, the unmanned vehicle is equipped with a camera and a lidar. The camera is usually an RGB or grayscale camera, used to acquire rich texture and color information in front of the unmanned vehicle. The lidar is usually a 16-line, 32-line, or solid-state lidar, used to acquire high-precision three-dimensional environmental point clouds in front of the unmanned vehicle.

[0025] The camera and lidar are pre-calibrated in a spatiotemporal manner, including spatial calibration and temporal calibration. The spatial calibration is used to determine the relative position and attitude relationship between the camera and lidar, and to establish a precise transformation relationship between the two sensor coordinate systems. The temporal calibration is used to ensure that the image frames and lidar point cloud frames obtained at the same time describe the scene state of the physical world at the same time.

[0026] Step S2: Perform multi-target detection and tracking on consecutive frame images to obtain the image tracking ID of each person target and the corresponding image detection box in each frame image; wherein, in the initial frame image, a person target is designated as the guide person, and its corresponding image tracking ID is used as the lock ID.

[0027] Specifically, deep learning-based object detection models, such as the YOLO series or Faster R-CNN models, which have been trained on large-scale pedestrian datasets, are used to detect people in real time for each frame of the image. This yields two-dimensional bounding boxes and corresponding confidence scores for each person in the image. Subsequently, multi-object tracking algorithms, such as BoT-SORT, DeepSORT, and ByteTrack, are used as input to perform cross-frame data association, using the detection results of the consecutive frames as input, to achieve continuous tracking of the person target. For each frame, a tracking result list is output, where each element includes not only the detection box information but also a persistent image tracking ID. This ID remains unchanged throughout the entire visible period of the same target, and ideally, it can be recovered to the same ID even if it is briefly occluded during motion and then reappears.

[0028] More specifically, in the initial frame when image tracking begins, a specific person target is designated as the guide that the unmanned vehicle needs to follow through human interaction, and its corresponding image tracking ID is marked as the lock ID, which is the target that needs to be continuously tracked in the future.

[0029] Step S3: Based on each frame of LiDAR point cloud, extract personnel category clusters in the environment; this includes processing each frame of LiDAR point cloud using the following steps S31-S33: Step S31: Preprocess the lidar point cloud in the current frame and cluster the preprocessed obstacle points to obtain multiple independent obstacle clusters.

[0030] Specifically, ground segmentation is performed on the current frame's LiDAR point cloud to remove point clouds belonging to the ground, thereby removing ground interference and avoiding false detection of the ground as obstacles. The ground segmentation algorithm can use a ray projection-based ground segmentation algorithm (Ray Ground Filter) or the PatchWork++ algorithm.

[0031] Spatial clustering is performed on the segmented non-ground point cloud data, i.e. obstacle point cloud, to aggregate nearby point clouds in space to obtain several independent clusters. The spatial clustering method can be the Euclidean clustering algorithm (DBSCAN).

[0032] Step S32: Extract and classify features from each of the independent obstacle clusters to obtain several initial personnel category clusters.

[0033] Specifically, a set of multidimensional feature vectors is calculated for each independent obstacle cluster. These multidimensional feature vectors include the cluster's geometry (such as the size, volume, and surface area of ​​its three-dimensional bounding box), point cloud distribution (such as density distribution in the vertical direction, horizontal cross-sectional shape, and centroid height), and statistical properties (such as the number of points). The extracted multidimensional feature vectors are input into a pre-trained binary support vector machine (SVM) model. This SVM model outputs whether the cluster belongs to the "personnel category" or "other category" based on the position of the input multidimensional feature vectors in the feature space. It should be noted that the binary support vector machine (SVM) model is a pre-trained model using a large number of point cloud clustering samples labeled "pedestrian" and "non-pedestrian."

[0034] Step S33: Based on several initial personnel category clusters, obtain personnel category clusters in the environment.

[0035] In this embodiment, the identification and tracking of guide personnel is divided into a tracking initialization phase and a tracking phase. The tracking initialization phase is the process that begins after the image detection box corresponding to the locked ID starts matching with the personnel category cluster in the LiDAR point cloud, and continues until the image detection box corresponding to the locked ID and the personnel category cluster in the LiDAR point cloud have been successfully matched continuously for a preset number of times (exemplarily, 5 frames). In the tracking initialization phase, the main task of the system is to establish a one-to-one mapping relationship between the locked ID maintained by the image tracker and the target detected by the LiDAR. Therefore, during the tracking initialization process, a breadth-first search strategy is preferred. That is, in the LiDAR point cloud processing, all initial personnel category clusters within a first preset spatial range corresponding to the current position of the unmanned vehicle are retained as the identified personnel category clusters. The first preset spatial range can be a radius of 10 meters, which is used to ensure that the real guide personnel target can be included in the candidate set when the precise location of the target is unknown, and to avoid the inability to initialize successfully due to an excessively narrow search range.

[0036] The tracking phase is the process from the completion of initialization until the target is continuously lost and the system resets to the initial state. During this phase, the system's main tasks are to continuously output accurate 3D state information and effectively maintain continuous tracking of the guide personnel in the face of occlusion, sensor failure, and other problems. Therefore, a depth-first spatial search strategy is preferred in the tracking phase. Specifically, in the LiDAR point cloud processing, the initial personnel category cluster that is closest to the predicted location information of the autonomous vehicle guide personnel within a second preset spatial range is retained as the identified personnel category cluster. This is based on the predicted location information of the guide personnel in the next frame fed back by the state estimator during the tracking phase. It retains all initial personnel category clusters whose center point is within the second preset spatial range of the predicted location information, and further selects the initial personnel category cluster closest to the predicted location from these clusters.

[0037] Wherein, the first preset space range is larger than the second preset space range. For example, the second preset space range can be a range with a radius of 3 meters.

[0038] It should be noted that there is no strict execution order between steps S2 and S3. Those skilled in the art should understand that the processing of the above two branches is logically and temporally independent, and their execution order should not constitute a limitation on the scope of protection of this invention.

[0039] Step S4: Project the clusters of personnel categories identified in the LiDAR point cloud frame by frame onto the image plane and match them with the image detection boxes of each personnel target in the corresponding frame image.

[0040] Specifically, due to inherent defects in single-modal tracking methods, image sensors, while capable of extracting rich semantic features, lack three-dimensional spatial information and cannot output accurate target positions. Although lidar can provide three-dimensional coordinates, its point cloud scanning is sparse, making it difficult to extract stable target features in single-modal mode, leading to errors in target matching between adjacent frames. In this embodiment, the personnel category clusters identified in the lidar point cloud are matched frame by frame with the image detection boxes of each personnel target in the corresponding frame image. Lidar compensates for the inaccurate ranging problem of image sensors, and image sensors compensate for the "feature sparsity" defect of lidar. After the two are fused, accurate output of the target's three-dimensional position can be achieved.

[0041] Furthermore, the step of projecting the clusters of personnel categories identified in the lidar point cloud onto the image plane and matching them with the image detection boxes of each personnel target in the corresponding frame image includes steps S41-S43: Step S41: Project the personnel category clusters in the lidar point cloud of the frame onto the two-dimensional space of the image to obtain the detection boxes of each personnel category cluster.

[0042] Specifically, the coordinates of the eight vertices of the smallest bounding cube of the personnel category cluster identified in step S3 in the current frame are transformed from the lidar coordinate system to the camera coordinate system using a pre-calibrated camera-liDAR extrinsic parameter matrix. Then, the coordinates of these eight vertices in the camera coordinate system are projected onto the image pixel plane using the camera's intrinsic parameter matrix, resulting in eight two-dimensional pixels. The minimum and maximum values ​​of these eight projected points in the X and Y directions of the image pixel coordinate system are taken as the personnel category cluster detection box in the two-dimensional image. .

[0043] Understandably, mapping the target in three-dimensional space to the same two-dimensional pixel coordinate system as the image detection box allows for direct geometric comparison between the two.

[0044] Step S42: Perform a similarity measurement on the cluster detection boxes of each person category and the image detection boxes of each person target in the corresponding frame image; wherein, the similarity measurement includes spatial overlap measurement and geometric similarity measurement.

[0045] Specifically, the similarity between the cluster detection boxes of each person category obtained in step S41 and the image detection boxes of each person target in the frame image obtained in step S2 is measured.

[0046] Furthermore, the spatial overlap metric is the intersection-union ratio (IUGR) of the personnel category cluster detection box and the image detection box, which measures the degree of overlap between the two boxes in spatial location. The higher the IUGR, the more consistent the coverage area of ​​the target detected by the LiDAR and the target detected by the image is on the two-dimensional image.

[0047] The geometric similarity metric is the aspect ratio of the personnel category cluster detection box and the image detection box. The aspect ratio of the personnel category cluster detection box is divided by the aspect ratio of the image detection box to obtain a ratio value. It is then determined whether the ratio value is within a preset range, for example, 0.6 to 1.5. If it is within the preset range, the two detection boxes are determined to be geometrically similar; otherwise, the two detection boxes are determined to be geometrically dissimilar.

[0048] Understandably, by measuring the similarity in shape between two boxes, even if they are in close proximity, the aspect ratio of the projected box and the detection box of a tall, thin person and a short, wide car will be significantly different, thus effectively filtering out false matches.

[0049] Step S43: When the spatial overlap metric is greater than a preset threshold and the geometric similarity metric is within a preset range, the match is successful; otherwise, the match fails.

[0050] Specifically, a match is considered successful when the metrics of the two matching bounding boxes simultaneously meet their respective conditions. It should be noted that the preset threshold for the spatial overlap metric can be set to 0.5.

[0051] Understandably, a single metric (such as spatial overlap) is prone to misjudgment due to sensor noise, calibration errors, and partial occlusion of the target. Spatial overlap ensures that the positions are basically aligned, while geometric similarity ensures that the physical size ratio of the detection box is reasonable. Combining the two can form a more robust matching judgment.

[0052] Step S5: After the image detection box corresponding to the locked ID begins to match the personnel category cluster of the LiDAR point cloud, the tracking initialization stage begins.

[0053] Step S6: When the image detection box corresponding to the locked ID successfully matches the personnel category cluster of the LiDAR point cloud for a preset number of consecutive times, the tracking stage begins.

[0054] Specifically, when the detection box corresponding to the lock ID and the LiDAR projection box successfully match for the first time in a certain frame, the system enters the tracking initialization phase and begins to continuously track the matching pair. In each subsequent frame, as long as the detection box corresponding to the lock ID continues to successfully match the personnel category cluster of the LiDAR, the counter is incremented by 1. If it fails, the counter is reset to zero. When the number of consecutive matching frames is greater than the preset number of frames, for example, the preset number of frames is 5 frames, it is considered that the LiDAR can stably observe the guiding personnel in the detection box corresponding to the lock ID. At this time, the initialization is completed and the system enters the stable tracking phase.

[0055] Understandably, successful continuous matching requires that the same image ID must be continuously associated with the LiDAR target whose motion trajectory is coherent in three-dimensional space. This reduces the probability of matching an incorrect three-dimensional target due to accidental errors in single-frame matching. After initialization, continuous and reliable position observations of the tracked target in three-dimensional space are obtained, providing a high-quality historical observation sequence for the subsequent motion state estimator.

[0056] Meanwhile, after the tracking initialization is completed, the range for identifying personnel category clusters in the lidar point cloud of subsequent frames is also changed from the first preset spatial range with the unmanned vehicle as the origin to the second preset spatial range with the predicted position of the unmanned vehicle guiding the personnel.

[0057] Step S7: For each frame in the tracking phase, extract the three-dimensional state information of the personnel category cluster in the lidar point cloud, and obtain the position information of the personnel guided by the unmanned vehicle based on the three-dimensional state information of the personnel category cluster that is successfully matched with the image detection box corresponding to the locked ID.

[0058] Specifically, during the stable tracking phase, for each frame of data, the three-dimensional state information of the personnel category clusters in the lidar point cloud is extracted, including: Extract the minimum bounding cuboid of the personnel category cluster point cloud in three-dimensional space.

[0059] Specifically, the personnel category cluster point cloud consists of a set of points including three-dimensional coordinates. By traversing the three-dimensional coordinates of each point in the cluster, the minimum and maximum values ​​on the X, Y, and Z coordinate axes are found respectively. These six values ​​directly define the minimum bounding box of the personnel category cluster point cloud.

[0060] Based on the minimum circumscribed cuboid, the three-dimensional coordinates, length, width, and height of its center point are extracted as the three-dimensional state information of the personnel category cluster.

[0061] Specifically, based on the coordinates of the eight vertices of the minimum circumscribed cuboid, the three-dimensional coordinates of its center point are obtained using the following formula. : ; in, and These represent the minimum and maximum values ​​of the clusters on the X-axis of the lidar coordinate system, respectively. and These represent the minimum and maximum values ​​of the clusters on the Y-axis of the lidar coordinate system, respectively. and These represent the minimum and maximum values ​​of the clusters on the Z-axis of the lidar coordinate system, respectively.

[0062] The length l, width w, and height h of the personnel category clusters are obtained using the following formula: .

[0063] More specifically, the three-dimensional state information of the LiDAR personnel who successfully matched the locked ID is used as the observation value input to a pre-constructed motion state estimator, such as a Kalman filter. The motion state estimator performs a prediction-update loop according to a preset motion model, such as a uniform motion model. This includes: using the observation value of the current frame to correct the prediction result of the previous frame to obtain the smoothed three-dimensional position and velocity information of the guide personnel in the current frame, and based on the corrected three-dimensional position and velocity information of the current frame, obtaining the predicted three-dimensional position of the guide personnel in the next frame.

[0064] Furthermore, in the initial frame at the start of the tracking phase, an image occlusion set is initialized. This set is used in subsequent tracking phases to dynamically record other image tracking IDs that overlap with the locked ID when occlusion occurs. It also provides historical interference information for target re-identification and secure switching of tracking identity after occlusion ends. In each frame of the tracking phase, based on the image detection box corresponding to the locked ID, it is detected whether there is occlusion; when there is occlusion, the occlusion flag of the current frame is set to occlusion, and the image recognition IDs of all occluded image detection boxes are recorded.

[0065] Furthermore, the step of detecting whether there is occlusion based on the image detection box corresponding to the locked ID includes: Iterate through all other image detection boxes in the current frame image except for the locked ID, and calculate the intersection-over-union (IoU) ratio of each IoU with the image detection box of the locked ID. If the IoU ratio of any other IoU with the image detection box of the locked ID exceeds a preset occlusion threshold, then the current frame is determined to be occluded. If the IoU ratio of any other IoU with the image detection box of the locked ID does not exceed the preset occlusion threshold, then the current frame is determined not to be occluded. For example, the preset occlusion threshold can be 0.3.

[0066] It should be noted that when there is no occlusion and no image detection box corresponding to the lock ID is detected in the current frame, the occlusion flag of the current frame is not set.

[0067] Furthermore, in complex dynamic scenarios during the tracking phase, the guide may be obstructed by obstacles or other pedestrians, the LiDAR may fail to detect due to sparse point clouds or environmental interference, and the image tracker may lose image tracking due to strong light, backlight, or motion blur. Based on the above reasons, the matching between the lock ID in the current frame and the personnel category cluster in the LiDAR point cloud fails. At this time, it is determined whether there is an image detection box corresponding to the lock ID in the current frame image. If there is an image detection box corresponding to the lock ID, it indicates that the image tracker has still successfully locked the specified guide, and step S71 is executed. If there is no image detection box corresponding to the lock ID, it indicates that the image tracker has lost the specified guide, and the image detection box of the lock ID cannot be obtained from the image, and step S72 is executed.

[0068] Step S71: If the image detection box corresponding to the locked ID exists in the current frame image, determine whether there is a personnel category cluster in the current frame lidar point cloud.

[0069] Specifically, if it is confirmed that there is a locked ID detection box in the current frame image, the detection status of the LiDAR is diagnosed first, that is, to determine whether there is a valid personnel category cluster in the LiDAR point cloud of the current frame.

[0070] If there is no personnel category cluster in the current frame of the LiDAR point cloud, the LiDAR point cloud is re-clustered based on the image detection box to obtain the three-dimensional state information of the re-clustered personnel category cluster.

[0071] Specifically, if no personnel category cluster is found in the current frame's LiDAR point cloud, it is determined that the LiDAR has temporarily missed detection or failed. Utilizing the reliable prior information that the image tracker has still locked onto the target, local re-clustering of the LiDAR based on image frame guidance is performed. This guides the LiDAR to re-cluster within a local 3D space that may contain the target, including the following steps S701-S704: Step S701: Based on the calibration parameters of the camera and LiDAR, map the image detection box to the LiDAR coordinate system, and extract the LiDAR point cloud subset in the image detection box.

[0072] Specifically, the four corner points of the image detection box with the locked ID in the current frame are back-projected into the camera coordinate system using the inverse of the camera intrinsic parameter matrix, resulting in four three-dimensional rays originating from the camera optical center and pointing towards the corner points of the image box. These four rays form an infinitely extending quadrangular pyramid in three-dimensional space, i.e., the view frustum. Using the inverse of the calibrated camera-LiDAR extrinsic parameter matrix, the view frustum is transformed from the camera coordinate system to the LiDAR coordinate system, obtaining the corresponding search space in the LiDAR coordinate system. Each three-dimensional point in the current frame's LiDAR point cloud is traversed, and points within the view frustum in the LiDAR coordinate system are extracted to form a subset of the LiDAR point cloud.

[0073] Step S702: Cluster the lidar point cloud subset to obtain at least one candidate cluster.

[0074] Step S703: Perform feature extraction and classification on each of the candidate clusters to obtain the personnel category clusters; Step S704: Extract the three-dimensional state information of the personnel category clusters as the result of re-clustering.

[0075] Specifically, for the lidar point cloud subset obtained in step S701, the point cloud processing steps as in steps S32-S33 are used to sequentially perform point cloud clustering, feature extraction and classification, and three-dimensional state information extraction steps to obtain the personnel category clusters and their corresponding three-dimensional state information in the lidar point cloud subset, which are used as the result of re-clustering of lidar detection in the current frame.

[0076] More specifically, if the personnel category clusters and their corresponding three-dimensional state information can be successfully extracted, the three-dimensional state information is used as the observation value, and the prediction result of the previous frame is corrected by the motion state estimator to obtain the predicted three-dimensional position of the guiding personnel in the next frame.

[0077] If the personnel category cluster and its corresponding 3D state information cannot be successfully extracted, the current frame will not provide 3D observation data to the motion state estimator and will wait for the next frame of laser point cloud data input.

[0078] If there is a cluster of people categories in the current frame of the LiDAR point cloud, determine whether the occlusion flag of the current frame is occlusion.

[0079] Specifically, if there is a cluster of people categories in the current frame of the LiDAR point cloud, but it fails to match the image detection box with the locked ID, it indicates that the cross-modal association matching has failed. At this time, the LiDAR has detected the target or other pedestrians, but failed to correctly associate them with the image detection box of the locked ID. It is then necessary to determine whether there is an image detection box in the current frame that occludes the image detection box of the locked ID.

[0080] When the occlusion flag in the current frame is occlusion, the location information of the personnel guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame's lidar point cloud.

[0081] Specifically, in occlusion scenarios, the LiDAR will inevitably lock onto the closer occluder, while the image will still lock onto the occluded guide, thus causing the matching to fail in this frame. However, since both the image tracker and the LiDAR can process normally, it indicates that the occlusion is only temporary. At this time, it is only necessary to directly use the 3D position information provided by the LiDAR as an approximate estimate of the guide's current position to maintain the continuity of tracking during the occlusion period and avoid following the wrong target due to accidental switching.

[0082] When the occlusion flag in the current frame is not occluded, image tracking ID rematching is performed based on the personnel category clusters of the current frame's LiDAR point cloud.

[0083] Specifically, when the image detection box corresponding to the locked ID is not occluded by other image detection boxes, the LiDAR detects the target and there are no interfering image detection boxes in the image, but they are not correctly associated. Therefore, it is necessary to actively perform rematching to repair the incorrect matching state by re-establishing the association between the LiDAR projection box and the image detection box.

[0084] Furthermore, the image tracking ID rematching based on the personnel category clusters of the current frame's LiDAR point cloud includes the following steps S711-S715: Step S711: If the LiDAR point cloud identifies a cluster of personnel categories in K consecutive frames, then rematch the personnel category clusters in the current frame LiDAR point cloud with the image detection boxes in the current frame image.

[0085] Specifically, if the lidar point cloud identifies a cluster of people categories for K consecutive frames, it is considered that the lidar has established a stable and continuous observation of the targets in the current field of view, which is reliable information. At this time, the cluster of people categories based on the lidar point cloud of the current frame is matched again with the image detection box.

[0086] Step S712: When the image tracking ID of the rematched image detection box does not exist in the image occlusion set, the matched image tracking ID is set as the new locking ID, and the image occlusion set is cleared. Based on the three-dimensional state information of the personnel category cluster of the current frame lidar point cloud, the position information of the unmanned vehicle guiding the personnel is obtained.

[0087] Specifically, if the image tracking ID that is successfully matched does not exist in the image occlusion set, it means that the target matched at this time is an image tracking ID that has not been recorded as occlusion, and it is likely to be the target of the lock ID. At this time, the lock ID is reset using the matched image tracking ID, and the image occlusion set is cleared.

[0088] Step S713: When the image tracking ID of the rematched image detection box exists in the image occlusion set, the locking ID remains unchanged. Based on the three-dimensional state information of the personnel category cluster of the current frame lidar point cloud, the position information of the unmanned vehicle guiding the personnel is obtained.

[0089] Specifically, if the successfully matched image tracking ID exists in the image occlusion set, it means that this person was in front of / next to the guide during the occlusion period, blocking the designated guide. At this time, the matched image tracking ID is highly likely to be the occluder, not the guide himself. Therefore, the locked ID is kept unchanged, and the three-dimensional state information of the current frame is output to the motion state estimator.

[0090] Step S714: If matching fails again, obtain the location information of the unmanned vehicle guiding the personnel based on the three-dimensional state information of the personnel category clusters in the current frame's LiDAR point cloud.

[0091] Specifically, if a suitable image detection box is still not found for the current frame's personnel category cluster after the rematch operation, the reason may be that the target pose changes drastically, causing the projection box to be distorted, the image tracker misses a detection briefly, or there is a small calibration error. At this time, there is no image ID available for switching, so the current locked ID state remains unchanged, and only the three-dimensional state information of the current frame is output to the motion state estimator to maintain tracking continuity.

[0092] Step S715: If the LiDAR point cloud does not identify a cluster of personnel categories in K consecutive frames, obtain the location information of the personnel guided by the unmanned vehicle based on the three-dimensional state information of the personnel category clusters in the current frame of the LiDAR point cloud.

[0093] Specifically, if the lidar point cloud does not identify a cluster of personnel categories in K consecutive frames, it means that the lidar observation state is not very stable. In this case, only the three-dimensional state information of the current frame is input to the motion state estimator, without performing ID locking processing.

[0094] It should be noted that the K-frame in this embodiment can be set to 10 frames.

[0095] Step S72: If the image detection box corresponding to the lock ID does not exist in the current frame image, determine whether the occlusion flag of the previous frame is an occlusion.

[0096] Specifically, if the image detection box corresponding to the locked ID does not exist in the current frame image, that is, the image tracker has lost the guide personnel, then there is no image tracking object in this frame image. It is necessary to determine whether there is an occluder in the previous frame image. At the same time, if there is a valid personnel category cluster in the current frame LiDAR point cloud, it indicates that the LiDAR has still locked a target.

[0097] When the occlusion flag of the current frame is occluded, the image recognition IDs of all image detection boxes that recorded occlusion in the previous frame are stored in the image occlusion set, and it is determined whether an image tracking ID that does not exist in the image occlusion set appears in the target tracking result of the current frame.

[0098] When an image tracking ID rematch occurs, it is performed based on the personnel category clusters in the current frame's LiDAR point cloud.

[0099] Specifically, when there is occlusion in the previous frame, that is, the tracked guide is lost after being occluded, the source of interference is most likely the occluder in the previous frame. At this time, if the target tracking result of the current frame image shows an image tracking ID that does not exist in the image occlusion set, it is considered that this image tracking ID is a new target that has not been identified as the occluder. It may be a real guide reappearance or an irrelevant person newly entering the scene. At this time, the rematching steps as in steps S711-S715 are performed to re-identify the guide.

[0100] When the person is not present, the location information of the person being guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame of the lidar point cloud.

[0101] Specifically, when all image tracking IDs in the current image detection frame are in the image occlusion set, it can be assumed that the real guide has not yet been reproduced. In this case, only the three-dimensional state information of the current frame is input to the motion state estimator.

[0102] When the occlusion marker in the current frame is not occluded, the location information of the personnel guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame's lidar point cloud.

[0103] Specifically, in scenarios where there is no image loss due to occlusion in the current frame, the tracking of the guide is maintained using only the lidar point cloud information of the current frame.

[0104] It should be noted that if the image detection box corresponding to the locked ID does not exist in the current frame image, and there is no valid personnel category cluster in the current frame LiDAR point cloud, the motion state estimator only performs prediction updates and does not perform observation corrections. Simultaneously, the counter for invalid LiDAR observation frames is accumulated. If the LiDAR detection information fails to input valid 3D state information to the motion state estimator for L consecutive frames, it is determined that the currently tracked target is lost. The entire process is then reset to the initial tracking initialization process, and step S4 is executed to restart the guided personnel tracking and identification process. In this embodiment, L frames can be set to 10 frames.

[0105] In summary, the unmanned vehicle guidance and personnel identification and tracking method based on multimodal sensors according to embodiments of the present invention has the following beneficial effects: 1. The embodiments of the present invention adopt a multimodal post-fusion architecture, firstly performing independent target detection and tracking processing on the LiDAR and image sensor data respectively, ensuring that each mode outputs valid detection and tracking results first, and then accurately matching and fusing the results of each sensor through a cross-modal fusion algorithm. At the same time, targeted fault-tolerant strategies for abnormal situations such as LiDAR detection failure and image tracking anomalies enable the tracking system to maintain continuous and stable target tracking even under complex abnormal working conditions.

[0106] 2. In the tracking initialization phase of this invention, the target is searched in a large spatial range with the unmanned vehicle as the center to ensure that the guide personnel can be captured. After the initialization is completed and the stable tracking phase is entered, the state estimator is used to predict the target position of the next frame and perform a high-precision search in a small range around the predicted position. This not only significantly improves the real-time performance of tracking, but also filters out irrelevant interfering targets in the distance, reducing the probability of mismatch and misfollowing in dense crowds or cross-movement scenarios.

[0107] 3. In this embodiment of the invention, when a target is detected to be occluded, the tracking IDs of other images that caused the occlusion are recorded in the cache list. The system does not immediately switch the tracking target, but continuously monitors the continuity of the LiDAR detection. Only when the LiDAR maintains stable detection for multiple consecutive frames, and subsequently matches a new image tracking ID that does not exist in the cache list, does the system determine that the new ID is the correct target reproduction and safely update the tracking target. This ensures that target switching is only performed under extremely high confidence, effectively preventing the system from mistakenly tracking the previous occluder or other interfering targets.

[0108] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0109] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for guiding and tracking people in unmanned vehicles based on multimodal sensors, characterized in that, Includes the following steps: Acquire continuous frame images and continuous frame LiDAR point clouds of the unmanned vehicle based on timestamp alignment; Multi-target detection and tracking are performed on consecutive frame images to obtain the image tracking ID of each person target and the corresponding image detection box in each frame image; wherein, a person target is designated as the guide in the initial frame image, and its corresponding image tracking ID is used as the lock ID. Based on the point cloud of each frame of LiDAR, extract the clusters of personnel categories in the environment; The clusters of personnel categories identified in the LiDAR point cloud are projected onto the image plane frame by frame and matched with the image detection boxes of each personnel target in the corresponding frame image; Once the image detection box corresponding to the locked ID begins to match the personnel category cluster of the LiDAR point cloud, the tracking initialization phase begins. When the image detection box corresponding to the locked ID successfully matches the personnel category cluster of the LiDAR point cloud for a preset number of consecutive times, the tracking phase begins. For each frame in the tracking phase, the three-dimensional state information of the personnel category clusters in the LiDAR point cloud is extracted, and the position information of the personnel guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters that are successfully matched with the image detection box corresponding to the locked ID.

2. The method according to claim 1, characterized in that, The step of projecting the clusters of personnel categories identified in the lidar point cloud onto the image plane and matching them with the image detection boxes of each personnel target in the corresponding frame image includes: The personnel category clusters in the lidar point cloud of this frame are projected onto the two-dimensional space of the image to obtain the detection boxes of each personnel category cluster; The similarity measurement is performed between the cluster detection boxes of each of the aforementioned personnel categories and the image detection boxes of each personnel target in the corresponding frame image; wherein, the similarity measurement includes spatial overlap measurement and geometric similarity measurement; A match is considered successful when the spatial overlap metric is greater than a preset threshold and the geometric similarity metric is within a preset range; otherwise, a match is considered unsuccessful.

3. The method according to claim 2, characterized in that, The spatial overlap metric is the intersection-union ratio of the personnel category cluster detection boxes and the image detection boxes; The geometric similarity metric is the aspect ratio of the personnel category cluster detection box and the image detection box.

4. The method according to claim 1, characterized in that, The step of extracting personnel category clusters in the environment based on each frame of LiDAR point cloud includes: performing the following processing on each frame of LiDAR point cloud: The lidar point cloud in the current frame is preprocessed, and the preprocessed obstacle points are clustered to obtain multiple independent obstacle clusters; Feature extraction and classification are performed on each of the independent obstacle clusters to obtain several initial personnel category clusters; During the initialization of tracking, all initial personnel category clusters within a first preset spatial range relative to the current location of the unmanned vehicle are retained as the identified personnel category clusters; During the tracking phase, the initial personnel category cluster that is closest to the predicted location information of the autonomous vehicle guide within a second preset spatial range is retained as the identified personnel category cluster; wherein, the first preset spatial range is larger than the second preset spatial range.

5. The method according to claim 4, characterized in that, Extracting the three-dimensional state information of the personnel category clusters includes: Extract the minimum bounding cube of the personnel category cluster point cloud in three-dimensional space; Based on the minimum circumscribed cuboid, the three-dimensional coordinates, length, width, and height of its center point are extracted as the three-dimensional state information of the personnel category cluster.

6. The method according to claim 1, characterized in that, At the beginning of the tracking phase, the image occlusion set is initialized in the initial frame. In each frame of the tracking phase, based on the image detection box corresponding to the lock ID, it is detected whether there is an occlusion. When there is an occlusion, the occlusion flag of the current frame is set to occlusion, and the image recognition IDs of all occluded image detection boxes are recorded.

7. The method according to claim 6, characterized in that, During the tracking phase, if the personnel category cluster in the current frame LiDAR point cloud fails to match the image detection box corresponding to the lock ID in the current frame image, the following operation is performed: Determine whether the image detection box corresponding to the lock ID exists in the current frame image; If an image detection box corresponding to the locked ID exists in the current frame image, determine whether there is a personnel category cluster in the current frame LiDAR point cloud; If there is no personnel category cluster in the current frame of the LiDAR point cloud, the LiDAR point cloud is re-clustered based on the image detection box to obtain the three-dimensional state information of the re-clustered personnel category cluster. If there is a cluster of people categories in the current frame of the lidar point cloud, determine whether the occlusion flag of the current frame is occlusion; When the occlusion flag in the current frame is occlusion, the location information of the unmanned vehicle guiding the personnel is obtained based on the three-dimensional state information of the personnel category clusters in the current frame's lidar point cloud. When the occlusion flag in the current frame is not occluded, image tracking ID rematching is performed based on the personnel category clusters of the current frame's lidar point cloud. If the image detection box corresponding to the lock ID does not exist in the current frame image, determine whether the occlusion flag of the previous frame is an occlusion; When the occlusion flag of the current frame is occlusion, the image recognition IDs of all image detection boxes that recorded occlusion in the previous frame are stored in the image occlusion set, and it is determined whether an image tracking ID that does not exist in the image occlusion set appears in the target tracking result of the current frame. When this occurs, image tracking ID rematching is performed based on the personnel category clusters in the current frame's LiDAR point cloud; When the person does not appear, the location information of the person being guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame of the lidar point cloud. When the occlusion marker in the current frame is not occluded, the location information of the personnel guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame's lidar point cloud.

8. The method according to claim 7, characterized in that, The image tracking ID rematching based on the personnel category clusters of the current frame's LiDAR point cloud includes: If a cluster of people categories is identified in K consecutive frames of the LiDAR point cloud, the cluster of people categories in the current frame of the LiDAR point cloud is rematched with the image detection box of the current frame image. When the image tracking ID of the rematched image detection box does not exist in the image occlusion set, the matched image tracking ID is set as the new locking ID, and the image occlusion set is cleared. Based on the three-dimensional state information of the personnel category cluster of the current frame LiDAR point cloud, the position information of the unmanned vehicle guiding the personnel is obtained. When the image tracking ID of the rematched image detection box exists in the image occlusion set, the locked ID remains unchanged. Based on the three-dimensional state information of the personnel category cluster of the current frame LiDAR point cloud, the position information of the unmanned vehicle guiding the personnel is obtained. If the matching fails again, the location information of the people guided by the unmanned vehicle is obtained based on the three-dimensional state information of the personnel category clusters in the current frame of the LiDAR point cloud. If no consecutive K frames identify a cluster of personnel categories in the LiDAR point cloud, the location information of the personnel guided by the autonomous vehicle can be obtained based on the three-dimensional state information of the personnel category clusters in the current frame of the LiDAR point cloud.

9. The method according to any one of claims 6-8, characterized in that, The step of detecting whether there is occlusion based on the image detection box corresponding to the locked ID includes: Based on the image detection box corresponding to the locked ID and all the image detection boxes in the current frame image, perform intersection-union calculation; When the intersection-union ratio is greater than the preset occlusion threshold, all image detection boxes that occlude the image detection box corresponding to the locked ID are obtained.

10. The method according to claim 9, characterized in that, The re-clustering of LiDAR point clouds based on the image detection bounding box includes: Based on the calibration parameters of the camera and LiDAR, the image detection box is mapped to the LiDAR coordinate system, and a subset of the LiDAR point cloud in the image detection box is extracted. Clustering the lidar point cloud subset yields at least one candidate cluster; For each of the candidate clusters, feature extraction and classification are performed to obtain the personnel category clusters; The three-dimensional state information of the personnel category clusters is extracted and used as the result of re-clustering.

Images