Automatic marking, detection model training and target recognition method, and electronic device

Abstract

Embodiments of the present application provide an automatic marking method, a detection model training method, a target identification method and an electronic device, wherein the automatic marking method comprises: determining a tracking target that is not tracked and matched according to a detection result of target detection on adjacent frame 3D point cloud data; setting a tracking task for the tracking target that is not tracked and matched and continuously tracking until there is no 3D point cloud data of the tracking target in a detection box corresponding to the tracking target or the tracking target exceeds a preset range according to the 3D point cloud data; and marking data for the tracking target that is not tracked and matched according to a tracking result. Through the embodiments of the present application, the data marking can be more accurate, the marking quality is also higher, and the detection model training effect is better. When applied to the target identification process, a more accurate target identification result can be obtained.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an automatic labeling method, a detection model training method, a target recognition method, and corresponding electronic devices. Background Technology

[0002] With the development of LiDAR technology, it has been widely used in various scenarios such as autonomous driving, vehicle-to-everything (V2X) communication, and cleaning robots. In these scenarios, by performing target detection on the 3D point cloud data collected by LiDAR, information such as the type, scale, and distance of obstacles around the device or vehicle can be perceived in real time, thus providing effective support for decision-making, planning, and control in autonomous driving, V2X communication, and cleaning robots. This target detection is mostly achieved through detection models; therefore, target detection models based on 3D point cloud data can be considered the cornerstone of LiDAR's effectiveness in various scenarios.

[0003] For object detection models, training on 3D point cloud data is necessary. This training data includes not only the 3D point cloud data itself but also labeled 3D point cloud data. The more labeled 3D point cloud data there is, the better the training performance and the higher the robustness of the object detection model. Therefore, the quality of the 3D point cloud data labeling directly affects the training effect of the object detection model.

[0004] However, due to the complexity and characteristics of 3D point cloud data itself, as well as the differences in LiDAR usage scenarios, the marking quality of 3D point cloud data is poor and the marking information is inaccurate.

[0005] Therefore, how to efficiently and accurately label 3D point cloud data has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of this application provide an automatic labeling and detection model training scheme to at least partially solve the above problems.

[0007] According to a first aspect of the embodiments of this application, an automatic labeling method is provided, comprising: determining unmatched tracking targets based on the detection results of target detection on adjacent frame 3D point cloud data; setting a tracking task for the unmatched tracking targets and continuously tracking them until there is no 3D point cloud data of the tracking target in the detection box corresponding to the tracking target, or determining from the 3D point cloud data that the tracking target exceeds a preset range; and labeling the unmatched tracking targets with data based on the tracking results.

[0008] According to a second aspect of the embodiments of this application, a method for training a detection model is provided. The detection model includes a backbone network, a detection head network, and an auxiliary network. The method includes: acquiring 3D point cloud sample data; extracting features from the 3D point cloud sample data using the backbone network of the detection model to obtain corresponding feature vectors; performing target detection based on the feature vectors using the detection head network of the detection model to obtain detection results; predicting whether each position in the 3D point cloud sample data belongs to a target object sample based on the feature vectors using the auxiliary network of the detection model, and outputting prediction information; obtaining a first loss value based on the detection results and their corresponding first loss function; obtaining a second loss value based on the prediction information and its corresponding second loss function; and training the detection model based on the first loss value and the second loss value.

[0009] According to a third aspect of the embodiments of this application, a target recognition method is provided, comprising: acquiring a real-time acquired 3D point cloud data stream; determining a tracking target that matches the target and a tracking target that does not match the target based on the detection results of target detection on adjacent frames of 3D point cloud data in the 3D point cloud data stream; tracking the tracking target that matches the target based on the detection results, and setting a tracking task for the tracking target that does not match the target and continuously tracking it until there is no 3D point cloud data of the tracking target in the detection frame corresponding to the tracking target, or determining from the 3D point cloud data that the tracking target exceeds a preset range; and identifying the tracking target based on the tracking results of the tracking target that matches the target and the tracking results of the tracking target that does not match the target.

[0010] According to a fourth aspect of the embodiments of this application, an electronic device is provided, including: an image acquisition device, a lidar, a display, a processor, a communication interface, and a communication bus, wherein the image acquisition device, the lidar, the display, the processor, and the communication interface communicate with each other through the communication bus; wherein: the lidar is used to acquire 3D point cloud data of the surrounding environment in real time to form a 3D point cloud data stream; the processor is used to determine the tracking target that matches the target and the tracking target that does not match the target based on the detection result of target detection of adjacent frames of 3D point cloud data in the 3D point cloud data stream; and to track the tracking target that matches the target based on the detection result. The system sets a tracking task for the unmatched tracking target and continuously tracks it until the 3D point cloud data of the tracking target is no longer present in the detection box corresponding to the tracking target, or the tracking target is determined to be outside a preset range based on the 3D point cloud data; it identifies the tracking target based on the tracking results of the matched tracking target and the tracking results of the unmatched tracking target; it overlays the tracking target identification result onto a preset image, or performs 3D modeling based on the tracking target identification result and the image acquired in real time by the image acquisition device to obtain a 3D virtual scene; the display is used to display the image overlaid with the tracking target identification result, or to display the 3D virtual scene.

[0011] According to a fifth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first, second, or third aspect.

[0012] According to a sixth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform an operation corresponding to the method described in the first, second, or third aspect.

[0013] According to the automatic labeling scheme provided in this application embodiment, based on the detection results of 3D point cloud data, it is determined whether there are any untrackable targets. Unlike the traditional method of deleting targets if they are not detected in consecutive N frames, this application embodiment scheme continuously tracks these targets until their corresponding detection frame no longer contains 3D point cloud data of the target or they have exceeded a preset distance range. Therefore, by combining tracking with multi-frame 3D point cloud data for detection, the detection results are more accurate, effectively avoiding missed detections and false detections caused by target occlusion or sparse point clouds. Consequently, the data labeling based on this is also more accurate and of higher quality.

[0014] According to the detection model training scheme provided in this application, an auxiliary network is added to the detection model to detect whether each position in the 3D point cloud sample data belongs to the target object sample. This assists in the training of the entire detection model, enabling the trained detection model to extract more detailed and comprehensive 3D point cloud data features, resulting in better detection performance and robustness. Consequently, subsequent detection and tracking of 3D point cloud data using this detection model is more accurate, and the 3D point cloud data labeling based on the detection and tracking results is also more accurate and of higher quality. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0016] Figure 1 This is a schematic diagram of an exemplary system applicable to embodiments of this application;

[0017] Figure 2A This is a flowchart illustrating the steps of an automatic marking method according to Embodiment 1 of this application;

[0018] Figure 2B for Figure 2A A schematic diagram of a reverse tracking detection in the illustrated embodiment;

[0019] Figure 3A This is a flowchart illustrating the steps of a detection model training method according to Embodiment 2 of this application;

[0020] Figure 3B for Figure 3A A schematic diagram of the structure of a detection model in the embodiment shown;

[0021] Figure 3C for Figure 2A A schematic diagram of a scenario example in the illustrated embodiment;

[0022] Figure 4 This is a flowchart illustrating the steps of a target recognition method according to Embodiment 3 of this application;

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device according to Embodiment 4 of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0025] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0026] Figure 1 An exemplary system applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 1 The example in the text shows multiple user devices.

[0027] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to automatically label 3D point cloud data. As an optional example, in some embodiments, the cloud server 102 can determine unmatched tracking targets based on the detection results of the 3D point cloud data; then, a tracking task is set for the tracking target and tracking is continued until there is no 3D point cloud data of the tracking target in the corresponding detection box or the tracking target is within a preset range; then, the tracking results are used to automatically label the tracking target. As another example, in some embodiments, the cloud server 102 can be equipped with a detection model to perform target detection and target tracking on the 3D point cloud data. As another example, in some embodiments, the cloud server 102 can also train the detection model. In some embodiments, the cloud server 102 can label the 3D point cloud data according to a request sent by the user device 106. In some embodiments, the cloud server 102 may also use the tracking method in automatic labeling to track and identify targets.

[0028] In some embodiments, communication network 104 may be any suitable combination of one or more wired and / or wireless networks. For example, communication network 104 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User equipment 106 may be connected to communication network 104 via one or more communication links (e.g., communication link 112), and communication network 104 may be linked to cloud server 102 via one or more communication links (e.g., communication link 114). Communication links may be any communication link suitable for transmitting data between user equipment 106 and cloud server 102, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0029] User device 106 may include any one or more user devices suitable for interacting with users, presenting information, and inputting data. In some embodiments, user device 106 may send a request to cloud server 102 to label 3D point cloud data and receive the labeling results from cloud server 102. The 3D point cloud data may be sent by user device 106 to cloud server 102, or the user device may include the storage address of the 3D point cloud data in the request, and cloud server 102 may retrieve the data from that storage address and then perform labeling. In some embodiments, user device 106 may also send a request to cloud server 102 to perform target tracking and identification on the 3D point cloud data stream formed by real-time acquired 3D point cloud data and receive the results from cloud server 102. In some embodiments, user device 106 may include any suitable type of device. For example, in some embodiments, user device 106 may include any suitable type of user device such as a mobile device, tablet computer, laptop computer, or desktop computer.

[0030] Based on the above system, the solution provided in this application will be described below through several embodiments.

[0031] Example 1

[0032] Reference Figure 2A The diagram shows a flowchart of an automatic marking method according to Embodiment 1 of this application.

[0033] The automatic marking method in this embodiment includes the following steps:

[0034] Step S202: Based on the detection results of target detection on adjacent frame 3D point cloud data, determine the unmatched tracking targets.

[0035] In many scenarios involving autonomous driving, such as self-driving cars, vehicle-to-everything (V2X) communication, and cleaning robots, it is necessary to acquire perception data through LiDAR, specifically 3D point cloud data collected by LiDAR. Based on 3D point cloud data, autonomous driving control and decision-making can be effectively achieved, but this requires a high-performance detection model. To obtain such a high-performance detection model, the training sample data used for training it needs accurate labels (Label). The process of generating these labels for the training sample data is the labeling process described in the embodiments of this application.

[0036] To achieve accurate labeling, this embodiment first performs target detection on adjacent 3D point cloud data frames. Generally, the labeling effect is good for detected and trackable targets. However, for targets that fail to be tracked, these targets may be temporarily completely or partially occluded, or the data may be sparse in some frames, easily leading to missed detections and false detections. To address this, this embodiment first determines those untracked targets based on the detection results of target detection on adjacent frames (usually two adjacent frames, but not limited to these; if it is frame-by-frame detection, then two adjacent detection frames).

[0037] While target detection on adjacent 3D point cloud data can be achieved using conventional detection models, to achieve more accurate and effective detection, an improved detection model provided in this application can be used in one feasible approach. This detection model is a machine learning model trained on a point cloud segmentation auxiliary task. This improved detection model, which performs target detection on 3D point cloud data, can be used to detect targets on the 3D point cloud data. The point cloud segmentation auxiliary task is used to predict whether each location in the 3D point cloud data belongs to a target object sample.

[0038] The detection model can be divided into a backbone network, a detection head network, and an auxiliary network. Based on this, the detection model can be trained using 3D point cloud sample data. During training, the backbone network is used to extract features from the 3D point cloud sample data and output the corresponding feature vectors; the detection head network is used to perform target detection based on the feature vectors output by the backbone network; and the auxiliary network is used to predict whether each location in the 3D point cloud sample data belongs to the target object sample based on the feature vectors output by the backbone network and output the prediction information. Specifically, 3D point cloud sample data can be input into the detection model to be trained; the backbone network of the detection model extracts features from the 3D point cloud sample data and outputs the corresponding feature vector; the detection head network of the detection model performs target detection based on the feature vector to obtain the detection result; the auxiliary network of the detection model predicts whether each position in the 3D point cloud sample data belongs to the target object sample based on the feature vector and outputs the prediction information; a first loss value is obtained based on the detection result and its corresponding first loss function; a second loss value is obtained based on the prediction information and its corresponding second loss function; and the detection model is trained based on the first and second loss values. The specific training process of the detection model described above will be described in detail in Example 2 and will not be elaborated here.

[0039] Based on the target detection results of adjacent 3D point cloud data and the temporal relationship between adjacent frames, the tracking targets that match and those that do not match (the IoU of the corresponding detection boxes is below the threshold) can be determined.

[0040] An exemplary tracking process may include three stages: state prediction, target association, and state update.

[0041] In the state prediction phase, for each detected target, its state can be represented as a 10-dimensional state vector (x, y, z, θ, l, w, h, v_x, v_y, v_z). The first three vectors (x, y, z) represent the center position of the target; the fourth vector θ represents the horizontal angle of the corresponding 3D bounding box; the fifth, sixth, and seventh vectors (l, w, h) represent the length, width, and height of the corresponding 3D bounding box; and the eighth, ninth, and tenth vectors (v_x, v_y, v_z) represent the target's velocity in the three directions. This velocity is calculated during tracking after the target's tracking and matching in the current and previous frames. Therefore, based on the target's position and velocity in the previous frame, the position of the target in the current frame can be predicted.

[0042] In the target association stage, the detection result (detection box) of the target in the current frame is matched with the predicted result (predicted box) in the aforementioned state prediction, and an affinity matrix is ​​generated based on the corresponding 3D IoU. For example, the Hungarian algorithm can be used for matching, and a threshold is set; if the IoU is lower than the threshold, it is considered not matched. This threshold can be appropriately set by those skilled in the art according to actual needs, and this application embodiment does not impose any limitations on it; for example, it can be 0.3.

[0043] Therefore, the tracked and matched targets and the untracked and matched targets can be identified. This application focuses on the untracked and unmatched targets. In addition, for subsequent detection and tracking, the state of the target needs to be updated, i.e., entering the state update stage.

[0044] During the state update phase, the target's state can be updated to (x', y', z', θ', l', w', h', v_x', v_y', v_z') using a weighted summation method. The weights are determined by uncertainty, such as through Kalman filtering. This updated state becomes the state of the current frame, allowing the system to return to the state prediction phase and continue the target tracking process.

[0045] Step S204: Set up a tracking task for the unmatched tracking target and perform continuous tracking until the 3D point cloud data of the tracking target does not exist in the detection box corresponding to the tracking target, or the tracking target is determined to be outside the preset range based on the 3D point cloud data.

[0046] Unlike traditional tracking methods where unmatched targets are deleted if they are not detected in the next N frames, this embodiment assigns a corresponding tracking task to unmatched targets and continuously tracks them. For example, a new track can be immediately assigned to the detection box of any unmatched target, and tracking can continue. This significantly improves the recall rate of tracked targets.

[0047] For moving targets, tracking can continue until the corresponding detection bounding box for the target contains no 3D point cloud data for that target, or until the target exceeds a preset range. This ensures that tracking continues even if the target fails to match the detection bounding box multiple times, further improving recall. The preset range can be appropriately set by those skilled in the art based on the actual scenario. For example, in scenarios with high density of people, vehicles, and objects, the preset range can be set to 10 meters; in scenarios with low density, the preset range can be set further, such as 50-100 meters, and so on.

[0048] For stationary targets, tracking will continue as long as the target remains within the preset range. Therefore, even if the target is obscured for an extended period, tracking can continue. As mentioned earlier, this preset range can be appropriately set by those skilled in the art based on the actual scenario. For example, in scenarios with a high density of people, vehicles, and objects, the preset range can be set to 10 meters; in scenarios with a low density of people, vehicles, and objects, the preset range can be set further, such as 50-100 meters, and so on.

[0049] This allows for relatively accurate detection and tracking results. However, to further improve the accuracy of detection and tracking, one feasible approach is to process the tracking boxes obtained during tracking, including: obtaining a sequence of tracking boxes corresponding to the tracked target based on the continuous tracking process; determining whether to filter out the tracking boxes corresponding to the sequence based on the tracking information; performing corresponding operations on the tracking box sequence based on the determination result; and determining the tracking result for unmatched targets based on the operation result. This allows for the filtering out of invalid sequences from the tracking box sequence, thereby improving subsequent labeling efficiency.

[0050] Among them, determining whether to perform a filtering operation on the tracking boxes corresponding to the tracking box sequence based on the tracking information corresponding to the tracking box sequence can be achieved by one or more of the following methods combined.

[0051] Method 1: Based on the relationship between the matching information of the tracking boxes and the corresponding detection boxes in the tracking box sequence and the preset matching threshold, determine whether to filter out all tracking boxes in the tracking box sequence.

[0052] The matching information can be the number of matches or the percentage of matches in all matches. Correspondingly, the preset matching threshold can be a preset matching count threshold or a preset matching percentage threshold. The specific settings of these two thresholds can be appropriately set by those skilled in the art according to actual needs; this application embodiment does not impose any limitations on this. For example, the preset matching percentage threshold can be 0.3. In this approach, the fewer the number of matches between the tracking box and the detection box or the lower the percentage, the lower the probability that it may be the target object; therefore, the tracking box sequence can be filtered out.

[0053] Method 2: Determine whether to filter out all tracking boxes in the tracking box sequence based on the relationship between the duration of the tracking box sequence and the preset time length threshold.

[0054] The preset time length threshold can be appropriately set by those skilled in the art according to the actual situation, for example, such as 0.5 seconds. In this method, the shorter the duration of the tracking box sequence, the more likely it is to be an invalid tracking target, and it can be filtered out.

[0055] Method 3: Based on the relationship between the number of 3D point cloud data of the tracked target in each tracking box in the tracking box sequence and the preset number threshold, determine whether to filter out all tracking boxes in the tracking box sequence.

[0056] The preset number threshold can be appropriately set by those skilled in the art according to the actual situation, for example, 15 points. In this method, if the number of 3D point cloud data of the tracked target in each tracking box in the tracking box sequence is consistently lower than the preset number threshold, it indicates that it may be an invalid tracking target, such as noise, and therefore, it can be filtered out.

[0057] By using the methods described above, a valid sequence of tracking boxes can be obtained. Based on this sequence, tracking boxes that have significant reference value for subsequent labeling can be identified. For example, the retained tracking boxes can be determined based on the operation results, and the tracking boxes whose 3D point cloud data quantity meets the preset sorting criteria can be selected from them. Based on the selected tracking boxes, the tracking results for unmatched tracking targets can be determined.

[0058] For example, the following operations can be performed on the obtained tracking box sequence: (1) If the calculated hit ratio (the proportion of times the tracking box matches the detection box) is less than 0.3 for a certain tracking box sequence track list, then delete the track list; (2) If the time length of a certain track list is less than 0.5 seconds, then delete the track list; (3) If the number of points in a tracking box in a certain track list never exceeds 15 points, then delete the track list; (4) Among the remaining tracking box sequences, select the top 3 tracking boxes with the most points, as their scale estimation is the most accurate. The average of these three boxes can be used as the scale estimation of all tracking boxes in the sequence to significantly improve the accuracy of the tracking box scale estimation. Based on this, the remaining tracking boxes can be used as tracking results for subsequent data labeling.

[0059] Step S206: Label the unmatched tracking targets based on the tracking results.

[0060] Based on the aforementioned tracking process, the results of multi-frame 3D point cloud data detection can be combined to determine the tracking results (such as detection boxes or tracking boxes) corresponding to unmatched tracking targets. Thus, the unmatched tracking targets can be accurately labeled in these frames, i.e., a label can be set. The information of the label includes, but is not limited to, the position information, orientation information, and scale information of the tracking target. This information can be obtained based on the state vector corresponding to the tracking target in the corresponding frame.

[0061] Furthermore, in many cases, the target moves from a distance towards the nearest point. However, due to the sparse 3D point cloud data at a distance, the target is difficult to detect, while detection at a closer distance is much easier. Based on this, embodiments of this application also provide a reverse tracking method to improve the recall rate of such targets with sparse point clouds at a distance, moving from far to near.

[0062] Specifically, this can be implemented as follows: if, based on the detection results of target detection on adjacent frames of 3D point cloud data, the target to be tracked is determined to be a target moving from far to near, then the sequence of tracking boxes corresponding to the target is obtained; the initial tracking box in the sequence of tracking boxes is determined, as well as the initial time of the 3D point cloud data frame corresponding to the initial tracking box; a preset number of 3D point cloud data frames with times earlier than the initial time are obtained; based on the initial tracking box, the preset number of 3D point cloud data frames are tracked; and based on the tracking results, the tracking targets in the preset number of 3D point cloud data frames are labeled with data.

[0063] An example of reverse tracking detection is as follows: Figure 2B As shown, Figure 2B The top row represents target detection using the traditional method, and the bottom row represents target detection using the reverse tracking method.

[0064] Depend on Figure 2B As can be seen, using the traditional method, the 3D point cloud data is relatively sparse when the target is at a distance (corresponding to the first 3 frames marked with an X), making it difficult to detect until it moves closer (corresponding to frames 4-7). However, in reality, the target also exists in the first 3 frames, but cannot be detected using the traditional method.

[0065] The reverse tracking detection scheme, after detecting the target in the 4th frame, re-detects the previous 3 frames (marked by dashed boxes). Because it uses the detection results from later frames as a basis, it can predict and track targets in earlier frames based on existing results. Therefore, it can obtain the target's detection result from the relatively sparse 3D point cloud data in the earlier frames. This improves target recall and detection accuracy. Consequently, the data labeling based on this is also more accurate and has higher labeling quality.

[0066] This embodiment determines whether there are untrackable targets based on the detection results of 3D point cloud data. Unlike the traditional approach of deleting targets if they are not detected in consecutive N frames, this embodiment continuously tracks them until their corresponding detection frame no longer contains 3D point cloud data or they have exceeded a preset distance range. Therefore, by combining tracking with multi-frame 3D point cloud data for detection, the detection results are more accurate, effectively avoiding missed and false detections caused by target occlusion or sparse point clouds. Consequently, the data labeling based on this approach is also more accurate and of higher quality.

[0067] Example 2

[0068] This embodiment focuses on training the detection model used in Embodiment 1 to describe the solution of this application. For example, the results of the detection model are first introduced, such as... Figure 3B As shown.

[0069] from Figure 3B As can be seen, the detection model includes a backbone network (shown as Backbone in the figure), a detection head network (shown as Detection head in the figure), and an auxiliary network (shown as Auxiliary network in the figure).

[0070] The backbone network is used to extract features from 3D point cloud sample data and output the corresponding feature vectors. In one example, the backbone network can be implemented as an encoder structure, which includes at least convolutional layers and pooling layers, and can extract robust 3D point cloud features from the 3D point cloud data.

[0071] The detection head network is used for target detection based on the feature vectors output by the backbone network. In one example, the detection head network can be implemented as any suitable target detector, including but not limited to the SSD (SingleShot MultiBox Detector) detector, or a detector capable of matching prior boxes with ground truth using 2D joint cross-sections (IoU). Exemplarily, a structure similar to the backbone network and detection head in the Pointpillar model can be adopted. The detection head can output the target location and target category based on the 3D point cloud features extracted by the backbone.

[0072] To make the features extracted by the backbone network more robust and further improve the target detection performance, the detection model in this embodiment introduces a point cloud segmentation auxiliary task to help the backbone extract more robust 3D point cloud features from the original point cloud data. This auxiliary network part is used to predict whether each position in the 3D point cloud sample data belongs to the target object sample based on the feature vector output by the backbone network part, and outputs the prediction information.

[0073] like Figure 3B As shown, the scale at the Backbone input is set to N*N. Due to pooling in the pooling layer, the scale at the Backbone output becomes N / 4*N / 4. In the introduced auxiliary network, deconvolution is added to restore the scale to N*N. Subsequently, semantic segmentation is used to determine whether each position in N*N belongs to the target object sample, such as an obstacle. Specifically, a sigmoid activation function is added at the end of the auxiliary network, converting the output at each position x into a probability form P(x) of [0,1].

[0074] In the point cloud segmentation auxiliary task, the following loss function is used:

[0075] E=∑-w(x)log(y*P(x)+(1-y)*(1-P(x)))

[0076] Where y represents the ground truth of the detection box, i.e. whether position x belongs to the target object sample such as an obstacle; P(x) represents the probability predicted by the detection model that the position belongs to the target object sample such as an obstacle; w(x) represents the weight, which gives different weights to different positions in the loss function for class balancing of the target object sample's corresponding category.

[0077] During training, the aforementioned loss function assists in the training of the entire detection model, significantly improving its overall performance. Although this auxiliary network is no longer used in subsequent applications, its participation in training allows the entire detection model to acquire more robust 3D point cloud features.

[0078] Based on this detection model structure, the flowchart of a detection model training method provided in this application embodiment is as follows: Figure 3A As shown. The training method for this detection model includes the following steps:

[0079] Step S302: Obtain 3D point cloud sample data, and extract features from the 3D point cloud sample data through the backbone network of the detection model to be trained to obtain the corresponding feature vector.

[0080] As mentioned earlier, this backbone network portion can be implemented as an encoder structure, for example, as the backbone network structure in the Pointpillar model. Through this backbone network portion, 3D point cloud features, i.e., the feature vectors, can be obtained.

[0081] Step S304: Target detection is performed based on the feature vector using the detection head network part of the detection model to obtain the detection result.

[0082] As mentioned earlier, the detection head network can be implemented as a detection head structure in the Pointpillar model. This detection head network can perform target detection and obtain corresponding detection results. A first loss function is set in this part, including classification and regression components. The classification loss function obtains the target's category, and the regression loss function obtains the target's location. It should be noted that the specific form of this first loss function is not limited in this embodiment; it only needs to possess the aforementioned functionality.

[0083] Step S306: Using the auxiliary network part of the detection model, based on the feature vector, predict whether each position in the 3D point cloud sample data belongs to the target object sample, and output the prediction information.

[0084] As mentioned earlier, the auxiliary network part may include deconvolution layers (or upsampling layers) and semantic segmentation layers. In one feasible approach, semantic segmentation is performed based on the feature vector by the auxiliary network part of the detection model; based on the result of semantic segmentation, it is determined whether each position in the 3D point cloud sample data belongs to the target object sample, and prediction information is output. Optionally, semantic segmentation based on the feature vector can be implemented as follows: upsampling the feature vector to obtain a feature vector of the same dimension as the 3D point cloud sample data input to the backbone network part; semantic segmentation is performed on the 3D point cloud sample data based on the feature vector of the same dimension.

[0085] For example, by combining the structure of this auxiliary network, the feature vectors output by the backbone network can be scaled back by deconvolution layers, and the scaled feature vectors can be semantically segmented by semantic segmentation layers to determine whether each location point in the 3D point cloud data belongs to the target object sample.

[0086] In addition, a corresponding second loss function is set in this auxiliary network part, such as the loss function E=∑-w(x)log(y*P(x)+(1-y)*(1-P(x))) mentioned above.

[0087] The meanings of the parameters in the second loss function are as described above and will not be repeated here.

[0088] Step S308: Obtain the first loss value based on the detection result and its corresponding first loss function; obtain the second loss value based on the prediction information and its corresponding second loss function.

[0089] It should be noted that the operations of obtaining the first loss value and obtaining the second loss value can be performed in any order, or in parallel.

[0090] Step S310: Train the detection model based on the first loss value and the second loss value.

[0091] By combining two loss values ​​to train the detection model, the model can extract more robust 3D point cloud features, thereby improving the overall performance of the detection model.

[0092] Training the detection model is an iterative process that continues until the training termination condition is met, such as reaching a preset number of training iterations or the loss value meeting a preset threshold.

[0093] Once the detection model is trained, it can be used for detection and tracking as described in Example 1. For example, the detection model is used to perform target detection on adjacent frames of 3D point cloud data to obtain detection results; then, based on the detection results, unmatched tracking targets are identified; a tracking task is set for the unmatched tracking targets and continuous tracking is performed until the 3D point cloud data of the tracking target does not exist in the detection box corresponding to the tracking target, or it is determined from the 3D point cloud data that the tracking target is outside a preset range; then, data labeling is performed on the unmatched tracking targets based on the tracking results.

[0094] According to the detection model training scheme provided in this embodiment, an auxiliary network component is added to the detection model to detect whether each position in the 3D point cloud sample data belongs to the target object sample. This assists in the training of the entire detection model, enabling the trained detection model to extract more detailed and comprehensive 3D point cloud data features, resulting in better detection performance and robustness. Consequently, subsequent detection and tracking of 3D point cloud data using this detection model is more accurate, and the 3D point cloud data labeling based on the detection and tracking results is also more accurate and of higher quality.

[0095] The following example uses a specific scenario to illustrate the automatic labeling process based on the above detection model. Figure 3C As shown.

[0096] Figure 3CIn this process, N consecutive 3D point cloud data frames are input to the detection model. Assuming that based on the detection model's results for frames 30 and 31, the target X bounded by detection box 1 in frame 30 does not appear in frame 31, then target X is identified as an untracked target (illustrated as a car in the diagram). A new track task is assigned to target X, and it is continuously tracked. Assuming it reappears in frame 48 and persists in frames 49-100, the following steps are taken: First, it is determined whether the hit ratio in the track list corresponding to frames 30-100 is less than 0.3 (assuming it is 0.7 in this example). The track list corresponding to target X in frames 30-100 is retained. Next, the duration of this track list is determined (assuming it is 2 seconds in this example), which is greater than the preset duration threshold of 0.5 seconds. The track list is then retained. Furthermore, determine the number of 3D point data points corresponding to target X in each tracking box in the track list. In this example, it is assumed that each has more than 15 point cloud data points, and the track list is retained.

[0097] However, we need to determine the number of 3D point cloud data points corresponding to target X in each tracking box in the track list. Let's assume that frames 90, 92, and 95 have the most 3D point cloud data points. Then, we select three tracking boxes corresponding to target X in these three frames and label target X based on these three tracking boxes. For example, we can obtain the position, orientation, and scale of target X based on the 3D point cloud data in these three tracking boxes.

[0098] Of course, the above example only uses one target X as a simple example. The actual implementation is much more complex than the example above. However, for each detected target, especially untracked and unmatched targets, the above method can be used to detect and track the target, and then automatically label it based on the tracking results.

[0099] Practice has proven that the above-mentioned point cloud segmentation-based auxiliary task can effectively improve the performance of the detection model. Furthermore, the tracking method described in this application embodiment for unmatched tracking targets allows for optimization of labeling results based on multi-frame point cloud data. Applying these two improvements to the Kitti dataset, and comparing them with traditional 3D point cloud detection models such as the Pointpillar model, the accuracy of the detection results is shown in the table below:

[0100]

[0101] It is evident that incorporating point cloud segmentation assistance into the 3D detection model can significantly improve its detection performance. Furthermore, by employing multi-frame continuous tracking in this embodiment and using the tracking results for labeling, model performance can be further enhanced. Consequently, the labeling of 3D data becomes more accurate and of higher quality.

[0102] Example 3

[0103] Reference Figure 4 The flowchart illustrates the steps of a target recognition method according to Embodiment 3 of this application.

[0104] In this embodiment, the tracking method used in Embodiment 1 is applied to the target tracking and identification process, which can improve the accuracy of target tracking and identification.

[0105] The target recognition method in this embodiment includes the following steps:

[0106] Step S402: Acquire the real-time 3D point cloud data stream.

[0107] For devices equipped with LiDAR, including but not limited to autonomous vehicles, autonomous robots, and some AR (Augmented Reality) and VR (Virtual Reality) devices, they will collect 3D point cloud data of the surrounding environment in real time when they are in operation, thereby forming a 3D point cloud data stream. That is, the data stream is formed based on the collection of multiple consecutive frames of 3D point cloud data.

[0108] Step S404: Based on the detection results of target detection on adjacent frames of 3D point cloud data in the 3D point cloud data stream, determine the tracking targets that match and the tracking targets that do not match.

[0109] Since this embodiment is mainly used for accurate target tracking, it will identify not only unmatched tracking targets but also matched tracking targets. The specific method for determining matched and unmatched tracking targets based on the detection results can be found in the relevant description in Embodiment 1, and will not be repeated here.

[0110] Step S406: Track the matched targets based on the detection results; and set up tracking tasks for unmatched targets and continue tracking them until the 3D point cloud data of the target does not exist in the detection box corresponding to the target, or the target is determined to be outside the preset range based on the 3D point cloud data.

[0111] For targets that match the tracking criteria, conventional tracking methods can be used; while for targets that do not match, the method described in Example 1 can be used. Corresponding tracking results are then obtained for each.

[0112] Step S408: Based on the tracking results of the matched tracking targets and the tracking results of the unmatched tracking targets, identify the tracking targets.

[0113] Whether the tracking target is matched or not, the tracking results contain the same information, such as the information in the previously mentioned 10-dimensional state vector (x,y,z,θ,l,w,h,v_x,v_y,v_z), or at least position information, orientation information, and scale information. Based on this, target recognition can be achieved, including but not limited to corresponding position recognition, orientation recognition, scale recognition, and recognition such as category recognition (e.g., people, vehicles, obstacles), travel path recognition, etc.

[0114] Optionally, these identification results can also be applied to subsequent practical applications as needed. Therefore, in one feasible embodiment, this embodiment may further include the following step S410.

[0115] Step S410: Overlay the result of the target recognition onto the preset image and display it; or, perform 3D modeling based on the result of the target recognition and the image acquired in real time by the image acquisition device to obtain a 3D virtual scene and display it.

[0116] For example, during driving planning or navigation, the results of the aforementioned target identification can be overlaid on an electronic map to more clearly display information about the environment in which the vehicle, robot, or AR / VR device is located.

[0117] Alternatively, if the device incorporates both LiDAR and image acquisition equipment such as a camera, the LiDAR can acquire 3D point cloud data of the surrounding environment in real time, while the image acquisition equipment simultaneously acquires image data of the surrounding environment. Based on this, the aforementioned target recognition results can be combined with the image data to achieve the desired application.

[0118] For example, during driving planning or navigation, the results of the aforementioned target identification can be overlaid onto the images captured in real time by the image acquisition device, and the image data of the current environment of the vehicle, robot or AR / VR device and the corresponding information of each tracked target in the image can be displayed more clearly on the screen.

[0119] For example, in certain virtual reality scenarios, such as simulated scene displays or game scenes, 3D modeling can be performed based on the results of the aforementioned target tracking and recognition and the real-time image acquisition by the image acquisition device to obtain the corresponding 3D virtual scene; then, the 3D virtual scene can be displayed through a monitor so that users can obtain an immersive experience in the corresponding scene through VR devices.

[0120] As can be seen, by applying the tracking method for unmatched targets provided in this application to a tracking and identification scenario, the tracking and identification accuracy can be improved, resulting in better tracking performance.

[0121] Example 4

[0122] Reference Figure 5 This diagram illustrates the structure of an electronic device according to Embodiment 4 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device. Exemplarily, it can be implemented as a vehicle, robot, AR device, or VR device, etc.

[0123] like Figure 5 As shown, the electronic device may include: an image acquisition device 502, a lidar 504, a display 506, a processor 508, a communications interface 510, and a communications bus 512.

[0124] in:

[0125] Image acquisition device 502 is used to acquire image data of the surrounding environment in real time.

[0126] The LiDAR 504 is used to collect 3D point cloud data of the surrounding environment in real time and form a 3D point cloud data stream.

[0127] The processor 508 is configured to: determine matching and non-matching tracking targets based on the detection results of target detection on adjacent frames of 3D point cloud data in the 3D point cloud data stream; track the matching targets based on the detection results; and set tracking tasks for the non-matching targets and continuously track them until the 3D point cloud data of the target does not exist in the detection box corresponding to the target, or the target is determined to be outside a preset range based on the 3D point cloud data; identify the tracking targets based on the tracking results of the matching targets and the non-matching targets; overlay the identified target results onto a preset image (including images acquired by the image acquisition device 502 and other images acquired by non-image acquisition devices 502, such as electronic maps), or perform 3D modeling based on the identified target results and images acquired in real time by the image acquisition device to obtain a 3D virtual scene.

[0128] Display 506 is used to display an image overlaid with the results of the tracked target recognition, or to display a 3D virtual scene.

[0129] Image acquisition device 502, lidar 504, display 506, processor 508, communication interface 510, and communication between them are completed through communication bus 512.

[0130] Communication interface 510 is used to communicate with other electronic devices or cloud servers.

[0131] The processor 508 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0132] Optionally, the electronic device of this embodiment may further include a memory 514 for storing various data in the electronic device, such as image data, 3D point cloud data, various intermediate data generated during tracking, etc. The memory 514 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0133] The specific implementation of the above-described operations of the electronic device in this embodiment can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device and module described above can be referred to the corresponding process description in the foregoing method embodiments, and will not be repeated here.

[0134] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0135] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0136] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0137] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0138] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. An automatic marking method, comprising: Based on the detection results of target detection on adjacent 3D point cloud data, identify unmatched tracking targets; Set a tracking task for the unmatched tracking target and perform continuous tracking until the 3D point cloud data of the tracking target is no longer present in the detection box corresponding to the tracking target, or the tracking target is determined to be outside the preset range based on the 3D point cloud data; Data is labeled based on the tracking results for targets that are not matched by the tracking target; The method further includes, prior to labeling the 3D point cloud data based on the tracking results: Based on the continuous tracking process, the sequence of tracking boxes corresponding to the tracking target is obtained; Based at least on the tracking information corresponding to the tracking box sequence, determine whether to perform a filtering operation on the tracking boxes corresponding to the tracking box sequence; Based on the judgment result, perform corresponding operations on the tracking box sequence, and determine the tracking result for the unmatched tracking target based on the operation result; Wherein, determining whether to perform a filtering operation on the tracking boxes corresponding to the tracking box sequence based at least on the tracking information corresponding to the tracking box sequence includes at least one of the following: Based on the relationship between the matching information of the tracking boxes and the corresponding detection boxes in the tracking box sequence and the preset matching threshold, it is determined whether to filter out all tracking boxes in the tracking box sequence; the matching information includes the number of matches, or the proportion of the number of matches in all matches; Based on the relationship between the duration of the tracking box sequence and a preset time length threshold, determine whether to filter out all tracking boxes in the tracking box sequence; Based on the relationship between the number of 3D point cloud data of the tracked target in each tracking box in the tracking box sequence and a preset number threshold, it is determined whether to filter out all tracking boxes in the tracking box sequence.

2. The method according to claim 1, wherein, Determining the tracking result for the unmatched tracking target based on the operation result includes: Based on the operation results, determine the remaining tracking frames, and select the tracking frames whose 3D point cloud data quantity meets the preset sorting criteria. Based on the selected tracking box, the tracking result for the unmatched tracking target is determined.

3. The method according to claim 1 or 2, wherein, The method further includes: If, based on the detection results of target detection on adjacent 3D point cloud data, it is determined that the tracking target to be matched is a tracking target that is moving from far to near, then the tracking box sequence corresponding to the tracking target is obtained; Determine the initial tracking box in the tracking box sequence, and the initial time of the 3D point cloud data frame corresponding to the initial tracking box; Acquire 3D point cloud data frames whose time is earlier than the initial time by a preset number of frames; Based on the initial tracking frame, a preset number of 3D point cloud data frames are tracked; Based on the tracking results, data labels are applied to the tracking targets in the preset number of 3D point cloud data frames.

4. The method according to claim 1 or 2, wherein, Before determining the unmatched tracking target based on the detection results of target detection on adjacent 3D point cloud data, the method further includes: The detection model based on 3D point cloud data is used to detect objects in 3D point cloud data; wherein the detection model is a machine learning model trained on a point cloud segmentation auxiliary task.

5. The method according to claim 4, wherein, The method further includes: The detection model was trained using 3D point cloud sample data; The detection model comprises a backbone network, a detection head network, and an auxiliary network. The backbone network is used to extract features from 3D point cloud sample data and output corresponding feature vectors. The detection head network is used to perform target detection based on the feature vectors output by the backbone network. The auxiliary network is used to predict whether each position in the 3D point cloud sample data belongs to a target object sample based on the feature vectors output by the backbone network and output prediction information.

6. The method according to claim 5, wherein, The step of training the detection model using 3D point cloud sample data includes: The 3D point cloud sample data is input into the detection model to be trained; The backbone network of the detection model is used to extract features from the 3D point cloud sample data and output the corresponding feature vectors. The detection head network of the detection model performs target detection based on the feature vector to obtain the detection result. The auxiliary network part of the detection model predicts whether each position in the 3D point cloud sample data belongs to the target object sample based on the feature vector, and outputs the prediction information. A first loss value is obtained based on the detection result and its corresponding first loss function; a second loss value is obtained based on the prediction information and its corresponding second loss function. The detection model is trained based on the first loss value and the second loss value.

7. The method according to claim 6, wherein, The auxiliary network portion of the detection model, based on the feature vector, predicts whether each location in the 3D point cloud sample data belongs to the target object sample and outputs prediction information, including: Semantic segmentation is performed based on the feature vectors through the auxiliary network part of the detection model; Based on the semantic segmentation results, determine whether each position in the 3D point cloud sample data belongs to the target object sample, make a prediction, and output the prediction information.

8. The method according to claim 7, wherein, The semantic segmentation based on the feature vector includes: The feature vector is upsampled to obtain a feature vector of the same dimension as the 3D point cloud sample data input to the backbone network. Based on the feature vectors of the same dimension, semantic segmentation is performed on 3D point cloud sample data.

9. A target recognition method, comprising: Acquire real-time 3D point cloud data streams; Based on the target detection results of adjacent frames of 3D point cloud data in the 3D point cloud data stream, the tracking targets that match and the tracking targets that do not match are determined. Based on the detection results, the tracking targets that are matched are tracked, and a tracking task is set for the tracking targets that are not matched and they are tracked continuously until there is no 3D point cloud data of the tracking target in the detection box corresponding to the tracking target, or the tracking target is determined to be outside the preset range based on the 3D point cloud data. Based on the continuous tracking process of the unmatched tracking targets, a sequence of tracking boxes corresponding to the unmatched tracking targets is obtained; Based at least on the tracking information corresponding to the tracking box sequence, determine whether to perform a filtering operation on the tracking boxes corresponding to the tracking box sequence; Based on the judgment result, perform corresponding operations on the tracking box sequence, and determine the tracking result for the unmatched tracking target based on the operation result; Based on the tracking results of matched targets and the tracking results of unmatched targets, target identification is performed. The step of determining whether to perform a filtering operation on the tracking boxes corresponding to the tracking box sequence based at least on the tracking information corresponding to the tracking box sequence includes at least one of the following: Based on the relationship between the matching information of the tracking boxes and the corresponding detection boxes in the tracking box sequence and the preset matching threshold, it is determined whether to filter out all tracking boxes in the tracking box sequence; the matching information includes the number of matches, or the proportion of the number of matches in all matches; Based on the relationship between the duration of the tracking box sequence and a preset time length threshold, determine whether to filter out all tracking boxes in the tracking box sequence; Based on the relationship between the number of 3D point cloud data of the tracked target in each tracking box in the tracking box sequence and a preset number threshold, it is determined whether to filter out all tracking boxes in the tracking box sequence.

10. The method according to claim 9, wherein, The method further includes: The results of target tracking and identification are overlaid onto a preset image and displayed. or, Based on the results of target tracking and recognition and the images acquired in real time by the image acquisition device, 3D modeling is performed to obtain and display a 3D virtual scene.

11. An electronic device, comprising: The system includes an image acquisition device, a lidar, a display, a processor, a communication interface, and a communication bus. The image acquisition device, the lidar, the display, the processor, and the communication interface communicate with each other via the communication bus. in: The lidar is used to collect 3D point cloud data of the surrounding environment in real time and form a 3D point cloud data stream. The processor is configured to: determine matching and non-matching tracking targets based on the detection results of target detection on adjacent frames of 3D point cloud data in the 3D point cloud data stream; track matching targets based on the detection results; and set tracking tasks for non-matching targets and continuously track them until no 3D point cloud data of the target exists in the detection box corresponding to the target, or the target is determined to be outside a preset range based on the 3D point cloud data; obtain a sequence of tracking boxes corresponding to the non-matching targets based on the continuous tracking process; determine whether to perform a filtering operation on the tracking boxes corresponding to the sequence of tracking boxes based at least on the tracking information corresponding to the sequence of tracking boxes; perform corresponding operations on the sequence of tracking boxes based on the determination result, and determine the tracking result for the non-matching targets based on the operation result; and determine the tracking result for the matching targets and the non-matching targets based on the tracking results for the matching targets and the non-matching targets. The tracking results are used to identify the tracking target; the results of the tracking target identification are superimposed on a preset image, or 3D modeling is performed based on the results of the tracking target identification and the images acquired in real time by the image acquisition device to obtain a 3D virtual scene; the step of determining whether to filter the tracking boxes corresponding to the tracking box sequence based on the tracking information corresponding to the tracking box sequence includes at least one of the following: determining whether to filter all tracking boxes in the tracking box sequence based on the relationship between the matching information of the tracking boxes and the corresponding detection boxes in the tracking box sequence and a preset matching threshold; the matching information includes the number of matches, or the proportion of the number of matches in all matches; determining whether to filter all tracking boxes in the tracking box sequence based on the relationship between the duration of the tracking box sequence and a preset duration threshold; determining whether to filter all tracking boxes in the tracking box sequence based on the relationship between the number of 3D point cloud data of the tracking target in each tracking box in the tracking box sequence and a preset number threshold; The display is used to display an image overlaid with the results of the tracking target recognition, or to display the 3D virtual scene.

Images