Target tracking method and system in point cloud continuous frames, electronic device and storage medium

Abstract

Embodiments of the present application provide a target tracking method and system in point cloud continuous frames, electronic equipment and storage medium. The method comprises: obtaining a continuous frame of a three-dimensional point cloud of a target to be tracked and a labeled bounding box in a frame of the continuous frame of the target to be tracked; for each frame in the continuous frame except the frame labeled with the labeled bounding box, determining selected point clouds in the frame based on at least the position parameters of the labeled bounding box of the target to be tracked, wherein the point clouds in the frame corresponding to the position of the labeled bounding box are part of the selected point clouds; performing target segmentation on the selected point clouds to determine segmented point clouds; performing tracking operation on the point clouds in the labeled bounding box to determine a predicted bounding box of the target to be tracked in the frame; and determining target tracking point clouds of the target to be tracked in the segmented point clouds based on the predicted bounding box. The scheme can significantly improve the accuracy of target tracking in point cloud continuous frames, and the efficiency of target tracking is also high.

Description

Target tracking methods, systems, electronic devices, and storage media in continuous point cloud frames Technical Field

[0001] This application relates to the field of point cloud processing technology, specifically to a target tracking method in continuous frames of point cloud, a target tracking system in continuous frames of point cloud, an electronic device, and a storage medium. Background Technology

[0002] Point clouds are discrete sets of points distributed in N-dimensional space, primarily in three dimensions. They are discrete samples of information about the surface of an object, typically acquired by LiDAR scanning, and play an important role in fields such as autonomous driving.

[0003] In the field of point cloud processing, target tracking within consecutive frames of a point cloud is required in various scenarios. For example, when annotating targets in a 3D point cloud, the target is first tracked within consecutive frames of the 3D point cloud, and then labeled based on the tracking results. Taking autonomous driving scenarios as an example, autonomous driving environmental perception systems have high requirements for 3D spatial perception capabilities and multi-frame collaborative processing capabilities. Therefore, it is necessary to track the same target in multiple consecutive frames and assign it the same label information.

[0004] Some existing methods employ traditional frame-by-frame manual tracking. This method is inefficient and has high labor costs. Other methods rely on difference prediction or in-journey prediction for tracking. For difference prediction or in-journey prediction, the target in specified preceding and following frames needs to be manually marked. For example, the target's position in the first and last frames is manually marked, and then, based on the difference in target position between these two frames, the target's position in the intermediate frames is estimated using fixed numerical calculations, thus completing target tracking across consecutive frames. This approach has low tracking accuracy. For example, it is more suitable for situations where the vehicle's direction of travel is straight, but for situations with complex vehicle trajectories, it is clearly difficult to accurately track vehicles across consecutive frames. Summary of the Invention

[0005] To at least partially address the aforementioned problems in the prior art, according to one aspect of this application, a method for target tracking in continuous frames of a point cloud is provided, comprising:

[0006] Acquire consecutive frames of the 3D point cloud of the target to be tracked, as well as the bounding boxes of the target within a single frame of the consecutive frames;

[0007] For each frame in a consecutive sequence, except for those frames marked with bounding boxes.

[0008] Based at least on the position parameters of the bounding box of the target to be tracked, the selected point cloud in the frame is determined, wherein the point cloud within the bounding box corresponding to the position of the bounding box in the frame is a part of the selected point cloud;

[0009] The selected point cloud is segmented to determine the segmented point cloud;

[0010] Tracking operations are performed on the point cloud within the labeled bounding box to determine the predicted bounding box of the target object in that frame; and

[0011] Based on the predicted bounding box, the target tracking point cloud of the target to be tracked is determined in the segmented point cloud.

[0012] For example, performing tracking operations on the point cloud within the labeled bounding box includes:

[0013] Within the selected point cloud area, tracking calculations are performed on the point cloud within the labeled bounding box to determine the predicted bounding box in the selected point cloud.

[0014] For example, determining the selected point cloud in the frame based at least on the position parameters of the labeled bounding box of the target to be tracked includes:

[0015] Based on the position parameters of the labeled bounding box, the corresponding box in the frame is determined, and the movement direction of the target to be tracked is determined;

[0016] Expand the corresponding bounding box at least along the direction of movement, so that the expanded bounding box serves as the selection bounding box; and

[0017] The point cloud within the bounding box is defined as the selected point cloud in this frame.

[0018] For example, determining the direction of movement of the target to be tracked includes:

[0019] The length direction of the annotation bounding box is determined as the movement direction. The annotation bounding box is a cuboid, and its length is greater than its width and height.

[0020] For example, expanding the position-corresponding box at least along the direction of movement includes:

[0021] The corresponding position box is expanded along both the length and width directions to use the expanded bounding box as the selection bounding box. The expansion factor of the selection bounding box in the length direction is greater than the expansion factor in the width direction.

[0022] For example, determining the target point cloud of the object to be tracked in the segmented point cloud based on the predicted bounding box includes:

[0023] Based on the predicted bounding boxes, determine the number of points in each segmented point cloud that fall within the predicted bounding boxes; and

[0024] The segmented point cloud with the largest number of points is identified as the target point cloud.

[0025] For example, acquiring consecutive frames of the 3D point cloud of the target to be tracked includes:

[0026] Receives the URL of consecutive frames input by the user; and

[0027] Utilizing multiple threads to obtain consecutive frames based on URLs.

[0028] For example, obtaining the labeled bounding box in one frame of a series of frames of the target to be tracked includes:

[0029] For the point cloud to be labeled in the first region of a frame in a series of frames, ground point cloud filtering is performed on the point cloud to be labeled based on the density of points in different parts of the point cloud to be labeled, so as to obtain the first point cloud after removing the ground point cloud.

[0030] The filtered first point cloud is segmented to determine the target point cloud of the target to be tracked, and the bounding box of the target point cloud is used as the annotation bounding box.

[0031] and / or

[0032] The selected point cloud is segmented to determine the segmented point cloud, including:

[0033] Based on the density of points in different parts of the selected point cloud, ground point cloud filtering is performed on the selected point cloud to obtain a second point cloud after removing the ground point cloud.

[0034] Target segmentation is performed on the filtered second point cloud to determine the segmented point cloud.

[0035] For example, ground point cloud filtering of the point cloud to be processed includes:

[0036] Based on the point density of different parts of the point cloud to be processed, the point cloud to be processed is divided into sparse point cloud part and dense point cloud part.

[0037] Progressive morphological filtering is performed on the sparse point cloud portion to remove ground point clouds from the sparse point cloud portion and obtain the filtering result, wherein the filtering result includes the filtered sparse point cloud portion after removing ground point clouds from the sparse point cloud portion.

[0038] The dense point cloud portion and the filtered sparse point cloud portion are merged to obtain the merged point cloud; and

[0039] The filtered point cloud is determined based on the merged point cloud;

[0040] The point cloud to be processed includes the point cloud to be labeled and / or the selected point cloud, and the filtered point cloud includes the first point cloud and / or the second point cloud.

[0041] According to another aspect of this application, a target tracking system in continuous frames of a point cloud is also provided, comprising:

[0042] The acquisition module is used to acquire consecutive frames of the 3D point cloud of the target to be tracked, as well as the bounding boxes of the target within a single frame of the consecutive frames.

[0043] The selection module is used to determine the selection point cloud in each frame except for the frame with a bounding box in a continuous frame, based at least on the position parameters of the bounding box of the target to be tracked. The point cloud within the bounding box corresponding to the position of the bounding box in the frame is a part of the selection point cloud.

[0044] The segmentation module is used to segment the selected point cloud into regions to determine the segmented point cloud, wherein each segmented point cloud includes a target.

[0045] The tracking calculation module is used to perform tracking calculations on the point cloud within the labeled bounding box to determine the predicted bounding box of the target object in that frame; and

[0046] The determination module is used to determine the target tracking point cloud of the target to be tracked in the segmented point cloud based on the predicted bounding box.

[0047] According to another aspect of this application, an electronic device is also provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the above-described target tracking method in consecutive frames of a point cloud.

[0048] According to another aspect of this application, a storage medium is also provided, on which program instructions are stored, which are used to execute the target tracking method in the continuous frames of the point cloud described above when the program instructions are run.

[0049] According to the above scheme, the target to be tracked in non-reference frames outside the frame is automatically tracked based on the bounding box of the target in a frame of a continuous frame of the acquired 3D point cloud. On the one hand, the selected point cloud in each non-reference frame is determined based on the position parameters of the bounding box, and then the segmented point cloud is determined by segmenting the target in the selected point cloud. On the other hand, the predicted bounding box of the target tracking point cloud in each non-reference frame is determined by performing tracking calculations on the point cloud in the bounding box. Finally, the target tracking point cloud in the segmented point cloud is accurately determined based on the determined predicted bounding box. This scheme can significantly improve the tracking accuracy of targets in continuous frames of point cloud. At the same time, the execution logic of this scheme is relatively simple and the computational load is relatively small, so the tracking efficiency is also high. Moreover, this scheme has good applicability and can be applied to target tracking of continuous frames of 3D point cloud in various scenarios. In addition, as mentioned above, the automatic tracking of targets in continuous frames in the prior art mostly involves manually annotating the position of the target to be tracked in the specified preceding and following frames, and then using methods such as difference prediction or in-journey prediction to predict and annotate the position of the target to be tracked in the continuous frames. This approach requires manual annotation of targets in at least two frames, resulting in high labor costs. Furthermore, its applicability is limited, particularly in terms of accuracy when tracking targets in continuous point cloud frames within large scenes. In contrast, this embodiment only requires target annotation in one frame of a continuous sequence, and then uses a target tracking algorithm to determine the target's position in the non-reference frame. This saves on manual annotation costs and improves tracking accuracy to some extent.

[0050] The invention description introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. This description is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0051] The advantages and features of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0052] The following drawings, which are incorporated herein by reference and are used to understand this application, illustrate embodiments of the invention and their descriptions to explain the principles of the invention. In the drawings,

[0053] Figure 1 shows a schematic flowchart of a target tracking method in consecutive frames of a point cloud according to an embodiment of this application;

[0054] Figure 2a shows a schematic diagram of point cloud target segmentation results according to an embodiment of the present application;

[0055] Figure 2b shows a schematic diagram of the tracking calculation results of the target to be tracked according to an embodiment of the present application;

[0056] Figure 2c shows a schematic diagram of the superposition of Figure 2a and Figure 2b;

[0057] Figure 3 shows a schematic flowchart of a target tracking method in consecutive frames of point cloud according to another embodiment of this application;

[0058] Figure 4 shows a schematic block diagram of a target tracking system in consecutive frames of a point cloud according to an embodiment of this application; and

[0059] Figure 5 shows a schematic block diagram of an electronic device according to an embodiment of the present application. Detailed Implementation

[0060] In the following description, numerous details are provided to enable a thorough understanding of this application. However, those skilled in the art will appreciate that the following description merely illustrates preferred embodiments of the application, and that the application can be implemented without one or more of these details. Furthermore, to avoid confusion with this application, some technical features well-known in the art have not been described in detail.

[0061] To at least partially solve the above-mentioned technical problems, according to one aspect of this application, a target tracking method in continuous frames of point clouds is provided.

[0062] Figure 1 shows a schematic flowchart of a target tracking method 100 in a continuous frame of a point cloud according to an embodiment of the present application. As shown in Figure 1, the target tracking method 100 in a continuous frame of a point cloud includes steps S110, S130, S150, S170 and S190.

[0063] Step S110: Obtain consecutive frames of the 3D point cloud of the target to be tracked and the bounding box of one frame of the consecutive frames of the target to be tracked.

[0064] According to embodiments of this application, a continuous frame of a 3D point cloud can be a 3D point cloud of the target to be tracked that is acquired sequentially over a period of time using any existing or future-developed point cloud acquisition device. It should be understood that the continuous frame can be a relative concept; it can be a series of continuously acquired original frames, or it can be a series of relatively continuous frames obtained by performing processes such as uniform frame subtraction on the original continuous frames acquired by the point cloud acquisition device.

[0065] The number of frames in a consecutive frame can be any number greater than or equal to 2. For example, the number of frames can be 10. All 10 frames of the point cloud can include the target to be tracked, or only some frames can include the target. The target to be tracked can be any suitable object to be tracked, such as any suitable dynamic object, like a human body, face, animal, or vehicle. Each frame in a consecutive frame of a 3D point cloud can include one target to be tracked, or it can include multiple targets.

[0066] In a specific example, the target to be tracked could be a moving car. The consecutive frames of the 3D point cloud are, for example, consecutive frames of the 3D point cloud obtained by the LiDAR of a moving car A over a period of time, capturing images of its external environment. It is understood that the external environment of car A may include pedestrians, roads, and other vehicles around it. Therefore, each frame of the consecutive frames of the 3D point cloud can include objects in this environment. In this example, the target to be tracked could be other vehicles located around car A. If multiple vehicles are surrounding car A, the target to be tracked could be one of those vehicles or multiple vehicles, depending on the specific requirements.

[0067] The continuous frames of a 3D point cloud can be raw point cloud data directly acquired using any existing or future suitable point cloud acquisition device, or point cloud data after preprocessing operations on the raw point cloud data. These preprocessing operations can include various suitable operations that facilitate subsequent processing, such as coordinate transformation of the raw point cloud, point cloud downsampling, and point cloud filtering.

[0068] According to the embodiments of this application, in this step, continuous frames of a 3D point cloud can be directly acquired using a point cloud acquisition device, or pre-stored continuous frames can be indirectly retrieved. For example, a user can send the Uniform Resource Locator (URL) of the continuous frame to be tracked to the server using a client, and in this step, the continuous frame can be retrieved from network storage using multiple threads based on the received continuous frame URL.

[0069] The operations of fetching consecutive frames and obtaining the bounding boxes of a single frame within the consecutive frames of the target to be tracked can be performed in various suitable execution orders. Optionally, the bounding boxes can be pre-annotated bounding boxes of the target to be tracked. In this step, the pre-annotated bounding boxes and the URLs of the consecutive frames to be tracked can be obtained first, and then the consecutive frames can be fetched based on the consecutive frame URLs. Alternatively, the consecutive frames to be tracked can be fetched first, and then the bounding boxes of a single frame within the consecutive frames of the target to be tracked can be obtained. For example, after fetching the consecutive frames, the user-input bounding boxes of the target to be tracked in the reference frames can be received, or the bounding boxes of the target can be determined using a suitable target annotation algorithm.

[0070] In one example, a frame in a series of frames can be any frame in the series, as long as it includes the target to be tracked. In another example, a frame in a series of frames can also be a specific frame within the series. This specific frame could be, for example, the middle frame or the first frame in the series. For simplicity, the frame including the bounding box of the target to be tracked will be referred to as the reference frame. According to an embodiment of this application, the bounding box can be a stereo bounding box of the target point cloud corresponding to the target to be tracked in the reference frame. The bounding box can be any suitable stereo bounding box, including but not limited to axis-aligned bounding boxes, directed bounding boxes, and point cloud convex hulls. Preferably, the bounding box can be the smallest directed bounding box of the target point cloud corresponding to the target to be tracked in the reference frame.

[0071] Steps S130 to S190 are described below. These steps can be performed on every frame in a continuous series of frames except for those with labeled bounding boxes, in order to track the target to be tracked in these frames, that is, to determine the target tracking point cloud in these frames.

[0072] Step S130: For each frame in a continuous series of frames, excluding those with labeled bounding boxes, determine the selected point cloud in that frame based at least on the position parameters of the labeled bounding boxes of the target to be tracked. The point cloud within the bounding box corresponding to the position of the labeled bounding box in that frame is a portion of the selected point cloud.

[0073] It is understood that the bounding box of the target to be tracked in the reference frame is the true bounding box of the target to be tracked in the reference frame. In a series of frames acquired over a period of time, the position of the target to be tracked may move. For example, for a moving car A, its external environment is constantly changing. Specifically, taking car B located next to car A as the target to be tracked, the relative positional relationship between car A and car B may change in a short period of time. Therefore, in a series of frames capturing the external environment of car A, the position of car B may be different in different frames. It is understood that the positional change of the same target in a series of frames is usually gradual. Therefore, after determining the position of the target in a certain frame of a series of frames, the approximate position range of the target in other frames can be estimated based on the position of the target in that frame. For simplicity, in the following description, frames other than the reference frame in a series of frames will be referred to as non-reference frames. According to the embodiments of this application, after determining the bounding box of the target to be tracked in the reference frame in step S110, the target to be tracked in each non-reference frame can be tracked based on the position parameters of the bounding box in the reference frame. Specifically, in this step, the selected point cloud in each non-reference frame can be determined based on the position parameters of the bounding box in the reference frame. The selected point cloud can be a point cloud representing the selected area where the target to be tracked may be located. This selected area can be a region that includes and is larger than the actual location area of ​​the target to be tracked in the non-reference frame.

[0074] According to embodiments of this application, the selected point cloud in each non-reference frame can be determined using various suitable determination logics based on the position parameters of the bounding box in the reference frame. Exemplarily, but not limitingly, a selected region in each non-reference frame, larger than the region containing the position corresponding box of the bounding box, can be determined based on the position parameters of the bounding box in the reference frame, and the point cloud falling within the determined selected region is defined as the selected point cloud. It is understood that the position of the position corresponding box in the non-reference frame is the same as the position of the bounding box in the reference frame. The selected point cloud includes all point clouds within the position corresponding box. For example, in the embodiment of continuous frames of the external environment of car A described above, if the bounding box of car B in the determined reference frame is represented as "200≤x≤700, 100≤y≤330 and 100≤z≤330", then a selection region can be determined in each non-reference frame. This selection region can be an area in the non-reference frame that includes and is larger than the position-corresponding box "200≤x≤700, 100≤y≤330 and 100≤z≤330". This selection region is, for example, a large area obtained by extending along a preset direction of the position-corresponding box. This preset direction is, for example, the determined movement direction of the target to be tracked. For example, any suitable method can be used to determine the movement direction of the target to be tracked. Specifically, for example, the bounding box is a rectangle, and the movement direction can be the direction of the long side of the rectangle. After determining the movement direction, the position-corresponding box can be expanded outward along this movement direction, and the expanded area is used as the selection region, thereby determining the selection point cloud in each reference frame that falls within the selection region. Alternatively, the preset direction can also be the direction of the three-dimensional coordinate axis of the point cloud, such as expanding the position corresponding box outward along at least one dimension to obtain the selection area, and determining the selected point cloud that falls within the selection area.

[0075] Step S150: Perform target segmentation on the selected point cloud to determine the segmented point cloud.

[0076] It is understood that the selected point cloud may include not only the target point cloud of the target to be tracked, but also non-target point clouds surrounding the target. For example, in the embodiment of continuous frames of the external environment of car A mentioned above, the selected point cloud in each determined non-reference frame may include not only the target point cloud corresponding to car B, but also non-target point clouds near car B. For example, the selected point cloud may also include ground point clouds located on the ground and point clouds corresponding to car C. In this step, any suitable method can be used to segment the selected point cloud to determine the segmented point cloud as the segmentation result. This segmented point cloud corresponds to the target to be tracked. Optionally, segmentation can be performed directly using a suitable point cloud segmentation algorithm, either existing or developed in the future, to obtain the segmentation result. These point cloud segmentation algorithms may include, but are not limited to, any one of the following: random sample consensus segmentation algorithm, Euclidean clustering segmentation algorithm, and region growing segmentation algorithm. For example, Euclidean clustering segmentation can be used directly to segment the selected point cloud to obtain the segmented point cloud. Since the computational cost of using the Euclidean clustering segmentation algorithm is lower, the segmentation result is also more accurate. Alternatively, the selected point cloud can be first filtered using ground point cloud techniques, and then a segmentation algorithm such as Euclidean clustering can be used to segment the filtered point cloud to obtain the segmented point cloud. This approach can more accurately segment the desired point cloud.

[0077] It is understandable that the number of segmented point clouds determined in this step may be one or more. In one example, if the selected point cloud only includes the target point cloud corresponding to car B and a small number of surrounding interfering point clouds, then in this step, a method such as Euclidean clustering can be used to segment the selected point cloud, resulting in a set of segmented point clouds corresponding to car B. In another example, if the selected point cloud includes the target point cloud corresponding to car B, the point cloud corresponding to car C, and nearby ground point clouds, then in this step, the selected point cloud can first be filtered for ground point clouds, resulting in filtered point clouds. It is understood that the filtered point clouds may include the point clouds corresponding to car B and the point clouds corresponding to car C. Then, a method such as Euclidean clustering can be used to segment the point cloud after the ground point cloud filtering, resulting in segmented point clouds containing only car B and segmented point clouds containing only car C, thus obtaining two sets of segmented point clouds. Similarly, if the segmented point cloud includes more than two cars, then this step can produce more than two sets of segmented point clouds. Of course, in some cases, the number of segmented point clouds determined in this step may be 0. For example, there may be no target to be tracked in some non-reference frames, in which case the number of segmented point clouds determined in this step may be 0.

[0078] Step S170: Perform tracking calculations on the point cloud within the labeled bounding box to determine the predicted bounding box of the target to be tracked in the frame.

[0079] According to the embodiments of this application, the point cloud within the labeled bounding box of the reference frame can be used as the point cloud to be tracked, determining the position of the target to be tracked in each non-reference frame in a continuous series of frames. For example, the predicted bounding box of the target to be tracked in each non-reference frame can be determined. For example, the points contained in the labeled bounding box of the target to be tracked in the reference frame can be calculated, and point cloud data can be constructed as the point cloud to be tracked. Then, the point cloud to be tracked can be input into the Point Cloud Library (PCL) tracking algorithm module, and the tracking task for the continuous frames can be started. Optionally, the tracking of the point cloud to be tracked can be performed directly on the point cloud of each non-reference frame. For example, the point cloud of each non-reference frame can be input into the PCL tracking algorithm module, and the predicted bounding box of the target to be tracked in each frame can be output. Alternatively, the tracking of the point cloud to be tracked can be performed on the selected point cloud in each non-reference frame determined in step S150 above. For example, only the selected point cloud in each non-reference frame can be input into the PCL tracking algorithm module, and the predicted bounding box of the target to be tracked in the selected point cloud can be output. This approach can significantly reduce computational load, improve target tracking efficiency, and increase tracking accuracy.

[0080] According to the embodiments of this application, any existing or future suitable target tracking algorithm can be used to implement target tracking in this step. This includes, but is not limited to, target tracking methods based on the Hungarian algorithm, target tracking methods based on Kalman filtering, and target tracking methods based on correlation filtering.

[0081] Step S190: Based on the predicted bounding box, determine the target tracking point cloud of the target to be tracked in the segmented point cloud.

[0082] It is understandable that target tracking alone can achieve target tracking in consecutive frames, but target tracking algorithms often have special requirements for the tracking scene, especially since the accuracy of target tracking algorithms is poor when tracking objects in large scenes. According to the embodiments of this application, on the one hand, step S150 above involves target segmentation of the selected point cloud in each non-reference frame, determining the segmented point cloud of the object; on the other hand, step S170 also involves tracking the point cloud to be tracked within the labeled bounding box in the reference frame, determining the predicted bounding box of the target to be tracked in each non-reference frame. In other words, the target to be tracked in each non-reference frame is determined by both target segmentation and target tracking, resulting in two different results. In step S190, the results of target segmentation and target tracking are combined to determine the final target tracking point cloud. This scheme can both compensate for the accuracy problem of a single target tracking scheme and save computation to a certain extent, thus achieving a greater balance between target tracking accuracy and tracking efficiency.

[0083] According to embodiments of this application, a target tracking point cloud can be determined based on the relative positional relationship between the segmented point cloud determined by target segmentation in step S150 and the predicted bounding box. Specifically, the target tracking point cloud can be a group of the determined segmented point clouds. In other words, the segmented point clouds obtained from the target segmentation result can be filtered using the predicted bounding box obtained from the target tracking result to determine the final target tracking point cloud. For example, for each non-reference frame, a group of segmented point clouds that meets a preset standard can be filtered out as the target tracking point cloud based on the relative positional relationship between the predicted bounding box and each group of segmented point clouds. This relative positional relationship can be determined by various methods.

[0084] In one example, the relative positional relationship can be represented by the number of points falling within the predicted bounding box in each group of segmented point clouds, thereby determining the target tracking point cloud. For instance, if multiple groups of segmented point clouds are determined in step S150, the number of points falling within the predicted bounding box in each group can be determined separately, and the segmented point cloud with the largest number of points can be directly identified as the target tracking point cloud. Specifically, in this case, if only one group of segmented point clouds is determined in step S150, it can be directly identified as the target tracking point cloud. Alternatively, a number threshold can be set. If the number of points is greater than or equal to the number threshold, the group of segmented point clouds can be used as a candidate target tracking point cloud, and then the group with the largest number of points from the candidate target tracking point clouds can be identified as the target tracking point cloud. This number threshold can be set arbitrarily according to actual needs, for example, it can be set to 10, 20, 50, etc. In this example, if the number of points falling within the predicted bounding box in each group of segmented point clouds is less than the preset threshold, a prompt message such as "pending review" can be returned, allowing for further manual review to determine whether a target tracking point cloud exists in the non-reference frame.

[0085] In another example, the bounding boxes of each group of segmented point clouds can be determined first based on the target segmentation results of step S150. Then, the target tracking point cloud can be determined based on the relative positional relationship between the bounding boxes and the predicted bounding boxes of each group of segmented point clouds. For example, the area of ​​the intersection region between the bounding boxes and the predicted bounding boxes of each group of segmented point clouds can be calculated. The segmented point cloud with the largest intersection area can be determined as the target tracking point cloud. Alternatively, an area threshold can be set, and segmented point clouds with intersection areas greater than the area threshold can be used as candidate segmented point clouds, from which the segmented point cloud with the largest intersection area can be determined as the target tracking point cloud.

[0086] Of course, any other suitable method can be used to determine the relative positional relationship between the predicted bounding box and each group of segmented point clouds, thereby determining the target tracking point cloud.

[0087] For example, after determining the target tracking point cloud in each non-reference frame, the determined target tracking point cloud's position information can be output, thereby enabling target tracking annotation in consecutive frames of the point cloud. According to embodiments of this application, the position information of each target tracking point cloud can be determined as annotation data. For example, the position information of the target tracking point cloud can be represented by the position information of the minimum bounding box of the target tracking point cloud. Prior to this step, any suitable method can be used to determine the minimum bounding box of each target tracking point cloud. This minimum bounding box is, for example, the minimum directed bounding box.

[0088] According to the embodiments of this application, the target tracking method 100 in continuous point cloud frames described above can be implemented in any existing or future suitable point cloud basic processing platform. For example, a browser / server (B / S architecture) model can be adopted, providing HTTP API calls. The point cloud processing algorithms involved in each step (such as point cloud segmentation algorithms and target tracking algorithms) can be implemented through various related basic processing algorithms in PCL. The solution based on a mature point cloud processing library has a smaller computational load, higher processing efficiency, and stronger scalability.

[0089] According to the above scheme, the target to be tracked in non-reference frames outside the frame is automatically tracked based on the bounding box of the target in a frame of a continuous frame of the acquired 3D point cloud. On the one hand, the selected point cloud in each non-reference frame is determined based on the position parameters of the bounding box, and then the segmented point cloud is determined by segmenting the target in the selected point cloud. On the other hand, the predicted bounding box of the target tracking point cloud in each non-reference frame is determined by performing tracking calculations on the point cloud in the bounding box. Finally, the target tracking point cloud in the segmented point cloud is accurately determined based on the determined predicted bounding box. This scheme can significantly improve the tracking accuracy of targets in continuous frames of point cloud. At the same time, the execution logic of this scheme is relatively simple and the computational load is relatively small, so the tracking efficiency is also high. Moreover, this scheme has good applicability and can be applied to target tracking of continuous frames of 3D point cloud in various scenarios. In addition, as mentioned above, the automatic tracking of targets in continuous frames in the prior art mostly involves manually annotating the position of the target to be tracked in the specified preceding and following frames, and then using methods such as difference prediction or in-journey prediction to predict and annotate the position of the target to be tracked in the continuous frames. This approach requires manual annotation of targets in at least two frames, resulting in high labor costs. Furthermore, its applicability is limited, particularly in terms of accuracy when tracking targets in continuous point cloud frames within large scenes. In contrast, this embodiment only requires target annotation in one frame of a continuous sequence, and then uses a target tracking algorithm to determine the target's position in the non-reference frame. This saves on manual annotation costs and improves tracking accuracy to some extent.

[0090] For example, step S110, acquiring consecutive frames of the 3D point cloud of the target to be tracked, may include steps S111 and S112. Step S111 involves receiving the URL of the consecutive frames input by the user. Step S112 involves acquiring the consecutive frames using multithreading based on the URL.

[0091] As mentioned earlier, a B / S architecture can be adopted, using an HTTP API call to utilize the PCL system to perform target tracking in consecutive frames. For example, a user can send the URLs of the consecutive frames to be tracked to the server using a client. The server can then use multiple threads to simultaneously download each frame from the consecutive frames stored in the network storage based on the received URLs, and convert them into point cloud data that the PCL system can process. Furthermore, target tracking in the consecutive frames of the point cloud can then be performed within the PCL system.

[0092] This approach can significantly reduce the load on client computers, alleviating the cost and workload of system maintenance and upgrades. Therefore, it greatly simplifies user operation, improves the efficiency of continuous frame tracking processing, and provides a better user experience.

[0093] For example, the bounding box of the target to be tracked in the reference frame can be automatically annotated. Step S110, which involves obtaining the bounding box of the target in a frame of consecutive frames, may include steps S113 and S114.

[0094] Step S113: For the point cloud to be labeled in the first region of a frame in a continuous frame, ground point cloud filtering is performed on the point cloud to be labeled based on the density of points in different parts of the point cloud to be labeled, so as to obtain the first point cloud after removing the ground point cloud.

[0095] The first region can be of any size and shape. Optionally, the first region can be a preset fixed region. For example, the first region can be a region within a preset coordinate range around a preset calibration position in a reference frame. Alternatively, the first region can also be determined based on region parameters input by the user. For example, a human-computer interaction interface can be provided on the client side to display the 3D point cloud visualization window, in which the user can input the desired first region for annotation via a toolbar such as "Region Selection". Specifically, in an embodiment where the continuous frame of the point cloud is the point cloud of the surrounding environment of an automatically driving car A, the 3D point cloud of the surrounding environment of car A from a top-down perspective can be displayed in the human-computer interaction interface, and the user can select a rectangular region as the first region using a rectangle box such as one in the interface. For example, if the user wants to track car B around car A, they can roughly draw a rectangle around car B by controlling the left mouse button and input it into the system by clicking the "OK" button control in the interface. It can be understood that the point cloud to be annotated can be the part of the 3D point cloud that falls within the first region. For example, in the example above where the user defines a rectangular frame around car B as the first region, the point cloud to be labeled can include all points within the rectangular frame.

[0096] It can be understood that the first region is the approximate area to be labeled. The point cloud within the first region includes not only the target point cloud of the tracked object but also the ground point cloud surrounding the target. To accurately label the location of the tracked object, this step involves filtering the point cloud to be labeled using ground point cloud filtering to obtain the first point cloud after removing the ground point cloud filter.

[0097] According to embodiments of this application, the first region may include one target to be tracked, or it may include multiple targets to be tracked. If the user wishes to track multiple targets, the multiple targets can be labeled one by one, or the multiple targets can be labeled at once. Optionally, the user can input a first region including one target to be tracked through, for example, the display interface of a reference frame on a client. In step S113, the server can perform ground point cloud filtering on the point clouds to be labeled falling into the first region to obtain target point clouds corresponding to the target to be tracked. Alternatively, the user can also input a first region including multiple targets to be tracked. In step S113, the server can perform ground point cloud filtering on the point clouds to be labeled falling into the first region to obtain multiple target point clouds corresponding to each target to be tracked.

[0098] Typically, in the 3D point cloud acquired by a point cloud acquisition device, the density of points in the ground portion of the point cloud is relatively low, meaning the point distribution is sparse. Conversely, the density of points in the target point cloud corresponding to the target to be tracked is relatively high, meaning the point distribution is dense. Therefore, ground point cloud filtering can be performed on the point cloud to be labeled based on the point density of different parts of the point cloud. Any suitable filtering logic can be used, and any existing or future suitable point cloud filtering algorithm can be employed for ground point cloud filtering of the point cloud to be labeled.

[0099] By way of example, and not limitation, the point cloud to be labeled in the first region of the reference frame can first be divided into an appropriate number of point cloud parts based on the density of points in different parts of the point cloud to be labeled. For example, the point cloud to be labeled can be divided into a point cloud part with a higher point density and a point cloud part with a lower point density. The part with a higher point density can be called a sparse point cloud part with a sparser point distribution. The part with a higher point density can be called a dense point cloud part with a denser point distribution. Specifically, various suitable division criteria can be used for point cloud division. For example, the number of points in a preset neighborhood of each point in the point cloud to be labeled can be determined. If the number is not less than a first number threshold, the point can be regarded as a dense point; otherwise, if the number is less than the first number threshold, the point can be regarded as a sparse point. Furthermore, the set of dense points determined can be regarded as the dense point cloud part of the point cloud to be labeled, and the set of sparse points determined can be regarded as the sparse point cloud part of the point cloud to be labeled. After determining the point cloud parts with different densities, for example, the dense point cloud part and the sparse point cloud part are determined. The two point cloud portions can be filtered separately using appropriate filtering methods to remove ground point clouds from each portion. Finally, the two filtered point cloud portions are merged to obtain the first filtered point cloud with ground point clouds removed. Alternatively, only one point cloud portion can be filtered; for example, only the sparse point cloud portion can be filtered to obtain the filtered sparse point cloud portion. Then, the filtered sparse point cloud portion and the segmented dense point cloud portion can be merged, and the first filtered point cloud with ground point clouds removed can be determined based on the merged point cloud. In this step, various suitable filtering algorithms can be used to filter the segmented point cloud portions for ground point clouds. Exemplarily, but not limitingly, the filtering algorithm may include random sample consistency filtering algorithms, progressive morphological filtering algorithms, etc.

[0100] Step S114: Perform target segmentation on the filtered first point cloud to determine the target point cloud of the target to be tracked, and use the bounding box of the target point cloud as the annotation bounding box.

[0101] In this step, any existing or future point cloud segmentation algorithm can be used to segment the filtered point cloud. For example, random sample consensus segmentation, Euclidean clustering segmentation, and region growing segmentation algorithms can be used.

[0102] It is understandable that ground point cloud filtering of the point cloud located in the first region of the reference frame can remove the influence of interfering point clouds around the target. The filtered first point cloud can only include the target point cloud corresponding to the target to be tracked. For example, if the target to be tracked is car B, the filtered point cloud can include the point cloud corresponding to car B. As another example, if the targets to be tracked are car B and car C, the filtered point cloud can include the target point cloud corresponding to car B and the target point cloud corresponding to car C. In other examples, the filtered point cloud may include a small number of interfering point clouds, such as line point clouds, in addition to the target point cloud corresponding to the target to be tracked. In this example, by performing target segmentation on the filtered point cloud, the filtered point cloud can be divided into target point clouds and line point clouds corresponding to each target to be tracked. Exemplarily, methods such as random sampling consistency can be further used to filter out the line point clouds to obtain the target point clouds corresponding to each target to be tracked.

[0103] After determining the target point cloud for each target to be tracked, an appropriate point cloud bounding box determination method can be used to determine the bounding box of each target point cloud. For example, the minimum directed bounding box of each target point cloud can be determined, and this minimum directed bounding box can be used as the corresponding labeled bounding box of the target to be tracked in the reference frame.

[0104] According to the above scheme, the target point cloud and its bounding box for each target can be quickly and accurately determined by sequentially filtering the ground point cloud of the point cloud to be labeled in the first region of the reference frame and then segmenting the point cloud after removing the ground. Simultaneously, the ground point cloud filtering scheme based on the density of points in different parts of the point cloud to be labeled can accurately remove ground point clouds, improving the accuracy of the identified bounding boxes for the target point clouds. Furthermore, this target labeling scheme has low computational cost. Therefore, it can also improve the tracking efficiency and accuracy of targets in consecutive frames.

[0105] For example, ground point cloud filtering of the point cloud to be processed may include the following steps. The following description takes the point cloud to be processed as the point cloud to be labeled as an example. That is, ground point cloud filtering of the point cloud to be labeled in step S113 includes steps S113.1 to S113.4.

[0106] Step S113.1: Based on the point density of different parts of the point cloud to be labeled, the point cloud to be labeled is divided into a sparse point cloud portion and a dense point cloud portion. Any existing or future suitable method for dividing the point cloud to be labeled into sparse and dense point cloud portions can be used. For example, a radius filter can be used to filter the point cloud to be labeled, and the filtered point cloud can be considered as the dense point cloud portion. Then, the other point clouds outside the dense point cloud portion of the point cloud to be labeled can be identified as the sparse point cloud portion.

[0107] Step S113.2 involves performing progressive morphological filtering on the sparse point cloud portion to remove ground point clouds and obtain a filtering result. This filtering result includes the filtered sparse point cloud portion after removing ground point clouds. It is understood that since the distribution of points in the ground point cloud is typically sparse, the sparse point cloud portion obtained in step S113.1 can include most of the ground point clouds. Therefore, ground point cloud filtering can be performed on the sparse point cloud portion. Specifically, in this step, a progressive morphological filtering method can be used to remove ground point clouds from the sparse point cloud portion. Appropriate parameters such as an elevation threshold and a gradually increasing window size can be set, and based on these parameters, the progressive morphological filtering algorithm in the PCL library can be used to obtain the ground point clouds and non-ground point clouds in the sparse point cloud portion. Furthermore, by deleting the ground point clouds from the sparse point cloud portion, the filtered sparse point cloud portion after removing ground point clouds can be obtained.

[0108] Progressive morphological filtering is applicable to ground point cloud filtering in various scenarios and achieves good filtering results, removing most ground point clouds from sparse point cloud regions. However, this method is slow in processing dense point clouds. This step, by applying progressive morphological filtering to the sparse point cloud region, ensures both faster processing speed and better results.

[0109] Step S113.3: Merge the dense point cloud portion and the filtered sparse point cloud portion to obtain the first merged point cloud. Thus, all points in the dense point cloud portion and the filtered sparse point cloud portion together constitute the first merged point cloud.

[0110] Step S113.4: Determine the filtered point cloud based on the first merged point cloud. Optionally, the first merged point cloud can be directly used as the filtered point cloud. Alternatively, the first merged point cloud can be further processed. For example, various suitable methods can be used to perform secondary ground point cloud filtering on the first merged point cloud, and the point cloud after secondary ground point cloud filtering can be used as the filtered point cloud for subsequent target segmentation.

[0111] The above scheme achieves the goal of quickly removing most of the ground point cloud from the point cloud to be processed by dividing the point cloud into sparse and dense point cloud components and then filtering the sparse point cloud component using ground point cloud filtering. The ground point cloud filtering has high accuracy and ensures the integrity of the point cloud other than the ground point cloud in the point cloud to be processed. When the point cloud to be processed is an unannotated point cloud, it can improve the tracking efficiency and accuracy of targets in consecutive frames.

[0112] For example, step S130 determines the selected point cloud in the frame based at least on the position parameters of the bounding box of the target to be tracked, including steps S131 to S133.

[0113] Step S131: Based on the position parameters of the bounding box, determine the corresponding box in the frame and determine the movement direction of the target to be tracked.

[0114] The positional parameters of the bounding box can be its positional parameters within the reference frame. Taking the bounding box as a minimal directed bounding box (OBB) as an example, these positional parameters can include the center coordinates of the OBB, the unit vectors of the OBB in its three directions (length, width, and height), and the dimensions of the OBB in its three directions. Accordingly, a corresponding bounding box with the same positional parameters can be determined in each non-reference frame of a consecutive frame.

[0115] Furthermore, the movement direction of the target can be determined based on these positional parameters and the shape of the target. The movement direction can be determined in any suitable manner. For example, the movement direction can be determined based on the determined positional bounding box. Optionally, one of the three directions of the positional bounding box can be determined as the movement direction. Specifically, the movement direction can be determined based on the shape and motion characteristics of the target. For example, if the target is a car, since cars typically move along their length, and the length of the car is usually larger than its width and height, the length direction can be determined based on the positional parameters of the positional bounding box. Specifically, the direction with the larger dimension among the three-dimensional directions of the positional bounding box can be used as the length direction and determined as the movement direction. As another example, if the target is a human body, the thickness direction can be initially determined as the movement direction. Specifically, the direction with the smaller dimension among the three-dimensional directions of the positional bounding box can be used as the thickness direction and determined as the movement direction. Of course, in other examples, directions other than the three directions of the positional bounding box can also be determined as the movement direction.

[0116] Step S132: Expand the position-corresponding box at least along the movement direction so that the expanded bounding box is used as the selection bounding box.

[0117] It is understandable that within the short time frame of consecutive frame acquisition, the target to be tracked typically moves along its direction of movement. Therefore, after determining the bounding box of the target to be tracked in one frame of the consecutive frames, the position range of the target to be tracked in the consecutive frames can be roughly delineated based on the position of the bounding box and the direction of movement of the target. According to the embodiments of this application, this position range is the range in which the target to be tracked may be located in all consecutive frames as a whole, which is a large position range. This position range can be represented by an outer bounding box, i.e., a selected bounding box. It is understood that the position area of ​​each selected bounding box in each non-reference frame includes and is larger than the area of ​​the box corresponding to the position of the bounding box in that frame.

[0118] According to embodiments of this application, for each non-reference frame, various suitable expansion methods can be employed to expand the position-corresponding box at least along the movement direction, and the expanded larger area of ​​the position box is used as the selection bounding box. In one example, the position-corresponding box can be expanded by a factor of m along the movement direction to obtain the selection bounding box. m can be any suitable number, which can be set according to the movement speed of the target to be tracked and the acquisition frame rate of consecutive frames. Preferably, m is a number greater than or equal to 2. For example, if the movement direction is the length direction of the position-corresponding box, the position-corresponding box can be expanded by a factor of 2 along the length direction, that is, expanded by a factor of 1 in both the forward and backward directions to obtain the selection bounding box. It can be understood that in this example, the length of the selection bounding box is 3 times the length of the marked bounding box. In another example, the position-corresponding box can also be expanded simultaneously along the movement direction and other preset directions. Other preset directions are, for example, perpendicular to the movement direction. For example, if the movement direction is the length direction of the position-corresponding box, then the other preset directions can be the width direction, or the width and height directions. For example, the position-corresponding box can be expanded by a factor of m along the length direction and by a factor of n along the width direction, where n can be any suitable number. Alternatively, the position bounding box can be expanded by a factor of m along the length direction, a factor of n along the width direction, and a factor of p along the height direction, where p can be any suitable number. It is understood that m, n, and p can all be set according to the moving speed of the target being tracked and the frame rate of consecutive frames.

[0119] Step S133: The point cloud within the bounding box is determined as the selected point cloud in this frame.

[0120] For each non-reference frame, after determining the bounding box, the point cloud composed of points falling within the bounding box can be defined as the selected point cloud for that frame. Then, target segmentation can be performed on the selected point cloud to obtain a segmented point cloud. Finally, based on the target tracking results and the segmented point cloud, the target tracking point cloud is determined.

[0121] In the above scheme, the movement direction of the target to be tracked is determined based on the parameters of the bounding box. The corresponding bounding boxes in each non-reference frame are then expanded according to the movement direction to obtain the selected bounding box, thereby determining the selected point cloud. This scheme can quickly and accurately determine the approximate location of the target in the selected point cloud in each non-reference frame. This significantly reduces the computational load of target segmentation and helps to accurately segment the complete target point cloud. Therefore, it can significantly improve the tracking efficiency and processing accuracy of targets in consecutive frames of the point cloud.

[0122] For example, determining the movement direction of the target to be tracked in step S131 includes step S131.1, which determines the length direction of the bounding box as the movement direction, wherein the bounding box is a regularly shaped cuboid, and the length of the bounding box is greater than its width and height. Specifically, the bounding box can be the smallest directed rectangular 3D bounding box of the target to be tracked in the reference frame.

[0123] According to embodiments of this application, the length direction of the bounding box can be determined as the movement direction, and the length direction is greater than the width and height directions. Taking a car as an example, the movement direction of the car is usually its body length direction, and the dimension of the body length direction is usually greater than the dimensions of the car body width and height directions. The bounding box of the car is, for example, the minimum directed rectangular solid bounding box of the car. Therefore, by comparing the dimensions of the minimum directed rectangular solid bounding box in three directions, the length direction with the largest dimension can be determined as the movement direction.

[0124] This method can quickly and accurately determine the direction of movement, thereby enabling rapid and precise identification of the selected point cloud within the bounding box. This significantly improves the tracking efficiency and processing accuracy of targets in consecutive frames of the point cloud.

[0125] For example, step S132 expands the position-corresponding box at least along the movement direction, including step S132.1, expanding the position-corresponding box along its length and width directions respectively, so as to use the expanded bounding box as the selection bounding box. Wherein, the expansion factor of the selection bounding box in the length direction is greater than the expansion factor in the width direction.

[0126] Taking a car as an example, it's understandable that a car typically moves along its length in a short period, and may also change position laterally in its width direction, such as when turning. However, the magnitude of its movement along its length is greater than its movement along its width. Therefore, the bounding box can be expanded along both its length and width directions to obtain the selection bounding box. Furthermore, the factor by which the bounding box is expanded in the length direction can be set larger, while the factor by which it is expanded in the width direction can be set smaller.

[0127] The bounding box of a car is, for example, the smallest directed rectangular 3D bounding box of the car. In this step, the corresponding position box can be expanded by a factor of m along its length and by a factor of n along its width, and the expanded bounding box is used as the selection bounding box. Furthermore, m is greater than n. That is, the expansion factor of the corresponding position box along the aforementioned movement direction is greater than the expansion factor along other directions. m and n can be arbitrarily set according to actual needs. For example, and not limitingly, if m is 10 and n is 1, the corresponding position box can be expanded by a factor of 5 both forward and backward along the length of the car body, and by a factor of 0.5 both left and right along the width of the car body, resulting in the selection bounding box. It can be understood that in this example, the length of the selection bounding box is 11 times that of the bounding box, and the width is twice that of the bounding box.

[0128] The above scheme considers not only the positional changes of the target in the direction of movement but also its potential positional changes in the perpendicular direction. Based on the different magnitudes of movement in these two directions, the corresponding bounding boxes are expanded by different factors along the length and width directions to obtain the selected bounding boxes. This scheme further improves the accuracy of the determined bounding boxes for the target in consecutive frames, while also requiring less computation and exhibiting higher processing efficiency. Therefore, it can improve the tracking efficiency and accuracy of targets in consecutive frames. Furthermore, this scheme also has good scalability.

[0129] For example, step S150 performs target segmentation on the selected point cloud to determine the segmented point cloud, which includes steps S151 and S152. Step S151 involves filtering the selected point cloud using ground point cloud based on the density of points in different parts of the selected point cloud to obtain a filtered point cloud with ground point cloud removed. Step S152 involves performing target segmentation on the filtered point cloud to determine the segmented point cloud.

[0130] The implementation of step S151 is similar to that of step S113 described above, and the implementation of step S152 is similar to that of step S114 described above. Those skilled in the art can understand the various implementations of steps S151 and S152 by reading the aforementioned steps S113 and S114. For the sake of brevity, they will not be described in detail here.

[0131] According to the above scheme, the point cloud of the target to be tracked can be quickly, effectively, and accurately segmented in each non-reference frame of a continuous frame. This improves the efficiency and precision of target tracking in continuous point cloud frames.

[0132] For example, similar to step S113, step S151 performs ground point cloud filtering on the selected point cloud, including steps S151.1 to S151.4. Step S151.1: Based on the point density of different parts of the selected point cloud, divide the selected point cloud into a sparse point cloud portion and a dense point cloud portion. Step S151.2: Perform progressive morphological filtering on the sparse point cloud portion to remove ground point clouds from the sparse point cloud portion and obtain a filtering result. The filtering result includes the filtered sparse point cloud portion after removing ground point clouds from the sparse point cloud portion. Step S151.3: Merge the dense point cloud portion and the filtered sparse point cloud portion to obtain a second merged point cloud. Step S151.4: Determine the filtered point cloud based on the second merged point cloud.

[0133] The implementation of steps S151.1 to S151.4 is similar to that of steps S113.1 to S113.4 mentioned above, and will not be repeated here for the sake of brevity.

[0134] The above approach significantly improves the processing efficiency and accuracy of ground point cloud filtering for selected point clouds, resulting in a better user experience. Consequently, it can significantly improve the tracking efficiency and accuracy of targets in consecutive frames.

[0135] For example, step S170, which involves tracking the point cloud within the labeled bounding box, includes step S171. Step S171 involves performing tracking calculations on the point cloud within the labeled bounding box within the selected point cloud area to determine a predicted bounding box in the selected point cloud.

[0136] According to the embodiments of this application, the point cloud within the labeled bounding box can be used as the target point cloud to be tracked. Target tracking calculations are performed on the selected point cloud in each non-reference frame within the determined consecutive frames to determine the predicted bounding box of the target to be tracked in the selected point cloud. For example, tracking can be implemented based on the tracking algorithm module in the PCL system. After obtaining the labeled bounding box of the target to be tracked in the reference frame in step S110, the points contained in the labeled bounding box in the reference frame can be calculated to construct point cloud data as the point cloud to be tracked. Then, the point cloud to be tracked is input into the PCL tracking algorithm module to start the tracking task of the target to be tracked in the consecutive frames. Specifically, for each non-reference frame, the determined selected point cloud can be input into the PCL tracking algorithm module, and the module outputs the predicted bounding box of the target to be tracked in that non-reference frame. Furthermore, based on the relative positional relationship between the determined predicted bounding box and the segmented point cloud determined in step S150, the target tracking point cloud of the target to be tracked in that non-reference frame can be finally determined.

[0137] The above scheme can significantly reduce the computational load of target tracking while ensuring the accuracy of target tracking, thereby significantly improving tracking efficiency while maintaining prediction accuracy.

[0138] For example, step S190 determines the target point cloud of the target to be tracked in the segmented point cloud based on the predicted bounding box, including steps S191 and S192.

[0139] Step S191: Based on the predicted bounding box, determine the number of points in each segmented point cloud that fall within the predicted bounding box. Step S192: Determine the segmented point cloud with the largest number of points as the target tracking point cloud.

[0140] As mentioned earlier, the relative positional relationship between two points can be represented by the number of points in each segmented point cloud that fall within the predicted bounding box. It can be understood that the more points in a segmented point cloud fall within the predicted bounding box, the greater the likelihood that the segmented point cloud is the target tracking point cloud. Therefore, the segmented point cloud with the largest number of points can be identified as the target point cloud. The method of counting the points in the point cloud is computationally less complex and more accurate.

[0141] The following explanation uses a car as an example to illustrate the target to be tracked. Figure 2a shows a schematic diagram of the point cloud target segmentation result of the selected point cloud according to an embodiment of this application. Figure 2b shows a schematic diagram of the tracking operation result of the target to be tracked according to an embodiment of this application. Figure 2c shows a superimposed schematic diagram of Figures 2a and 2b. As shown in Figure 2a, by segmenting the selected point cloud in the figure in step S130, two sets of segmented point clouds can be obtained. One set is the point cloud located in the small bounding box on the left, which can be seen to be a local point cloud of a car. The other set is the point cloud located in the large bounding box on the right. This point cloud can be the overall point cloud of another car. Figure 2b can be regarded as the predicted bounding box of the target to be tracked (shown by the black bounding box in the figure) obtained by performing tracking operation on the selected point cloud of Figure 2a. Furthermore, in Figure 2c, the final target tracking point cloud can be determined according to the relative positional relationship between each set of segmented point clouds and the predicted bounding box. As shown in Figure 2c, the car point cloud in the large bounding box on the right intersects with the predicted bounding box of the target to be tracked, and a large number of its points fall within the predicted bounding box. However, the car point cloud in the small bounding box on the left does not intersect with the predicted bounding box of the target to be tracked, and zero of its points fall within the predicted bounding box. Therefore, in step S192, the car point cloud in the large bounding box on the right can be identified as the target tracking point cloud.

[0142] This approach can accurately determine the target tracking point cloud while greatly reducing the amount of computation, thereby significantly improving the tracking efficiency and accuracy of targets in consecutive frames and providing a better user experience.

[0143] Figure 3 shows a schematic flowchart of a target tracking method in a continuous point cloud frame according to another embodiment of this application. The continuous point cloud frame is, for example, a continuous point cloud frame of the external environment of car A in autonomous driving. Exemplarily, a B / S architecture can be adopted, and an HTTP API call can be provided to use a point cloud processing algorithm in PCL to track one or more target vehicles in the continuous point cloud frame. As shown, upon receiving an HTTP request, input parameters for continuous frame tracking can be constructed in response to operations on the continuous frame interface displayed on the client.

[0144] According to embodiments of this application, the input parameters include the position parameters of the bounding box of the target to be tracked in one frame of a continuous frame. For example, a user can select one frame in the continuous frame interface displayed on the client as a reference frame. A rectangular area can be selected using operable controls in the display interface of that frame. The server can then determine the bounding box of the target to be tracked based on the rectangular area input by the user. Exemplarily, but not limitingly, the point cloud falling within the rectangular area in the reference frame can be calculated based on the position of the rectangular area, and PCL point cloud data can be constructed as the point cloud to be tracked. The point cloud to be tracked can be downsampled first, and the number of points in the downsampled point cloud can be calculated. If the number is less than a threshold, the random sampling consistency filtering method in PCL can be directly used to remove the ground point cloud, obtaining the filtered point cloud to be labeled. If the number is not less than the threshold, the point cloud to be labeled can be divided into a dense point cloud portion and a sparse point cloud portion using, for example, a radius filter. A progressive morphological filtering method is applied to the sparse point cloud portion to remove ground point clouds, resulting in a filtered sparse point cloud. Simultaneously, the height range of the ground point clouds can be determined. The filtered sparse and dense point cloud portions are then merged to obtain a merged point cloud. Based on the determined height range of the ground point clouds, the remaining ground point clouds in the merged point cloud are filtered out, resulting in a filtered first point cloud. Euclidean clustering is then used to segment the filtered first point cloud to identify car point clouds within the point cloud to be labeled. Furthermore, the bounding boxes of these car point clouds can be determined, such as the minimum directed bounding box (e.g., the minimum rectangular bounding box), serving as the labeled bounding boxes. Of course, the target to be tracked can also be multiple cars; in this case, the labeled bounding box for each car can be determined.

[0145] According to embodiments of this application, the input parameters may further include point cloud continuous frame URLs. Users can select the continuous frames they wish to track using a selection control, for example. The client can send the user-selected continuous frame URLs to the server. After receiving the continuous frame URLs, the server can retrieve the continuous frame files from network storage using multiple threads, load them into memory, and create a point cloud storage data structure.

[0146] After determining the input parameters for continuous frame tracking, each target in the continuous frames can be tracked based on these parameters. For the bounding box of each target, the points within the bounding box in the reference frame can be calculated to construct point cloud data, which serves as the target point cloud and is input into the PCL tracking algorithm module. Then, a multi-threaded tracking task for all target point clouds in the continuous frames can be initiated.

[0147] Specifically, for any non-reference frame in a series of frames, the corresponding bounding box of the labeled bounding box can be determined within that frame. Then, based on the corresponding bounding box, the body length direction of each target (vehicle) to be tracked can be determined. The direction of the larger of the x-axis and y-axis dimensions of the corresponding bounding box can be set as the body length direction. Then, the corresponding bounding box can be enlarged by a factor of r (r > 1) along the body length direction and by a factor of 1 in the other directions to obtain a selected bounding box. The point cloud falling within this selected bounding box is used as the selected point cloud. Target segmentation is performed on the selected point cloud to obtain the segmentation result. If the segmentation result is empty, for example, if no car point cloud is segmented, an invalid bounding box with a size of 0 can be output. If the segmentation result is not empty, the selected point cloud can be further input into the tracking module to obtain the predicted bounding box of the target to be tracked in the selected point cloud. For cases where the segmentation result is greater than 1 (the number of segments in the point cloud is greater than 1), the number of points in each segmented point cloud that fall within the predicted bounding box can be calculated. The segmented point cloud with the most points falling within the predicted bounding box is then identified as the point cloud of the target to be tracked, and the position parameters of the minimum directed bounding box for that point cloud can be output. For cases where the segmentation result is equal to 1, the segmented point cloud can be directly identified as the point cloud of the target to be tracked, and the minimum directed bounding box for that point cloud can be output.

[0148] Traverse the continuous frames and repeat the target tracking steps above. Once the minimum directed bounding box of all targets to be tracked in each non-reference frame is determined, that is, after all asynchronous tasks are completed, assemble the results of all tasks, return the requested result, and complete the target tracking task in the continuous point cloud frames.

[0149] According to another aspect of this application, a target tracking system in continuous frames of a point cloud is also provided. Figure 4 shows a schematic block diagram of a target tracking system 400 in continuous frames of a point cloud according to an embodiment of this application. Exemplarily, the system 400 includes an acquisition module 410, a selection module 420, a segmentation module 430, a tracking calculation module 440, and a determination module 450.

[0150] The acquisition module 410 is used to acquire consecutive frames of the 3D point cloud of the target to be tracked and the bounding box of one frame of the consecutive frames of the target to be tracked.

[0151] The selection module 420 is used to determine the selection point cloud in each frame of a continuous series of frames, excluding those with labeled bounding boxes, based at least on the position parameters of the labeled bounding boxes of the target to be tracked. The point cloud within the bounding box corresponding to the position of the labeled bounding box in that frame is a part of the selection point cloud.

[0152] The segmentation module 430 is used to segment the selected point cloud to determine the segmented point cloud.

[0153] The tracking calculation module 440 is used to perform tracking calculations on the point cloud in the labeled bounding box to determine the predicted bounding box of the target to be tracked in the frame.

[0154] The determination module 450 is used to determine the target tracking point cloud of the target to be tracked in the segmented point cloud based on the predicted bounding box.

[0155] This target tracking system in continuous point cloud frames is used to implement the aforementioned target tracking method 100 in continuous point cloud frames. Those skilled in the art can easily understand the implementation structure, working principle, and beneficial effects of the target tracking system in continuous point cloud frames by reading the above description of the target tracking method 100 in continuous point cloud frames. For the sake of brevity, further details are omitted here.

[0156] According to another aspect of this application, an electronic device is also provided. FIG5 shows a schematic block diagram of an electronic device 500 according to an embodiment of this application. The electronic device 500 includes a processor 510 and a memory 520, wherein the memory 520 stores computer program instructions, which are executed by the processor 510 to perform the target tracking method 100 in consecutive point cloud frames described above.

[0157] According to another aspect of this application, a storage medium is also provided, on which program instructions are stored, which are used to execute the target tracking method 100 in the continuous frames of the point cloud described above when the program instructions are run.

[0158] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0160] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0161] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0162] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in a target tracking system in continuous frames of point clouds according to embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0163] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0164] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A method for target tracking in continuous frames of a point cloud, characterized in that, include: Acquire consecutive frames of the 3D point cloud of the target to be tracked, as well as the bounding box of the target in one frame of the consecutive frames; For each frame in the continuous frames other than those marked with the labeled bounding box, at least based on the position parameters of the labeled bounding box of the target to be tracked, the selected point cloud in that frame is determined, wherein the point cloud within the position corresponding box of the labeled bounding box in that frame is a part of the selected point cloud; target segmentation is performed on the selected point cloud to determine the segmented point cloud; tracking operations are performed on the point cloud within the labeled bounding box to determine the predicted bounding box of the target to be tracked in that frame; and based on the predicted bounding box, the target tracking point cloud of the target to be tracked in the segmented point cloud is determined, wherein determining the selected point cloud in that frame at least based on the position parameters of the labeled bounding box of the target to be tracked includes: determining the position corresponding box in that frame based on the position parameters of the labeled bounding box; expanding the position corresponding box at least along a preset direction to use the expanded bounding box as the selected bounding box; and determining the point cloud within the selected bounding box as the selected point cloud in that frame.

2. The target tracking method in continuous frames of a point cloud as described in claim 1, characterized in that, The step of tracking the point cloud within the labeled bounding box includes: performing tracking calculations on the point cloud within the selected point cloud to determine the predicted bounding box in the selected point cloud.

3. The target tracking method in continuous frames of a point cloud as described in claim 1, characterized in that, The step of determining the selected point cloud in the frame based at least on the position parameters of the bounding box of the target to be tracked further includes: determining the movement direction of the target to be tracked, wherein the preset direction includes the movement direction.

4. The target tracking method in continuous frames of a point cloud as described in claim 3, characterized in that, Determining the movement direction of the target to be tracked includes: determining the length direction of the bounding box as the movement direction, wherein the bounding box is a cuboid, and the length of the bounding box is greater than the width and height of the bounding box.

5. The target tracking method in continuous frames of a point cloud as described in claim 4, characterized in that, The step of expanding the position corresponding box at least along a preset direction includes: expanding the position corresponding box along the length direction and the width direction respectively, so as to use the expanded bounding box as the selection bounding box, wherein the multiple by which the selection bounding box is expanded in the length direction is greater than the multiple by which it is expanded in the width direction.

6. The target tracking method in continuous frames of point clouds as described in any one of claims 1 to 5, characterized in that, The step of determining the target tracking point cloud of the target to be tracked in the segmented point cloud based on the predicted bounding box includes: determining the number of points in each segmented point cloud that fall within the predicted bounding box based on the predicted bounding box; and determining the segmented point cloud with the most determined points as the target tracking point cloud.

7. The target tracking method in continuous frames of point clouds as described in any one of claims 1 to 5, characterized in that, The process of acquiring continuous frames of the 3D point cloud of the target to be tracked includes: receiving the URL of the continuous frames input by the user; and acquiring the continuous frames using multi-threading based on the URL.

8. The target tracking method in continuous frames of a point cloud as described in any one of claims 1 to 5, characterized in that, Obtaining the bounding box of the target to be tracked in a frame of the continuous frames includes: for the point cloud to be labeled in a first region of the frame of the continuous frames, performing ground point cloud filtering on the point cloud to be labeled based on the density of points in different parts of the point cloud to obtain a first point cloud after removing ground point cloud; performing target segmentation on the filtered first point cloud to determine the target point cloud of the target to be tracked, and using the bounding box of the target point cloud as the bounding box; and / or performing target segmentation on the selected point cloud to determine the segmented point cloud includes: performing ground point cloud filtering on the selected point cloud based on the density of points in different parts of the selected point cloud to obtain a second point cloud after removing ground point cloud; performing target segmentation on the filtered second point cloud to determine the segmented point cloud.

9. The target tracking method in continuous frames of a point cloud as described in claim 8, characterized in that, Ground point cloud filtering of the point cloud to be processed includes: dividing the point cloud to be processed into a sparse point cloud portion and a dense point cloud portion based on the point density of different parts of the point cloud to be processed; performing progressive morphological filtering on the sparse point cloud portion to remove ground point clouds in the sparse point cloud portion and obtaining a filtering result, wherein the filtering result includes a filtered sparse point cloud portion after removing ground point clouds from the sparse point cloud portion; merging the dense point cloud portion and the filtered sparse point cloud portion to obtain a merged point cloud; and determining a filtered point cloud based on the merged point cloud; wherein the point cloud to be processed includes the point cloud to be labeled and / or the selected point cloud, and the filtered point cloud includes the first point cloud and / or the second point cloud.

10. A target tracking system in continuous frames of a point cloud, comprising: The acquisition module is used to acquire consecutive frames of the 3D point cloud of the target to be tracked, as well as the bounding box of the target to be tracked in one frame of the consecutive frames. The selection module is used to determine the selection point cloud in each frame of the continuous frames, excluding frames marked with the marked bounding box, based at least on the position parameters of the marked bounding box of the target to be tracked, wherein the point cloud within the position corresponding box of the marked bounding box in the frame is a part of the selection point cloud; the segmentation module is used to perform region segmentation on the selection point cloud to determine segmented point clouds, wherein each segmented point cloud includes a target; the tracking calculation module is used to perform tracking calculation on the point cloud in the marked bounding box to determine the predicted bounding box of the target to be tracked in the frame; and the determination module is used to determine the target tracking point cloud of the target to be tracked in the segmented point cloud based on the predicted bounding box, wherein determining the selection point cloud in the frame based at least on the position parameters of the marked bounding box of the target to be tracked includes: determining the position corresponding box in the frame based on the position parameters of the marked bounding box; expanding the position corresponding box at least along a preset direction to use the expanded bounding box as the selection bounding box; and determining the point cloud within the selection bounding box as the selection point cloud in the frame.

11. An electronic device comprising a processor and a memory, wherein, The memory stores computer program instructions, which, when executed by the processor, are used to perform the target tracking method in consecutive point cloud frames as described in any one of claims 1 to 9.

12. A computer-readable storage medium having program instructions stored thereon, the program instructions being configured to perform the target tracking method in consecutive frames of a point cloud as described in any one of claims 1 to 9 when executed.

Images