Target tracking method and device, vehicle, chip and storage medium

Abstract

The present disclosure provides a target tracking method and device, a vehicle, a chip and a storage medium. The method comprises: acquiring a collection image of an environment in which a target vehicle is located collected by a vehicle-mounted camera at a target time, and performing target detection on the collection image to obtain at least one detection region; acquiring a point cloud data set of the environment collected by a vehicle-mounted radar at the target time; and performing a clustering process according to the at least one detection region and the point cloud data set to extract a target data set obtained by clustering from the point cloud data set; wherein the target data set is used for target tracking. Thus, by combining the image detection result of the vehicle-mounted camera with the point cloud data set of the vehicle-mounted radar and performing a clustering process based on the detection region, the target data set used for target tracking is extracted, which significantly improves the accuracy of target tracking, enhances the robustness of target tracking, and thus effectively improves the safety and reliability of an autonomous driving system.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to a target tracking method, device, vehicle, chip, and storage medium. Background Technology

[0002] With the continuous advancement of computer vision, artificial intelligence, and sensor technologies, autonomous driving systems need to achieve continuous and accurate tracking of multiple targets in the surrounding environment to obtain crucial information such as their position, speed, and direction. This information is essential for the decision-making, planning, and control of autonomous driving systems, helping vehicles to make safe and reasonable driving behaviors in complex and ever-changing road environments. Therefore, accurately achieving target tracking is of paramount importance. Summary of the Invention

[0003] This disclosure aims to address, to some extent, one of the technical problems in the related technologies.

[0004] To this end, this disclosure proposes a target tracking method, device, vehicle, chip, and storage medium. By combining the image detection results of the vehicle-mounted camera with the point cloud dataset of the vehicle-mounted radar and performing a clustering process based on the detection region, a target dataset for target tracking is extracted. This significantly improves the accuracy and robustness of target tracking, thereby effectively enhancing the safety and reliability of the autonomous driving system.

[0005] One embodiment of this disclosure proposes a target tracking method, comprising: acquiring an image of the environment in which a target vehicle is located, captured by an onboard camera at a target time, and performing target detection on the captured image to obtain at least one detection region; acquiring a point cloud dataset of the environment captured by an onboard radar at the target time; and performing a clustering process based on the at least one detection region and the point cloud dataset to extract a clustered target dataset from the point cloud dataset; wherein the target dataset is used for target tracking.

[0006] Another embodiment of this disclosure proposes a target tracking device, comprising: a first acquisition module, configured to acquire an image of the environment in which a target vehicle is located, captured by an onboard camera at a target time, and perform target detection on the acquired image to obtain at least one detection region; a second acquisition module, configured to acquire a point cloud dataset of the environment captured by an onboard radar at the target time; and a processing module, configured to perform a clustering process based on the at least one detection region and the point cloud dataset to extract a clustered target dataset from the point cloud dataset; wherein the target dataset is used for target tracking.

[0007] Another embodiment of this disclosure provides a vehicle including: a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method as described in one embodiment.

[0008] Another embodiment of this disclosure provides a chip including processing circuitry configured to perform the methods described in the above-described embodiment.

[0009] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in the above-described aspect of the embodiment.

[0010] Another embodiment of this disclosure provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the method described in the foregoing embodiment.

[0011] The target tracking method proposed in this disclosure acquires images of the environment in which the target vehicle is located, captured by an onboard camera at a target time, and performs target detection on the acquired images to obtain at least one detection region; acquires a point cloud dataset of the environment captured by an onboard radar at the target time; and performs a clustering process based on the at least one detection region and the point cloud dataset to extract the clustered target dataset from the point cloud dataset. The target dataset is used for target tracking. Thus, by combining the image detection results from the onboard camera with the point cloud dataset from the onboard radar and performing a clustering process based on the detection region, a target dataset for target tracking is extracted, significantly improving the accuracy and robustness of target tracking, thereby effectively enhancing the safety and reliability of the autonomous driving system.

[0012] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0013] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0014] Figure 1 This is a schematic flowchart of a target tracking method provided in an embodiment of the present disclosure;

[0015] Figure 2 This is a flowchart illustrating another target tracking method provided in an embodiment of the present disclosure;

[0016] Figure 3This is a schematic diagram illustrating the matching of point cloud data and detection areas provided in an embodiment of this disclosure;

[0017] Figure 4 This is a schematic diagram of point cloud clustering provided in an embodiment of the present disclosure;

[0018] Figure 5 This is a flowchart illustrating another target tracking method provided in an embodiment of the present disclosure;

[0019] Figure 6 This is a schematic diagram illustrating the principle of a target tracking method provided in an embodiment of this disclosure;

[0020] Figure 7 This is a schematic diagram of point cloud clustering association provided in an embodiment of the present disclosure;

[0021] Figure 8 This is a schematic diagram of a point cloud clustering secondary association provided in an embodiment of the present disclosure;

[0022] Figure 9 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of the present disclosure;

[0023] Figure 10 This is a block diagram illustrating a vehicle according to an exemplary embodiment;

[0024] Figure 11 This is a schematic diagram of the chip structure proposed in the embodiments of this disclosure. Detailed Implementation

[0025] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0026] In practical applications, intelligent vehicles typically have multiple sensors such as millimeter-wave radar and cameras. Millimeter-wave radar generates millimeter-wave point cloud information related to the environment by sending and receiving millimeter-wave signals. Currently, intelligent driving systems obtain information such as the position and speed of targets in the environment (such as vehicles and pedestrians) by clustering these point cloud information, thereby enabling the detection and tracking of targets in the environment around the vehicle and providing accurate environmental perception capabilities for the intelligent driving system.

[0027] Because millimeter-wave radar can only provide a limited number of point clouds, the point clouds generated by target reflections cannot fully reflect the geometric and positional characteristics of the target. For example, a large vehicle on the side may only have point cloud distribution at the front and rear. This problem makes it difficult for traditional point cloud clustering algorithms to completely cluster the point clouds corresponding to the target, affecting the subsequent processing of millimeter-wave radar perception, interfering with the overall operational stability of the intelligent driving system and the user's driving experience.

[0028] Therefore, in related technologies, the overall environmental perception capability of vehicles is improved by combining millimeter-wave point cloud information and visual information. Currently, the methods of combining millimeter-wave point cloud information and visual information are mainly divided into two types: feature-level fusion and target-level fusion. Feature-level fusion refers to extracting multi-dimensional features related to the environment from millimeter-wave point cloud information and visual information respectively through neural networks, and simultaneously inputting the two types of features into the target detection neural network to obtain the corresponding detection results. Target-level fusion first detects targets in the environment from point cloud information and visual information respectively, and then fuses the target detection results obtained by the two methods. However, neither of the above two methods utilizes the visual detection results to optimize millimeter-wave point cloud clustering, resulting in limited performance of existing millimeter-wave point cloud-based target detection.

[0029] To address the aforementioned issues, this disclosure proposes a target tracking method, apparatus, vehicle, chip, and storage medium.

[0030] The target tracking method, apparatus, vehicle, chip, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0031] Figure 1 This is a schematic flowchart of a target tracking method provided in an embodiment of the present disclosure.

[0032] This embodiment illustrates the example of a target tracking method configured in a target tracking device, which can be applied to autonomous vehicles.

[0033] like Figure 1 As shown, the target tracking method may include the following steps:

[0034] Step 101: Acquire the image of the environment of the target vehicle captured by the vehicle-mounted camera at the target time, and perform target detection on the captured image to obtain at least one detection area.

[0035] In order to achieve perception and control of the surrounding environment, in this embodiment of the disclosure, the vehicle-mounted camera can capture images of the environment in which the target vehicle is located in real time, thereby acquiring the captured images of the environment in which the target vehicle is located at a specified time (e.g., the target time) by the vehicle-mounted camera. Then, target detection is performed on the images captured at the target time to obtain at least one detection box, and at least one detection region is determined based on the at least one detection box, wherein each detection region includes one detection box.

[0036] Step 102: Obtain the point cloud dataset of the environment collected by the vehicle-mounted radar at the target time.

[0037] To achieve three-dimensional perception of the environment surrounding a vehicle, onboard radar (such as LiDAR or millimeter-wave radar) scans the environment around the vehicle in real time, generating a point cloud dataset composed of discrete points. This allows the acquisition of a point cloud dataset of the environment collected by the onboard radar at the target time. Each point in the point cloud dataset represents information reflected back after the radar beam encounters an object in a certain direction.

[0038] It should be noted that the execution order of steps 101 and 102 is not limited in this embodiment. Figure 1 The example only shows that step 102 is executed after step 101. In actual application, steps 101 and 102 can be executed simultaneously, or step 102 can be executed before step 101.

[0039] Step 103: Based on at least one detection region and a point cloud dataset, perform a clustering process to extract the clustered target dataset from the point cloud dataset; wherein the target dataset is used for target tracking.

[0040] To improve the accuracy of target tracking, in this embodiment, the image detection results are combined with the point cloud dataset of the vehicle-mounted radar, and a clustering process based on the detection region is performed to extract the target dataset for target tracking. It should be noted that, to further improve the accuracy of target tracking, the point cloud data needs to be preprocessed before combining the image detection results with the point cloud dataset. This preprocessing includes noise reduction, filtering, and sampling of the point cloud data.

[0041] The target tracking method of this disclosure acquires an image of the environment in which the target vehicle is located, captured by an onboard camera at a target time, and performs target detection on the captured image to obtain at least one detection region; acquires a point cloud dataset of the environment captured by an onboard radar at the target time; and performs a clustering process based on the at least one detection region and the point cloud dataset to extract the clustered target dataset from the point cloud dataset. The target dataset is used for target tracking. Thus, by combining the image detection results from the onboard camera with the point cloud dataset from the onboard radar and performing a clustering process based on the detection region, a target dataset for target tracking is extracted, significantly improving the accuracy and robustness of target tracking, thereby effectively enhancing the safety and reliability of the autonomous driving system.

[0042] To clearly illustrate how the clustering process is implemented in the above embodiments, this disclosure proposes another target tracking method.

[0043] Figure 2 This is a schematic flowchart of another target tracking method provided in an embodiment of the present disclosure.

[0044] like Figure 2 As shown, the target tracking method may include the following steps:

[0045] Step 201: Acquire the image of the environment of the target vehicle captured by the vehicle-mounted camera at the target time, and perform target detection on the captured image to obtain at least one detection area.

[0046] Step 202: Obtain the point cloud dataset of the environment collected by the vehicle-mounted radar at the target time.

[0047] Step 203: For any detection region, determine whether there exists at least one first intermediate dataset located within any detection region in the point cloud dataset.

[0048] It should be understood that, since vehicles may undergo dynamic changes such as movement and turning during driving, the target position of the detection result of the acquired image may not be completely consistent with the actual position of the radar point cloud. Therefore, in order to better adapt to the above dynamic changes and improve the accuracy of matching, in this embodiment of the disclosure, it is determined whether there is point cloud data in the point cloud dataset that matches any detection area; wherein, the detection area is the area after the detection box is expanded.

[0049] As an example, for any detection region, determine whether there exists a point cloud dataset in the point cloud data that matches any detection region, that is, whether there exists at least one first intermediate dataset located in any detection region.

[0050] For example, such as Figure 3 As shown, Figure 3The left-hand figure shows that there exists a point cloud dataset that matches the corresponding detection region, meaning there is a first intermediate dataset located within that detection region. Figure 3 The figure on the right shows that there is no point cloud dataset in the point cloud dataset that successfully matches the corresponding detection region, that is, there is no first intermediate dataset located within the detection region.

[0051] Step 204: If yes, cluster the point cloud dataset based on at least one first intermediate dataset to obtain at least one first cluster.

[0052] The first cluster includes a first intermediate dataset.

[0053] To improve the completeness and accuracy of target tracking, as a possible implementation, for any first intermediate dataset, based on the attribute information (such as spatial location and velocity attributes) of each point cloud in the point cloud dataset other than the first intermediate dataset, it is determined whether the point clouds are adjacent, and all adjacent points are treated as a cluster, i.e., the first cluster, where the first cluster includes the corresponding first intermediate dataset.

[0054] For example, such as Figure 4 As shown, Figure 4 The data includes adjacent points and non-adjacent points in the point cloud. The adjacent points in the point cloud are considered as a cluster, i.e., the first cluster.

[0055] Step 205: Based on the size information of the detection frame in any detection area, determine whether there is a second cluster to be merged in at least one first cluster.

[0056] To improve the completeness and accuracy of target tracking, in this embodiment of the disclosure, point cloud data of the same target in the image detection results are merged.

[0057] In this embodiment of the disclosure, based on the size information of the detection box within any detection area, it is determined whether there is a correlation between the first clusters, that is, whether they are point cloud data of the same target.

[0058] As an example, a reference cluster is determined from at least one first cluster based on the location information of the target vehicle at the target time; wherein the distance between the reference cluster and the target vehicle is less than a threshold; for any cluster in the first cluster other than the reference cluster, the target distance between the center of the any cluster and the center of the reference cluster is determined; based on the comparison result of the target distance and the size information of the detection box in any detection area, it is determined whether any cluster is a second cluster to be merged.

[0059] In other words, for any detection area, the cluster closest to the target vehicle's location is determined from at least one first cluster corresponding to that detection area, i.e., the reference cluster. For other clusters in the first cluster besides the reference cluster, the target distance between the center of the other cluster and the center of the reference cluster is determined. This target distance is greater than the diagonal length of the detection frame within any detection area (e.g., the detection frame has a length of a, a width of b, and a diagonal length of b). When the target cluster is less than or equal to the diagonal length of the detection frame within any detection area, the other cluster is determined to be associated with the reference cluster, meaning the other cluster is the second cluster to be merged.

[0060] Step 206: If yes, merge the second clusters to be merged to obtain the first target dataset.

[0061] Then, the second cluster to be merged is merged to obtain the first target dataset, that is, the clustering results associated with the same target are merged, and the merged point cloud dataset is used as different components of the same target, such as the front and rear of a large vehicle.

[0062] As one possible implementation, for at least one detection region other than any other detection region, it is determined whether there is at least one third intermediate dataset located within the other detection regions in the second intermediate dataset; wherein, the second intermediate dataset includes point cloud data in the point cloud dataset other than the first target dataset, or includes point cloud data in the point cloud dataset other than the first cluster, or includes point cloud data in the point cloud dataset other than the first intermediate dataset; if so, the second intermediate dataset is clustered based on at least one third intermediate dataset to obtain at least one third cluster; wherein, the third cluster includes one third intermediate dataset; based on the size information of the detection boxes in the other detection regions, it is determined whether there is a fourth cluster to be merged in the at least one third cluster; if so, the fourth cluster to be merged is merged to obtain the second target dataset.

[0063] In other words, point cloud data that fails to match the corresponding detection area, is not adjacent, or is not successfully associated is subjected to secondary clustering and association. For example, when the first cluster includes the second cluster to be merged, the second intermediate dataset includes point cloud data in the point cloud dataset excluding the first target dataset (i.e., unmatched point cloud data, non-adjacent point cloud data, and unassociated point cloud data). Similarly, when the first cluster (a cluster formed by adjacent point clouds) does not include the second cluster to be merged (i.e., point cloud clusters are not associated), the second intermediate dataset includes point cloud data in the point cloud dataset excluding the first cluster. Furthermore, when there is no point cloud data adjacent to the matched point cloud data, the second intermediate dataset includes point cloud data in the point cloud dataset excluding the first intermediate dataset (matched point cloud data). Then, the second intermediate dataset is used to determine... The second intermediate dataset is used to determine whether it matches other detection regions outside the corresponding detection region. Specifically, it is used to determine whether there is at least one third intermediate dataset located in other detection regions within the second intermediate dataset. If there is at least one third intermediate dataset located in other detection regions within the second intermediate dataset, the second intermediate dataset is clustered based on the third intermediate dataset to obtain at least one third cluster. That is, the point clouds in the third intermediate dataset that are adjacent to those in the second intermediate dataset are clustered. Then, based on the size information of the detection boxes in the matched detection regions, it is determined whether there is a fourth cluster with a correlation in the third cluster. Clusters with a correlation are those whose center distance is less than the diagonal length of the detection box in the corresponding detection region. If there is a fourth dataset to be merged, the fourth dataset to be merged is merged to obtain the second target dataset.

[0064] As another possible implementation, a general clustering is performed on the remaining point cloud data after successful secondary association and merging in the point cloud dataset. That is, for the remaining point cloud data in the second intermediate dataset other than the second target dataset, the remaining point cloud data is clustered according to the attribute information (such as spatial location and velocity) of the remaining point cloud data to obtain at least one third target data. For example, point cloud data with similar spatial location and similar velocity are grouped into one cluster.

[0065] As another possible implementation, if there is no point cloud data in the second intermediate dataset that successfully matches other detection areas, clustering is performed. That is, if there is no third intermediate dataset in the second intermediate dataset, the points in the second intermediate dataset are clustered according to the attribute information of each point in the second intermediate dataset to obtain at least one fourth target dataset.

[0066] As another possible implementation, when the third cluster does not include the associated fourth cluster, a general clustering is performed on the unassociated point cloud data. That is, based on the point cloud data in the second intermediate dataset excluding the third cluster, the point cloud data in the second intermediate dataset excluding the third cluster are clustered to obtain at least one fifth target dataset.

[0067] As another possible implementation, when there is no point cloud data in the point cloud dataset that matches the detection area, the point cloud data in the point cloud dataset is clustered according to the attribute information of the point cloud data in the point cloud dataset to obtain at least one sixth target dataset.

[0068] It should be noted that one or more of the above possible implementations may be executed in actual applications, and this disclosure does not impose any specific limitations.

[0069] Furthermore, it should be noted that the execution process of steps 201 to 202 can be implemented in any of the various embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0070] The target tracking method of this disclosure, for any detection area, determines whether there is at least one first intermediate dataset in the point cloud dataset located within any detection area; if so, it clusters the point cloud dataset based on the at least one first intermediate dataset to obtain at least one first cluster; wherein the first cluster includes a first intermediate dataset; based on the size information of the detection box in any detection area, it determines whether there is a second cluster to be merged in the at least one first cluster; if so, it merges the second cluster to be merged to obtain a first target dataset. Thus, the clustering operation based on the first intermediate dataset can more accurately classify point cloud data belonging to the same target into one category, which can effectively reduce the problem of false detection and missed detection caused by insufficient local information, ensure comprehensive coverage of the target, and help to achieve continuous and stable tracking of the target.

[0071] To clearly illustrate how target detection is performed on image information in the above embodiments to obtain at least one detection region, this disclosure proposes another target tracking method.

[0072] Figure 5 This is a schematic flowchart of another target tracking method provided in an embodiment of the present disclosure.

[0073] like Figure 5 As shown, the target tracking method may include the following steps:

[0074] Step 501: Perform target detection on the acquired image to obtain at least one detection box.

[0075] In this embodiment of the disclosure, a target detection algorithm is used to detect targets in the acquired image in order to identify targets in the acquired image and draw a bounding box (i.e., a detection box) for each detected target.

[0076] Step 502: For any detection frame, expand the size of any detection frame to obtain an expanded detection frame.

[0077] Since the target position of the detection result of the acquired image is not completely consistent with the actual position of the radar point cloud, in order to better adapt to the above dynamic changes and improve the accuracy of matching, in this embodiment of the disclosure, the size of each detection box is expanded to obtain an expanded detection box, and then it is determined whether there is point cloud data in the point cloud dataset that matches the region of the expanded detection box.

[0078] In this embodiment of the disclosure, the width and height of the detection frame are increased by a certain proportion or a fixed pixel value.

[0079] As an example, the horizontal expansion of the actual area corresponding to the detection box is specified by a distance (e.g., two meters). That is, based on the specified horizontal expansion distance of the actual area, the distance to be expanded in the detection box is calculated. The vertical expansion distance of the detection box = exp(0.03 * vertical side length of the detection box).

[0080] Step 503: Take any region within the expanded detection box in the acquired image as a detection region.

[0081] Furthermore, the area extended from any detection box is regarded as a detection region. That is, after object detection is performed on the acquired image, the initially determined detection box is not used directly. Instead, the size of the detection box is expanded according to specific rules (such as increasing a certain margin or based on the possible range of motion of the object), thereby forming a larger region containing more contextual information as the final detection region.

[0082] Step 504: Obtain the point cloud dataset of the environment collected by the vehicle-mounted radar at the target time.

[0083] Step 505: Based on at least one detection region and a point cloud dataset, perform a clustering process to extract the clustered target dataset from the point cloud dataset. The target dataset is used for target tracking.

[0084] It should be noted that the execution process of steps 504 to 505 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0085] To improve the accuracy of target tracking, a perception module is used to track targets based on the target dataset.

[0086] In this embodiment of the disclosure, it is determined whether there is a detection box in each detection box that matches the target dataset; if there is, the target dataset and the attribute information of the target in the matching detection box are sent to the perception module so that the perception module can perform target tracking based on the target dataset and the attribute information of the target in the matching detection box; if there is no target, the target dataset is sent to the perception module so that the perception module can perform target tracking based on the target dataset.

[0087] In other words, the system determines whether there is a detection result in the acquired image that matches the point cloud cluster (target dataset). If so, it means that the detection box has successfully identified and located the target detected by the radar. The target dataset and the attribute information of the target within the matching detection box are then sent to the perception module. The perception module uses the target dataset and the attribute information of the target within the matching detection box to perform more accurate target tracking and behavior prediction, thereby ensuring that the vehicle can make safer and more reasonable driving decisions. If no detection box matching the target dataset is found, it may mean that the vehicle camera failed to identify the target detected by the radar in the current frame (possibly due to occlusion, viewing angle limitations, etc.). In this case, only the target dataset itself is sent to the perception module. The perception module then performs target tracking based on the target dataset provided by the vehicle radar. Although there is a lack of visual information support from the vehicle camera, it can still maintain basic target tracking capabilities through the accurate distance and speed information provided by the radar, ensuring the stability and reliability of the system.

[0088] The target tracking method of this disclosure performs target detection on the acquired image to obtain at least one detection box; for any detection box, the size of any detection box is expanded to obtain an expanded detection box; the region within the expanded detection box in the acquired image is taken as a detection region. Thus, by performing target detection on the acquired image and expanding the detection box to obtain the detection region, the accuracy and robustness of target recognition and matching are improved, while providing richer contextual information and improving the effect of target tracking.

[0089] In any embodiment of this disclosure, the implementation process of this disclosure can be as follows: Figure 6 As shown, the main steps include:

[0090] 1. Real-time monitoring and judgment of the behavior of traffic participants around the vehicle through vehicle-mounted cameras, that is, acquiring images of the environment in which the target vehicle is located at the target time by the vehicle-mounted camera, and performing target detection on the acquired images to obtain the detection area; wherein, a detection area includes a detection box;

[0091] 2. Obtain point cloud information corresponding to the environment, and perform point cloud preprocessing on the point cloud information transmitted back by the vehicle radar;

[0092] 3. Match the detection area with the radar point cloud information in sequence. Perform point cloud clustering operation based on the detection area on the successfully matched point cloud information. Perform general point cloud clustering operation on the point cloud information that failed to match. For the unassociated clusters corresponding to a certain detection area, perform secondary clustering association operation with other detection areas.

[0093] 3.1 Considering the certain positioning difference between the detection box (image detection result) and the radar point cloud information, the matching area corresponding to the detection box is first expanded based on the relative position of the detection box and the vehicle. For example, in actual calculation, the matching area is usually expanded horizontally by 0.5 meters, and the vertical expansion is related to the position of the detection box. The calculation method is: vertical expansion distance = exp(0.03 * vertical position of detection result);

[0094] 4. Referring to the corresponding detection box, perform point cloud clustering operations based on the detection box for the successfully matched point cloud information in sequence. Then, associate the clustering results (determine whether the distance between the clustering results is less than the diagonal length of the detection box). If the association is successful (determine whether the distance between the clustering results is less than the diagonal length of the detection box), then merge all the successfully associated clusters. If the association fails, then perform a secondary clustering association operation with other detection regions for all the unassociated clusters.

[0095] 4.1 Point cloud clustering based on detection boxes will refer to the target size of the corresponding detection box, and combine the spatial position and velocity attributes of the point cloud itself to determine whether the points are adjacent, and all adjacent points will be treated as a cluster.

[0096] 4.2. Referring to the target size of the detection box, calculate the spatial distance between clustering results to determine whether the clustering results are related. Figure 7 As shown;

[0097] 5. Merge clustering results associated with the same detection box, treating them as different components of the same target, such as the front and rear of a large vehicle. The merged clustering results are output as the matching detection results and clustering part in the overall clustering algorithm output.

[0098] 6. Perform a general point cloud clustering operation on the point cloud information that fails to match (does not match any detection area, i.e. is not located in any detection area). The clustering result is used as the remaining point cloud clustering part in the overall clustering algorithm output.

[0099] 6.1 General point cloud clustering combines the spatial location and velocity attributes of the point cloud itself to determine whether the points are adjacent, and treats all adjacent points as a cluster;

[0100] 7. For example Figure 8As shown, a secondary clustering association operation is performed on all unassociated clusters corresponding to a certain detection area and other detection boxes. If successful, it is output as the matching detection area and cluster part in the overall clustering algorithm; if unsuccessful, it is output as the unmatched detection area and cluster part.

[0101] 8. The overall clustering algorithm takes the output of the above three parts as the clustering result and outputs it to the subsequent perception module of the intelligent driving system.

[0102] To achieve the above embodiments, this disclosure also proposes a target tracking device.

[0103] Figure 9 This is a schematic diagram of the structure of a target tracking device provided in an embodiment of the present disclosure.

[0104] like Figure 9 As shown, the target tracking device 900 includes: a first acquisition module 910, a second acquisition module 920, and a processing module 930.

[0105] The first acquisition module 910 is used to acquire images of the environment where the target vehicle is located, captured by the vehicle-mounted camera at the target time, and to perform target detection on the acquired images to obtain at least one detection area; the second acquisition module 920 is used to acquire a point cloud dataset of the environment captured by the vehicle-mounted radar at the target time; the processing module 930 is used to perform a clustering process based on at least one detection area and the point cloud dataset to extract the clustered target dataset from the point cloud dataset; wherein, the target dataset is used for target tracking.

[0106] As one possible implementation, the processing module 930 is used to determine, for any detection region, whether there is at least one first intermediate dataset in the point cloud dataset located within the detection region; if so, to cluster the point cloud dataset based on at least one first intermediate dataset to obtain at least one first cluster; wherein the first cluster includes a first intermediate dataset; and to determine, based on the size information of the detection box within any detection region, whether there is a second cluster to be merged in the at least one first cluster; if so, to merge the second cluster to be merged to obtain a first target dataset.

[0107] As one possible implementation, the processing module 930 is used to determine a reference cluster from at least one first cluster based on the location information of the target vehicle at the target time; wherein the distance between the reference cluster and the target vehicle is less than a threshold; for any cluster in the first cluster other than the reference cluster, determine the target distance between the center of any cluster and the center of the reference cluster; and determine whether any cluster is a second cluster to be merged based on the comparison result of the target distance and the size information of the detection box in any detection area.

[0108] As one possible implementation, the processing module 930 is configured to, for the at least one detection region other than any of the detection regions, determine whether there exists at least one third intermediate dataset located within the other detection regions in the second intermediate dataset; wherein, the second intermediate dataset includes point cloud data in the point cloud dataset other than the first target dataset, or includes point cloud data in the point cloud dataset other than the first cluster, or includes point cloud data in the point cloud dataset other than the first intermediate dataset; if so, cluster the second intermediate dataset based on the at least one third intermediate dataset to obtain at least one third cluster; wherein, the third cluster includes one third intermediate dataset; and determine whether there exists a fourth cluster to be merged in the at least one third cluster based on the size information of the detection boxes in the other detection regions; if so, merge the fourth cluster to be merged to obtain the second target dataset.

[0109] As one possible implementation, the processing module 930 is used to cluster the remaining point cloud data based on the attribute information of the remaining point cloud data in the second intermediate dataset other than the second target dataset, so as to obtain at least one third target dataset.

[0110] As one possible implementation, the processing module 930 is used to cluster the points in the second intermediate dataset according to the attribute information of each point in the second intermediate dataset if the third intermediate dataset does not exist in the second intermediate dataset, so as to obtain at least one fourth target dataset.

[0111] As one possible implementation, the processing module 930 is used to cluster the point cloud data in the second intermediate dataset other than at least one third cluster if there is no fourth cluster in at least one third cluster, so as to obtain at least one fifth target dataset.

[0112] As one possible implementation, if there is no point cloud data located within at least one detection region in the point cloud dataset, then the point cloud dataset is clustered according to the attribute information of each point in the point cloud dataset to obtain at least one sixth target dataset.

[0113] As one possible implementation, the first acquisition module 910 is used to perform target detection on the acquired image to obtain at least one detection box; for any detection box, the size of any detection box is expanded to obtain an expanded detection box; and the area within the expanded detection box in the acquired image is taken as a detection region.

[0114] As one possible implementation, the target tracking device 900 also includes a judgment module and a transmission module.

[0115] The system includes a judgment module for determining whether a detection box in each detection box matches the target dataset; and a sending module for sending the target dataset and the attribute information of the target within the matching detection box to the perception module if a match exists, so that the perception module can perform target tracking based on the target dataset and the attribute information of the target within the matching detection box; and a sending module for sending the target dataset to the perception module if a match does not exist, so that the perception module can perform target tracking based on the target dataset.

[0116] The target tracking device proposed in this disclosure acquires images of the environment where the target vehicle is located, captured by an onboard camera at a target time, and performs target detection on the acquired images to obtain at least one detection region; acquires a point cloud dataset of the environment captured by an onboard radar at the target time; and performs a clustering process based on the at least one detection region and the point cloud dataset to extract the clustered target dataset from the point cloud dataset. The target dataset is used for target tracking. Thus, by combining the image detection results from the onboard camera with the point cloud dataset from the onboard radar and performing a clustering process based on the detection region, a target dataset for target tracking is extracted, significantly improving the accuracy and robustness of target tracking, thereby effectively enhancing the safety and reliability of the autonomous driving system.

[0117] To implement the above embodiments, this disclosure also proposes a vehicle, including a processor and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the target tracking method as described in the foregoing method embodiments.

[0118] To implement the above embodiments, this disclosure proposes a chip including a processing circuit configured to perform the target tracking method as described in the foregoing method embodiments.

[0119] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the target tracking method as described in the foregoing method embodiments.

[0120] To implement the above embodiments, this disclosure also proposes a computer program product having a computer program stored thereon, which, when executed by a processor, implements the target tracking method as described in the foregoing method embodiments.

[0121] Figure 10This is a block diagram illustrating a vehicle 1000 according to an exemplary embodiment. For example, vehicle 1000 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 1000 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0122] Reference Figure 10 The vehicle 1000 may include various subsystems, such as an infotainment system 1010, a perception system 1020, a decision control system 1030, a drive system 1040, and a computing platform 1050. The vehicle 1000 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the vehicle 1000 can be interconnected via wired or wireless means.

[0123] In some embodiments, the infotainment system 1010 may include a communication system, an entertainment system, and a navigation system, etc.

[0124] The perception system 1020 may include several types of sensors for sensing information about the environment surrounding the vehicle 1000. For example, the perception system 1020 may include a global positioning system (which may be a GPS system, a BeiDou system, or another positioning system), an inertial measurement unit (IMU), a lidar, a millimeter-wave radar, an ultrasonic radar, and a camera device.

[0125] The decision control system 1030 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0126] The drive system 1040 may include components that provide powered motion to the vehicle 1000. In one embodiment, the drive system 1040 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0127] Some or all of the functions of the vehicle 1000 are controlled by a computing platform 1050. The computing platform 1050 may include at least one processor 1051 and a memory 1052, the processor 1051 being able to execute instructions 1053 stored in the memory 1052.

[0128] The processor 1051 can be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0129] The memory 1052 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0130] In addition to instruction 1053, memory 1052 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 1052 can be used by computing platform 1050.

[0131] In this embodiment of the disclosure, processor 1051 may execute instruction 1053 to complete all or part of the steps of the above method embodiments.

[0132] Figure 11 This is a schematic diagram of the chip structure proposed in an embodiment of this disclosure. See also... Figure 11 The diagram shown is a schematic representation of the structure of chip 1100, but it is not limited to this.

[0133] Chip 1100 includes processing circuitry 1101, which is configured to perform any of the above methods.

[0134] In some embodiments, chip 1100 further includes one or more interface circuits 1102. Optionally, the interface circuit 1102 is connected to memory 1103, and the interface circuit 1102 can be used to receive signals from memory 1103 or other devices, and the interface circuit 1102 can be used to send signals to memory 1103 or other devices. For example, the interface circuit 1102 can read instructions stored in memory 1103 and send the instructions to processing circuit 1101.

[0135] In some embodiments, the interface circuit 1102 performs at least one of the communication steps such as sending and / or receiving in the above method, while the processing circuit 1101 performs other steps.

[0136] In some embodiments, the terms interface circuit, interface, transceiver pin, transceiver, etc., can be used interchangeably.

[0137] In some embodiments, chip 1100 further includes one or more memories 1103 for storing instructions. Optionally, all or part of the memories 1103 may be located outside of chip 1100.

[0138] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0139] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0140] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0142] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0143] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0144] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0145] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A target tracking method, characterized in that, include: The system acquires images of the environment in which the target vehicle is located, captured by the vehicle-mounted camera at a target time, and performs target detection on the acquired images to obtain at least one detection area. Acquire the point cloud dataset of the environment collected by the vehicle-mounted radar at the target time; Based on the at least one detection region and the point cloud dataset, a clustering process is performed to extract the clustered target dataset from the point cloud dataset; wherein the target dataset is used for target tracking.

2. The method according to claim 1, characterized in that, The clustering process includes: For any detection region, determine whether there exists at least one first intermediate dataset in the point cloud dataset located within the detection region; If so, based on the at least one first intermediate dataset, the point cloud dataset is clustered to obtain at least one first cluster; wherein, the first cluster includes a first intermediate dataset; Based on the size information of the detection frame within any detection area, determine whether there is a second cluster to be merged in the at least one first cluster; If so, the second cluster to be merged is merged to obtain the first target dataset.

3. The method according to claim 2, characterized in that, The step of determining whether there is a second cluster to be merged in the at least one first cluster based on the size information of the detection frame within any detection area includes: Based on the location information of the target vehicle at the target time, a reference cluster is determined from the at least one first cluster; wherein the distance between the reference cluster and the target vehicle is less than a threshold. For any cluster in the first cluster other than the reference cluster, determine the target distance between the center of the any cluster and the center of the reference cluster; Based on the comparison result between the target distance and the size information of the detection frame in any detection area, it is determined whether any cluster is a second cluster to be merged.

4. The method according to claim 2, characterized in that, The clustering process also includes: For the other detection regions in the at least one detection region besides any of the detection regions, determine whether there is at least one third intermediate dataset located in the other detection regions in the second intermediate dataset; wherein, the second intermediate dataset includes point cloud data in the point cloud dataset other than the first target dataset, or includes point cloud data in the point cloud dataset other than the first cluster, or includes point cloud data in the point cloud dataset other than the first intermediate dataset. If so, based on the at least one third intermediate dataset, the second intermediate dataset is clustered to obtain at least one third cluster; wherein, the third cluster includes a third intermediate dataset; Based on the size information of the detection frames in the other detection areas, determine whether there is a fourth cluster to be merged in the at least one third cluster; If so, the fourth cluster to be merged is merged to obtain the second target dataset.

5. The method according to claim 4, characterized in that, The clustering process also includes: Based on the attribute information of the remaining point cloud data in the second intermediate dataset other than the second target dataset, the remaining point cloud data is clustered to obtain at least one third target dataset.

6. The method according to claim 4, characterized in that, The clustering process also includes: If the third intermediate dataset is not present in the second intermediate dataset, then the points in the second intermediate dataset are clustered according to the attribute information of each point in the second intermediate dataset to obtain at least one fourth target dataset.

7. The method according to claim 4, characterized in that, The clustering process also includes: If the fourth cluster is not present in the at least one third cluster, then the point cloud data in the second intermediate dataset other than the at least one third cluster are clustered to obtain at least one fifth target dataset.

8. The method according to claim 2, characterized in that, The clustering process also includes: If there is no point cloud data located within the at least one detection region in the point cloud dataset, then the point cloud dataset is clustered according to the attribute information of each point in the point cloud dataset to obtain at least one sixth target dataset.

9. The method according to any one of claims 1-8, characterized in that, The step of performing target detection on the image information to obtain at least one detection region includes: Target detection is performed on the acquired image to obtain at least one detection box; For any detection frame, the size of the detection frame is expanded to obtain the expanded detection frame. The region within any of the expanded detection boxes in the acquired image is taken as a detection region.

10. The method according to claim 9, characterized in that, The method further includes: Determine whether any of the detection boxes contains a detection box that matches the target dataset; If it exists, the target dataset and the attribute information of the target within the matched detection box are sent to the perception module so that the perception module can perform target tracking based on the target dataset and the attribute information of the target within the matched detection box. If the target dataset does not exist, the target dataset is sent to the perception module so that the perception module can perform target tracking based on the target dataset.

11. A target tracking device, characterized in that, include: The first acquisition module is used to acquire images of the environment where the target vehicle is located, captured by the vehicle-mounted camera at the target time, and to perform target detection on the acquired images to obtain at least one detection area. The second acquisition module is used to acquire the point cloud dataset of the environment collected by the vehicle-mounted radar at the target time; The processing module is configured to perform a clustering process based on the at least one detection region and the point cloud dataset to extract the clustered target dataset from the point cloud dataset; wherein the target dataset is used for target tracking.

12. A vehicle, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-10.

13. A chip, characterized in that, The chip includes processing circuitry configured to perform the method as described in any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

Images