Obstacle tracking method, apparatus, and vehicle

Abstract

This disclosure relates to an obstacle tracking method, apparatus, and electronic device. The method includes: acquiring 3D bounding box information of various objects surrounding a vehicle at the current time point, and 3D detection bounding box information of various obstacles surrounding the vehicle at various historical time points; for each obstacle, inputting the 3D detection bounding box information of the obstacle at various historical time points into the obstacle tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point; determining the obstacle to which each 3D bounding box information belongs based on the 3D bounding box information and the 3D prediction bounding box information of each obstacle at the current time point, for updating the tracking filter of each obstacle, and determining the updated 3D prediction bounding box information of the obstacle at the current time point, thereby performing obstacle tracking processing. This method enables 3D obstacle detection in multiple shooting device scenarios, improves obstacle detection efficiency, and thus improves obstacle tracking efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to an obstacle tracking method, device, and vehicle. Background Technology

[0002] Current obstacle tracking methods mainly involve acquiring surrounding images at various time points captured by a single camera on the vehicle; performing target detection processing on the surrounding images at each time point to determine the 2D detection bounding box information of obstacles at each time point; and performing matching processing on the 2D detection bounding box information of obstacles at adjacent time points to determine whether they correspond to the same obstacle.

[0003] The above solution is suitable for 2D obstacle detection based on a single shooting device, but it is difficult to apply to 3D obstacle detection in scenarios with multiple shooting devices. Summary of the Invention

[0004] This disclosure provides an obstacle tracking method, apparatus, and vehicle.

[0005] According to a first aspect of the present disclosure, an obstacle tracking method is provided, the method comprising: acquiring 3D bounding box information of each object surrounding a vehicle at a current time point, and 3D detection bounding box information of each obstacle surrounding the vehicle at each historical time point; the 3D bounding box information being determined based on multiple surrounding images of the vehicle at the current time point; for each obstacle, inputting the 3D detection bounding box information of the obstacle at each historical time point into the obstacle tracking filter to determine 3D prediction bounding box information of the obstacle at the current time point; and based on the 3D bounding box information of each object surrounding the vehicle at the current time point and the 3D detection bounding box information of each obstacle surrounding the vehicle at each historical time point, determining ... determining 3D prediction bounding box information of the obstacle at the current time point based on the 3D bounding box information of each object surrounding the vehicle at the current time point and the 3D detection bounding box information of each obstacle surrounding the vehicle at each historical time point. The 3D predicted bounding box information of the object is used to determine the obstacle to which each 3D target bounding box belongs at the current time point. Based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target bounding box belongs, the tracking filter of each obstacle is updated to obtain the updated tracking filter of each obstacle. For each obstacle, the 3D detection bounding box information of the obstacle at the current time point and at each previous historical time point is input into the updated tracking filter of the obstacle to obtain the updated 3D predicted bounding box information of the obstacle at the current time point, which is used for tracking the obstacle.

[0006] In one embodiment of this disclosure, obtaining 3D target bounding box information around a vehicle at a current time point includes: obtaining multiple surrounding images of the vehicle at the current time point; the multiple surrounding images are captured by multiple imaging devices on the vehicle at the current time point; obtaining an obstacle detection model; inputting the multiple surrounding images into the obstacle detection model to obtain multiple 3D candidate bounding box information and the confidence scores corresponding to the 3D candidate bounding box information; and filtering the multiple 3D candidate bounding box information based on the confidence scores to obtain the 3D target bounding box information at the current time point.

[0007] In one embodiment of this disclosure, the obstacle detection model is trained by combining multiple surrounding images of the vehicle at historical time points and obstacle information around the vehicle in the vehicle coordinate system at the historical time points. The step of inputting multiple surrounding images into the obstacle detection model to obtain multiple 3D candidate box information and the corresponding confidence scores of the 3D candidate box information includes: inputting multiple surrounding images into the obstacle detection model to obtain multiple 3D candidate box information and corresponding confidence scores in the vehicle coordinate system; and performing coordinate transformation processing by combining the vehicle's position information in the world coordinate system at the current time point and the multiple 3D candidate box information in the vehicle coordinate system to obtain multiple 3D candidate box information in the world coordinate system.

[0008] In one embodiment of this disclosure, the step of filtering multiple 3D candidate bounding box information based on the confidence level to obtain 3D target bounding box information at the current time point includes: performing ground projection processing on each 3D candidate bounding box information to obtain ground 2D candidate bounding box information corresponding to each 3D candidate bounding box information; sorting each ground 2D candidate bounding box information in descending order according to the confidence level corresponding to each 3D candidate bounding box information to obtain a sorting result; extracting the ground 2D candidate bounding box information with the highest confidence level from the sorting result, and deleting ground 2D candidate bounding box information in the sorting result whose intersection-union ratio with the ground 2D candidate bounding box information is greater than or equal to a first intersection-union ratio threshold; repeating the above steps until the ground 2D candidate bounding box information in the sorting result is processed; and using the 3D candidate bounding box information corresponding to the extracted ground 2D candidate bounding box information as the 3D target bounding box information at the current time point.

[0009] In one embodiment of this disclosure, the 3D target bounding box information and the 3D detection bounding box information include obstacle categories; determining the obstacle to which each 3D target bounding box information belongs at the current time point based on each 3D target bounding box information and each obstacle's 3D prediction bounding box information at the current time point includes: for each obstacle, obtaining the 3D target bounding box information to be compared from each 3D target bounding box information; the obstacle category in the 3D target bounding box information to be compared is consistent with the category of the obstacle; determining the matching degree between the 3D prediction bounding box information of the obstacle and each 3D target bounding box information to be compared; determining the obstacle to which the 3D target bounding box information to be compared belongs corresponding to the largest matching degree among multiple matching degrees, which is the obstacle.

[0010] In one embodiment of this disclosure, the 3D target bounding box information and the 3D predicted bounding box information further include position information; determining the matching degree between the 3D predicted bounding box information of the obstacle and each 3D target bounding box information to be compared includes: performing ground projection processing on the 3D predicted bounding box information of the obstacle and each 3D target bounding box information to be compared to obtain ground 2D predicted bounding box information corresponding to the 3D predicted bounding box information of the obstacle, and ground 2D target bounding box information corresponding to each 3D target bounding box information to be compared; for each 3D target bounding box information to be compared, based on the position information in the 3D target bounding box information to be compared... Based on the location information in the 3D predicted bounding box information of the obstacle, the distance data between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared is determined; based on the ground 2D target bounding box information corresponding to the 3D target bounding box information to be compared, and the ground 2D predicted bounding box information corresponding to the 3D predicted bounding box information of the obstacle, the ground intersection-over-union ratio (COUNG) data between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared is determined; based on the distance data and the ground COUNG data, the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared is determined.

[0011] In one embodiment of this disclosure, determining the matching degree between the 3D predicted bounding box information of the obstacle and each 3D target bounding box information to be compared further includes: combining the position information of the vehicle in the world coordinate system at the current time point, and the position information of multiple shooting devices on the vehicle relative to the vehicle, to determine the image 2D predicted bounding box information corresponding to the 3D predicted bounding box information of the obstacle, and the image 2D target bounding box information corresponding to each 3D target bounding box information to be compared; combining multiple surrounding images of the vehicle at the current time point, to determine the image region corresponding to each image 2D target bounding box information, and the image region corresponding to the image 2D predicted bounding box information; for each The process involves: determining the intersection-over-union (IoU) ratio between the 2D target bounding box information and the 2D predicted bounding box information, and the image similarity data between the image region corresponding to the 2D target bounding box information and the image region corresponding to the 2D predicted bounding box information; and determining the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared based on the distance data and the ground IoU data, including: determining the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared based on the distance data, the ground IoU data, the IoU data, and the image similarity data.

[0012] In one embodiment of this disclosure, the step of determining the image 2D prediction box information corresponding to the 3D prediction box information of the obstacle and the image 2D target box information corresponding to each 3D target box information to be compared, by combining the vehicle's position information in the world coordinate system at the current time point and the position information of multiple imaging devices on the vehicle relative to the vehicle, includes: determining the position information of multiple imaging devices in the world coordinate system at the current time point by combining the vehicle's position information in the world coordinate system at the current time point and the position information of multiple imaging devices on the vehicle relative to the vehicle; determining the imaging device corresponding to each 3D target box information to be compared and the imaging device corresponding to the 3D prediction box information of the obstacle based on the position information of the multiple imaging devices; determining the coordinate transformation relationship between the image coordinate system and the world coordinate system of each imaging device based on the position information of each imaging device; and determining the image 2D prediction box information corresponding to the 3D prediction box information of the obstacle and the image 2D target box information corresponding to each 3D target box information to be compared, by combining the coordinate transformation relationship between the image coordinate system and the world coordinate system of each imaging device.

[0013] In one embodiment of this disclosure, the method further includes: determining the updated 3D prediction bounding box information of the obstacle at the current time point as the 3D detection bounding box information of the obstacle at the current time point.

[0014] According to a second aspect of the present disclosure, an obstacle tracking device is also provided. The device includes: an acquisition module, configured to acquire 3D target bounding box information of each object surrounding a vehicle at a current time point, and 3D detection bounding box information of each obstacle surrounding the vehicle at each historical time point; the 3D target bounding box information is determined based on multiple surrounding images of the vehicle at the current time point; a first determination module, configured to, for each obstacle, input the 3D detection bounding box information of the obstacle at each historical time point into the obstacle's tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point; and a second determination module, configured to, based on the 3D target bounding box information at the current time point... The system includes a first module and a second module. The first module determines the obstacle to which each 3D target box belongs at the current time point, based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target box belongs. The second module updates the tracking filter of each obstacle based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target box belongs, resulting in an updated tracking filter for each obstacle. The third module determines the obstacle by inputting the 3D detection bounding box information of the obstacle at the current time point and previous historical time points into the updated tracking filter of the obstacle, resulting in updated 3D predicted bounding box information of the obstacle at the current time point, which is used for tracking the obstacle.

[0015] According to a third aspect of the present disclosure, a vehicle is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: implement the obstacle tracking method as described above.

[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is also provided, which, when instructions in the storage medium are executed by a processor, enables the processor to perform the obstacle tracking method as described above.

[0017] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0018] By acquiring 3D bounding box information of the vehicle's surroundings at the current time point, and 3D detection bounding box information of obstacles around the vehicle at various historical time points; the 3D bounding box information is determined based on multiple surrounding images of the vehicle at the current time point; for each obstacle, the 3D detection bounding box information of the obstacle at various historical time points is input into the obstacle tracking filter to determine the obstacle's 3D prediction bounding box information at the current time point; based on the 3D bounding box information and the 3D prediction bounding box information of each obstacle at the current time point, the obstacle to which each 3D bounding box information belongs at the current time point is determined; based on the current... The 3D predicted bounding box information of each obstacle at the previous time point, as well as the obstacle to which each 3D target bounding box belongs, are used to update the tracking filter of each obstacle, resulting in an updated tracking filter for each obstacle. For each obstacle, the 3D detection bounding box information of the obstacle at the current time point and at each previous historical time point are input into the updated tracking filter of the obstacle to obtain the updated 3D predicted bounding box information of the obstacle at the current time point, which is used for obstacle tracking. This enables 3D obstacle detection in multiple shooting device scenarios, improves obstacle detection efficiency, and thus improves obstacle tracking efficiency.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0021] Figure 1 This is a flowchart of an obstacle tracking method according to an embodiment of the present disclosure;

[0022] Figure 2 This is a flowchart of an obstacle tracking method according to another embodiment of the present disclosure;

[0023] Figure 3 This is a flowchart of an obstacle tracking method according to another embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram of the structure of an obstacle tracking device according to an embodiment of the present disclosure;

[0025] Figure 5 This is a structural block diagram of a vehicle according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0027] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] Current obstacle tracking methods mainly involve acquiring surrounding images at various time points captured by a single camera on the vehicle; performing target detection processing on the surrounding images at each time point to determine the 2D detection bounding box information of obstacles at each time point; and performing matching processing on the 2D detection bounding box information of obstacles at adjacent time points to determine whether they correspond to the same obstacle.

[0029] The above solution is suitable for 2D obstacle detection based on a single shooting device, but it is difficult to apply to 3D obstacle detection in scenarios with multiple shooting devices.

[0030] Figure 1 This is a flowchart of an obstacle tracking method according to an embodiment of the present disclosure. It should be noted that the obstacle tracking method of this embodiment can be applied to an obstacle tracking device, which can be configured in an electronic device to enable the electronic device to perform obstacle tracking functions.

[0031] The electronic device can be any device with computing capabilities, such as a personal computer (PC), mobile terminal, server, or controller in a vehicle. A mobile terminal can be a vehicle-mounted device, mobile phone, tablet computer, personal digital assistant, wearable device, or other hardware device with various operating systems, touchscreens, and / or displays. The following embodiments use an electronic device as an example for illustration.

[0032] like Figure 1 As shown, the method includes the following steps:

[0033] Step 101: Obtain the 3D bounding box information of each object around the vehicle at the current time point, and the 3D detection bounding box information of each obstacle around the vehicle at each historical time point; the 3D bounding box information is determined based on multiple images of the vehicle's surroundings at the current time point.

[0034] In this embodiment of the disclosure, multiple camera devices may be installed on the vehicle, and the positions and shooting angles of these multiple camera devices on the vehicle may differ. These multiple camera devices can acquire images of the vehicle's surroundings in real time.

[0035] In this embodiment of the disclosure, the 3D target bounding box information around the vehicle at the current time point can be determined by the electronic device in combination with the vehicle's position information at the current time point, the position information of multiple shooting devices on the vehicle, the shooting angle range of the shooting devices, and the 2D detection box information in multiple surrounding images.

[0036] In this embodiment of the disclosure, the 3D target bounding box information may include at least one of the following: position information, size information, heading angle information, and category information. The position information can be represented by the 3D coordinates of each corner of the 3D target bounding box; the size information can refer to the length, width, and height of the 3D target bounding box. The heading angle can be the angle information of the 3D target bounding box relative to the vehicle. The category information, i.e., the obstacle category, can be, for example, a person, a vehicle, a static object, etc., and can be set according to actual needs.

[0037] The 3D detection box information may also include at least one of the following: position information, size information, heading angle information, and category information.

[0038] Step 102: For each obstacle, input the 3D detection bounding box information of the obstacle at each historical time point into the obstacle tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point.

[0039] In this embodiment of the disclosure, a tracking filter can be provided for each obstacle. The number of tracking filters for each obstacle can be one or more, such as a 2D frame filter, a 3D frame velocity filter, a 3D frame acceleration filter, a 3D frame size filter, etc.

[0040] In one example, the input to the 2D box filter can be the ground coordinates, length, and width of a corner of the 3D detection box information of the obstacle at each historical time point; the output of the 2D box filter can be the ground coordinates, length, and width of a corner of the 3D prediction box information of the obstacle at the current time point. In another example, the input to the 2D box filter can be the ground coordinates, length, and width of the center point of the 3D detection box information of the obstacle at each historical time point; the output of the 2D box filter can be the ground coordinates, length, and width of the center point of the 3D prediction box information of the obstacle at the current time point.

[0041] The input to the 3D frame velocity filter can be the position information and heading angle information of the 3D detection frame information of the obstacle at each historical time point; the output of the 3D frame velocity filter can be the position information, heading angle information and velocity information of the 3D prediction frame information of the obstacle at the current time point.

[0042] The input to the 3D bounding box acceleration filter can be the position information and heading angle information of the 3D detection bounding box information of the obstacle at each historical time point; the output of the 3D bounding box acceleration filter can be the position information, heading angle information, velocity information and acceleration information of the 3D prediction bounding box information of the obstacle at the current time point.

[0043] The input to the 3D bounding box size filter can be the size information of the 3D detection bounding box information of the obstacle at each historical time point; the output of the 3D bounding box size filter can be the size information of the 3D prediction bounding box information of the obstacle at the current time point.

[0044] In cases where there are multiple tracking filters for obstacles, the output results of multiple tracking filters can be combined to comprehensively determine the 3D prediction box information of obstacles at the current time point.

[0045] Step 103: Based on the 3D target bounding box information and the 3D predicted bounding box information of each obstacle at the current time point, determine the obstacle to which each 3D target bounding box information belongs at the current time point.

[0046] In this embodiment of the disclosure, the electronic device can determine the matching degree between each 3D target box information and each 3D prediction box information based on each 3D target box information and each 3D prediction box information at the current time point; and determine the obstacle to which each 3D target box information belongs at the current time point by combining the matching degree.

[0047] Step 104: Based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target bounding box belongs, update the tracking filter of each obstacle to obtain the updated tracking filter of each obstacle.

[0048] In this embodiment of the disclosure, the electronic device may perform step 104 as follows: for each obstacle, determine the difference information between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information belonging to the obstacle; based on the difference information, update the tracking filter of the obstacle to obtain the updated tracking filter of the obstacle.

[0049] Step 105: For each obstacle, input the 3D detection bounding box information of the obstacle at the current time point and at each previous historical time point into the updated tracking filter of the obstacle to obtain the updated 3D prediction bounding box information of the obstacle at the current time point, which is used for obstacle tracking processing.

[0050] In this embodiment, it should be noted that the obstacle tracking filter can be a Kalman filter; the input of the Kalman filter can involve multiple time points, and the output can also involve multiple time points. That is, by inputting the 3D detection box information of obstacles at the current time point and previous historical time points into the updated Kalman filter, the 3D prediction box information of obstacles at the current time point and multiple future time points can be predicted, thereby obtaining the updated 3D prediction box information of obstacles at the current time point.

[0051] In this embodiment of the disclosure, after step 105, the electronic device may further perform the following process: determining the updated 3D prediction bounding box information of the obstacle at the current time point as the 3D detection bounding box information of the obstacle at the current time point. Thus, the electronic device can obtain the 3D detection bounding box information of the obstacle at each time point and perform obstacle tracking processing.

[0052] In the obstacle tracking method of this disclosure, the following steps are taken: 1) Obtain 3D bounding box information of each obstacle surrounding the vehicle at the current time point, and 3D detection bounding box information of each obstacle surrounding the vehicle at each historical time point. 2) The 3D bounding box information is determined based on multiple surrounding images of the vehicle at the current time point. 3) For each obstacle, the 3D detection bounding box information of the obstacle at each historical time point is input into the obstacle tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point. 4) Based on the 3D bounding box information and the 3D prediction bounding box information of each obstacle at the current time point, the method to which each 3D bounding box information belongs is determined. The system detects obstacles; based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target bounding box belongs, the tracking filter of each obstacle is updated to obtain the updated tracking filter for each obstacle; for each obstacle, the 3D detection bounding box information of the obstacle at the current time point and previous historical time points are input into the updated tracking filter of the obstacle to obtain the updated 3D predicted bounding box information of the obstacle at the current time point, which is used for obstacle tracking. This system can realize 3D obstacle detection in multiple shooting device scenarios, improve obstacle detection efficiency, and thus improve obstacle tracking efficiency.

[0053] Figure 2This is a flowchart illustrating another embodiment of an obstacle tracking method disclosed herein. It should be noted that the obstacle tracking method of this embodiment can be applied to an obstacle tracking device, which can be configured in an electronic device to enable the electronic device to perform obstacle tracking functions.

[0054] The electronic device can be any device with computing capabilities, such as a personal computer (PC), mobile terminal, server, or controller in a vehicle. A mobile terminal can be a vehicle-mounted device, mobile phone, tablet computer, personal digital assistant, wearable device, or other hardware device with various operating systems, touchscreens, and / or displays. The following embodiments use an electronic device as an example for illustration.

[0055] like Figure 2 As shown, the method includes the following steps:

[0056] Step 201: Obtain multiple surrounding images of the vehicle at the current time point; the multiple surrounding images are obtained by multiple camera devices on the vehicle at the current time point.

[0057] Step 202: Obtain the obstacle detection model.

[0058] In this embodiment, the obstacle detection model can be pre-trained by combining surrounding images of the vehicle at multiple historical time points and obstacle information around the vehicle. The obstacle information includes, for example, the position, size, heading angle, and category information of the detection box containing the obstacle. The obstacle information around the vehicle can refer to the obstacle information around the vehicle in the vehicle coordinate system.

[0059] The obstacle detection model can be trained by electronic devices or other equipment. Taking the training of the obstacle detection model by electronic devices as an example, the training process can be as follows: acquire multiple surrounding images of the vehicle at a historical time point, as well as laser point cloud data at that historical time point; combine the laser point cloud data at that historical time point to determine the 3D detection bounding box information of the obstacle in the vehicle coordinate system at that historical time point; establish the correspondence between the 3D detection bounding box information of the obstacle and the multiple surrounding images of the vehicle at that historical time point to obtain training data; use the training data to train the initial obstacle detection model to obtain the trained obstacle detection model.

[0060] Step 203: Input multiple surrounding images into the obstacle detection model to obtain multiple 3D candidate box information and the confidence level corresponding to the 3D candidate box information.

[0061] In this embodiment, the obstacle detection model is trained by combining multiple surrounding images of the vehicle at historical time points and obstacle information around the vehicle in the vehicle coordinate system at historical time points. Correspondingly, the electronic device performing step 203 can be, for example, by inputting multiple surrounding images into the obstacle detection model to obtain multiple 3D candidate box information and corresponding confidence scores in the vehicle coordinate system; and by combining the vehicle's position information in the world coordinate system at the current time point and the multiple 3D candidate box information in the vehicle coordinate system with coordinate transformation processing to obtain multiple 3D candidate box information in the world coordinate system.

[0062] Since the vehicle's coordinate system may differ at different points in time during its operation, it is necessary to convert the information of multiple 3D candidate boxes in the vehicle's coordinate system to the world coordinate system for processing. This ensures that the information of multiple 3D candidate boxes is located in the same coordinate system, thereby further improving the accuracy of subsequent tracking processing.

[0063] Step 204: Combine the confidence scores to filter the information of multiple 3D candidate boxes and obtain the information of each 3D target box at the current time point.

[0064] In this embodiment of the disclosure, the electronic device may perform step 204 as follows: perform ground projection processing on each 3D candidate box information to obtain ground 2D candidate box information corresponding to each 3D candidate box information; sort each ground 2D candidate box information in descending order according to the confidence level corresponding to each 3D candidate box information to obtain a sorting result; extract the ground 2D candidate box information with the highest confidence level from the sorting result, and delete the ground 2D candidate box information in the sorting result whose intersection-union ratio with the ground 2D candidate box information is greater than or equal to the first intersection-union ratio threshold; repeat the above steps until the ground 2D candidate box information in the sorting result is processed; and use the extracted ground 2D candidate box information corresponding to the 3D candidate box information as the 3D target box information at the current time point.

[0065] Among them, ground projection processing of 3D candidate box information can refer to removing the height information or the Z-axis coordinate information from the 3D candidate box information.

[0066] The process of determining the intersection-union ratio of two ground 2D candidate bounding boxes can be, for example, as follows: obtain the intersection of the regions occupied by the two ground 2D candidate bounding boxes; obtain the union of the regions occupied by the two ground 2D candidate bounding boxes; and determine the ratio of the intersection region to the union region as the intersection-union ratio of the two ground 2D candidate bounding boxes.

[0067] Among them, filtering multiple 3D candidate box information can remove duplicate 3D candidate box information, ensuring that only one 3D candidate box information is retained for each obstacle, which facilitates subsequent matching processing.

[0068] Step 205: Obtain 3D detection frame information of various obstacles around the vehicle at each historical time point.

[0069] Step 206: For each obstacle, input the 3D detection bounding box information of the obstacle at each historical time point into the obstacle tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point.

[0070] Step 207: Based on the 3D target bounding box information and the 3D predicted bounding box information of each obstacle at the current time point, determine the obstacle to which each 3D target bounding box information belongs at the current time point.

[0071] Step 208: Based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target bounding box belongs, update the tracking filter of each obstacle to obtain the updated tracking filter of each obstacle.

[0072] Step 209: For each obstacle, input the 3D detection bounding box information of the obstacle at the current time point and at each previous historical time point into the updated tracking filter of the obstacle to obtain the updated 3D prediction bounding box information of the obstacle at the current time point, which is used for obstacle tracking processing.

[0073] It should be noted that the descriptions of steps 206 to 209 can be found in [reference needed]. Figure 1 The detailed description of steps 102 to 105 in the illustrated embodiment will not be repeated here.

[0074] In the obstacle tracking method of this embodiment, multiple surrounding images of the vehicle at the current time point are acquired; these multiple surrounding images are captured by multiple imaging devices on the vehicle at the current time point; an obstacle detection model is acquired; the multiple surrounding images are input into the obstacle detection model to acquire multiple 3D candidate box information and the confidence level corresponding to the 3D candidate box information; the multiple 3D candidate box information is filtered based on the confidence level to obtain the 3D target box information at the current time point; 3D detection box information of each obstacle around the vehicle at each historical time point is acquired; for each obstacle, the 3D detection box information of the obstacle at each historical time point is input into the obstacle tracking filter to determine the 3D prediction box information of the obstacle at the current time point; based on the 3D target box information at the current time point... Based on the bounding box information and the 3D predicted bounding box information of each obstacle, the obstacle to which each 3D target bounding box belongs at the current time point is determined. Based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target bounding box belongs, the tracking filter for each obstacle is updated to obtain the updated tracking filter for each obstacle. For each obstacle, the 3D detection bounding box information of the obstacle at the current time point and previous historical time points are input into the updated tracking filter to obtain the updated 3D predicted bounding box information of the obstacle at the current time point, which is used for obstacle tracking. This method can accurately determine each 3D target detection box at the current time point and perform 3D obstacle tracking based on each 3D target detection box, further improving obstacle tracking efficiency.

[0075] Figure 3 This is a flowchart illustrating another embodiment of an obstacle tracking method disclosed herein. It should be noted that the obstacle tracking method of this embodiment can be applied to an obstacle tracking device, which can be configured in an electronic device to enable the electronic device to perform obstacle tracking functions.

[0076] The electronic device can be any device with computing capabilities, such as a personal computer (PC), mobile terminal, server, or controller in a vehicle. A mobile terminal can be a vehicle-mounted device, mobile phone, tablet computer, personal digital assistant, wearable device, or other hardware device with various operating systems, touchscreens, and / or displays. The following embodiments use an electronic device as an example for illustration.

[0077] like Figure 3 As shown, the method includes the following steps:

[0078] Step 301: Obtain the 3D target bounding box information of each vehicle around the vehicle at the current time point, and the 3D detection bounding box information of each obstacle around the vehicle at each historical time point; the 3D target bounding box information is determined based on multiple surrounding images of the vehicle at the current time point; the 3D target bounding box information and the 3D detection bounding box information include the obstacle category.

[0079] Step 302: For each obstacle, input the 3D detection bounding box information of the obstacle at each historical time point into the obstacle tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point.

[0080] Step 303: For each obstacle, obtain the 3D target box information to be compared from the 3D target box information; the obstacle category in the 3D target box information to be compared is consistent with the obstacle category.

[0081] Step 304: Determine the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared.

[0082] In this embodiment of the disclosure, the 3D target bounding box information and the 3D predicted bounding box information further include location information. Correspondingly, the electronic device performing step 304 may, for example, perform ground projection processing on the 3D predicted bounding box information of the obstacle and each 3D target bounding box information to be compared, to obtain ground 2D predicted bounding box information corresponding to the 3D predicted bounding box information of the obstacle, and ground 2D target bounding box information corresponding to each 3D target bounding box information to be compared; for each 3D target bounding box information to be compared, determine the distance data between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared based on the location information in the 3D target bounding box information to be compared and the location information in the 3D predicted bounding box information of the obstacle; determine the ground intersection-union ratio (DIU) data between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared based on the ground 2D target bounding box information corresponding to the 3D target bounding box information to be compared and the ground 2D predicted bounding box information corresponding to the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared based on the distance data and the ground DIU data; and determine the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared based on the distance data and the ground DIU data.

[0083] The distance data between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared can be obtained by calculating the distance based on the 3D coordinate information of a corner in the 3D predicted bounding box information of the obstacle and the 3D coordinate information of the corresponding corner in the 3D target bounding box information to be compared; or, it can be obtained by calculating the distance based on the 3D coordinate information of the center point of the 3D predicted bounding box information of the obstacle and the 3D coordinate information of the center point of the 3D target bounding box information to be compared.

[0084] The process of determining the ground intersection-union ratio (CUI) data between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared can be, for example, obtaining the intersection between the area occupied by the 3D predicted bounding box information and the area occupied by the 3D target bounding box information to be compared; obtaining the union between the area occupied by the 3D predicted bounding box information and the area occupied by the 3D target bounding box information to be compared; and determining the ratio of the intersection to the union as the ground CUI data.

[0085] In this embodiment of the disclosure, the process by which the electronic device determines the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared based on distance data and ground intersection-union comparison data can be as follows: determining the distance matching degree based on distance data and a preset distance threshold; determining the ground intersection-union comparison matching degree based on ground intersection-union comparison data and a second intersection-union comparison threshold; and combining the distance matching degree and the ground intersection-union comparison matching degree to determine the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared.

[0086] Specifically, when the distance data is greater than or equal to a preset distance threshold, the distance matching degree is determined to be zero; when the distance data is less than the preset distance threshold, the distance matching degree is determined based on the distance data; the smaller the distance data, the greater the distance matching degree.

[0087] Specifically, when the ground crossover ratio data is less than or equal to the second crossover ratio threshold, the ground crossover ratio matching degree is determined to be zero; when the ground crossover ratio data is greater than the second crossover ratio threshold, the ground crossover ratio matching degree is determined based on the ground crossover ratio data; and the larger the ground crossover ratio data, the larger the ground crossover ratio matching degree.

[0088] The electronic device can determine the weights corresponding to the distance matching degree and the ground intersection-union ratio matching degree; perform weighted summation on the distance matching degree and the ground intersection-union ratio matching degree according to the weights to obtain the weighted summation result; and use the weighted summation result as the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared.

[0089] In this embodiment of the disclosure, the electronic device may further perform the following processes: combining the vehicle's position information in the world coordinate system at the current time point, and the position information of multiple imaging devices on the vehicle relative to the vehicle, to determine the image 2D prediction box information corresponding to the 3D prediction box information of the obstacle, and the image 2D target box information corresponding to each 3D target box information to be compared; combining multiple surrounding images of the vehicle at the current time point, to determine the image region corresponding to each image 2D target box information, and the image region corresponding to the image 2D prediction box information; for each image 2D target box information, to determine the intersection-over-union (IoU) data between the image 2D target box information and the image 2D prediction box information, and the image similarity data between the image region corresponding to the image 2D target box information and the image region corresponding to the image 2D prediction box information. Correspondingly, the electronic device can determine the matching degree between the 3D prediction box information of the obstacle and the 3D target box information to be compared based on distance data, ground IoU data, IoU data, and image similarity data.

[0090] Among them, the calculation strategies for image similarity data can be, for example, feature similarity calculation, calculation of the number of identical pixels, and structural similarity (SSIM).

[0091] Step 305: Determine the obstacle to which the 3D target box information to be compared belongs among the multiple matching degrees with the highest matching degree.

[0092] Step 306: Based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target bounding box belongs, update the tracking filter of each obstacle to obtain the updated tracking filter of each obstacle.

[0093] Step 307: For each obstacle, input the 3D detection bounding box information of the obstacle at the current time point and at each previous historical time point into the updated tracking filter of the obstacle to obtain the updated 3D prediction bounding box information of the obstacle at the current time point, which is used for obstacle tracking processing.

[0094] In the obstacle tracking method of this disclosure, the following steps are taken: 1) Obtaining 3D target bounding box information of the vehicle's surroundings at the current time point, and 3D detection bounding box information of obstacles around the vehicle at various historical time points. 2) The 3D target bounding box information is determined based on multiple surrounding images of the vehicle at the current time point. 3) The 3D target bounding box information and the 3D detection bounding box information include obstacle categories. 3) For each obstacle, the 3D detection bounding box information of the obstacle at various historical time points is input into the obstacle tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point. 4) For each obstacle, the 3D target bounding box information to be compared is obtained from the 3D target bounding box information. 5) The obstacle category in the 3D target bounding box information to be compared is consistent with the obstacle category. 6) The matching degree between the 3D prediction bounding box information of the obstacle and the 3D target bounding box information to be compared is determined. 7) The matching degree corresponding to the largest matching degree among multiple matching degrees is determined. The obstacle to which the 3D target bounding box information to be compared belongs is the obstacle. Based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target bounding box information belongs, the tracking filter of each obstacle is updated to obtain the updated tracking filter of each obstacle. For each obstacle, the 3D detection bounding box information of the obstacle at the current time point and at each previous historical time point is input into the updated tracking filter of the obstacle to obtain the updated 3D predicted bounding box information of the obstacle at the current time point, which is used for obstacle tracking. In this way, the matching degree between the 3D predicted bounding box information of each obstacle and the 3D target bounding box information to be compared can be combined to determine the obstacle to which each 3D target bounding box information belongs. This can accurately determine the obstacle corresponding to the 3D target bounding box information, further improve obstacle tracking efficiency, and achieve accurate obstacle tracking.

[0095] Figure 4 This is a schematic diagram of the structure of an obstacle tracking device according to an embodiment of the present disclosure.

[0096] like Figure 4 As shown, the obstacle tracking device may include: an acquisition module 401, a first determination module 402, a second determination module 403, an update module 404, and a third determination module 405.

[0097] The acquisition module 401 is used to acquire 3D target bounding box information of the vehicle's surroundings at the current time point, and 3D detection bounding box information of obstacles around the vehicle at various historical time points; the 3D target bounding box information is determined based on multiple surrounding images of the vehicle at the current time point; the first determination module 402 is used to input the 3D detection bounding box information of the obstacle at various historical time points into the obstacle's tracking filter for each obstacle, and determine the 3D prediction bounding box information of the obstacle at the current time point; the second determination module 403 is used to determine the 3D prediction bounding box information of the obstacle based on the 3D target bounding box information and the 3D prediction bounding box information of each obstacle at the current time point. The system first determines the obstacles to which each 3D target bounding box belongs at the current time point; the second module 404 updates the tracking filter of each obstacle based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacles to which each 3D target bounding box information belongs, to obtain the updated tracking filter of each obstacle; the third module 405 inputs the 3D detection bounding box information of the obstacle at the current time point and previous historical time points into the updated tracking filter of the obstacle for each obstacle to obtain the updated 3D predicted bounding box information of the obstacle at the current time point, and uses it for tracking the obstacle.

[0098] In one embodiment of this disclosure, the acquisition module 401 is specifically used to: acquire multiple surrounding images of the vehicle at the current time point; the multiple surrounding images are captured by multiple imaging devices on the vehicle at the current time point; acquire an obstacle detection model; input the multiple surrounding images into the obstacle detection model to acquire multiple 3D candidate box information and the confidence level corresponding to the 3D candidate box information; and filter the multiple 3D candidate box information based on the confidence level to obtain the 3D target box information at the current time point.

[0099] In one embodiment of this disclosure, the obstacle detection model is trained by combining multiple surrounding images of the vehicle at historical time points and obstacle information around the vehicle in the vehicle coordinate system at the historical time points; the acquisition module 401 is further configured to input multiple surrounding images into the obstacle detection model to obtain multiple 3D candidate box information and corresponding confidence scores in the vehicle coordinate system; and perform coordinate transformation processing by combining the vehicle's position information in the world coordinate system at the current time point and the multiple 3D candidate box information in the vehicle coordinate system to obtain multiple 3D candidate box information in the world coordinate system.

[0100] In one embodiment of this disclosure, the acquisition module 401 is further configured to: perform ground projection processing on each 3D candidate box information to obtain ground 2D candidate box information corresponding to each 3D candidate box information; sort each ground 2D candidate box information in descending order according to the confidence level corresponding to each 3D candidate box information to obtain a sorting result; extract the ground 2D candidate box information with the highest confidence level from the sorting result, and delete the ground 2D candidate box information in the sorting result whose intersection-union ratio with the ground 2D candidate box information is greater than or equal to a first intersection-union ratio threshold; repeat the above steps until the ground 2D candidate box information in the sorting result is processed; and use the 3D candidate box information corresponding to the extracted ground 2D candidate box information as each 3D target box information at the current time point.

[0101] In one embodiment of this disclosure, the 3D target bounding box information and the 3D detection bounding box information include an obstacle category; the second determining module 403 is specifically used to: for each obstacle, obtain the 3D target bounding box information to be compared from each 3D target bounding box information; the obstacle category in the 3D target bounding box information to be compared is consistent with the category of the obstacle; determine the matching degree between the 3D prediction box information of the obstacle and each 3D target bounding box information to be compared; determine the obstacle to which the 3D target bounding box information to be compared belongs, corresponding to the largest matching degree among multiple matching degrees.

[0102] In one embodiment of this disclosure, the 3D target bounding box information and the 3D predicted bounding box information further include position information; the second determining module 403 is further configured to perform ground projection processing on the 3D predicted bounding box information of the obstacle and each 3D target bounding box information to be compared, to obtain ground 2D predicted bounding box information corresponding to the 3D predicted bounding box information of the obstacle, and ground 2D target bounding box information corresponding to each 3D target bounding box information to be compared; for each 3D target bounding box information to be compared, based on the position information in the 3D target bounding box information to be compared and the 3D predicted bounding box information of the obstacle... The location information in the predicted bounding box information is used to determine the distance data between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared; based on the ground 2D target bounding box information corresponding to the 3D target bounding box information to be compared and the ground 2D predicted bounding box information corresponding to the 3D predicted bounding box information of the obstacle, the ground intersection-over-union ratio (COUNG) data between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared is determined; based on the distance data and the ground COUNG data, the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared is determined.

[0103] In one embodiment of this disclosure, the second determining module 403 is further configured to: combine the vehicle's position information in the world coordinate system at the current time point, and the position information of multiple imaging devices on the vehicle relative to the vehicle, to determine the image 2D prediction box information corresponding to the 3D prediction box information of the obstacle, and the image 2D target box information corresponding to each 3D target box information to be compared; combine multiple surrounding images of the vehicle at the current time point to determine the image region corresponding to each image 2D target box information, and the image region corresponding to the image 2D prediction box information; for each image 2D target box information, to determine the intersection-union ratio (IUGR) data between the image 2D target box information and the image 2D prediction box information, and the image similarity data between the image region corresponding to the image 2D target box information and the image region corresponding to the image 2D prediction box information; correspondingly, the second determining module 403 is further configured to: determine the matching degree between the 3D prediction box information of the obstacle and the 3D target box information to be compared based on the distance data, the ground IUGR data, the IUGR data, and the image similarity data.

[0104] In one embodiment of this disclosure, the second determining module 403 is further configured to: determine the position information of multiple shooting devices in the world coordinate system at the current time point by combining the position information of the vehicle in the world coordinate system at the current time point and the position information of multiple shooting devices on the vehicle relative to the vehicle; determine the shooting device corresponding to each 3D target bounding box to be compared and the shooting device corresponding to the 3D prediction bounding box information of the obstacle based on the position information of each shooting device; determine the coordinate transformation relationship between the image coordinate system and the world coordinate system of each shooting device based on the position information of each shooting device; and determine the image 2D prediction bounding box information corresponding to the 3D prediction bounding box information of the obstacle and the image 2D target bounding box information corresponding to each 3D target bounding box information to be compared by combining the coordinate transformation relationship between the image coordinate system and the world coordinate system of each shooting device.

[0105] In one embodiment of this disclosure, the apparatus further includes: a fourth determining module, configured to determine the updated 3D prediction box information of the obstacle at the current time point as the 3D detection box information of the obstacle at the current time point.

[0106] In the obstacle tracking device of this embodiment, the following steps are taken: 1) Obtaining 3D target bounding box information of the vehicle's periphery at the current time point, and 2) Obtaining 3D detection bounding box information of obstacles around the vehicle at each historical time point; 3) Obtaining 3D target bounding box information based on multiple images of the vehicle's periphery at the current time point; 4) For each obstacle, inputting the 3D detection bounding box information of the obstacle at each historical time point into the obstacle tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point; 5) Determining the 3D predicted bounding box information of the obstacle at the current time point based on the 3D target bounding box information and the 3D predicted bounding box information of each obstacle at the current time point. The system detects obstacles; based on the 3D predicted bounding box information of each obstacle at the current time point and the obstacle to which each 3D target bounding box belongs, the tracking filter of each obstacle is updated to obtain the updated tracking filter for each obstacle; for each obstacle, the 3D detection bounding box information of the obstacle at the current time point and previous historical time points are input into the updated tracking filter of the obstacle to obtain the updated 3D predicted bounding box information of the obstacle at the current time point, which is used for obstacle tracking. This system can realize 3D obstacle detection in multiple shooting device scenarios, improve obstacle detection efficiency, and thus improve obstacle tracking efficiency.

[0107] According to a third aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to implement the obstacle tracking method as described above.

[0108] To implement the above embodiments, this disclosure also proposes a storage medium.

[0109] When the instructions in the storage medium are executed by the processor, the processor is able to perform the obstacle tracking method as described above.

[0110] To implement the above embodiments, this disclosure also provides a computer program product.

[0111] When the computer program product is executed by the processor of the electronic device, it enables the electronic device to perform the above-described method.

[0112] Figure 5 This is a structural block diagram of a vehicle 500 according to an exemplary embodiment of the present disclosure. For example, vehicle 500 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. Vehicle 500 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0113] Reference Figure 5The vehicle 500 may include various subsystems, such as an infotainment system 510, a perception system 520, a decision control system 530, a drive system 540, and a computing platform 550. The vehicle 500 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and component of the vehicle 500 can be interconnected via wired or wireless means.

[0114] In some embodiments, the infotainment system 510 may include a communication system, an entertainment system, and a navigation system, etc.

[0115] The perception system 520 may include several sensors for sensing information about the environment surrounding the vehicle 500. For example, the perception system 520 may include a global positioning system (which may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and a camera device.

[0116] The decision control system 530 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0117] The drive system 540 may include components that provide powered motion to the vehicle 500. In one embodiment, the drive system 540 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0118] Some or all of the functions of vehicle 500 are controlled by computing platform 550. Computing platform 550 may include at least one processor 551 and memory 552, and processor 551 may execute instructions 553 stored in memory 552.

[0119] Processor 551 can be any conventional processor, such as a commercially available CPU. Processors may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0120] The memory 552 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0121] In addition to instruction 553, memory 552 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 552 can be used by computing platform 550.

[0122] In this embodiment of the disclosure, processor 551 may execute instructions 553 to complete all or part of the steps of the obstacle tracking method described above.

[0123] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0124] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0125] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0126] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An obstacle tracking method, characterized in that, The method includes: Acquire 3D bounding box information of each object around the vehicle at the current time point, and 3D detection bounding box information of each obstacle around the vehicle at each historical time point; the 3D bounding box information is determined based on multiple images of the vehicle's surroundings at the current time point. For each obstacle, the 3D detection bounding box information of the obstacle at each historical time point is input into the obstacle's tracking filter to determine the 3D prediction bounding box information of the obstacle at the current time point; Based on the 3D target bounding box information and the 3D predicted bounding box information of each obstacle at the current time point, determine the obstacle to which each 3D target bounding box information belongs at the current time point; Based on the 3D predicted bounding box information of each obstacle at the current time point, and the obstacle to which each 3D target bounding box information belongs, the tracking filter of each obstacle is updated to obtain the updated tracking filter of each obstacle. For each obstacle, the 3D detection bounding box information of the obstacle at the current time point and at each previous historical time point is input into the updated tracking filter of the obstacle to obtain the updated 3D prediction bounding box information of the obstacle at the current time point, which is used for tracking the obstacle.

2. The method according to claim 1, characterized in that, Obtain the 3D bounding boxes around the vehicle at the current time point, including: Acquire multiple surrounding images of the vehicle at the current time point; the multiple surrounding images are captured by multiple imaging devices on the vehicle at the current time point; Obtain an obstacle detection model; Multiple surrounding images are input into the obstacle detection model to obtain multiple 3D candidate box information and the confidence level corresponding to the 3D candidate box information; Based on the confidence level, the information of multiple 3D candidate boxes is filtered to obtain the information of each 3D target box at the current time point.

3. The method according to claim 2, characterized in that, The obstacle detection model is trained by combining multiple surrounding images of the vehicle at historical time points, as well as obstacle information around the vehicle in the vehicle coordinate system at those historical time points. The step of inputting multiple surrounding images into the obstacle detection model to obtain multiple 3D candidate bounding box information and the confidence scores corresponding to the 3D candidate bounding box information includes: Multiple surrounding images are input into the obstacle detection model to obtain multiple 3D candidate bounding boxes in the vehicle coordinate system and their corresponding confidence scores. By combining the vehicle's position information in the world coordinate system at the current time point with multiple 3D candidate bounding box information in the vehicle body coordinate system, coordinate transformation processing is performed to obtain multiple 3D candidate bounding box information in the world coordinate system.

4. The method according to claim 2, characterized in that, The process of filtering multiple 3D candidate bounding box information based on the confidence level to obtain the 3D target bounding box information at the current time point includes: Perform ground projection processing on each 3D candidate box information to obtain the ground 2D candidate box information corresponding to each 3D candidate box information; Based on the confidence level of each 3D candidate bounding box, the ground 2D candidate bounding box information is sorted in descending order to obtain the sorting result. Extract the ground 2D candidate box information with the highest confidence from the sorting results, and delete the ground 2D candidate box information in the sorting results whose intersection-union ratio with the ground 2D candidate box information is greater than or equal to the first intersection-union ratio threshold; Repeat the above steps until the ground 2D candidate box information in the sorting result is processed; The 3D candidate box information corresponding to the extracted 2D candidate box information on the ground is used as the 3D target box information at the current time point.

5. The method according to claim 1, characterized in that, The 3D target bounding box information and the 3D detection bounding box information include obstacle categories; determining the obstacle to which each 3D target bounding box belongs at the current time point is determined based on the 3D target bounding box information and the 3D prediction bounding box information of each obstacle at the current time point includes: For each obstacle, acquire the 3D bounding box information to be compared from each 3D bounding box information; the obstacle category in the 3D bounding box information to be compared is consistent with the obstacle category. Determine the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared; The obstacle to which the 3D target bounding box information to be compared belongs is identified among multiple matching degrees with the highest matching degree.

6. The method according to claim 5, characterized in that, The 3D target bounding box information and the 3D predicted bounding box information also include location information; determining the matching degree between the 3D predicted bounding box information of the obstacle and each 3D target bounding box information to be compared includes: Ground projection processing is performed on the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared, to obtain the ground 2D predicted bounding box information corresponding to the 3D predicted bounding box information of the obstacle and the ground 2D target bounding box information corresponding to the 3D target bounding box information to be compared. For each 3D target bounding box to be compared, the distance data between the 3D predicted bounding box of the obstacle and the 3D target bounding box to be compared is determined based on the position information in the 3D target bounding box to be compared and the position information in the 3D predicted bounding box of the obstacle. Based on the ground 2D target bounding box information corresponding to the 3D target bounding box information to be compared, and the ground 2D prediction bounding box information corresponding to the 3D prediction bounding box information of the obstacle, the ground intersection-over-union ratio data between the 3D prediction bounding box information of the obstacle and the 3D target bounding box information to be compared is determined; Based on the distance data and the ground intersection-over-union data, the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared is determined.

7. The method according to claim 6, characterized in that, The determination of the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared further includes: Combining the vehicle's position information in the world coordinate system at the current time point, and the position information of multiple shooting devices on the vehicle relative to the vehicle, the image 2D prediction box information corresponding to the 3D prediction box information of the obstacle, and the image 2D target box information corresponding to each 3D target box information to be compared are determined. By combining multiple surrounding images of the vehicle at the current time point, the image region corresponding to the 2D target bounding box information of each image and the image region corresponding to the 2D prediction bounding box information of the image are determined. For each 2D target bounding box information in an image, determine the intersection-over-union ratio (IoU) between the 2D target bounding box information and the 2D predicted bounding box information, as well as the image similarity data between the image region corresponding to the 2D target bounding box information and the image region corresponding to the 2D predicted bounding box information. The step of determining the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared based on the distance data and the ground intersection-over-union data includes: Based on the distance data, the ground intersection-over-union (IoU) data, the IoU data, and the image similarity data, the matching degree between the 3D predicted bounding box information of the obstacle and the 3D target bounding box information to be compared is determined.

8. The method according to claim 7, characterized in that, The step of combining the vehicle's position information in the world coordinate system at the current time point, and the position information of multiple imaging devices on the vehicle relative to the vehicle, to determine the image 2D prediction box information corresponding to the 3D prediction box information of the obstacle, and the image 2D target box information corresponding to each 3D target box information to be compared, includes: By combining the vehicle's position information in the world coordinate system at the current time point, and the position information of multiple camera devices on the vehicle relative to the vehicle, the position information of multiple camera devices in the world coordinate system at the current time point is determined; Based on the location information of multiple shooting devices, determine the shooting device corresponding to each 3D target bounding box information to be compared, and the shooting device corresponding to the 3D prediction bounding box information of the obstacle; Based on the location information of each shooting device, determine the coordinate transformation relationship between the image coordinate system and the world coordinate system of each shooting device; By combining the coordinate transformation relationship between the image coordinate system and the world coordinate system of each shooting device, the image 2D prediction box information corresponding to the 3D prediction box information of the obstacle, and the image 2D target box information corresponding to each 3D target box information to be compared are determined.

9. The method according to claim 1, characterized in that, The method further includes: The updated 3D prediction bounding box information of the obstacle at the current time point is determined as the 3D detection bounding box information of the obstacle at the current time point.

10. An obstacle tracking device, characterized in that, The device includes: The acquisition module is used to acquire the 3D target bounding box information of each vehicle around the vehicle at the current time point, and the 3D detection bounding box information of each obstacle around the vehicle at each historical time point; the 3D target bounding box information is determined based on multiple surrounding images of the vehicle at the current time point. The first determining module is used to input the 3D detection box information of the obstacle at each historical time point into the tracking filter of the obstacle for each obstacle, and determine the 3D prediction box information of the obstacle at the current time point; The second determining module is used to determine the obstacle to which each 3D target box information belongs at the current time point based on the 3D target box information and the 3D predicted box information of each obstacle at the current time point. The update module is used to update the tracking filter of each obstacle based on the 3D prediction box information of each obstacle at the current time point and the obstacle to which each 3D target box information belongs, so as to obtain the updated tracking filter of each obstacle. The third determining module is used to input the 3D detection box information of the obstacle at the current time point and at each previous historical time point into the updated tracking filter of the obstacle to obtain the updated 3D prediction box information of the obstacle at the current time point, which is used to perform tracking processing on the obstacle.

11. A vehicle, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured as follows: The steps of implementing the obstacle tracking method as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor, enable the processor to perform the obstacle tracking method as described in any one of claims 1 to 9.

Images