Target tracking method and device based on multi-source visual fusion, product and vehicle

Abstract

The invention provides a target tracking method and device based on multi-source visual fusion, a product and a vehicle, and belongs to the technical field of automatic driving. The method comprises the steps that each detection result in a plurality of detection results at the current moment is matched with each current track, and the detection results comprise a look-around detection result and a monocular detection result; and when any track is successfully matched with one look-around detection result and one monocular detection result at the same time, executing a detection result fusion strategy, determining a fusion detection result corresponding to the look-around detection result and the monocular detection result, and updating the track according to the fusion detection result. The objective of the invention is to improve the accuracy of target tracking.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of autonomous driving, and more specifically, to a target tracking method, device, product, and vehicle based on multi-source visual fusion. Background Art

[0002] In the field of intelligent transportation technology, especially autonomous driving technology, target detection and tracking technology based on multi-source sensors is particularly important. The existing multi-source visual detection and tracking methods limit the overall performance of target detection and tracking, and the accuracy of target tracking is low. Summary of the Invention

[0003] The embodiments of the present application provide a target tracking method, device, product and vehicle based on multi-source visual fusion, aiming to improve the accuracy of target tracking.

[0004] In a first aspect, an embodiment of the present application provides a target tracking method based on multi-source visual fusion, the method comprising: Matching each of the multiple detection results at the current moment with each current trajectory, wherein the detection results include surround detection results and monocular detection results; When any trajectory successfully matches a surround view detection result and a monocular detection result at the same time, the detection result fusion strategy is executed to determine the fused detection result corresponding to the surround view detection result and the monocular detection result, and the trajectory is updated according to the fused detection result.

[0005] In a second aspect, an embodiment of the present application provides an electronic device comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the target tracking method based on multi-source visual fusion described in the first aspect of the embodiment is executed.

[0006] In a third aspect, an embodiment of the present application provides a non-volatile readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, it executes the target tracking method based on multi-source visual fusion described in the first aspect of the embodiment.

[0007] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the target tracking method based on multi-source visual fusion described in the first aspect of the embodiment.

[0008] In a fifth aspect, an embodiment of the present application provides a vehicle, which includes the target tracking method based on multi-source visual fusion described in the first aspect of the embodiment.

[0009] Beneficial effects: When tracking a target, this method obtains multiple detection results at the current moment, including surround view detection results and monocular detection results from different detection sources, and then matches each of the multiple detection results with each current trajectory. When any trajectory successfully matches a surround view detection result and a monocular detection result at the same time, a detection result fusion strategy is executed to determine the fused detection result corresponding to the surround view detection result and the monocular detection result. Based on the fused detection result, the trajectory is updated. By fusing the detection results of different detection sources and integrating the strengths of the detection results of different detection sources, an update of the visual target trajectory with better performance and stronger stability is achieved, thereby improving the accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 This is a flowchart of the steps of a target tracking method based on multi-source visual fusion proposed in one embodiment of the present application; Figure 2 This is a flow chart of the preprocessing of the test results proposed in one embodiment of the present application; Figure 3 is a schematic diagram of BEV ray filtering provided by an embodiment of the present application; Figure 4 This is a flow chart of matching detection results and trajectories proposed in one embodiment of the present application; Figure 5 This is a schematic diagram of ray direction distance calculation proposed in an embodiment of the present application; Figure 6 This is a schematic diagram of a preset type matching provided in an embodiment of the present application; Figure 7 This is a schematic diagram of the detection result fusion proposed in one embodiment of the present application; Figure 8 This is a flow chart of the detection result fusion process proposed in one embodiment of the present application; Figure 9 This is a schematic diagram of size update proposed in one embodiment of the present application; Figure 10 This is a schematic diagram of trajectory status update provided by an embodiment of the present application; Figure 11 Schematic diagram of trajectory state update in an occlusion scenario proposed in one embodiment of the present application; Figure 12This is an overall flow chart of a target tracking method based on multi-source visual fusion proposed in one embodiment of the present application; Figure 13 This is a functional module diagram of a target tracking device based on multi-source visual fusion proposed in one embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0013] In the field of intelligent transportation technology, especially autonomous driving technology, target detection and tracking technology of multi-source sensors is particularly important. There are only a few existing multi-source visual detection and tracking methods, and the characteristics of the visual detection source are rarely considered in the target association and matching stage. Some of them are indirectly fused based on the error distribution under a single detection source, or each visual detection source is treated equally in the fusion stage of multiple detection sources, which limits the overall performance of target detection and tracking and reduces the accuracy of target tracking.

[0014] For example, common target detection and tracking technologies include the following: Using a 2D detection and tracking method, the 3D position of the target is first estimated using the prior target size and camera parameters. Assuming that the positioning error of the target under each camera follows a Gaussian distribution and that the fused state still follows a Gaussian distribution, the mean and variance of the position under a single camera are first estimated, and then a new mean is obtained by fusion. This method is significantly affected by differences in target size, occlusion, and truncation on the 2D image. In addition, the mean and variance of the filter output under a single detection source lag, resulting in a large overall error.

[0015] There is also target features extracted by the encoder to perform fusion tracking of lidar and camera targets. However, on the one hand, the cost of lidar is relatively high. On the other hand, when performing detection and tracking tasks, there is a conflict between the training goal of the detection task is to maximize the inter-class feature distance, while the goal of the tracking task is to maximize the intra-class feature distance. Directly using the detection features will lead to performance degradation, and separately adding a decoupling module and outputting target features will significantly increase the amount of calculation and delay.

[0016] Alternatively, a single camera is first tracked for the target of each camera source, and similarity is calculated based on position and size, and cross-camera correlation matching and target fusion are performed. This method increases the amount of calculation and cumulative error due to multiple tracking. In addition, the characteristics of the visual detection source are not considered during matching, and each visual detection source is treated equally in the fusion stage, resulting in incomplete trajectories and large position deviations.

[0017] Currently, most target detection and tracking are based on the fusion of millimeter-wave radar / lidar targets and visual targets. Millimeter-wave radar can relatively accurately detect distant moving objects and speeds, but cannot accurately capture target categories, and has poor performance for stationary targets. It is also prone to false alarms and inflation for highly reflective targets, especially in scenes such as tunnels or road signs. Lidar has more accurate ranging capabilities, but cannot accurately capture target categories and attributes. Telephoto cameras can capture targets at greater distances. In addition, mechanical lidars that can detect 360° targets are much more expensive than cameras. Cameras have inferior ranging performance to millimeter-wave / lidars, but have fewer other inherent limitations, and their target characteristics mainly depend on how they are used.

[0018] Therefore, the embodiment of the present application provides a target tracking method based on multi-source visual fusion, which can improve the overall performance of target detection and tracking and improve the accuracy of target tracking.

[0019] Reference Figure 1 , shows a flowchart of the steps of a target tracking method based on multi-source visual fusion in an embodiment of the present application, and the method may specifically include the following steps: S101: Matching each of a plurality of detection results at the current moment with each current trajectory, wherein the detection results include surround detection results and monocular detection results.

[0020] Specifically, during the target tracking process, multiple detection results are obtained at the current moment. The multiple detection results include detection results from different detection sources. The detection results include surround detection results and monocular detection results. Among them, the surround detection result is the BEV (Bird's-Eye View) detection result.

[0021] For example, the monocular detection result can be the detection result obtained after 3D target detection is performed on the image captured by a telephoto camera; the surround detection result can be the detection result obtained after 3D target detection is performed on the images acquired by multiple cameras, such as the surround view image capture by 7 cameras including front width, front narrow, left front, left rear, right front, right rear and rear, and then the 360-degree detection result of the main vehicle's driving environment is output after 3D target detection.

[0022] In the actual implementation process, when performing 3D target detection based on the collected images, a conventional 3D target detection model can be used to process the images, and the 3D target detection model outputs the detection results. The type of 3D target detection model can be selected according to the needs of the actual application.

[0023] In actual implementation, the surround view detection result may be a 10 Hz BEV surround view detection result, where one surround view detection result includes a 3D detection result of obstacles in the main vehicle coordinate system.

[0024] The monocular detection result can be a 10Hz long-focus monocular detection result. A monocular detection result includes the 2D detection result of the target in the long-focus monocular image, the 3D detection result of the target in the main vehicle coordinate system, and the visual ratio of the detection result. The visual ratio indicates the completeness of the target in the long-focus monocular image.

[0025] Among them, the 3D detection results in the surround detection results and monocular detection results include parameters in multiple dimensions such as the target's xyz coordinates, length, width, height, angle, type, and detection score. Among them, the detection score is used to characterize the reliability of the detection results output by the 3D target detection model.

[0026] In the actual implementation process, when performing target tracking, it is also necessary to obtain the 100Hz positioning information of the main vehicle in the world coordinate system. The positioning information is used to represent the position and direction of the vehicle.

[0027] Reference Figure 2 , shows a schematic diagram of the process of preprocessing the detection results provided by an embodiment of the present application. In a feasible implementation, after obtaining multiple detection results at the current moment, the multiple detection results can be preprocessed. The preprocessing process includes: First, a score filtering strategy is executed based on the detection scores corresponding to the detection results at the current moment. The score filtering strategy includes: filtering the surround detection results whose detection scores are less than the first filtering threshold; and filtering the monocular detection results whose detection scores are less than the second filtering threshold.

[0028] The first filtering threshold and the second filtering threshold may be set according to the upstream 3D object detection model used in the actual implementation process. For example, the first filtering threshold may be set to 0.27, and the second filtering threshold may be set to 0.5.

[0029] Through the score filtering strategy, detection results with low detection reliability can be filtered out, thereby avoiding the detection results with low detection reliability occupying processing resources in the subsequent processing process. In addition, the detection results with low detection reliability have a greater interference with target tracking. After filtering out the detection results with low detection reliability, the reliability and accuracy of target tracking can be further improved.

[0030] Furthermore, a ray filtering strategy can be implemented. Due to the characteristics of BEV, there are usually multiple repeated frames in the ray direction of the target, and these multiple repeated frames actually represent the same target. Therefore, repeated surround detection results can be filtered.

[0031] Specifically, after filtering the surround view detection results whose detection scores are less than the first filtering threshold, multiple surround view detection results are sorted in descending order according to the detection scores, and then a ray filtering strategy is executed on each surround view detection result in turn. The ray filtering strategy is used to filter duplicate surround view detection results.

[0032] In a feasible implementation, executing the ray filtering strategy includes the following steps: A1: For each surround view detection result, calculate the occupancy angle between the surround view detection result and the host vehicle.

[0033] Specifically, for any surround view detection result, the angle between each corner point in the surround view detection result and the main vehicle is calculated. The calculation formula for the angle between each corner point and the main vehicle is as follows:

[0034] in, angle is the angle between the corner point and the host vehicle; x , y is the coordinate of any corner point in the main vehicle coordinate system.

[0035] After calculating the angles between all corner points of the 3D box in the surround view detection result and the main vehicle, the range between the minimum and maximum angles can be used as the occupancy angle between the surround view detection result and the main vehicle.

[0036] A2: Calculate the overlap between the occupancy angle of the surround view detection result and the occupancy angle of each surround view detection result in the surround view detection result queue.

[0037] Specifically, the calculation formula for the overlap between the occupied angles of any two surround view detection results is:

[0038] Among them, the occupancy angle of any surround detection result is θ max arrive θ min , θ 1 and θ 2 represent the two occupied angles used to calculate the coincidence; iou Indicates the degree of overlap; for example, the occupied angle θ1 is θ 1min -θ 1max , the occupied angle θ2 is θ 2min - θ 2max .

[0039] A3: When there is any surround view detection result in the surround view detection result queue whose overlap with the surround view detection result is greater than the overlap threshold, and the distance between the two surround view detection results is less than the target ratio of the detection result depth, the surround view detection result is characterized as a duplicate detection result and is discarded.

[0040] For example, the overlap threshold can be set according to the type of the target. For example, when the target is a vehicle, the overlap threshold is set to 0.9; when the target is other objects other than a vehicle, the overlap threshold is set to 0.1.

[0041] The depth of the detection result is used to characterize the distance from the target to the vehicle. If the targets of two detection results are close enough and the overlap of the occupied angles is greater than the overlap threshold, the two detection results are considered to be duplicate detection results. For example, the target ratio of the detection result depth can be set to 4%.

[0042] When the degree of overlap between the occupied angle of the surround view detection result and the occupied angle of any surround view detection result in the surround view detection result queue is greater than the overlap threshold, and the distance between the detections is less than 4% of the detection result depth, the surround view detection result is a duplicate detection result and is discarded.

[0043] A4: When there is no surround view detection result in the surround view detection result queue and the overlap between the surround view detection result and the surround view detection result is greater than the overlap threshold, add the surround view detection result to the surround view detection result queue.

[0044] When there is no surround view detection result in the surround view detection result queue and the overlap between the surround view detection result and the surround view detection result is greater than the overlap threshold, it indicates that the surround view detection result and the surround view detection result in the surround view detection result queue are not repeated, and the surround view detection result is added to the surround view detection result queue.

[0045] After filtering out surround view detection results with lower detection reliability based on the detection score, the surround view detection results with the highest detection scores can be retained among multiple surround view detection results of repeated detections in descending order of detection scores. This not only removes redundancy from repeated detections, but also improves the accuracy and reliability of subsequent target tracking and reduces the processing resources occupied by repeated detection results.

[0046] Reference Figure 3 , showing a schematic diagram of BEV ray filtering provided by an embodiment of the present application, Figure 3The figure visualizes the corresponding occupied angle ranges of surround view detection results 1 and 2. The overlap of the occupied angles between surround view detection results 1 and 2 is 0, and surround view detection results 1 and 2 are not repeated detections; while the overlap of the occupied angles of surround view detection results 2 and 3 is large, which means they are repeated detections. After executing the ray filtering strategy, one of the detection results with a low detection score will be filtered out and discarded.

[0047] Reference Figure 4 , shows a flow chart of matching the detection results and trajectories provided in an embodiment of the present application. In a feasible implementation, the detection results and trajectories can undergo three rounds of matching processes.

[0048] In the first round of matching: the pre-processed detection results are respectively matched with all stable trajectories to perform the ray direction matching strategy. The successfully matched detection results and trajectories do not participate in the subsequent second round of matching process.

[0049] In the second round of matching, for each detection result that does not match a stable trajectory, a ray direction matching strategy is performed with the stable trajectory and unstable trajectory that do not match the detection result. The detection results and trajectories that successfully match do not participate in the third round of matching. In this embodiment, unstable trajectories include unstable trajectories, continued trajectories, and hidden trajectories.

[0050] In the third round of matching: For monocular detection results that are not matched to a track, the 2D matching strategy is executed with the track that is not matched to the detection result.

[0051] Among them, each surround view detection result and each monocular detection result must be matched with the trajectory separately. Because the surround view detection result does not include the 2D detection result, the surround view detection result only needs to perform the first and second rounds of matching, and only the 2D detection result in the monocular detection result performs the third round of matching.

[0052] In a feasible implementation, any detection result includes a surround detection result and a monocular detection result, and the process of executing a ray direction matching strategy with any trajectory may include the following steps: B1: Calculate the distance between each detection result and each trajectory.

[0053] Specifically, the ray direction distance, angular distance, size distance and type distance between each detection result and each trajectory can be determined respectively; then, the distance between each detection result and each trajectory is calculated based on the ray direction distance, the angular distance, the size distance and the type distance.

[0054] Specifically, the process of calculating the distance between a detection result and a trajectory includes the following sub-steps: B11: Determine the ray direction distance between each detection result and each trajectory.

[0055] Reference Figure 5 , shows a schematic diagram of ray direction distance calculation provided by an embodiment of the present application, such as Figure 5 As shown, the ray distances from the points on the ellipse to the trajectory are the same, and the distance between the detection result and the trajectory can be decomposed into the ray direction and the vertical direction between the main vehicle and the detection result.

[0056] Then, the ray direction distance between the detection result and the trajectory is calculated based on the distance between the ray direction and the perpendicular direction to the ray direction.

[0057] For example, the formula for ray direction distance is:

[0058] in, track Indicates trajectory; det Indicates the test results; track x and track y The coordinates of the nearest corner of the trajectory in the main vehicle coordinate system; det x and det y The coordinates of the nearest corner point of the detection result in the main vehicle coordinate system; d x and d y Represent the coordinate difference between the nearest corner point of the trajectory and the detection result in the main vehicle coordinate system; d_dir x and d_dir y Respectively represent the decomposed distances in the ray direction of the main vehicle and the detection result; heading Indicates the angle in the test result; d Indicates the ray direction distance between the trajectory and the detection result.

[0059] B12: Determine the angular distance between each detection result and each trajectory.

[0060] Specifically, the angular distance actually evaluates the lateral deviation between the trajectory and the detection result caused by the angle. The angular distance between the trajectory and the detection result can be calculated based on the angular difference between the trajectory and the detection result and the length of the trajectory size.

[0061] The calculation formula of the angular distance is:

[0062] in, d° Indicates angular distance; size track.l The length indicating the track size; heading track represents the trajectory angle; heading det Indicates the angle of the test result; abs () is the absolute value function.

[0063] B13: Determine the size distance between each detection result and each trajectory.

[0064] Specifically, the size distance can be calculated based on the difference in length between the trajectory size and the detection result size; The calculation formula of the size distance includes:

[0065] in, d size Indicates size distance; size track.l The length indicating the track size; size det.l Indicates the length of the test result size; abs () is the absolute value function.

[0066] B14: Determine the type distance between each detection result and each trajectory.

[0067] Specifically, the detection result includes the type of the target, and it is determined whether the type of the trajectory is the same as the type of the detection result.

[0068] When the type of the detection result is the same as the type of the trajectory, the type distance between the detection result and the trajectory is 0.

[0069] When the type of the detection result is different from the type of the trajectory, and the type of the detection result and the type of the trajectory belong to a preset type match, the type distance between the detection result and the trajectory is 1.

[0070] When the type of the detection result is different from the type of the trajectory, and the type of the detection result and the type of the trajectory do not belong to the preset type match, it is determined that the detection result and the trajectory are not matched successfully. For example, the type distance can be set to infinity. If the type distance is infinity, it is deemed that the detection result is not allowed to match the trajectory.

[0071] In actual implementation, the preset type matching can be set according to the needs of the actual application.

[0072] Reference Figure 6, shows a schematic diagram of the preset type matching provided by the implementation of this application, such as Figure 6 As shown, the preset type matching is: between types with direct connections, the type distance can be set to 1.

[0073] Specifically, the type distances between cars, trucks, or buses are all 1; the type distances between tricycles and cars, trucks, and buses are all 1; the type distance between tricycles and two-wheeled vehicles is 1; and the type distance between two-wheeled vehicles and pedestrians is 1.

[0074] For example, if the type of a surround view detection result or monocular detection result is a pedestrian, if the type of the track is a two-wheeled vehicle, the type distance between the detection result and the track is 1; if the type of the track is any of the three-wheeled vehicles, cars, trucks, or buses, the type distance between the detection result and the track is infinite, that is, the detection result cannot be successfully matched with the track.

[0075] B15: Calculate the distance between each detection result and each trajectory according to the ray direction distance, the angle distance, the size distance, and the type distance.

[0076] For example, the distance between the detection result and the trajectory may be obtained by summing the ray direction distance, the angle distance, the size distance, and the type distance.

[0077] The ray direction distance, the angular distance, the size distance and the type distance may also be weightedly summed to obtain the distance between the detection result and the trajectory; when using the weighted summation, the weights corresponding to the ray direction distance, the angular distance, the size distance and the type distance may be customized according to the needs of the actual application.

[0078] B2: Among all detection results whose corresponding distance to any track is less than the distance threshold, determine the detection result that successfully matches the track, and calculate the matching score between the track and the successfully matched detection result.

[0079] The distance threshold is set according to actual application requirements. For example, in this implementation, the distance threshold is set to 5.

[0080] In the actual implementation process, for a trajectory, there may be multiple detection results whose distances are less than the distance threshold. Therefore, for all detection results corresponding to any trajectory whose distances are less than the distance threshold, the Hungarian matching algorithm can be used to determine the detection results that successfully match the trajectory in different detection sources.

[0081] For example, if the detection results of a trajectory with distances less than the distance threshold include multiple surround view detection results and / or multiple monocular detection results, then all surround view detection results and all monocular detection results are classified according to different detection sources, and the Hungarian matching algorithm is used to determine the successfully matched detection results among all surround view detection results and the successfully matched detection results among all monocular detection results.

[0082] The Hungarian matching algorithm can use the minimum cost to determine the detection result that successfully matches the trajectory. Moreover, the detection result that successfully matches the trajectory determined by the Hungarian matching algorithm is unique, so that each trajectory can only be matched with at most one detection result from the same detection source.

[0083] For example, if a track 1 successfully matches a detection result, it includes: track 1 successfully matches a monocular detection result, or track 1 successfully matches a surround view detection result, or track 1 successfully matches a surround view detection result and a monocular detection result.

[0084] Specifically, after determining the detection result that matches the trajectory, the matching score between the trajectory and the successfully matched detection result is calculated based on the distance threshold and the distance between the detection result that successfully matches the trajectory and the trajectory.

[0085] The formula for calculating the matching score is:

[0086] in, score dist represents the matching score; dist is the distance between the detection result and the trajectory; threshold is the distance threshold.

[0087] In a feasible implementation, the process of executing a 2D matching strategy on any monocular detection result that is not matched to a track and the track that is not matched to the detection result includes: For any track that is not matched to a detection result, the 2D intersection-over-union (IoU) ratio between the track and the 2D detection results in each monocular detection result that is not matched to the track is calculated respectively. Based on multiple monocular detection results whose 2D IoU ratio corresponding to the track is greater than the IoU threshold, the monocular detection result that successfully matches the track is determined based on the Hungarian matching algorithm, and the matching score between the track and the successfully matched monocular detection result is calculated.

[0088] Specifically, the intersection-over-union (IoU) threshold can be customized according to the needs of the actual application. In this embodiment, the IoU threshold is set to 0.4. That is, for monocular detection results with a 2D IoU greater than 0.4, a monocular detection result that successfully matches the trajectory is determined based on the Hungarian matching algorithm. The matching score between the monocular detection result and the trajectory is calculated based on the distance between the monocular detection result and the trajectory after executing the ray direction matching strategy.

[0089] For example, monocular detection result 1 fails to match trajectory 1 successfully in the first and second rounds of matching after executing the ray direction matching strategy, but the distance between monocular detection result 1 and trajectory 1 has been calculated. If, in the third round of matching, monocular detection result 1 and trajectory 1 are successfully matched after executing the 2D matching strategy, the matching score between the monocular detection result and the trajectory is calculated based on the calculated distance and matching score formula.

[0090] If the monocular detection result still does not match any track in the third round of matching, the monocular detection result matching fails.

[0091] For example, it is assumed that a stable trajectory 1, a stable trajectory 2, an unstable trajectory 1, and a continuous trajectory 1 are included, and the detection results include a surround view detection result 1, a surround view detection result 2, a monocular detection result 1, and a monocular detection result 2.

[0092] In the first round of matching between detection results and stable results, surround view detection result 1 is successfully matched with stable trajectory 1 and stable trajectory 2 after being matched with surround view detection result 1, and the matching score is calculated as A. Surround view detection result 2, monocular detection result 1, and monocular detection result 2 are all matched with stable trajectory 1 and stable trajectory 2 respectively, but no match is successful. At this time, surround view detection result 2, monocular detection result 1, and monocular detection result 2 are the detection results that need to participate in the second round of matching. The trajectories participating in the second round of matching include stable trajectory 1, which was not successfully matched in the first round, and other unstable trajectories: unstable trajectory 1 and continued trajectory 1.

[0093] Assuming that surround view detection result 2 does not match any track in the second round of matching, surround view detection result 2 fails to match.

[0094] Assume that monocular detection result 1 is successfully matched to unstable trajectory 1 in the second round of matching, while monocular detection result 2 is not matched to any trajectory. Monocular detection result 2 enters the third round of matching. The trajectories entering the third round of matching include stable trajectory 1 and continued trajectory 1.

[0095] If the monocular detection result 2 still does not match any track in the third round of matching, the monocular detection result 2 fails to match.

[0096] This embodiment uses three rounds of matching to prioritize 3D matching between surround view detection results and monocular detection results and stable trajectories, which can improve matching efficiency. Successfully matched detection results and trajectories do not need to undergo subsequent matching.

[0097] The stable trajectories and detection results that failed to match in the first round can be matched again in the second round, so that the detection results that failed to match can be matched again with the stable trajectories that failed to match, avoiding matching omissions.

[0098] S102: When any trajectory successfully matches a surround view detection result and a monocular detection result at the same time, a detection result fusion strategy is executed to determine a fused detection result corresponding to the surround view detection result and the monocular detection result, and the trajectory is updated according to the fused detection result.

[0099] Reference Figure 7 , showing a schematic diagram of the detection result fusion provided by an embodiment of the present application. If, during the matching process, a trajectory successfully matches both a surround view detection result and a monocular detection result, it indicates that both the surround view detection result and the monocular detection result are detection results suitable for the trajectory. The surround view detection result and the monocular detection result can be fused so that the fused detection result has the detection performance of different detection sources at the same time. By combining the strengths of different detection sources, better performance and more stable visual target trajectory tracking can be achieved.

[0100] Reference Figure 8 , shows a schematic diagram of the process of detection result fusion provided in an embodiment of the present application. Specifically, when executing the detection result fusion strategy, the corner points, sizes and angles of the surround detection results and the monocular detection results are fused respectively, and finally the final fused detection result is obtained.

[0101] In a feasible implementation, when any trajectory successfully matches a surround view detection result and a monocular detection result at the same time, executing the detection result fusion strategy to determine the fused detection result corresponding to the surround view detection result and the monocular detection result includes the following steps: C1: Calculate the target scores corresponding to the surround detection result and the monocular detection result.

[0102] Specifically, the target scores corresponding to the surround view detection result and the monocular detection result are calculated based on their respective detection scores, the matching scores with the trajectory, and the visible ratio in the detection results.

[0103] For example, the formula for calculating the target score is:

[0104] in, score calib Indicates the target score corresponding to any detection result, including the target score of the surround detection result score bev And the target score of the monocular detection result score mono ; score det Indicates the detection score in the detection result; score dist Represents the matching score between the detection result and the trajectory; visible_ratio Indicates the visible ratio in the detection results.

[0105] Specifically, the visible ratio of the monocular detection result is directly output by the 3D object detection model that generates the monocular detection result, and the visible ratio is included in the monocular detection result. The surround view detection result is generated based on the surround view images collected by multiple cameras. The detection result of any target is complete, and there is no problem of occlusion or loss of collection. Therefore, the visible ratio of the surround view detection result is always 1.

[0106] In the actual implementation process, score det This parameter can reflect the reliability of the detection result. In addition to using the detection score of the detection result, it can also be the difference between the detection score in a detection result and the corresponding threshold. For example, the difference between the detection score of the surround detection result and the first filtering threshold (such as 0.27) and the difference between the detection score of the monocular detection result and the second filtering threshold (such as 0.5) can be used.

[0107] The target score is calculated by the detection score corresponding to each detection result, the matching score between the detection result and the trajectory, and the visible ratio in the detection result. It can reflect the detection result with better performance among the surround detection results and monocular detection results matched by the trajectory at the same time. The target score will participate in the subsequent fusion process as the fusion weight of each detection result, so that the fusion process can be based on the detection result of the detection source with better performance, and then combined with the strengths of the other detection result to achieve higher accuracy and better effect in target tracking.

[0108] C2: According to the target score, determine the corner fusion result, size fusion result, and angle fusion result corresponding to the surround detection result and the monocular detection result respectively.

[0109] Specifically, the method includes the following sub-steps: C21: Determine the corner point fusion result corresponding to the surround detection result and the monocular detection result.

[0110] First, the corner point of the surround view detection result closest to the main vehicle and the corner point of the monocular detection result closest to the main vehicle are selected as the fusion corner points of their respective detection results.

[0111] Secondly, the target scores corresponding to the surround detection result and the monocular detection result are used as the corner point fusion weights corresponding to the detection results.

[0112] Then, according to the corner fusion weights and fusion corner points corresponding to the surround view detection result and the monocular detection result, a corner fusion result corresponding to the surround view detection result and the monocular detection result is calculated.

[0113] For example, the calculation formula of the corner fusion result is:

[0114] in, corner Indicates the corner fusion result; corner bev Indicates the fusion corner point of the surround detection result; corner mono Indicates the fusion corner point of the monocular detection result; score bev is the target score of the surround detection result; score mono is the target score of the monocular detection result.

[0115] C22: Determine the size fusion result corresponding to the surround detection result and the monocular detection result.

[0116] According to all historical matching detection results of the trajectory, according to the target scores corresponding to each historical matching detection result, the historical matching detection results of the target number with the highest target score are selected, and the size average of the historical matching detection results of the target number is used as the size fusion result.

[0117] Reference Figure 9 , shows a size update schematic diagram provided by an embodiment of the present application. Based on all historical matching detection results of the trajectory, the size average of the detection results with the highest number of targets is selected as the detection size at the current moment. For example, the number of targets can be set to 10, that is, the 10 detection results with the highest target scores are selected, and the size average of the 10 detection results is used as the size at the current moment.

[0118] The detection results with the highest number of targets may include both surround detection results and monocular detection results.

[0119] C23: Determine an angle fusion result corresponding to the surround view detection result and the monocular detection result.

[0120] First, according to the angle of the detection result matched by the trajectory in the previous frame, the angles of the surround detection result and the monocular detection result are corrected to obtain the corrected angles of each detection result.

[0121] That is, based on the angle of the detection result matched by the trajectory in the previous frame, the cases where the angle is obviously problematic are filtered out first. For example, the formula for correcting the angle is:

[0122] in, heading' Indicates the angle after the test result is corrected; heading is the angle for the test result; last_ heading is the angle of the detection result matched by the trajectory in the previous frame; N is the angle deviation value.

[0123] In this embodiment, the angle deviation value can be set according to the needs of actual application. In this embodiment, the angle deviation value is set to 8°, that is, when the angle of the detection result is obviously inconsistent with the angle of the trajectory, the trajectory angle can be set to the angle of the detection result.

[0124] Then, according to the corrected angles corresponding to the surround view detection result and the monocular detection result, the target scores corresponding to the surround view detection result and the monocular detection result are used as angle fusion weights to calculate the angle fusion result corresponding to the surround view detection result and the monocular detection result.

[0125] For example, the calculation formula of the angle fusion result is:

[0126] in, heading* Indicates the angle fusion result; score bev is the target score of the surround detection result; score mono is the target score of the monocular detection result; heading bev The angle after correction of surround detection result; heading mono is the corrected angle of the monocular detection result.

[0127] C3: Determine a fusion detection result corresponding to the surround detection result and the monocular detection result based on the corner fusion result, the size fusion result, and the angle fusion result.

[0128] For example, based on the corner fusion result, the size fusion result, and the angle fusion result, a calculation formula for determining the fusion detection result corresponding to the surround view detection result and the monocular detection result is:

[0129] in, c x and c y Respectively represent the coordinates of each corner point in the fusion detection result; corner x and corner y is the coordinate of the corner point fusion result; size l and size w Respectively represent the length and width of the size fusion result; heading* Indicates the angle fusion result.

[0130] In the actual implementation process, the positive and negative signs in the above formula are determined according to the specific positions of the corner points. Because the size changes during the fusion of the two detection results, it is necessary to adjust the corner points in the fusion detection results according to the size change. The corner point is a rectangular projection of the 3D detection result on the BEV top view. The corner points include the left front, left rear, right front, and right rear. A set of positive and negative signs in the above formula can determine the corner point in a direction, such as c x and c y When both are positive, they represent the right front corner point. c x and c y If both are negative, it represents the left rear corner point.

[0131] After determining a fusion detection result corresponding to the surround view detection result and the monocular detection result, the trajectory is updated according to the fusion detection result.

[0132] Specifically, each trajectory has a corresponding filter, which is used to smooth the detection results and provide a speed estimate. By inputting the fused detection results into the filter of the trajectory, the state of the filter is updated. Based on the fused detection results and the filter of the trajectory, the speed estimation result and convergence status of the trajectory at the current moment are output. For example, the speed estimation result includes the position, velocity and acceleration of the target.

[0133] The convergence status is determined by the covariance of the filter estimate. The convergence status is used to characterize whether the target speed is accurate. For example, the convergence status can be judged based on the covariance and the covariance threshold. If the covariance is less than the covariance threshold, the convergence status is converged, indicating that the speed trajectory result is reliable; if the covariance is greater than or equal to the covariance threshold, the convergence status is not converged, indicating that the speed trajectory result is unreliable.

[0134] In this embodiment, the trajectory filter adopts Kalman filtering. In actual implementation, other filters may also be adopted.

[0135] After matching each detection result with each trajectory, if any trajectory only successfully matches one detection result, such as a surround view detection result or a monocular detection result, the detection result matched by the trajectory is input into the trajectory filter. Based on the detection result matched by the trajectory and the trajectory filter, the velocity estimation result and convergence status of the trajectory at the current moment are output.

[0136] After matching each detection result with each trajectory, if any detection result fails to match any trajectory, a new trajectory is generated based on the detection result. Based on the filter of the new trajectory and the detection result, the velocity estimation result and convergence status of the new trajectory at the current moment are output.

[0137] In a feasible implementation, after matching each of the preprocessed detection results with each current trajectory, the method further includes: updating a status of each trajectory, where the status of the trajectory is used to indicate whether the trajectory is a stable trajectory.

[0138] In this embodiment, the trajectory is divided into five types according to its status: stable trajectory, unstable trajectory, continuous trajectory, hidden trajectory and extinct trajectory, among which: A stable trajectory refers to a stable and reliable trajectory. The sources of stable trajectories include initialized trajectories, i.e., newly created trajectories, and any of unstable trajectories, continued trajectories, and hidden trajectories can be updated to stable trajectories.

[0139] Unstable trajectories refer to trajectories that may contain false positives. The sources of unstable trajectories include initialized trajectories, i.e., newly created trajectories.

[0140] A continued trajectory refers to a trajectory that has not matched any detection results in a short period of time. It is a form of continuation of a stable trajectory. The source of the continued trajectory is the stable trajectory.

[0141] Hidden trajectories refer to trajectories that have not been matched to detection results in multiple consecutive frames. They are another form of existence of stable trajectories. The source of hidden trajectories is the continued trajectory.

[0142] Extinct trajectories refer to unreliable trajectories that will be discarded. The sources of extinct trajectories are hidden trajectories or unstable trajectories.

[0143] In a feasible implementation, stable trajectories and continued trajectories can be output, and disappeared trajectories can be discarded; unstable trajectories and hidden trajectories are not output; all retained trajectories, including all stable trajectories, unstable trajectories, continued trajectories, and hidden trajectories, are used as the trajectories of the next frame, and these retained trajectories will participate in the matching of the detection results of the next frame.

[0144] Reference Figure 10 , which shows a schematic diagram of track status update provided by an embodiment of the present application. The process of updating the status of each track includes: When any trajectory is initialized, that is, when any detection result does not match an existing trajectory and a new trajectory is created, if the trajectory is initialized by a high-scoring detection result, the trajectory is a stable trajectory; if the trajectory is initialized by a low-scoring detection result, the trajectory is an unstable trajectory.

[0145] The judgment of high-score detection results and low-score detection results can be made based on the detection score and score threshold of the detection result. Specifically, when the trajectory is generated based on the detection result with a detection score greater than or equal to the score threshold, the trajectory is a stable trajectory; when the trajectory is generated based on the detection result with a detection score less than the score threshold, the trajectory is an unstable trajectory.

[0146] For example, the score threshold can be set according to the needs of actual applications. In this embodiment, the score threshold of the surround detection result is set to 0.35, and the score threshold of the monocular detection result is set to 0.6.

[0147] If the trajectory is initialized based on the surround view detection result, a detection score greater than 0.35 in the surround view detection result is a high-score detection result, and the trajectory is stable. A detection score less than or equal to 0.35 is a low-score detection result, and the trajectory is unstable.

[0148] If the trajectory is initialized based on the monocular detection result, a detection score greater than 0.6 in the monocular detection result is a high-score detection result, and the trajectory is a stable trajectory; a detection score less than or equal to 0.6 is a low-score detection result, and the trajectory is an unstable trajectory.

[0149] When any stable trajectory fails to successfully match any detection result within the first time period, the state of the stable trajectory is updated to a continued trajectory.

[0150] When any continued trajectory fails to successfully match any detection result within multiple consecutive frames, the status of the continued trajectory is updated to a hidden trajectory. For example, it can be updated to a hidden trajectory when it fails to successfully match any detection result within two consecutive frames.

[0151] When any hidden track fails to match any detection result within the second time period, or any unstable track fails to match any detection result within the third time period, the status is updated to an extinct track and discarded.

[0152] During actual implementation, the first time period, the number of consecutive multiple frames, the second time period, and the third time period can be set according to actual application requirements, and this embodiment does not impose any restrictions.

[0153] When any continued trajectory or hidden trajectory successfully matches any detection result, the status is updated to a stable trajectory.

[0154] When any unstable trajectory matches any detection result in multiple consecutive frames, or the detection score of the successfully matched detection result is greater than or equal to the score threshold, the status of the unstable trajectory is updated to a stable trajectory.

[0155] Reference Figure 11 , shows a schematic diagram of the trajectory state update in the occlusion scenario provided by an embodiment of the present application. For the main vehicle, the trajectory of a target is a stable trajectory. When the target is moving, it is temporarily unable to obtain detection results to match the trajectory due to occlusion by obstacles, etc. Therefore, during the period of being blocked by obstacles, the target's trajectory will be updated from a stable trajectory to a continued trajectory or a hidden trajectory. When the target is no longer blocked by obstacles, the continued trajectory or the hidden trajectory will be updated to a stable trajectory because it matches the detection results again.

[0156] Therefore, by setting up a continued trajectory or hidden trajectory as a way to maintain a stable trajectory, the target trajectory can still be used by downstream services in short-term occlusion scenarios, providing a buffer stage for the existence of the target trajectory and avoiding the problem of needing to rebuild the trajectory before and after the occlusion in short-term occlusion scenarios.

[0157] Reference Figure 12 , showing the overall flow chart of the target tracking method based on multi-source visual fusion provided by an embodiment of the present application. In a feasible implementation, after obtaining multiple detection results at the current moment, including surround detection results and monocular detection results, the method preprocesses the detection results and then matches each detection result with each retained track.

[0158] When the trajectory successfully matches the detection result of a detection source, the filter is updated according to the detection result, that is, the filter of the trajectory is updated to obtain the updated trajectory and output it; When the trajectory successfully matches the detection results of different detection sources, the surround detection results and the monocular detection results are fused, and the filter is updated based on the fused detection results, that is, the filter of the trajectory is updated to obtain the updated trajectory and output it; When the detection result does not successfully match any trajectory, a new trajectory is created and output based on the detection result; When the track does not successfully match any detection result, the track status is updated, which may be updated to a continued track, a hidden track, or an extinct track.

[0159] Finally, the stable and continued trajectories at the current moment are output. For example, at the current moment, the trajectory status that matches the detection result has been updated to a stable trajectory, and the newly created trajectory is created based on the high-scoring detection results retained after preprocessing, which is also a stable trajectory. The special state of the still existing stable trajectory, namely the continued trajectory, can also be output.

[0160] The method provided in this embodiment performs target tracking based on the detection results of different detection sources, including multi-source visual target detection for BEV 3D target detection based on surround view images and monocular 3D detection of a single camera image. Compared with target tracking relying on a single detection source, this method is more reliable and accurate.

[0161] In addition, the detection results are preprocessed, the detection results with low detection scores are filtered, and the detection results of repeated detections are filtered. This can not only reduce the data processing volume and processing pressure, but also reduce the interference of unreliable detection results, and further improve the reliability and accuracy of target tracking.

[0162] In the process of matching trajectories and detection results, the characteristics of the visual detection source are combined to achieve stable matching of visual trajectories with visual targets from different detection sources. The 3D target detection model is used to predict the target's 3D position, size, and orientation. The 3D distance components parallel / perpendicular to the ray direction from the target to the main vehicle are calculated for matching. Cascaded 2D matching is performed on unmatched targets. This method does not rely on prior size and is less affected by occlusion and truncation in the 2D image. It achieves accurate matching results at a low computational cost, is simple and robust, and provides more complete trajectories.

[0163] When a trajectory is matched to the detection results of different detection sources at the same time, when the detection results of different detection sources are fused, the detection results of the detection source with better performance can be combined with the strengths of other target sources to achieve tracking of visual target trajectories with better performance and greater stability.

[0164] This method can achieve surround target detection and ultra-distant target detection at a distance of 200m by combining the advantages of different visual target detection methods. It can also verify the blind spot filling effect through multiple detection sources. Compared with the vision-millimeter wave radar / lidar fusion solution, it can achieve the same or better perception performance at a lower cost configuration. In the target tracking process based on this method, taking the vehicle category as an example, the fusion result well combines the advantages of both detection sources, can achieve stable target detection and tracking, and can ensure a detection rate of more than 93% even with a monocular detection range of more than 120 meters, reaching a detection distance of 200 meters. At the same time, BEV surround view detection is used to detect surrounding targets, and extremely high detection rate and accuracy can be guaranteed for targets to the side / rear. At the same time, when the detection source detection results are dynamically selected through the matching process, a smooth transition between different detection sources can be achieved, reducing the average ranging error to only 2.8%.

[0165] Reference Figure 13 , shows a functional module diagram of a target tracking device based on multi-source visual fusion provided by an embodiment of the present application, the device comprising: A matching module 100 is configured to match each of a plurality of detection results at a current moment with each current trajectory, wherein the detection results include surround view detection results and monocular detection results; The detection fusion and tracking module 200 is used to execute the detection result fusion strategy when any trajectory successfully matches a surround view detection result and a monocular detection result, determine the fused detection result corresponding to the surround view detection result and the monocular detection result, and update the trajectory based on the fused detection result.

[0166] Optionally, each test result carries a test score corresponding to the test result, and the apparatus further includes: The first filtering module is used to execute a score filtering strategy based on the detection scores corresponding to the various detection results at the current moment; wherein the score filtering strategy includes: filtering the surround detection results whose detection scores are less than the first filtering threshold; and filtering the monocular detection results whose detection scores are less than the second filtering threshold.

[0167] Optionally, the device further comprises: The second filtering module is used to sort the multiple surround view detection results in descending order according to the detection scores, and then execute a ray filtering strategy on each surround view detection result in turn, where the ray filtering strategy is used to filter out duplicate surround view detection results.

[0168] Optionally, the second filtering module executes a ray filtering strategy, including: An occupancy angle calculation unit is used to calculate the occupancy angle between each surround view detection result and the host vehicle; a coincidence calculation unit, configured to respectively calculate the coincidence between the occupied angle of the surround view detection result and the occupied angles of each surround view detection result in the surround view detection result queue; a rejection unit, configured to, when a degree of overlap between any surround view detection result in the surround view detection result queue and the surround view detection result is greater than a degree of overlap threshold, and the distance between the two surround view detection results is less than a target ratio of the detection result depth, characterize the surround view detection result as a duplicate detection result and reject the surround view detection result; The retaining unit is configured to add the surround view detection result to the surround view detection result queue when there is no surround view detection result in the surround view detection result queue and the overlap between the surround view detection result and the surround view detection result is greater than the overlap threshold.

[0169] Optionally, the coincidence calculation unit calculates the coincidence between the occupied angles of any two surround view detection results using the following calculation formula:

[0170] Among them, the occupancy angle of any surround detection result is θ max arrive θ min , θ 1 and θ 2 represent the two occupied angles used to calculate the coincidence; iou Indicates the degree of overlap.

[0171] Optionally, the matching module includes: The first matching unit is configured to execute a ray direction matching strategy for each pre-processed detection result and all stable trajectories respectively; for each detection result that is not matched to a stable trajectory, execute a ray direction matching strategy for each detection result that is not matched to the stable trajectory and the unstable trajectory respectively; The second matching unit is used to execute a 2D matching strategy on the monocular detection results that are not matched to the trajectory and the trajectory that is not matched to the detection result.

[0172] Optionally, the first matching unit includes: A distance calculation subunit, used to calculate the distance between each detection result and each trajectory; The matching score calculation subunit is used to determine the detection result that successfully matches any track among all detection results whose corresponding distance is less than the distance threshold, and calculate the matching score between the track and the successfully matched detection result.

[0173] Optionally, the distance calculation subunit is used to: Determine the ray direction distance, angle distance, size distance and type distance between each detection result and each trajectory respectively; The distance between each detection result and each track is calculated based on the ray direction distance, the angle distance, the size distance, and the type distance.

[0174] Optionally, the distance calculation subunit determines the ray direction distance between each detection result and each trajectory using the following formula:

[0175] in, track Indicates trajectory; det Indicates the test results; track x and track y The coordinates of the nearest corner of the trajectory in the main vehicle coordinate system; det x and det y The coordinates of the nearest corner point of the detection result in the main vehicle coordinate system; d x and d y Represent the coordinate difference between the nearest corner point of the trajectory and the detection result in the main vehicle coordinate system; d_dir x and d_dir y Respectively represent the decomposed distances in the ray direction of the main vehicle and the detection result; heading Indicates the angle in the test result; d Indicates the ray direction distance between the trajectory and the detection result.

[0176] Optionally, the distance calculation subunit is configured to: Calculate the angular distance between the trajectory and the detection result based on the angular difference between the trajectory and the detection result and the length of the trajectory size; The calculation formula of the angular distance is:

[0177] in, d° Indicates angular distance; size track.l The length of the trajectory size; heading track represents the trajectory angle; heading det Indicates the angle of the test result; abs () is the absolute value function.

[0178] Optionally, the distance calculation subunit is used to calculate the size distance based on the difference in length between the trajectory size and the detection result size; The calculation formula of the size distance includes:

[0179] Among them, d size Indicates size distance; size track.l The length indicating the track size; size det.l Indicates the length of the test result size; abs () is the absolute value function.

[0180] Optionally, the distance calculation subunit is configured to: Determine whether the type of trajectory is the same as the type of detection result; When the type of the detection result is the same as the type of the trajectory, the type distance between the detection result and the trajectory is 0; When the type of the detection result is different from the type of the trajectory, and the type of the detection result and the type of the trajectory belong to the preset type match, the type distance between the detection result and the trajectory is 1; When the type of the detection result is different from the type of the trajectory, and the type of the detection result and the type of the trajectory do not belong to a preset type match, it is determined that the detection result and the trajectory are not successfully matched.

[0181] Optionally, the distance calculation subunit is configured to: Summing the ray direction distance, the angle distance, the size distance, and the type distance to obtain the distance between the detection result and the trajectory; Alternatively, a weighted sum is performed on the ray direction distance, the angle distance, the size distance, and the type distance to obtain the distance between the detection result and the trajectory.

[0182] Optionally, the matching score calculation subunit is configured to: For all detection results whose distance to any trajectory is less than the distance threshold, the Hungarian matching algorithm is used to determine the detection results that successfully match the trajectory in different detection sources. Calculating a matching score between the trajectory and the successfully matched detection result based on the distance threshold and the distance between the trajectory and the detection result that successfully matched the trajectory; The formula for calculating the matching score is:

[0183] in, score dist represents the matching score; distis the distance between the detection result and the trajectory; threshold is the distance threshold.

[0184] Optionally, the second matching unit is configured to: For any track that is not matched to a detection result, calculate the 2D intersection-over-union ratio between the track and the 2D detection results in each monocular detection result that is not matched to the track; Based on multiple monocular detection results whose 2D intersection-over-union (IoU) ratios corresponding to the trajectory are greater than the IoU threshold, the Hungarian matching algorithm is used to determine the monocular detection result that successfully matches the trajectory, and the matching score between the trajectory and the successfully matched monocular detection result is calculated.

[0185] Optionally, the detection fusion and tracking module includes: A target score calculation unit is configured to calculate the target scores corresponding to the surround view detection result and the monocular detection result, respectively, based on the detection scores corresponding to the surround view detection result and the monocular detection result, the matching score between the surround view detection result and the monocular detection result, and the visible ratio in the detection results when any track successfully matches a surround view detection result and a monocular detection result. A detection fusion unit is configured to determine, based on the target score, a corner fusion result, a size fusion result, and an angle fusion result corresponding to the surround view detection result and the monocular detection result; and determine, based on the corner fusion result, the size fusion result, and the angle fusion result, a fusion detection result corresponding to the surround view detection result and the monocular detection result.

[0186] Optionally, the detection fusion unit is used to: Select the corner point closest to the main vehicle in the surround view detection result and the corner point closest to the main vehicle in the monocular detection result as the fusion corner points of the detection results; The target scores corresponding to the surround detection result and the monocular detection result are used as the corner point fusion weights corresponding to the detection results; The corner fusion result corresponding to the surround view detection result and the monocular detection result is calculated according to the corner fusion weights and fusion corner points corresponding to the surround view detection result and the monocular detection result.

[0187] Optionally, the detection fusion unit calculates the target score using the formula:

[0188] in, score calib Indicates the target score corresponding to any detection result, including the target score of the surround detection result score bev And the target score of the monocular detection result scoremono ; score det Indicates the detection score in the detection result; score dist Represents the matching score between the detection result and the trajectory; visible_ratio Indicates the visible ratio in the detection results.

[0189] Optionally, the calculation formula for the corner fusion result calculated by the detection fusion unit is:

[0190] in, corner Indicates the corner fusion result; corner bev Indicates the fusion corner point of the surround detection result; corner mono Indicates the fusion corner point of the monocular detection result; score bev is the target score of the surround detection result; score mono is the target score of the monocular detection result.

[0191] Optionally, the detection fusion unit is used to: According to all historical matching detection results of the trajectory, according to the target scores corresponding to each historical matching detection result, the historical matching detection results of the target number with the highest target score are selected, and the size average of the historical matching detection results of the target number is used as the size fusion result.

[0192] Optionally, the detection fusion unit is used to: According to the angle of the detection result matched by the trajectory in the previous frame, the angles of the surround detection result and the monocular detection result are corrected to obtain the corrected angles of each detection result; According to the corrected angles corresponding to the surround view detection result and the monocular detection result, the target scores corresponding to the surround view detection result and the monocular detection result are used as angle fusion weights to calculate the angle fusion result corresponding to the surround view detection result and the monocular detection result.

[0193] Optionally, the formula for the detection fusion unit to obtain the corrected angle of each detection result is:

[0194] in, heading' Indicates the angle after the test result is corrected; heading is the angle for the test result; last_ heading is the angle of the detection result matched by the trajectory in the previous frame; N is the angle deviation value.

[0195] Optionally, the calculation formula used by the detection fusion unit to determine the angle fusion result is:

[0196] in, heading* Indicates the angle fusion result; score bev is the target score of the surround detection result; score mono is the target score of the monocular detection result; heading bev The angle after correction of surround detection result; heading mono is the corrected angle of the monocular detection result.

[0197] Optionally, the detection fusion unit determines a calculation formula for a fusion detection result corresponding to the surround detection result and the monocular detection result:

[0198] in, c x and c y Respectively represent the coordinates of each corner point in the fusion detection result; corner x and corner y is the coordinate of the corner point fusion result; size l and size w Respectively represent the length and width of the size fusion result; heading* Indicates the angle fusion result.

[0199] Optionally, the detection fusion and tracking module includes: A first filtering unit is configured to input a fusion detection result corresponding to the surround view detection result and the monocular detection result into a filter of the trajectory when any trajectory successfully matches a surround view detection result and a monocular detection result at the same time; According to the fusion detection result and the filter of the trajectory, the velocity estimation result and convergence status of the trajectory at the current moment are output.

[0200] Optionally, the device further includes a second filtering unit, configured to: When any track successfully matches a surround detection result or a monocular detection result, the detection result matched by the track is input into the filter of the track; According to the detection result matched to the trajectory and the filter of the trajectory, the velocity estimation result and convergence status of the trajectory at the current moment are output.

[0201] Optionally, the device further includes a third filtering unit, configured to: When any detection result fails to match any trajectory, a new trajectory is generated based on the detection result, and the velocity estimation result and convergence status of the new trajectory at the current moment are output based on the filter of the new trajectory and the detection result.

[0202] Optionally, the device further comprises: The trajectory state updating module is used to update the state of each trajectory, where the state of the trajectory is used to indicate whether the trajectory is a stable trajectory.

[0203] Optionally, the device further includes a trajectory state initialization module, configured to: When any trajectory is initialized, if the trajectory is generated based on the detection results with a detection score greater than or equal to the score threshold, the trajectory is considered a stable trajectory; When the trajectory is generated based on a detection result with a detection score less than a score threshold, the trajectory is an unstable trajectory.

[0204] Optionally, the trajectory state updating module includes a first trajectory state updating unit, configured to: When any stable trajectory fails to successfully match any detection result within the first time period, the state of the stable trajectory is updated to a continued trajectory; When any continued trajectory fails to successfully match any detection result within multiple consecutive frames, the status of the continued trajectory is updated to a hidden trajectory; When any hidden track fails to match any detection result within the second time period, or any unstable track fails to match any detection result within the third time period, the status is updated to an extinct track and discarded.

[0205] Optionally, the trajectory state updating module includes a second trajectory state updating unit, configured to: When any continued trajectory or hidden trajectory successfully matches any detection result, the status is updated to a stable trajectory; When any unstable trajectory matches any detection result in multiple consecutive frames, or the detection score of the successfully matched detection result is greater than or equal to the score threshold, the status of the unstable trajectory is updated to a stable trajectory.

[0206] Optionally, the device further comprises: The trajectory retention module is used to take all stable trajectories, unstable trajectories, continued trajectories, and hidden trajectories as the trajectories of the next frame.

[0207] An embodiment of the present application also provides an electronic device, comprising: at least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, the target tracking method based on multi-source visual fusion described in the embodiment is executed.

[0208] An embodiment of the present application further provides a non-volatile readable storage medium, which stores a computer program. When the computer program is executed by a processor, the target tracking method based on multi-source visual fusion described in the embodiment is executed.

[0209] An embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the target tracking method based on multi-source visual fusion described in the embodiment.

[0210] An embodiment of the present application further provides a vehicle, which includes the target tracking device based on multi-source visual fusion described in the embodiment, and executes the target tracking method based on multi-source visual fusion described in the embodiment.

[0211] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0212] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, apparatuses, or computer program products. Therefore, the embodiments of the present application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware. Furthermore, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0213] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0214] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0216] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0217] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0218] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A target tracking method based on multi-source visual fusion, characterized in that: The method comprises: Matching each of the multiple detection results at the current moment with each current trajectory, wherein the detection results include surround detection results and monocular detection results; When any trajectory successfully matches a surround view detection result and a monocular detection result at the same time, the detection result fusion strategy is executed to determine the fused detection result corresponding to the surround view detection result and the monocular detection result, and the trajectory is updated according to the fused detection result.

2. The method according to claim 1, characterized in that Before matching each of the multiple detection results at the current moment with each current trajectory, the method further includes: Execute the score filtering strategy based on the detection scores corresponding to each detection result at the current moment; The score filtering strategy includes: Filtering surround detection results whose detection scores are less than a first filtering threshold; The monocular detection results whose detection scores are less than the second filtering threshold are filtered.

3. The method according to claim 2, characterized in that After filtering the surround view detection results having a detection score less than the first filtering threshold, the method further includes: After sorting the multiple surround view detection results in descending order according to the detection scores, a ray filtering strategy is executed on each surround view detection result in turn, where the ray filtering strategy is used to filter out duplicate surround view detection results.

4. The method according to claim 3, characterized in that The ray filtering strategy is executed on each surround detection result in turn, including: For each surround view detection result, calculate the occupancy angle between the surround view detection result and the host vehicle; Calculate the overlap between the occupancy angle of the surround view detection result and the occupancy angle of each surround view detection result in the surround view detection result queue respectively; When the overlap between any surround view detection result in the surround view detection result queue and the surround view detection result is greater than the overlap threshold, and the distance between the two surround view detection results is less than the target ratio of the detection result depth, the surround view detection result is characterized as a duplicate detection result and is discarded; When there is no surround view detection result in the surround view detection result queue and the overlap between the surround view detection result and the surround view detection result is greater than the overlap threshold, the surround view detection result is added to the surround view detection result queue.

5. The method according to claim 4, characterized in that The calculation formula for the overlap between the occupied angles of any two surround view detection results is: Among them, the occupancy angle of any surround detection result is θ max arrive θ min , θ 1 and θ 2 represent the two occupied angles used to calculate the coincidence; iou Indicates the degree of overlap.

6. The method according to claim 1, characterized in that Matching each of the multiple detection results at the current moment with each current trajectory includes: Execute ray direction matching strategy for each detection result and all stable trajectories; For each detection result that is not matched to a stable trajectory, a ray direction matching strategy is performed with the stable trajectory that is not matched to the detection result and the unstable trajectory respectively; For monocular detection results that are not matched to a track, a 2D matching strategy is performed on the track that is not matched to the detection result.

7. The method according to claim 6, characterized in that The process of executing the ray direction matching strategy includes: Calculate the distance between each detection result and each trajectory; Among all detection results whose distances corresponding to any track are less than the distance threshold, a detection result that successfully matches the track is determined, and a matching score between the track and the successfully matched detection result is calculated.

8. The method according to claim 7, characterized in that Calculating the distance between each detection result and each track includes: Determine the ray direction distance, angle distance, size distance and type distance between each detection result and each trajectory respectively; The distance between each detection result and each track is calculated based on the ray direction distance, the angle distance, the size distance, and the type distance.

9. The method according to claim 8, characterized in that The formula for determining the ray direction distance between each detection result and each trajectory is: in, track Indicates trajectory; det Indicates the test results; track x and track y The coordinates of the nearest corner of the trajectory in the main vehicle coordinate system; det x and det y The coordinates of the nearest corner point of the detection result in the main vehicle coordinate system; d x and d y They represent the coordinate difference between the nearest corner point of the trajectory and the detection result in the main vehicle coordinate system; d_dir x and d_dir y Respectively represent the decomposed distances in the ray direction of the main vehicle and the detection result; heading Indicates the angle in the test result; d Indicates the ray direction distance between the trajectory and the detection result.

10. The method according to claim 8, characterized in that The process of determining the angular distance between each detection result and each trajectory includes: Calculate the angular distance between the trajectory and the detection result based on the angular difference between the trajectory and the detection result and the length of the trajectory size; The calculation formula of the angular distance is: in, d° Indicates angular distance; size track.l The length indicating the track size; heading track represents the trajectory angle; heading det Indicates the angle of the test result; abs () is the absolute value function.

11. The method according to claim 8, characterized in that The process of determining the size distance between each detection result and each track includes: Calculate the size distance based on the difference in length between the trajectory size and the detection result size; The calculation formula of the size distance includes: in, d size Indicates size distance; size track.l The length indicating the track size; size det.l Indicates the length of the test result size; abs () is the absolute value function.

12. The method according to claim 8, characterized in that The process of determining the type distance between each detection result and each track includes: Determine whether the type of trajectory is the same as the type of detection result; When the type of the detection result is the same as the type of the trajectory, the type distance between the detection result and the trajectory is 0; When the type of the detection result is different from the type of the trajectory, and the type of the detection result and the type of the trajectory belong to the preset type match, the type distance between the detection result and the trajectory is 1; When the type of the detection result is different from the type of the trajectory, and the type of the detection result and the type of the trajectory do not belong to a preset type match, it is determined that the detection result and the trajectory are not successfully matched.

13. The method according to claim 8, characterized in that The calculating the distance between each detection result and each track according to the ray direction distance, the angle distance, the size distance, and the type distance includes: Summing the ray direction distance, the angle distance, the size distance, and the type distance to obtain the distance between the detection result and the trajectory; Alternatively, a weighted sum is performed on the ray direction distance, the angle distance, the size distance, and the type distance to obtain the distance between the detection result and the trajectory.

14. The method according to claim 7, characterized in that The method of determining a detection result that successfully matches any trajectory among all detection results whose distances to the trajectory are less than a distance threshold, and calculating a matching score between the trajectory and the successfully matched detection result, includes: For all detection results whose distance to any trajectory is less than the distance threshold, the Hungarian matching algorithm is used to determine the detection results that successfully match the trajectory in different detection sources. Calculating a matching score between the trajectory and the successfully matched detection result based on the distance threshold and the distance between the detection result successfully matched with the trajectory and the trajectory; The formula for calculating the matching score is: in, score dist represents the matching score; dist is the distance between the detection result and the trajectory; threshold is the distance threshold.

15. The method according to claim 6, characterized in that For the monocular detection results that are not matched to the track, respectively executing the 2D matching strategy with the track that is not matched to the detection result, including: For any track that is not matched to a detection result, calculate the 2D intersection-over-union ratio between the track and the 2D detection results in each monocular detection result that is not matched to the track; Based on multiple monocular detection results whose 2D intersection-over-union (IoU) ratios corresponding to the trajectory are greater than the IoU threshold, the Hungarian matching algorithm is used to determine the monocular detection result that successfully matches the trajectory, and the matching score between the trajectory and the successfully matched monocular detection result is calculated.

16. The method according to claim 1, wherein When any trajectory successfully matches a surround view detection result and a monocular detection result at the same time, executing the detection result fusion strategy to determine the fusion detection result corresponding to the surround view detection result and the monocular detection result, including: When any track successfully matches a surround view detection result and a monocular detection result at the same time, the target score corresponding to the surround view detection result and the monocular detection result is calculated based on their corresponding detection scores, the matching score with the track, and the visible ratio in the detection results; Determine, according to the target score, a corner fusion result, a size fusion result, and an angle fusion result corresponding to the surround detection result and the monocular detection result; According to the corner fusion result, the size fusion result and the angle fusion result, a fusion detection result corresponding to the surround detection result and the monocular detection result is determined.

17. The method according to claim 16, characterized in that Determining the corner point fusion result corresponding to the surround detection result and the monocular detection result, including: Select the corner point closest to the main vehicle in the surround view detection result and the corner point closest to the main vehicle in the monocular detection result as the fusion corner points of the detection results; The target scores corresponding to the surround detection result and the monocular detection result are used as the corner point fusion weights corresponding to the detection results; The corner fusion result corresponding to the surround view detection result and the monocular detection result is calculated according to the corner fusion weights and fusion corner points corresponding to the surround view detection result and the monocular detection result.

18. The method according to claim 17, characterized in that The formula for calculating the target score is: in, score calib Indicates the target score corresponding to any detection result, including the target score of the surround detection result score bev And the target score of the monocular detection result score mono ; score det Indicates the detection score in the detection result; score dist Represents the matching score between the detection result and the trajectory; visible_ratio Indicates the visible ratio in the detection results.

19. The method according to claim 18, characterized in that The calculation formula of the corner fusion result is: in, corner Indicates the corner fusion result; corner bev Indicates the fusion corner point of the surround detection result; corner mono Indicates the fusion corner point of the monocular detection result; score bev is the target score of the surround detection result; score mono is the target score of the monocular detection result.

20. The method according to claim 16, wherein Determining the size fusion result corresponding to the surround detection result and the monocular detection result, including: According to all historical matching detection results of the trajectory, according to the target scores corresponding to each historical matching detection result, the historical matching detection results of the target number with the highest target score are selected, and the size average of the historical matching detection results of the target number is used as the size fusion result.

21. The method according to claim 16, wherein Determining an angle fusion result corresponding to the surround view detection result and the monocular detection result, including: According to the angle of the detection result matched by the trajectory in the previous frame, the angles of the surround detection result and the monocular detection result are corrected to obtain the corrected angles of each detection result; According to the corrected angles corresponding to the surround view detection result and the monocular detection result, the target scores corresponding to the surround view detection result and the monocular detection result are used as angle fusion weights to calculate the angle fusion result corresponding to the surround view detection result and the monocular detection result.

22. The method according to claim 21, characterized in that The angles of the surround detection result and the monocular detection result are corrected according to the angle of the detection result matched by the trajectory in the previous frame. The formula for obtaining the corrected angles of each detection result is: in, heading' Indicates the angle after the test result is corrected; heading is the angle for the test result; last_ heading is the angle of the detection result matched by the trajectory in the previous frame; N is the angle deviation value.

23. The method according to claim 22, characterized in that The calculation formula of the angle fusion result is: in, heading* Indicates the angle fusion result; score bev is the target score of the surround detection result; score mono is the target score of the monocular detection result; heading bev The angle after correction of surround detection result; heading mono is the corrected angle of the monocular detection result.

24. The method according to claim 16, wherein According to the corner fusion result, size fusion result and angle fusion result, the calculation formula for determining the fusion detection result corresponding to the surround detection result and the monocular detection result is: in, c x and c y Respectively represent the coordinates of each corner point in the fusion detection result; corner x and corner y is the coordinate of the corner point fusion result; size l and size w Respectively represent the length and width of the size fusion result; heading* Indicates the angle fusion result.

25. The method according to claim 1, wherein The updating of the trajectory according to the fusion detection result includes: When any track successfully matches a surround view detection result and a monocular detection result at the same time, the fusion detection result corresponding to the surround view detection result and the monocular detection result is input into the filter of the track; According to the fusion detection result and the filter of the trajectory, the velocity estimation result and convergence status of the trajectory at the current moment are output.

26. The method according to claim 1, wherein After matching each of the multiple detection results at the current moment with each current trajectory, the method further includes: When any track successfully matches a surround detection result or a monocular detection result, the detection result matched by the track is input into the filter of the track; According to the detection result matched to the trajectory and the filter of the trajectory, the velocity estimation result and convergence status of the trajectory at the current moment are output.

27. The method according to claim 1, wherein After matching each of the multiple detection results at the current moment with each current trajectory, the method further includes: When any detection result fails to match any trajectory, a new trajectory is generated based on the detection result, and the velocity estimation result and convergence status of the new trajectory at the current moment are output based on the filter of the new trajectory and the detection result.

28. The method according to claim 1, wherein After matching each of the multiple detection results at the current moment with each current trajectory, the method further includes: The status of each trajectory is updated, where the status of the trajectory is used to indicate whether the trajectory is a stable trajectory.

29. The method according to claim 28, characterized in that The method further comprises: When any trajectory is initialized, if the trajectory is generated based on the detection results with a detection score greater than or equal to the score threshold, the trajectory is considered a stable trajectory; When the trajectory is generated based on a detection result with a detection score less than a score threshold, the trajectory is an unstable trajectory.

30. The method according to claim 28, wherein The updating of the status of each track includes: When any stable trajectory fails to successfully match any detection result within the first time period, the state of the stable trajectory is updated to a continued trajectory; When any continued trajectory fails to successfully match any detection result within multiple consecutive frames, the status of the continued trajectory is updated to a hidden trajectory; When any hidden track fails to match any detection result within the second time period, or any unstable track fails to match any detection result within the third time period, the status is updated to an extinct track and discarded.

31. The method according to claim 30, characterized in that The updating of the status of each track includes: When any continued trajectory or hidden trajectory successfully matches any detection result, the status is updated to a stable trajectory; When any unstable trajectory matches any detection result in multiple consecutive frames, or the detection score of the successfully matched detection result is greater than or equal to the score threshold, the status of the unstable trajectory is updated to a stable trajectory.

32. The method according to claim 31, wherein After updating the status of each track, the method further includes: All stable trajectories, unstable trajectories, continued trajectories, and hidden trajectories are used as the trajectories of the next frame.

33. An electronic device, characterized in that: include: At least one processor, and a memory, wherein the memory stores a computer program that can be run on the processor, wherein when the processor executes the computer program, it executes the target tracking method based on multi-source visual fusion according to any one of claims 1 to 32.

34. A non-volatile readable storage medium, characterized in that The non-volatile readable storage medium stores a computer program, wherein, when the computer program is executed by a processor, the target tracking method based on multi-source visual fusion according to any one of claims 1 to 32 is executed.

35. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the target tracking method based on multi-source visual fusion described in any one of claims 1 to 32 is implemented.

36. A vehicle, characterized in that: The vehicle is used to execute the target tracking method based on multi-source vision fusion described in any one of claims 1-32.

Images