Track data fusion method, electronic equipment and vehicle

Abstract

The invention discloses a trajectory data fusion method, electronic equipment and a vehicle. The trajectory data fusion method comprises the following steps: acquiring first trajectory data acquired by at least two radars arranged on the vehicle, and acquiring second trajectory data acquired by a camera arranged on the vehicle; performing fusion processing on the first trajectory data acquired by the at least two radars to obtain third trajectory data after fusion processing; and carrying out fusion processing on the second trajectory data and the third trajectory data to obtain fourth trajectory data after fusion processing. The technical problem that the accuracy and anti-interference performance of a fusion result are relatively weak due to the fact that information collected by all sensors cannot be fully utilized in an existing sensor combination fusion method can be solved.

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and specifically relates to a trajectory data fusion method, electronic equipment and vehicle, as well as a trajectory data fusion device, a computer-readable storage medium and a computer program product. Background Art

[0002] In the related technology, vehicles are usually equipped with multiple types of sensors such as corner radars, front radars and cameras. The above sensors will collect target trajectory data separately and perform fusion processing on the collected target trajectory data. One of the solutions for fusion processing is to adopt a combined fusion method, that is, select the target trajectory data of a sensor in the corner radar or front radar according to the location of the target, and then fuse it with the target trajectory data collected by the camera. The above combined fusion method cannot fully utilize the information collected by all sensors, which leads to the technical problem that the accuracy and anti-interference ability of the fusion result will be relatively weak. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a trajectory data fusion method, an electronic device and a vehicle, as well as a target trajectory data fusion device, a computer-readable storage medium and a computer program product, which can solve the technical problem that the existing combined fusion method cannot fully utilize the information collected by all sensors, thereby resulting in the relatively weak accuracy and anti-interference of the fusion result.

[0004] In a first aspect, an embodiment of the present application provides a trajectory data fusion method, including:

[0005] Acquire first track data respectively collected by at least two radars provided on the vehicle, and acquire second track data collected by a camera provided on the vehicle;

[0006] Performing fusion processing on the first trajectory data respectively collected by the at least two radars to obtain fused third trajectory data, where the third trajectory data includes the first fused trajectory data of at least one radar observation target;

[0007] The second trajectory data and the third trajectory data are fused to obtain fused fourth trajectory data, where the fourth trajectory data includes the second fused trajectory data of at least one observed target.

[0008] In some embodiments, the at least two radars provided on the vehicle include at least two of a front radar, a first corner radar located on a first side of the front radar, and a second corner radar located on a second side of the front radar.

[0009] In some embodiments, the first trajectory data includes observation trajectory data of at least one radar observation target, and the obtaining of the first trajectory data respectively collected by at least two radars provided on the vehicle includes:

[0010] Acquire observation trajectory data of at least one radar observation target collected by at least two radars in a current collection cycle;

[0011] Based on the observation trajectory data of at least one radar observation target collected in a current collection cycle, the historical observation trajectory data of the radar observation target obtained by at least two radars in a historical collection cycle are updated to obtain first trajectory data.

[0012] In some embodiments, the second trajectory data includes observation trajectory data of at least one visual observation target, and the second trajectory data collected by a camera disposed on the vehicle is obtained, including:

[0013] Obtaining observation trajectory data of at least one visual observation target collected by the camera in a current collection cycle;

[0014] Based on the observation trajectory data of at least one visual observation target collected in the current collection cycle, the historical trajectory data of the visual observation target obtained by the camera in the historical collection cycle is updated to obtain second trajectory data.

[0015] In some embodiments, after obtaining the observation trajectory data of at least one radar observation target collected by at least two radars in the current collection cycle, the method further includes:

[0016] Based on a preset trajectory screening condition, the observation trajectory data of at least one radar observation target collected by at least two radars in a current collection cycle are screened to retain the observation trajectory data of the radar observation target that meets the preset trajectory screening condition; and / or

[0017] After obtaining the observation trajectory data of at least one visual observation target collected by the camera in the current collection cycle, the method further includes:

[0018] The observation trajectory data of at least one visual observation target collected by the camera in a current collection cycle is filtered based on the preset trajectory filtering condition to retain the observation trajectory data of the visual observation target that meets the preset trajectory filtering condition.

[0019] In some embodiments, the preset trajectory screening condition includes: at least one of a relative position threshold, a relative speed threshold, a target confidence threshold, a target type, a target identifier, and a validity identifier.

[0020] In some embodiments, after filtering the observation trajectory data of at least one radar observation target collected by at least two radars in the current collection cycle based on the preset trajectory screening condition, the method further includes:

[0021] performing coordinate system conversion on observation trajectory data of at least one radar observation target; and / or,

[0022] After filtering the observation trajectory data of at least one visual observation target collected by the camera in the current collection cycle based on the preset trajectory screening condition, the method further includes:

[0023] The observation trajectory data of at least one visual observation target is transformed into a coordinate system.

[0024] In some embodiments, the observation trajectory data includes position data and speed data, and the coordinate system conversion of the observation trajectory data of at least one radar observation target includes:

[0025] converting the position data of at least one radar observation target into a vehicle rear axle coordinate system;

[0026] converting velocity data of at least one radar observed target into an absolute coordinate system; and / or,

[0027] The coordinate system conversion of the observation trajectory data of at least one visual observation target includes:

[0028] The position data of at least one visual observation target is converted to a vehicle rear axle coordinate system.

[0029] In some embodiments, it also includes:

[0030] If the observation trajectory data of at least one radar observation target is not collected in the current collection period, the observation trajectory data of at least one radar observation target in the current collection period is determined based on the historical observation trajectory data of at least one radar observation target in the previous two collection periods; and / or,

[0031] If the observation trajectory data of at least one visual observation target is not collected in the current collection cycle, the observation trajectory data of at least one visual observation target in the current collection cycle is determined based on the historical observation trajectory data of at least one visual observation target in the previous two collection cycles.

[0032] In some embodiments, it also includes:

[0033] If the observation trajectory data of at least one radar observation target is not collected for N consecutive collection cycles, the historical observation trajectory data of at least one radar observation target is deleted; and / or,

[0034] If the observation trajectory data of at least one visual observation target is not collected for N consecutive collection cycles, the historical observation trajectory data of at least one visual observation target is deleted;

[0035] Wherein, N is a positive integer greater than or equal to 2.

[0036] In some embodiments, the at least two radars include a first radar and a second radar, and before fusing the first trajectory data respectively collected by the at least two radars, the process further includes:

[0037] Determine the first radar observation target and the second radar observation target that are associated with each other based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar;

[0038] The above-mentioned fusing processing of the first trajectory data respectively collected by at least two radars includes:

[0039] The observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target that are mutually related are fused to obtain first fused trajectory data of at least one radar observation target.

[0040] In some embodiments, the determining of the mutually associated first radar observation target and second radar observation target based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar includes:

[0041] Calculating a trajectory distance between the first radar observation target and the second radar observation target based on the observation trajectory data of at least one first radar observation target and the observation trajectory data of at least one second radar observation target;

[0042] Based on the track distance between the first radar observation target and the second radar observation target, the second radar observation target and the second radar observation target associated with each other are determined.

[0043] In some embodiments, the above-mentioned calculating the trajectory distance between the first radar observation target and the second radar observation target based on the observation trajectory data of at least one first radar observation target and the observation trajectory data of at least one second radar observation target includes:

[0044] The trajectory distance between the first radar observation target and the second radar observation target is determined based on the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

[0045] In some embodiments, the first radar is a first corner radar, and the second radar is a second corner radar. Based on the Mahalanobis distance between the observed trajectory data of the target observed by the first radar and the observed trajectory data of the target observed by the second radar, and at least one of the distance variance cost, the cycle number related cost, the speed cost, and the near-point target penalty, determining the trajectory distance between the target observed by the first radar and the target observed by the second radar comprises:

[0046] determining a track distance between the target observed by the first radar and the target observed by the second radar based on a Mahalanobis distance between the observed track data of the target observed by the first radar and the observed track data of the target observed by the second radar, an additional defined distance between the first corner radar and the second corner radar, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty,

[0047] In some embodiments, the determining of the trajectory distance between the first radar observed target and the second radar observed target based on the Mahalanobis distance between the observed trajectory data of the first radar observed target and the observed trajectory data of the second radar observed target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty, includes:

[0048] Dist1=(1+MagnifyCost1)*MDist1+TraSizeCost1+SpeedCost1+NearbyCost1

[0049] Among them, Dist1 is the trajectory distance between the first radar observation target and the second radar observation target, MDist1 is the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, MagnifyCost1 is the distance variance cost of the trajectory distance between the first radar observation target and the second radar observation target, TraSizeCost1 is the cost related to the number of cycles of the first radar observation target and the second radar observation target, SpeedCost1 is the speed cost of the first radar observation target and the second radar observation target, and NearbyCost1 is the near-point target penalty of the first radar observation target and the second radar observation target.

[0050] In some embodiments, the determining of the first radar observation target and the second radar observation target associated with each other based on the track distance between the first radar observation target and the second radar observation target includes:

[0051] If the track distances between the first radar observation target and at least two second radar observation targets are calculated, it is determined that the first radar observation target and the second radar observation target with the smallest track distance are associated with each other.

[0052] In some embodiments, before determining the mutually associated first radar observation target and second radar observation target based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar, the method further includes:

[0053] Determine the first radar observation target and the second radar observation target that are not associated with each other based on the speed information and / or the distance information of the first radar observation target and the second radar observation target;

[0054] In some embodiments, the speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

[0055] In some embodiments, before the fusion processing of the second trajectory data and the third trajectory data is performed, the method further includes:

[0056] Determining mutually associated radar observation targets and visual observation targets based on first fused trajectory data of at least one radar observation target and observation trajectory data of at least one visual observation target;

[0057] The above-mentioned fusing processing of the second trajectory data and the third trajectory data includes:

[0058] The first fused trajectory data of the mutually related radar observation targets and the observation trajectory data of the visual observation targets are fused to obtain the second fused trajectory data of at least one observation target.

[0059] In some embodiments, the determining of the mutually associated radar observation targets and visual observation targets based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target includes:

[0060] Calculating a trajectory distance between the radar observation target and the visual observation target based on first fused trajectory data of at least one radar observation target and observation trajectory data of at least one visual observation target;

[0061] Based on the track distance between the radar observation target and the visual observation target, the radar observation target and the visual observation target that are associated with each other are determined.

[0062] In some embodiments, the calculating of the trajectory distance between the radar observation target and the visual observation target based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target includes:

[0063] The trajectory distance between the radar observed target and the visually observed target is determined based on the Mahalanobis distance between the first fused trajectory data of the radar observed target and the observed trajectory data of the visually observed target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

[0064] In some embodiments, the determining of the trajectory distance between the radar observed target and the visually observed target based on the Mahalanobis distance between the first fused trajectory data of the radar observed target and the observed trajectory data of the visually observed target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty, includes:

[0065] Dist2=(1+MagnifyCost2)*MDist2+TraSizeCost2+SpeedCost2+NearbyCost2

[0066] Among them, Dist2 is the trajectory distance between the radar observation target and the visual observation target, MDist2 is the Mahalanobis distance between the first fused trajectory data of the radar observation target and the observed trajectory data of the visual observation target, MagnifyCost2 is the distance variance cost of the trajectory distance between the radar observation target and the visual observation target, TraSizeCost2 is the cost related to the number of cycles of the radar observation target and the visual observation target, SpeedCost2 is the speed cost of the radar observation target and the visual observation target, and NearbyCost2 is the near-point target penalty of the radar observation target and the visual observation target.

[0067] In some embodiments, the determining of the mutually associated radar observation target and visual observation target based on the track distance between the radar observation target and the visual observation target includes:

[0068] If the track distances between the radar observation target and at least two visual observation targets are calculated, it is determined that the radar observation target and the visual observation target with the smallest track distance are associated with each other.

[0069] In some embodiments, before determining the mutually associated radar observation targets and visual observation targets based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target, the method further includes:

[0070] Determine radar observation targets and visual observation targets that are not associated with each other based on speed information and / or distance information of the radar observation targets and the visual observation targets;

[0071] In some embodiments, the speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

[0072] In some embodiments, the second trajectory data includes observation trajectory data of at least one second visual observation target collected in the current stage, and observation trajectory data of at least one third visual observation target collected in the historical stage, and the method further includes:

[0073] Determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target;

[0074] The observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target that are related to each other are fused to obtain third fused trajectory data of at least one visual observation target.

[0075] In some embodiments, the determining of the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target includes:

[0076] Based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target, calculating the trajectory distance between the second visual observation target and the third visual observation target;

[0077] Based on the track distance between the second visual observation target and the third visual observation target, the second visual observation target and the third visual observation target associated with each other are determined.

[0078] In some embodiments, the above-mentioned calculation of the trajectory distance between the second visual observation target and the third visual observation target based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target includes:

[0079] The trajectory distance between the second visual observation target and the third visual observation target is determined based on the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

[0080] In some embodiments, the determining of the trajectory distance between the second visual observation target and the third visual observation target based on the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty, includes:

[0081] Dist3=(1+MagnifyCost3)*MDist3+TraSizeCost3+SpeedCost3+NearbyCost3

[0082] Among them, Dist3 is the trajectory distance between the second visual observation target and the third visual observation target, MDist3 is the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, MagnifyCost3 is the distance variance cost of the trajectory distance between the second visual observation target and the third visual observation target, TraSizeCost3 is the cost related to the number of cycles of the second visual observation target and the third visual observation target, SpeedCost3 is the speed cost of the second visual observation target and the third visual observation target, and NearbyCost3 is the near-point target penalty of the second visual observation target and the third visual observation target.

[0083] In some embodiments, the determining of the second visual observation target and the third visual observation target associated with each other based on the track distance between the second visual observation target and the third visual observation target includes:

[0084] If the track distances between the second visual observation target and at least two third visual observation targets are calculated, it is determined that the second visual observation target and the third visual observation target having the smallest track distance are associated with each other.

[0085] In some embodiments, before determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target, the method further includes:

[0086] Based on the speed information and / or distance information of the second visual observation target and the third visual observation target, the second visual observation target and the third visual observation target that do not satisfy the mutual association condition are determined.

[0087] In some embodiments, the speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; the distance information includes at least one of a longitudinal distance and a lateral distance. In some embodiments, before determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target, the method further includes:

[0088] If the second visual observation target and the third visual observation target have the same observation target identifier, it is determined that the second visual observation target and the third visual observation target having the same observation target identifier are associated with each other.

[0089] In a second aspect, an embodiment of the present application provides a trajectory data fusion device, including:

[0090] A data acquisition module, used to acquire first trajectory data respectively collected by at least two radars provided on the vehicle, and to acquire second trajectory data collected by a camera provided on the vehicle;

[0091] A first fusion processing module is used to perform fusion processing on the first trajectory data collected by at least two radars respectively to obtain third trajectory data after fusion processing, where the third trajectory data includes the first fusion trajectory data of at least one radar observation target;

[0092] The second fusion processing module is used to perform fusion processing on the second trajectory data and the third trajectory data to obtain fused fourth trajectory data, where the fourth trajectory data includes the second fused trajectory data of at least one observed target.

[0093] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the trajectory data fusion method described in the first aspect are implemented.

[0094] In a fourth aspect, an embodiment of the present application provides a vehicle, comprising the above-mentioned electronic device, wherein the memory of the electronic device stores a program or instruction that can be run on a processor, and when the program or instruction is executed by the processor, the steps of the trajectory data fusion method as described in the first aspect are implemented.

[0095] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the trajectory data fusion method described in the first aspect are implemented.

[0096] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed by a processor of a vehicle, implements the steps of the trajectory data fusion method as described in the first aspect.

[0097] The technical solution provided by the present application requires first obtaining first trajectory data collected by at least two radars respectively arranged on the vehicle, and obtaining second trajectory data collected by a camera arranged on the vehicle; then fusing the first trajectory data collected by at least two radars respectively to obtain third trajectory data after fusion processing; finally fusing the second trajectory data and the third trajectory data to obtain fourth trajectory data after fusion processing, wherein the first trajectory data includes observation trajectory data of at least one radar observation target, the second trajectory data includes observation trajectory data of at least one visual observation target, the third trajectory data includes first fused trajectory data of at least one radar observation target, and the fourth trajectory data includes second fused trajectory data of at least one observation target. When performing trajectory data fusion, the above technical solution first performs radar trajectory data fusion, and then fuses the fused radar trajectory data with the visual trajectory data, so that the information collected by all sensors is fully utilized, thereby enhancing the accuracy and anti-interference of the trajectory data fusion result, thereby improving the vehicle's environmental perception ability and driving decision accuracy, and enabling the vehicle to adapt to complex and changeable driving environments.

[0098] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0100] Figure 1 A schematic diagram of a flow chart of a trajectory data fusion method in an embodiment of the present application;

[0101] Figure 2 for Figure 1 A schematic diagram of a specific execution flow of obtaining first trajectory data in step 101 of the illustrated embodiment;

[0102] Figure 3 A schematic diagram of a logical structure for updating historical observation trajectory data of a radar observation target in an embodiment of the present application;

[0103] Figure 4 for Figure 1 A schematic diagram of a specific execution flow of obtaining the second trajectory data in step 101 of the illustrated embodiment;

[0104] Figure 5 A schematic diagram of a logical structure for updating historical observation trajectory data of a visual observation target in an embodiment of the present application;

[0105] Figure 6A schematic diagram of a logic flow of predicting observation trajectory data of a radar observation target in an embodiment of the present application;

[0106] Figure 7 A schematic diagram of a logical structure for predicting observation trajectory data of a visual observation target in an embodiment of the present application;

[0107] Figure 8 A schematic diagram of the logical structure for deleting the historical observation trajectory data of the radar observation target in an embodiment of the present application;

[0108] Fig. 9 A schematic diagram of the logical structure for deleting the historical observation trajectory data of the visual observation target in an embodiment of the present application;

[0109] Fig.10 for Figure 1 A schematic diagram of a specific execution flow of step 102 of the illustrated embodiment;

[0110] Fig.11 for Fig.10 A schematic diagram of a specific execution flow of step 601 of the illustrated embodiment;

[0111] Fig.12 This is a schematic diagram of a process for fusing observation trajectory data collected by different radars in an embodiment of the present application;

[0112] Fig.13 for Figure 1 A schematic diagram of a specific execution flow of step 103 of the illustrated embodiment;

[0113] Fig.14 for Fig.13 A schematic diagram of a specific execution flow of step 801 of the illustrated embodiment;

[0114] Fig.15 A schematic diagram of a logical structure for fusing first fused trajectory data of a radar observation target with observation trajectory data of a visual observation target collected by a camera in an embodiment of the present application;

[0115] Fig.16 A schematic diagram of a process for fusing observation trajectory data of a visual observation target collected by a camera at different stages in an embodiment of the present application;

[0116] Fig.17 for Fig.16 A schematic diagram of a specific execution flow of step 1201 of the illustrated embodiment;

[0117] Fig.18 A schematic diagram of a logical structure for fusing observation trajectory data of a visual observation target collected by a camera at different stages in an embodiment of the present application;

[0118] Fig.19 This is a schematic diagram of the structure of a trajectory data fusion device in an example of this application;

[0119] Fig. 20 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0120] Fig.21 It is a schematic diagram of the structure of a vehicle in an embodiment of the present application. DETAILED DESCRIPTION

[0121] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0122] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0123] As described in the background technology section, in the related art, a combined fusion method is used to process the target trajectory data collected by the sensors on the vehicle, that is, a combined method of fusing a camera and a single radar is often used. However, for a vehicle using 5R1V for sensor layout, that is, four corner radars, a front radar and a top camera are provided, a large number of targets will be detected by the front radar, the corner radar and the camera at the same time. The above-mentioned combined fusion method cannot fully utilize the information of all sensors, and cannot fully exert the advantages of multi-sensor fusion. In addition, when only two types of sensors are fused, if one type of sensor fails or the detection error is large, it will have a greater impact on the fusion result, resulting in the accuracy and anti-interference ability of the fusion result being relatively weak.

[0124] In addition, when the target trajectory data collected by each sensor is subjected to data association matching by the combined fusion method, only one frame of observation data is used, that is, the target trajectory data collected within one sampling period is used for association calculation. This will ignore the motion state of the target and be more sensitive to abnormal speed. If two different targets within the observation range intersect in space at a certain moment, it is easy to cause misassociation of the targets. When the data of a certain data source is abnormal or erroneous, it may have an adverse effect on the entire association result, thereby affecting the fusion of subsequent data, resulting in low accuracy of the fusion result.

[0125] Therefore, in the technical solution provided in the embodiment of the present application, when fusing the trajectory data, it is necessary to first obtain the first trajectory data collected by at least two radars respectively arranged on the vehicle, and obtain the second trajectory data collected by the camera arranged on the vehicle; then, the first trajectory data collected by at least two radars respectively are first fused to obtain the third trajectory data after fusion processing; finally, the second trajectory data and the third trajectory data are fused to obtain the fourth trajectory data after fusion processing, wherein the first trajectory data includes the observation trajectory data of at least one radar observation target, the second trajectory data includes the observation trajectory data of at least one visual observation target, the third trajectory data includes the first fused trajectory data of at least one radar observation target, and the fourth trajectory data includes the second fused trajectory data of at least one observation target. When fusing the trajectory data, the above technical solution first performs radar trajectory data fusion, and then fuses the fused radar trajectory data with the visual trajectory data, so that the information collected by all sensors is fully utilized, thereby enhancing the accuracy and anti-interference of the trajectory data fusion result, thereby improving the vehicle's environmental perception ability and driving decision accuracy, so that the vehicle can adapt to complex and changeable driving environments.

[0126] Figure 1 FIG. 1 is a flow chart of a trajectory data fusion method in an embodiment of the present application. The trajectory data fusion method can be implemented by a corresponding trajectory data fusion device, such as Figure 1 As shown, the method comprises the following steps:

[0127] Step 101: acquiring first trajectory data respectively collected by at least two radars provided on the vehicle, and acquiring second trajectory data collected by a camera provided on the vehicle;

[0128] In the technical solution provided in the embodiment of the present application, the target trajectory data collected by the sensors installed on the vehicle is acquired. The above-mentioned sensors may include radars and cameras. For the radar installed on the vehicle, it may include a front radar and a corner radar. For example, the vehicle with the above-mentioned 5R1V sensor arrangement, the corner radars include two corner radars at the front of the vehicle and two corner radars at the rear of the vehicle. For the corner radars at the front of the vehicle, they are usually arranged on both sides of the front radar, and the collection ranges of the two are relatively consistent, so the demand for trajectory data fusion is relatively large. Therefore, the at least two radars in the embodiment of the present application include at least two of the front radar, the first corner radar located on the first side of the front radar, and the second corner radar located on the second side of the front radar. Among them, the front radar can detect targets in front, and its detection distance is usually farther and the resolution is higher; the corner radar can detect targets in adjacent lanes and blind spots, and its detection distance is shorter, generally within tens of meters.

[0129] Step 102: fusing the first trajectory data respectively collected by at least two radars to obtain third trajectory data after fusion processing, where the third trajectory data includes the first fused trajectory data of at least one radar observation target;

[0130] On the basis of the above step 101, the embodiment of the present application can obtain first trajectory data respectively collected by at least two radars, and then perform data fusion on the obtained first trajectory data to obtain third trajectory data.

[0131] Step 103: Fusing the second trajectory data and the third trajectory data to obtain fused fourth trajectory data, where the fourth trajectory data includes the second fused trajectory data of at least one observed target.

[0132] This step is to fuse the second trajectory data and the third trajectory data obtained in step 101 and step 102 to obtain fourth trajectory data. The fourth trajectory data includes the second fused trajectory data of at least one observed target, which is the data that fuses the information collected by the radar and the camera and is finally sent to the downstream algorithm module.

[0133] As described above, by first fusing the observation trajectory data of at least two radars and then fusing the fused first fused trajectory data with the observation trajectory data of the camera, the information collected by all sensors can be fully utilized, thereby enhancing the accuracy and anti-interference of the trajectory data fusion result, thereby improving the vehicle's environmental perception ability and driving decision accuracy, and enabling the vehicle to adapt to complex and changeable driving environments.

[0134] In some embodiments, the first trajectory data includes observation trajectory data of at least one radar observation target, specifically, Figure 2 for Figure 1 The schematic diagram of the specific execution flow of obtaining the first trajectory data in step 101 of the embodiment shown in FIG. Figure 2 As shown, the following steps are included:

[0135] Step 201, obtaining observation trajectory data of at least one radar observation target collected by at least two radars in a current collection cycle;

[0136] The collection period in this step may also be referred to as a frame period. A frame of data is collected in each sampling period. The frame of data includes observation trajectory data of at least one radar observation target.

[0137] Step 202: Based on the observation trajectory data of at least one radar observation target collected in the current collection cycle, the historical observation trajectory data of the radar observation target obtained by at least two radars in the historical collection cycle are updated to obtain first trajectory data.

[0138] In an embodiment of the present application, based on the observation trajectory data of at least one radar observation target obtained in step 201, the historical observation trajectory data of the radar observation target obtained by at least two radars in the historical acquisition period is updated. When the first trajectory data is obtained, two situations may also be included. One of the situations is that if the target identifier of at least one radar observation target collected in the current acquisition period can match the target identifier of the radar observation target obtained in the historical acquisition period, the observation trajectory data of at least one radar observation target collected in the current acquisition period is stored as the latest data in the historical observation trajectory data of the radar observation target that matches it, and the length of the historical observation trajectory of the stored radar observation target is increased, but the length of the historical observation trajectory of the stored radar observation target cannot exceed the set observation trajectory length. If it exceeds the set observation trajectory length, trajectory length, the earliest stored observation trajectory data of the radar observation target will be deleted; the second case is that if the target identification of at least one radar observation target collected in the current acquisition cycle cannot match the target identification of the radar observation target obtained in the historical acquisition cycle, in some cases, the observation trajectory data of the above radar observation target can be regarded as a new observation trajectory of the radar observation target for storage, that is, a new observation trajectory data of the radar observation target is added. For example, when the target identification of the radar observation target collected in the current acquisition cycle has not been stored in the historical observation trajectory data of the radar observation target, it means that the observation trajectory data of the radar observation target collected in the current acquisition cycle is the observation trajectory data of the new target, and it is stored as a new observation trajectory data of the radar observation target. The specific execution flow chart can be shown as follows Figure 3 As shown, the following steps are included:

[0139] Step 301: Acquire observation trajectory data of at least one radar observation target collected in the current collection cycle;

[0140] The observation trajectory data of the radar observation target in this step is the observation trajectory data of at least one radar observation target collected in the current collection cycle.

[0141] Step 302: Acquire historical observation trajectory data of the radar observation target obtained within the historical collection period;

[0142] The historical observation trajectory data of the radar observation target in this step is the observation trajectory data of the radar observation target that has been stored.

[0143] Step 303, determining whether the target identification of the radar observation target can be matched;

[0144] Specifically, a judgment is made based on the target identification of at least one radar observation target collected in the current collection cycle and the target identification of the radar observation target obtained in the historical collection cycle. When the two match, step 304 is executed to update the historical observation trajectory data based on the observation trajectory data of at least one radar observation target collected in the current collection cycle; otherwise, step 305 is executed.

[0145] Step 304: Update historical observation trajectory data;

[0146] This step is to update the historical observation trajectory data of the radar observation target after determining in step 303 that the target identification of at least one radar observation target collected in the current collection cycle matches the target identification of the radar observation target obtained in the historical collection cycle.

[0147] Step 305, determining whether the target identifier of the observation trajectory data of at least one radar observation target collected in the current collection period is stored in the historical observation trajectory data;

[0148] This step is to determine whether the target identification of at least one radar observation target collected in the current collection cycle is stored in the historical observation trajectory data after the target identification is determined not to match in step 303. If the target identification is not stored in the historical observation trajectory data of the radar observation target, step 306 is executed.

[0149] Step 306: Add observation trajectory data of at least one radar observation target collected in the current collection cycle.

[0150] After it is determined in the above step 305 that the target identifier of at least one radar observation target collected in the current collection cycle is not stored in the historical observation trajectory data, a new observation trajectory data of the radar observation target is added.

[0151] In some embodiments, the second trajectory data includes observation trajectory data of at least one visual observation target, specifically, Figure 4 for Figure 1 The specific execution flow diagram of obtaining the second trajectory data in step 101 of the embodiment shown in FIG. Figure 4 As shown, the following steps are included:

[0152] Step 401: Obtain observation trajectory data of at least one visual observation target collected by a camera in a current collection cycle;

[0153] Step 402: Based on the observation trajectory data of at least one visual observation target collected in the current collection cycle, the historical observation trajectory data of the visual observation target obtained by the camera in the historical collection cycle is updated to obtain second trajectory data.

[0154] In an embodiment of the present application, the historical observation trajectory data of the visual observation target obtained by the camera in the historical acquisition period is updated based on the observation trajectory data of at least one visual observation target obtained in step 401. When the second trajectory data is obtained, two situations are also included. One of the situations is that if the target identifier of at least one visual observation target collected by the camera in the current acquisition period can match the target identifier of the visual observation target obtained by the camera in the historical acquisition period, the observation trajectory data of at least one visual observation target collected by the camera in the current acquisition period is stored as the latest data in the historical observation trajectory data of the visual observation target that matches it, and the length of the stored historical observation trajectory of the visual observation target is increased, but the stored historical observation trajectory of the visual observation target is The trajectory length cannot exceed the set observation trajectory length. If it exceeds the set observation trajectory length, the trajectory data of the earliest stored visual observation target will be deleted; the second case is that if the target identifier of at least one visual observation target collected by the camera in the current acquisition cycle cannot match the target identifier of the visual observation target obtained by the camera in the historical acquisition cycle, and the target identifier of at least one visual observation target collected by the camera in the current acquisition cycle has not been stored in the historical observation trajectory data of the visual observation target, it means that the observation trajectory data of at least one visual observation target collected by the camera in the current acquisition cycle is the observation trajectory data of the new target, and it will be stored as a new observation trajectory data of the visual observation target. The specific execution flow chart can be shown as follows Figure 5 As shown, the following steps are included:

[0155] Step 501: Obtain observation trajectory data of at least one visual observation target collected by a camera in a current collection cycle;

[0156] The observation trajectory data of the visual observation target in this step is the observation trajectory data of at least one visual observation target collected by the camera in the current collection cycle.

[0157] Step 502: Acquire historical observation trajectory data of the visual observation target obtained by the camera in a historical acquisition cycle;

[0158] The historical observation trajectory data of the visual observation target in this step is the observation trajectory data of the visual observation target that has been stored.

[0159] Step 503: determine whether the target identifier can be matched;

[0160] Specifically, a judgment is made based on the target identification of at least one visual observation target collected by the camera in the current acquisition cycle and the target identification of the visual observation target obtained by the camera in the historical acquisition cycle. When the two match, step 504 is executed to update the historical observation trajectory data based on the observation trajectory data of at least one visual observation target collected by the camera in the current acquisition cycle; otherwise, step 505 is executed.

[0161] Step 504: Update historical observation trajectory data;

[0162] This step is to update the historical observation trajectory data of the visual observation target after determining in step 503 that the target identifier of at least one visual observation target collected by the camera in the current collection cycle matches the target identifier of the visual observation target obtained by the camera in the historical collection cycle.

[0163] Step 505: determine whether the target identification of at least one visual observation target collected by the camera in the current collection cycle is stored in the historical observation trajectory data;

[0164] This step is to judge whether the target identification of at least one visual observation target collected by the camera in the current collection cycle is stored in the historical observation trajectory data after determining that the target identification does not match in step 503. If the above target identification is not stored in the historical observation trajectory data of the visual observation target, execute step 506.

[0165] Step 506: Add the observation trajectory data collected by the camera in the current collection cycle.

[0166] After determining in the above step 505 that the target identifier of at least one visual observation target collected by the camera in the current collection cycle is not stored in the historical observation trajectory data, a new observation trajectory data of the visual observation target is added.

[0167] In the embodiment of the present application, the front radar, corner radar, camera and other sensors configured on the vehicle may be affected by electromagnetic, similar objects, lighting conditions, bad weather and other factors when sensing the surrounding environment, and may collect some erroneous data or some data that affects the vehicle's driving. Therefore, before fusing the observed trajectory data, the above data needs to be pre-processed, for example, filtering and screening invalid or erroneous data. Specifically, it may include at least one of the following situations:

[0168] First, after obtaining the observation trajectory data of at least one radar observation target collected by at least two radars in the current acquisition cycle, it is also necessary to filter the observation trajectory data of at least one radar observation target collected by at least two radars in the current acquisition cycle based on preset trajectory filtering conditions to retain the observation trajectory data of the radar observation target that meets the preset trajectory filtering conditions.

[0169] Second, after obtaining the observation trajectory data of at least one visual observation target collected by the camera in the current acquisition cycle, it is necessary to filter the observation trajectory data of at least one visual observation target collected by the camera in the current acquisition cycle based on preset trajectory filtering conditions to retain the observation trajectory data of the visual observation target that meets the preset trajectory filtering conditions.

[0170] In some embodiments, the preset trajectory screening conditions include at least one of a relative position threshold, a relative speed threshold, a target confidence threshold, a target type, a target identification, and a validity identifier. These trajectory screening conditions are mainly used to screen which radar observation targets and / or visual observation targets' observation trajectory data are retained. Specifically, the relative position threshold is mainly used to limit the distance between the observation target and the vehicle, that is, the observation trajectory data contained in the observation target exceeding the relative position threshold will be screened out, the relative speed threshold is used to limit the relative speed between the observation target and the vehicle, and the observation target whose relative speed exceeds the relative speed threshold can be screened out, and the relative speed and relative position mutually confirm the data that needs to be filtered; and the target confidence threshold, target type, target identification, and validity identifier are mainly used to determine the observation target that the vehicle clearly needs to pay attention to. If it meets this type of screening conditions, it can be screened out. In this way, it is possible to prevent the observation trajectory data of erroneous clutter or invalid observation targets from occupying computing power.

[0171] In some embodiments, before the observation trajectory data is fused, the observation trajectory data of the observation target collected by each sensor needs to be converted into a coordinate system, that is, the coordinate system where the observation trajectory data of the observation target collected by the sensor is located is rotated and translated so that it completely overlaps with another coordinate system. This is mainly because the coordinate systems of the data collected by multiple types of sensors such as corner radar, front radar and camera equipped on the vehicle are different. For example, the coordinate system where the data collected by millimeter-wave radars such as corner radar and front radar is located generally takes the millimeter-wave radar as the coordinate origin, the irradiation direction of the millimeter-wave radar as the x-axis, the left side of the radar irradiation direction as the y-axis direction, and the vertical upward as the z-axis direction; while the coordinate system where the data collected by the camera is located, the coordinate origin is at the optical center position, the z-axis faces forward, the x-axis points to the right, and the y-axis points downward. Therefore, it is necessary to convert the observation trajectory data of the observation target collected by the sensor so that the observation trajectory data of the observation target collected by each sensor are represented in the same coordinate system, so as to facilitate the subsequent fusion of the observation trajectory data of the observation target.

[0172] The coordinate system conversion in the above embodiment may include at least one of the following situations:

[0173] First, after the observation trajectory data of at least one radar observation target collected by at least two radars in the current collection cycle are filtered based on preset filtering conditions, the observation trajectory data of at least one radar observation target needs to be converted into a coordinate system.

[0174] Second, after filtering the observation trajectory data of at least one visual observation target collected by the camera in the current collection cycle based on preset filtering conditions, it is also necessary to perform coordinate system conversion on the observation trajectory data of at least one visual observation target.

[0175] In an embodiment of the present application, the above-mentioned observation trajectory data includes position data and speed data of the observation target. Therefore, when the observation trajectory data of at least one radar observation target is converted into a coordinate system, the position data and speed data of the radar observation target need to be converted into a coordinate system respectively.

[0176] When the coordinate system conversion is performed on the position data of at least one radar observation target, the position data of at least one radar observation target is converted to the vehicle rear axle coordinate system. Specifically, the position data of the radar observation target collected by radar sensors such as the front radar and the corner radar is converted from the front bumper coordinate system to the vehicle rear axle coordinate system based on the center of the rear axle of the vehicle. The conversion relationship for converting the position data of the radar observation target from the front bumper coordinate system to the vehicle rear axle coordinate system can be set as:

[0177] X POS =X POS+DistRearAxleToFrontBumper

[0178] Among them, X POS It is the longitudinal position of the target observed by the front radar and corner radar, and DistRearAxleToFrontBumper is the distance from the front bumper to the rear axle.

[0179] When the speed data of at least one radar observation target is converted to an absolute coordinate system, the speed data of at least one radar observation target is converted to an absolute coordinate system. Specifically, the speed data of the radar observation target collected by radar sensors such as the front radar and the corner radar is converted from a relative vehicle coordinate system to an absolute coordinate system, wherein the absolute coordinate system can also be regarded as a ground coordinate system, which is a fixed coordinate system relative to the surface of the earth. In this coordinate system, the position of the vehicle is measured relative to a fixed point on the surface of the earth. The above conversion relationship for converting the speed data of the radar observation target from the relative vehicle coordinate system to the absolute coordinate system can be set as:

[0180] Vel x =((-YawRate×Y POS )+Vel x )+(Speed)

[0181] Vel y =((-YawRate×Y POS )+Vel x )

[0182] Acc x =((-YawRate 2 ×Y POS )-(2×YawRate×Vel y )+Acc x +Acc)

[0183] Acc y =((-YawRate 2 ×Y POs )+(2×YawRate×Vel x )+Acc y +(YawRate×Speed))

[0184] Among them, Vel x Vel is the longitudinal velocity of the target observed by the radar, y is the lateral velocity of the radar observation target, YawRate is the vehicle yaw rate, Speed ​​is the vehicle speed, Acc x is the longitudinal acceleration of the radar observation target, Acc yis the lateral acceleration of the radar observation target, and Acc is the acceleration of the vehicle.

[0185] In the above embodiment, the coordinate system conversion of the observation trajectory data of at least one radar observation target includes the coordinate system conversion of the position data and the speed data of the radar observation target. In contrast, when the coordinate system conversion of the observation trajectory data of at least one visual observation target is performed, since general cameras can directly output speed data relative to the absolute coordinate system, it is only necessary to convert the position data of the at least one visual observation target to the vehicle rear axle coordinate system, and there is no need to perform coordinate system conversion on the speed data of the at least one visual observation target.

[0186] The above-mentioned conversion of the position data of at least one visual observation target to the vehicle rear axle coordinate system refers to converting the position data of the visual observation target collected by the camera from the camera installation position coordinate system to the vehicle rear axle coordinate system. The above-mentioned conversion relationship of converting the position data of the visual observation target from the camera installation position coordinate system to the vehicle rear axle coordinate system can be set as:

[0187] X POS =X POS +DistCameraToRearAxle

[0188] Y POS =Y POS +DistCameraToRearXAxis

[0189] Among them, X POS is the longitudinal position of the visual observation target, Y POS The lateral position of the visual observation target, DistCameraToRearAxle is the longitudinal distance between the camera and the rear axle, and DistCameraToRearXAxis is the lateral distance between the camera and the longitudinal center axis of the vehicle.

[0190] In the above Figure 2 to Figure 3 In an embodiment, the observation trajectory data of at least one radar observation target is collected in the current acquisition cycle, and then the historical observation trajectory data is updated or the new trajectory data is added for explanation; however, in some other embodiments, if the observation trajectory data of at least one radar observation target is not collected in the current acquisition cycle, the observation trajectory data of at least one radar observation target in the current acquisition cycle can be determined based on the historical observation trajectory data of at least one radar observation target in the previous two acquisition cycles.

[0191] Specifically, if the observation trajectory data of at least one radar observation target is not collected in the current collection period, it means that the observation trajectory data of at least one radar observation target has not been newly input in the current collection period. If the above situation occurs for the first time, the observation trajectory data of at least one radar observation target in the current collection period can be determined by any one of the following two methods:

[0192] The first method is that if the observation trajectory data collected by at least one radar observation target in at least two historical collection cycles have been stored, the observation trajectory data collected in the first two historical collection cycles are used for data extrapolation, so as to predict the observation trajectory data of at least one radar observation target that may be collected in the current collection cycle;

[0193] The second method is that the observation trajectory data collected by at least one radar observation target in only one historical acquisition cycle has been stored. At this time, it is only necessary to copy the observation trajectory data collected by the above-mentioned at least one radar observation target in only one historical acquisition cycle, and store the copied observation trajectory data as the observation trajectory data of at least one radar observation target collected in the current acquisition cycle.

[0194] Specifically, Figure 6 As shown, in Figure 3 Based on the embodiment shown, it also includes:

[0195] Step 307, determining whether the target identifier of the radar observation target that has been stored as historical trajectory data can be matched with the same target identifier in at least one radar observation target collected in the current collection period;

[0196] This step is after step 305, to determine whether the target identifier of the radar observation target that has been stored as historical trajectory data can match the same target identifier in at least one radar observation target collected in the current collection period. If the same target identifier is not matched, step 308 is executed.

[0197] Step 308, determining whether the observation trajectory data of at least one radar observation target is not collected more than twice;

[0198] After determining in the above step 307 that the same target identifier is not matched, it is determined whether the observation trajectory data of at least one radar observation target has not been collected for more than two times. If it is the first time, step 309 is executed.

[0199] Step 309: extrapolate the historical observation trajectory data of the previous two acquisition cycles to predict the observation trajectory data of at least one radar observation target in the current acquisition cycle;

[0200] After determining in the above step 308 that it is the first time that the observation trajectory data of at least one radar observation target has not been collected, the observation trajectory data collected in the first two historical collection cycles are used for data extrapolation to predict the observation trajectory data of at least one radar observation target that may be collected in the current collection cycle; if the historical observation trajectory data of at least one radar observation target only includes the observation trajectory data of at least one radar observation target collected in one historical collection cycle, the historical observation trajectory data of the at least one radar observation target is copied, and the copied observation trajectory data is stored as the observation trajectory data of the radar observation target collected in the current collection cycle.

[0201] In some embodiments, if the observation trajectory data of at least one radar observation target is not collected for N consecutive collection cycles, the historical observation trajectory data of at least one radar observation target is deleted, where N is a positive integer greater than or equal to 2.

[0202] Specifically, Figure 8 As shown, in Figure 6 Based on the embodiment shown, it also includes:

[0203] Step 310: Delete historical observation trajectory data of at least one radar observation target.

[0204] This step is to delete the historical observation trajectory data of at least one radar observation target after it is determined in step 308 that the observation trajectory data of at least one radar has not been collected for more than two times. Figure 4 to Figure 5 In an embodiment, the observation trajectory data of at least one visual observation target collected by the camera in the current acquisition cycle is used, and then the historical observation trajectory data is updated or the new trajectory data is added for explanation; however, in some other embodiments, if the observation trajectory data of at least one visual observation target is not collected in the current acquisition cycle, the observation trajectory data of at least one visual observation target in the current acquisition cycle can be determined based on the historical observation trajectory data of at least one visual observation target in the previous two acquisition cycles.

[0205] Specifically, if the observation trajectory data of at least one visual observation target is not collected in the current collection period, it means that the observation trajectory data of at least one visual observation target has not been newly input in the current collection period. If the above situation occurs for the first time, the observation trajectory data of at least one visual observation target in the current collection period can be determined by any one of the following two methods:

[0206] The first method is that the observation trajectory data collected by at least one visual observation target in at least two historical collection cycles have been stored, and then the observation trajectory data collected in the first two historical collection cycles are used for data extrapolation, so as to predict the observation trajectory data of at least one visual observation target that may be collected in the current collection cycle;

[0207] The second method is that the observation trajectory data of at least one visual observation target collected only in one historical collection cycle has been stored. At this time, it is only necessary to copy the observation trajectory data of the at least one visual observation target collected only in one historical collection cycle, and store the copied observation trajectory data as the observation trajectory data of at least one visual observation target collected in the current collection cycle.

[0208] Specifically, Figure 7 As shown, in Figure 5 Based on the embodiment shown, it also includes:

[0209] Step 507: determine whether the target identifier of the visual observation target that has been stored as historical trajectory data can be matched with the same target identifier in at least one visual observation target collected in the current collection period;

[0210] This step is after step 505, to determine whether the target identification of the visual observation target that has been stored as historical trajectory data can match the same target identification in the visual observation target collected in the current collection period. If the same target identification is not matched, step 508 is executed.

[0211] Step 508, determining whether the observation trajectory data of at least one visual observation target is not collected more than twice;

[0212] After determining in the above step 507 that the same target identifier is not matched, it is determined whether the observation trajectory data of at least one visual observation target is not collected for more than two times. If it is the first time, step 509 is executed.

[0213] Step 509: extrapolate the historical observation trajectory data of the previous two acquisition cycles to predict the observation trajectory data of at least one visual observation target in the current acquisition cycle.

[0214] After determining in the above step 508 that it is the first time that the observation trajectory data of at least one visual observation target has not been collected, the observation trajectory data collected in the first two historical collection cycles are used for data extrapolation to predict the observation trajectory data of at least one visual observation target that may be collected in the current collection cycle; if the historical observation trajectory data of at least one visual observation target only contains the observation trajectory data of at least one visual observation target collected in one historical collection cycle, the historical observation trajectory data of the at least one visual observation target is copied, and the copied observation trajectory data is stored as the observation trajectory data of at least one visual observation target collected in the current collection cycle.

[0215] In some embodiments, if observation trajectory data of at least one visual observation target is not collected for N consecutive collection cycles, the historical observation trajectory data of the at least one visual observation target is deleted, where N is a positive integer greater than or equal to 2.

[0216] Specifically, Fig. 9 As shown, in Figure 7 Based on the embodiment shown, it also includes:

[0217] Step 5010: Delete historical observation trajectory data of at least one visual observation target.

[0218] This step is to delete the historical observation trajectory data of at least one visual observation target after it is determined in step 508 that the observation trajectory data of at least one visual observation target has not been collected for more than two times.

[0219] In the above embodiment, if the corresponding observation trajectory data of at least one radar observation target or at least one visual observation target is not collected in at least two consecutive acquisition cycles, it means that the observation trajectory data of at least one radar observation target or at least one visual observation target has not received new input in at least two consecutive historical acquisition cycles, that is, the observation trajectory data of at least one radar observation target or at least one visual observation target has not been updated in at least two consecutive historical acquisition cycles, and the observation trajectory data of at least one radar observation target or at least one visual observation target that has not been updated is deleted.

[0220] As described above, by updating the observation trajectory data collected by the sensor in the current acquisition cycle, when the sensor has not collected at least one radar or at least one visual observation trajectory data in the current acquisition cycle, the historical observation trajectory data of the sensor in the previous two acquisition cycles is used to predict the observation trajectory data of at least one radar observation target or at least one visual observation target in the current acquisition cycle, or when the sensor has not collected observation trajectory data in N consecutive acquisition cycles, the corresponding historical observation trajectory data collected by the sensor is deleted, etc., thereby achieving the purpose of managing the observation trajectory data collected by the sensor, thereby facilitating the fusion of subsequent observation trajectory data.

[0221] In the above embodiment of the present application, the observation trajectory data of the radar observation target collected by the radar is first fused. The specific implementation method can be found in the following embodiment. In this embodiment, for the convenience of description, at least two radars are respectively referred to as the first radar and the second radar, wherein the first radar can be any one of the front radar and the corner radar, and the second radar can be any one of the corner radar and the front radar. The two can be different radars, and there is no limitation in this embodiment.

[0222] In an embodiment of the present application, it is necessary to first determine the mutually related first radar observation target and the second radar observation target based on the first trajectory data collected by at least two radars respectively, and then fuse the observation trajectory data of the mutually related first radar observation target and the observation trajectory data of the second radar observation target. This can ensure that the subsequent fusion of the first trajectory data is based on the related data source, thereby improving the effectiveness and accuracy of the fusion result.

[0223] Specifically, Fig.10 for Figure 1 The specific execution flow diagram of step 102 of the embodiment shown in FIG. Fig.10 As shown, the following steps are included:

[0224] Step 601: determining a first radar observation target and a second radar observation target that are associated with each other based on observation trajectory data of at least one first radar observation target collected by a first radar and observation trajectory data of at least one second radar observation target collected by a second radar;

[0225] This step is to determine the first radar observation target and the second radar observation target that can be associated with each other by using corresponding calculation processing after the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar. The first radar observation target can be a radar observation target identified by some target recognition algorithms from the radar data collected by the first radar, as the first radar observation target, and the observation trajectory data of the first radar observation target can be obtained; the second radar observation target can be a radar target identified by some target recognition algorithms from the radar data collected by the second radar, as the second radar target, and the observation trajectory data of the second radar target can be obtained.

[0226] Step 602: Fusing the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target that are mutually related to each other to obtain first fused trajectory data of at least one radar observation target.

[0227] This step is based on step 601, and the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target that are mutually related are fused to obtain first fused trajectory data of at least one radar observation target. The fusion processing can use a Kalman filter algorithm to fuse the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target that are mutually related. The Kalman filter algorithm is an algorithm that uses a linear system state equation to optimally estimate the system state through system input and output observation data.

[0228] Fig.11 for Fig.10 The specific execution flow diagram of step 601 of the embodiment shown in FIG. Fig.11 As shown, the following steps are included:

[0229] Step 701: Calculate a trajectory distance between the first radar observation target and the second radar observation target based on the observation trajectory data of at least one first radar observation target and the observation trajectory data of at least one second radar observation target;

[0230] In an embodiment of the present application, the trajectory distance between two radar observation targets may be obtained, and the correlation between the two radar observation targets may be determined by the trajectory distance.

[0231] In some embodiments, the above-mentioned trajectory distance can be determined by the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, and at least one of the distance variance cost, the cycle number related cost, the speed cost and the near-point target penalty.

[0232] For example, the trajectory distance between the first radar observation target and the second radar observation target may be calculated by the following formula:

[0233] Dist1=(1+MagnifyCost1)*MDist1+TraSizeCost1+SpeedCost1+NearbyCost1

[0234] Among them, Dist1 is the trajectory distance between the first radar observation target and the second radar observation target, MDist1 is the inter-trajectory Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, MagnifyCost1 is the distance variance cost of the trajectory distance between the first radar observation target and the second radar observation target, TraSizeCost1 is the cost related to the number of cycles of the first radar observation target and the second radar observation target, SpeedCost1 is the speed cost of the first radar observation target and the second radar observation target, and NearbyCost1 is the near-point target penalty of the first radar observation target and the second radar observation target.

[0235] Among them, the inter-trajectory Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target is mainly used to measure the similarity between the first radar observation target and the second radar observation target, which can be set as:

[0236]

[0237] Wherein, MDist1 is the inter-trajectory Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, μ is the mean of the observation target, ∑ is the covariance matrix of all observation targets, and X is any one of the distance in the x direction, the distance in the y direction, and the speed in the x direction between the first radar observation target and the second radar observation target.

[0238] Among them, the distance variance cost is mainly used to reflect the stability of the trajectory distance between the first radar observation target and the second radar observation target during the acquisition cycle. If it is unstable, it means that the first radar observation target and the second radar observation target are more likely to be not the same observation target, and their correlation is weak. The above distance variance cost can be set as:

[0239]

[0240] Among them, MagnifyCost1 is the distance variance cost between the first radar observation target and the second radar observation target, n is the number of acquisition cycles, MDist iis the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target in each acquisition cycle, It is the average Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target within n acquisition cycles.

[0241] Among them, the cycle number related cost is mainly used to reduce the impact of the inconsistent trajectory data lengths of the first radar observation target and the second radar observation target on the trajectory distance between the first radar observation target and the second radar observation target, which can be set as:

[0242] TraSizeCost1=FrameGainA1+FrameGainB1+FrameGainC1

[0243] Among them, TraSizeCost1 is the cycle number related cost of the first radar observation target and the second radar observation target, and FrameGainA1, FrameGainB1, and FrameGainC1 are all correction values ​​of the acquisition cycle.

[0244] In this application example, it can be set to store at least the observation trajectory data of the first radar observation target or the observation trajectory data of the second radar observation target within one acquisition cycle, or to store at most the observation trajectory data of the first radar observation target or the observation trajectory data of the second radar observation target within three acquisition cycles.

[0245] Among them, the speed cost is mainly used to make up for the defects in calculating the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target due to the small position difference between the first radar observation target and the second radar observation target. Therefore, it is necessary to increase the horizontal speed cost and the longitudinal speed cost as an effective compensation for the possible deficiencies in the Mahalanobis distance calculation. The above speed cost can be set as:

[0246]

[0247] Among them, SpeedCost1 is the speed cost of the first radar observation target and the second radar observation target, n is the number of acquisition cycles, VecDiffi i is the lateral speed or longitudinal speed of the first radar observation target and the second radar observation target in each acquisition cycle, It is the average lateral velocity or average longitudinal velocity of the first radar observation target and the second radar observation target within n acquisition cycles.

[0248] In the above embodiment, if the first radar observation target and the second radar observation target are within the specified near-point range in the x-direction of the vehicle, the first radar observation target and the second radar observation target are set as near-point observation targets, and then the speed cost of the first radar observation target and the second radar observation target is increased to ensure the accuracy of subsequent observation trajectory fusion. For example, if the first radar observation target and the second radar observation target are within 19m in the x-direction of the vehicle, the first radar observation target and the second radar observation target are set as near-point observation targets, and the speed cost of the first radar observation target and the second radar observation target is doubled.

[0249] Among them, the near-point target penalty is mainly applicable to the situation when the near-point observation target is a pedestrian and the above-mentioned speed cost cannot take effect. In the embodiment of the present application, when the near-point observation target is a pedestrian, the near-point target penalty is added, and its value is set to 0.5gate. Gate is the penalty threshold value, and its value is set differently in different sensor combination fusion methods. For example, in the combined fusion method based on the front radar and the corner radar, the gate value is set to 6, and in the combined fusion method based on the radar and the camera, the gate value is set to 12.

[0250] In some embodiments, if the first radar is a corner radar and the second radar is another corner radar, it is necessary to consider the additional limited distance caused by the position offset between the two radars. At this time, when determining the trajectory distance between the target observed by the first radar and the target observed by the second radar, it is necessary to base it on the inter-trajectory Mahalanobis distance between the observation trajectory data of the target observed by the first radar and the observation trajectory data of the target observed by the second radar, the additional limited distance between the first radar and the second radar, and at least one of the distance variance cost, cycle number related cost, speed cost and near-point target penalty.

[0251] Step 702: Determine the first radar observation target and the second radar observation target that are associated with each other based on the track distance between the first radar observation target and the second radar observation target.

[0252] This step is based on step 701 and uses the calculated track distance between the first radar observation target and the second radar observation target to determine the first radar observation target and the second radar observation target that are associated with each other.

[0253] In some embodiments, if the trajectory distance between the first radar observation target and at least two second radar observation targets has been calculated, the first radar observation target and the second radar observation target with the smallest trajectory distance are determined to be associated with each other. In this embodiment of the present application, the nearest neighbor association algorithm can be used to determine the first radar observation target and the second radar observation target with the smallest trajectory distance.

[0254] In an embodiment of the present application, before determining the mutually associated first radar observation target and second radar observation target based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar, the first radar observation target and the second radar observation target that do not meet the mutual correlation condition can also be determined based on the speed information and / or distance information of the first radar observation target and the second radar observation target. That is, the first radar observation target and the second radar observation target that obviously do not meet the mutual correlation condition can be determined in advance. After the two radar observation targets that do not meet the mutual correlation condition are determined in advance, the trajectory distance calculation using the above algorithm can be no longer performed on them, thereby avoiding unnecessary calculations and saving computing resources.

[0255] In some embodiments, the speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

[0256] In the above embodiment, when determining the first radar observation target and the second radar observation target that do not meet the mutual correlation condition according to the speed information, it is possible to judge whether the two observation targets meet the mutual correlation condition according to the longitudinal speed value, the lateral speed value, and the longitudinal speed difference value. In some cases, when judging whether the two observation targets meet the mutual correlation condition according to the longitudinal speed value, the mutual correlation condition is set as follows: one of the observation targets does not have a pseudo longitudinal motion state, that is, the absolute value of its longitudinal absolute speed does not exceed 2m / s; and the other observation target does not have a pseudo longitudinal static state, that is, the absolute value of its longitudinal absolute speed is less than 1m / s. For example, if the absolute value of the longitudinal absolute speed of the first radar observation target is greater than 2m / s, it is regarded as an observation target with a pseudo longitudinal motion state, and if the absolute value of the longitudinal absolute speed of the second radar observation target is less than 1m / s, it is regarded as an observation target with a pseudo longitudinal static state, and then it is determined that the two observation targets do not meet the above mutual correlation condition, that is, it indicates that the first radar observation target and the second radar observation target are not correlated with each other. After that, the speed threshold value is directly assigned to the two observation targets that do not meet the mutual correlation condition. The speed threshold value will vary according to different sensor combination fusion methods. For example, in the combination fusion method based on the front radar and the corner radar, its value is set to 6, and in the combination fusion method based on the radar and the camera, its value is set to 12. For example, if the first radar observation target is an observation target in a pseudo-longitudinal motion state, and the second radar observation target is an observation target in a pseudo-longitudinal stationary state, the first radar observation target and the second radar observation target do not meet the mutual correlation condition. At this time, the speed threshold value of 6 is assigned to the first radar observation target and the second radar observation target.

[0257] In addition, in some cases, it is also possible to determine whether the first radar observation target and the second radar observation target meet the mutual correlation condition based on the lateral speed. At this time, the mutual correlation condition is set as follows: the two observation targets do not belong to opposite observation targets, and the above-mentioned opposite observation targets refer to the lateral speed of one observation target being greater than 2m / s, and the lateral speed of the other observation target being less than -2m / s. For example, if the lateral speed of the first radar observation target is greater than 2m / s, and the lateral speed of the second radar observation target is less than -2m / s, then the above-mentioned two observation targets are regarded as two opposite observation targets, and thus do not meet the above-mentioned mutual correlation condition, that is, it indicates that the first radar observation target and the second radar observation target are not related to each other. Subsequently, the speed threshold value is directly assigned to the above-mentioned two observation targets that do not meet the mutual correlation condition.

[0258] Alternatively, in some other cases, when judging whether the first radar observation target and the second radar observation target meet the mutual correlation condition based on the longitudinal speed difference, the mutual correlation condition is set as: the longitudinal speed difference of the two observation targets does not exceed 12m / s. For example, if the longitudinal speed difference between the first radar observation target and the second radar observation target exceeds 12m / s, it means that the two observation targets do not meet the mutual correlation condition, that is, the first radar observation target and the second radar observation target are not related to each other, and then the speed threshold value is directly assigned to the two observation targets that do not meet the mutual correlation condition.

[0259] When determining the first radar observation target and the second radar observation target that do not satisfy the mutual correlation according to the distance information, it is possible to judge whether the two observation targets satisfy the mutual correlation condition according to the longitudinal distance and the lateral distance of the two observation targets. At this time, the mutual correlation condition is set as: the longitudinal distance of the two observation targets does not exceed 12m or the lateral distance does not exceed 2m. For example, if the longitudinal distance of the first radar observation target and the second radar observation target exceeds 12m or the lateral distance exceeds 2m, it is judged that the above two observation targets do not satisfy the above mutual correlation condition, that is, it indicates that the first radar observation target and the second radar observation target are not mutually correlated. Subsequently, the distance threshold value is directly assigned to the two observation targets that do not satisfy the mutual correlation condition. The above distance threshold value will vary according to different sensor combination fusion methods. For example, in the combination fusion method based on the front radar and the corner radar, its value is set to 6, and in the combination fusion method based on the radar and the camera, its value is set to 12.

[0260] In the above-mentioned various situations, the first radar observation target and the second radar observation target that do not satisfy the mutual correlation condition are determined by using the above-mentioned speed information and distance information. One or several situations can be selected for judgment, and when multiple situations are selected, the execution order can be adjusted to determine the first radar observation target and the second radar observation target that do not satisfy the mutual correlation condition. Ultimately, the first radar observation target and the second radar observation target that do not satisfy the mutual correlation condition are not involved in the subsequent trajectory distance calculation, thereby reducing the amount of calculation.

[0261] In the above embodiment, the first radar and the second radar may be a corner radar and a front radar. Fig.12 In the illustrated embodiment, the observation trajectory data of the first radar observation target collected by the corner radar is input and fused with the observation trajectory data of the second radar observation target collected by the front radar, and with the observation trajectory data of the second radar observation target collected by another corner radar as an example to illustrate the fusion of observation trajectory data collected by different radars. Fig.12 This is a flow chart of a specific embodiment of the present application. Fig.12 As shown, the following steps are included:

[0262] Step 801, inputting observation trajectory data of a first radar observation target collected by a corner radar;

[0263] The observation trajectory data of the first radar observation target collected by the corner radar in this step includes the observation trajectory data of at least one first radar observation target collected by the corner radar.

[0264] Step 802, determining whether the observation trajectory data of the first radar observation target collected by the corner radar has been traversed;

[0265] On the basis of the above step 801, it is judged whether the traversal of the observation trajectory data of the first radar observation target collected by the diagonal radar is completed. If the traversal of the observation trajectory data of the above first radar observation target is completed, step 8021 is executed, otherwise step 803 is executed.

[0266] Step 803, storing the observation trajectory data of the first radar observation target collected by the corner radar;

[0267] After determining in the above step 802 that the observation trajectory data of the first radar observation target collected by the corner radar has not been traversed, the observation trajectory data of the first radar observation target that has been traversed is stored.

[0268] Step 804, determining whether the observation trajectory data of the observation target collected by the stored sensor has been traversed;

[0269] This step is to judge whether the traversal of the observation trajectory data of the observation target collected by the stored sensor is completed after the observation trajectory data of the first radar observation target collected by the corner radar in the above embodiment is stored. If the traversal of the observation trajectory of the above observation target is completed, execute step 805, otherwise execute step 806.

[0270] Step 805, traverse the observation trajectory data of the first radar observation target collected by the next corner radar;

[0271] After it is determined in the above step 804 that the observation trajectory data of the observation target collected by the stored sensor has been traversed, the observation trajectory data of the first radar observation target collected by the next corner radar is traversed.

[0272] Step 806, determining whether the stored observation trajectory data of the observation target collected by the sensor is the observation trajectory data of the observation target of the second radar collected by the front radar;

[0273] After it is determined in the above step 804 that the observation trajectory data of the observation target collected by the stored sensor has not been traversed, it is determined whether the observation trajectory data of the observation target collected by the stored sensor is the observation trajectory data of the observation target collected by the second radar collected by the front radar. If the observation trajectory data of the observation target collected by the above stored sensor is the observation trajectory data of the observation target collected by the second radar collected by the front radar, execute step 807; otherwise, execute step 8012.

[0274] Step 807: Preliminarily determine whether the observed target is associated based on the speed information and distance information of the observed target;

[0275] The observation targets in this step include the first radar observation target and the second radar observation target, that is, at least one of the speed information and distance information of the first radar observation target and the second radar observation target is used to determine whether the two observation targets are associated. The specific process of the determination can refer to the above embodiment. The speed information includes at least one of the longitudinal speed value, the lateral speed value and the longitudinal speed difference value; the distance information includes at least one of the longitudinal distance and the lateral distance. If the two observation targets are determined to be associated with each other based on the above conditions, step 809 is executed, otherwise step 808 is executed.

[0276] Step 808: The first radar observation target and the second radar observation target are not related to each other;

[0277] Step 809, calculating the track distance between the first radar observation target and the second radar observation target;

[0278] After preliminarily determining that the first radar observation target and the second radar observation target are related to each other based on the speed information and distance information of the first radar observation target and the second radar observation target in the step 807, the trajectory distance between the first radar observation target and the second radar observation target is calculated using corresponding variables, wherein the variables include the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, and at least one of the distance variance cost, the cycle number related cost, the speed cost and the near-point target penalty.

[0279] Step 8010: determine whether the trajectory distance between the first radar observation target and the second radar observation target is less than a threshold;

[0280] This step is to compare the trajectory distance between the first radar observation target and the second radar observation target with a preset threshold value after the trajectory distance between the first radar observation target and the second radar observation target is calculated in step 809 to determine whether the above trajectory distance is less than the threshold value, the above threshold value is the threshold value gate, if the above trajectory distance is less than the threshold value, execute step 8020, otherwise execute step 8011.

[0281] Step 8011: The first radar observation target and the second radar observation target are not related to each other;

[0282] Step 8012: determine whether the stored observation trajectory data of the observation target collected by the sensor is the observation trajectory data of the second radar observation target collected by another corner radar;

[0283] After determining in the above step 806 that the observation trajectory data of the observation target collected by the stored sensor is not the observation trajectory data of the second radar observation target collected by the front radar, then determine whether the observation trajectory data of the observation target stored by the sensor is the observation trajectory data of the second radar observation target collected by another corner radar. If the observation trajectory data of the above observation target is the observation trajectory data of the second radar observation target, execute step 8013.

[0284] Step 8013: Preliminarily determine whether the observed target is associated based on the speed information and distance information of the observed target;

[0285] The observation targets in this step include the observation targets of one corner radar and the observation targets of another corner radar, that is, at least one of the speed information and distance information of the two corner radar observation targets is used to determine whether the two observation targets are related. The specific process of the determination can refer to the above embodiment. The speed information includes at least one of the longitudinal speed value, the lateral speed value and the longitudinal speed difference value; the distance information includes at least one of the longitudinal distance and the lateral distance. If the two observation targets are determined to be related to each other based on the above conditions, step 8015 is executed, otherwise step 8014 is executed.

[0286] Step 8014: the first radar observation target and the second radar observation target are not related to each other;

[0287] Step 8015: track distance between the first radar observation target and the second radar observation target;

[0288] After preliminarily determining that the first radar observation target and the second radar observation target are related to each other based on the speed information and distance information of the first radar observation target and the second radar observation target in the above step 8013, the trajectory distance between the above two observation targets is calculated using corresponding variables, and the above variables include the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, the additional limited distance between the first radar observation target and the second radar observation target, and at least one of the distance variance cost, the cycle number related cost, the speed cost and the near-point target penalty.

[0289] Step 8016: determine whether the trajectory distance between the first radar observation target and the second radar observation target is less than a threshold;

[0290] After calculating the trajectory distance between the first radar observation target and the second radar observation target in the above step 8015, the trajectory distance between the first radar observation target and the second radar observation target is compared with a preset threshold to determine whether the above trajectory distance is less than the threshold. The above threshold is the threshold value gate. If the above trajectory distance is less than the threshold, execute step 8020, otherwise execute step 8017.

[0291] Step 8017: The first radar observation target and the second radar observation target are not related to each other;

[0292] Step 8018: storing the track distance between the first radar observation target and the second radar observation target into a track distance matrix;

[0293] After it is determined in the above step 8016 that the track distance between the first radar observation target and the second radar observation target is less than a preset threshold, the above track distance is stored in the track distance matrix.

[0294] Step 8019, traverse the next stored observation trajectory data of the observation target collected by the sensor;

[0295] This step is to traverse the next stored observation trajectory data of the observation target collected by the sensor after the above trajectory distance is stored in the trajectory matrix in step 8018.

[0296] Step 8020: Determine the first radar observation target and the second radar observation target that are finally correlated with each other;

[0297] After the traversal of the observation trajectory data of the observation target collected by the sensor in the above step 802 is completed, the nearest neighbor association algorithm can be used to determine that the first radar observation target and the second radar observation target with the smallest trajectory distance are the two observation targets that are finally associated with each other.

[0298] Step 8021, fusing the observation trajectory data of the two observation targets that are finally related to each other;

[0299] This step is to fuse the observation trajectory data of the first radar observation target and the second radar observation target that are finally correlated with each other in step 8020. For example, the observation trajectory data of the two observation targets can be fused using a Kalman filter algorithm.

[0300] Step 8022: Output the first fused trajectory data of the radar observation target.

[0301] In the above step 8021, after the observation trajectory data of the two final mutually related radar observation targets are fused with the help of the Kalman filter algorithm, the fusion result, that is, the first fused trajectory data of the radar observation target, is output.

[0302] In an embodiment of the present application, after the observation trajectory data of the radar observation target collected by the radar is fused to obtain the third trajectory data, the second trajectory data collected by the camera can be fused with the above-mentioned third trajectory data. The above-mentioned second trajectory data includes the observation trajectory data of at least one visual observation target, and the third trajectory data includes the first fused trajectory data of at least one radar observation target. The specific implementation method can be referred to the following embodiment.

[0303] In an embodiment of the present application, it is necessary to first determine the mutually related radar observation targets and visual observation targets based on the above-mentioned second trajectory data and third trajectory data, and then fuse the observation trajectory data of the mutually related radar observation targets and the observation trajectory data of the visual observation targets. This can ensure that the subsequent fusion of the second trajectory data and the third trajectory data is based on related data sources, thereby improving the effectiveness and accuracy of the fusion results.

[0304] Specifically, Fig.13 for Figure 1 The specific execution flow diagram of step 103 of the embodiment shown in FIG. Fig.13 As shown, the following steps are included:

[0305] Step 901: determining mutually associated radar observation targets and visual observation targets based on first fused trajectory data of at least one radar observation target and observation trajectory data of at least one visual observation target;

[0306] This step is based on step 401, and can be Fig.11 In the illustrated embodiment, the first fused trajectory data of the radar observed target is obtained to determine the mutually correlated radar observed target and visual observed target.

[0307] Step 902: Fusing the first fused trajectory data of the mutually related radar observation targets and the observation trajectory data of the visual observation targets to obtain second fused trajectory data of at least one observation target.

[0308] This step is to fuse the first fused trajectory data of the mutually related radar observation targets and the observation trajectory data of the visual observation targets after determining the mutually related radar observation targets and the visual observation targets in step 901, so as to obtain the second fused trajectory data of at least one observation target, wherein the fusion processing can use the Kalman filtering algorithm to fuse the first fused trajectory data of the mutually related radar observation targets and the observation trajectory data of the visual observation targets.

[0309] As described above, the second fused trajectory data of the observed target obtained by this method can better utilize the data collected by the radar and the camera, thereby enhancing the accuracy and anti-interference of the fusion result.

[0310] Specifically, Fig.14 for Fig.13 The specific execution flow diagram of step 901 of the embodiment shown in FIG. Fig.14 As shown, the following steps are included:

[0311] Step 1001: Calculate a trajectory distance between a radar observation target and a visual observation target based on first fused trajectory data of at least one radar observation target and observation trajectory data of at least one visual observation target;

[0312] This step is to calculate the trajectory distance between the radar observation target and the visual observation target using corresponding variables after obtaining the first fused trajectory data of the radar observation target and the observation trajectory data of at least one visual observation target collected by the camera.

[0313] In an embodiment of the present application, the trajectory distance between the radar observation target and the visual observation target can be obtained, and the correlation between the radar observation target and the visual observation target can be determined by the trajectory distance.

[0314] In some embodiments, the above-mentioned trajectory distance can be determined by the Mahalanobis distance between the first fused trajectory data of the radar observed target and the observed trajectory data of the visually observed target, and at least one of the distance variance cost, the cycle number related cost, the speed cost and the near-point target penalty.

[0315] For example, the trajectory distance between the radar observation target and the visual observation target can be calculated by the following formula:

[0316] Dist2=(1+MagnifyCost2)*MDist2+TraSizeCost2+SpeedCost2+NearbyCost2

[0317] Among them, Dist2 is the trajectory distance between the radar observation target and the visual observation target, MDist2 is the Mahalanobis distance between the first fused trajectory data of the radar observation target and the observed trajectory data of the visual observation target, MagnifyCost2 is the distance variance cost of the trajectory distance between the radar observation target and the visual observation target, TraSizeCost2 is the cost related to the number of cycles of the radar observation target and the visual observation target, SpeedCost2 is the speed cost of the radar observation target and the visual observation target, and NearbyCost2 is the near-point target penalty of the radar observation target and the visual observation target.

[0318] Among them, the Mahalanobis distance between the first fusion trajectory data of the radar observation target and the observation trajectory data of the visual observation target is mainly used to measure the similarity between the radar observation target and the visual observation target, which can be set as:

[0319]

[0320] Wherein, MDist2 is the inter-track Mahalanobis distance between the first fused trajectory data of the radar observation target and the observation trajectory data of the visual observation target, μ is the mean of the observation target, ∑ is the covariance matrix of all observation targets, and X is any one of the distance in the x direction, the distance in the y direction, and the speed in the x direction between the radar observation target and the visual observation target.

[0321] Among them, the distance variance cost is mainly used to reflect the stability of the trajectory distance between the radar observation target and the visual observation target during the acquisition cycle. If it is unstable, it means that the radar observation target and the visual observation target are more likely to be not the same observation target, and their correlation is weak. The above distance variance cost can be set as:

[0322]

[0323] Among them, MagnifyCost2 is the distance variance cost of radar observation target and visual observation target, n is the number of acquisition cycles, MDist i is the Mahalanobis distance between the first fused trajectory data of the radar observation target and the observation trajectory data of the visual observation target in each acquisition cycle, It is the average Mahalanobis distance between the first fused trajectory data of the radar observation target and the observation trajectory data of the visual observation target within n acquisition cycles.

[0324] Among them, the cycle number related cost is mainly used to reduce the impact of the inconsistent trajectory data length of the radar observation target and the visual observation target on the trajectory distance between the radar observation target and the visual observation target, which can be set as:

[0325] TraSizeCost2=FrameGainA2+FrameGainB2+FrameGainC2

[0326] Among them, TraSizeCost2 is the cycle number related cost of radar observation target and visual observation target, and FrameGainA2, FrameGainB2, and FrameGainC2 are all correction values ​​of the acquisition cycle.

[0327] In the present application example, it can be set to store at least the observation trajectory data of the radar observation target or the observation trajectory data of the visual observation target within one acquisition cycle, or to store at most the observation trajectory data of the radar observation target or the observation trajectory data of the visual observation target within three acquisition cycles.

[0328] Among them, the speed cost is mainly used to compensate for the defects in calculating the Mahalanobis distance between the first fusion trajectory data of the radar observation target and the observation trajectory data of the visual observation target due to the small position difference between the radar observation target and the visual observation target. Therefore, it is necessary to increase the lateral speed cost and the longitudinal speed cost as an effective compensation for the possible deficiencies in the Mahalanobis distance calculation. The above speed cost can be set as:

[0329]

[0330] Among them, SpeedCost2 is the speed cost of radar observation target and visual observation target, n is the number of acquisition cycles, VecDiffi i For each acquisition cycle, the lateral speed or longitudinal speed of the radar observation target and the visual observation target, It is the average lateral velocity or average longitudinal velocity of the radar observation target and the visual observation target within n acquisition cycles.

[0331] In the above embodiment, if the radar observation target and the visual observation target are within the near-point range specified in the x-direction of the vehicle, the above-mentioned radar observation target and the visual observation target are set as near-point observation targets, and then the speed cost of the radar observation target and the visual observation target is increased to ensure the accuracy of subsequent observation trajectory fusion. For example, if the radar observation target and the visual observation target are within 19m of the x-direction of the vehicle, the above-mentioned radar observation target and the visual observation target are set as near-point observation targets, and the speed cost of the radar observation target and the visual observation target is doubled.

[0332] Among them, the near-point target penalty is mainly applicable to the situation when the near-point observation target is a pedestrian and the above-mentioned speed cost cannot take effect. In the embodiment of the present application, when the near-point observation target is a pedestrian, the near-point target penalty is added, and its value is set to 0.5gate. Gate is the penalty threshold value, and its value is set differently in different sensor combination fusion methods. For example, in the combined fusion method based on the front radar and the corner radar, the gate value is set to 6, and in the combined fusion method based on the radar and the camera, the gate value is set to 12.

[0333] Step 1002: Determine mutually associated radar observation targets and visual observation targets based on the track distance between the radar observation targets and the visual observation targets.

[0334] This step is based on step 1001, and uses the calculated trajectory distance between the radar observation target and the visual observation target to determine the mutually associated radar observation target and visual observation target.

[0335] In some embodiments, if the trajectory distance between the radar observation target and at least two visual observation targets has been calculated, the radar observation target and the visual observation target with the smallest trajectory distance are determined to be associated with each other. In an embodiment of the present application, a nearest neighbor association algorithm can be used to determine the radar observation target and the visual observation target with the smallest trajectory distance.

[0336] In an embodiment of the present application, before determining the mutually associated radar observation targets and visual observation targets based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target, the radar observation targets and visual observation targets that do not meet the mutual association conditions can also be determined based on the speed information and / or distance information of the radar observation targets and the visual observation targets. That is, the radar observation targets and visual observation targets that obviously do not meet the mutual association conditions can be determined in advance. After the radar observation targets and visual observation targets that do not meet the mutual association conditions are determined in advance, the trajectory distance calculation using the above algorithm can be stopped, thereby avoiding unnecessary calculations and saving computing resources.

[0337] In some embodiments, the speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

[0338] In the above embodiment, when determining the radar observation target and the visual observation target that do not meet the mutual correlation condition according to the speed information, it can be judged whether they meet the mutual correlation condition according to the longitudinal speed value, the lateral speed value, and the longitudinal speed difference value of the two observation targets. In some cases, when judging whether the above two observation targets meet the mutual correlation condition according to the longitudinal speed value, the mutual correlation condition is set as follows: one of the observation targets does not have a pseudo-longitudinal motion state, that is, the absolute value of its longitudinal absolute speed does not exceed 2m / s; and the other observation target does not have a pseudo-longitudinal static state, that is, the absolute value of its longitudinal absolute speed is less than 1m / s. For example, if the absolute value of the longitudinal absolute speed of the radar observation target is greater than 2m / s, it is regarded as an observation target with a pseudo-longitudinal motion state, and the absolute value of the longitudinal absolute speed of the visual observation target is less than 1m / s, it is regarded as an observation target with a pseudo-longitudinal static state, and then it is determined that the above two observation targets do not meet the above mutual correlation condition, that is, it indicates that the radar observation target and the visual observation target are not correlated with each other. After that, the speed threshold value is directly assigned to the two observation targets that do not meet the mutual correlation condition. The speed threshold value will vary according to different sensor combination fusion methods. For example, in the combination fusion method based on the front radar and the corner radar, its value is set to 6, and in the combination fusion method based on the radar and the camera, its value is set to 12. For example, if the radar observation target is an observation target in a pseudo-longitudinal motion state, and the visual observation target is an observation target in a pseudo-longitudinal stationary state, the first radar observation target and the visual observation target do not meet the mutual correlation condition. At this time, the speed threshold value of 12 is assigned to the radar observation target and the visual observation target.

[0339] In addition, in some cases, it is also possible to determine whether the radar observation target and the visual observation target meet the mutual correlation condition based on the lateral speed. At this time, the mutual correlation condition is set as follows: the two observation targets do not belong to the opposite observation targets, and the above-mentioned opposite observation targets refer to the lateral speed of one observation target being greater than 2m / s, and the lateral speed of the other observation target being less than -2m / s. For example, if the lateral speed of the radar observation target is greater than 2m / s and the lateral speed of the visual observation target is less than -2m / s, then the above-mentioned two observation targets are regarded as two opposite observation targets, thereby not meeting the above-mentioned mutual correlation condition, that is, indicating that the radar observation target and the visual observation target are not correlated with each other. Subsequently, the speed threshold value is directly assigned to the two observation targets that do not meet the mutual correlation condition.

[0340] Alternatively, in other cases, when judging whether the radar observation target and the visual observation target meet the mutual correlation condition based on the longitudinal speed difference, the mutual correlation condition is set as: the longitudinal speed difference of the two observation targets does not exceed 12m / s. For example, if the longitudinal speed difference between the radar observation target and the visual observation target exceeds 12m / s, it means that the above two observation targets do not meet the mutual correlation condition, that is, it indicates that the radar observation target and the visual observation target are not correlated with each other, and then the speed threshold value is directly assigned to the above two observation targets that do not meet the mutual correlation condition.

[0341] When determining that the radar observation target and the visual observation target do not satisfy the mutual correlation according to the distance information, it is possible to judge whether they satisfy the mutual correlation condition according to the longitudinal distance and the lateral distance of the two observation targets. At this time, the mutual correlation condition is set as: the longitudinal distance of the two observation targets does not exceed 12m or the lateral distance does not exceed 2m. For example, if the longitudinal distance between the radar observation target and the visual observation target exceeds 12m or the lateral distance exceeds 2m, it is judged that the above two observation targets do not satisfy the above mutual correlation condition, that is, it indicates that the radar observation target and the visual observation target are not mutually correlated. Subsequently, the distance threshold value is directly assigned to the two observation targets that do not satisfy the mutual correlation condition. The above distance threshold value will vary according to different sensor combination fusion methods. For example, in the method based on the combination fusion of the front radar and the corner radar, its value is set to 6, and in the method based on the combination fusion of the radar and the camera, its value is set to 12.

[0342] In the above-mentioned various situations, the above-mentioned speed information and distance information are used to determine the first radar observation target and the second radar observation target that do not meet the mutual correlation condition. One or several situations can be selected for judgment, and when multiple situations are selected, the execution order can be adjusted to determine the radar observation targets and visual observation targets that do not meet the mutual correlation condition. Ultimately, the radar observation targets and visual observation targets that are determined to not meet the mutual correlation condition do not participate in the subsequent trajectory distance calculation, thereby reducing the amount of calculation.

[0343] In the above embodiment, the radar observation target may include at least one of the observation targets of each corner radar and the observation target of the front radar. Fig.15 In the embodiment shown, the observation trajectory data of the visual observation target collected by the camera is input and Fig.12 In the embodiment shown, the fusion of the first fused trajectory data of the radar observation target is taken as an example to illustrate the fusion of the first fused trajectory data of the radar observation target and the observation trajectory data of the visual observation target collected by the camera. Fig.15 This is a flow chart of a specific embodiment of the present application. Fig.15 As shown, the following steps are included:

[0344] Step 1101, inputting observation trajectory data of a visual observation target collected by a camera;

[0345] The observation trajectory data of the visual observation target collected by the camera includes observation trajectory data of at least one visual observation target collected by the camera.

[0346] Step 1102, determining whether the observation trajectory data of the visual observation target collected by the camera has been traversed to completion;

[0347] On the basis of the above step 1101, it is judged whether the observation trajectory data of the visual observation target collected by the camera has been traversed. If the observation trajectory data of the above visual observation target has been traversed, step 1122 is executed, otherwise step 1103 is executed.

[0348] Step 1103, storing the observation trajectory data of the visual observation target collected by the camera;

[0349] After determining in the above step 1102 that the observation trajectory data of the visual observation target collected by the camera has not yet been traversed, the observation trajectory data of the above visual observation target that has been traversed is stored.

[0350] Step 1104, storing the first fused trajectory data of the radar observation target into the observation trajectory data of the observation target collected by the stored camera;

[0351] This step is in Fig.12 In the illustrated embodiment, after the first fused trajectory data of the radar observation target is obtained, the first fused trajectory data is stored in the observation trajectory data of the observation target collected by the camera.

[0352] Step 1105, determining whether the traversal of the observation trajectory data of the observation target collected by the stored sensor is completed;

[0353] This step is to judge whether the traversal of the observation trajectory data of the visual observation target collected by the camera in the above embodiment is completed after the traversal of the observation trajectory data of the observation target collected by the stored sensor is completed. If the traversal of the observation trajectory of the above visual observation target is completed, execute step 1106, otherwise execute step 1107.

[0354] Step 1106, traverse the observation trajectory data of the visual observation target collected by the next camera;

[0355] After it is determined in the above step 1105 that the stored observation trajectory data of the observation target collected by the sensor has been traversed, the next observation trajectory data of the visual observation target collected by the camera is traversed.

[0356] Step 1107, determining whether the stored observation trajectory data of the observation target collected by the sensor is the observation trajectory data of the radar observation target collected by the radar;

[0357] After it is determined in the above step 1105 that the stored observation trajectory data of the observation target collected by the sensor has not been traversed, it is determined whether the stored observation trajectory data of the observation target collected by the sensor is the observation trajectory data of the radar observation target collected by the radar. If the above stored observation trajectory data of the observation target collected by the sensor is the observation trajectory data of the radar observation target, execute step 1108, otherwise execute step 1113.

[0358] Step 1108: Preliminarily determine whether the observed target is associated based on the speed information and distance information of the observed target;

[0359] The observation targets in this step include radar observation targets and visual observation targets, that is, at least one of the speed information and distance information of the radar observation target and the visual observation target is used to determine whether the two observation targets are associated. The specific process of the determination can refer to the above embodiment. The speed information includes at least one of the longitudinal speed value, the lateral speed value and the longitudinal speed difference value; the distance information includes at least one of the longitudinal distance and the lateral distance. If the two observation targets are determined to be associated with each other based on the above conditions, step 1110 is executed, otherwise step 1109 is executed.

[0360] Step 1109: The radar observation target and the visual observation target are not related to each other;

[0361] Step 1110, calculating the trajectory distance between the radar observation target and the visual observation target;

[0362] In the above step 1108, after preliminarily determining that the above observation targets are related to each other based on the speed information and distance information of the radar observation target and the visual observation target, the trajectory distance between the radar observation target and the visual observation target is calculated using corresponding variables, and the above variables include the Mahalanobis distance between the first fused trajectory data of the radar observation target and the observation trajectory data of the visual observation target, and at least one of the distance variance cost, the cycle number related cost, the speed cost and the near-point target penalty.

[0363] Step 1111, determining whether the trajectory distance between the radar observation target and the visual observation target is less than a threshold;

[0364] This step is to compare the trajectory distance between the radar observation target and the visual observation target with a preset threshold after calculating the trajectory distance between the radar observation target and the visual observation target in step 1110 to determine whether the above trajectory distance is less than the threshold. The above threshold is the threshold value gate. If the above trajectory distance is less than the threshold, execute step 1113, otherwise execute step 1112.

[0365] Step 1112: The radar observation target and the visual observation target are not related to each other;

[0366] Step 1113, storing the trajectory distance between the radar observation target and the visual observation target in a trajectory distance matrix;

[0367] After determining in the above step 1111 that the track distance between the radar observation target and the visual observation target is less than a preset threshold, the above track distance is stored in a track distance matrix.

[0368] Step 1114, traverse the next stored observation trajectory data of the observation target collected by the sensor;

[0369] This step is to traverse the next stored observation trajectory data of the observation target collected by the sensor after the above trajectory distance is stored in the trajectory matrix in step 1113.

[0370] Step 1115, determining the radar observation target and the visual observation target that are finally correlated with each other;

[0371] After the traversal of the observation trajectory data of the observation target collected by the sensor is determined to be stored in the above step 1102, the nearest neighbor association algorithm can be used to determine that the radar observation target and the visual observation target with the smallest trajectory distance are the two observation targets that are finally associated with each other.

[0372] Step 1116, fusing the observation trajectory data of the two observation targets that are finally related to each other;

[0373] This step is to fuse the observation trajectory data of the two observation targets after the final mutually related radar observation targets and visual observation targets are determined in step 1115. For example, the observation trajectory data of the two observation targets can be fused using the Kalman filter algorithm.

[0374] Step 1117: output the second fused trajectory data of the observed target.

[0375] In the above step 1116, after the observation trajectory data of the final mutually related radar observation target and the visual observation target are fused with the help of the Kalman filter algorithm, the fusion result, that is, the second fused trajectory data of the observation target, is output.

[0376] In the above embodiment, if the observation trajectory data of the radar observation target collected by the radar is invalid due to bad weather, equipment interference, etc., and cannot be fused with the observation trajectory data of the observation target collected by the camera, the observation trajectory data of the visual observation target collected by the camera at different stages can be fused. The specific implementation process is as follows: Fig.16 As shown, the following steps are included:

[0377] Step 1201: determining a second visual observation target and a third visual observation target that are associated with each other based on observation trajectory data of at least one second visual observation target and observation trajectory data of at least one third visual observation target;

[0378] This step is to obtain the observation trajectory data of at least one second visual observation target collected by the camera in the current stage and the observation trajectory data of at least one third visual observation target collected by the camera in the historical stage in step 101, and then use corresponding calculation processing to determine the mutually related second visual observation targets and third visual observation targets, wherein the stage includes multiple collection cycles.

[0379] Step 1202: Fusing the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target that are related to each other to obtain third fused trajectory data of at least one visual observation target.

[0380] In this step, after the mutually related second visual observation target and the third visual observation target are determined in step 1201, the observation trajectory data of the mutually related second visual observation target and the observation trajectory data of the third visual observation target are fused to obtain third fused trajectory data of at least one visual observation target. For example, the Kalman filtering algorithm can be used to fuse the observation trajectory data of the mutually related second visual observation target and the observation trajectory data of the third visual observation target.

[0381] As described above, the third fused trajectory data of the visual observation target obtained by this method can better utilize the data collected by the camera, and at the same time enable the vehicle system to operate normally with or without the participation of the radar, making it adjustable, so that the vehicle can flexibly adjust the working mode according to different usage scenarios and needs, and cope with various complex and changeable road environments in the best state.

[0382] Specifically, Fig.17 for Fig.16 The specific execution flow diagram of step 1201 of the embodiment shown in FIG. Fig.17 As shown, the following steps are included:

[0383] Step 1301: Calculate a trajectory distance between the second visual observation target and the third visual observation target based on observation trajectory data of at least one second visual observation target and observation trajectory data of at least one third visual observation target;

[0384] In this step, after the camera collects observation trajectory data of at least one second visual observation target in the current stage, and the camera collects observation trajectory data of at least one third visual observation target in the historical stage, the corresponding variables are used to calculate the trajectory distance between the second visual observation target and the third visual observation target.

[0385] In an embodiment of the present application, the trajectory distance between the second visual observation target and the third visual observation target may be obtained, and the correlation between the second visual observation target and the third visual observation target may be determined through the trajectory distance.

[0386] In some embodiments, the above-mentioned trajectory distance can be determined by the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, and at least one of the distance variance cost, the cycle number related cost, the speed cost and the near-point target penalty.

[0387] For example, the trajectory distance between the second visual observation target and the third visual observation target can be calculated by the following formula:

[0388] Dist3=(1+MagnifyCost3)*MDist3+TraSizeCost3+SpeedCost3+NearbyCost3

[0389] Among them, Dist3 is the trajectory distance between the second visual observation target and the third visual observation target, MDist3 is the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, MagnifyCost3 is the distance variance cost of the trajectory distance between the second visual observation target and the third visual observation target, TraSizeCost3 is the cost related to the number of cycles of the second visual observation target and the third visual observation target, SpeedCost3 is the speed cost of the second visual observation target and the third visual observation target, and NearbyCost3 is the near-point target penalty of the second visual observation target and the third visual observation target.

[0390] Among them, the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target is mainly used to measure the similarity between the second visual observation target and the third visual observation target, which can be set as:

[0391]

[0392] Among them, MDist3 is the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, μ is the mean of the observation targets, ∑ is the covariance matrix of all observation targets, and X is any one of the distance in the x direction, the distance in the y direction, and the speed in the x direction between the radar observation target and the visual observation target.

[0393] Among them, the distance variance cost is mainly used to reflect the stability of the trajectory distance between the second visual observation target and the third visual observation target during the acquisition cycle. If it is unstable, it means that the second visual observation target and the third visual observation target are more likely to be not the same observation target, and their correlation is weak. The above distance variance cost can be set as:

[0394]

[0395] Among them, MagnifyCost3 is the distance variance cost between the second visual observation target and the third visual observation target, n is the number of acquisition cycles, and MDist i is the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target in each acquisition cycle, is the average Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target within n acquisition cycles.

[0396] Among them, the cycle number related cost is mainly used to reduce the impact of the inconsistent trajectory data length of the second visual observation target and the third visual observation target on the trajectory distance between the second visual observation target and the third visual observation target, which can be set as:

[0397] TraSizeCost3=FrameGainA3+FrameGainB3+FrameGainC3

[0398] Among them, TraSizeCost3 is the cycle number related cost of the second visual observation target and the third visual observation target, and FrameGainA3, FrameGainB3, and FrameGainC3 are all correction values ​​of the acquisition cycle.

[0399] In this application example, it can be set to store at least the observation trajectory data of the second visual observation target or the observation trajectory data of the third visual observation target within one acquisition cycle, or to store at most the observation trajectory data of the second visual observation target or the observation trajectory data of the third visual observation target within three acquisition cycles.

[0400] Among them, the speed cost is mainly used to compensate for the defects in calculating the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target due to the small position difference between them. Therefore, it is necessary to increase the horizontal speed cost and the longitudinal speed cost as an effective compensation for the possible deficiencies in the Mahalanobis distance calculation. The above speed cost can be set as:

[0401]

[0402] Among them, SpeedCost3 is the speed cost of the second visual observation target and the third visual observation target, n is the number of acquisition cycles, VecDiff i is the lateral speed or longitudinal speed of the second visual observation target and the third visual observation target in each acquisition cycle, is the average lateral velocity or average longitudinal velocity of the second visual observation target and the third visual observation target within n acquisition cycles.

[0403] In the above embodiment, if the second visual observation target and the third visual observation target are within the near-point range specified in the x-direction of the vehicle, the second visual observation target and the third visual observation target are set as near-point observation targets, and then the speed costs of the second visual observation target and the third visual observation target are increased to ensure the accuracy of subsequent observation trajectory fusion. For example, if the second visual observation target and the third visual observation target are within 19m of the vehicle in the x-direction, the second visual observation target and the third visual observation target are set as near-point observation targets, and the speed costs of the second visual observation target and the third visual observation target are doubled.

[0404] Among them, the near-point target penalty is mainly applicable to the situation when the near-point observation target is a pedestrian and the above-mentioned speed cost cannot take effect. In the embodiment of the present application, when the near-point observation target is a pedestrian, the near-point target penalty is added, and its value is set to 0.5gate. Gate is the penalty threshold value, and its value is set differently in different sensor combination fusion methods. For example, in the combined fusion method based on the front radar and the corner radar, the gate value is set to 6, and in the combined fusion method based on the radar and the camera, the gate value is set to 12.

[0405] Step 1302: Determine the second visual observation target and the third visual observation target that are associated with each other based on the trajectory distance between the second visual observation target and the third visual observation target.

[0406] This step is based on step 1301 and uses the calculated trajectory distance between the second visual observation target and the third visual observation target to determine the second visual observation target and the third visual observation target that are associated with each other.

[0407] In some embodiments, if the trajectory distance between the second visual observation target and the third visual observation target has been calculated, the second visual observation target and the third visual observation target with the smallest trajectory distance are determined to be associated with each other. In this embodiment of the present application, the nearest neighbor association algorithm can be used to determine the second visual observation target and the third visual observation target with the smallest trajectory distance.

[0408] In an embodiment of the present application, before determining the second visual observation target and the third visual observation target that are mutually associated based on the trajectory distance between the second visual observation target and the third visual observation target, the second visual observation target and the third visual observation target that do not meet the mutual association condition can also be determined based on the speed information and / or distance information of the second visual observation target and the third visual observation target. That is, the second visual observation target and the third visual observation target that obviously do not meet the mutual association condition can be determined in advance. After the two visual observation targets that do not meet the mutual association condition are determined in advance, the trajectory distance calculation using the above algorithm can be no longer performed on them, thereby avoiding unnecessary calculations and saving computing resources.

[0409] The speed information in the above embodiment includes at least one of a longitudinal speed value, a lateral speed value and a longitudinal speed difference value; the distance information includes at least one of a longitudinal distance and a lateral distance.

[0410] In some embodiments, before determining the second visual observation target and the third visual observation target that are mutually related based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target, it is also possible to judge whether the two visual observation targets are mutually related based on the target identifiers of the above two visual observation targets, that is, if the above second visual observation target and the third visual observation target have the same observation target identifier, then it is determined that the second visual observation target and the third visual observation target with the same observation target identifier are mutually related. The target identifier of the observation target is the code of the observation target, that is, the ID of the observation target, which is unique. By using the uniqueness of the target identifier of the observation target, it is possible to determine in advance whether the second visual observation target and the third visual observation target are mutually related, and then it is no longer necessary to calculate the trajectory distance for the observation trajectory data of the above two visual observation targets, thereby avoiding unnecessary calculations and saving computing resources.

[0411] In the above embodiment, when determining the second visual observation target and the third visual observation target that do not meet the mutual correlation condition according to the speed information, it is possible to judge whether they meet the mutual correlation condition according to the longitudinal speed value, the lateral speed value, and the longitudinal speed difference value of the two observation targets. In some cases, when judging whether the above two observation targets meet the mutual correlation condition according to the longitudinal speed value, the mutual correlation condition is set as follows: one of the observation targets does not have a pseudo longitudinal motion state, that is, the absolute value of its longitudinal absolute speed does not exceed 2m / s; and the other observation target does not have a pseudo longitudinal static state, that is, the absolute value of its longitudinal absolute speed is less than 1m / s. For example, if the absolute value of the longitudinal absolute speed of the second visual observation target is greater than 2m / s, it is regarded as an observation target with a pseudo longitudinal motion state, and the absolute value of the longitudinal absolute speed of the third visual observation target is less than 1m / s, it is regarded as an observation target with a pseudo longitudinal static state, and then it is determined that the above two observation targets do not meet the above mutual correlation condition, that is, it indicates that the radar observation target and the visual observation target, the second visual observation target and the third visual observation target are not correlated with each other. Afterwards, the speed threshold value is directly assigned to the two observation targets that do not meet the mutual correlation condition. The speed threshold value will vary according to different sensor combination fusion methods. For example, in the combination fusion method based on the front radar and the corner radar, its value is set to 6, and in the combination fusion method based on the radar and the camera, its value is set to 12.

[0412] In addition, in some cases, it is also possible to determine whether the second visual observation target and the third visual observation target meet the mutual correlation condition based on the lateral speed. At this time, the mutual correlation condition is set as follows: the two observation targets do not belong to opposite observation targets, and the above-mentioned opposite observation targets refer to the lateral speed of one observation target being greater than 2m / s, and the lateral speed of the other observation target being less than -2m / s. For example, if the lateral speed of the second visual observation target is greater than 2m / s, and the lateral speed of the third visual observation target is less than -2m / s, then the above-mentioned two observation targets are regarded as two opposite observation targets, and thus do not meet the above-mentioned mutual correlation condition, which means that the second visual observation target and the third visual observation target are not mutually correlated. Subsequently, the speed threshold value is directly assigned to the two observation targets that do not meet the mutual correlation condition.

[0413] Alternatively, in some other cases, when judging whether the second visual observation target and the third visual observation target meet the mutual correlation condition based on the longitudinal speed difference, the mutual correlation condition is set as: the longitudinal speed difference between the two observation targets does not exceed 12m / s. For example, if the longitudinal speed difference between the second visual observation target and the third visual observation target exceeds 12m / s, it means that the two observation targets do not meet the mutual correlation condition, that is, the second visual observation target and the third visual observation target are not related to each other, and then the speed threshold value is directly assigned to the two observation targets that do not meet the mutual correlation condition.

[0414] When determining the second visual observation target and the third visual observation target that do not satisfy the mutual correlation based on the distance information, it is possible to judge whether they satisfy the mutual correlation condition based on the longitudinal distance and lateral distance of the two observation targets. At this time, the mutual correlation condition is set as: the longitudinal distance of the two observation targets does not exceed 12m or the lateral distance does not exceed 2m. For example, if the longitudinal distance between the second visual observation target and the third visual observation target exceeds 12m or the lateral distance exceeds 2m, it is judged that the above two observation targets do not satisfy the above mutual correlation condition, which indicates that the second visual observation target and the third visual observation target are not mutually correlated. Subsequently, the distance threshold value is directly assigned to the two observation targets that do not satisfy the mutual correlation condition. The above distance threshold value will vary according to different sensor combination fusion methods. For example, in the combination fusion method based on the front radar and the corner radar, its value is set to 6, and in the combination fusion method based on the radar and the camera, its value is set to 12.

[0415] In addition, in some other cases, if the second visual observation target and the third visual observation target have the same target identifier, it is determined that the second visual observation target and the third visual observation target are associated with each other, and a target identifier threshold value is assigned to their direct target identifiers. The target identifier threshold value is generally set to 0. Otherwise, it is determined that the second visual observation target and the third visual observation target do not meet the mutual association condition, that is, the second visual observation target and the third visual observation target are not associated with each other. The above mutual association condition is that the target identifiers of the two observation targets are the same.

[0416] In the above-mentioned various situations, the second visual observation target and the third visual observation target that do not meet the mutual correlation condition are determined by utilizing the above-mentioned speed information, distance information and target identification. One or several situations can be selected for judgment, and when multiple situations are selected, the execution order can be adjusted to determine the second visual observation target and the third visual observation target that do not meet the mutual correlation condition. Ultimately, the second visual observation target and the third visual observation target that do not meet the mutual correlation condition do not participate in the subsequent trajectory distance calculation, thereby reducing the amount of calculation.

[0417] In the above embodiment, the observation trajectory data of the second visual observation target collected by the camera in the current stage can be input and fused with the observation trajectory data of the third visual observation target collected by the camera in the historical stage as an example to illustrate the fusion of the observation trajectory data of the visual observation targets collected by the camera at different stages. Fig.18 This is a flow chart of a specific embodiment of the present application. Fig.18 As shown, the following steps are included:

[0418] Step 1401, inputting observation trajectory data of a second visual observation target collected by the camera at the current stage;

[0419] The observation trajectory data of the visual observation target collected by the camera at the current stage includes the observation trajectory data of at least one second visual observation target collected by the camera at the current stage.

[0420] Step 1402, determining whether the observation trajectory data of the first visual observation target collected by the camera at the current stage has been traversed;

[0421] On the basis of the above step 1401, it is judged whether the observation trajectory data of the second visual observation target collected by the camera at the current stage has been traversed. If the observation trajectory data of the second visual observation target has been traversed, step 1414 is executed, otherwise step 1403 is executed.

[0422] Step 1403, storing the observation trajectory data of the second visual observation target collected by the camera at the current stage;

[0423] After determining in the above step 1402 that the observation trajectory data of the second visual observation target collected by the camera in the current stage has not been traversed, the observation trajectory data of the above second visual observation target that has been traversed is stored.

[0424] Step 1404, determining whether the traversal of the observation trajectory data of the observation target collected by the stored sensor is completed;

[0425] This step is to judge whether the traversal of the observation trajectory data of the observation target collected by the stored sensor is completed after the observation trajectory data of the second visual observation target collected by the camera at the current stage in the above embodiment is stored. If the traversal of the observation trajectory data of the above observation target is completed, execute step 1405, otherwise execute step 1406.

[0426] Step 1405, traverse the next observation trajectory data of the second visual observation target collected by the camera at the current stage;

[0427] After it is determined in the above step 1404 that the observation trajectory data of the observation target collected by the stored sensor has been traversed, the next observation trajectory data of the second visual observation target collected by the camera at the current stage is traversed.

[0428] Step 1406: determine whether the stored observation trajectory data of the observation target collected by the sensor is the observation trajectory data of the third observation target collected by the camera in the historical stage;

[0429] After it is determined in the above step 1404 that the observation trajectory data of the observation target collected by the stored sensor has not been traversed, it is determined whether the observation trajectory data of the observation target collected by the stored sensor is the observation trajectory data of the third observation target collected by the camera in the historical stage. If the observation trajectory data of the above observation target is the observation trajectory data of the third observation target, execute step 1407.

[0430] Step 1407: Preliminarily determine whether the observed targets are related to each other based on the speed information, distance information and target identification;

[0431] After determining in the above step 1406 that the observation trajectory data of the observation target collected by the stored sensor is the observation trajectory data of the third observation target collected by the camera in the historical stage, based on at least one of the speed information, distance information and target identification of the second visual observation target and the third visual observation target of the camera, it is preliminarily determined whether the above two observation targets are related to each other, wherein the speed information includes at least one of the longitudinal speed value, the lateral speed value and the longitudinal speed difference; the distance information includes at least one of the longitudinal distance and the lateral distance. If it is preliminarily determined that the observation targets are related to each other based on the speed information, the distance information and the target identification, step 1409 is executed, otherwise step 1408 is executed.

[0432] Step 1408: the second visual observation target and the third visual observation target are not associated with each other;

[0433] Step 1409: Calculate the trajectory distance between the second visual observation target and the third visual observation target;

[0434] In the above step 1407, after preliminarily judging that the above observation targets are related to each other based on the speed information, distance information and target identification of the second visual observation target and the third visual observation target, the trajectory distance between the second visual observation target and the third visual observation target is calculated using corresponding variables, and the above variables include the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, and at least one of the distance variance cost, the cycle number related cost, the speed cost and the near-point target penalty.

[0435] Step 1410, determining whether the track break distance between the second visual observation target and the third visual observation target is less than a threshold;

[0436] This step is to compare the trajectory distance between the second visual observation target and the third visual observation target with a preset threshold after calculating it in step 1409 to determine whether the above trajectory distance is less than the threshold. The above threshold is the threshold value gate. If the above trajectory distance is less than the threshold, execute step 1412, otherwise execute step 1411.

[0437] Step 1411: the second visual observation target and the third visual observation target are not associated with each other;

[0438] Step 1412, storing the track distance between the second visual observation target and the third visual observation target into a track distance matrix;

[0439] After determining in the above step 1410 that the trajectory distance between the second visual observation target and the third visual observation target is less than a preset threshold, the above trajectory distance is stored in a trajectory distance matrix.

[0440] Step 1413, traverse the next stored observation trajectory data of the observation target collected by the sensor in the historical stage;

[0441] This step is to traverse the next stored observation trajectory data of the observation target collected by the sensor after the above trajectory distance is stored in the trajectory matrix in step 1412.

[0442] Step 1414: determining the second visual observation target and the third visual observation target that are finally associated with each other;

[0443] After the traversal of the observation trajectory data of the observation target collected by the sensor in the above step 1402 is completed, the nearest neighbor association algorithm can be used to determine that the second visual observation target and the third visual observation target with the smallest trajectory distance are the two observation targets that are finally associated with each other.

[0444] Step 1415, fusing the observation trajectory data of the two observation targets that are finally related to each other;

[0445] This step is to fuse the observation trajectory data of the second visual observation target and the third visual observation target that are finally correlated with each other in step 1414. For example, the observation trajectory data of the two observation targets can be fused using a Kalman filter algorithm.

[0446] Step 1416: output the third fused trajectory data of the visual observation target.

[0447] In the above step 1415, after the observation trajectory data of the final mutually related second visual observation target and the third visual observation target are fused with the help of the Kalman filter algorithm, the fusion result, that is, the third fused trajectory data of the visual observation target, is output.

[0448] With the above Figure 1-Figure 18 Corresponding to any provided trajectory data fusion method, the embodiments of the present application provide a corresponding device. Fig.19 This is a schematic diagram of the structure of a trajectory data fusion device in an example of this application, such as Fig.19 As shown, the trajectory data fusion device includes a data acquisition module 11, a first fusion processing module 12, and a second fusion processing module 13, wherein the data acquisition module 11 is used to acquire first trajectory data respectively collected by at least two radars arranged on the vehicle, and acquire second trajectory data collected by a camera arranged on the vehicle; the first fusion processing module 12 is used to perform fusion processing on the first trajectory data respectively collected by the at least two radars to obtain third trajectory data after fusion processing, and the third trajectory data includes first fused trajectory data of at least one radar observation target; the second fusion processing module 13 is used to perform fusion processing on the second trajectory data and the third trajectory data to obtain fourth trajectory data after fusion processing, and the fourth trajectory data includes second fused trajectory data of at least one observation target.

[0449] In some embodiments, the at least two radars provided on the vehicle include at least two of a front radar, a first corner radar located on a first side of the front radar, and a second corner radar located on a second side of the front radar.

[0450] In some embodiments, the first trajectory data includes observation trajectory data of at least one radar observation target, and the obtaining of the first trajectory data respectively collected by at least two radars provided on the vehicle includes:

[0451] Acquire observation trajectory data of at least one radar observation target collected by at least two radars in a current collection cycle;

[0452] Based on the observation trajectory data of at least one radar observation target collected in a current collection cycle, the historical observation trajectory data of the radar observation target obtained by at least two radars in a historical collection cycle are updated to obtain first trajectory data.

[0453] In some embodiments, the second trajectory data includes observation trajectory data of at least one visual observation target, and the second trajectory data collected by a camera disposed on the vehicle is obtained, including:

[0454] Obtaining observation trajectory data of at least one visual observation target collected by the camera in a current collection cycle;

[0455] Based on the observation trajectory data of at least one visual observation target collected in the current collection cycle, the historical trajectory data of the visual observation target obtained by the camera in the historical collection cycle is updated to obtain second trajectory data.

[0456] In some embodiments, after obtaining the observation trajectory data of at least one radar observation target collected by at least two radars in the current collection cycle, the method further includes:

[0457] Based on a preset trajectory screening condition, the observation trajectory data of at least one radar observation target collected by at least two radars in a current collection cycle are screened to retain the observation trajectory data of the radar observation target that meets the preset trajectory screening condition; and / or

[0458] After obtaining the observation trajectory data of at least one visual observation target collected by the camera in the current collection cycle, the method further includes:

[0459] The observation trajectory data of at least one visual observation target collected by the camera in a current collection cycle is filtered based on the preset trajectory filtering condition to retain the observation trajectory data of the visual observation target that meets the preset trajectory filtering condition.

[0460] In some embodiments, the preset trajectory screening condition includes: at least one of a relative position threshold, a relative speed threshold, a target confidence threshold, a target type, a target identifier, and a validity identifier.

[0461] In some embodiments, after filtering the observation trajectory data of at least one radar observation target collected by at least two radars in the current collection cycle based on the preset trajectory screening condition, the method further includes:

[0462] performing coordinate system conversion on observation trajectory data of at least one radar observation target; and / or,

[0463] After filtering the observation trajectory data of at least one visual observation target collected by the camera in the current collection cycle based on the preset trajectory screening condition, the method further includes:

[0464] The observation trajectory data of at least one visual observation target is transformed into a coordinate system.

[0465] In some embodiments, the observation trajectory data includes position data and speed data, and the coordinate system conversion of the observation trajectory data of at least one radar observation target includes:

[0466] converting the position data of at least one radar observation target into a vehicle rear axle coordinate system;

[0467] converting velocity data of at least one radar observed target into an absolute coordinate system; and / or,

[0468] The coordinate system conversion of the observation trajectory data of at least one visual observation target includes:

[0469] The position data of at least one visual observation target is converted to a vehicle rear axle coordinate system.

[0470] In some embodiments, it also includes:

[0471] If the observation trajectory data of the first radar observation target is not collected in the current collection period, the observation trajectory data of the first radar observation target in the current collection period is determined based on the historical observation trajectory data of the first radar observation target in the previous two collection periods; and / or,

[0472] If the observation trajectory data of at least one visual observation target is not collected in the current collection cycle, the observation trajectory data of at least one visual observation target in the current collection cycle is determined based on the historical observation trajectory data of at least one visual observation target in the previous two collection cycles.

[0473] In some embodiments, it also includes:

[0474] If the observation trajectory data of the first radar observation target is not collected for N consecutive collection cycles, the historical observation trajectory data of the first radar observation target is deleted; and / or,

[0475] If the observation trajectory data of at least one visual observation target is not collected for N consecutive collection cycles, the historical observation trajectory data of at least one visual observation target is deleted;

[0476] Wherein, N is a positive integer greater than or equal to 2.

[0477] In some embodiments, the at least two radars include a first radar and a second radar, and before fusing the first trajectory data respectively collected by the at least two radars, the method further includes:

[0478] Determine the first radar observation target and the second radar observation target that are associated with each other based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar;

[0479] The above-mentioned fusing processing of the first trajectory data respectively collected by at least two radars includes:

[0480] The observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target that are mutually related are fused to obtain first fused trajectory data of at least one radar observation target.

[0481] In some embodiments, the determining of the mutually associated first radar observation target and second radar observation target based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar includes:

[0482] Calculating a trajectory distance between the first radar observation target and the second radar observation target based on the observation trajectory data of at least one first radar observation target and the observation trajectory data of at least one second radar observation target;

[0483] Based on the track distance between the first radar observation target and the second radar observation target, the first radar observation target and the second radar observation target associated with each other are determined.

[0484] In some embodiments, the above-mentioned calculating the trajectory distance between the first radar observation target and the second radar observation target based on the observation trajectory data of at least one first radar observation target and the observation trajectory data of at least one second radar observation target includes:

[0485] The trajectory distance between the first radar observation target and the second radar observation target is determined based on the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

[0486] In some embodiments, the first radar is a first corner radar, and the second radar is a second corner radar. Based on the Mahalanobis distance between the observed trajectory data of the target observed by the first radar and the observed trajectory data of the target observed by the second radar, and at least one of the distance variance cost, the cycle number related cost, the speed cost, and the near-point target penalty, determining the trajectory distance between the target observed by the first radar and the target observed by the second radar comprises:

[0487] determining a track distance between the target observed by the first radar and the target observed by the second radar based on a Mahalanobis distance between the observed track data of the target observed by the first radar and the observed track data of the target observed by the second radar, an additional defined distance between the first corner radar and the second corner radar, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty,

[0488] In some embodiments, the determining of the trajectory distance between the first radar observed target and the second radar observed target based on the Mahalanobis distance between the observed trajectory data of the first radar observed target and the observed trajectory data of the second radar observed target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty, includes:

[0489] Dist1=(1+MagnifyCost1)*MDist1+TraSizeCost1+SpeedCost1+NearbyCost1

[0490] Among them, Dist1 is the trajectory distance between the first radar observation target and the second radar observation target, MDist1 is the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, MagnifyCost1 is the distance variance cost of the trajectory distance between the first radar observation target and the second radar observation target, TraSizeCosr1 is the cost related to the number of cycles of the first radar observation target and the second radar observation target, SpeedCost1 is the speed cost of the first radar observation target and the second radar observation target, and NearbyCost1 is the near-point target penalty of the first radar observation target and the second radar observation target.

[0491] In some embodiments, the determining of the first radar observation target and the second radar observation target associated with each other based on the track distance between the first radar observation target and the second radar observation target includes:

[0492] If the track distances between the first radar observation target and at least two second radar observation targets are calculated, it is determined that the first radar observation target and the second radar observation target with the smallest track distance are associated with each other.

[0493] In some embodiments, before determining the mutually associated first radar observation target and second radar observation target based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar, the method further includes:

[0494] Determine the first radar observation target and the second radar observation target that are not associated with each other based on the speed information and / or the distance information of the first radar observation target and the second radar observation target;

[0495] In some embodiments, the speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

[0496] In some embodiments, before the fusion processing of the second trajectory data and the third trajectory data is performed, the method further includes:

[0497] Determining mutually associated radar observation targets and visual observation targets based on first fused trajectory data of at least one radar observation target and observation trajectory data of at least one visual observation target;

[0498] The above-mentioned fusing processing of the second trajectory data and the third trajectory data includes: fusing the first fused trajectory data of the mutually related radar observation target and the observation trajectory data of the visual observation target to obtain the second fused trajectory data of at least one observation target.

[0499] In some embodiments, the determining of the mutually associated radar observation targets and visual observation targets based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target includes:

[0500] Calculating a trajectory distance between the radar observation target and the visual observation target based on first fused trajectory data of at least one radar observation target and observation trajectory data of at least one visual observation target;

[0501] Based on the track distance between the radar observation target and the visual observation target, the radar observation target and the visual observation target that are associated with each other are determined.

[0502] In some embodiments, the calculating of the trajectory distance between the radar observation target and the visual observation target based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target includes:

[0503] The trajectory distance between the radar observed target and the visually observed target is determined based on the Mahalanobis distance between the first fused trajectory data of the radar observed target and the observed trajectory data of the visually observed target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

[0504] In some embodiments, the determining of the trajectory distance between the radar-observed target and the visually-observed target based on the Mahalanobis distance between the first fused trajectory data of the radar-observed target and the observed trajectory data of the visually-observed target, and at least one of a distance variance cost, a cycle number-related cost, a speed cost, and a near-point target penalty, includes:

[0505] Dist2=(1+MagnifyCost2)*MDist2+TraSizeCost2+SpeeCost2+NearbyCost2

[0506] Among them, Dist2 is the trajectory distance between the radar observation target and the visual observation target, MDist2 is the Mahalanobis distance between the first fused trajectory data of the radar observation target and the observed trajectory data of the visual observation target, MagnifyCost2 is the distance variance cost of the trajectory distance between the radar observation target and the visual observation target, TraSizeCost2 is the cost related to the number of cycles of the radar observation target and the visual observation target, SpeedCost2 is the speed cost of the radar observation target and the visual observation target, and NearbyCost2 is the near-point target penalty of the radar observation target and the visual observation target.

[0507] In some embodiments, the determining of the mutually associated radar observation target and visual observation target based on the track distance between the radar observation target and the visual observation target includes:

[0508] If the track distances between the radar observation target and at least two visual observation targets are calculated, it is determined that the radar observation target and the visual observation target with the smallest track distance are associated with each other.

[0509] In some embodiments, before determining the mutually associated radar observation targets and visual observation targets based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target, the method further includes:

[0510] Determine radar observation targets and visual observation targets that are not associated with each other based on speed information and / or distance information of the radar observation targets and the visual observation targets;

[0511] In some embodiments, the speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

[0512] In some embodiments, the second trajectory data includes observation trajectory data of at least one second visual observation target collected in the current stage, and observation trajectory data of at least one third visual observation target collected in the historical stage, and the method further includes:

[0513] Determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target;

[0514] The observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target that are related to each other are fused to obtain third fused trajectory data of at least one visual observation target.

[0515] In some embodiments, the determining of the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target includes:

[0516] Based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target, calculating the trajectory distance between the second visual observation target and the third visual observation target;

[0517] Based on the track distance between the second visual observation target and the third visual observation target, the second visual observation target and the third visual observation target associated with each other are determined.

[0518] In some embodiments, the above-mentioned calculation of the trajectory distance between the second visual observation target and the third visual observation target based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target includes:

[0519] The trajectory distance between the second visual observation target and the third visual observation target is determined based on the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

[0520] In some embodiments, the determining of the trajectory distance between the second visual observation target and the third visual observation target based on the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty, includes:

[0521] Dist3=(1+MagnifyCost3)*MDist3+TraSizeCost3+SpeedCost3+NearbyCost3

[0522] Among them, Dist3 is the trajectory distance between the second visual observation target and the third visual observation target, MDist3 is the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, MagnifyCost3 is the distance variance cost of the trajectory distance between the second visual observation target and the third visual observation target, TraSizeCost3 is the cost related to the number of cycles of the second visual observation target and the third visual observation target, SpeedCost3 is the speed cost of the second visual observation target and the third visual observation target, and NearbyCost3 is the near-point target penalty of the second visual observation target and the third visual observation target.

[0523] In some embodiments, the determining of the second visual observation target and the third visual observation target associated with each other based on the track distance between the second visual observation target and the third visual observation target includes:

[0524] If the track distances between the second visual observation target and at least two third visual observation targets are calculated, it is determined that the second visual observation target and the third visual observation target having the smallest track distance are associated with each other.

[0525] In some embodiments, before determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target, the method further includes:

[0526] Based on the speed information and / or distance information of the second visual observation target and the third visual observation target, the second visual observation target and the third visual observation target that do not satisfy the mutual association condition are determined.

[0527] In some embodiments, the speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; the distance information includes at least one of a longitudinal distance and a lateral distance. In some embodiments, before determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of at least one second visual observation target and the observation trajectory data of at least one third visual observation target, the method further includes:

[0528] If the second visual observation target and the third visual observation target have the same observation target identifier, it is determined that the second visual observation target and the third visual observation target having the same observation target identifier are associated with each other.

[0529] As described above, by first fusing the trajectory data of at least two radars and then fusing the fused radar trajectory data with the trajectory data of the camera, the information collected by all sensors can be fully utilized, thereby enhancing the accuracy and anti-interference of the trajectory data fusion results, thereby improving the vehicle's environmental perception ability and driving decision accuracy, and enabling the vehicle to adapt to complex and changeable driving environments.

[0530] The present application also provides an electronic device, such as Fig. 20 As shown, the electronic device includes: a processor 21 and a memory 22, the memory 22 stores programs or instructions that can be run on the processor, and the above-mentioned programs or instructions are executed by the processor to achieve the following Figure 1-Figure 18 The steps of the trajectory data fusion method.

[0531] The present application also provides a vehicle, such as Fig.21 As shown, the vehicle includes Fig. 20 The electronic device 31, wherein the memory of the electronic device stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the following is achieved: Figure 1-Figure 18 The steps of the trajectory data fusion method.

[0532] The present application also provides a computer-readable storage medium, wherein a program or instruction is stored on the computer-readable storage medium, and when the program or instruction is executed by a processor, the following is implemented: Figure 1-Figure 18 The steps of the trajectory data fusion method.

[0533] The present application also provides a computer program product, which, when executed by a processor of a vehicle, implements the following Figure 1-Figure 18 The steps of the trajectory data fusion method.

[0534] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0535] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

[0536] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims

1. A trajectory data fusion method, characterized in that: include: Acquire first track data respectively collected by at least two radars provided on the vehicle, and acquire second track data collected by a camera provided on the vehicle; Performing fusion processing on the first trajectory data respectively collected by the at least two radars to obtain fused third trajectory data, wherein the third trajectory data includes the first fused trajectory data of at least one radar observation target; The second trajectory data and the third trajectory data are fused to obtain fused fourth trajectory data, where the fourth trajectory data includes the second fused trajectory data of at least one observed target.

2. The method according to claim 1, characterized in that The at least two radars disposed on the vehicle include at least two of a front radar, a first corner radar located on a first side of the front radar, and a second corner radar located on a second side of the front radar.

3. The method according to claim 2, characterized in that The first trajectory data includes observation trajectory data of at least one radar observation target, and the obtaining of the first trajectory data respectively collected by at least two radars arranged on the vehicle includes: Acquire observation trajectory data of at least one radar observation target collected by the at least two radars in a current collection cycle; Based on the observation trajectory data of at least one radar observation target collected in the current collection cycle, the historical observation trajectory data of the radar observation target obtained by the at least two radars in the historical collection cycle is updated to obtain the first trajectory data.

4. The method according to claim 3, characterized in that: The second trajectory data includes observation trajectory data of at least one visual observation target, and the acquiring of the second trajectory data collected by a camera provided on the vehicle includes: Acquire observation trajectory data of at least one visual observation target collected by the camera in a current collection cycle; Based on the observation trajectory data of at least one visual observation target collected in the current collection cycle, the historical trajectory data of the visual observation target obtained by the camera in the historical collection cycle is updated to obtain the second trajectory data.

5. The method according to claim 4, characterized in that After acquiring the observation trajectory data of at least one radar observation target collected by the at least two radars in the current collection cycle, the method further includes: Based on a preset trajectory screening condition, the observation trajectory data of at least one radar observation target collected by the at least two radars in a current collection cycle are screened to retain the observation trajectory data of the radar observation target that meets the preset trajectory screening condition; and / or After acquiring the observation trajectory data of at least one visual observation target collected by the camera in the current collection cycle, the method further includes: The observation trajectory data of at least one visual observation target collected by the camera in a current collection cycle is filtered based on a preset trajectory screening condition to retain the observation trajectory data of the visual observation target that meets the preset trajectory screening condition.

6. The method according to claim 5, characterized in that The preset trajectory screening condition includes: at least one of a relative position threshold, a relative speed threshold, a target confidence threshold, a target type, a target identifier, and a validity identifier.

7. The method according to claim 4, characterized in that After screening the observation trajectory data of at least one radar observation target collected by the at least two radars in the current collection cycle based on the preset trajectory screening condition, the method further includes: performing coordinate system conversion on the observation trajectory data of the at least one radar observation target; and / or, After filtering the observation trajectory data of at least one visual observation target collected by the camera in the current collection cycle based on the preset trajectory screening condition, the method further includes: The observation trajectory data of the at least one visual observation target is converted into a coordinate system.

8. The method according to claim 7, characterized in that The observation trajectory data includes position data and speed data, and the coordinate system conversion of the observation trajectory data of the at least one radar observation target includes: Converting the position data of the at least one radar observation target to a vehicle rear axle coordinate system; converting the velocity data of the at least one radar observation target into an absolute coordinate system; and / or, The performing coordinate system conversion on the observation trajectory data of the at least one visual observation target comprises: The position data of the at least one visual observation target is converted to a vehicle rear axle coordinate system.

9. The method according to claim 4, characterized in that Also includes: If the observation trajectory data of at least one radar observation target is not collected in the current collection cycle, the observation trajectory data of the at least one radar observation target in the current collection cycle is determined based on the historical observation trajectory data of the at least one radar observation target in the previous two collection cycles; and / or, If the observation trajectory data of at least one visual observation target is not collected in the current collection cycle, the observation trajectory data of the at least one visual observation target in the current collection cycle is determined based on the historical observation trajectory data of the at least one visual observation target in the previous two collection cycles.

10. The method according to claim 9, characterized in that Also includes: If the observation trajectory data of at least one radar observation target is not collected for N consecutive collection cycles, the historical observation trajectory data of the at least one radar observation target is deleted; and / or, If the observation trajectory data of at least one visual observation target is not collected for N consecutive collection cycles, the historical observation trajectory data of the at least one visual observation target is deleted; Wherein, N is a positive integer greater than or equal to 2.

11. The method according to any one of claims 1 to 10, characterized in that: The at least two radars include a first radar and a second radar. Before fusing the first trajectory data respectively collected by the at least two radars, the method further includes: Determine the first radar observation target and the second radar observation target that are associated with each other based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar; The fusing and processing the first trajectory data respectively collected by the at least two radars comprises: The observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target that are mutually related are fused to obtain first fused trajectory data of at least one radar observation target.

12. The method according to claim 11, characterized in that The determining of the mutually associated first radar observation target and second radar observation target based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar comprises: Calculating a track distance between the first radar observation target and the second radar observation target based on the observation track data of the at least one first radar observation target and the observation track data of the at least one second radar observation target; Based on the track distance between the first radar observation target and the second radar observation target, the first radar observation target and the second radar observation target that are associated with each other are determined.

13. The method according to claim 12, characterized in that The calculating, based on the observation trajectory data of the at least one first radar observation target and the observation trajectory data of the at least one second radar observation target, a trajectory distance between the first radar observation target and the second radar observation target comprises: The trajectory distance between the first radar observed target and the second radar observed target is determined based on the Mahalanobis distance between the observed trajectory data of the first radar observed target and the observed trajectory data of the second radar observed target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

14. The method according to claim 13, characterized in that The first radar is a first corner radar, and the second radar is a second corner radar. The determining of the track distance between the target observed by the first radar and the target observed by the second radar based on the Mahalanobis distance between the observed track data of the target observed by the first radar and the observed track data of the target observed by the second radar, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty comprises: The track distance between the target observed by the first radar and the target observed by the second radar is determined based on the Mahalanobis distance between the observed track data of the target observed by the first radar and the observed track data of the target observed by the second radar, the additional limited distance between the first corner radar and the second corner radar, and at least one of the distance variance cost, the cycle number related cost, the speed cost and the near-point target penalty.

15. The method according to claim 13, characterized in that The determining the trajectory distance between the first radar observation target and the second radar observation target based on the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty, comprises: Dist1=(1+MagnifyCost1)*MDist1+TraSizeCost1+SpeedCost1+NearbyCost1 Among them, Dist1 is the trajectory distance between the first radar observation target and the second radar observation target, MDist1 is the Mahalanobis distance between the observation trajectory data of the first radar observation target and the observation trajectory data of the second radar observation target, MagnifyCost1 is the distance variance cost of the trajectory distance between the first radar observation target and the second radar observation target, TraSizeCost1 is the cost related to the number of cycles of the first radar observation target and the second radar observation target, SpeedCost1 is the speed cost of the first radar observation target and the second radar observation target, and NearbyCost1 is the near-point target penalty of the first radar observation target and the second radar observation target.

16. The method according to claim 12, characterized in that The determining, based on the track distance between the first radar observation target and the second radar observation target, of the first radar observation target and the second radar observation target that are associated with each other comprises: If the track distance between the first radar observation target and at least two second radar observation targets is calculated, it is determined that the first radar observation target and the second radar observation target with the smallest track distance are associated with each other.

17. The method according to claim 11, characterized in that Before determining the mutually associated first radar observation target and second radar observation target based on the observation trajectory data of at least one first radar observation target collected by the first radar and the observation trajectory data of at least one second radar observation target collected by the second radar, the method further includes: Based on the speed information and / or distance information of the first radar observation target and the second radar observation target, the first radar observation target and the second radar observation target that do not satisfy the mutual correlation condition are determined.

18. The method according to claim 17, characterized in that The speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

19. The method according to any one of claims 1 to 10, characterized in that: Before the fusion processing of the second trajectory data and the third trajectory data is performed, the method further includes: Determining mutually associated radar observation targets and visual observation targets based on first fused trajectory data of at least one radar observation target and observation trajectory data of at least one visual observation target; The fusing the second trajectory data and the third trajectory data comprises: The first fused trajectory data of the mutually related radar observation targets and the observation trajectory data of the visual observation targets are fused to obtain the second fused trajectory data of at least one observation target.

20. The method according to claim 19, characterized in that The determining of the mutually associated radar observation targets and visual observation targets based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target comprises: Calculating a trajectory distance between the radar observation target and the visual observation target based on the first fused trajectory data of the at least one radar observation target and the observation trajectory data of the at least one visual observation target; Based on the track distance between the radar observation target and the visual observation target, the radar observation target and the visual observation target that are associated with each other are determined.

21. The method according to claim 20, characterized in that The calculating, based on the first fused trajectory data of the at least one radar observed target and the observation trajectory data of the at least one visually observed target, the trajectory distance between the radar observed target and the visually observed target comprises: The trajectory distance between the radar observation target and the visual observation target is determined based on the Mahalanobis distance between the first fused trajectory data of the radar observation target and the observation trajectory data of the visual observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

22. The method according to claim 20, characterized in that The determining of the trajectory distance between the radar observed target and the visually observed target based on the Mahalanobis distance between the first fused trajectory data of the radar observed target and the observed trajectory data of the visually observed target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty, comprises: Dist2=(1+MagnifyCost2)*MDist2+TraSizeCost2+SpeedCost2+NearbyCost2 Among them, Dist2 is the trajectory distance between the radar observation target and the visual observation target, MDist2 is the Mahalanobis distance between the first fused trajectory data of the radar observation target and the observed trajectory data of the visual observation target, MagnifyCost2 is the distance variance cost of the trajectory distance between the radar observation target and the visual observation target, TraSizeCost2 is the cost related to the number of cycles of the radar observation target and the visual observation target, SpeedCost2 is the speed cost of the radar observation target and the visual observation target, and NearbyCost2 is the near-point target penalty of the radar observation target and the visual observation target.

23. The method according to claim 20, characterized in that The step of determining the mutually associated radar observation target and visual observation target based on the track distance between the radar observation target and the visual observation target comprises: If the track distances between the radar observation target and at least two visual observation targets are calculated, it is determined that the radar observation target and the visual observation target with the smallest track distance are associated with each other.

24. The method according to claim 19, characterized in that Before determining the mutually associated radar observation targets and visual observation targets based on the first fused trajectory data of at least one radar observation target and the observation trajectory data of at least one visual observation target, the method further includes: Based on the speed information and / or distance information of the radar observation target and the visual observation target, the radar observation target and the visual observation target that do not meet the mutual correlation condition are determined.

25. The method according to claim 24, characterized in that The speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

26. The method according to any one of claims 1 to 10, characterized in that: The second trajectory data includes observation trajectory data of at least one second visual observation target collected in the current stage, and observation trajectory data of at least one third visual observation target collected in the historical stage, and the method further includes: Determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of the at least one second visual observation target and the observation trajectory data of the at least one third visual observation target; The observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target that are related to each other are fused to obtain third fused trajectory data of at least one visual observation target.

27. The method according to claim 26, characterized in that The determining of the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of the at least one second visual observation target and the observation trajectory data of the at least one third visual observation target comprises: Calculating a trajectory distance between the second visual observation target and the third visual observation target based on the observation trajectory data of the at least one second visual observation target and the observation trajectory data of the at least one third visual observation target; Based on the track distance between the second visual observation target and the third visual observation target, the second visual observation target and the third visual observation target associated with each other are determined.

28. The method according to claim 27, characterized in that The calculating the trajectory distance between the second visual observation target and the third visual observation target based on the observation trajectory data of the at least one second visual observation target and the observation trajectory data of the at least one third visual observation target comprises: Determine a trajectory distance between the second visual observation target and the third visual observation target based on the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty.

29. The method according to claim 28, characterized in that The determining of the trajectory distance between the second visual observation target and the third visual observation target based on the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, and at least one of a distance variance cost, a cycle number related cost, a speed cost, and a near-point target penalty, comprises: Dist3=(1+MagnifyCost3)*MDist3+TraSizeCost3+SpeedCost3+NearbyCost3 Among them, Dist3 is the trajectory distance between the second visual observation target and the third visual observation target, MDist3 is the Mahalanobis distance between the observation trajectory data of the second visual observation target and the observation trajectory data of the third visual observation target, MagnifyCost3 is the distance variance cost of the trajectory distance between the second visual observation target and the third visual observation target, TraSizeCost3 is the cost related to the number of cycles of the second visual observation target and the third visual observation target, SpeedCost3 is the speed cost of the second visual observation target and the third visual observation target, and NearbyCost3 is the near-point target penalty of the second visual observation target and the third visual observation target.

30. The method according to claim 27, characterized in that The determining the second visual observation target and the third visual observation target that are associated with each other based on the track distance between the second visual observation target and the third visual observation target includes: If the track distances between the second visual observation target and at least two third visual observation targets are calculated, it is determined that the second visual observation target and the third visual observation target having the smallest track distance are associated with each other.

31. The method according to claim 26, characterized in that Before determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of the at least one second visual observation target and the observation trajectory data of the at least one third visual observation target, the method further includes: Based on the speed information and / or distance information of the second visual observation target and the third visual observation target, the second visual observation target and the third visual observation target that do not satisfy the mutual association condition are determined.

32. The method according to claim 31, characterized in that The speed information includes at least one of a longitudinal speed value, a lateral speed value, and a longitudinal speed difference value; and the distance information includes at least one of a longitudinal distance and a lateral distance.

33. The method according to claim 26, characterized in that Before determining the second visual observation target and the third visual observation target that are associated with each other based on the observation trajectory data of the at least one second visual observation target and the observation trajectory data of the at least one third visual observation target, the method further includes: If the second visual observation target and the third visual observation target have the same observation target identifier, it is determined that the second visual observation target and the third visual observation target having the same observation target identifier are associated with each other.

34. A trajectory data fusion device, characterized in that: include: A data acquisition module, used to acquire first trajectory data respectively collected by at least two radars provided on the vehicle, and to acquire second trajectory data collected by a camera provided on the vehicle; A first fusion processing module is used to perform fusion processing on the first trajectory data respectively collected by the at least two radars to obtain third trajectory data after fusion processing, wherein the third trajectory data includes the first fused trajectory data of at least one radar observation target; The second fusion processing module is used to perform fusion processing on the second trajectory data and the third trajectory data to obtain fused fourth trajectory data, where the fourth trajectory data includes the second fused trajectory data of at least one observed target.

35. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the trajectory data fusion method according to any one of claims 1 to 33 are implemented.

36. A vehicle, characterized in that: An electronic device comprising the electronic device of claim 35, wherein the memory of the electronic device stores a program or instruction that can be run on a processor, and the program or instruction, when executed by the processor, implements the steps of the trajectory data fusion method as described in any one of claims 1 to 33.

37. A computer-readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the trajectory data fusion method according to any one of claims 1 to 31 are implemented.

38. A computer program product, characterized in that When the program product is executed by a processor of a vehicle, the steps of the trajectory data fusion method as described in any one of claims 1 to 33 are implemented.

Images