Target fusion method and device, electronic equipment and storage medium

Abstract

The present application relates to a kind of target fusion method, device, electronic equipment and storage medium, the method comprises: obtaining multiple targets corresponding to multiple sensors, wherein multiple targets include the first target corresponding to laser radar, the second target corresponding to millimeter wave radar and the third target corresponding to visual sensor;Second target and third target are respectively associated with the first target, and data association result is obtained;According to data association result, the target of successful association is fused, and fusion target is obtained.The present application can use the first target corresponding to laser radar as the basis for data association and fusion, which avoids the problem of low accuracy of fusion results caused by using the third target corresponding to visual sensor as the basis for data association and fusion in the prior art, thereby improving the accuracy of target fusion results when the vehicle drives in special scenarios such as curves.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and specifically to a target fusion method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of autonomous driving technology, autonomous vehicles are becoming increasingly common in people's daily lives. The accuracy of target information fusion is crucial to the safety of autonomous driving; if the target information fusion error is large, it will seriously affect the safety of autonomous vehicles.

[0003] In existing technologies, target information fusion is typically based on visual sensors and millimeter-wave radar. However, visual sensors suffer from limitations in specific scenarios such as curves. These limitations include a short target recognition distance, poor recognition stability, and significant discrepancies between target attributes and reality. Even the relative position of the target and lane lines can change with the curve, especially with significant curve curvature. Inaccurate lateral motion models in visual sensors lead to inaccurate lateral velocity decomposition in curves, resulting in a greater discrepancy between the output target and reality as the curve widens. Therefore, existing target information fusion methods have low accuracy in specific scenarios like curves. Summary of the Invention

[0004] One objective of this invention is to provide a target fusion method to solve the problem of low accuracy of fusion results in special scenarios such as curves in existing target information fusion methods; a second objective is to provide a target fusion device; a third objective is to provide an electronic device; and a fourth objective is to provide a storage medium.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A target fusion method includes: acquiring multiple targets corresponding to multiple sensors, wherein the multiple targets include a first type of target corresponding to a lidar, a second type of target corresponding to a millimeter-wave radar, and a third type of target corresponding to a vision sensor; associating the second type of target and the third type of target with the first type of target respectively to obtain a data association result; and fusing the successfully associated targets according to the data association result to obtain a fused target.

[0007] Based on the aforementioned technical methods, the second type of target corresponding to millimeter-wave radar and the third type of target corresponding to visual sensors can be correlated and fused with the first type of target corresponding to lidar, respectively. In other words, the first type of target corresponding to lidar is used as the basis for data correlation and fusion. This fully utilizes the high-precision ranging characteristics and highly stable target output characteristics of lidar in curved scenarios, avoiding the low accuracy of fusion results caused by using the third type of target corresponding to visual sensors as the basis for data correlation and fusion in existing technologies. This improves the accuracy of target fusion results when vehicles are driving in special scenarios such as curves, thereby enhancing vehicle safety.

[0008] Further, the step of associating the second type of target and the third type of target with the first type of target respectively to obtain a data association result includes: determining the first type of target as the initial target; based on the initial target and a preset elliptic threshold, selecting a set of candidate targets from the second type of target and the third type of target; determining the Mahalanobis distance, motion trend, and target type matching result between each candidate target in the candidate target set and the initial target; determining the optimal candidate target corresponding to the initial target based on the Mahalanobis distance, the motion trend, and the target type matching result; and associating the initial target and the optimal candidate target with data to obtain the data association result.

[0009] Based on the above technical means, the first type of target corresponding to the lidar can be used as the basis for data association. The optimal candidate target can be selected from the second type of target corresponding to the millimeter-wave radar and the third type of target corresponding to the vision sensor to be associated with the first type of target. This will minimize the error interference of the third type of target corresponding to the vision sensor in the curved scene and improve the accuracy of target data association.

[0010] Furthermore, the step of associating the initial target and the optimal candidate target with data to obtain the data association result includes: acquiring the association matching algorithm and association parameters pre-designed for each sensor, wherein the association matching algorithm and association parameters are different for different sensors; and associating the initial target and the optimal candidate target with data according to the association matching algorithm and association parameters corresponding to the initial target and the optimal candidate target to obtain the data association result.

[0011] Based on the above technical means, different association matching algorithms and association parameters can be set for different sensors. Therefore, in the data association process, both the association algorithm common to each sensor and the association matching algorithm and association parameters corresponding to each sensor can be executed, which solves the problem of association difficulties caused by the huge differences in the working principles and actual performance of each sensor.

[0012] Furthermore, the step of fusing the successfully associated targets based on the data association results to obtain the fused target includes: obtaining the road scene where the vehicle is currently located, wherein the road scene is a curve scene or a straight road scene; adjusting the target attribute weight values ​​of the initial target and the optimal candidate target in the data association results based on the road scene; and fusing the initial target and the optimal candidate target based on the adjusted target attribute weight values ​​to obtain the fused target.

[0013] Based on the above technical means, the target attribute weight values ​​of the initial target and the target attribute weight values ​​of the optimal candidate target can be dynamically adjusted under different road scenarios, so that the overall performance of the fused target reaches a high level.

[0014] Furthermore, adjusting the target attribute weight values ​​of the initial target and the optimal candidate target in the data association result based on the road scene includes:

[0015] When the road scene is a curved scene, the position attribute weight value of the initial target and the velocity attribute weight value of the optimal candidate target are increased, and the position attribute weight value of the optimal candidate target and the velocity attribute weight value of the initial target are decreased, wherein the position attribute weight value and the velocity attribute weight value are both part of the target attribute weight value.

[0016] Based on the above technical means, in a curve scenario, the position attribute weight value of the initial target and the velocity attribute weight value of the optimal candidate target can be increased, while the position attribute weight value of the optimal candidate target and the velocity attribute weight value of the initial target can be decreased, thus making full use of the advantages of each sensor on different target attributes.

[0017] Furthermore, after fusing the successfully associated targets based on the data association results to obtain fused targets, the method further includes:

[0018] When the fusion target incorporates data from the third type of target, it is determined whether the center point of the fusion target falls within the visual target bounding box, where the visual target bounding box refers to the two-dimensional box obtained by projecting the third type of target onto the visual image projection plane; if it is determined that the center point of the fusion target falls within the visual target bounding box, the fusion target is retained; if it is determined that the center point of the fusion target does not fall within the visual target bounding box, the fusion target is discarded.

[0019] Based on the above technical means, visual target bounding boxes can be used to judge the rationality of fused targets and realize virtual target verification.

[0020] Furthermore, before associating the second type of target and the third type of target with the first type of target respectively to obtain a data association result, the method further includes: converting the data corresponding to the first type of target, the second type of target, and the third type of target according to a predefined data format; and synchronizing the first type of target, the second type of target, and the third type of target after format conversion in a spatiotemporal manner. Associating the second type of target and the third type of target with the first type of target respectively to obtain a data association result includes: associating the spatiotemporally synchronized first type of target, the second type of target, and the third type of target with data to obtain the data association result.

[0021] Based on the aforementioned technical means, data format conversion and spatiotemporal synchronization can be performed on the first, second, and third types of targets before data association, thereby improving the accuracy of data association.

[0022] A target fusion device includes: an acquisition module for acquiring multiple targets corresponding to multiple sensors, wherein the multiple targets include a first type of target corresponding to a lidar, a second type of target corresponding to a millimeter-wave radar, and a third type of target corresponding to a vision sensor; an association module for data association between the second type of target and the third type of target and the first type of target, respectively, to obtain a data association result; and a fusion module for fusing the successfully associated targets according to the data association result to obtain a fused target.

[0023] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements the steps of the aforementioned target fusion method.

[0024] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned target fusion method.

[0025] The beneficial effects of this invention are:

[0026] (1) This invention can associate and fuse the second type of target corresponding to millimeter-wave radar and the third type of target corresponding to visual sensor with the first type of target corresponding to lidar, respectively, that is, using the first type of target corresponding to lidar as the basis for data association and fusion. In this way, the high-precision ranging characteristics and high-stability target output characteristics of lidar in curved scenarios can be fully utilized, avoiding the problem of low accuracy of fusion results caused by using the third type of target corresponding to visual sensor as the basis for data association and fusion in the prior art, thereby improving the accuracy of target fusion results when the vehicle is driving in special scenarios such as curves, and thus improving vehicle safety;

[0027] (2) This invention can set different association matching algorithms and association parameters for different sensors. Therefore, in the data association process, it can execute the association algorithm that is common to each sensor, as well as the association matching algorithm and association parameters corresponding to each sensor itself, thus solving the problem of association difficulties caused by the huge differences in the working principles and actual performance of each sensor.

[0028] (3) The present invention can use visual target boxes to make reasonable judgments on the fusion target and realize virtual target verification. Attached Figure Description

[0029] Figure 1 A flowchart illustrating a target fusion method provided in an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the structure of a vehicle provided in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the structure of a target fusion device provided in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0034] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0035] See Figure 1 , Figure 1 This is a flowchart illustrating a target fusion method provided in an embodiment of the present invention. Figure 1 As shown, the target fusion method may include the following steps:

[0036] Step 101: Obtain multiple targets corresponding to multiple sensors, including the first type of target corresponding to the lidar, the second type of target corresponding to the millimeter-wave radar, and the third type of target corresponding to the vision sensor.

[0037] Specifically, the aforementioned multiple sensors may include, but are not limited to, various sensors such as lidar, millimeter-wave radar, and vision sensors. Among them, lidar can use lasers to detect targets and obtain the first type of target; millimeter-wave radar can use millimeter waves to detect targets and obtain the second type of target; and vision sensors can use image visual analysis to detect targets and obtain the third type of target.

[0038] Step 102: Associate the second and third types of targets with the first type of target to obtain the data association results.

[0039] It should be noted that, based on engineering experience, in straight-line scenarios, the differences in target position and velocity between various sensors are not significant, resulting in relatively accurate fusion information. However, in curved scenarios without LiDAR, the third type of target generally has a larger distance error than the second type of target. This often leads to difficulties in associating the third type of target newly created using visual sensors with the corresponding second type of target from the millimeter-wave radar. Consequently, the final fused target position information cannot be corrected using millimeter-wave information with smaller distance errors, resulting in a large error in the final fused target output on curves. Due to the increased position error of the third type of target, no matter how the corresponding association parameters are adjusted, the accuracy of matching in curved and ordinary scenarios cannot be guaranteed. Therefore, in this embodiment of the invention, LiDAR is introduced to use the first type of target created by the LiDAR for data association. When performing data association in a curved scenario, the second and third types of targets can be associated with the first type of target respectively. Since the second type of target has a smaller deviation from the first type of target, while the third type of target has a larger deviation from the first type of target, the first type of target is usually associated with the second type of target. This avoids the problem of low accuracy of the fusion result caused by using the third type of target corresponding to the visual sensor as the basis for data association and fusion.

[0040] Step 103: Based on the data association results, merge the successfully associated targets to obtain the merged targets.

[0041] After the data association is completed, each data association result contains targets from each sensor. Therefore, it is necessary to fuse the targets from each sensor in each data association result to obtain the fused target.

[0042] In this embodiment, the second type of target corresponding to the millimeter-wave radar and the third type of target corresponding to the visual sensor can be associated and fused with the first type of target corresponding to the lidar, respectively. That is, the first type of target corresponding to the lidar is used as the basis for data association and fusion. This fully utilizes the high-precision ranging characteristics and highly stable target output characteristics of lidar in curved scenarios, avoiding the problem of low accuracy in fusion results caused by using the third type of target corresponding to the visual sensor as the basis for data association and fusion in existing technologies. This improves the accuracy of target fusion results when the vehicle is driving in special scenarios such as curves, thereby enhancing vehicle safety.

[0043] Furthermore, in step 102 above, the second and third types of targets are respectively associated with the first type of target to obtain the data association results, including:

[0044] The first type of target is identified as the initial target;

[0045] Based on the initial objective and the preset elliptic threshold, a set of candidate objectives is selected from the second and third types of objectives;

[0046] Determine the Mahalanobis distance, motion trend, and target type matching results between each candidate target in the candidate target set and the initial target;

[0047] Based on the Mahalanobis distance, motion trend, and target type matching results, determine the optimal candidate target corresponding to the initial target;

[0048] Data association is performed between the initial target and the optimal candidate target to obtain the data association results.

[0049] Specifically, the size of the preset elliptical threshold can be set according to actual conditions, and this application embodiment does not impose a specific limitation. The elliptical threshold is used to filter candidate targets from the second type of target and the third type of target to form a candidate target set.

[0050] It should be noted that, since there is no LiDAR in the current technology, the process of obtaining the optimal candidate target is roughly as follows: In the first frame of the program, the third type of target newly created by the visual sensor is used as the initial target. Starting from the second frame, using the initial target as the midpoint, elliptical thresholds that can be adjusted according to the magnitude of longitudinal distance are used to initially screen potentially matching candidate targets, forming a candidate target set. In the candidate target set, the Mahalanobis distance between the initial target and each candidate target is calculated sequentially. The candidate target with the smallest Mahalanobis distance is selected as the optimal candidate target, completing the association step.

[0051] In this embodiment, the first type of target newly created by the lidar is used as the initial target. Due to its high positional accuracy, it can search for candidate targets from sensors such as vision and millimeter-wave radar within a relatively small elliptical threshold, forming a candidate target set. Third type targets corresponding to vision sensors are often eliminated at this stage; if not, they are used normally and the candidate step is performed. Then, the Mahalanobis distance between the initial target and each candidate target in the candidate target set is calculated. The Mahalanobis distance calculation dimension consists of horizontal and vertical distance and horizontal and vertical velocity. Simultaneously, it is necessary to consider whether the initial target and each candidate target in the candidate target set have the same motion trend and the same target type. Finally, the target with the smallest Mahalanobis distance, the same motion trend, and the same target type is selected as the optimal candidate target. At this stage, since the detection accuracy of lidar and millimeter-wave radar is similar, a relatively optimal situation of correlation between lidar and millimeter-wave radar is often formed. Adding the target from the millimeter-wave radar to the fused target further improves the output accuracy and stability.

[0052] In this way, the first type of target corresponding to the lidar can be used as the basis for data association. The optimal candidate target can be selected from the second type of target corresponding to the millimeter-wave radar and the third type of target corresponding to the vision sensor to be associated with the first type of target. This will minimize the error interference of the third type of target corresponding to the vision sensor in the curved scene and improve the accuracy of target data association.

[0053] Furthermore, the above steps involve data association between the initial target and the optimal candidate target to obtain data association results, including:

[0054] Obtain the pre-designed association matching algorithm and association parameters for each sensor, where the association matching algorithm and association parameters are different for different sensors;

[0055] Based on the association matching algorithm and association parameters corresponding to the initial target, and the association matching algorithm and association parameters corresponding to the optimal candidate target, data association is performed on the initial target and the optimal candidate target to obtain the data association result.

[0056] In one embodiment, due to the significant differences in the working principles and actual performance of each sensor, the association matching algorithm and association parameters corresponding to each sensor can be different. For example, millimeter-wave radar has high longitudinal ranging accuracy, so when using elliptical threshold screening, the threshold range can be appropriately reduced to reduce the number of candidate targets entering the candidate target set. For the third type of targets corresponding to visual sensors, there may be overlapping target boxes. Therefore, during association, logic can be added to check whether a new target overlaps with an existing target. If an overlap is detected, it is likely due to a visual false target, and therefore no association is performed. When designing the association matching algorithm for each sensor, an abstract association function can be designed using a C++ abstract class. The association matching algorithm for each sensor inherits from this abstract class, and the sensor association algorithm is executed sequentially in each cycle. This ensures that both the general association algorithm and each sensor's specific algorithm and parameters are executed.

[0057] In this way, different association matching algorithms and association parameters can be set for different sensors. Therefore, during the data association process, both the association algorithm common to all sensors and the association matching algorithm and association parameters corresponding to each sensor can be executed, which solves the problem of association difficulties caused by the huge differences in the working principles and actual performance of various sensors.

[0058] Furthermore, in step 103 above, based on the data association results, the successfully associated targets are fused to obtain fused targets, including:

[0059] Obtain the current road scene where the vehicle is located, where the road scene is either a curved scene or a straight scene;

[0060] Based on the road scenario, the target attribute weight values ​​of the initial target and the optimal candidate target in the data association results are adjusted;

[0061] Based on the adjusted target attribute weights, the initial target and the optimal candidate target are fused to obtain the fused target.

[0062] In one embodiment, each fusion target contains candidate target information from multiple sensors, which can be used to perform candidate data fusion. The general approach to target fusion is as follows: based on engineering experience, the target attribute weights of each sensor can be dynamically adjusted according to the current road scene of the vehicle, based on different target attributes (such as position, speed, etc.). In this way, the target attribute weights of the initial target and the optimal candidate target can be dynamically adjusted under different road scenes, so that the overall performance of the fused target reaches a high level.

[0063] Furthermore, based on the road scenario, the above steps adjust the target attribute weight values ​​of the initial target and the optimal candidate target in the data association results, including:

[0064] In the case of a road scene with curves, the weight values ​​of the position attribute of the initial target and the velocity attribute of the optimal candidate target are increased, while the weight values ​​of the position attribute of the optimal candidate target and the velocity attribute of the initial target are decreased. The position attribute weight value and the velocity attribute weight value are both target attribute weight values.

[0065] In one embodiment, because the LiDAR is accurate in the longitudinal and lateral positions of the target in a curved scenario, but the speed convergence is slow and large speed errors often occur, while the millimeter-wave radar is more accurate in longitudinal position, speed and lateral speed, the fusion algorithm increases the weight of the LiDAR in the final fused target position attribute and increases the weight of the millimeter-wave radar in the speed attribute, making full use of the advantages of each sensor in different target attributes, so that the overall performance of the output target reaches a high level.

[0066] Furthermore, after step 103 above, where the successfully associated targets are fused based on the data association results to obtain the fused targets, the method further includes:

[0067] When the fusion target includes data of a third type of target, determine whether the center point of the fusion target falls within the visual target bounding box. The visual target bounding box refers to the two-dimensional box obtained by projecting the third type of target onto the visual image projection plane.

[0068] If the center point of the fusion target falls within the visual target bounding box, the fusion target is preserved.

[0069] If the center point of the target to be merged does not fall within the visual target bounding box, the target to be merged will be removed.

[0070] In one embodiment, because the point cloud information of millimeter-wave radar is easily affected by environmental interference in curved scenarios, and lasers are prone to producing false targets, a visual target bounding box (provided that the third type of target enters the candidate target set and is eventually associated) can be introduced to associate with the first type of target. Although the position of the third type of target differs significantly from that of the first type of target in the top view, the projection of the first type of target point can often still be projected into the visual target bounding box of the third type of target on the visual image projection plane. Utilizing this characteristic, a corresponding association strategy is developed: in each running cycle, it is determined whether the midpoint of the first type of target falls within the visual target bounding box, and this is used as the matching association standard. After verifying the projection of the third type of target through this association, the errors of lasers and millimeter-wave radars identifying large and continuous obstacles such as road guardrails and bushes as vehicles or other targets in curved scenarios can be largely avoided.

[0071] In this embodiment, the visual target bounding box is used to judge the rationality of the fused target and screen out and remove false targets, which can make up for the impact of false targets of lidar and millimeter-wave radar on the function.

[0072] Furthermore, before step 102 above, which involves associating the second and third types of targets with the first type of target to obtain the data association results, the method further includes:

[0073] According to the predefined data format, the data corresponding to the first type of target, the second type of target, and the third type of target are converted into their respective formats.

[0074] The first, second, and third types of targets after format conversion are spatiotemporally synchronized.

[0075] The second and third types of targets are respectively associated with the first type of target to obtain the data association results, including:

[0076] Data association was performed on the first, second, and third types of targets after spatiotemporal synchronization to obtain the data association results.

[0077] In one embodiment, each sensor typically emits raw information such as target location, target speed, and target type. However, due to differences in sensor type and model, the format and representation of this raw information vary. Therefore, a unified data format needs to be predefined. The received raw signals are assigned to corresponding internal data variables according to their content, thereby unifying the data format and achieving hardware-software isolation. Simultaneously, all sensor data is timestamped with the vehicle's system time to achieve time synchronization. Furthermore, since the sensors are installed at different locations on the vehicle body and use different coordinate systems for target detection, it is also necessary to move the measurement points of each sensor to the center point of the front bumper of the vehicle through rotation and translation to achieve spatial synchronization.

[0078] Based on the aforementioned technical means, data format conversion and spatiotemporal synchronization can be performed on the first, second, and third types of targets before data association, thereby improving the accuracy of data association.

[0079] In one embodiment, the vehicle may include a front camera, a front millimeter-wave radar, an angle radar, a panoramic camera, a lidar, a vehicle information data receiving module, information preprocessing modules corresponding to each sensor, data association modules corresponding to each sensor, a data fusion module, a rationality judgment module, and a fusion target output module, such as... Figure 2 As shown, the front camera, front millimeter-wave radar, corner radar, panoramic camera, LiDAR, and vehicle information serve as the raw input information sources for the fusion target. The information preprocessing module is responsible for converting data of different formats into a unified data format within the fusion algorithm, and is also responsible for the spatiotemporal alignment of all data. The data association module is responsible for matching the output results of each sensor at the same time, matching the target from different sensor data sources together for data fusion, and adding association strategies for LiDAR and millimeter-wave radar, ultimately obtaining accurate target information even under curves, providing a reliable data source for subsequent autonomous driving function modules.

[0080] Specifically, the front camera can be an 8MP variable focal length camera used to detect all vehicles and pedestrians within a 120° range in front of the vehicle. The target recognition algorithm for the front camera can utilize data from the Horizon Robotics J5 BEV framework to achieve routine identification of vehicles, pedestrians, animals, and cyclists on structured roads. This algorithm has passed automotive-grade safety certification and possesses engineering capabilities while meeting performance requirements. The algorithm can also output target information via Ethernet, which may include, but is not limited to: target position, target speed, target dimensions, target tracking number, target heading information, and target type.

[0081] The front millimeter-wave radar uses the Doppler effect to detect targets within a 120° range in front of the vehicle. The greater the radial velocity difference between the target object and the vehicle, the clearer the detection and the more accurate the target output. The millimeter-wave radar needs to use a bus communication interface to acquire the vehicle's speed and heading information and calibrate the output target attributes in real time. Because the front millimeter-wave radar uses the radar cross section (RCS) for target identification and tracking, it is significantly affected by the environment, resulting in larger lateral position and velocity errors, but more accurate longitudinal attributes. Furthermore, for stationary targets, it can only output a point without dimensions.

[0082] The corner radar is essentially the same as the front millimeter-wave radar, both using the Doppler effect to detect targets. The difference is that the corner radar is installed on both sides of the front bumper of the vehicle, with a detection range of 100° to the left and right. The corner radar needs to use a bus communication interface to obtain the vehicle's speed and heading information, and calibrate the output target attributes in real time. At the same time, in order to avoid road noise interference, algorithms such as trajectory estimation and empirical thresholds are used to filter the target. The target information here may include, but is not limited to: target position, target speed, target length and width, target tracking number, etc.

[0083] The panoramic camera, a 2MP sensor, is mounted on both sides of the vehicle. Its primary function is to detect all vehicles and pedestrians within a 120° radius to the left and right of the vehicle. Its detection range includes targets close to the vehicle and outputs target-level information based on target attributes. The panoramic camera uses the Horizon J5BEV framework as its target recognition algorithm, demonstrating good performance for target output on structured roads. However, it often encounters issues such as false targets and large position and velocity errors in urban scenes.

[0084] LiDAR can be installed at the front of a vehicle to detect and identify targets in front of the vehicle. Specifically, a 128-line LiDAR can be used to scan targets in front every 100ms and return 700,000 point cloud data. The LiDAR backend uses a combination of deep learning and traditional clustering algorithms in real time to identify point cloud features, output target information, and transmit it to the backend module according to a predefined protocol.

[0085] In the multi-sensor system comprised of a front-facing camera, front millimeter-wave radar, corner radar, panoramic camera, and lidar, the output characteristics of each sensor differ significantly when detecting the same target. Often, at a given moment, one sensor may fail to detect the target while others do. Therefore, to address this characteristic, a module for effectively statistically analyzing sensor output information can be developed. This module retrieves the output information of all sensors at the same time. An empty output indicates that the corresponding sensor did not detect the target, and the module will label this sensor as "not detected." Subsequent information fusion will then skip this sensor, significantly reducing data indexing time.

[0086] When performing data association and target matching, the data association module follows the matching order based on the flag bits of each sensor. A successful match indicates that the target is the same. The backend then uses this sensor data for data fusion to output target data with higher information redundancy. Because the lateral velocity component of targets on curves is generally inaccurately estimated, the position and velocity attributes output by visual sensors in this scenario differ significantly from reality, sometimes even resulting in non-identification or false targets. In this situation, it is difficult to distinguish between the accuracy of different targets. Therefore, during association matching, visual and millimeter-wave radar targets often differ too much to be associated as the same target. Furthermore, visual sensors often have large detection errors for stationary targets, frequently misidentifying them as dynamic targets, ultimately leading to errors in the data attributes of the backend fusion algorithm. However, in this embodiment of the invention, the accuracy and small discrepancy in the position attribute detection of the first type of target corresponding to the lidar and the second type of target corresponding to the millimeter-wave radar are utilized. The matching logic of the millimeter-wave radar is introduced into the lidar association algorithm. When the lidar detects a target, the fusion can output the target, and the millimeter-wave radar data can correct the inaccuracy of the lidar velocity, ultimately outputting target information with more accurate position and velocity on curves.

[0087] See Figure 3 , Figure 3 This is a schematic diagram of a target fusion device provided in an embodiment of the present invention. Figure 3 As shown, the target fusion device 300 includes:

[0088] The acquisition module 301 is used to acquire multiple targets corresponding to multiple sensors, including a first type of target corresponding to the lidar, a second type of target corresponding to the millimeter-wave radar, and a third type of target corresponding to the vision sensor;

[0089] The association module 302 is used to associate the second type of target and the third type of target with the first type of target respectively to obtain the data association result;

[0090] The fusion module 303 is used to fuse the successfully associated targets based on the data association results to obtain the fused target.

[0091] Furthermore, the associated module 302 includes:

[0092] The first determination submodule is used to determine the first type of target as the initial target;

[0093] The filtering submodule is used to filter out a set of candidate targets from the second and third types of targets based on the initial target and the preset elliptical threshold.

[0094] The second determination submodule is used to determine the Mahalanobis distance, motion trend and target type matching results between each candidate target in the candidate target set and the initial target;

[0095] The third determination submodule is used to determine the optimal candidate target corresponding to the initial target based on Mahalanobis distance, motion trend and target type matching results;

[0096] The data association submodule is used to perform data association between the initial target and the optimal candidate target to obtain the data association results.

[0097] Furthermore, the data association submodule includes:

[0098] The acquisition unit is used to acquire the pre-designed association matching algorithm and association parameters for each sensor, wherein the association matching algorithm and association parameters are different for different sensors;

[0099] The data association unit is used to perform data association between the initial target and the optimal candidate target based on the association matching algorithm and association parameters corresponding to the initial target and the association matching algorithm and association parameters corresponding to the optimal candidate target, so as to obtain the data association result.

[0100] Furthermore, the fusion module 303 includes:

[0101] The acquisition submodule is used to acquire the road scene where the vehicle is currently located, where the road scene is either a curved scene or a straight scene;

[0102] The adjustment submodule is used to adjust the target attribute weight values ​​of the initial target and the optimal candidate target in the data association results based on the road scenario.

[0103] The fusion submodule is used to fuse the initial target and the best candidate target based on the adjusted target attribute weight values ​​to obtain the fused target.

[0104] Furthermore, the adjustments to the sub-modules include:

[0105] The adjustment unit is used to increase the position attribute weight value of the initial target and the velocity attribute weight value of the optimal candidate target when the road scene is a curve scene, and to decrease the position attribute weight value of the optimal candidate target and the velocity attribute weight value of the initial target. The position attribute weight value and the velocity attribute weight value are both target attribute weight values.

[0106] Furthermore, the device 300 includes:

[0107] The determination module is used to determine whether the center point of the fused target falls within the visual target bounding box when the fused target contains data of a third type of target. The visual target bounding box refers to the two-dimensional box obtained by projecting the third type of target onto the visual image projection plane.

[0108] The retention module is used to retain the fusion target if the center point of the fusion target falls within the visual target bounding box.

[0109] The elimination module is used to eliminate the target to be merged if the center point of the target does not fall within the visual target bounding box.

[0110] Furthermore, the device 300 includes:

[0111] The format conversion module is used to convert the data corresponding to the first type of target, the second type of target, and the third type of target according to the predefined data format.

[0112] The spatiotemporal synchronization module is used to perform spatiotemporal synchronization on the first, second, and third types of targets after format conversion.

[0113] The association module 302 is also used to perform data association on the first type of target, the second type of target and the third type of target after spatiotemporal synchronization, and obtain the data association result.

[0114] It should be noted that the device 300 can implement the steps of the target fusion method provided in any of the aforementioned method embodiments and achieve the same technical effect, which will not be elaborated here.

[0115] like Figure 4 As shown, this embodiment of the invention also provides an electronic device, including a processor 411, a communication interface 412, a memory 413, and a communication bus 414, wherein the processor 411, the communication interface 412, and the memory 413 communicate with each other through the communication bus 414.

[0116] Memory 413 is used to store computer programs;

[0117] In one embodiment of the present invention, when the processor 411 executes the program stored in the memory 413, it implements the target fusion method provided in any of the foregoing method embodiments, including:

[0118] Acquire multiple targets corresponding to multiple sensors, including a first type of target corresponding to lidar, a second type of target corresponding to millimeter-wave radar, and a third type of target corresponding to vision sensor;

[0119] The second and third types of targets are respectively associated with the first type of target to obtain the data association results;

[0120] Based on the data association results, the successfully associated targets are merged to obtain the merged targets.

[0121] In addition, embodiments of the present invention also provide a vehicle that includes the aforementioned electronic equipment, which can implement the steps of the target fusion method provided in any of the foregoing method embodiments.

[0122] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the target fusion method provided in any of the foregoing method embodiments.

[0123] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A target fusion method, characterized by, The method comprises: obtaining a plurality of targets corresponding to a plurality of sensors, wherein the plurality of targets comprise a first type of target corresponding to a laser radar, a second type of target corresponding to a millimeter wave radar, and a third type of target corresponding to a visual sensor; data associating the second type of target and the third type of target with the first type of target respectively to obtain a data association result; fusing the targets associated successfully according to the data association result to obtain a fused target; wherein the data associating the second type of target and the third type of target with the first type of target respectively to obtain a data association result comprises: determining the first type of target as an initial target; screening a candidate target set from the second type of target and the third type of target based on the initial target and a preset elliptical threshold; determining Mahalanobis distance, motion trend and target type matching result between each candidate target in the candidate target set and the initial target respectively; determining an optimal candidate target corresponding to the initial target according to the Mahalanobis distance, the motion trend and the target type matching result; data associating the initial target and the optimal candidate target to obtain the data association result.

2. The method of claim 1, wherein, The data associating the initial target and the optimal candidate target to obtain the data association result comprises: obtaining an association matching algorithm and an association parameter designed in advance by each sensor, wherein the association matching algorithm and the association parameter corresponding to different sensors are different; data associating the initial target and the optimal candidate target according to the association matching algorithm and the association parameter corresponding to the initial target, and the association matching algorithm and the association parameter corresponding to the optimal candidate target to obtain the data association result.

3. The method of claim 1, wherein, The fusing the targets associated successfully according to the data association result to obtain a fused target comprises: obtaining a road scene in which a vehicle currently locates, wherein the road scene is a curved road scene or a straight road scene; adjusting target attribute weight values of the initial target and the optimal candidate target in the data association result according to the road scene; fusing the initial target and the optimal candidate target based on the adjusted target attribute weight values to obtain the fused target.

4. The method of claim 3, wherein, The adjusting target attribute weight values of the initial target and the optimal candidate target in the data association result according to the road scene comprises: in the case that the road scene is a curved road scene, increasing a position attribute weight value of the initial target and a speed attribute weight value of the optimal candidate target, and decreasing the position attribute weight value of the optimal candidate target and the speed attribute weight value of the initial target, wherein the position attribute weight value and the speed attribute weight value both belong to the target attribute weight value.

5. The method of claim 1, wherein, After the fusing the targets associated successfully according to the data association result to obtain a fused target, the method further comprises: In a case where the fusion target is fused with data of the third type of target, it is determined whether a center point of the fusion target falls within a visual target frame, wherein the visual target frame refers to a two-dimensional frame obtained by projecting the third type of target on a visual image projection plane; In a case where it is determined that the center point of the fusion target falls within the visual target frame, the fusion target is retained; In a case where it is determined that the center point of the fusion target does not fall within the visual target frame, the fusion target is eliminated.

6. The method of claim 1, wherein, Before the data of the second type of target and the data of the third type of target are respectively associated with the data of the first type of target to obtain a data association result, the method further comprises: format conversion is performed on the data corresponding to the first type of target, the second type of target and the third type of target respectively according to a pre-defined data format; the first type of target, the second type of target and the third type of target after format conversion are time-space synchronized; the data association of the first type of target, the second type of target and the third type of target after time-space synchronization is performed to obtain the data association result. The apparatus comprises:

7. A target fusion device, characterized by, an acquisition module configured to acquire a plurality of targets corresponding to a plurality of sensors, wherein the plurality of targets comprises a first type of target corresponding to a laser radar, a second type of target corresponding to a millimeter wave radar and a third type of target corresponding to a visual sensor; an association module configured to respectively associate the second type of target and the third type of target with the first type of target to obtain a data association result; a fusion module configured to fuse the targets successfully associated according to the data association result to obtain a fusion target; The association module comprises: a first determination sub-module configured to determine the first type of target as an initial target; a screening sub-module configured to screen a candidate target set from the second type of target and the third type of target based on the initial target and a pre-set elliptical threshold; a second determination sub-module configured to determine a Mahalanobis distance, a motion trend and a target type matching result between each candidate target in the candidate target set and the initial target; a third determination sub-module configured to determine an optimal candidate target corresponding to the initial target according to the Mahalanobis distance, the motion trend and the target type matching result; a data association sub-module configured to perform data association on the initial target and the optimal candidate target to obtain the data association result. The apparatus comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; 8. An electronic device, comprising: The memory is configured to store a computer program; The processor is configured to execute the program stored on the memory to implement the steps of the target fusion method according to any one of claims 1-6. The computer program is executed by the processor to implement the steps of the target fusion method according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the target fusion method according to any one of claims 1-6.

Images