Multi-sensor association method and auxiliary driving system

Abstract

The invention relates to a multi-sensor association method. The method comprises the following steps: a receiving step: receiving targets detected by a plurality of sensors; a centralized association step: performing centralized association on the targets detected by the plurality of sensors and prior trajectory information to generate a centralized association result; and a dispersive association step of performing dispersive association on the targets detected by the plurality of sensors to generate a dispersive association result. According to the multi-sensor association method, the accuracy and success rate of sensor association can be improved.

Description

A method for correlating multiple sensors and an advanced driver assistance system Technical Field

[0001] This invention relates to autonomous driving technology, and more specifically to a method for associating multiple sensors and an assisted driving system. Background Technology

[0002] Sensor fusion technology is one of the core technologies of Advanced Driver Assistance Systems (ADAS) and Autonomous Driving Systems (AD). With the development of autonomous driving technology, vehicles need to rely on various sensors to perceive their surroundings, including other vehicles, pedestrians, road signs, and obstacles, to ensure safe driving and efficient navigation. These sensors include, but are not limited to, radar, LiDAR, cameras, and ultrasonic sensors (USS).

[0003] The purpose of sensor fusion technology is to correlate and fuse data from different sensors, thereby improving the accuracy and reliability of environmental perception. However, traditional fusion techniques still suffer from insufficient accuracy. Summary of the Invention

[0004] In view of the problems in the prior art, the present invention aims to provide a multi-sensor association method and an assisted driving system that can improve the accuracy and success rate of sensor association.

[0005] The multi-sensor correlation method of one aspect of the present invention includes the following steps:

[0006] The receiving step involves receiving targets detected by multiple sensors.

[0007] The centralized association step involves centrally associating the targets detected by the multiple sensors with prior trajectory information to generate a centralized association result; and

[0008] The distributed correlation step involves performing distributed correlation on the targets detected by the multiple sensors to generate distributed correlation results. Attached Figure Description

[0009] The described and other objects and advantages of this application will become more fully clear from the following detailed description taken in conjunction with the accompanying drawings, wherein the same or similar elements are denoted by the same reference numerals.

[0010] Figure 1 is a schematic diagram of a simple fusion system.

[0011] Figure 2 is a schematic diagram of a multi-sensor association module.

[0012] Figure 3 is a schematic diagram of a fusion system.

[0013] Figure 4 illustrates an example of a clustered association result.

[0014] Figure 5 is a schematic diagram illustrating a multi-sensor association method according to an embodiment of the present invention.

[0015] Figure 6 shows an example of the distributed association results.

[0016] Figure 7 shows an example of a combined association table.

[0017] Figure 8 is a schematic diagram of a multi-sensor association method according to an embodiment.

[0018] Figure 9 is a schematic diagram of the centralized association results of the embodiment shown in Figure 8.

[0019] Figure 10 is a schematic diagram of the distributed correlation results of the embodiment shown in Figure 8. Detailed Implementation

[0020] The following are some of the various embodiments of the present invention, intended to provide a basic understanding of the invention, and not intended to identify key or decisive elements of the invention or to limit the scope of protection.

[0021] First, the inventors conducted the following research on existing technologies for sensors, sensor correlation methods, and sensor fusion systems.

[0022] The sensors used in this invention include, but are not limited to, cameras, lidar, radar, and ultrasonic sensors.

[0023] Sensor association methods were proposed to address the problem that, assuming two sensors detect a vehicle / obstacle, they would typically output two different targets with different positions, speeds, types, sizes, etc. By applying an association method, these two targets can be associated as a single target. Generally, such sensor association methods are applied in fusion systems with multiple sensors.

[0024] The following is an example of a simple fusion system.

[0025] Figure 1 is a schematic diagram of a simple fusion system.

[0026] The fusion system has two sensors (a camera and a radar). In Figure 1, XY represents the coordinate system, ● represents the target detected by the camera, ■ represents the target detected by the radar, and ◆ represents the target after fusing the target detected by the camera and the target detected by the radar. It can be seen that although the camera and the radar detect the same vehicle, they will output two different targets. The two sensors are associated through the association module (not shown) in the fusion system to obtain the fused target.

[0027] Figure 2 is a schematic diagram of a multi-sensor association module.

[0028] As shown in Figure 2, the inputs of the multi-sensor association module are targets 11-14 detected by the sensors and prior trajectory information 15. The module outputs association information 17 between these input targets by applying an association algorithm 16 to these inputs. In this example, targets 11-14 detected by the sensors are target 11 detected by the camera, target 12 detected by the radar, target 13 detected by the lidar, and target 14 detected by the ultrasonic sensor, respectively.

[0029] Figure 3 is a schematic diagram of a simplified fusion system.

[0030] As shown in Figure 3, the simplified fusion system 20 includes: an association module 21, an update module 22, and a post-processing module 23. The fusion system 20 takes as input targets A1 detected by multiple sensors (e.g., cameras, radar, lidar, and ultrasonic sensors) and outputs targets A2 fused from these targets. These targets A2 are then provided to, for example, ADAS or AD.

[0031] The association method in Figures 2 and 3 is to associate prior trajectory information with the target detected by the current input sensor. This is called "centralized association". As an example, it can be understood as associating all targets detected by all sensors with a trajectory list.

[0032] Figure 4 illustrates an example of a clustered association result.

[0033] In Figure 4, t-1, t-2, t-3, ..., tN represent prior trajectories 1, 2, 3, ..., N, respectively; sx represents sensor x; and o-1, o-3, o-6, o-7, o-9, and o-12 represent targets 1, 3, 6, 7, 9, and 12, respectively.

[0034] Figure 4 shows the association of prior trajectories 1 to N with targets detected by a sensor x. However, through studying Figure 4, the inventors of this invention discovered that no associations were generated between different sensors in Figure 4, which are crucial for improving the accuracy and success rate of the associations.

[0035] In view of the above findings, this invention proposes an innovative multi-sensor association method that includes not only "centralized association" but also "distributed association".

[0036] Figure 5 is a schematic diagram illustrating a multi-sensor association method according to an embodiment of the present invention.

[0037] As shown in Figure 5, a multi-sensor association method according to an embodiment of the present invention includes:

[0038] The centralized association step involves centrally associating the detection data (i.e., the detected targets) from multiple sensors with prior trajectory information and generating a centralized association result. Specifically, it involves centrally associating the target 31 detected by the camera, the target 32 ​​detected by the radar, the target 33 detected by the lidar, and the target 34 detected by the ultrasonic sensor with prior trajectory information 35, generating a centralized association result 37.

[0039] The distributed association step involves distributed association of the detection data (i.e., the detected targets) from multiple sensors and generating distributed association results. Specifically, the target 31 detected by the camera, the target 32 ​​detected by the radar, the target 33 detected by the lidar, and the target 34 detected by the ultrasonic sensor are distributed and associated 38, and then the distributed association results 40 are generated after post-processing 39.

[0040] Optionally, after the dispersed association step, a fusion step is further included, in which the centralized association result 37 and the dispersed association result 40 are fused to generate a final association result (not shown in Figure 5).

[0041] Optionally, as an example, the centralized association result 37 and the decentralized association result 40 will be used simultaneously, which can be understood as a union that will be used by the subsequent level.

[0042] The prior trajectory information uses the final association result from the previous time period, meaning the prior trajectory information includes both centralized association result 37 and decentralized association result 40. Here, an example is given to illustrate how centralized association is implemented.

[0043] In this example, the centralized association mainly includes the following steps:

[0044] The data preprocessing steps include time synchronization and spatial calibration of data from various sensors to ensure that data from different sensors are within the same time frame and spatially consistent; and

[0045] The feature extraction step extracts key features from the data from various sensors, such as position, velocity, acceleration, shape, and size. Association algorithms, such as Multiple Hypothesis Tracking (MHT) and Joint Probabilistic Data Association (JPDA), are then used to match the targets detected by the sensors with prior trajectory information to obtain matching results.

[0046] The results generation step updates the prior trajectory information based on the matching results, generating a centralized association result that includes the latest sensor data and trajectory predictions.

[0047] The prior trajectory information here generally refers to all the target information maintained in the sensor fusion system. This information is obtained by receiving and fusing the detection results from different sensors. These fusion results are then used as the prior trajectory information for the next fusion cycle. In other words, the prior trajectory information of this frame is the fusion result of the previous frame.

[0048] Next, we will explain the dispersed associations.

[0049] Distributed correlation, also known as "sensor-sensor correlation," refers to associating targets detected by any two sensors and generating a correlation result between those two sensors. As an example, the distributed correlation process includes the following steps:

[0050] Targets detected by two different sensors are paired and associated using an association algorithm to obtain the association results. Examples of association algorithms include global nearest neighbor algorithms and local nearest neighbor algorithms.

[0051] The pairwise correlation results of targets detected by two different sensors are post-processed to obtain a more optimized distributed correlation result.

[0052] Figure 6 shows an example of the distributed association results.

[0053] Figure 6 shows an example of pairwise association results between sensors x and m. sx represents sensor x, sm represents sensor m, and o-1, o-3, o-6, o-7, o-9, and o-12 represent targets 1, 3, 6, 7, 9, and 12, respectively. Specifically, pairwise associations between sensors x and m yield the following pairs: target 1 with targets 1 and 6, target 2 with target 3, target 3 with targets 7 and 12, and so on, with target N and target 9.

[0054] In Figure 6, after the association results of two different sensors x and m are obtained, a post-processing algorithm is used to construct a comprehensive association table of all sensor association results.

[0055] Figure 7 shows an example of a comprehensive association table.

[0056] In the comprehensive association table shown in Figure 7, SS represents the sensor-sensor association result, s-1, s-2, s-3, ..., sN represent sensor 1, sensor 2, sensor 3, ..., sensor N, and id represents the target identifier.

[0057] If target 1 (id-1) of sensor 1 (s-1 in Figure 7) is associated with target 5 (id-5) of sensor 2 (s2 in Figure 7), and target 5 of sensor 2 is associated with target 2 (id-2) of sensor 3 (s-3 in Figure 7), and so on, then the post-processing above will yield targets with id = 1, 5, 2, ..., 7. Although they come from different sensors, they all represent the same target.

[0058] Next, an embodiment of the multi-sensor association method of the present invention will be described in detail.

[0059] Figure 8 is a schematic diagram of a multi-sensor association method according to an embodiment.

[0060] This embodiment uses one camera and two radars, for a total of three sensors. As shown in Figure 8, the multi-sensor association method in this embodiment includes:

[0061] The centralized association step involves centrally associating the detection data from the three sensors with prior trajectory information and generating centralized association results. Specifically, the target 41 detected by the camera, the target 42 detected by the first radar, and the target 43 detected by the second radar are centrally associating with prior trajectory information 44, generating centralized association results 45; and

[0062] The distributed correlation step involves performing distributed correlation on the detection data from the three sensors and generating distributed correlation results. Specifically, the target 41 detected by the camera, the target 42 detected by the first radar, and the target 43 detected by the second radar are respectively subjected to pairwise distributed correlation 47 to obtain distributed correlation results. After post-processing 48 on the distributed correlation results, a comprehensive distributed correlation result 49 is generated.

[0063] As a preferred embodiment, the multi-sensor association method may further include a fusion step (not shown) to fuse the centralized association results and the decentralized association results to obtain a more robust final association result.

[0064] Optionally, as an example, the centralized association results and the decentralized association results will be used simultaneously, which can be understood as a union that will be used by the subsequent level. Figure 9 shows a schematic diagram of the centralized association results of the embodiment shown in Figure 8.

[0065] First, as shown in Figure 9, by applying "centralized association," the targets detected by the three sensors are associated with prior trajectory information to obtain three centralized association results, specifically including:

[0066] The prior trajectory information P and the target S1 detected by the camera (there can be multiple targets, the same below) are centrally correlated to obtain the first central correlation result CR1;

[0067] The prior trajectory information P and the target S2 detected by the first radar are centrally correlated to obtain the second central correlation result CR2.

[0068] The prior trajectory information P and the target S3 detected by the second radar are centrally correlated to obtain the third central correlation result CR3.

[0069] Figure 10 is a schematic diagram of the distributed correlation results of the embodiment shown in Figure 8.

[0070] Secondly, as shown in Figure 10, the detection results of every two sensors are correlated pairwise to obtain three discrete correlation results, specifically including:

[0071] The first decentralized correlation result DR1 is obtained by associating target S1 detected by the associated camera and target S2 detected by the first radar.

[0072] The second decentralized correlation result DR2 is obtained by associating the target S1 detected by the associated camera and the target S3 detected by the second radar.

[0073] The third decentralized correlation result DR3 is obtained by correlating target S2 detected by the first radar and target S3 detected by the second radar; and

[0074] Post-processing combines the first dispersed association result DR1 to the third dispersed association result DR3 into a single dispersed association result.

[0075] As a preferred approach, it can be further configured to merge centralized and decentralized association results to obtain a more robust final association result.

[0076] The present invention also provides a domain controller, including a storage module, a processor, and a computer program stored on the storage module and executable on the processor, wherein the processor executes the computer program to implement the above-described multi-sensor association method.

[0077] The present invention also provides a computer program product, including a computer program that, when executed by a processor, performs the above-described multi-sensor association method.

[0078] The present invention also provides a driver assistance system, comprising: a plurality of sensors and at least one domain controller communicatively connected to the plurality of sensors, the domain controller being configured to determine a target by employing the multi-sensor association method described above to achieve driver assistance functions.

[0079] Among the aforementioned driver assistance functions are: Adaptive Cruise Control (ACC), Automatic Emergency Braking (AEB), Traffic Jam Assist (TJA), Lane Keeping Assist (LKA), etc.

[0080] As described above, the multi-sensor association method of the present invention combines centralized association and decentralized association, which utilizes both the global optimization capability of centralized association and the local fine-tuning of decentralized association, thereby significantly improving the accuracy and reliability of data fusion.

[0081] Moreover, according to the multi-sensor association method of the present invention, higher accuracy and consistency in target tracking and state estimation are achieved through multi-level association of prior trajectories and sensor data.

[0082] Furthermore, according to the multi-sensor association method of the present invention, by using two association methods and unifying the results of the two association methods through a post-processing algorithm, the problem of missing relationships between sensors in the prior art can be solved, ensuring the high quality of the final association result. Moreover, the application of the post-processing algorithm can effectively unify the data association results of different sensors, enhancing the robustness and adaptability of the system.

[0083] According to the present invention, more accurate perception fusion results can be provided for auxiliary functions such as adaptive cruise control (ACC), automatic emergency braking (AEB), traffic jam assist (TJA), and lane keeping assist (LKA).

[0084] The multi-sensor association method of this invention is not only applicable to advanced driver assistance systems (ADAS) and autonomous driving systems (AD), but can also be extended to other scenarios that require multi-sensor data fusion, such as robot navigation and intelligent monitoring. In these application scenarios, it can improve the fusion effect of multiple sensor data and enhance the versatility and practicality of the system.

[0085] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Those skilled in the art can conceive of other feasible variations or substitutions based on the technical scope disclosed in this application, and such variations or substitutions are all covered within the scope of protection of this application. Where there is no conflict, the embodiments and features described in the embodiments of this application can also be combined with each other. The scope of protection of this application is determined by the claims.

Claims

1. A method for correlating multiple sensors, characterized in that, Includes the following steps: The receiving step involves receiving targets detected by multiple sensors. The centralized association step involves centrally associating the targets detected by the multiple sensors with prior trajectory information to generate a centralized association result. The process includes a distributed correlation step, which performs distributed correlation on the targets detected by the multiple sensors to generate distributed correlation results.

2. The multi-sensor correlation method as described in claim 1, characterized in that, The process further includes a fusion step following the dispersed association step, which merges the centralized association result and the dispersed association result to generate a final association result.

3. The multi-sensor correlation method as described in claim 1, characterized in that, The decentralized association step includes the following sub-steps: performing decentralized association on the targets detected by the multiple sensors respectively; And post-process the results obtained from the distributed association to generate the distributed association result.

4. The multi-sensor correlation method as described in claim 3, characterized in that, The step of dispersing and associating the targets detected by the multiple sensors includes: associating the targets detected by each pair of sensors using an association algorithm.

5. The multi-sensor correlation method as described in claim 4, characterized in that, The association algorithm can be any of the following: global nearest neighbor algorithm, local nearest neighbor algorithm, etc.

6. The multi-sensor correlation method as described in claim 4, characterized in that, The step of post-processing the results obtained from the decentralized association to generate decentralized association results includes: using a post-processing algorithm to construct a comprehensive association table from the results obtained from the decentralized association of all sensors to generate decentralized association results.

7. The multi-sensor correlation method as described in claim 1, characterized in that, The prior trajectory information adopts the final association result of the previous time period.

8. A domain controller, comprising a storage module, a processor, and a computer program stored on the storage module and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-sensor association method according to any one of claims 1 to 8.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the multi-sensor association method according to any one of claims 1 to 7.

10. A driver assistance system, comprising: A plurality of sensors and at least one domain controller communicatively connected to the plurality of sensors, characterized in that the domain controller is configured to determine a target to achieve an assisted driving function by employing the multi-sensor association method according to any one of claims 1 to 7.

11. The driver assistance system as described in claim 10, characterized in that, The driver assistance functions include one or more of the following: adaptive cruise control, automatic emergency braking, traffic jam assist, and lane keeping assist.

Images