A multi-sensor target fusion method, system, storage medium and vehicle

Abstract

The application discloses a multi-sensor target fusion method, comprising the steps of: obtaining current target recognition results of multiple sensors in an intelligent driving system in real time; performing position fusion processing on the recognized targets in the multiple sensors to obtain fused target information; establishing an identification framework according to the categories and category confidence degrees of each sensor corresponding to each fused target, and obtaining the fused basic probability assignment function value of each category corresponding to each fused target by using an evidence theory method; and determining the final output category of each fused target. The application further discloses a corresponding system, a storage medium and a vehicle. By implementing the application, conflicting evidences can be effectively fused, and the accuracy of target detection and recognition is improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a multi-sensor target fusion method, system, storage medium, and vehicle. Background Technology

[0002] Target detection and recognition is a crucial component of autonomous driving technology. However, due to their inherent application characteristics, single sensors cannot consistently deliver reliable detection results across various scenarios and weather conditions. For example, cameras are significantly affected by external conditions (light and shadow), resulting in poor recognition performance in low-light environments; LiDAR is highly sensitive to environmental factors, such as noise from rain; and while millimeter-wave radar has strong penetration capabilities through fog, smoke, and dust, its target recognition ability is relatively weak. In this context, target recognition based on multi-sensor information fusion technology can leverage the strengths of various sensors, thereby enhancing the stability and accuracy of target detection and recognition.

[0003] However, in practical applications, due to the influence of the sensor's own physical characteristics, its own malfunctions, and external interference, different sensors will have different confidence levels in the detection and recognition results of the target, and may even have different degrees of conflict. Therefore, a stable and reliable target fusion method is particularly important for the accuracy of the final target detection and recognition. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multi-sensor target fusion method, system, storage medium and vehicle that can effectively fuse conflicting evidence and improve the accuracy of target detection and identification.

[0005] To address the aforementioned technical problems, as one aspect of the present invention, a multi-sensor target fusion method is provided, applied to an intelligent driving system, comprising at least the following steps:

[0006] The system obtains the current target recognition results from multiple sensors in the intelligent driving system in real time. The current target recognition results of each sensor include at least: the target's corresponding identifier, position, position covariance, category, category confidence, and speed information.

[0007] The targets identified by the various sensors are subjected to position fusion processing to obtain fused target information. The fused target information includes at least: the identifier, position, and velocity information of the fused target, as well as the category and category confidence level identified by each sensor.

[0008] Based on the categories and category confidence levels identified by each sensor corresponding to the fusion target at each location, an identification framework is established, and the basic probability allocation function values ​​of each category corresponding to each fusion target are obtained by using evidence theory methods.

[0009] Based on the basic probability allocation function values ​​of each category after fusion corresponding to each fusion target, and combined with the velocity information of the fusion target, the final output category of each fusion target is determined.

[0010] The step of obtaining the current target recognition results of multiple sensors in the intelligent driving system in real time includes:

[0011] During driving, the intelligent driving system collects real-time data from various sensors, including cameras, millimeter-wave radar, and lidar.

[0012] By employing the corresponding detection, recognition, and tracking algorithms pre-set for each sensor, the real-time data collected by each sensor is identified to obtain the target recognition result corresponding to each sensor.

[0013] The step of performing position fusion processing on the targets identified by the multiple sensors to obtain fused target information further includes:

[0014] Convert the data collected by each sensor to the vehicle's coordinate system;

[0015] After obtaining the target detection and tracking results of each sensor, one of the sensors is selected as the master sensor. When the tracking result of the master sensor is received for the first time, a fusion sequence is established based on the result of the master sensor as the first frame fusion result. After the target tracking results of other sensors are received, the Hungarian matching algorithm is used to match the target of the previous frame fusion result, and the Kalman filtering algorithm is used to predict the previous frame fusion result. The prediction result is then updated with the tracking results of the matched sensors.

[0016] Each matched target is associated with the data identified by each sensor to obtain a fusion result. The fusion result includes the identifier, position, and velocity information of each fusion target, as well as the category and category confidence level identified by each sensor.

[0017] The step of establishing an identification framework based on the categories and category confidence levels identified by each sensor corresponding to the fusion target at each location, and obtaining the fusion-based basic probability allocation function values ​​for each category corresponding to each fusion target using evidence theory methods, further includes:

[0018] A recognition framework Θ = {pedestrian, bicycle, motor vehicle} is established based on the predefined classification attributes of each matching target; a basic probability assignment m is performed on the different target categories corresponding to each sensor in the recognition framework based on the confidence level of the categories identified by each sensor. i (A j ), m i (Aj ) represents the basic probability assignment of the i-th sensor to the j-th class of the matched target;

[0019] Obtain the recognition accuracy α of different sensors for different target categories from historical data. ij ;

[0020] The basic probability allocation function value in the recognition framework is corrected using the recognition accuracy of different sensors for different categories using the following formula:

[0021] m′ i (A j )=α ij ·m i (A j )

[0022] The corrected basic probability assignment function values ​​are normalized to obtain the processed target category probability data:

[0023]

[0024] The processed target category probability data are fused using the Dempster synthesis rule to obtain the fused basic probability allocation function value for each category corresponding to each fused target.

[0025] Among them, the recognition accuracy α of different sensors for different target categories in historical data was obtained. ij The steps further include:

[0026] Based on the data annotation sets of each sensor, the recognition accuracy α of different sensors for different target categories is calculated using the following formula. ij :

[0027]

[0028] Where, α ij Let TP be the recognition accuracy of the i-th sensor for the j-th type of target. ij TN represents the number of targets of type j correctly identified by the i-th sensor. ij To determine the number of negative classes identified as negative, FP ij FN is the number of negative classes identified as positive classes. ij The number of positive classes identified as negative classes; when calculating the recognition accuracy of the j-th class target, all other classes are considered as negative classes.

[0029] The step of determining the final output category of each target based on the fused basic probability allocation function values ​​for each category corresponding to the target, and in conjunction with the target's velocity information, further includes:

[0030] The basic probability assignment function values ​​for each category of each target are sorted, and the top two basic probability assignment function values ​​are compared.

[0031] If the difference between the two is greater than or equal to a preset comparison threshold, then the category corresponding to the maximum basic probability assignment function value is taken as the classification of the target.

[0032] If the difference between the two is less than the preset comparison threshold, the speed of the target is matched with both, and the category that matches the speed is used as the classification of the target.

[0033] Accordingly, as another aspect of the present invention, a multi-sensor target fusion system is also provided, applied in an intelligent driving system, comprising at least:

[0034] The target recognition result acquisition unit is used to obtain the current target recognition results of multiple sensors in the intelligent driving system in real time. The current target recognition result of each sensor includes at least: the identification number, position, position covariance, category, category confidence, and speed information of each target.

[0035] The position fusion processing unit is used to perform position fusion processing on the targets identified by the multiple sensors to obtain fused target information. The fused target information includes at least: the identifier, position, and velocity information of the fused target, as well as the category and category confidence level identified by each sensor.

[0036] The evidence fusion processing unit is used to establish an identification framework based on the categories and category confidence levels identified by each sensor corresponding to the fusion target at each location, and to obtain the fusion basic probability allocation function values ​​of each category corresponding to each fusion target using evidence theory methods.

[0037] The output category decision unit is used to determine the final output category of each fusion target based on the basic probability allocation function value of each category after fusion corresponding to each fusion target, and in combination with the speed information of the fusion target.

[0038] The target recognition result acquisition unit further includes:

[0039] The data acquisition unit is used to collect real-time data from various sensors in the intelligent driving system during driving. These sensors include cameras, millimeter-wave radar, and lidar.

[0040] The target recognition unit is used to identify the real-time data collected by each sensor using the corresponding detection, recognition and tracking algorithms pre-set for each sensor, and obtain the target recognition result corresponding to each sensor.

[0041] The location fusion processing unit further includes:

[0042] The coordinate transformation unit is used to uniformly transform the data collected by various sensors to the vehicle's coordinate system.

[0043] The position fusion unit is used to select one of the sensors as the master sensor after obtaining the target detection and tracking results of each sensor. When the tracking result of the master sensor is received for the first time, a fusion sequence is established based on the result of the master sensor as the first frame fusion result. After receiving the target tracking results of other sensors, the Hungarian matching algorithm is used to match the target of the previous frame fusion result, and the Kalman filtering algorithm is used to predict the previous frame fusion result. The prediction result is updated with the tracking results of the matched sensors.

[0044] The location fusion result acquisition unit is used to associate each matched target with the data identified by each sensor to obtain the fusion result. The fusion result includes the identifier, position, and velocity information of the fusion target corresponding to each fusion target, as well as the category and category confidence level identified by each sensor.

[0045] The evidence fusion processing unit further includes:

[0046] The recognition framework building unit is used to build a recognition framework Θ = {pedestrian, bicycle, motor vehicle} based on the predefined classification attributes of each matched target; and to perform basic probability assignment m on the different target categories corresponding to each sensor in the recognition framework based on the confidence scores of the categories identified by each sensor. i (A j ), m i (A j ) represents the basic probability assignment of the i-th sensor to the j-th class of the matched target;

[0047] The recognition accuracy acquisition unit is used to obtain the recognition accuracy α of different sensors for different target categories in historical data. ij ;

[0048] The evidence correction unit is used to correct the basic probability allocation function value in the recognition framework using the recognition accuracy of different sensors for different categories, according to the following formula:

[0049] m′ i (A j )=α ij ·m i (A j )

[0050] The normalization unit is used to normalize the corrected basic probability assignment function values ​​to obtain the processed target category probability data.

[0051]

[0052] The evidence fusion calculation unit is used to perform fusion calculation on the processed target category probability data using the Dempster synthesis rule to obtain the fused basic probability allocation function value of each category corresponding to each fused target.

[0053] The recognition accuracy acquisition unit further includes:

[0054] The computing unit is used to calculate the recognition accuracy α of different sensors for different target categories based on the data annotation sets of each sensor using the following formula. ij :

[0055]

[0056] Where, α ij Let TP be the recognition accuracy of the i-th sensor for the j-th type of target. ij TN represents the number of targets of type j correctly identified by the i-th sensor. ij To determine the number of negative classes identified as negative, FP ij FN is the number of negative classes identified as positive classes. ij The number of positive classes identified as negative classes; when calculating the recognition accuracy of the j-th class target, all other classes are considered as negative classes.

[0057] The output category decision unit further includes:

[0058] The sorting and comparison unit is used to sort the basic probability assignment function values ​​of each category for each target and compare the top two basic probability assignment function values.

[0059] The decision unit is used to compare the comparison result of the ranking comparison unit with a preset comparison threshold. If the difference between the two is greater than or equal to the preset comparison threshold, the category corresponding to the maximum basic probability assignment function value is taken as the classification of the target; otherwise, the speed of the target is matched with the two, and the category that matches the speed is taken as the classification of the target.

[0060] Accordingly, as another aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the sensor target fusion method as described above.

[0061] Accordingly, as another aspect of the present invention, a vehicle is also provided, wherein an intelligent driving system is provided, characterized in that the intelligent driving system has multiple types of sensors and integrates a multi-sensor target fusion system as described above.

[0062] Implementing the embodiments of the present invention has the following beneficial effects:

[0063] This invention provides a multi-sensor target fusion method, system, storage medium, and vehicle. By employing an improved evidence theory approach to fuse target category attributes, it fully utilizes the statistical information verifiable in the dataset from different sensor detection and recognition algorithms. Specifically, it introduces the recognition accuracy of different sensors to correct the support levels of different evidence, enabling more effective fusion of conflicting evidence and improving the accuracy of target detection and recognition.

[0064] Secondly, in the embodiments of the present invention, a rich set of decision rules are used in the decision-making process of category judgment, and a comprehensive judgment is made considering the speed of matching the target, which avoids the misjudgment problem caused by a single decision rule and further improves the accuracy of category judgment.

[0065] In addition, in this embodiment of the invention, the Kalman filter algorithm is used to realize the association and position fusion of targets between different sensors, which solves the problem of target flashing and jitter caused by inaccurate or discontinuous sensing distance of a single sensor.

[0066] This invention can be applied to autonomous driving systems, enabling effective and stable detection and identification of targets, which helps improve the safety, reliability, and comfort of autonomous driving. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, obtaining other drawings based on these drawings without creative effort still falls within the scope of the present invention.

[0068] Figure 1 This is a schematic diagram of the main flow of an embodiment of a multi-sensor target fusion method provided by the present invention;

[0069] Figure 2 for Figure 1 A more detailed step diagram of step S12 in the diagram;

[0070] Figure 3 This is a schematic diagram of a structure of an embodiment of a multi-sensor target fusion system provided by the present invention;

[0071] Figure 4 This is a schematic diagram of the structure of the three target recognition result acquisition units;

[0072] Figure 5 for Figure 3A schematic diagram of the structure of the mid-position fusion processing unit;

[0073] Figure 6 for Figure 3 A schematic diagram of the structure of the evidence fusion processing unit;

[0074] Figure 7 for Figure 3 A schematic diagram of the structure of the output category decision unit. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0076] like Figure 1 The diagram shown illustrates the main flow of an embodiment of a multi-sensor target fusion method provided by the present invention. (In conjunction with...) Figure 2 As shown, in this embodiment, the method includes the following steps:

[0077] Step S10: Obtain the current target recognition results of multiple sensors in the intelligent driving system in real time. The current target recognition results of each sensor include at least: the identifier number, position, position covariance, category, category confidence, and speed information of each target.

[0078] In a specific example, step S10 further includes:

[0079] During driving, the intelligent driving system collects real-time data from various sensors, including cameras, millimeter-wave radar, and lidar; more specifically, the lidar outputs point cloud data, the camera outputs image data, and the millimeter-wave radar outputs a target list.

[0080] The system employs pre-configured detection, recognition, and tracking algorithms for each sensor to identify the real-time data collected by each sensor, thereby obtaining the target recognition result for each sensor. The detection, recognition, and tracking algorithms are pre-built, and the target detection and tracking results for each sensor include, but are not limited to, attribute information such as target identification number, position, position covariance, category, category confidence level, size, size variance, velocity, and heading angle.

[0081] By collecting labeled data, a test dataset is constructed. Based on the test dataset, various performance indicators of target detection and recognition algorithms corresponding to different sensors can be calculated, such as accuracy.

[0082] Step S11: Perform position fusion processing on the targets identified by the multiple sensors to obtain fused target information. The fused target information includes at least: the identifier, position, and velocity information of the fused target, as well as the category and category confidence level identified by each sensor.

[0083] In a specific example, step S11 further includes:

[0084] Step S110: Convert the data collected by each sensor to the vehicle coordinate system.

[0085] Different sensors typically output data based on their own sensor coordinate systems. Therefore, it is necessary to convert data from different sensor coordinate systems to the vehicle's own coordinate system. The vehicle's coordinate system uses the direction of the vehicle's front as the positive X-axis, the direction of the left side of the vehicle as the positive Y-axis, the direction perpendicular to the roof and upwards as the positive Z-axis, and the center point of the vehicle's logo at the front as the origin of the coordinate system.

[0086] The various attributes of the target detection information are transformed from the sensor coordinate system to the vehicle's coordinate system. Assume the coordinates in different sensor coordinate systems are (x... i ,y i ,z i The coordinates of the vehicle in its coordinate system are (x...). v ,y v ,z v The conversion formula is as follows:

[0087]

[0088] Where i is the i-th type of sensor, 1≤i≤N, and R i It is a rotation matrix, a 3x3 matrix, T i It is a translation matrix, a 3x1 matrix. R i and T i It can be obtained through pre-calibration.

[0089] Step S111: After obtaining the target detection and tracking results from each sensor, select one sensor as the master sensor. When the tracking result from the master sensor is received for the first time, a fusion sequence is established based on the master sensor's result as the first frame fusion result. Subsequently, when the target tracking result from another sensor is received, the previous frame fusion result is predicted and updated accordingly. Specifically, after receiving target tracking results from other sensors, target matching is performed on the previous frame fusion result using a matching algorithm based on the tracking ID and a Hungarian matching algorithm. The previous frame fusion result is then predicted using a Kalman filter algorithm, and the predicted result is updated using the matched sensor tracking results.

[0090] The Hungarian matching algorithm mainly performs the following: (1) Calculate the correlation matrix: calculate the distance between the center point of each fused target and each detected target to form the correlation matrix; (2) Segment the subgraph: in order to improve its computational performance, the original bipartite graph is segmented into subgraphs by deleting vertices whose distance is greater than a reasonable maximum distance threshold; (3) Calculate the best match: for each subgraph, the Hungarian algorithm is used to find the best detection and tracking match by minimizing the distance cost; if the distance between the two nodes after matching meets the requirements (less than 4 meters), the two are considered to be successfully matched.

[0091] Kalman filtering requires minimal computation, providing an efficient and computationally achievable method for estimating process states while minimizing the mean square error of the estimation. It is one of the most important and commonly used data fusion algorithms today. This invention employs a Kalman-filter-based CA model for motion estimation.

[0092] Step S112 involves associating each matched target with the data identified by each sensor to obtain a fusion result. The fusion result includes the identifier, position, and velocity information of each fused target, as well as the category and category confidence level identified by each sensor. It is understood that by recording which sensor data each fused target was obtained from in the fusion result, the relevant attributes of the target in different sensor detection and recognition algorithms can be traced.

[0093] Step S12: Based on the categories and category confidence levels identified by each sensor corresponding to the fusion target at each location, establish an identification framework and use evidence theory to obtain the fusion basic probability allocation function values ​​for each category corresponding to the fusion target.

[0094] In a specific example, step S12 further includes:

[0095] Step S120: Establish an identification framework Θ = {pedestrian, bicycle, motor vehicle} based on the predefined classification attributes of each matching target; within the identification framework Θ (sample space), a complete and mutually exclusive set of propositions forms a power set 2. Θ ;

[0096] It is understandable that the basic probability assignment based on the recognition framework Θ is a 2^T. Θ The function m (basic confidence assignment function) in the range [0,1] is called the mass function, and it satisfies the following conditions: and Here, A represents any proposition (subset) in the recognition framework Θ, and m(A) represents the degree of support of the evidence for proposition A. Furthermore, the A that makes m(A) > 0 is called the focal element of the evidence.

[0097] In this invention, three target attributes (pedestrians, bicycles, and motor vehicles) are defined in this method. Therefore, these three target attributes form the recognition framework Θ = {pedestrians, bicycles, and motor vehicles}. Simultaneously, each sensor used for target detection and recognition is equivalent to a piece of evidence, and the detection and recognition result based on the data provided by each sensor is equivalent to evidence. This method uses three types of sensors: lidar, camera, and millimeter-wave radar to detect and identify targets. Therefore, the mass functions m1, m2, and m3 represent the basic probability allocations of lidar, camera, and millimeter-wave radar for different target categories, respectively.

[0098] Therefore, it is necessary to perform basic probability assignment on the different target categories corresponding to each sensor in the recognition framework based on the confidence level of the categories identified by each sensor, to obtain m. i (A j );m i (A j ) represents the basic probability assignment of the i-th sensor to the j-th class of the matched target;

[0099] In one example, suppose that for the matching target corresponding to the current location, the confidence scores of the categories identified by each sensor are shown in Table 1 below:

[0100] Table 1

[0101] Assumption Camera millimeter-wave radar LiDAR Determined to be a pedestrian 0.86 0.50 0.75 It was identified as a bicycle. 0.13 0.25 0.20 Determined to be a motor vehicle 0.01 0.25 0.05

[0102] After the aforementioned steps, the camera has an 86%, 13%, and 1% probability of identifying a matching target as a pedestrian, bicycle, or motor vehicle, respectively; the millimeter-wave radar has a 50%, 25%, and 25% probability of identifying it as a pedestrian, bicycle, or motor vehicle, respectively; and the lidar has a 75%, 20%, and 5% probability of identifying it as a pedestrian, bicycle, or motor vehicle, respectively.

[0103] This allows us to construct a recognition framework Θ = {pedestrians, bicycles, motor vehicles}, and assign basic probabilities to each proposition based on the information in the table above. For example, m1(A1) represents the probability that the first sensor (camera) determines the current matching target to be in the first category (pedestrians) as 0.86, and so on.

[0104] Step S121: Obtain the recognition accuracy α of different sensors for different target categories in historical data. ij ;

[0105] Understandably, since different sensors operate independently, when a sensor malfunctions or encounters extreme scenarios, the target information obtained by different sensors may conflict to varying degrees. In such cases, directly fusing the target information obtained from different sensors using evidence theory can easily lead to incorrect results. Therefore, this method improves evidence theory by introducing the recognition accuracy of different sensors to correct the support levels of different pieces of evidence.

[0106] Specifically, based on the data annotation sets of each sensor, the recognition accuracy α of different sensors for different target categories is calculated using the following formula. ij :

[0107]

[0108] Where, α ij Let TP be the recognition accuracy of the i-th sensor for the j-th type of target. ij TN represents the number of targets of type j correctly identified by the i-th sensor. ij To determine the number of negative classes identified as negative, FP ij FN is the number of negative classes identified as positive classes. ij The number of positive classes identified as negative classes; when calculating the recognition accuracy of the j-th class target, all other classes are considered as negative classes.

[0109] For example, in one instance, α is obtained through the above calculation. 11 It is 0.6, α 12 It is 0.75, α 13 0.95; represents the camera's accuracy in recognizing pedestrians, bicycles, and vehicles, respectively; while α 21 It is 0.9, α 22 It is 0.65, α 23 It is 0.85; α 31 It is 0.8, α 32 It is 0.75, α 33 It is 0.9.

[0110] Step S122: The original evidence (basic probability assignment function value) in the recognition framework is corrected using the following formula based on the recognition accuracy of different sensors for different categories:

[0111] m′ i (A j )=α ij ·m i (A j )

[0112] Corresponding to the example above, the corrected m′ i (A j As shown in Table 2 below:

[0113] Assumption Camera millimeter-wave radar LiDAR Determined to be a pedestrian 0.86*0.6＝0.516 0.50*0.9＝0.45 0.75*0.8＝0.60 It was identified as a bicycle. 0.13*0.75＝0.0975 0.25*0.65＝0.1625 0.20*0.75＝0.15 Determined to be a motor vehicle 0.01*0.95＝0.0095 0.25*0.85＝0.2125 0.05*0.9＝0.045

[0114] Step S123, because in subsequent calculations, it is necessary to satisfy... and Therefore, it is necessary to normalize the corrected basic probability assignment function values ​​to obtain the processed target category probability data:

[0115]

[0116] Specifically, the mass function of each sensor was normalized, and the results are shown in Table 3 below:

[0117] Assumption Camera millimeter-wave radar LiDAR Determined to be a pedestrian 0.828 0.545 0.755 It was identified as a bicycle. 0.157 0.197 0.189 Determined to be a motor vehicle 0.015 0.258 0.056

[0118] Step S124: The processed target category probability data is fused using the Dempster synthesis rule to obtain the fused basic probability allocation function value for each category corresponding to each fused target.

[0119] It is understandable that for the case where there are n pieces of evidence, i.e., m1, m2, ..., m n 2 Θ Given n mutually independent basic probability assignments, the composition rule is:

[0120]

[0121]

[0122] Therefore, the basic probability allocation function value after fusion of each category can be calculated according to the above formula.

[0123] Let's illustrate this with an example from the steps above:

[0124] First, calculate the normalization constant k:

[0125] K=m″1(A1)*m″2(A1)*m″3(A1)+m″1(A2)*m″2(A2)*m″3(A2)+m″1(A3)*m″2(A3)*m″3(A3)

[0126] = 0.828*0.545*0.755 + 0.157*0.197*0.189 + 0.015*0.258*0.056

[0127] =0.3407 + 0.0058 + 0.0002

[0128] =0.3467

[0129] Then, the base probability assignment function value after fusing the three classifications is calculated:

[0130] m(A1)=m″1(A1)*m″2(A1)*m″3(A1) / K=0.3407 / 0.3467=0.9827

[0131] m(A2)=m″1(A2)*m″2(A2)*m″3(A2) / K=0.0058 / 0.3467=0.0167

[0132] m(A3)=m″1(A3)*m″2(A3)*m″3(A3) / K=0.0002 / 0.3467=0.0006

[0133] The fusion results are shown in Table 4 below:

[0134] Table 4

[0135] Assumption Fusion results Determined to be a pedestrian 0.9827 It was identified as a bicycle. 0.0167 Determined to be a motor vehicle 0.0006

[0136] Step S13: Based on the basic probability allocation function values ​​of each category after fusion corresponding to each fusion target, and combined with the velocity information of the fusion target, determine the final output category of each fusion target.

[0137] In a specific example, step S13 further includes:

[0138] Step S130: Sort the basic probability assignment function values ​​of each category of each matching target, and compare the basic probability assignment function values ​​of the top two.

[0139] Step S131: If the difference between the two is greater than or equal to a preset comparison threshold, then the category corresponding to the maximum basic probability assignment function value is taken as the classification of the target.

[0140] If the following equation is satisfied, that is, if there exists a maximum basic probability assignment function value, meaning the basic probability assignment function value of a certain category is the maximum value among all category outputs, and the basic probability assignment function values ​​of this category are sufficiently different from those of other categories, then the target is determined to be the category corresponding to the maximum basic probability assignment function value:

[0141]

[0142] Where ε is a hyperparameter (comparison threshold) set based on experience.

[0143] For example, assuming ε is 0.2 in one example, then in the example above, the maximum basic probability assignment function value (0.9827) that determines the target as a pedestrian satisfies the above relationship, so the classification corresponding to the matching target is determined to be a pedestrian.

[0144] If the difference between the two is less than the preset comparison threshold, the speed of the target is matched with both, and the category that matches the speed is used as the classification of the target.

[0145] Specifically, when the difference between the largest basic probability assignment function value and the basic probability assignment function values ​​of other categories is not significant enough, speed information is introduced to compare the speed of the target with the given speed ranges of different categories. The given speed ranges for pedestrians, bicycles, and motor vehicles are [1m / s, 2m / s], [4m / s, 6m / s], and [8m / s, 35m / s], respectively. If the category corresponding to the speed of the target is consistent with the category corresponding to one of the two largest basic probability assignment function values ​​of the target, then the target is determined to be the category that matches its speed.

[0146] For example, in one instance, the basic probability assignment function values ​​for a certain matching target, namely a motor vehicle and a bicycle, are 0.50 and 0.44, respectively; the difference between the two is less than 0.2; therefore, further judgment is needed based on the speed of the matching target; at this time, the speed is 10m / s, which is within the speed range of motor vehicles, so the category of this matching target is determined to be a motor vehicle.

[0147] Understandably, if the target category cannot be determined through the above decision-making steps, the matching target will be classified as an unknown object.

[0148] like Figure 3 The diagram shown illustrates a structural schematic of an embodiment of a multi-sensor target fusion system provided by the present invention. (In conjunction with...) Figures 4 to 7 As shown, in this embodiment, the multi-sensor target fusion system 1 is applied to an intelligent driving system, and it includes at least:

[0149] The target recognition result acquisition unit 10 is used to obtain the current target recognition results of multiple sensors in the intelligent driving system in real time. The current target recognition result of each sensor includes at least: the identification number, position, position covariance, category, category confidence, and speed information of each target.

[0150] The position fusion processing unit 11 is used to perform position fusion processing on the targets identified by the multiple sensors to obtain fused target information. The fused target information includes at least: the identifier, position, and velocity information of the fused target, as well as the category and category confidence level identified by each sensor.

[0151] The evidence fusion processing unit 12 is used to establish an identification framework based on the categories and category confidence levels identified by each sensor corresponding to the fusion target at each location, and to obtain the fusion basic probability allocation function values ​​of each category corresponding to each fusion target using evidence theory methods.

[0152] The output category decision unit 13 is used to determine the final output category of each fusion target based on the basic probability allocation function value of each category after fusion corresponding to each fusion target, and in combination with the speed information of the fusion target.

[0153] In a specific example, the target recognition result acquisition unit 10 further includes:

[0154] The data acquisition unit 100 is used to collect real-time data from various sensors in the intelligent driving system during driving. These various sensors include a camera, millimeter-wave radar, and lidar.

[0155] The target recognition unit 101 is used to identify the real-time data collected by each sensor by using the corresponding detection, recognition and tracking algorithms pre-set for each sensor, and to obtain the target recognition result corresponding to each sensor.

[0156] In a specific example, the location fusion processing unit 11 further includes:

[0157] The coordinate transformation unit 110 is used to uniformly transform the data collected by each sensor to the vehicle coordinate system.

[0158] The position fusion unit 111 is used to select one of the sensors as the master sensor after obtaining the target detection and tracking results of each sensor. When the tracking result of the master sensor is received for the first time, a fusion sequence is established based on the result of the master sensor as the first frame fusion result. After receiving the target tracking results of other sensors, the Hungarian matching algorithm is used to match the target of the previous frame fusion result, and the Kalman filtering algorithm is used to predict the previous frame fusion result. The prediction result is updated with the tracking results of the matched sensors.

[0159] The location fusion result acquisition unit 112 is used to associate each matched target with the data identified by each sensor to obtain a fusion result. The fusion result includes the identifier number, position, and velocity information of each fusion target, as well as the category and category confidence level identified by each sensor.

[0160] In a specific example, the evidence fusion processing unit 12 further includes:

[0161] The recognition framework establishment unit 120 is used to establish a recognition framework Θ = {pedestrian, bicycle, motor vehicle} based on the predefined classification attributes of each matched target; and to perform basic probability assignment m on the different target categories corresponding to each sensor in the recognition framework based on the confidence of the categories identified by each sensor. i (A j ), m i (A j ) represents the basic probability assignment of the i-th sensor to the j-th class of the matched target;

[0162] The recognition accuracy acquisition unit 121 is used to obtain the recognition accuracy α of different sensors for different target categories in historical data. ij ;

[0163] The evidence correction unit 122 is used to correct the original evidence (basic probability assignment function value) in the recognition framework using the recognition accuracy of different sensors for different categories according to the following formula:

[0164] m′ i (A j )=α ij ·m i (A j )

[0165] Normalization processing unit 123 is used to normalize the corrected basic probability allocation function value to obtain the processed target category probability data:

[0166]

[0167] The evidence fusion calculation unit 124 is used to perform fusion calculation on the processed target category probability data using the Dempster synthesis rule to obtain the fused basic probability allocation function value of each category corresponding to each fused target.

[0168] In a specific example, the recognition accuracy acquisition unit 121 further includes:

[0169] The computing unit is used to calculate the recognition accuracy α of different sensors for different target categories based on the data annotation sets of each sensor, using the following formula. ij :

[0170]

[0171] Where, α ij Let TP be the recognition accuracy of the i-th sensor for the j-th type of target. ij TN represents the number of targets of type j correctly identified by the i-th sensor. ij To determine the number of negative classes identified as negative, FP ijFN is the number of negative classes identified as positive classes. ij The number of positive classes identified as negative classes; when calculating the recognition accuracy of the j-th class target, all other classes are considered as negative classes.

[0172] In a specific example, the output category decision unit 13 further includes:

[0173] The sorting comparison unit 130 is used to sort the basic probability assignment function values ​​of each category of each target and compare the basic probability assignment function values ​​of the top two.

[0174] Decision unit 131 is used to compare the comparison result of the ranking comparison unit with a preset comparison threshold. If the difference between the two is greater than or equal to the preset comparison threshold, the category corresponding to the maximum basic probability allocation function value is taken as the classification of the target; otherwise, the speed of the target is matched with the two, and the category that matches the speed is taken as the classification of the target.

[0175] For more details, please refer to the above. Figure 1 and Figure 2 The description of that will not be repeated here.

[0176] In another aspect, the present invention provides a computer-readable medium having a computer program stored thereon, the computer program being executed by a processor to implement as follows: Figures 1 to 2 The steps of the sensor target fusion method described above. For more details, please refer to the preceding section. Figure 1 and Figure 2 The description of that will not be repeated here.

[0177] As another aspect of the present invention, a vehicle is also provided, wherein an intelligent driving system is provided, characterized in that the intelligent driving system has multiple types of sensors and integrates... Figures 3 to 7 The multi-sensor target fusion system is described above. For more details, please refer to the foregoing description. Figures 3 to 7 The description of that will not be repeated here.

[0178] Implementing the embodiments of the present invention has the following beneficial effects:

[0179] This invention provides a multi-sensor target fusion method, system, storage medium, and vehicle. By employing an improved evidence theory approach to fuse target category attributes, it fully utilizes the statistical information verifiable in the dataset from different sensor detection and recognition algorithms. Specifically, it introduces the recognition accuracy of different sensors to correct the support levels of different evidence, enabling more effective fusion of conflicting evidence and improving the accuracy of target detection and recognition.

[0180] Secondly, in the embodiments of the present invention, a rich set of decision rules are used in the decision-making process of category judgment, and a comprehensive judgment is made considering the speed of matching the target, which avoids the misjudgment problem caused by a single decision rule and further improves the accuracy of category judgment.

[0181] In addition, in this embodiment of the invention, the Kalman filter algorithm is used to realize the association and position fusion of targets between different sensors, which solves the problem of target flashing and jitter caused by inaccurate or discontinuous sensing distance of a single sensor.

[0182] This invention can be applied to autonomous driving systems, enabling effective and stable detection and identification of targets, which helps improve the safety, reliability, and comfort of autonomous driving.

[0183] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0185] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A multi-sensor target fusion method, applied in an intelligent driving system, characterized in that, It should include at least the following steps: The system obtains the current target recognition results from multiple sensors in the intelligent driving system in real time. The current target recognition result of each sensor includes at least: the target's corresponding identifier, location, category, category confidence level, and speed information. The targets identified by the various sensors are subjected to position fusion processing to obtain fused target information. The fused target information includes at least: the identifier, position, and velocity information of each fused target, as well as the category and category confidence level identified by each corresponding sensor. Based on the categories and category confidence levels identified by each sensor for each fusion target, an identification framework is established, and the basic probability allocation function values ​​of each category corresponding to each fusion target are obtained by using evidence theory methods. Based on the basic probability allocation function values ​​of each category after fusion corresponding to each fusion target, and combined with the velocity information of the fusion target, the final output category of each fusion target is determined.

2. The method as described in claim 1, characterized in that, The steps for obtaining the current target recognition results of multiple sensors in the intelligent driving system in real time include: During driving, the intelligent driving system collects real-time data from various sensors, including cameras, millimeter-wave radar, and lidar. By employing the corresponding detection, recognition, and tracking algorithms pre-set for each sensor, the real-time data collected by each sensor is identified to obtain the target recognition result corresponding to each sensor.

3. The method as described in claim 2, characterized in that, The step of performing position fusion processing on targets identified from multiple sensors to obtain fused target information further includes: Convert the data collected by each sensor to the vehicle's coordinate system; After obtaining the target detection and tracking results of each sensor, one of the sensors is selected as the master sensor. When the tracking result of the master sensor is received for the first time, a fusion sequence is established based on the result of the master sensor as the first frame fusion result. After the target tracking results of other sensors are received, the Hungarian matching algorithm is used to match the target of the previous frame fusion result, and the Kalman filtering algorithm is used to predict the previous frame fusion result. The prediction result is then updated with the tracking results of the matched sensors. Each matching target is associated with the data identified by each sensor to obtain the fusion result of the current frame. The fusion result includes the identifier, position, and velocity information of each fusion target, as well as the category and category confidence level identified by each sensor.

4. The method according to any one of claims 1 to 3, characterized in that, The step of establishing an identification framework based on the categories and category confidence levels identified by each sensor corresponding to the fusion target at each location, and obtaining the fusion-based basic probability allocation function values ​​for each category corresponding to each fusion target using evidence theory methods, further includes: A recognition framework Θ = {pedestrian, bicycle, motor vehicle} is established based on the predefined classification attributes of each matching target; basic probability assignment is performed on the different target categories corresponding to each sensor in the recognition framework based on the confidence scores of the categories identified by each sensor, obtaining m i (A j ), the m i (A j ) represents the basic probability assignment of the i-th sensor to the j-th class of the matched target; Obtain the recognition accuracy α of different sensors for different target categories from historical data. ij ; The basic probability allocation function value in the recognition framework is corrected using the recognition accuracy rate according to the following formula: m′ i (A j )=a ij ·m i (A j ) The corrected basic probability assignment function values ​​are normalized to obtain the processed target category probability data: The processed target category probability data are fused using the Dempster synthesis rule to obtain the fused basic probability allocation function value for each category corresponding to each fused target.

5. The method as described in claim 4, characterized in that, The accuracy α of different sensors in identifying different target categories in the obtained historical data. ij The steps further include: Based on the data annotation sets of each sensor, the recognition accuracy α of different sensors for different target categories is calculated using the following formula. ij : Where, α ij Let TP be the recognition accuracy of the i-th sensor for the j-th type of target. ij TN represents the number of targets of type j correctly identified by the i-th sensor. ij To determine the number of negative classes identified as negative, FP ij FN is the number of negative classes identified as positive classes. ij The number of positive classes identified as negative classes; when calculating the recognition accuracy of the j-th class target, all other classes are considered as negative classes.

6. The method as described in claim 5, characterized in that, The step of determining the final output category of each target based on the fused basic probability allocation function values ​​of each category corresponding to the target, and in combination with the target's velocity information, further includes: The basic probability assignment function values ​​for each category of each target are sorted, and the top two basic probability assignment function values ​​are compared. If the difference between the two is greater than or equal to a preset comparison threshold, then the category corresponding to the maximum basic probability assignment function value is taken as the classification of the target. If the difference between the two is less than the preset comparison threshold, the speed of the target is matched with the two, and the category that matches the speed is used as the classification of the target. Otherwise, the target will be classified as an unknown object.

7. A multi-sensor target fusion system, applied in an intelligent driving system, characterized in that, At least including: The target recognition result acquisition unit is used to obtain the current target recognition results of multiple sensors in the intelligent driving system in real time. The current target recognition result of each sensor includes at least: the identification number, position, category, category confidence, and speed information of each target. The position fusion processing unit is used to perform position fusion processing on the targets identified by the multiple sensors to obtain fused target information. Each fused target information includes at least: the identifier, position, and velocity information of the fused target, as well as the category and category confidence level identified by each corresponding sensor. The evidence fusion processing unit is used to establish an identification framework based on the categories and category confidence levels identified by each sensor corresponding to each fusion target, and to obtain the fusion basic probability allocation function values ​​of each category corresponding to each fusion target using evidence theory methods. The output category decision unit is used to determine the final output category of each fusion target based on the basic probability allocation function value of each category after fusion corresponding to each fusion target, and in combination with the speed information of the fusion target.

8. The system as described in claim 7, characterized in that, The target recognition result acquisition unit further includes: The data acquisition unit is used to collect real-time data from various sensors in the intelligent driving system during driving. These sensors include cameras, millimeter-wave radar, and lidar. The target recognition unit is used to identify the real-time data collected by each sensor using the corresponding detection, recognition and tracking algorithms pre-set for each sensor, and obtain the target recognition result corresponding to each sensor.

9. The system as described in claim 8, characterized in that, The location fusion processing unit further includes: The coordinate transformation unit is used to uniformly transform the data collected by various sensors to the vehicle's coordinate system. The position fusion unit is used to select one of the sensors as the master sensor after obtaining the target detection and tracking results of each sensor. When the tracking result of the master sensor is received for the first time, a fusion sequence is established based on the result of the master sensor as the first frame fusion result. After receiving the target tracking results of other sensors, the Hungarian matching algorithm is used to match the target of the previous frame fusion result, and the Kalman filtering algorithm is used to predict the previous frame fusion result. The prediction result is updated with the tracking results of the matched sensors. The location fusion result acquisition unit is used to associate each matched target with the data identified by each sensor to obtain the fusion result. The fusion result includes the identifier, position, and velocity information of the fusion target corresponding to each fusion target, as well as the category and category confidence level identified by each sensor.

10. The system according to any one of claims 7 to 9, characterized in that, The evidence fusion processing unit further includes: The recognition framework building unit is used to establish a recognition framework Θ = {pedestrian, bicycle, motor vehicle} based on the predefined classification attributes of each matched target; and to perform basic probability assignment on the different target categories corresponding to each sensor in the recognition framework based on the confidence of the categories identified by each sensor, thereby obtaining m. i (A j ), the m i (A j ) represents the basic probability assignment of the i-th sensor to the j-th class of the matched target; The recognition accuracy acquisition unit is used to obtain the recognition accuracy α of different sensors for different target categories in historical data. ij ; The evidence correction unit is used to correct the value of the basic probability allocation function in the recognition framework using the recognition accuracy according to the following formula: m′ i (A j )=a ij ·m i (A j ) The normalization unit is used to normalize the corrected basic probability assignment function values ​​to obtain the processed target category probability data. The evidence fusion calculation unit is used to perform fusion calculation on the processed target category probability data using the Dempster synthesis rule to obtain the fused basic probability allocation function value of each category corresponding to each fused target.

11. The system as claimed in claim 10, characterized in that, The recognition accuracy acquisition unit further includes: The computing unit is used to calculate the recognition accuracy α of different sensors for different target categories based on the data annotation sets of each sensor using the following formula. ij : Where, α ij Let TP be the recognition accuracy of the i-th sensor for the j-th type of target. ij TN represents the number of targets of type j correctly identified by the i-th sensor. ij To determine the number of negative classes identified as negative, FP ij FN is the number of negative classes identified as positive classes. ij The number of positive classes identified as negative classes; when calculating the recognition accuracy of the j-th class target, all other classes are considered as negative classes.

12. The system as claimed in claim 11, characterized in that, The output category decision unit further includes: The sorting and comparison unit is used to sort the basic probability assignment function values ​​of each category for each target and compare the top two basic probability assignment function values. The decision unit is used to compare the comparison result of the ranking comparison unit with a preset comparison threshold. If the difference between the two is greater than or equal to the preset comparison threshold, the category corresponding to the maximum basic probability assignment function value is taken as the classification of the target; otherwise, the speed of the target is matched with the two, and the category that matches the speed is taken as the classification of the target.

13. A computer-readable medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the multi-sensor target fusion method as described in any one of claims 1-6.

14. A vehicle equipped with an intelligent driving system, characterized in that, The intelligent driving system has multiple types of sensors and integrates a multi-sensor target fusion system as described in any one of claims 7 to 12.

Images