Sensor data fusion method and system

Abstract

The present invention is applicable to the field of data fusion technology and provides a sensor data fusion method and system. The method includes the following steps: calculating the hardware confidence Ch, semantic confidence Cs, and environmental confidence Ce of each sensor, wherein the hardware confidence is determined based on the signal-to-noise ratio, time decay factor, and frequency domain characteristics, and the semantic confidence is determined based on target detection confidence and target stability; determining the comprehensive confidence C of each sensor for different dimensional features; extracting the dimensional features of each sensor data, retrieving the comprehensive confidence of each sensor on the corresponding dimensional features, and calculating the fusion weight of each sensor on the corresponding dimensional features; and fusing the dimensional features based on the fusion weight of each sensor to obtain sensor fusion data. The fusion weights in the present invention are dynamically adjusted according to actual conditions to maximize the advantages of each sensor and significantly improve the accuracy and robustness of target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and in particular to a sensor data fusion method and system. Background Art

[0002] In modern target detection applications such as drone tracking and security monitoring, a single sensor often fails to meet the demands for precise detection in complex environments. Multiple sensors, such as cameras, radar, and lidar, play an irreplaceable role in target detection due to their unique physical properties and operating principles. Multi-sensor data fusion technology has emerged to improve the accuracy, robustness, and reliability of target detection by integrating data from diverse sensors. However, existing multi-sensor data fusion methods still face numerous challenges in handling the uncertainty and inconsistency of sensor data. Different sensors vary in data quality, reliability, and their ability to perceive the same target. Effectively evaluating and leveraging these differences is crucial for improving fusion performance. Currently, fixed weights or simple heuristic rules are often used in the fusion process, which are difficult to adapt to complex and changing environments and cannot dynamically adjust weights. Therefore, there is a need for a sensor data fusion method and system to address these challenges. Summary of the Invention

[0003] In view of the deficiencies in the prior art, the present invention aims to provide a sensor data fusion method and system to solve the problems existing in the above-mentioned background technology.

[0004] The present invention is implemented as follows: a sensor data fusion method, the method comprising the following steps:

[0005] Calculate the hardware confidence Ch, semantic confidence Cs, and environmental confidence Ce of each sensor. The hardware confidence is determined based on the signal-to-noise ratio, time decay factor, and frequency domain characteristics, and the semantic confidence is determined based on target detection confidence and target stability.

[0006] Determine the comprehensive confidence C of each sensor for different dimensional features, Cik = wik1×Ch+wik2×Cs+wik3×Ce, Cik represents the comprehensive confidence of sensor k when monitoring feature i, wik1, wik2 and wik3 are the hardware weight coefficient, semantic weight coefficient and environmental weight coefficient of sensor k when monitoring feature i respectively;

[0007] Extract the dimensional features of each sensor data, retrieve the comprehensive confidence of each sensor on the corresponding dimensional features, and calculate the fusion weight of each sensor on the corresponding dimensional features;

[0008] Based on the fusion weight of each sensor, the features of each dimension are fused to obtain sensor fusion data.

[0009] As a further solution of the present invention: the step of calculating the hardware confidence Ch, semantic confidence Cs and environment confidence Ce of each sensor specifically includes:

[0010] Calculate the hardware confidence Ch, Ch= ,in, represents the instantaneous signal-to-noise ratio of the sensor, represents the nominal maximum signal-to-noise ratio of the sensor, is the time interval of the last valid data, is the time constant, is the standard deviation of the signal-to-noise ratio, is the mean signal-to-noise ratio, is a definite coefficient, is the signal energy concentration in the frequency domain;

[0011] Calculate the semantic confidence Cs, Cs=α conf+β IoU+γ , α, β and γ are weight coefficients, conf is the target detection confidence, IoU is the intersection over union of the detection box and the predicted trajectory, is the size variance of the detection frame;

[0012] Collect environmental information and input it into the environmental confidence calculation model corresponding to each sensor to obtain the environmental confidence Ce.

[0013] As a further solution of the present invention: the IoU is calculated by the detection box and the predicted trajectory box, and the specific steps include:

[0014] Determine the center coordinates, width, height, and rotation angle of the detection frame, and determine the center coordinates, width, height, and rotation angle of the prediction trajectory frame;

[0015] Determine the four corner coordinates of the detection frame and the predicted trajectory frame based on the center coordinates, width value, height value and rotation angle, and obtain two polygons;

[0016] Calculate the intersection area and union area of ​​two polygons to get IoU, IoU = intersection area / union area.

[0017] As a further solution of the present invention: the predicted trajectory frame is obtained based on the data of the historical detection frame, and the specific steps include:

[0018] Retrieve the historical center coordinates, historical width, historical height, and historical rotation angle of nearly N detection boxes of the same target;

[0019] Based on the historical data of the last N detection frames, the predicted center coordinates, predicted width, predicted height, and predicted rotation angle are determined to obtain the predicted trajectory frame.

[0020] As a further solution of the present invention, the step of retrieving the comprehensive confidence of each sensor on the corresponding dimensional features and calculating the fusion weight of each sensor on the corresponding dimensional features specifically includes:

[0021] The comprehensive confidence of each sensor on each dimensional feature is retrieved in sequence, and several comprehensive confidences are normalized to obtain the fusion weight of each sensor on the dimensional feature;

[0022] Retrieving feature data of the dimensional feature and inputting the feature data into a weight correction library, wherein the weight correction library includes all dimensional features, each dimensional feature corresponds to a number of data ranges, each data range corresponds to a number of correction coefficients, and each correction coefficient corresponds to a sensor;

[0023] Output a number of correction coefficients, correct the fusion weights based on the correction coefficients, and normalize all the corrected fusion weights.

[0024] Another object of the present invention is to provide a sensor data fusion system, the system comprising:

[0025] A multi-dimensional confidence calculation module is used to calculate the hardware confidence Ch, semantic confidence Cs and environmental confidence Ce of each sensor. The hardware confidence is determined based on the signal-to-noise ratio, time decay factor and frequency domain characteristics, and the semantic confidence is determined based on the target detection confidence and target stability.

[0026] The comprehensive confidence determination module is used to determine the comprehensive confidence C of each sensor for different dimensional features, Cik = wik1×Ch+wik2×Cs+wik3×Ce, where Cik represents the comprehensive confidence of sensor k when monitoring feature i, and wik1, wik2, and wik3 are the hardware weight coefficient, semantic weight coefficient, and environmental weight coefficient of sensor k when monitoring feature i, respectively.

[0027] The fusion weight determination module is used to extract the dimensional features of each sensor data, retrieve the comprehensive confidence of each sensor on the corresponding dimensional features, and calculate the fusion weight of each sensor on the corresponding dimensional features;

[0028] The fusion data determination module is used to fuse the features of each dimension based on the fusion weight of each sensor to obtain sensor fusion data.

[0029] As a further solution of the present invention: the multidimensional confidence calculation module includes:

[0030] Hardware confidence unit, used to calculate hardware confidence Ch, Ch= ,in, represents the instantaneous signal-to-noise ratio of the sensor, represents the nominal maximum signal-to-noise ratio of the sensor, is the time interval of the last valid data, is the time constant, is the standard deviation of the signal-to-noise ratio, is the mean signal-to-noise ratio, is a definite coefficient, is the signal energy concentration in the frequency domain;

[0031] Semantic confidence unit, used to calculate semantic confidence Cs, Cs = α conf+β IoU+γ , α, β and γ are weight coefficients, conf is the target detection confidence, IoU is the intersection over union of the detection box and the predicted trajectory, is the size variance of the detection frame;

[0032] The environmental confidence unit is used to collect environmental information and input the environmental information into the environmental confidence calculation model corresponding to each sensor to obtain the environmental confidence Ce.

[0033] As a further solution of the present invention: the semantic confidence unit includes:

[0034] The frame information retrieval subunit is used to determine the center coordinates, width, height, and rotation angle of the detection frame, and the center coordinates, width, height, and rotation angle of the predicted trajectory frame;

[0035] The polygon determination subunit is used to determine the four corner coordinates of the detection frame and the predicted trajectory frame based on the center coordinates, width value, height value and rotation angle to obtain two polygons;

[0036] The IoU calculation subunit is used to calculate the intersection area and union area of ​​two polygons to obtain IoU, IoU = intersection area / union area.

[0037] As a further solution of the present invention: the semantic confidence unit further includes:

[0038] The historical data retrieval subunit is used to retrieve the historical center coordinates, historical width, historical height, and historical rotation angle of nearly N detection boxes of the same target;

[0039] The predicted trajectory frame subunit is used to determine the predicted center coordinates, predicted width value, predicted height value and predicted rotation angle based on the historical data of the last N detection frames to obtain the predicted trajectory frame.

[0040] As a further solution of the present invention: the fusion weight determination module includes:

[0041] A preliminary weight determination unit is used to sequentially retrieve the comprehensive confidence of each sensor on each dimensional feature, normalize the multiple comprehensive confidences, and obtain the fusion weight of each sensor on the dimensional feature;

[0042] A correction coefficient determination unit is used to retrieve feature data of the dimensional feature and input the feature data into a weight correction library, wherein the weight correction library includes all dimensional features, each dimensional feature corresponds to a number of data ranges, each data range corresponds to a number of correction coefficients, and each correction coefficient corresponds to a sensor;

[0043] The fusion weight correction unit is used to output a number of correction coefficients, correct the fusion weights based on the correction coefficients, and normalize all the corrected fusion weights.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] This method comprehensively considers each sensor's hardware confidence Ch, semantic confidence Cs, and environmental confidence Ce, and then determines the comprehensive confidence C of each sensor for different dimensional features. The same sensor has different comprehensive confidence levels when monitoring features in different dimensions, resulting in greater targeting and flexibility, maximizing the advantages of each sensor. The corresponding comprehensive confidence is retrieved based on the dimensional features of each sensor data, and the fusion weight of each sensor on the corresponding dimensional features is calculated. The fusion weight is determined by the comprehensive confidence and dynamically adjusted based on actual conditions. This results in more accurate and reliable sensor fusion data, significantly improving the precision and robustness of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flowchart of a sensor data fusion method.

[0047] Figure 2 A flowchart for calculating the multi-dimensional confidence of sensors in a sensor data fusion method.

[0048] Figure 3 Flowchart for calculating IoU in a sensor data fusion method.

[0049] Figure 4 Flowchart for determining predicted trajectory boxes in a sensor data fusion method.

[0050] Figure 5 The flowchart of calculating the fusion weight of sensors on dimensional features in a sensor data fusion method.

[0051] Figure 6 A structural diagram of a sensor data fusion system. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0054] like Figure 1 As shown, an embodiment of the present invention provides a sensor data fusion method, which includes the following steps:

[0055] S100, calculating the hardware confidence Ch, semantic confidence Cs, and environmental confidence Ce of each sensor, wherein the hardware confidence is determined based on the signal-to-noise ratio, the time decay factor, and the frequency domain characteristics, and the semantic confidence is determined based on the target detection confidence and the target stability;

[0056] S200, respectively determine the comprehensive confidence C of each sensor for different dimensional features, Cik = wik1 × Ch + wik2 × Cs + wik3 × Ce, Cik represents the comprehensive confidence of sensor k when monitoring feature i, wik1, wik2 and wik3 are the hardware weight coefficient, semantic weight coefficient and environmental weight coefficient of sensor k when monitoring feature i;

[0057] S300: extracting features of each dimension of each sensor data, retrieving the comprehensive confidence of each sensor on the corresponding dimension feature, and calculating the fusion weight of each sensor on the corresponding dimension feature;

[0058] S400: Perform feature fusion of each dimension based on the fusion weight of each sensor to obtain sensor fusion data.

[0059] It should be noted that existing multi-sensor data fusion methods still face numerous challenges when dealing with the uncertainty and inconsistency of sensor data. Different sensors vary in data quality, reliability, and their ability to perceive the same target. Effectively evaluating and leveraging these differences is key to improving fusion effectiveness. Currently, fixed weights or simple heuristic rules are often used in fusion processes. These methods struggle to adapt to complex and changing environments, and they cannot dynamically adjust weights, making it difficult to achieve optimal fusion results. The present invention aims to address these issues.

[0060] In an embodiment of the present invention, multiple sensors, such as cameras, ultrasonic radars, lidars, etc., are involved. After the sensor data is collected, the hardware confidence Ch, semantic confidence Cs and environmental confidence Ce of each sensor are calculated. The hardware confidence is determined based on the signal-to-noise ratio, time attenuation factor and frequency domain characteristics, the semantic confidence is determined based on the target detection confidence and target stability, and the environmental confidence is determined based on environmental factors (such as weather, lighting, etc.). In this way, the quality of sensor data can be comprehensively evaluated, the limitations of single-dimensional evaluation can be reduced, and the accuracy of data fusion can be significantly improved. The comprehensive confidence C for each sensor's different dimensional features is then determined: Cik = wik1 × Ch + wik2 × Cs + wik3 × Ce, where Cik represents the comprehensive confidence of sensor k when monitoring feature i. wik1, wik2, and wik3 are the hardware weighting coefficient, semantic weighting coefficient, and environmental weighting coefficient for sensor k when monitoring feature i, respectively. wik1, wik2, and wik3 are all pre-set fixed values. This allows the same sensor to have different comprehensive confidence levels when monitoring features in different dimensionalities, providing greater targeting and flexibility, maximizing the advantages of each sensor. The system then extracts the dimensional features of each sensor data (e.g., appearance, position, and velocity), retrieves the comprehensive confidence of each sensor for the corresponding dimensional features, and calculates the fusion weight for each sensor on the corresponding dimensional features. The fusion weight is determined by the comprehensive confidence and dynamically adjusted based on actual conditions, resulting in more accurate and reliable sensor fusion data. It significantly improves the accuracy and robustness of target detection and enhances the system's adaptability to environmental changes, providing strong technical support for applications such as drone target tracking, security monitoring, and intelligent driving.

[0061] like Figure 2 As shown, as a preferred embodiment of the present invention, the step of calculating the hardware confidence Ch, semantic confidence Cs and environment confidence Ce of each sensor specifically includes:

[0062] S101, calculate the hardware confidence Ch, Ch= ;

[0063] S102, calculate the semantic confidence Cs, Cs = α conf+β IoU+γ ;

[0064] S103 , collecting environmental information, and inputting the environmental information into the environmental confidence calculation model corresponding to each sensor to obtain the environmental confidence Ce.

[0065] In the embodiment of the present invention, the hardware confidence Ch is first calculated, Ch= ,in, represents the instantaneous signal-to-noise ratio of the sensor, represents the nominal maximum signal-to-noise ratio of the sensor, is the time interval of the last valid data, is the time constant, for example Take the typical value 0.5s, is the standard deviation of the signal-to-noise ratio, which is used to reflect stability. is the mean signal-to-noise ratio, is a definite coefficient, The signal frequency domain energy concentration, FT is between 0 and 1; the hardware confidence not only considers the signal-to-noise ratio, but also introduces the time attenuation factor and frequency domain characteristics to solve the problem of pulse interference. Then calculate the semantic confidence Cs, Cs = α conf+β IoU+γ , α, β, and γ are weight coefficients, α+β+γ=1, the specific value needs to be determined in advance, conf is the target detection confidence, the target detection confidence is the target detection model (such as YOLO, SSD, Faster R-CNN) for the probability of the target in the detection box, obtained directly by the detection model. IoU is the intersection over union ratio of the detection box and the predicted trajectory, To measure the size variance of the detection frame, we introduce a target stability indicator (size variance) to prevent interference from flickering objects, enhancing reliability. Finally, we collect environmental information and input it into the environmental confidence calculation model corresponding to each sensor to obtain the environmental confidence Ce. Each sensor has its own pre-built environmental confidence calculation model, which takes into account weather type, pressure gradient, relative humidity, temperature gradient, and other factors.

[0066] like Figure 3 As shown in FIG, as a preferred embodiment of the present invention, the IoU is calculated by the detection frame and the predicted trajectory frame, and the specific steps include:

[0067] S1021, determining the center coordinates, width, height, and rotation angle of the detection frame, and determining the center coordinates, width, height, and rotation angle of the prediction trajectory frame;

[0068] S1022, determining the four corner coordinates of the detection frame and the predicted trajectory frame based on the center coordinates, width value, height value, and rotation angle to obtain two polygons;

[0069] S1023: Calculate the intersection area and union area of ​​the two polygons to obtain IoU, where IoU = intersection area / union area.

[0070] In an embodiment of the present invention, to determine the intersection over union (IoU) of the sensor detection frame and the predicted trajectory, it is first necessary to extract the center coordinates, width, height, and rotation angle of the detection frame, and then determine the center coordinates, width, height, and rotation angle of the predicted trajectory frame. A coordinate transformation is then performed to determine the four corner coordinates of the detection frame and the predicted trajectory frame based on the center coordinates, width, height, and rotation angle, resulting in two polygons. The Sutherland-Hodgman algorithm is then used to calculate the intersection area of ​​the two polygons and determine the union area. The union area is calculated as: width × height of the detection frame + width × height of the predicted trajectory frame - intersection area. Finally, the IoU can be calculated.

[0071] Furthermore, rotation compensation is performed to make the result more accurate. The IoU after rotation compensation = IoU×exp(- ),when When the absolute value of is greater than π / 6, the rotation compensation calculation is performed. is the rotation angle of the detection frame, To predict the rotation angle of the trajectory frame, is the angle penalty factor, for example =π / 12.

[0072] like Figure 4 As shown, as a preferred embodiment of the present invention, the predicted trajectory frame is obtained based on the data of the historical detection frame, and the specific steps include:

[0073] S1024, retrieve the historical center coordinates, historical width values, historical height values, and historical rotation angles of nearly N detection boxes of the same target;

[0074] S1025: Determine the predicted center coordinates, predicted width, predicted height, and predicted rotation angle based on the historical data of the last N detection frames to obtain a predicted trajectory frame.

[0075] In an embodiment of the present invention, in order to obtain a predicted trajectory frame, the historical center coordinates, historical width values, historical height values, and historical rotation angles of several recent detection frames of the same detection target are retrieved. Then, according to the usage scenario, a linear motion model or a nonlinear model (such as an extended Kalman filter EKF) is selected to determine the predicted center coordinates, predicted width value, predicted height value, and predicted rotation angle, and finally the predicted trajectory frame is obtained.

[0076] like Figure 5 As shown, as a preferred embodiment of the present invention, the step of retrieving the comprehensive confidence of each sensor on the corresponding dimensional feature and calculating the fusion weight of each sensor on the corresponding dimensional feature specifically includes:

[0077] S301, sequentially retrieve the comprehensive confidence of each sensor on each dimensional feature, normalize the multiple comprehensive confidences, and obtain the fusion weight of each sensor on the dimensional feature;

[0078] S302: Retrieve feature data of the dimensional feature and input the feature data into a weight correction library. The weight correction library includes all dimensional features, each dimensional feature corresponds to a number of data ranges, each data range corresponds to a number of correction coefficients, and each correction coefficient corresponds to a sensor.

[0079] S303: Output a plurality of correction coefficients, correct the fusion weights based on the correction coefficients, and normalize all the corrected fusion weights.

[0080] In an embodiment of the present invention, the fusion weight of a dimensional feature is determined each time. Each time the fusion weight is determined, the comprehensive confidence of each sensor on the dimensional feature is retrieved, and several comprehensive confidences are normalized to obtain the fusion weight of each sensor on the dimensional feature. For example, if there are three sensors with comprehensive confidences of a, b, and c on the dimensional feature, the corresponding fusion weights are: a / (a+b+c), b / (a+b+c), and c / (a+b+c). Then the feature data of the dimensional feature is retrieved and input into the weight correction library. The weight correction library needs to be constructed in advance. The weight correction library includes all dimensional features, each dimensional feature corresponds to several data ranges, each data range corresponds to several correction coefficients, and each correction coefficient corresponds to a sensor. By determining the data range based on the feature data, several correction coefficients can be automatically output, and the fusion weight can be corrected based on the correction coefficient. When correcting, the correction coefficient can be directly multiplied by the corresponding fusion weight. Finally, all the corrected fusion weights need to be normalized.

[0081] like Figure 6 As shown, an embodiment of the present invention further provides a sensor data fusion system, the system comprising:

[0082] A multi-dimensional confidence calculation module 100 is used to calculate the hardware confidence Ch, semantic confidence Cs, and environmental confidence Ce of each sensor. The hardware confidence is determined based on the signal-to-noise ratio, time decay factor, and frequency domain characteristics, and the semantic confidence is determined based on target detection confidence and target stability.

[0083] The comprehensive confidence determination module 200 is used to determine the comprehensive confidence C of each sensor for different dimensional features, Cik = wik1×Ch+wik2×Cs+wik3×Ce, where Cik represents the comprehensive confidence of sensor k when monitoring feature i, and wik1, wik2 and wik3 are the hardware weight coefficient, semantic weight coefficient and environmental weight coefficient of sensor k when monitoring feature i, respectively;

[0084] The fusion weight determination module 300 is used to extract the dimensional features of each sensor data, retrieve the comprehensive confidence of each sensor on the corresponding dimensional features, and calculate the fusion weight of each sensor on the corresponding dimensional features;

[0085] The fusion data determination module 400 is used to perform feature fusion of each dimension based on the fusion weight of each sensor to obtain sensor fusion data.

[0086] As a preferred embodiment of the present invention, the multi-dimensional confidence calculation module 100 includes:

[0087] Hardware confidence unit, used to calculate hardware confidence Ch, Ch= ,in, represents the instantaneous signal-to-noise ratio of the sensor, represents the nominal maximum signal-to-noise ratio of the sensor, is the time interval of the last valid data, is the time constant, is the standard deviation of the signal-to-noise ratio, is the mean signal-to-noise ratio, is a definite coefficient, is the signal energy concentration in the frequency domain;

[0088] Semantic confidence unit, used to calculate semantic confidence Cs, Cs = α conf+β IoU+γ , α, β and γ are weight coefficients, conf is the target detection confidence, IoU is the intersection over union of the detection box and the predicted trajectory, is the size variance of the detection frame;

[0089] The environmental confidence unit is used to collect environmental information and input the environmental information into the environmental confidence calculation model corresponding to each sensor to obtain the environmental confidence Ce.

[0090] As a preferred embodiment of the present invention, the semantic confidence unit includes:

[0091] The frame information retrieval subunit is used to determine the center coordinates, width, height, and rotation angle of the detection frame, and the center coordinates, width, height, and rotation angle of the predicted trajectory frame;

[0092] The polygon determination subunit is used to determine the four corner coordinates of the detection frame and the predicted trajectory frame based on the center coordinates, width value, height value and rotation angle to obtain two polygons;

[0093] The IoU calculation subunit is used to calculate the intersection area and union area of ​​two polygons to obtain IoU, IoU = intersection area / union area.

[0094] As a preferred embodiment of the present invention, the semantic confidence unit further includes:

[0095] The historical data retrieval subunit is used to retrieve the historical center coordinates, historical width, historical height, and historical rotation angle of nearly N detection boxes of the same target;

[0096] The predicted trajectory frame subunit is used to determine the predicted center coordinates, predicted width value, predicted height value and predicted rotation angle based on the historical data of the last N detection frames to obtain the predicted trajectory frame.

[0097] As a preferred embodiment of the present invention, the fusion weight determination module 300 includes:

[0098] A preliminary weight determination unit is used to sequentially retrieve the comprehensive confidence of each sensor on each dimensional feature, normalize the multiple comprehensive confidences, and obtain the fusion weight of each sensor on the dimensional feature;

[0099] A correction coefficient determination unit is used to retrieve feature data of the dimensional feature and input the feature data into a weight correction library, wherein the weight correction library includes all dimensional features, each dimensional feature corresponds to a number of data ranges, each data range corresponds to a number of correction coefficients, and each correction coefficient corresponds to a sensor;

[0100] The fusion weight correction unit is used to output a number of correction coefficients, correct the fusion weights based on the correction coefficients, and normalize all the corrected fusion weights.

[0101] The above is only a detailed description of the preferred embodiments of the present invention, which is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0102] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0103] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0104] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the disclosure in the specification and examples. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A sensor data fusion method, characterized in that: The method comprises the following steps: Calculate the hardware confidence Ch, semantic confidence Cs, and environmental confidence Ce of each sensor. The hardware confidence is determined based on the signal-to-noise ratio, time decay factor, and frequency domain characteristics, and the semantic confidence is determined based on target detection confidence and target stability. Determine the comprehensive confidence C of each sensor for different dimensional features, Cik = wik1×Ch+wik2×Cs+wik3×Ce, Cik represents the comprehensive confidence of sensor k when monitoring feature i, wik1, wik2 and wik3 are the hardware weight coefficient, semantic weight coefficient and environmental weight coefficient of sensor k when monitoring feature i respectively; Extract the dimensional features of each sensor data, retrieve the comprehensive confidence of each sensor on the corresponding dimensional features, and calculate the fusion weight of each sensor on the corresponding dimensional features; Based on the fusion weight of each sensor, the features of each dimension are fused to obtain sensor fusion data; The step of calculating the hardware confidence Ch, semantic confidence Cs and environment confidence Ce of each sensor specifically includes: calculating the hardware confidence Ch, Ch= ,in, represents the instantaneous signal-to-noise ratio of the sensor, represents the nominal maximum signal-to-noise ratio of the sensor, is the time interval of the last valid data, is the time constant, is the standard deviation of the signal-to-noise ratio, is the mean signal-to-noise ratio, is a definite coefficient, is the signal frequency domain energy concentration; calculate the semantic confidence Cs, Cs=α conf+β IoU+γ , α, β and γ are weight coefficients, conf is the target detection confidence, IoU is the intersection over union of the detection box and the predicted trajectory, is the size variance of the detection frame; collect environmental information, input the environmental information into the environmental confidence calculation model corresponding to each sensor, and obtain the environmental confidence Ce.

2. The sensor data fusion method according to claim 1, characterized in that: The IoU is calculated by the detection frame and the predicted trajectory frame. The specific steps include: Determine the center coordinates, width, height, and rotation angle of the detection frame, and determine the center coordinates, width, height, and rotation angle of the prediction trajectory frame; Determine the four corner coordinates of the detection frame and the predicted trajectory frame based on the center coordinates, width value, height value and rotation angle, and obtain two polygons; Calculate the intersection area and union area of ​​two polygons to get IoU, IoU = intersection area / union area.

3. The sensor data fusion method according to claim 2, characterized in that: The predicted trajectory frame is obtained based on the data of the historical detection frame, and the specific steps include: Retrieve the historical center coordinates, historical width, historical height, and historical rotation angles of the latest N detection boxes of the same target; Based on the historical data of the most recent N detection frames, the predicted center coordinates, predicted width, predicted height, and predicted rotation angle are determined to obtain the predicted trajectory frame.

4. The sensor data fusion method according to claim 1, characterized in that: The step of retrieving the comprehensive confidence of each sensor on the corresponding dimensional feature and calculating the fusion weight of each sensor on the corresponding dimensional feature specifically includes: The comprehensive confidence of each sensor on each dimensional feature is retrieved in sequence, and several comprehensive confidences are normalized to obtain the fusion weight of each sensor on the dimensional feature; Retrieving feature data of the dimensional feature and inputting the feature data into a weight correction library, wherein the weight correction library includes all dimensional features, each dimensional feature corresponds to a number of data ranges, each data range corresponds to a number of correction coefficients, and each correction coefficient corresponds to a sensor; Output a number of correction coefficients, correct the fusion weights based on the correction coefficients, and normalize all the corrected fusion weights.

5. A sensor data fusion system, characterized in that: The system comprises: A multi-dimensional confidence calculation module is used to calculate the hardware confidence Ch, semantic confidence Cs and environmental confidence Ce of each sensor. The hardware confidence is determined based on the signal-to-noise ratio, time decay factor and frequency domain characteristics, and the semantic confidence is determined based on the target detection confidence and target stability. The comprehensive confidence determination module is used to determine the comprehensive confidence C of each sensor for different dimensional features, Cik = wik1×Ch+wik2×Cs+wik3×Ce, where Cik represents the comprehensive confidence of sensor k when monitoring feature i, and wik1, wik2, and wik3 are the hardware weight coefficient, semantic weight coefficient, and environmental weight coefficient of sensor k when monitoring feature i, respectively. The fusion weight determination module is used to extract the dimensional features of each sensor data, retrieve the comprehensive confidence of each sensor on the corresponding dimensional features, and calculate the fusion weight of each sensor on the corresponding dimensional features; The fusion data determination module is used to fuse the features of each dimension based on the fusion weight of each sensor to obtain sensor fusion data; The multi-dimensional confidence calculation module includes: a hardware confidence unit for calculating hardware confidence Ch, Ch = ,in, represents the instantaneous signal-to-noise ratio of the sensor, represents the nominal maximum signal-to-noise ratio of the sensor, is the time interval of the last valid data, is the time constant, is the standard deviation of the signal-to-noise ratio, is the mean signal-to-noise ratio, is a definite coefficient, is the signal frequency domain energy concentration; semantic confidence unit, used to calculate the semantic confidence Cs, Cs=α conf+β IoU+γ , α, β and γ are weight coefficients, conf is the target detection confidence, IoU is the intersection over union of the detection box and the predicted trajectory, is the size variance of the detection frame; the environment confidence unit is used to collect environmental information and input the environmental information into the environment confidence calculation model corresponding to each sensor to obtain the environment confidence Ce.

6. The sensor data fusion system according to claim 5, characterized in that: The semantic confidence unit includes: The frame information retrieval subunit is used to determine the center coordinates, width, height, and rotation angle of the detection frame, and the center coordinates, width, height, and rotation angle of the predicted trajectory frame; The polygon determination subunit is used to determine the four corner coordinates of the detection frame and the predicted trajectory frame based on the center coordinates, width value, height value and rotation angle to obtain two polygons; The IoU calculation subunit is used to calculate the intersection area and union area of ​​two polygons to obtain IoU, IoU = intersection area / union area.

7. The sensor data fusion system according to claim 6, characterized in that: The semantic confidence unit also includes: The historical data retrieval subunit is used to retrieve the historical center coordinates, historical width, historical height, and historical rotation angle of the latest N detection boxes of the same target; The predicted trajectory frame subunit is used to determine the predicted center coordinates, predicted width value, predicted height value and predicted rotation angle based on the historical data of the most recent N detection frames to obtain the predicted trajectory frame.

8. The sensor data fusion system according to claim 5, characterized in that: The fusion weight determination module includes: A preliminary weight determination unit is used to sequentially retrieve the comprehensive confidence of each sensor on each dimensional feature, normalize the multiple comprehensive confidences, and obtain the fusion weight of each sensor on the dimensional feature; A correction coefficient determination unit is used to retrieve feature data of the dimensional feature and input the feature data into a weight correction library, wherein the weight correction library includes all dimensional features, each dimensional feature corresponds to a number of data ranges, each data range corresponds to a number of correction coefficients, and each correction coefficient corresponds to a sensor; The fusion weight correction unit is used to output a number of correction coefficients, correct the fusion weights based on the correction coefficients, and normalize all the corrected fusion weights.

Images