Data fusion method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a kind of data fusion method, device, electronic equipment and storage medium, involve data processing technical field.The method is by obtaining the original data uploaded by multiple sensors;Through data quality online evaluation system, the quality of the original data uploaded by multiple sensors is evaluated respectively, obtain multiple quality evaluation results, and the original data uploaded by multiple sensors and multiple quality evaluation results are used as target to be fused data;Through multi-strategy perception fusion system, data fusion processing is carried out on target to be fused data, and target fusion result is obtained, which can ensure the accuracy, rationality and real-time of the data used in sensor perception fusion, and improve the robustness and real-time of sensor perception fusion scheme in real vehicle scene.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data fusion method, apparatus, electronic device, and storage medium. Background Technology

[0002] Sensor fusion refers to the process of combining data collected from multiple sensors. Sensors generate noise, requiring sensor fusion solutions to process this noise and improve the accuracy of the data collected. Sensor fusion solutions are deployed in autonomous driving systems.

[0003] For fusing data collected from different types of sensors or different numbers of data collected from the same type of sensors, current sensor perception fusion solutions typically employ a pre-defined logic, iteratively optimizing it on a specific offline dataset before migrating it to a real vehicle. In other words, current sensor perception fusion solutions, based on a specific offline dataset, can only fit a limited data distribution.

[0004] However, autonomous driving faces extremely diverse real-world scenarios, and sensor perception performance is highly dependent on real-world conditions such as temperature, climate, and lighting. Data collected by sensors can vary significantly across different scenarios. For example, data collected by sensors at night or in inclement weather is of poor quality. This leads to current sensor fusion solutions producing unreasonable fusion results from this poor-quality data, resulting in inaccurate obstacle detection results when relying on the fusion outcome, thus posing safety risks to autonomous driving. Summary of the Invention

[0005] This application provides a data fusion method, apparatus, electronic device, and storage medium to improve the above-mentioned problems.

[0006] In a first aspect, embodiments of this application provide a data fusion method. The method includes: acquiring raw data uploaded by multiple sensors; evaluating the quality of the raw data uploaded by the multiple sensors using an online data quality evaluation system to obtain multiple quality evaluation results; using the raw data uploaded by the multiple sensors and the multiple quality evaluation results as target data to be fused, wherein each sensor's uploaded raw data corresponds to one quality evaluation result; and performing data fusion processing on the target data to be fused using a multi-strategy perception fusion system to obtain a target fusion result.

[0007] Secondly, embodiments of this application provide a data fusion apparatus. The apparatus includes: a data acquisition module for acquiring raw data uploaded by multiple sensors; a data evaluation module for evaluating the quality of the raw data uploaded by the multiple sensors using an online data quality evaluation system, obtaining multiple quality evaluation results, and using the raw data uploaded by the multiple sensors and the multiple quality evaluation results as target data to be fused, wherein each sensor's uploaded raw data corresponds to one quality evaluation result; and a data fusion module for performing data fusion processing on the target data to be fused using a multi-strategy perception fusion system to obtain a target fusion result.

[0008] Thirdly, embodiments of this application provide an electronic device. The electronic device includes a memory, one or more processors, and one or more application programs. The one or more application programs are stored in the memory and configured to execute the methods provided in embodiments of this application when invoked by one or more processors.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium stores program code configured to execute the method provided in embodiments of this application when invoked by a processor.

[0010] This application provides a data fusion method, apparatus, electronic device, and storage medium. The method evaluates the quality of raw data uploaded by multiple sensors online in real time, and performs data fusion processing based on the raw data and its corresponding quality evaluation results. The resulting target fusion result takes into account the impact of unstable sensor perception performance on the perceived data during actual vehicle operation. This ensures the accuracy, rationality, and real-time performance of the data used in sensor perception fusion, and improves the robustness and real-time performance of the sensor perception fusion scheme in real vehicle scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating an application scenario of the data fusion method provided in an exemplary embodiment of this application;

[0013] Figure 2 This is a schematic flowchart of a data fusion method provided in an embodiment of this application;

[0014] Figure 3This is a schematic diagram illustrating the operation of an online data quality evaluation system provided in an exemplary embodiment of this application;

[0015] Figure 4 This is a schematic diagram of the operation of a multi-strategy perception fusion system provided in an exemplary embodiment of this application;

[0016] Figure 5 This is an exemplary embodiment provided by this application. Figure 4 The flowchart of the computational logic of the fusion operator is shown.

[0017] Figure 6 This is a flowchart illustrating a data fusion method provided in another embodiment of this application;

[0018] Figure 7 This is a flowchart illustrating a data fusion method provided in another embodiment of this application;

[0019] Figure 8 This is a schematic diagram of the computational logic of the fusion operator provided in an exemplary embodiment of this application;

[0020] Figure 9 This is a flowchart illustrating a method for constructing an online data quality evaluation system according to an embodiment of this application;

[0021] Figure 10 This is a flowchart illustrating a method for constructing a scoring model according to an embodiment of this application;

[0022] Figure 11 This is a structural block diagram of a data fusion apparatus provided in an embodiment of this application;

[0023] Figure 12 This is a structural block diagram of an electronic device provided in an embodiment of this application;

[0024] Figure 13 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of the data fusion method provided in an exemplary embodiment of this application. The data fusion system 100 includes multiple sensors 110, a data acquisition system 120, an online data quality evaluation system 130, and a multi-strategy perception fusion system 140.

[0027] Multiple sensors 110 are used to collect raw data corresponding to the sensors in the current environment during vehicle operation, and report the raw data to the data acquisition system 120. The data acquisition system 120 can sample the raw data uploaded by the multiple sensors 110, and send the sampled data as the sensor's raw data to the online data quality evaluation system 130 and the multi-strategy perception fusion system 140. The online data quality evaluation system 130 can evaluate the quality of the raw data separately, and send the quality evaluation results to the multi-strategy perception fusion system 140. The multi-strategy perception fusion system 140 performs data fusion processing on the raw data and the quality evaluation results, and sends the fusion result to the lower-level machine (…). Figure 1 (not shown in the image) so that the lower-level machine can perform the corresponding operation based on the fusion result.

[0028] The multiple sensors 110 may include multiple sensors of the same type, or multiple sensors of different types. The multiple sensors 110 can be configured according to actual needs. In some embodiments, the multiple sensors 110 can be configured to collect data for obstacle detection, in which case the raw data collected by the multiple sensors 110 is used for obstacle detection. For example... Figure 1 As shown, the multiple sensors 110 in this embodiment may include a lidar sensor 111, a camera sensor 112, and a millimeter-wave radar sensor 113. It should be noted that the number of sensors 111 can be one or more, the number of sensors 112 can be one or more, and the number of sensors 113 can be one or more.

[0029] The data acquisition system 120 can be mounted on multiple sensors 110 or set up independently of the multiple sensors 110. The data acquisition system 120 can be one or more electronic devices with computing and communication capabilities. For example, the electronic devices can be electronic control units (ECUs), microcontroller units (MCUs), central processing units (CPUs), programmable logic arrays (PLAs), or programmable logic controllers (PLCs).

[0030] In some implementations, the data acquisition system 120 may be configured with the same number of acquisition modules as the multiple sensors 110, with different acquisition modules used to acquire raw data uploaded by different sensors. For example... Figure 1As shown, the data acquisition system 120 may include a point cloud image acquisition module 121, a visual image acquisition module 122, and a point cloud data acquisition module 123. The point cloud image acquisition module 121 is used to acquire raw data uploaded by sensor 111. The visual image acquisition module 122 is used to acquire raw data uploaded by sensor 112. The point cloud data acquisition module 123 is used to acquire raw data uploaded by sensor 114.

[0031] In some implementations, if the system has strong computing power and resources, the data acquisition system 120 may not be required. Multiple sensors 110 can directly upload the raw data they collect to the online data quality evaluation system 130 and the multi-strategy perception fusion system 140.

[0032] The online data quality evaluation system 130 includes multiple scoring models. Different scoring models are used to score data uploaded by different types of sensors to evaluate the quality of data sensed by different types of sensors. These scoring models are pre-trained and stored in the online data quality evaluation system 130. In some embodiments, the online data quality evaluation system 130 includes scoring models corresponding to all types of vehicle-mounted sensors. In other embodiments, to save system resources, the online data quality evaluation system 130 may include scoring models corresponding to vehicle-mounted sensor types with higher importance. The online data quality evaluation system 130 can be one or more electronic devices with computing and communication capabilities, such as ECUs, MCUs, CPUs, PLAs, or PLCs.

[0033] The multi-strategy perception fusion system 140 includes multiple different perception fusion strategies. Different perception fusion strategies yield different fusion results when fusing the same data. The multi-strategy perception fusion system 140 employs a voting mechanism to determine the final fusion result from multiple fusion results obtained by fusing the same data using various perception fusion strategies. The multi-strategy perception fusion system 140 can be one or integrate multiple electronic devices with computing and communication capabilities; for example, the electronic devices can be ECUs, MCUs, CPUs, PLAs, or PLCs.

[0034] In some implementations, the data acquisition system 120, the online data quality evaluation system 130, and the multi-strategy perception fusion system 140 can be integrated into an electronic device with computing and communication capabilities, such as an ECU, MCU, or vehicle control unit (VCU).

[0035] Please see Figure 2 , Figure 2This is a schematic flowchart of a data fusion method provided in an embodiment of this application. This data fusion method can be applied to... Figure 1 The data fusion system 100 shown, or as will be mentioned later. Figure 11 The data fusion apparatus 200 shown. The data fusion method may include the following steps S110-S130.

[0036] Step S110: Obtain raw data uploaded by multiple sensors.

[0037] The multiple sensors can be a subset or all of the onboard sensors. The selection of multiple sensors can be based on actual needs; for example, multiple sensors can be installed around the vehicle to cover its 360-degree surroundings, ensuring the comprehensiveness of the data collected. As an example, if multiple sensors are configured to collect data for obstacle detection, they could include LiDAR sensors, camera sensors, and millimeter-wave radar sensors.

[0038] During actual vehicle operation, multiple sensors acquire raw data corresponding to the sensor in real time based on the current environment. For example, a lidar sensor can acquire laser signals, a camera sensor can acquire image signals, and a millimeter-wave radar sensor can acquire millimeter-wave signals. The raw data can include laser signals, image signals, and millimeter-wave signals. It should be noted that each sensor has a unique sensor identification number (ID). When the sensor outputs raw data, it writes the sensor ID into the raw data. Specifically, the sensor ID can be written at the beginning of the raw data so that other devices can retrieve the raw data collected by the corresponding sensor based on the sensor ID.

[0039] Each sensor can transmit the raw data it collects at a fixed data transmission frequency. However, different sensors have different data transmission frequencies.

[0040] In some implementations, if the data fusion system includes a data acquisition system, the data acquisition system acquires the raw data reported by multiple sensors according to the sensor ID and the data transmission frequency corresponding to the multiple sensors. The raw data reported by the multiple sensors is sampled according to the sensor ID, and the sampled data is used as the raw data of the multiple sensors. This sampled data is then input into the data quality evaluation system and the multi-strategy perception fusion system at the data transmission frequency corresponding to the multiple sensors.

[0041] In other implementations, if the data fusion system does not have a data acquisition system, the data quality evaluation system and the multi-strategy perception fusion system can directly obtain the raw data reported by multiple sensors based on the sensor IDs according to the data transmission frequency corresponding to the multiple sensors.

[0042] Step S120: The raw data uploaded by multiple sensors are evaluated using the online data quality evaluation system to obtain multiple quality evaluation results. The raw data uploaded by multiple sensors and the multiple quality evaluation results are used as the target data to be fused. Each raw data uploaded by each of the multiple sensors corresponds to one quality evaluation result.

[0043] The online data quality evaluation system can evaluate all raw data collected from multiple sensors, such as 3D point cloud data, 2D image data, and video stream data. Different evaluation metrics are used for data collected from different types of sensors; for example, point cloud density and intensity quality for point cloud data, sharpness and contrast for image data, and smoothness for video stream data.

[0044] Different evaluation metrics are used for data collected by different types of sensors, specifically implemented by different scoring models. That is, the online data quality evaluation system includes multiple scoring models, and different scoring models are used to score the raw data from different types of sensors. Multiple scoring models are pre-trained and stored in the data fusion system.

[0045] In some implementations, step S120 may include the following steps: evaluating the quality of the raw data uploaded by each of the multiple sensors using an online data quality evaluation system to obtain a quality score for each sensor; and calculating the ratio of the quality score for each sensor to the sum of the quality scores for the multiple sensors to obtain a quality evaluation result for each sensor.

[0046] The quality score can be set according to actual needs; it can be a specific numerical value or a specific ratio. For example, the quality score can be an integer value between 1 and 10. The quality score characterizes the quality of the data collected by the sensor. Generally, the higher the quality score, the higher the quality of the data collected by the sensor, and the higher the accuracy of subsequent operations performed based on the data collected by the sensor.

[0047] In some implementations, when the online data quality evaluation system acquires the raw sensor data (with the sensor ID written at the beginning of the raw data), it can query the on-board device table based on the sensor ID to determine the sensor type corresponding to the sensor ID as the target sensor type. The raw data corresponding to the sensor ID is then input into the scoring model corresponding to the target sensor type to obtain the quality score for that sensor. The on-board device table is a table stored in the vehicle at the time of manufacture, and it includes the device IDs and corresponding device types of all devices installed on the vehicle.

[0048] As an example, please refer to Figure 3 , Figure 3 This is a schematic flowchart illustrating the operation of an online data quality evaluation system provided in an exemplary embodiment of this application. The online data quality evaluation system can receive raw data (X1, X2, X3, ..., Xn) uploaded by multiple sensors according to the data transmission frequencies corresponding to those sensors. Here, X1, X2, X3, ..., Xn represent the raw data uploaded by different sensors, and n is a positive integer greater than 3. In this example, the multiple sensors include a lidar sensors, b camera sensors, and c millimeter-wave radar sensors.

[0049] Based on the sensor type, the raw data uploaded by the sensor is input into the scoring model corresponding to the sensor type to obtain the quality score (Y1, Y2, Y3, ..., Yn) for the raw data. Here, the quality score for X1 is Y1, for X2 it is Y2, for X3 it is Y3, and for Xn it is Yn. In this example, as... Figure 3 As shown, the online data quality evaluation system includes the following three scoring models: a LiDAR point cloud quality scoring model, a camera visual image quality scoring model, and a millimeter-wave radar point cloud quality scoring model. The LiDAR point cloud quality scoring model is used to score *a* LiDAR sensors. The camera visual image quality scoring model is used to score *b* camera sensors. The millimeter-wave radar point cloud quality scoring model is used to score *c* millimeter-wave radar sensors.

[0050] The online data quality evaluation system normalizes the quality scores (Y1, Y2, Y3, ..., Yn) corresponding to the original data (X1, X2, X3, ..., Xn). This involves calculating the ratio of the quality score for each sensor to the sum of the quality scores for all sensors, resulting in a normalized result (Q1, Q2, Q3, ..., Qn) for each sensor. Here, Q1 corresponds to the quality scores of X1 and Y1, Q2 to X2 and Y2, Q3 to X3 and Y3, and Qn to Xn and Yn. The online data quality evaluation system then transmits these normalized results to the multi-strategy perception fusion system at the same data transmission frequency. This allows the multi-strategy perception fusion system to perform data fusion processing on the normalized results and the original data uploaded from multiple sensors.

[0051] For example, a LiDAR point cloud quality scoring model scores a LiDAR sensors and outputs a first score (LS1, LS2, LS3, ..., LSM). A camera visual image quality scoring model scores b camera sensors and outputs a second score (CS1, CS2, CS3, ..., CSj). A millimeter-wave radar point cloud quality scoring model scores c millimeter-wave radar sensors and outputs a third score (RS1, RS2, RS3, ..., RSk).

[0052] The quality scores of each sensor can be normalized using the following expression: the ratio of the quality score of each sensor to the sum of the quality scores of all sensors is calculated to obtain the quality evaluation result for each sensor.

[0053]

[0054]

[0055]

[0056] Wherein, LSI represents the quality evaluation result corresponding to the i-th lidar sensor, and LSi represents the quality score corresponding to the i-th lidar sensor. CSJ represents the quality evaluation result corresponding to the j-th camera sensor, and CSj represents the quality score corresponding to the j-th camera sensor. RSP represents the quality evaluation result corresponding to the p-th millimeter-wave radar sensor, and RSp represents the quality score corresponding to the p-th millimeter-wave radar sensor. It represents the sum of the quality scores corresponding to multiple sensors. It represents the sum of the quality scores corresponding to multiple lidar sensors. It represents the sum of the quality scores corresponding to multiple camera sensors. This represents the sum of quality scores corresponding to multiple millimeter-wave radar sensors. a, b, and c represent the number of lidar sensors, camera sensors, and millimeter-wave radar sensors, respectively. i, j, and p are process variables.

[0057] It should be noted that, due to the different data transmission frame rates of different sensors, the multi-strategy perception fusion system needs to perform time synchronization and data alignment on the target data to be fused before performing data fusion processing. This ensures that the data is aligned in time, thereby improving data accuracy. Specifically, the multi-strategy perception fusion system can acquire the data transmission frame rates of multiple sensors; determine the sensor to be interpolated from among the multiple sensors based on the data transmission frame rates; perform frame interpolation processing on the raw data uploaded by the sensor to be interpolated, and use the interpolated data as the raw data of the sensor to be interpolated, so as to synchronize the raw data transmitted by multiple sensors in time. The quality evaluation result of the interpolated frame in the data after interpolation is the average of the quality evaluation results of the two frames before and after the interpolation.

[0058] In some implementations, the multi-strategy perception fusion system can send instruction information to each sensor to obtain the data transmission frame rate, and each sensor responds to the instruction information by uploading its own data transmission frame rate to the multi-strategy perception fusion system.

[0059] In some implementations, the specific implementation of determining the sensor to be inserted from multiple sensors based on the data transmission frame rate can be as follows: determine the sensor with the highest data transmission frame rate from the data transmission frame rates of multiple sensors, and determine the sensors other than the sensor with the highest data transmission frame rate from the multiple sensors as the sensors to be inserted.

[0060] In some implementations, the frame interpolation processing of the raw data uploaded by the sensor to be interpolated can be performed using algorithms such as nearest neighbor interpolation, linear interpolation, bilinear interpolation, or higher-order interpolation.

[0061] Step S130: The target data to be fused is processed by the multi-strategy perception fusion system to obtain the target fusion result.

[0062] The multi-strategy perception fusion system includes various perception fusion strategies. Different strategies yield different fusion results when processing the same data. These strategies can be pre-set and stored within the system. The specific number and content of these strategies can be flexibly configured based on the sensor types and computing capabilities of each platform, making it highly practical. Specifically, the content of each strategy is related to the intended use of the raw data from multiple sensors. Furthermore, the stored strategies can be modified during practical applications. For example, existing strategies can be modified or deleted, and new strategies can be added.

[0063] In some implementations, multiple sensors, including lidar sensors, camera sensors, and millimeter-wave radar sensors, are used for obstacle detection, with raw data from these sensors being used. At least three of the following perception fusion strategies can be deployed in a multi-strategy perception fusion system:

[0064] Strategy 1: Use a lidar sensor to detect targets (e.g., obstacles), use a camera sensor to supplement the lidar sensor's blind spots, and use a millimeter-wave radar sensor to correct the speed of the fusion result of the two.

[0065] Strategy 2: Use raw data from lidar sensors and camera sensors to perform data fusion, use the fused data to detect targets around the vehicle, and use millimeter-wave radar sensors to correct the detection results.

[0066] Strategy 3: Use both lidar and camera sensors to detect targets, and use millimeter-wave radar to correct targets in the overlapping areas of the two sensors.

[0067] In some implementations, the multi-strategy perception fusion system can employ multiple perception fusion strategies to perform data fusion processing on the target data to be fused, resulting in multiple fusion results. The multi-strategy perception fusion system can use a voting mechanism to determine the target fusion result from these multiple results. This voting mechanism can be implemented using a majority-based voting system.

[0068] Please refer to this as an example. Figure 3-5 , Figure 4 This is a schematic diagram illustrating the operation of a multi-strategy perception fusion system provided in an exemplary embodiment of this application. Figure 5 This is an exemplary embodiment provided by this application. Figure 4The diagram shows the computational logic of the fusion operator. (X1, X2, X3, ..., Xn) represent the raw data from multiple sensors. (Q1, Q2, Q3, ..., Qn) represent multiple quality evaluation results, each corresponding to the raw data from one sensor. (X1, X2, X3, ..., Xn) and (Q1, Q2, Q3, ..., Qn) constitute the target data to be fused. {Fi(Xi), Gi(Xi), Hi(Xi), ..., Ri(Xi)} represent multiple different perception fusion strategies.

[0069] like Figure 4 As shown, the multi-strategy perception fusion system can employ multiple fusion perception strategies in parallel to perform data fusion processing on the same target data to be fused, resulting in multiple fusion results. After each fusion perception strategy completes its processing, it outputs a fusion result to the fusion operator.

[0070] like Figure 5 As shown, the multi-strategy perception fusion system incorporates a voting mechanism in its fusion operator. The fusion operator uses this voting mechanism to determine the target fusion result from multiple fusion outcomes and outputs the target fusion result to the lower-level machine, enabling the lower-level machine to perform corresponding operations based on the target fusion result.

[0071] By using the quality of the raw data of each frame as a confidence level in the perception fusion strategy, the normalized weighting factor (i.e., the quality evaluation result corresponding to each sensor as normalized above) will make data with poor quality (i.e., small quality evaluation result) less important to the fusion result, and data with good quality (i.e., large quality evaluation result) more important to the fusion result. This can ensure the accuracy of the fusion result and improve the sensor perception performance.

[0072] Furthermore, compared to the current approach of using only one strategy to fuse raw data from multiple sensors, by setting up multiple different perception fusion strategies in a multi-strategy perception fusion system to process the same target data to be fused, the fusion result of the perception fusion strategy with the highest accuracy in the current real-vehicle scenario can be used as the target fusion result for different real-vehicle scenarios. This breaks through the bottleneck of perception fusion algorithms in autonomous driving being fixed in a finite-dimensional data space, which can improve the accuracy of the fusion result and thus effectively improve the perception performance of real-vehicle autonomous driving.

[0073] The data fusion method provided in this application evaluates the quality of raw data uploaded by multiple sensors online in real time, and performs data fusion processing based on the raw data and its corresponding quality evaluation results. The target fusion result takes into account the impact of unstable sensor perception performance on the perceived data during actual vehicle operation, which can ensure the accuracy, rationality and real-time performance of the data used in sensor perception fusion, and improve the robustness and real-time performance of the sensor perception fusion scheme in real vehicle scenarios.

[0074] Please see Figure 6 , Figure 6 This is a flowchart illustrating a data fusion method provided in another embodiment of this application. This data fusion method can be applied to... Figure 1 The data fusion system 100 shown, or as will be mentioned later. Figure 11 The data fusion apparatus 200 shown. The data fusion method may include the following steps S210-S240.

[0075] Step S210: Obtain raw data uploaded by multiple sensors.

[0076] Step S220: The raw data uploaded by multiple sensors are evaluated using an online data quality evaluation system to obtain multiple quality evaluation results. The raw data uploaded by multiple sensors and the multiple quality evaluation results are used as the target data to be fused. Each sensor's uploaded raw data corresponds to one quality evaluation result.

[0077] For a detailed description of steps S210-S220, please refer to the aforementioned steps S110-S120 respectively, and they will not be repeated here.

[0078] Step S230: Multiple perception fusion strategies are used in parallel to perform data fusion processing on the target data to be fused, resulting in multiple fusion results. Each perception fusion strategy in the multiple perception fusion strategies corresponds to one fusion result.

[0079] Each of the multiple perception fusion strategies can include one or more perception fusion algorithms. The perception fusion algorithm can be determined according to actual needs; for example, the perception fusion algorithm can include a pre-fusion algorithm and / or a post-fusion algorithm.

[0080] In some implementations, the specific implementation of using each perception fusion strategy to perform data fusion processing on the target data to be fused to obtain a fusion result may include the following steps: using the perception fusion algorithm corresponding to a perception fusion strategy to perform data fusion processing on the raw data of multiple sensors in the target data to be fused, initially obtaining multiple raw data fusion results, each corresponding to one sensor; setting multiple quality evaluation results in the target data to be fused as weights of the corresponding sensors; calculating the fusion results of each of the multiple sensors according to their respective raw data fusion results and weights, obtaining multiple preliminary fusion results, each corresponding to one sensor; calculating the sum of the multiple preliminary fusion results as the fusion result corresponding to the perception fusion strategy.

[0081] Specifically, the specific implementation method for calculating the raw data fusion result corresponding to each sensor may include the following steps: using a perception fusion algorithm corresponding to a perception fusion strategy to perform data fusion processing on the raw data corresponding to a sensor, and obtaining the raw data fusion result corresponding to that sensor.

[0082] Specifically, the specific implementation of calculating the fusion result of each sensor may include the following steps: calculating the product of the original data fusion result and the weight corresponding to a sensor, as the preliminary fusion result corresponding to that sensor.

[0083] As an example, such as Figure 4 As shown, the raw data and quality evaluation result corresponding to a sensor are X1 and Q1, respectively. Using the perception fusion algorithm Fi(Xi) in perception fusion strategy 1 to perform data fusion processing on X1, the data fusion result corresponding to this sensor is F1(X1). The product of F1(X1) and Q1, Q1*F1(X1), is calculated as the preliminary fusion result corresponding to this sensor. The sum of the preliminary fusion results corresponding to each sensor is calculated, resulting in "Q1*F1(X1)+Q2*F2(X2)+Q3*F3(X3)+…+Qn*Fn(Xn)", which is the fusion result corresponding to perception fusion strategy 1. The fusion results corresponding to other perception fusion strategies 2-n can be calculated in a similar manner to calculating the fusion result corresponding to perception fusion strategy 1.

[0084] Step S240: Use a voting mechanism to determine the target fusion result from multiple fusion results.

[0085] The voting mechanism can be implemented using a majority-based voting system. The specific details of the voting mechanism can be determined based on the intended use of the raw data from the multiple sensors.

[0086] For steps S230-240 that are not described in detail, please refer to step S130 above, and will not be repeated here.

[0087] The data fusion method provided in this application evaluates the quality of raw data uploaded by multiple sensors online in real time, and performs data fusion processing based on the raw data and its corresponding quality evaluation results. The target fusion result takes into account the impact of unstable sensor perception performance on the perceived data during actual vehicle operation, which can ensure the accuracy, rationality and real-time performance of the data used in sensor perception fusion, and improve the robustness and real-time performance of the sensor perception fusion scheme in real vehicle scenarios.

[0088] Please see Figure 7 , Figure 7 This is a flowchart illustrating a data fusion method provided in another embodiment of this application. This data fusion method can be applied to... Figure 1The data fusion system 100 shown, or as will be mentioned later. Figure 11 The data fusion apparatus 200 shown. The data fusion method may include the following steps S310-S340.

[0089] First, it should be noted that the raw data uploaded by the multiple sensors in this application embodiment is used for obstacle detection. As an example, the multiple sensors in this application embodiment may include one or more lidar sensors, one or more camera sensors, and one or more millimeter-wave radar sensors.

[0090] Step S310: Obtain raw data uploaded by multiple sensors.

[0091] Step S320: The raw data uploaded by multiple sensors are evaluated using the online data quality evaluation system to obtain multiple quality evaluation results. The raw data uploaded by multiple sensors and the multiple quality evaluation results are used as the target data to be fused. Each raw data uploaded by each of the multiple sensors corresponds to one quality evaluation result.

[0092] For a detailed description of steps S310-S320, please refer to the aforementioned steps S110-S120, which will not be repeated here.

[0093] Step S330: Multiple perception fusion strategies are used in parallel to perform data fusion processing on the target data to be fused, resulting in multiple obstacle categories and corresponding obstacle poses. The multiple obstacle categories and corresponding obstacle poses are multiple fusion results.

[0094] When raw data from multiple sensors is used for obstacle detection, multiple fusion results calculated by various perception fusion strategies based on the target data to be fused are also used for obstacle detection. Each fusion result corresponding to a perception fusion strategy includes one or more obstacle categories and one or more obstacle poses. The number of obstacles is the same as the number of obstacle poses, with each obstacle corresponding to one obstacle pose.

[0095] By employing multiple perception fusion strategies in parallel to process the target data to be fused, multiple obstacle categories and corresponding obstacle poses can be obtained.

[0096] Step S340: Using a voting mechanism, determine the category containing the most obstacles from multiple obstacle categories as the target obstacle category, and determine the average of the obstacle poses corresponding to the target obstacle category as the target obstacle pose. Here, the target obstacle category and the target obstacle pose are the target fusion results.

[0097] Please see Figure 8 , Figure 8This is a schematic diagram illustrating the computational logic of a fusion operator provided in an exemplary embodiment of this application. The multi-strategy perception fusion system can incorporate a majority-rule voting unit within the fusion operator. Upon receiving multiple fusion results, including obstacle categories and obstacle poses, from various perception fusion strategies (e.g., strategy 1-n), the system calculates the number of obstacles in each category and designates the category with the highest number of obstacles as the target obstacle category. The average obstacle pose corresponding to the target obstacle category is then calculated as the target obstacle pose. The target obstacle category and target obstacle pose are then used as the target fusion result and output to the lower-level machine, enabling the lower-level machine to perform obstacle detection operations based on the target fusion result.

[0098] In some implementations, if there are at least two categories that include the most obstacles, these at least two obstacle categories are used as preliminary obstacle categories. The mean of the quality evaluation results in the perception fusion strategy corresponding to the at least two preliminary obstacle categories is calculated, and the preliminary obstacle category with the largest mean is used as the target obstacle category.

[0099] As an example, the multiple perception fusion strategies include strategies 1-10. Multiple obstacle categories include categories A, B, C, and D. Category A is calculated using strategy 1 and includes one obstacle of category A. Category B is calculated using strategies 2-5 and includes four obstacles of category B. Category C is calculated using strategies 6-9 and includes four obstacles of category C. Category D is calculated using strategy 10 and includes one obstacle of category D. It is evident that categories B and C contain the most obstacles, thus the initial obstacle categories can be determined as categories B and C. Further, the mean of the quality evaluation results in strategies 2-5 corresponding to category B can be calculated as the first mean, and the mean of the quality evaluation results in strategies 6-9 corresponding to category C can be calculated as the second mean. If the first mean is greater than the second mean, then category B is determined as the target obstacle category. Conversely, if the first mean is less than the second mean, then category C is determined as the target obstacle category.

[0100] In some implementations, calculating the average pose of obstacles corresponding to the target obstacle category as the target obstacle pose may include the following steps: obtaining obstacles corresponding to the target obstacle category as target obstacles, and calculating the average pose of obstacles corresponding to the target obstacles as the target obstacle pose. For example, continuing from the previous example, assuming the target obstacle is of category B, four obstacles of category B can be obtained, and the average pose of obstacles corresponding to these four obstacles of category B can be calculated as the target obstacle pose.

[0101] For steps S330-S340 that are not described in detail, please refer to the aforementioned steps S130 and S230-S240, which will not be repeated here.

[0102] The data fusion method provided in this application evaluates the quality of raw data uploaded by multiple sensors online in real time, and performs data fusion processing based on the raw data and its corresponding quality evaluation results. The target fusion result takes into account the impact of unstable sensor perception performance on the perceived data during actual vehicle operation, which can ensure the accuracy, rationality and real-time performance of the data used in sensor perception fusion, and improve the robustness and real-time performance of the sensor perception fusion scheme in real vehicle scenarios.

[0103] Please see Figure 9 , Figure 9 This is a flowchart illustrating a method for constructing an online data quality evaluation system according to an embodiment of this application. The data fusion method provided in this embodiment includes... Figure 9 The method for constructing an online data quality evaluation system is shown. The method for constructing an online data quality evaluation system may include the following steps S410-S430.

[0104] Step S410: Obtain the types of multiple sensors.

[0105] In some implementations, as described above, the sensor type corresponding to the sensor ID can be obtained by querying the vehicle equipment table based on the sensor ID.

[0106] In other implementations, the mapping relationship between sensor ID and sensor type can be pre-set in the online data quality evaluation system so that the online data quality evaluation system can directly find the sensor type corresponding to the sensor ID.

[0107] In some other implementations, the sensor can write its type into the corresponding raw data so that the online data quality assessment system can obtain the sensor type.

[0108] Step S420: Construct multiple scoring models based on the types of multiple sensors, wherein different scoring models are used to score the raw data uploaded by different types of sensors.

[0109] In some implementations, please refer to Figure 10 , Figure 10 This is a flowchart illustrating a method for constructing a scoring model according to an embodiment of this application. The method for constructing a scoring model may include the following steps S421-S423.

[0110] Step S421: Select the same type of sensor from multiple sensors as the target sensor.

[0111] The system includes one or more target sensors. Each target sensor can be any type of sensor from the multiple sensors listed. The online data quality assessment system can sort the multiple sensor types in a predetermined order and sequentially build a corresponding scoring model for each type of sensor. This predetermined order can be set according to actual needs, such as sorting them according to the importance of each type of sensor. Alternatively, the online data quality assessment system can randomly build a corresponding scoring model for each type of sensor until a corresponding scoring model exists for all types of sensors.

[0112] The online data quality evaluation system can randomly select sensors of the same type from multiple sensors as the target sensor.

[0113] Step S422: Obtain the raw dataset collected by the target sensor and the corresponding quality score of the raw dataset, wherein the quality score is manually labeled.

[0114] The raw dataset from the target sensor is acquired synchronously in real time.

[0115] The quality scores for the original dataset are manually labeled, and the specific values ​​of the quality scores can be set according to actual needs. For example, the quality score can be a positive integer from 1 to 10. The higher the quality score, the higher the data quality, and the more accurate the summation results calculated from higher quality data.

[0116] In some implementations, the target sensor is a lidar sensor. Evaluators can score the point cloud acquired by the lidar sensor based on principles such as point cloud density, point cloud efficiency, and the presence of condensation, stratification, or high anti-dilatation phenomena, assigning a quality score to the point cloud. Here, a point cloud refers to a dataset where each point represents a set of X, Y, Z geometric coordinates and an intensity value. This intensity value records the strength of the returned signal based on the reflectivity of the object's surface. When these points are combined, they form a point cloud, which is a set of data points representing 3D shapes or objects in space. Point clouds can be automatically colored for more realistic visualization.

[0117] In other implementations, the target sensor is a camera sensor. Evaluators can score the images acquired by the camera sensor based on criteria such as image sharpness, whether they are overexposed or underexposed, whether they are distorted, and whether there are any missing parts, assigning a quality score to the images acquired by the camera sensor.

[0118] In some other implementations, the target sensor is a millimeter-wave radar sensor. Evaluators can score the images acquired by the millimeter-wave radar sensor based on principles such as point cloud density and point cloud accuracy, assigning quality scores to the images.

[0119] After scoring the target sensor, the scorers can upload the original dataset and the corresponding quality score to the online data quality evaluation system.

[0120] Step S423: Take the original dataset as input and the quality score corresponding to the original dataset as output. Use the TensorFlow framework and a preset deep learning scheme to train the scoring model corresponding to the target sensor to obtain the scoring model corresponding to the target sensor.

[0121] TensorFlow is an open-source machine learning framework based on Python that also offers programming languages ​​such as C / C++, Java, Go, and R.

[0122] The preset deep learning scheme may include one or more deep learning classification algorithms, such as DenseNet, ResNet, GoogleNet, and Vgg. It should be noted that scoring models for different types of sensors can be trained using different deep learning classification algorithms, or they can be trained using the same deep learning classification algorithm.

[0123] In some implementations, the original dataset is used as input, and the quality scores corresponding to the original dataset are used as output. The scoring model corresponding to the target sensor is trained repeatedly using the TensorFlow framework and a preset deep learning scheme until the accuracy of the scoring model reaches a preset accuracy. The scoring model with the preset accuracy is then used as the scoring model corresponding to the target sensor. The preset accuracy can be set according to the actual requirements for data accuracy.

[0124] By training the scoring model using the raw dataset collected by sensors and the quality scores of the raw dataset labeled by humans, the quality evaluation results obtained by the scoring model in evaluating the data quality can be close to the results of human judgment, thereby improving the accuracy of the quality evaluation results.

[0125] Step S430: Construct an online data quality evaluation system based on multiple scoring models.

[0126] After multiple scoring models have been trained, an online data quality evaluation system can be built to deploy these models.

[0127] Currently, intelligent driving systems directly input acquired sensor data into pre-designed perception fusion strategies. However, most offline-designed perception fusion strategies are based on offline datasets or data collection for specific scenarios, and cannot fully replicate the complexities of the real world. Furthermore, the hardware state of the sensors themselves is uncertain. For example, LiDAR sensors measure distance by the time between the laser pulse emitted by the laser and the reflection from the target object back to the receiver; different climates, lighting conditions, air humidity, temperature, and the target object itself are all key factors affecting reflection. Similarly, the image quality of camera sensors is strongly correlated with lighting conditions. Therefore, even the best current offline-designed perception fusion strategies are insufficient to accurately evaluate sensor performance in various real-world vehicle scenarios.

[0128] Compared to current perception fusion strategies, this application embodiment sets up an online data quality evaluation system to score the real-time raw data of each sensor, and applies the scoring results to the sensor's perception results in the multi-strategy perception fusion system. This allows real-time data quality to dynamically influence the effect of the perception fusion strategy, thereby ensuring the real-time performance and accuracy of the data and fusion results.

[0129] The method for constructing an online data quality evaluation system provided in this application constructs different scoring models according to different sensor types, enabling the online data quality evaluation system to evaluate the quality of raw data uploaded by different types of sensors, thus ensuring the comprehensiveness of the data quality evaluation.

[0130] Please see Figure 11 , Figure 11 This is a structural block diagram of a data fusion apparatus provided in an embodiment of this application. The data fusion apparatus 200 can be applied to the above-described... Figure 1 The data fusion system 100 shown includes a data fusion device 200, which may include a data acquisition module 210, a data evaluation module 220, and a data fusion module 230. The data acquisition module 210, the data evaluation module 220, and the data fusion module 230 are communicatively connected to enable data interaction.

[0131] The data acquisition module 210 is used to acquire raw data uploaded by multiple sensors.

[0132] The data evaluation module 220 is used to evaluate the quality of the raw data uploaded by the multiple sensors through an online data quality evaluation system, obtain multiple quality evaluation results, and use the raw data uploaded by the multiple sensors and the multiple quality evaluation results as target data to be fused. Each of the multiple sensors uploads raw data corresponding to one quality evaluation result.

[0133] The data fusion module 230 is used to perform data fusion processing on the target data to be fused through a multi-strategy perception fusion system to obtain the target fusion result.

[0134] In some implementations, the data fusion module 230 is further configured to perform data fusion processing on the target data to be fused in parallel using the multiple perception fusion strategies respectively, to obtain multiple fusion results, wherein each of the multiple perception fusion strategies corresponds to one fusion result; and to determine the target fusion result from the multiple fusion results using a voting mechanism.

[0135] In some implementations, the raw data uploaded by the multiple sensors is used for obstacle detection. The data fusion module 230 is further configured to perform data fusion processing on the target data to be fused using the multiple perception fusion strategies in parallel, obtaining multiple obstacle categories and corresponding obstacle poses, wherein the multiple obstacle categories and corresponding obstacle poses are the multiple fusion results; a voting mechanism is used to determine the category containing the most obstacles from the multiple obstacle categories as the target obstacle category, and the average of the obstacle poses corresponding to the target obstacle category is determined as the target obstacle pose, wherein the target obstacle category and the target obstacle pose are the target fusion result.

[0136] In some embodiments, before the target data to be fused is processed by the multi-strategy perception fusion system to obtain the target fusion result, the data fusion module 230 is further configured to acquire the data transmission frame rate of the multiple sensors; determine the sensor to be interpolated from the multiple sensors according to the data transmission frame rate; perform frame interpolation processing on the original data uploaded by the sensor to be interpolated, and use the data after the interpolation as the original data of the sensor to be interpolated, so as to synchronize the original data transmitted by the multiple sensors in time, wherein the quality evaluation result of the inserted frame in the data after the interpolation is the average of the quality evaluation results of the two frames before and after the interpolation.

[0137] In some embodiments, the data fusion device 200 further includes a model building module (not shown in the figure). The model building module is used to obtain the types of the plurality of sensors; construct a plurality of scoring models according to the types of the plurality of sensors, wherein different scoring models are used to score the raw data uploaded by different types of sensors; and construct the online data quality evaluation system according to the plurality of scoring models.

[0138] In some implementations, the model building module is further configured to: identify sensors of the same type as target sensors from the plurality of sensors; obtain the original dataset collected by the target sensor and the quality score corresponding to the original dataset, wherein the quality score is manually labeled; use the original dataset as input and the quality score corresponding to the original dataset as output, and train the scoring model corresponding to the target sensor using the TensorFlow framework and a preset deep learning scheme to obtain the scoring model corresponding to the target sensor.

[0139] In some implementations, the model building module is also used to send the built scoring model to the data evaluation module 220. The data evaluation module 220 is also used to receive the scoring model and deploy it in the data evaluation module 220.

[0140] In some implementations, the data evaluation module 220 is further configured to evaluate the quality of the raw data uploaded by each of the plurality of sensors through an online data quality evaluation system, thereby obtaining a quality score corresponding to each sensor; and to calculate the ratio of the quality score corresponding to each sensor to the sum of the quality scores corresponding to the plurality of sensors, thereby obtaining a quality evaluation result corresponding to each sensor.

[0141] Those skilled in the art will clearly understand that the data fusion apparatus 200 provided in the embodiments of this application can implement the data fusion method provided in the embodiments of this application. The specific working process of the above apparatus and modules can be found in the corresponding process of the data fusion method in the embodiments of this application, and will not be repeated here.

[0142] In the embodiments provided in this application, the coupling, direct coupling, or communication connection between the modules shown or discussed may be indirect coupling or communication coupling through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms. The embodiments of this application do not limit this.

[0143] Furthermore, the functional modules in the embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules, and this application embodiment does not impose any restrictions on this.

[0144] Please see Figure 12 , Figure 12This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device 300 may include one or more of the following components: a memory 310, one or more processors 320, and one or more application programs, wherein the one or more application programs may be stored in the memory 310 and configured to, when invoked by one or more processors 320, cause one or more processors 320 to execute the data fusion method provided in the embodiment of this application.

[0145] The processor 320 may include one or more processing cores. The processor 320 uses various interfaces and lines to connect to various parts of the entire electronic device 300, and is used to run or execute instructions, programs, code sets or instruction sets stored in the memory 310, as well as to call and run or execute data stored in the memory 310, and perform various functions of the electronic device 300 and process data.

[0146] In some implementations, the processor 320 may be implemented in at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA).

[0147] In some implementations, processor 320 may integrate one or a combination of several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 320 and may be implemented using a separate communication chip.

[0148] The memory 310 may include random access memory (RAM) or read-only memory (ROM). The memory 310 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 310 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc. The data storage area may store data created by the electronic device 300 during use.

[0149] Please see Figure 13 , Figure 13 This is a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable storage medium 400 stores code 410, which is configured to cause the processor to execute the data fusion method described above in an embodiment of this application when invoked by the processor.

[0150] The computer-readable storage medium 400 may be an electronic storage device such as flash memory, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), hard disk, or ROM. Optionally, the computer-readable storage medium 400 includes a non-volatile computer-readable storage medium (Non-TCRSM). The computer-readable storage medium 400 has storage space for code 410 that performs any of the method steps described above. This code 410 can be read from or written to one or more computer program products. The code 410 may be compressed in an appropriate form.

[0151] In summary, the embodiments of this application provide a data fusion method, apparatus, electronic device, and storage medium. This method evaluates the quality of raw data uploaded by multiple sensors online in real time, and performs data fusion processing based on the raw data and its corresponding quality evaluation results. The resulting target fusion result takes into account the impact of unstable sensor perception performance on the perceived data during actual vehicle operation. This ensures the accuracy, rationality, and real-time performance of the data used in sensor perception fusion, and improves the robustness and real-time performance of the sensor perception fusion scheme in real vehicle scenarios.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data fusion method, characterized in that, include: Acquire raw data uploaded by multiple sensors, including lidar sensors, camera sensors, and millimeter-wave radar sensors; The raw data uploaded by the multiple sensors are evaluated using an online data quality evaluation system to obtain multiple quality evaluation results. The raw data uploaded by the multiple sensors and the multiple quality evaluation results are used as the target data to be fused. Each of the multiple sensors uploads raw data corresponding to one quality evaluation result. The target data to be fused is processed by a multi-strategy perception fusion system to obtain the target fusion result; The step of performing data fusion processing on the target data to be fused through a multi-strategy perception fusion system to obtain the target fusion result includes: Multiple perception fusion strategies are employed in parallel to perform data fusion processing on the target data to be fused, resulting in multiple fusion results. Each of the multiple perception fusion strategies corresponds to one fusion result. The multiple perception fusion strategies include at least Strategy 1, Strategy 2, and Strategy 3. Strategy 1 uses a lidar sensor to detect targets, a camera sensor to supplement the detection of blind spots of the lidar sensor, and a millimeter-wave radar sensor to correct the fusion results of the lidar sensor and the camera sensor. Strategy 2 uses the raw data from the lidar sensor and the camera sensor to perform data fusion, uses the fused data to detect targets around the vehicle, and uses a millimeter-wave radar sensor to correct the detection results. Strategy 3 uses both the lidar sensor and the camera sensor to perform target detection, and uses millimeter-wave radar to correct targets in the overlapping areas of the lidar sensor and the camera sensor. A voting mechanism is used to determine the target fusion result with the highest accuracy from the multiple fusion results.

2. The data fusion method according to claim 1, characterized in that, The raw data uploaded by the multiple sensors is used for obstacle detection. The step of using multiple perception fusion strategies in parallel to perform data fusion processing on the target data to be fused, and obtaining multiple fusion results, includes: The target data to be fused is processed by employing the multiple perception fusion strategies in parallel to obtain multiple obstacle categories and corresponding obstacle poses, wherein the multiple obstacle categories and corresponding obstacle poses are the multiple fusion results; The step of determining the target fusion result with the highest accuracy from the multiple fusion results using a voting mechanism includes: A voting mechanism is used to determine the category containing the most obstacles from the multiple obstacle categories as the target obstacle category, and the average of the obstacle poses corresponding to the target obstacle category is determined as the target obstacle pose, wherein the target obstacle category and the target obstacle pose are the target fusion result.

3. The data fusion method according to any one of claims 1-2, characterized in that, Before performing data fusion processing on the target data to be fused through the multi-strategy perception fusion system to obtain the target fusion result, the method further includes: Obtain the data transmission frame rate of the multiple sensors; Based on the data transmission frame rate, determine the sensor to be inserted from the plurality of sensors; The raw data uploaded by the sensor to be interpolated is subjected to frame interpolation processing to achieve time synchronization of the raw data transmitted by the multiple sensors. The quality evaluation result of the inserted frame in the data after frame interpolation processing is the average of the quality evaluation results of the two frames before and after the inserted frame.

4. The data fusion method according to claim 1, characterized in that, The method for constructing the online data quality evaluation system includes: Obtain the types of the multiple sensors; Multiple scoring models are constructed based on the types of the multiple sensors, wherein different scoring models are used to score the raw data uploaded by different types of sensors; Based on the multiple scoring models, the online data quality evaluation system is constructed.

5. The data fusion method according to claim 4, characterized in that, Methods for building a scoring model include: Select sensors of the same type from the plurality of sensors as target sensors; Obtain the raw dataset collected by the target sensor and the corresponding quality score of the raw dataset, wherein the quality score is manually labeled; The original dataset is used as input, and the quality score corresponding to the original dataset is used as output. The scoring model corresponding to the target sensor is trained using the TensorFlow framework and a preset deep learning scheme to obtain the scoring model corresponding to the target sensor.

6. The data fusion method according to claim 1, characterized in that, The step of evaluating the quality of the raw data uploaded by the multiple sensors using an online data quality evaluation system to obtain multiple quality evaluation results includes: The quality of the raw data uploaded by each of the multiple sensors is evaluated using an online data quality evaluation system to obtain a quality score for each sensor. The ratio of the quality score corresponding to each sensor to the sum of the quality scores corresponding to the multiple sensors is calculated to obtain the quality evaluation result corresponding to each sensor.

7. A data fusion device, characterized in that, include: The data acquisition module is used to acquire raw data uploaded by multiple sensors, including a lidar sensor, a camera sensor, and a millimeter-wave radar sensor. The data evaluation module is used to evaluate the quality of the raw data uploaded by the multiple sensors through the online data quality evaluation system, obtain multiple quality evaluation results, and use the raw data uploaded by the multiple sensors and the multiple quality evaluation results as the target data to be fused. Each of the multiple sensors uploads raw data corresponding to one quality evaluation result. The data fusion module is used to perform data fusion processing on the target data to be fused through a multi-strategy perception fusion system to obtain the target fusion result; The step of performing data fusion processing on the target data to be fused through a multi-strategy perception fusion system to obtain the target fusion result includes: Multiple perception fusion strategies are employed in parallel to perform data fusion processing on the target data to be fused, resulting in multiple fusion results. Each of the multiple perception fusion strategies corresponds to one fusion result. The multiple perception fusion strategies include at least Strategy 1, Strategy 2, and Strategy 3. Strategy 1 uses a lidar sensor to detect targets, a camera sensor to supplement the detection of blind spots of the lidar sensor, and a millimeter-wave radar sensor to correct the fusion results of the lidar sensor and the camera sensor. Strategy 2 uses the raw data from the lidar sensor and the camera sensor to perform data fusion, uses the fused data to detect targets around the vehicle, and uses a millimeter-wave radar sensor to correct the detection results. Strategy 3 uses both the lidar sensor and the camera sensor to perform target detection, and uses millimeter-wave radar to correct targets in the overlapping areas of the lidar sensor and the camera sensor. A voting mechanism is used to determine the target fusion result with the highest accuracy from the multiple fusion results.

8. An electronic device, characterized in that, include: Memory; One or more processors; One or more applications, wherein the one or more applications are stored in the memory and configured to, when invoked by the one or more processors, cause the one or more processors to execute the data fusion method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that, when invoked by a processor, causes the processor to execute the data fusion method as described in any one of claims 1-6.

Images