Data fusion method and device, electronic equipment and storage medium

Abstract

The invention discloses a data fusion method and device, electronic equipment and a storage medium, the data fusion method can be applied to the technical field of aided driving, and the method comprises the steps: obtaining to-be-fused data from a multi-modal sensor; based on timestamps and space coordinate information in the to-be-fused data, verifying logic consistency of the to-be-fused data and historical associated data in time sequence and space relation to obtain a time sequence verification result; when the time sequence verification result is abnormal, determining a time sequence abnormal degree based on a timestamp deviation value between the to-be-fused data and the historical associated data and / or a space coordinate deviation value between the to-be-fused data and the historical associated data; repairing the to-be-fused data based on a repairing strategy corresponding to the time sequence abnormal degree; and performing fusion processing on the repaired data to be fused to generate an environment perception result. The data fusion method provided by the invention has real-time protection and dynamic fault-tolerant capabilities, and the accuracy of a fusion result is improved.

Description

Technical Field

[0001] This application relates to the field of driver assistance technology, and in particular to data fusion methods, devices, electronic devices and storage media. Background Technology

[0002] With the rapid development of fields such as intelligent driving, robotic perception, and smart cities, multimodal perception systems have become a core technology for achieving high-precision environmental understanding. These systems typically integrate data from various heterogeneous sensors, such as LiDAR, cameras, millimeter-wave radar, and GNSS (Global Navigation Satellite System), to compensate for the limitations of single-modal perception capabilities. However, due to significant differences in sampling frequency, coordinate system definition, time reference, and data format among different sensors, efficiently and reliably achieving accurate alignment and fusion of multi-source data across the spatiotemporal dimensions has become a crucial prerequisite for ensuring the overall performance of the system.

[0003] In current multimodal perception systems, spatiotemporal data fusion is a key step in achieving high-precision environmental perception and intelligent decision-making. Existing mainstream technical solutions mainly revolve around improving "data alignment accuracy," and their typical implementation paths include: at the hardware level, employing high-precision clock synchronization mechanisms (such as clock modules based on the IEEE 1588 Precision Time Protocol, PTP) to unify the time reference of multiple sensors; at the software level, relying on feature matching algorithms (such as SIFT (Scale-Invariant Feature Transform) feature point matching, point cloud and image edge alignment, etc.) to calibrate the spatial coordinate system, thereby ensuring the consistency of multi-source heterogeneous data in the temporal and spatial dimensions at an initial level.

[0004] While the aforementioned methods can meet basic spatiotemporal alignment requirements in ideal environments, they still have significant limitations in complex real-world scenarios. On one hand, multimodal sensors are susceptible to environmental interference; for example, point cloud data from LiDAR may be obscured or shifted in rain or snow, and image frame timestamps from cameras may become disordered under strong light. On the other hand, the system also faces the risk of malicious attacks from the outside, such as injecting forged timestamps or tampering with spatial coordinate information. However, existing fusion solutions typically only possess static verification capabilities for the integrity of the original input data, lacking a real-time reliable verification mechanism for the dynamic data flow during the fusion process. This makes it difficult to effectively identify and resist tampering with intermediate states during fusion, potentially leading to distorted fusion results and subsequent misjudgments by Advanced Driver Assistance Systems (ADAS) or autonomous driving decision-making modules.

[0005] Furthermore, the temporal logical correlation between multimodal data (e.g., the sampling periods of LiDAR point cloud frames and camera image frames must be strictly matched, and the temporal consistency between GNSS positioning information and millimeter-wave radar velocity data must be reasonable) is crucial for ensuring fusion accuracy. However, current technology systems have not yet built dedicated security protection mechanisms for "temporal logical consistency." Attackers can exploit this vulnerability to disrupt the inherent logical structure of the data by inserting delayed frames, swapping the order of data frames, or forging temporal relationships, causing the fusion system to output incorrect environmental perception results, seriously threatening the safety and reliability of intelligent driving systems. Summary of the Invention

[0006] To address the problems of existing technologies, embodiments of this application provide a data fusion method, apparatus, electronic device, and storage medium. The technical solution is as follows: On the one hand, a data fusion method is provided, the method comprising: Acquire data to be fused from multimodal sensors; the data to be fused is standardized data that has been spatiotemporally aligned, and the standardized data contains at least a unified timestamp and spatial coordinate information; Based on the timestamp and spatial coordinate information in the data to be fused, the logical consistency between the data to be fused and the historical associated data in terms of time sequence and spatial relationship is verified to obtain the time sequence verification result; the historical associated data and the data to be fused are continuous in time and / or adjacent in space. If the time series verification result is abnormal, the degree of time series abnormality is determined based on the timestamp deviation between the data to be fused and the historical associated data, and / or the spatial coordinate offset between the data to be fused and the historical associated data. Based on the repair strategy corresponding to the degree of time series anomaly, the data to be fused is repaired; The repaired data to be fused is fused to generate environmental perception results.

[0007] On the other hand, a data fusion apparatus is provided, the apparatus comprising: The data acquisition module is used to acquire data to be fused from multimodal sensors; the data to be fused is standardized data that has been spatiotemporally aligned, and the standardized data contains at least a unified timestamp and spatial coordinate information; The time sequence verification module is used to verify the logical consistency between the data to be merged and the historical associated data in terms of time sequence and spatial relationship based on the timestamp and spatial coordinate information in the data to be merged, and to obtain the time sequence verification result; the historical associated data and the data to be merged are continuous in time and / or adjacent in space. The anomaly determination module is used to determine the degree of time series anomaly based on the timestamp deviation between the data to be fused and the historical associated data, and / or the spatial coordinate offset between the data to be fused and the historical associated data, when the time series verification result is abnormal. The data repair module is used to repair the data to be fused based on the repair strategy corresponding to the degree of time series anomaly. The fusion perception module is used to fuse the repaired data to generate environmental perception results.

[0008] In one exemplary embodiment, the standardized data further includes data identifiers and sensor types; the timing verification module includes: The node addition module is used to add nodes corresponding to the data to be fused in the time-series correlation graph based on the data identifier, the timestamp, the spatial coordinate information, and the sensor type; the time-series correlation graph is a historical correlation data network constructed based on the nodes corresponding to historical data. The node query module is used to query the nodes in the time-series correlation map that are associated with the nodes corresponding to the data to be fused, based on the timestamp, spatial coordinate information and sensor type of the nodes corresponding to the data to be fused, as the nodes corresponding to the historical correlation data. The continuity judgment module is used to determine, based on the temporal correlation graph, whether the temporal continuity between the node corresponding to the data to be fused and the node corresponding to the historical correlation data meets the preset continuity conditions.

[0009] In one exemplary embodiment, the data repair module includes: The time period determination module is used to determine the target time period by tracing back a preset duration based on the timestamps in the data to be fused when the degree of time anomaly is at a first preset level. The correction execution module is used to correct the timestamps and / or spatial coordinate information in the data to be fused based on the time change trend and / or spatial coordinate pattern of historical data associated with the data to be fused within the target time period.

[0010] In one exemplary embodiment, the data repair module further includes: The replacement acquisition module is used to acquire replacement data corresponding to the data to be fused from the redundant backup data of the sensor corresponding to the data to be fused when the degree of time anomaly is at a second preset level; the second preset level is higher than the first preset level. The replacement and triggering module is used to replace the data to be fused based on the replacement data and trigger a fault diagnosis process for the sensor corresponding to the data to be fused.

[0011] In one exemplary implementation, the fusion sensing module includes: The post-repair verification module is used to verify the logical consistency between the repaired data to be fused and historical associated data in terms of time sequence and spatial relationship based on the timestamp and spatial coordinate information in the repaired data to be fused, and to obtain the post-repair time sequence verification result. The repaired fusion module is used to perform fusion processing on the repaired data to be fused when the timing verification result after repair is normal.

[0012] In one exemplary embodiment, the apparatus further includes a process switching module, which is used to switch to a backup fusion processing process when the timing verification result after the repair is abnormal; the backup fusion processing process generates a temporary environmental perception result based on historical reliable data.

[0013] In one exemplary embodiment, the timing verification module includes: The signature acquisition module is used to acquire the digital signature corresponding to the data to be merged; A signature verification module is used to verify the validity of the digital signature; The post-verification verification module is used to verify the logical consistency between the data to be merged and historical associated data in terms of time sequence and spatial relationship, based on the timestamp and spatial coordinate information in the data to be merged, provided that the digital signature has been verified as valid.

[0014] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data fusion method of any of the above aspects.

[0015] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the data fusion method as described in any of the above aspects.

[0016] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the data fusion method described above.

[0017] This application's embodiments add a real-time logical verification step based on historical correlation data to the critical path of the fusion process. By forcibly comparing the current data to be fused with historical contexts that are continuous in time and / or adjacent in space, it can detect in real time temporal logic anomalies caused by sensor dynamic interference (such as point cloud jumps caused by rain and snow) or malicious attacks (such as swapping data frame order or injecting delayed frames) during the fusion process. These anomalies include disordered time sequence or spatial relationships that violate continuity rules, thus achieving accurate identification of dynamic data tampering during the fusion process. Furthermore, based on the quantified degree of anomaly, intelligent matching and execution of repair strategies are performed to correct abnormal data online, effectively resisting attack modes that cause environmental perception failure by disrupting temporal logic. The modal data fusion method of this application no longer relies entirely on front-end alignment accuracy, possessing real-time protection and dynamic fault tolerance capabilities, significantly improving the accuracy of the fusion results and the robustness of the system in complex environments. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a data fusion method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating another data fusion method provided in an embodiment of this application; Figure 3 This is a structural block diagram of the fusion process security protection module provided in the embodiments of this application; Figure 4 This is an example diagram of the temporal correlation map of multimodal data provided in an embodiment of this application; Figure 5 This is an exemplary comparison diagram of dynamic repair effects provided in an embodiment of this application; Figure 6 This is a structural block diagram of a data fusion device provided in an embodiment of this application; Figure 7 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0022] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0023] It should be noted that the technical solution proposed in this application has broad applicability, and its application scenarios include, but are not limited to, intelligent driving, robot environmental perception, and IoT data integration, among other fields that require spatiotemporal alignment and fusion processing of multi-source heterogeneous data. To facilitate a clear and specific explanation of the technical principles and implementation methods, this application will primarily use multimodal sensor data fusion in a combined assisted driving scenario as an example. This example has typical spatiotemporal data correlation and verification requirements, and can fully demonstrate the core ideas and advantages of the method in this application. For other applicable fields, the implementation logic is the same as in this embodiment, and will not be elaborated upon further.

[0024] As integrated driver assistance technologies evolve towards higher levels (L4 / L5, autonomous driving levels), the systems place higher demands on the accuracy, real-time performance, and robustness of environmental perception. Multi-sensor fusion has become a core technological path to achieve highly reliable environmental perception. Currently, systems typically compensate for the limitations of single sensors by fusing heterogeneous data from multiple sources, such as LiDAR (providing high-precision 3D point clouds), cameras (images rich in semantic information), millimeter-wave radar (range and velocity measurement data with strong anti-interference capabilities), and GNSS / IMU (Inertial Measurement Unit) (positioning and attitude timing data). Against this backdrop, multimodal data fusion is evolving from the initial "single-dimensional data stitching" to "deep spatiotemporal fusion" that emphasizes dynamic correlation. Spatiotemporal data, including timestamps, spatial coordinates, and their inherent acquisition timing relationships, has become a key factor affecting fusion accuracy.

[0025] Current mainstream technical solutions mainly focus on improving spatiotemporal alignment accuracy. Common approaches include: using high-precision clock synchronization (such as the IEEE 1588 PTP protocol) at the hardware level, and using feature matching algorithms (such as SIFT, point cloud and image edge alignment) at the software level to correct spatial deviations and achieve spatiotemporal benchmark unification of multi-source data; then using fusion algorithms such as Kalman filtering, particle filtering, or attention mechanisms to extract and synthesize features from the aligned data to generate environmental perception results; in terms of security, traditional encryption algorithms (such as AES-256 (Advanced Encryption Standard-256)) and role-based access control (RBAC) are mainly relied upon to achieve static protection of the original data.

[0026] However, while existing solutions can basically meet the requirements of spatiotemporal alignment and static security, they are insufficient to address the dynamic and real-time requirements of advanced driver assistance systems (ADAS) for data security. Their shortcomings are mainly reflected in the following aspects: First, they lack protection for the dynamic integrity of spatiotemporal data during the fusion process, failing to effectively identify dynamic data corruption caused by sensor disturbances (such as rain and snow obstruction, or blinding due to strong light) or malicious injection (such as timestamp forgery or coordinate tampering). Second, they neglect security protection for the temporal correlation logic between multimodal data; attackers can disrupt the fusion logic by scrambling, delaying, or rearranging the order of data frames, leading to perception failure. Third, there is a conflict between security mechanisms and system real-time requirements. Traditional encryption and verification algorithms, due to their high computational overhead and long processing time (often exceeding 10ms), cannot meet the millisecond-level (10–50ms) real-time fusion requirements of ADAS, often forcing the system to make trade-offs between security and real-time performance, thus creating potential security vulnerabilities.

[0027] In view of this, this application proposes a data fusion method. This method provides systematic security for the entire process of multimodal data fusion by constructing three collaborative modules: spatiotemporal data preprocessing, security protection during the fusion process, and real-time verification and dynamic repair. Its operation is supported by a specific hardware architecture, mainly including: a combined auxiliary driving domain controller integrating a multi-core CPU (Central Processing Unit) and an FPGA (Field-Programmable Gate Array) acceleration unit (compatible with mainstream computing platforms such as NVIDIA Orin and Horizon Journey 5); a high-precision clock synchronization module capable of achieving microsecond-level (≤1μs) synchronization accuracy (supporting the IEEE 1588 PTP protocol); and a multimodal sensor group including LiDAR, camera, millimeter-wave radar, and GNSS / IMU. The core of this method lies in real-time detection and repair of potential dynamic tampering of spatiotemporal data at each stage of data calibration, fusion, and invocation, ensuring its integrity; establishing a specialized protection mechanism for temporal logical associations to effectively identify and block temporal disorder attacks and maintain the consistency of multi-source data collaboration; and achieving efficient protection of spatiotemporal data while strictly meeting the system's millisecond-level real-time requirements through the design of lightweight security algorithms. This addresses the issues of integrity loss in the dynamic fusion process of spatiotemporal data, the lack of security protection for temporal correlations, and the prominent contradiction between security measures and real-time requirements.

[0028] For details, please refer to Figure 1 The diagram shows a flowchart of a data fusion method provided in this application embodiment. This method is geared towards multimodal data processing scenarios on the vehicle side, and is as follows: S1, the multimodal sensor group synchronously collects raw data, performs spatiotemporal alignment and standardization, and outputs standardized data; S2, the standardized data is lightly encrypted and a digital signature is generated, forming a secure data packet including encrypted data and a signature; S3, the digital signature of the data packet is verified; if it fails, it is determined to be a severe anomaly, and the process jumps directly to S5; if it passes, its temporal logical continuity with historical data is further verified. If the temporal sequence is normal, the process proceeds to S4; if the temporal sequence is abnormal, the process proceeds to S5 according to its severity; S4, the verified secure data packet is decrypted, and the decrypted data is input into the multimodal fusion algorithm to generate the final environmental perception result for decision-making; S5, a graded repair is initiated based on the degree of anomaly: for minor anomalies, the data is corrected through the algorithm; for severe anomalies, the sensor redundancy backup is called for replacement, and the repaired data is returned to S3 for re-verification; if the repair fails, the system triggers a level one alarm, switches to the backup fusion model, and records the anomaly log.

[0029] To provide clear and implementable technical support for the above-mentioned method flow from S1 to S5, this application achieves this through the collaborative operation of three core modules: spatiotemporal data preprocessing, fusion process security protection, and real-time verification and dynamic repair. The specific functions and collaboration mechanisms of each module are described below. The spatiotemporal data preprocessing module is directly connected to the multimodal sensor group and is responsible for executing step S1 in the process flow. This module uses a high-precision clock synchronization module to provide a unified time reference for all sensors, calibrates spatial coordinate deviations through feature point matching algorithms (e.g., matching based on point cloud normal vectors and image texture features), outputs standardized data containing unique data identifiers, standard timestamps, precise spatial coordinates, and sensor type identifiers, and transmits it to subsequent stages. The fusion process security protection module is typically integrated within the FPGA acceleration unit of the domain controller, corresponding to step S2, and is responsible for receiving and protecting the standardized data. The core tasks of this module include two aspects: first, implementing lightweight encryption using a simplified and optimized AES-Lite (Advanced Encryption Standard Lite) algorithm to keep the encryption time for a single frame of data within 1 millisecond; second, generating a digital signature, which involves calculating a joint hash value for each frame of data based on "data ID (Identifier) ​​- timestamp - spatial coordinates - sensor key," using a simplified version of the SHA-256 (Secure Hash Algorithm-256) algorithm to ensure that the signature time does not exceed 0.5 milliseconds. At this point, the data is encapsulated into a secure data packet with a digital signature. The real-time verification and dynamic repair module runs through and dominates the decision-making logic of S3 and S5, and determines whether the process proceeds to S4. This module has two key sub-units. Before data fusion, the timing logic verification submodule first verifies the validity of the digital signature, and then verifies the temporal continuity between the current data node and historical nodes based on the constructed timing correlation graph (for example, requiring the time difference between data frames from different sensors to be within ±0.5 milliseconds of the sampling period). Only after all of the above passes can the data proceed to step S4 for decryption and fusion. Once an anomaly is detected during verification, the dynamic repair submodule immediately initiates graded repair in step S5 according to the anomaly type and severity. For minor anomalies such as timestamp deviation ≤2ms or spatial coordinate offset ≤3cm, the system performs linear fitting and coordinate correction based on the trend of the most recent multiple frames (e.g., 3 frames) of historical data. For severe anomalies such as signature failure, ID tampering, or coordinate offset >5cm, the system automatically calls the sensor redundancy backup unit (real-time caching of the most recent 5 frames of data) to replace the data and simultaneously triggers sensor fault diagnosis. The repaired data is returned to S3 for re-verification. If the final repair fails, the system will execute a security degradation strategy, including triggering a level 1 alarm, switching to a backup fusion model, and recording detailed anomaly logs.Through the above modular design and precise collaboration, this application constructs a closed-loop management system covering the entire process from data preprocessing, security hardening, real-time verification to dynamic anomaly repair. While strictly meeting the millisecond-level real-time requirements, it systematically ensures the integrity, temporal logic consistency and security of data during the dynamic fusion process.

[0030] By constructing a real-time verification and dynamic repair mechanism, spatiotemporal data tampering behaviors (such as timestamp forgery, coordinate offset, and data frame replacement) occurring during the fusion process can be accurately identified and automatically corrected, thus ensuring the accuracy and integrity of the data at the dynamic fusion level. Simultaneously, by establishing a temporal correlation security protection system and constructing a temporal correlation graph that binds and verifies data identifiers, timestamps, spatial coordinates, and sensor types, the system effectively defends against fusion logic disorder caused by temporal disruption attacks, ensuring the consistency of multi-source data collaboration. Based on achieving the above security objectives, lightweight encryption and verification algorithms, combined with hardware acceleration design, strictly control the total time of a single security processing step to within 5 milliseconds, meeting the real-time fusion requirement of ≤50 milliseconds for advanced driver assistance systems, thus successfully preventing security protection from becoming a system performance bottleneck. It breaks through the limitations of existing technologies that only provide static protection for raw data, and achieves dynamic and full-process security protection for the fusion process. Through lightweight design, it fundamentally reconciles the inherent contradiction between security and real-time performance, significantly improves the reliability, robustness and anti-attack capability of the combined assisted driving system's environmental perception, and effectively reduces driving risks caused by data security issues.

[0031] Please see Figure 2 The diagram illustrates a data fusion method according to an embodiment of this application. It should be noted that while this specification provides method steps as shown in the embodiments or flowcharts, more or fewer steps may be included based on conventional or non-inventive methods. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or product execution, the method can be executed sequentially according to the embodiments or accompanying drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include: S201, acquire the data to be fused from the multimodal sensor; the data to be fused is standardized data that has been spatiotemporally aligned, and the standardized data contains at least a unified timestamp and spatial coordinate information.

[0032] Specifically, the multimodal sensor group (including lidar, camera, millimeter-wave radar and GNSS / IMU) synchronously acquires raw data; the spatiotemporal data preprocessing module uses a high-precision clock synchronization mechanism to unify the time reference of each sensor, and uses a feature point matching algorithm to calibrate the spatial coordinates, outputting standardized data containing a unique ID, standard timestamp, accurate spatial coordinates and sensor identification.

[0033] S203, based on the timestamp and spatial coordinate information in the data to be merged, verify the logical consistency between the data to be merged and the historical related data in terms of time sequence and spatial relationship, and obtain the time sequence verification result.

[0034] In this system, historical associated data and the data to be fused are temporally continuous and / or spatially adjacent. Specifically, historical associated data can be stored in a circular buffer or a time-series database. The selection logic is as follows: in the time dimension, select the most recent frames whose timestamp difference with the data to be fused is less than a set threshold (e.g., twice the highest sampling interval of the sensor); in the spatial dimension, select data whose spatial coordinate distance from the data to be fused is less than a preset neighborhood radius (e.g., calculated based on the vehicle's maximum possible displacement speed).

[0035] The logical consistency verification specifically includes: determining whether the timestamp of the data to be fused is later than the timestamp of the associated historical data, and whether the time difference is within the expected range of the sensor sampling period; at the same time, determining whether the change of the spatial coordinates of the data to be fused relative to the coordinates of the historical associated data is within the reasonable motion trajectory tolerance range predicted based on historical velocity and acceleration information.

[0036] The timing verification result is a measure of whether the current data frame can be smoothly embedded into the spatiotemporal continuum composed of historical data. Its essence is to verify whether the mapping of the continuity of motion in the physical world (such as the gradual changes in vehicle position and speed) in the data stream is disrupted.

[0037] Specifically, obtain the digital signature corresponding to the data to be merged; verify the validity of the digital signature; and, if the digital signature is verified to be valid, verify the logical consistency between the data to be merged and historical related data in terms of time sequence and spatial relationship based on the timestamp and spatial coordinate information in the data to be merged.

[0038] Specifically, an additional checkpoint for verifying the authenticity and integrity of the data source is added before the timing logic verification. Digital signature verification can defend against attacks that could tamper with or forge data during transmission. Specifically, using the sensor's unique private key, a cryptographic hash operation is performed on the data packet containing core fields such as data identifiers, timestamps, and spatial coordinates. For example, a simplified version of the ECDSA (Elliptic Curve Digital Signature Algorithm) algorithm is used to generate a signature. During verification, the fusion system uses the corresponding public key to decrypt the signature and compares the calculated hash value with the actual hash value of the data packet. This process is completed before the data is passed to the timing logic verification module; if verification fails, the data is directly blocked from entering the subsequent fusion process.

[0039] In practice, the generation and verification of digital signatures can employ lightweight cryptographic algorithms, such as the elliptic curve digital signature algorithm ECDSA, to adapt to the computing power constraints of automotive-grade chips. The signature content can cover core fields such as data identifiers, timestamps, and coordinates.

[0040] In practice, digital signature verification can be linked to sensor identity. Each sensor has a unique key, and the central processing unit holds the corresponding public key. In this way, sensor identity authentication is completed simultaneously with signature verification, preventing unauthorized devices from accessing the data fusion network.

[0041] Specifically, the acquisition of digital signatures relies on a security protection module for the fusion process. Specifically, such as... Figure 3 As shown, the security protection module for this fusion process includes a data receiving unit, an AES-Lite encryption unit, a simplified SHA-256 signature unit, and a secure data packet output unit, with core computing power supported by an FPGA acceleration unit. Its workflow is as follows: after the data receiving unit acquires standardized data and sensor keys, the data is sequentially encrypted by the AES-Lite encryption unit, and then the SHA-256 signature unit generates a signature. The entire process is supported by parallel processing provided by the FPGA acceleration unit; the digital signature is output through the secure data packet output unit. By leveraging the hardware parallel processing capabilities of the FPGA, the total time for encryption and signing is optimized from 50ms in software implementation to less than 8ms, thereby significantly improving processing efficiency while ensuring the authenticity of the data source through signature verification.

[0042] As can be seen from the above technical solutions of the embodiments of this application, by using signature verification as a prerequisite for timing verification, a large amount of obvious malicious data can be filtered out in advance, preventing it from entering the more resource-intensive graph construction and query process, thus improving the overall system processing efficiency. Signature verification and timing verification complement each other, forming a dual protection against tampering and out-of-order execution, which can resist a wider attack surface.

[0043] Specifically, standardized data also includes data identifiers and sensor types. Based on data identifiers, timestamps, spatial coordinate information, and sensor types, nodes corresponding to the data to be fused are added to the temporal correlation graph. The temporal correlation graph is a historical correlation data network constructed based on the nodes corresponding to historical data. Based on the timestamps, spatial coordinate information, and sensor types of the nodes corresponding to the data to be fused, nodes associated with the nodes corresponding to the data to be fused in the temporal correlation graph are queried as the nodes corresponding to the historical correlation data. Based on the temporal correlation graph, it is determined whether the temporal continuity between the nodes corresponding to the data to be fused and the nodes corresponding to the historical correlation data meets the preset continuity conditions.

[0044] The temporal correlation graph transforms discrete historical data points into a network model containing spatiotemporal relationships. Each node is a multidimensional snapshot of a data entity, while edges represent transfer relationships that conform to physical laws (such as temporal sequence, spatial adjacency, and sensor collaboration). This graph-based representation enables the system to quickly find historical correlations and assess continuity through graph traversal or graph matching.

[0045] Specifically, the temporal correlation graph is stored in memory in real time as a graph data structure. In addition to data identifiers, timestamps, spatial coordinate information, and sensor types, node attributes may also include data confidence weights. After a new node is added, the system automatically creates or updates the edges between the node and existing nodes based on the proximity of timestamps and spatial coordinates (e.g., using the k-nearest neighbor algorithm). The weights of the edges reflect the strength of the spatiotemporal correlation.

[0046] The preset continuity conditions include, but are not limited to: the existence of a directed edge from a historical associated data node to the data node to be fused; the time interval represented by the edge is consistent with the inherent collaborative sampling period of the two types of sensors; and the deviation between the spatial coordinate change transmitted along the edge and the predicted value based on the physical motion model is less than a threshold.

[0047] Specifically, examples of time-series correlation graphs include... Figure 4 As shown, this graph uses nodes to represent multimodal data units with different attributes. Each node is associated with a unique data ID and a precise timestamp (e.g., 16:00:00.000), and carries multidimensional attributes such as sensor type, laser category, or specific radar model. Directed edges between nodes represent the temporal interactions and dependencies between different data units. The time interval labeled 'I' on the edge (e.g., 10ms, 20ms) quantifies the delay in the association between data units. For example, an edge labeled 10ms from ID001 to ID002 indicates a time difference of 10 milliseconds between the associations. This graph structure clearly presents and verifies the dynamic correlation and logical consistency of multi-source data in the time dimension.

[0048] In practice, edges in the temporal correlation graph can carry rich metadata, such as transition probabilities (based on historical statistics) and physical constraints (such as displacement ranges determined by velocity limits). When determining continuity, not only are timestamps checked for increments, but the rationality and probability of a transition can also be judged by the edge weights. For example, if the edge weight from LiDAR node A to camera node B is very high, it means that these two types of data have historically been highly correlated; if newly arrived data of type B cannot be correlated with the most recent data of type A through high-weighted edges, even if the timestamps are continuous, it may be judged as abnormal due to violation of the correlation rule.

[0049] In practice, the temporal correlation graph can be dynamically updated and pruned. Old historical nodes are removed based on strategies (such as time windows or importance) to control the graph size and ensure real-time query performance. Simultaneously, new relationships formed by successfully fused data are fed back into the temporal correlation graph to enhance its completeness and accuracy, enabling learning and evolution.

[0050] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application, by introducing a temporal correlation graph of multimodal data, realize multidimensional binding and dynamic verification of "data ID-timestamp-spatial coordinates-sensor type". This enables accurate identification of temporal disruption attacks such as data frame order swapping and delayed data insertion, effectively filling the gap in security protection at the temporal logic level in existing technologies, and fundamentally ensuring the coherence and stability of the multimodal fusion process logic.

[0051] S205, if the time series verification result is abnormal, determine the degree of time series abnormality based on the timestamp deviation between the data to be fused and the historical associated data, and / or the spatial coordinate offset between the data to be fused and the historical associated data.

[0052] Specifically, by calculating timestamp deviation and spatial coordinate offset, anomalies are moved from a qualitative to a quantitative level. This quantification, i.e., the degree of time-series anomalies, serves as the decision-making basis for subsequent graded repair strategies. This ensures that repair actions (such as correction or replacement) match the actual severity of the anomaly, avoiding resource waste from over-repair or security risks from under-repair.

[0053] S207, based on the repair strategy corresponding to the degree of time series anomaly, repair the data to be fused.

[0054] The repair strategy is directly correlated with the degree of temporal anomaly. Specifically, based on the credible historical context and the specific manifestation of the anomaly, an appropriate intervention method is intelligently selected to restore the credible state of the data, thus achieving dynamic fault tolerance.

[0055] Specifically, when the time-series anomaly level is at the first preset level, a target time period is determined by tracing back a preset duration based on the timestamps in the data to be fused; the timestamps and / or spatial coordinate information in the data to be fused are corrected based on the temporal change trends and / or spatial coordinate patterns of historical data associated with the data to be fused within the target time period. When the time-series anomaly level is at the second preset level, replacement data corresponding to the data to be fused is obtained from the redundant backup data of the sensor corresponding to the data to be fused; the second preset level is higher than the first preset level; the data to be fused is replaced based on the replacement data, and a fault diagnosis process for the sensor corresponding to the data to be fused is triggered.

[0056] The first preset threshold typically corresponds to minor, explainable, and predictable deviations caused by environmental noise, transient interference, etc.; the second preset threshold corresponds to significant, unreliable, and severe deviations that may be caused by sensor malfunctions, malicious data injection, etc. The threshold ranges for both can be dynamically configured through experimental statistics and security policies.

[0057] Specifically, the first and second preset severity levels are divided by preset anomaly threshold ranges. For example, when the absolute value of the timestamp deviation is ≤ T1 (e.g., 2ms) and the spatial coordinate offset magnitude is ≤ D1 (e.g., 3cm), it is judged as the first preset severity level (mild anomaly); when the absolute value of the timestamp deviation is > T1, or the spatial coordinate offset magnitude is > D1, or the digital signature verification fails, or the data identifier is illegal, it is judged as the second preset severity level (severe anomaly). Thresholds T1 and D1 can be experimentally calibrated or dynamically adjusted according to sensor performance indicators (e.g., calibration error, measurement noise) and specific application scenarios (e.g., highways, urban roads).

[0058] The forward tracing of a preset time period refers to using the principle of temporal locality for repair, meaning that changes in physical states over a short period are usually smooth and trend-like. By analyzing historical data within a short window preceding outlier data points, a temporal or spatial change model (such as linear regression or Kalman filter prediction) is fitted, and this model is used to correct the outliers. This method has low computational cost and fast response, making it suitable for scenarios with extremely high real-time requirements.

[0059] In practice, a predictive model can be introduced for the repair strategy corresponding to the first preset level. Specifically, in addition to simple linear trends, a lightweight recurrent neural network trained on historical data can be used to predict the reasonable range of data at the next moment, and the predicted values ​​can be used to correct outliers with weights.

[0060] In practice, for the second preset level of repair strategy, the sources of redundant backup data can be diverse: they can be hardware redundancy (backup sensors of the same type), temporal redundancy (data frames cached earlier by the sensor that have been verified), or modal redundancy (equivalent information derived from other types of sensors, such as image patch features of the corresponding location derived from radar data). After the fault diagnosis process is triggered, the diagnosis results can be fed back to the fusion system for long-term adjustment of the trust level in the sensor.

[0061] Specifically, verification and repair are achieved through the collaboration of the fusion process security protection module and the real-time verification and dynamic repair module. Specifically, after receiving standardized data, the fusion process security protection module encrypts it using the FPGA-accelerated AES-Lite algorithm and generates a digital signature using a simplified SHA-256 algorithm. This data is then packaged into a secure data packet containing the encrypted data and the digital signature and transmitted to subsequent stages. The timing logic verification submodule in the real-time verification and dynamic repair module takes over this secure data packet and verifies the validity of the digital signature (by comparing it with the pre-stored key on the sensor). If the signature verification fails, it is immediately identified as a severe anomaly, and the process jumps directly to the dynamic repair and alarm stage. If the signature verification passes, this submodule will further perform a continuity verification based on the timing correlation graph. Only after the timing logic is confirmed to be normal will the data be allowed to enter the normal fusion processing flow; if the verification finds a timing anomaly, the system will guide the process back to the dynamic repair and alarm stage according to the specific severity of the anomaly (mild or severe). The dynamic repair submodule addresses minor anomalies (such as single-frame data timestamp deviation ≤ 2ms, spatial coordinate offset ≤ 3cm) by employing linear fitting and coordinate correction algorithms to repair the data. For severe anomalies (such as data ID tampering, spatial coordinate offset > 5cm, signature verification failure), it directly replaces the data with redundant backup data from the sensor. All repaired or replaced data must be returned to the real-time verification and dynamic repair module for re-verification, forming a closed loop. If both repair and replacement fail (e.g., redundant data is also unavailable), the system will trigger a Level 1 alarm: suspending the current main fusion algorithm and switching to a backup fusion model based on historical reliable data to output temporary sensing results, ensuring basic system operation. Simultaneously, detailed logs are recorded, including the anomaly time, sensor type, and data characteristics, providing a basis for subsequent troubleshooting.

[0062] In practical implementation, adjustments can be made to accommodate different hardware configurations and security standards. For example, at the encryption algorithm level, the AES-Lite algorithm can be replaced with the lightweight SM4 algorithm compliant with national cryptographic standards. Calculations show that this replacement increases the single-frame data encryption time by no more than 0.5ms, fully meeting system real-time requirements while maintaining the same level of security. Furthermore, if the deployed vehicle does not have a dedicated sensor redundancy backup unit, the system can use a software repair scheme based on multi-sensor cross-validation as an alternative. This scheme utilizes the physical consistency between different sensor data for mutual verification and correction (e.g., using millimeter-wave radar distance data to correct the spatial offset of lidar point clouds, or using GNSS positioning data to verify the timestamps of camera image frames). Its repair accuracy decreases by no more than 3% compared to hardware redundancy-based schemes, without incurring additional hardware costs, thus making it well-suited for cost-sensitive combined driver assistance systems.

[0063] To verify the actual effectiveness of the aforementioned graded restoration mechanism, this application's embodiments use the spatial restoration effect of lidar point clouds as an example for quantitative evaluation. For example... Figure 5 As shown, the spatial offset of the point cloud before repair reached 8 cm, resulting in significant positional deviations in the scene presentation. After triggering a light repair, the system corrected based on historical data patterns, reducing the offset to 1.2 cm in just 0.8 ms, significantly improving spatial accuracy. In the heavy repair scenario, the system called redundant backup data for replacement. Although the time increased slightly to 1.2 ms, the offset was further optimized to 0.8 cm, achieving more accurate correction. The results show that this repair scheme can effectively reduce spatial offset within millisecond-level latency, balancing processing efficiency and data accuracy, verifying the feasibility and effectiveness of the hierarchical repair strategy.

[0064] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application achieve efficient self-repair capability through hierarchical repair. Minor repair uses computational correction, saving the overhead of calling backup resources; severe repair decisively switches the data source to ensure input quality, and simultaneously initiates diagnostics to prevent problems before they occur. This significantly improves data reliability, ensures the continuous and stable operation of the system, and optimizes the system's computational and resource utilization efficiency while ensuring security. A fault isolation and handling mechanism (triggered diagnostics) is clearly defined when data is unavailable, linking data-level repair with sensor device-level health management, thereby improving the maintainability and reliability of the entire sensing system. It supports two repair schemes: sensor redundancy backup and multi-sensor cross-verification, enhancing its universality and industrialization potential across different vehicle configurations. Furthermore, through the synergistic optimization of algorithm lightweighting and hardware acceleration, the processing time of the entire safety process is strictly controlled, successfully eliminating potential real-time bottlenecks in safety procedures and fully meeting the stringent requirements of advanced driver assistance systems for instantaneous response. In addition, it supports flexible switching of encryption algorithms compliant with domestic and international standards, adapting to global market compliance requirements. Its hardware acceleration design is compatible with mainstream automotive-grade computing platforms, eliminating the need for customized dedicated hardware, effectively reducing deployment costs and accelerating industrial application.

[0065] In practice, repair strategies can include information-level repair and trust-level repair. Information-level repair directly corrects timestamps or coordinate values. Trust-level repair is more in-depth; for example, if a camera exhibits mild timing anomalies in multiple consecutive frames, although the data is usable after repair, the system can dynamically reduce the weight coefficient of that sensor in the fusion algorithm, thereby achieving soft repair at the decision-making level.

[0066] S209: Perform fusion processing on the repaired data to be fused to generate environmental perception results.

[0067] Specifically, the fusion process security protection module decrypts the transmitted security data packets to recover usable standardized data; this data is then input into a multimodal fusion algorithm (such as an attention-based fusion model) for integrated computation; the algorithm outputs environmental perception results containing key information such as obstacle positions and driving path references, which will directly serve subsequent combined assisted driving decisions and path planning.

[0068] Specifically, based on the timestamps and spatial coordinates in the repaired data to be fused, the logical consistency between the repaired data to be fused and historical related data in terms of temporal order and spatial relationship is verified to obtain the repaired temporal verification result. If the repaired temporal verification result is normal, the repaired data to be fused is fused. If the repaired temporal verification result is abnormal, the process switches to the backup fusion processing flow. The backup fusion processing flow generates temporary environmental perception results based on reliable historical data.

[0069] In practice, the timing verification after repair employs the same or more stringent verification logic as step S203. If the repaired data still fails verification, or if N consecutive repair attempts (e.g., 3 times) fail, the timing verification result after repair is deemed abnormal. At this point, the system switches to the backup fusion processing flow. This flow, for example, can disable data sources that have recently exhibited abnormal behavior, and only use historical data that has consistently proven reliable over a period of time (e.g., data from millimeter-wave radar and IMU that have not triggered alarms) to generate a conservative but robust environmental perception result based on simplified motion extrapolation or obstacle tracking algorithms. This result is used to trigger a safe stop of the vehicle or alert the driver to take over, while simultaneously recording and reporting fault codes.

[0070] In practice, the backup fusion processing flow is usually a simplified but highly reliable fusion model. For example, when the main multimodal fusion algorithm cannot run due to data quality issues, it can be switched to a basic obstacle detection and tracking flow that only relies on millimeter-wave radar and vehicle dynamics models. Although the perception dimension is reduced, it is sufficient to support emergency safety operations (such as deceleration and stopping).

[0071] In practice, the switching action can be designed as a smooth transition. For example, the outputs of the main fusion algorithm and the backup algorithm can be weighted and mixed for a period of time. The weights are dynamically adjusted according to the abnormal state of the input data of the main algorithm, so as to avoid the impact of the change in the perception result on the downstream control module.

[0072] Specifically, the repaired data must be re-verified to prevent the repair process itself from introducing new errors or causing incomplete repairs. In the event of a repair failure, ensure the system still provides basic functionality.

[0073] It significantly enhances the system's resilience and survivability. Even in the face of continuous or complex attacks / failures, the system can recover through a cycle of repair and retesting, avoid complete failure through degraded operation, buy time and the possibility for safe stopping or alerting the driver to take over, and provide a definite safety bottom line for the system.

[0074] As can be seen from the above technical solutions of the embodiments of this application, the embodiments of this application, by constructing a dynamic security protection system covering the entire process from data preprocessing, lightweight encryption, real-time verification to fusion processing, break through the limitations of existing technologies that only provide static protection for raw data. Specifically, by comprehensively utilizing technologies such as lightweight encryption, data signature, and real-time integrity verification, various data tampering and injection attacks occurring during the fusion process are effectively identified and blocked, thereby significantly improving the dynamic credibility of multi-source data. On this basis, time-series verification greatly enhances the system's ability to resist time-series disturbance attacks, improving the overall accuracy and robustness of environmental perception results, and providing a solid data security guarantee for reliable decision-making in advanced driver assistance systems.

[0075] Corresponding to the data fusion methods provided in the above embodiments, this application also provides a data fusion device. Since the data fusion device provided in this application corresponds to the data fusion methods provided in the above embodiments, the implementation methods of the aforementioned data fusion methods are also applicable to the data fusion device provided in this embodiment, and will not be described in detail in this embodiment.

[0076] Please see Figure 6 The diagram shown is a structural schematic of a data fusion device provided in an embodiment of this application. This device has the function of implementing the data fusion method described in the above-described method embodiments. This function can be implemented by hardware or by hardware executing corresponding software. Figure 6 As shown, the device may include: The data acquisition module 610 is used to acquire data to be fused from multimodal sensors; the data to be fused is standardized data that has been spatiotemporally aligned, and the standardized data contains at least a unified timestamp and spatial coordinate information; The timing verification module 620 is used to verify the logical consistency of the data to be merged and historical related data in terms of time sequence and spatial relationship based on the timestamp and spatial coordinate information in the data to be merged, and to obtain the timing verification result; the historical related data and the data to be merged are continuous in time and / or adjacent in space. The anomaly determination module 630 is used to determine the degree of time series anomaly based on the timestamp deviation between the data to be fused and the historical associated data, and / or the spatial coordinate offset between the data to be fused and the historical associated data, when the time series verification result is abnormal. The data repair module 640 is used to repair the data to be fused based on the repair strategy corresponding to the degree of time series anomaly. The fusion perception module 650 is used to perform fusion processing on the repaired data to be fused, and generate environmental perception results.

[0077] In one exemplary implementation, the standardized data further includes data identifiers and sensor types; the timing verification module includes: The node addition module is used to add nodes corresponding to the data to be fused in the time-series correlation graph based on data identifiers, timestamps, spatial coordinate information, and sensor types; the time-series correlation graph is a historical correlation data network constructed based on the nodes corresponding to historical data; The node query module is used to query nodes in the time series correlation map that are associated with the nodes corresponding to the nodes of the data to be fused, based on the timestamp, spatial coordinate information and sensor type of the nodes corresponding to the nodes of the data to be fused, as the nodes corresponding to the historical correlation data. The continuity judgment module is used to determine whether the temporal continuity between the corresponding node of the data to be fused and the corresponding node of the historical associated data meets the preset continuity conditions based on the temporal correlation graph.

[0078] In one exemplary implementation, the data repair module includes: The time period determination module is used to determine the target time period by tracing back a preset duration based on the timestamps in the data to be fused when the degree of time anomaly is at the first preset level. The correction execution module is used to correct the timestamps and / or spatial coordinate information in the data to be fused based on the time change trends and / or spatial coordinate patterns of historical data associated with the data to be fused within the target time period.

[0079] In one exemplary embodiment, the data repair module further includes: The replacement acquisition module is used to acquire replacement data corresponding to the data to be fused from the redundant backup data of the sensor corresponding to the data to be fused when the degree of timing anomaly is at a second preset level; the second preset level is higher than the first preset level. The replacement and triggering module is used to replace the data to be fused based on the replacement data and trigger the fault diagnosis process for the sensor corresponding to the data to be fused.

[0080] In one exemplary implementation, the fusion sensing module includes: The post-repair verification module is used to verify the logical consistency between the repaired data to be merged and historical related data in terms of time sequence and spatial relationship based on the timestamp and spatial coordinate information in the repaired data to be merged, and to obtain the post-repair time sequence verification result. The post-repair fusion module is used to perform fusion processing on the repaired data to be fused when the timing verification result after repair is normal.

[0081] In one exemplary embodiment, the apparatus further includes a process switching module, which is used to switch to a backup fusion processing process when the timing verification result after repair is abnormal; the backup fusion processing process generates temporary environmental perception results based on historical reliable data.

[0082] In one exemplary embodiment, the timing verification module includes: The signature acquisition module is used to acquire the digital signature corresponding to the data to be merged; The signature verification module is used to verify the validity of digital signatures; The post-verification verification module is used to verify the logical consistency of the data to be merged with historical related data in terms of time sequence and spatial relationship, based on the timestamp and spatial coordinate information in the data to be merged, provided that the digital signature has been verified as valid.

[0083] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0084] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement any of the data fusion methods provided in the above method embodiments.

[0085] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0086] The method embodiments provided in this application can be executed in a computer terminal, server or similar computing device, that is, the above-mentioned electronic device may include a computer terminal, server or similar computing device. Figure 7This is a hardware structure block diagram of a computer device for running a data fusion method provided in an embodiment of the present invention, such as... Figure 7 As shown, the internal structure of this computer device may include, but is not limited to, a processor, a network interface, and a memory. The processor, network interface, and memory within the computer device can be connected via a bus or other means, as illustrated in the embodiments of this specification. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0087] The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device. The network interface may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.). Memory is the storage device in the computer device used to store programs and data. It is understood that the memory here can be a high-speed RAM storage device, or a non-volatile storage device, such as at least one disk storage device; optionally, it can also be at least one storage device located remotely from the aforementioned processor. The memory provides storage space, which stores the operating system of the electronic device, including but not limited to: Windows (an operating system), Linux (an operating system), Android (a mobile operating system), iOS (a mobile operating system), etc., which are not limited in this invention; and the storage space also stores one or more instructions suitable for being loaded and executed by the processor, which can be one or more computer programs (including program code). In the embodiments of this specification, the processor loads and executes one or more instructions stored in the memory to implement the data fusion method provided in the above method embodiments.

[0088] Embodiments of this application also provide a computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a data fusion method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the data fusion methods provided in the above-described method embodiments.

[0089] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0090] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0092] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0093] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data fusion method characterized by, The method comprises: acquiring to-be-fused data from a multi-modal sensor; the to-be-fused data is spatio-temporally aligned standardized data, and the standardized data at least contains uniform timestamp and spatial coordinate information; based on the timestamp and spatial coordinate information in the to-be-fused data, checking logical consistency of the to-be-fused data and historical associated data in time sequence and spatial relationship to obtain a time sequence checking result; the historical associated data is continuous in time and / or adjacent in space with the to-be-fused data; in the case that the time sequence checking result is abnormal, determining a time sequence abnormality degree based on a timestamp deviation value between the to-be-fused data and the historical associated data, and / or a spatial coordinate offset value between the to-be-fused data and the historical associated data; based on a repair strategy corresponding to the time sequence abnormality degree, repairing the to-be-fused data; performing fusion processing on the to-be-fused data that has completed repair to generate an environment perception result.

2. The method of claim 1, wherein, The standardized data further comprises data identification and sensor type; and the checking of logical consistency of the to-be-fused data and historical associated data in time sequence and spatial relationship based on the timestamp and spatial coordinate information in the to-be-fused data comprises: based on the data identification, the timestamp, the spatial coordinate information and the sensor type, adding a node corresponding to the to-be-fused data in a time sequence association graph; the time sequence association graph is a historical associated data network constructed based on nodes corresponding to historical data; based on the timestamp, the spatial coordinate information and the sensor type of the node corresponding to the to-be-fused data, querying nodes associated with the node corresponding to the to-be-fused data in the time sequence association graph as nodes corresponding to the historical associated data; based on the time sequence association graph, judging whether time sequence continuity between the node corresponding to the to-be-fused data and the node corresponding to the historical associated data satisfies a preset continuity condition.

3. The method of claim 1, wherein, The repairing of the to-be-fused data based on the repair strategy corresponding to the time sequence abnormality degree comprises: in the case that the time sequence abnormality degree belongs to a first preset degree, tracing back a target time period based on the timestamp in the to-be-fused data by a preset time length; based on a time variation trend and / or spatial coordinate rule of historical data associated with the to-be-fused data in the target time period, correcting the timestamp and / or spatial coordinate information in the to-be-fused data.

4. The method of claim 3, wherein, The repairing of the to-be-fused data based on the repair strategy corresponding to the time sequence abnormality degree further comprises: in the case that the time sequence abnormality degree belongs to a second preset degree, acquiring replacement data corresponding to the to-be-fused data from redundant backup data of a sensor corresponding to the to-be-fused data; the second preset degree is higher than the first preset degree; based on the replacement data, replacing the to-be-fused data and triggering a fault diagnosis process for the sensor corresponding to the to-be-fused data.

5. The method according to any one of claims 1 to 4, characterized in that, The fusion processing on the to-be-fused data that has completed repair comprises: The method further comprises: In a case where the repaired time sequence checking result is normal, the repaired to-be-fused data is fused.

6. The method of claim 5, wherein, The method further comprises: In a case where the repaired time sequence checking result is abnormal, a backup fusion processing procedure is switched to; the backup fusion processing procedure generates a temporary environment perception result based on historical reliable data.

7. The method of claim 1, wherein, The method further comprises: The method further comprises: The method further comprises: The method further comprises:

8. A data fusion apparatus characterized by comprising: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises:

9. An electronic device, comprising: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The method further comprises: The

Images