Data fusion method and device of sensor

Abstract

The invention provides a data fusion method and device of a sensor, relates to the technical field of data acquisition of an automotive electronic system, is applied to an AIO architecture, and comprises the following steps: adding a target clock timestamp to at least one kind of sensor data from a hardware layer through a kernel space, and transmitting the sensor data after time synchronization to a user layer; discarding the data packet in the first abnormal state according to the target clock timestamp and the preset length requirement of each type of sensor data, and reporting an error report; extracting at least one target clock timestamp from the sensor data, discarding a data packet corresponding to the target clock timestamp in the second abnormal state, and reporting an error report; performing aggregation processing on the at least one sensor data according to the target clock timestamp and the error report, and performing fusion according to a preset fusion rule to generate a standard interface for providing access for the target application; the technical problem that the data fusion reliability of heterogeneous sensors cannot be realized in the prior art is relieved.

Description

Technical Field

[0001] The present invention relates to the technical field of automotive electronic system data acquisition, and in particular to a sensor data fusion method and device. Background Art

[0002] Currently, a variety of sensors are widely used in automotive electronic systems, such as Inertial Measurement Unit (IMU), Global Navigation Satellite System (GNSS) receivers, Ultrasonic Sensor System (USS), LiDAR, etc., to obtain information data such as vehicle status and surrounding environment.

[0003] The inventors have found that each sensor has a different data sampling rate, format and time base, and each sensor has an independent data stream and time base for collection and processing. Therefore, the current sensor data fusion application scenario cannot achieve accurate data synchronization of different types of sensor data, which makes such data fusion application scenarios limited to specific sensor combinations and lacks versatility and scalability. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a sensor data fusion method and device, which performs precise time synchronization through the AIO architecture, and monitors and corrects abnormal sensor data and timestamps at the same time, thereby alleviating the technical problem that the reliability of heterogeneous sensor data fusion cannot be achieved in the prior art.

[0005] In a first aspect, an embodiment provides a sensor data fusion method, which is applied to an AIO architecture, wherein the AIO architecture includes a hardware layer, a kernel space, and a user layer, and the method includes:

[0006] Adding a target clock timestamp to at least one sensor data from the hardware layer through the kernel space for time synchronization, and transmitting the at least one first sensor data after time synchronization to the user layer;

[0007] Read the first sensor data, and detect a first abnormal state of the first sensor data according to the target clock timestamp and a preset length requirement of each type of the first sensor data; discard the data packet of the first abnormal state and report an error report, and report the second sensor data; wherein the first sensor data includes a plurality of data packets, the second sensor data includes at least one data packet of a normal state, and each data packet carries a target clock timestamp;

[0008] Extract at least one of the target clock timestamps from the second sensor data, and perform a second abnormal state detection; discard a data packet corresponding to the target clock timestamp of the second abnormal state and report an error report, and report the third sensor data; wherein the third sensor data includes at least one data packet in a normal state, and each of the data packets carries a target clock timestamp in a normal state;

[0009] Aggregate at least one of the third sensor data according to the target clock timestamp and the error report, and then fuse according to a preset fusion rule to generate a standard interface for providing access to the target application.

[0010] In an optional implementation, the step of detecting the first abnormal state of the first sensor data according to the target clock timestamp and a preset length requirement of each type of the first sensor data includes:

[0011] Compare the target clock timestamp carried in each data packet of the first sensor data of the first sensor type currently read with the target clock timestamp carried in each data packet of the first sensor data of the first sensor type previously read;

[0012] A data packet whose comparison result exceeds a first preset time threshold is identified as a first abnormal state; wherein the first preset time threshold corresponds to the first sensor type.

[0013] In an optional implementation, the step of detecting the first abnormal state of the first sensor data according to the target clock timestamp and the preset length requirement of each type of the first sensor data further includes:

[0014] Comparing the length of each data packet in the first sensor data of the first sensor type currently read with a preset data length;

[0015] The data packet whose comparison result does not meet the preset requirement is identified as a first abnormal state.

[0016] In an optional implementation, the step of extracting at least one of the target clock timestamps from the second sensor data and performing a second abnormal state detection includes:

[0017] Extracting the target clock timestamp carried by each data packet in the second sensor data;

[0018] Convert each of the target clock timestamps into a standard format timestamp;

[0019] Based on the continuity of the standard format timestamps, determining whether each of the standard format timestamps has an abnormality;

[0020] If so, the standard format timestamp with the jump is identified as a second abnormal state.

[0021] In an optional implementation, based on the continuity of the standard format timestamp, the step of determining whether each of the standard format timestamps has an abnormality includes: repeatedly executing the following process until all data packets collected by the current sensor are traversed:

[0022] Comparing the difference between the current standard format timestamp and the previous standard format timestamp with a second preset time threshold;

[0023] If the difference does not exceed the second preset time threshold, the current standard format timestamp is considered to be in a normal state;

[0024] If the difference exceeds the second preset time threshold, and the comparison result between the difference and the second preset time threshold satisfies the preset difference range, the second preset time threshold is adaptively adjusted so that under the effect of the adjusted second preset time threshold, the current standard format timestamp is in a normal state;

[0025] If the difference exceeds the second preset time threshold, and the comparison result between the difference and the second preset time threshold exceeds the preset difference range, the current standard format timestamp is identified as a second abnormal state;

[0026] Set the next standard format timestamp as the new current standard format timestamp.

[0027] In an optional implementation, the step of aggregating at least one of the third sensor data according to the target clock timestamp and the error report, and then fusing according to a preset fusion rule to generate a standard interface for providing access to the target application includes:

[0028] storing each type of the third sensor data in a time window buffer; wherein the time window buffer includes each type of historical third sensor data in a preset time period;

[0029] Determine a highest sampling frequency from sampling frequencies of various third sensor data, and perform a frame interpolation operation on each third sensor data other than the highest sampling frequency according to the highest sampling frequency, to obtain at least one fourth sensor data;

[0030] Based on the target clock timestamp of each data packet in the fourth sensor data, determine the same-frame data packets at the same target clock timestamp and the sensor type corresponding to each of the same-frame data packets;

[0031] Aggregate the same-frame data packets of each sensor type at the same target clock timestamp to obtain an aggregated data packet corresponding to each target clock timestamp;

[0032] Determining historical third sensor data from the time window buffer according to the reported error report, and correcting the aggregated data packet having the abnormal target clock timestamp based on the historical third sensor data;

[0033] According to a preset fusion rule, at least one fourth sensor data in each corrected aggregate data packet is fused to obtain a fusion result corresponding to each target clock timestamp;

[0034] The fusion result and the modified aggregated data packet are transmitted to a target application.

[0035] In an optional implementation, the step of adding a target clock timestamp to at least one sensor data from the hardware layer through the kernel space for time synchronization, and transmitting the at least one first sensor data after time synchronization to the user layer includes:

[0036] The target clock timestamp of the current moment is read from the target clock module in the Linux system through the kernel space and added to each sensor data, and then uploaded to the user layer; wherein the target clock timestamp includes the system clock timestamp, the navigation clock timestamp, and the Beidou satellite clock timestamp.

[0037] In a second aspect, an embodiment provides a sensor data fusion device, which is applied to an AIO architecture, wherein the AIO architecture includes a hardware layer, a kernel space, and a user layer, and the device includes:

[0038] an acquisition module, adding a target clock timestamp to at least one sensor data from the hardware layer through the kernel space for time synchronization, and transmitting the at least one first sensor data after time synchronization to the user layer;

[0039] A data reading module, reading the first sensor data, and detecting a first abnormal state of the first sensor data according to the target clock timestamp and a preset length requirement of each type of the first sensor data; discarding a data packet of the first abnormal state and reporting an error report, and reporting the second sensor data; wherein the first sensor data includes a plurality of data packets, the second sensor data includes at least one data packet of a normal state, and each data packet carries a target clock timestamp;

[0040] A timestamp parsing module extracts at least one of the target clock timestamps from the second sensor data and performs a second abnormal state detection; discards a data packet corresponding to the target clock timestamp of the second abnormal state and reports an error report, and reports the third sensor data; wherein the third sensor data includes at least one data packet in a normal state, and each of the data packets carries a target clock timestamp in a normal state;

[0041] The data aggregation module aggregates at least one of the third sensor data according to the target clock timestamp and the error report, and then fuses them according to a preset fusion rule to generate a standard interface for providing access to the target application.

[0042] In a third aspect, an embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method described in any of the aforementioned implementation methods are implemented.

[0043] In a fourth aspect, an embodiment provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method described in any one of the aforementioned implementations.

[0044] A sensor data fusion method and device provided by an embodiment of the present invention implements time synchronization by stamping target clock timestamps on various sensor data based on the kernel space of an AIO architecture, and then detects and discards data packets in abnormal states in each type of sensor data through the application layer, and at the same time detects the abnormal state of the target clock timestamp carried by each data packet, and discards the data packet corresponding to the target clock timestamp in the abnormal state, while the data packets and timestamps of the remaining sensor data are normal. At this time, while aligning various types of sensor data according to the target clock timestamp, the aforementioned abnormal data packets are corrected according to the error report reported by the detected abnormal state, and then integration processing and fusion operations are performed to obtain accurate integrated data packets and fusion results that can be accessed by the target application for subsequent processing.

[0045] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0046] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flow chart of a sensor data fusion method provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of an AIO architecture provided by an embodiment of the present invention;

[0050] Figure 3 A functional module diagram of a sensor data fusion device provided by an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of the hardware architecture of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Currently, cars are usually equipped with a variety of sensors, such as IMU, GPS, wheel speed meter, laser radar, etc. The data of these sensors need to be fused so that different applications of the vehicle system can perform subsequent operations based on the fusion results. It can be seen that the accuracy of the fusion results will affect the accuracy of different subsequent vehicle applications. However, the inventors have found that the heterogeneous data fusion of multiple sensors not only requires time synchronization, but also involves data anomaly detection and correction processing during the transmission process, so that the data of multiple sensors with different sampling frequencies and formats can be accurately fused.

[0054] Based on this, an embodiment of the present invention provides a sensor data fusion method and device, which realizes accurate target clock synchronization through the AIO architecture, and then detects and processes sensor data reading anomalies and timestamp anomalies through the application layer, and corrects them again according to the aforementioned abnormal conditions before fusion, thereby ensuring the accuracy of the fusion result.

[0055] To facilitate understanding of this embodiment, a sensor data fusion method disclosed in an embodiment of the present invention is first introduced in detail. The method is applied to the Automobile I / O (AIO) architecture, which is a subsystem in the Linux kernel for managing sensors. It provides a set of standard interfaces for user space, simplifies sensor driver development, and provides a consistent user space API.

[0056] Figure 2 A schematic diagram of an AIO architecture provided by an embodiment of the present invention.

[0057] The Automobile I / O (AIO) architecture includes a hardware layer, a kernel space and a user layer; there is a public memory buffer between the kernel space and the user layer, and the public memory buffer includes a character device Char device, a file reading system sysfs and a configuration system configs; the user layer includes a LibAio module, and the LibAio module includes a data reading module, a timestamp parsing module and a data aggregation engine; the kernel space includes a driver layer and a core layer.

[0058] The Automobile I / O (AIO) architecture supports sensor data collection using multiple transmission protocols (I2C, SPI, UART, etc.). Multiple sensors can be added to a driver at one time. The AIO architecture automatically adds a target clock timestamp, such as a PHC timestamp, to the acquired sensor data, and then passes it to a shared memory buffer. The LibAio module in the user layer reads the data collected by each sensor from the shared memory buffer and aggregates and fuses the time-synchronized sensor data into a unified format based on the target clock timestamp.

[0059] In this AIO architecture, different heterogeneous sensors can be pre-registered according to the fusion needs of actual application scenarios, and the timestamp range threshold (preset time threshold) corresponding to each sensor can be defined in the configuration system; the fusion result format to be unified, the preset data length and preset conversion format of each sensor data, and the specific types of sensors required by actual business needs can also be pre-set according to actual business needs;

[0060] Various heterogeneous sensors at the hardware layer transmit the collected data and timestamps (timer or GPIO pin collection) of various heterogeneous sensors to the kernel space through transmission protocols such as I2C, SPI, and UART;

[0061] The sensor data collected by the hardware layer is transmitted to the core layer through the bus driver BUSDriver of the driver layer, and the timestamp is transmitted to the core layer through the high-precision timer trigger of AIO;

[0062] The core layer adds a target clock timestamp, such as a PHC timestamp, to each sensor data. If the sensor data includes IMU Data--GNSSData--USSData, then the sensor data after adding the PHC timestamp are at the same time, and the time difference between the timestamps of each sensor data meets the preset timestamp range threshold.

[0063] Transmit the sensor data at the same time to the LibAio module at the user layer;

[0064] Among them, the LibAio module can read the data of each sensor through the character device char device, the LibAio module can read the data of a target sensor through the file system sysfs, and the LibAio module can also configure the high-precision timer interrupt Hrtimer trigger by configuring configs;

[0065] The LibAio module can be understood as an API architecture. Its main function is to provide applications with a standardized interface for accessing aggregated data composed of various sensor data. It can achieve the following functions:

[0066] a. The LibAio architecture API itself can define functions and methods for data access, such as obtaining the latest sensor data set processed by the data aggregation engine with a target clock timestamp; for example, the LibAio architecture can also enable the target application to continuously receive a specific type of sensor data, establish a direct correspondence with the original sensor data, and achieve the purpose of subscribing to the data stream.

[0067] b. Through the timestamp parsing module in the LibAio architecture API, it is also possible to find data of specific sensor types (such as IMU, GNSS, USS, etc.) in a specific time period required by the upper-level target application (such as assisted driving system ADAS system, etc.) based on the target clock timestamp carried by the sensor data.

[0068] c. The LibAio architecture obtains various sensor data after time synchronization through a common memory buffer, which can optimize the performance of frequent access.

[0069] d. The data reading module, timestamp parsing module and data aggregation engine of the LibAio architecture are all integrated with error handling and exception reporting functions;

[0070] e. The data reading module of the LibAio architecture converts the data of each sensor after time synchronization according to the conversion format preset for each sensor; for example, the data of sensor A is converted into binary data, the data of sensor B is converted into floating point data, and so on.

[0071] f.LibAio architecture also has a security module ( Figure 2 ), is linked to the data aggregation engine to perform security audits on the aggregation results and fusion results again to further improve the reliability of providing data for the target application.

[0072] LibAio modules include Data Reading Module (DRM), Timestamp Parsing Module (TPM), and Data Aggregation Engine (DAE).

[0073] Data Reading Module (DRM):

[0074] Main function: Read the first sensor data with target clock timestamp from kernel space.

[0075] Specific tasks: a. The data reading module interacts with the AIO subsystem including the kernel space, and the data reading module reads the first sensor data after time synchronization through the AIO character device char device or the shared memory buffer. b. The data reading module determines whether there is an abnormality in the sensor data after time synchronization after reading; if there is an abnormality such as time jump or time rollback caused by a reading operation error, the read sensor data is synchronized or reported again. c. The data reading module determines whether the preset data length of each sensor data after time synchronization matches that of each sensor data, and then determines whether the current sensor data is complete. d. The data reading module can also convert each second sensor data into its corresponding preset data format; for example, the second sensor data can be converted into a data format with two decimal places by performing floating-point operations. Among them, the shared memory buffer is an area between the kernel space and the application layer, which is used to efficiently transmit a large amount of sensor data after time synchronization.

[0076] Each second sensor data processed by the data reading module is transmitted to the timestamp parsing module in the form of a data stream.

[0077] Timestamp Parsing Module (TPM):

[0078] Main function: Parse and process the PHC timestamp attached to the second sensor data.

[0079] Specific tasks: a. The timestamp parsing module extracts the PHC timestamp information from each second sensor data after time synchronization preprocessing in the form of data stream. b. The timestamp parsing module converts the PHC timestamp into a standard format timestamp (such as a UNIX timestamp). c. The timestamp parsing module checks whether there is an abnormality in the standard format timestamp, such as a time jump. d. The timestamp parsing module can correct the timestamp of the second abnormal state by interpolating the jump standard format timestamp, processing the missing timestamp, or adaptively adjusting the preset time threshold, thereby correcting the problem of timestamp jump.

[0080] Output: Transmit each third sensor data to the data aggregation engine.

[0081] Data Aggregation Engine (DAE)

[0082] Main function: Aggregate data from different sensors into a unified data packet based on timestamp.

[0083] Specific tasks: a. Maintain a time window buffer to store all third sensor data in the recent period. b. Perform data interpolation operations on various third sensor data according to the sampling frequency, and the data interpolation operations include nearest neighbor interpolation, linear interpolation, etc. c. Aggregate at least one third sensor data according to the target clock timestamp. d. According to the reported error report, search the buffer for the third sensor data corresponding to the previous target clock timestamp for interpolation correction, or do not take the correction operation and report the error report together with the fusion results of the subsequent steps. e. Fusion of data that may be initially aggregated, such as calculating the average or weighted sum of multiple sensors.

[0084] Output: Aggregate data packets and / or fusion results of multiple sensors after time synchronization. Aggregate data packets such as |--IMUData--GNSSData--USSData--Timestamp|.

[0085] Sensor data is uploaded from the hardware layer to the user layer via the IO architecture, and control instructions are transmitted from the user layer to the hardware layer via the IO architecture.

[0086] Figure 1 A flow chart of a sensor data fusion method provided by an embodiment of the present invention.

[0087] like Figure 1 As shown, the method comprises the following steps:

[0088] Step S102: adding a target clock timestamp to at least one sensor data from the hardware layer through the kernel space for time synchronization, and transmitting the at least one first sensor data after time synchronization to the user layer.

[0089] The target clock timestamp of the current moment is read from the target clock module in the Linux system through the kernel space and added to each sensor data, and then uploaded to the user layer; among them, the target clock timestamp includes the system clock timestamp, navigation clock timestamp, Beidou satellite clock timestamp, such as PHC timestamp, PTP timestamp, etc.

[0090] The time-synchronized first sensor data may be understood as one or more data packets from different types of sensors at the current moment.

[0091] Step S104, read the first sensor data, and detect the first abnormal state of the first sensor data according to the target clock timestamp and the preset length requirement of each first sensor data; discard the data packet of the first abnormal state and report an error report, and report the second sensor data.

[0092] The first sensor data includes multiple data packets. To ensure data application security, the data in the first abnormal state may be discarded after the first sensor data is detected and an error report may be reported. The second sensor data includes at least one data packet in a normal state. Each data packet carries a target clock timestamp. The second sensor data in a normal state is reported to participate in subsequent operations.

[0093] As an optional embodiment, in order to avoid the problem of excessive discarding of data packets, the data packets in the first abnormal state may be corrected; for example, the previous data packet / the next data packet of the data packet in the first abnormal state is selected to replace the data packet in the first abnormal state according to the target clock timestamp carried by each data packet;

[0094] Preferably, the target clock timestamp of the data packet in the first abnormal state can be used to calculate the time difference with the target clock timestamp corresponding to the previous data packet and the next data packet, and the previous data packet or the next data packet with the smaller time difference can be selected to replace the data packet in the first abnormal state.

[0095] Step S106, extracting at least one target clock timestamp from the second sensor data, and performing a second abnormal state detection; discarding a data packet corresponding to the target clock timestamp of the second abnormal state and reporting an error report, and reporting the third sensor data.

[0096] Here, based on the removal of abnormal state data packets from sensor data in the aforementioned embodiment, the abnormal state of the target clock timestamp carried by each data packet is detected.

[0097] Among them, in order to ensure data application security, the second sensor data corresponding to the timestamp in the second abnormal state can be discarded and an error report can be reported after the target clock timestamp is detected, and the third sensor data includes at least one data packet in a normal state, and each data packet carries a target clock timestamp in a normal state;

[0098] As an optional embodiment, in order to avoid the problem of excessive packet discarding, the data packet corresponding to the timestamp in the second abnormal state may also be corrected; for example, for the target clock timestamp carried by each data packet, the average value of the previous data packet and the next data packet of the data packet in the first abnormal state is selected to replace the data packet in the first abnormal state.

[0099] Step S108 , aggregating at least one third sensor data according to the target clock timestamp and the error report, and then fusing according to a preset fusion rule to generate a standard interface for providing access to the target application.

[0100] Here, the target clock timestamp can align, aggregate and fuse the data of multiple sensors; according to the error report reported in the steps of the aforementioned embodiment, it can be known which data packet or packets of each sensor have experienced anomalies, and further correction processing is performed to ensure application reliability.

[0101] Among them, the target application can be understood as the target application program on the upper layer of the data aggregation engine. The target application may include at least one target application that is different from the vehicle application scenario, and different target applications require aggregated data packets and fusion results of different sensor data; the embodiment of the present invention provides an access interface for outputting the aggregated data packets and fusion results of the sensor data required by the target application. Based on the output of the application layer data aggregation engine of the embodiment of the present invention, the target application can access the aggregated data packets and fusion results of the sensor data required by it.

[0102] In a preferred embodiment of actual application, various types of sensor data are marked with target clock timestamps according to the kernel space of the AIO architecture to achieve time synchronization, and then the application layer detects and discards the data packets in abnormal status in each type of sensor data, and at the same time, the abnormal status of the target clock timestamp carried by each data packet is detected, and the data packet corresponding to the abnormal target clock timestamp is discarded. The data packets and timestamps of the remaining sensor data are normal. At this time, the various types of sensor data are aligned according to the target clock timestamp, and the aforementioned abnormal data packets are corrected according to the error report reported by the detected abnormal status, and then the integration processing and fusion operations are performed to obtain accurate integrated data packets and fusion results that can be accessed by the target application for subsequent processing.

[0103] In some embodiments, the first sensor data after time synchronization can be read from a shared memory buffer. The setting of the shared memory buffer can alleviate the cumbersome defects of frequent reading from the hardware layer. However, in actual applications, the sensor data reading operation may cause some anomalies, causing the original first sensor data after time synchronization to have time jumps, lags and other problems. On this basis, step S104 detects the first abnormal state generated by the first sensor data through a reading operation, and performs corresponding operations on the first sensor data in the first abnormal state to ensure the reliability of subsequent data applications, including:

[0104] Step 1.1), compare the target clock timestamp carried by each data packet in the first sensor data of the first sensor type currently read with the target clock timestamp carried by each data packet of the first sensor data of the first sensor type previously read; and identify the data packet whose comparison result exceeds the first preset time threshold as a first abnormal state.

[0105] The first preset time threshold corresponds to the first sensor type, that is, each sensor corresponds to a corresponding preset time threshold, and the number of data packets collected by each sensor at a time is fixed; in addition, if a certain sensor collects multiple data packets, the order of each data packet in the collected multiple data packets can be known based on the target clock timestamp it carries;

[0106] For example, the 10 data packets 1 currently read by sensor A have each target clock timestamp t1...t10, and the 10 data packets 2 previously read by sensor A have each target clock timestamp T1...T10; according to the order of each data packet in the multi-data packet set, the target clock timestamp corresponding to each data packet 1 and the target clock timestamp corresponding to each data packet 2 are calculated in turn, and the difference comparison result of each data packet 1 is compared with the first preset time threshold. If the difference comparison result exceeds the first preset time threshold, it is considered that the data packet 1 corresponding to the difference comparison result is in the first abnormal state, that is, the reading operation has caused an abnormality to the first sensor data.

[0107] Step 1.2), comparing the length of each data packet in the first sensor data of the first sensor type currently read with the preset data length; and identifying the data packet whose comparison result does not meet the preset requirement as a first abnormal state.

[0108] Among them, each sensor corresponds to a preset data length; in addition to step 1.1), the first sensor data abnormality caused by the reading operation can also be determined by whether the length of each data packet is consistent with the preset data length. If not, the inconsistent data is considered to be the first abnormal state.

[0109] For example, the 10 data packets 1 currently read by sensor A correspond to data lengths L1 . . . L10 respectively, and the preset data length is 5. Then, data packets that do not meet the length of 5 are all determined to be in the first abnormal state.

[0110] Based on the above-mentioned embodiment, in addition to detecting the abnormal state of the sensor data caused by the reading operation, the state of the target clock timestamp of each data packet can also be detected to ensure the reliability of subsequent data application; illustratively, step S106 can determine whether the target clock timestamp of each data packet is abnormal by the continuity of each data packet from the same sensor read at the same time, including:

[0111] Step 2.1), extracting the target clock timestamp carried by each data packet in the second sensor data.

[0112] Here, each data packet carries its corresponding target clock timestamp, and the target clock timestamp of each data packet can be extracted by text recognition, keyword recognition, etc.

[0113] Step 2.2), convert each target clock timestamp into a standard format timestamp.

[0114] Here, in order to more accurately identify and detect anomalies of the target clock timestamp, each target clock timestamp is converted into a standard format, such as a UNIX timestamp.

[0115] Step 2.3), based on the continuity of the standard format timestamps, determine whether each standard format timestamp has an anomaly.

[0116] If it exists, the standard format timestamp with the jump is identified as the second abnormal state.

[0117] If not, the third sensor data with the target clock timestamp and the data packet both in normal state is uploaded.

[0118] Exemplarily, step 2.3) is to determine whether there is an abnormality based on the continuity of the standard format timestamps between the data packets collected by the same sensor at the same time; for example, the 10 data packets 1 currently read by sensor A, according to the difference between the timestamps of adjacent data packets 1 in the 10 data packets 1, determine whether there is an abnormality in the target clock timestamp, including: repeatedly executing the following process until all the data packets collected by the current sensor are traversed:

[0119] Step 2.3.1), comparing the difference between the current standard format timestamp and the previous standard format timestamp with a second preset time threshold.

[0120] The second preset time threshold also corresponds to the sensor type, that is, different sensor types may correspond to corresponding second preset time thresholds; the second preset time threshold may be the same as or different from the first preset time threshold.

[0121] Step 2.3.2), if the difference does not exceed the second preset time threshold, the current standard format timestamp is considered to be in a normal state.

[0122] Here, the difference between the current standard format timestamp of the current data packet and the previous standard format timestamp of the previous data packet is 49us and does not exceed the second preset time threshold such as 50us, so the current standard format timestamp is in a normal state.

[0123] Step 2.3.3), if the difference exceeds the second preset time threshold, and the comparison result between the difference and the second preset time threshold satisfies the preset difference range, the second preset time threshold is adaptively adjusted so that under the effect of the adjusted second preset time threshold, the current standard format timestamp is in a normal state.

[0124] Here, the difference between the current standard format timestamp of the current data packet and the previous standard format timestamp of the previous data packet is 51us, which exceeds the second preset time threshold such as 50us, but the comparison result between the two is 1us, which satisfies the preset difference range such as 5us. At this time, the second preset time threshold is adaptively adjusted to 51us, so that the difference between the current standard format timestamp of the current data packet and the previous standard format timestamp of the previous data packet is 51us, which does not exceed the second preset time threshold 51us after adaptive adjustment.

[0125] Step 2.3.4), if the difference exceeds the second preset time threshold, and the comparison result between the difference and the second preset time threshold exceeds the preset difference range, the current standard format timestamp is identified as the second abnormal state.

[0126] Here, the difference between the current standard format timestamp of the current data packet and the previous standard format timestamp of the previous data packet is 56us, which exceeds the second preset time threshold such as 50us, and the comparison result between the two is 6us, which satisfies the preset difference range such as 5us. At this time, the current standard format timestamp is in the second abnormal state, and the data packet corresponding to the timestamp in the second abnormal state can be discarded or corrected accordingly.

[0127] Step 2.3.5), taking the next standard format timestamp as the new current standard format timestamp.

[0128] Here, after the current standard format timestamp of the current data packet is determined, the next standard format timestamp of the next data packet is used as the new current standard format timestamp to repeat steps 2.3.1) to 2.3.5) until all data packets collected by the sensor this time are traversed and terminated.

[0129] On the basis of the above-mentioned embodiment, the third sensor data whose data packets and target clock timestamps are normal are corrected to further ensure the reliability of the aggregation and fusion results; step S108 can obtain the abnormal data packets through the error report reported in the above-mentioned embodiment and correct them, which specifically includes the following steps:

[0130] Step 3.1), storing each type of third sensor data in a time window buffer.

[0131] The time window buffer includes each type of historical third sensor data of a preset time period; for example, the time window buffer stores the third sensor data of sensors AF from the current moment to the past 6 hours; the time window buffer is integrated in the data aggregation engine.

[0132] Step 3.2), determining the highest sampling frequency from the sampling frequencies of various third sensor data, and performing frame interpolation operation on each third sensor data not having the highest sampling frequency according to the highest sampling frequency, to obtain at least one fourth sensor data.

[0133] Here, the sampling frequency of each sensor may be different, such as sensor A collects 10 data packets per second, and sensor B collects 20 data packets per second. For the reliability of subsequent aggregation and fusion operations, the third sensor data of other sensors are interpolated based on the sampling frequency corresponding to the sensor with the highest sampling frequency. For example, based on the fact that sensor B collects 20 data packets per second, the third sensor data of sensor A is interpolated, and the interpolation is performed according to the sensor data of 20-10.

[0134] As an optional embodiment, the average of the 10 data packets originally collected by sensor A can be selected as the object of linear interpolation, and the third sensor data of sensor A can be inserted; after the interpolation operation, the third sensor data of sensor A, that is, the sampling frequency of the fourth sensor data (20 data packets collected per second) is the same as that of sensor B.

[0135] Step 3.3), based on the target clock timestamp of each data packet in the fourth sensor data, determine the data packets of the same frame with the same target clock timestamp and the sensor type corresponding to each data packet of the same frame.

[0136] It can be understood that each type of sensor data is stamped with a target clock timestamp through the kernel space of AIO, and at this time each type of sensor data corresponds to the same type of target clock timestamp; and in step 3.3), each same-frame data packet with the same timestamp value and the sensor types corresponding to these same-frame data packets can be determined.

[0137] Step 3.4), aggregate the same frame data packets of each sensor type at the same target clock timestamp to obtain an aggregated data packet corresponding to each target clock timestamp.

[0138] It should be noted that the timestamp values ​​collected by different sensors may not be exactly the same. Here, if the difference between the target clock timestamp 1 collected by sensor A and the target clock timestamp 2 collected by sensor B meets the preset requirements, it is determined that the target clock timestamp 1 of sensor A and the target clock timestamp 2 of sensor B have the same values ​​and are at the same time; the data packet m corresponding to the target clock timestamp 1 and the data packet n corresponding to the target clock timestamp 2 are data packets of the same frame, and an aggregation operation is performed to obtain the aggregated data packet |sensor A data packet m-sensor B data packet n-target clock timestamp 1 / 2 (value)|.

[0139] Step 3.5), determining historical third sensor data from the time window buffer according to the reported error report, and correcting the aggregated data packet with the abnormal target clock timestamp based on the historical third sensor data.

[0140] Exemplarily, according to the error report uploaded in the aforementioned embodiment, the priority order of the aforementioned abnormal data packets in the data packets collected by each sensor at a single time can be known; based on the priority order of the abnormal data packets in each sensor, the target data packets with the corresponding priority order in the historical third sensor data of the corresponding sensor are searched from the time window buffer; based on the mean calculation result of the target data packets, the abnormal data packets of the corresponding sensor are corrected / replaced.

[0141] Step 3.6), according to a preset fusion rule, fuse at least one fourth sensor data in each corrected aggregate data packet to obtain a fusion result corresponding to each target clock timestamp.

[0142] Here, according to the actual application requirements / target application requirements, at least one fourth sensor data can be selectively fused; wherein the preset fusion rules may include performing average calculation, weighted sum calculation, etc. on the fourth sensor data of the sensor type required by the target application to obtain the fusion result.

[0143] In step 3.7), the fusion result and the corrected aggregated data packet are transmitted to the target application.

[0144] Here, according to actual needs, the target application can obtain the fusion result and the above-mentioned aggregated data packet through access.

[0145] The embodiment of the present invention realizes accurate time synchronization of heterogeneous data from different sensors (such as IMU, GNSS, USS, etc.) through a unified time reference and data aggregation engine; optimizes data transmission and processing flow and improves overall system efficiency through a design that spans kernel space and user space; supports multiple sensor interfaces (I2C, SPI, UART, etc.), is easy to integrate new sensors, and adapts to future technological development; LIBAIO architecture API provides a standardized interface, simplifies the application development process, and improves code reusability; through an optimized kernel-user space communication mechanism, ensures real-time processing and low-latency response of data; integrates precision time protocol (PTP) support at the kernel layer to provide a unified high-precision time reference for all sensor data; through a data aggregation engine (DAE), accurate time alignment and fusion of different types of sensor data are realized, providing more comprehensive and reliable information; LIBAIO architecture includes a modular design including a data reading module (DRM), a timestamp parsing module (TPM) and a data aggregation engine (DAE), which improves the maintainability and scalability of the system. Considering the particularity of the vehicle environment, such as temperature changes, vibrations and other factors, the stability and reliability of the system in practical applications are improved. It provides strong technical support for applications such as advanced driver assistance systems (ADAS) and autonomous driving technologies that require high-precision multi-sensor fusion.

[0146] In some embodiments, Figure 3 As shown, an embodiment of the present invention further provides a sensor data fusion device 200, which is applied to an AIO architecture. The AIO architecture includes a hardware layer, a kernel space, and a user layer. The device includes:

[0147] The acquisition module 201 adds a target clock timestamp to at least one sensor data from the hardware layer through the kernel space for time synchronization, and transmits the at least one first sensor data after time synchronization to the user layer;

[0148] The data reading module 202 reads the first sensor data, and detects the first abnormal state of the first sensor data according to the target clock timestamp and the preset length requirement of each type of the first sensor data; discards the data packet of the first abnormal state and reports an error report, and reports the second sensor data; wherein the first sensor data includes a plurality of data packets, the second sensor data includes at least one data packet of a normal state, and each data packet carries a target clock timestamp;

[0149] The timestamp parsing module 203 extracts at least one of the target clock timestamps from the second sensor data and performs a second abnormal state detection; discards the data packet corresponding to the target clock timestamp of the second abnormal state and reports an error report, and reports the third sensor data; wherein the third sensor data includes at least one data packet in a normal state, and each of the data packets carries a target clock timestamp in a normal state;

[0150] The data aggregation module 204 aggregates at least one of the third sensor data according to the target clock timestamp and the error report, and then fuses them according to a preset fusion rule to generate a standard interface for providing access to the target application.

[0151] Furthermore, the data reading module 202 is also specifically used to compare the target clock timestamp carried by each data packet in the first sensor data of the first sensor type currently read with the target clock timestamp carried by each data packet of the first sensor data of the first sensor type previously read; and identify a data packet whose comparison result exceeds a first preset time threshold as a first abnormal state; wherein the first preset time threshold corresponds to the first sensor type.

[0152] Furthermore, the data reading module 202 is also specifically used to compare the length of each data packet in the first sensor data of the first sensor type currently read with a preset data length; and identify the data packet whose comparison result does not meet the preset requirement as a first abnormal state.

[0153] Furthermore, the timestamp parsing module 203 is also specifically used to extract the target clock timestamp carried by each data packet in the second sensor data; convert each of the target clock timestamps into a standard format timestamp; based on the continuity of the standard format timestamps, determine whether each of the standard format timestamps has an abnormality; if so, identify the standard format timestamp with a jump as a second abnormal state.

[0154] Furthermore, the timestamp parsing module 203 is also specifically used to repeatedly execute the following process until all data packets collected by the current sensor are traversed: comparing the difference between the current standard format timestamp and the previous standard format timestamp with a second preset time threshold; if the difference does not exceed the second preset time threshold, the current standard format timestamp is identified as a normal state; if the difference exceeds the second preset time threshold, and the comparison result of the difference with the second preset time threshold satisfies the preset difference range, the second preset time threshold is adaptively adjusted so that under the effect of the adjusted second preset time threshold, the current standard format timestamp is in a normal state; if the difference exceeds the second preset time threshold, and the comparison result of the difference with the second preset time threshold exceeds the preset difference range, the current standard format timestamp is identified as a second abnormal state; and the next standard format timestamp is used as the new current standard format timestamp.

[0155] In some embodiments, the data aggregation module 204 is further specifically used to store each type of the third sensor data in a time window buffer; wherein the time window buffer includes each type of historical third sensor data in a preset time period; determine the highest sampling frequency from the sampling frequencies of various third sensor data, and perform frame insertion operation on each type of third sensor data other than the highest sampling frequency according to the highest sampling frequency to obtain at least one fourth sensor data; based on the target clock timestamp of each data packet in the fourth sensor data, determine the same-frame data packets at the same target clock timestamp and the sensor type corresponding to each of the same-frame data packets; aggregate the same-frame data packets at the same target clock timestamp of each sensor type to obtain an aggregated data packet corresponding to each target clock timestamp; determine the historical third sensor data from the time window buffer according to the reported error report, and correct the aggregated data packet with an abnormal target clock timestamp based on the historical third sensor data; according to a preset fusion rule, fuse at least one fourth sensor data in each corrected aggregated data packet to obtain a fusion result corresponding to each target clock timestamp; transmit the fusion result and the corrected aggregated data packet to the target application.

[0156] In some embodiments, the acquisition module 201 is also specifically used to read the target clock timestamp of the current moment from the target clock module in the Linux system through the kernel space and add it to each sensor data, and then upload it to the user layer; wherein the target clock timestamp includes the system clock timestamp, the navigation clock timestamp, and the Beidou satellite clock timestamp.

[0157] Figure 4Schematic diagram of the hardware architecture of the electronic device 300 provided in an embodiment of the present invention. Figure 4 As shown, the electronic device 300 includes: a machine-readable storage medium 301 and a processor 302, and may also include a non-volatile storage medium 303, a communication interface 304 and a bus 305; wherein the machine-readable storage medium 301, the processor 302, the non-volatile storage medium 303 and the communication interface 304 communicate with each other through the bus 305. The processor 302 can execute the sensor data fusion method described in the above embodiment by reading and executing the machine-executable instructions for sensor data fusion in the machine-readable storage medium 301.

[0158] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0159] The non-volatile medium may be a non-volatile memory, a flash memory, a storage drive (such as a hard drive), any type of storage disk (such as a CD, DVD, etc.), or a similar non-volatile storage medium, or a combination thereof.

[0160] It can be understood that the specific operation methods of each functional module in this embodiment can refer to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0161] The computer-readable storage medium provided in the embodiment of the present invention stores a computer program. When the computer program code is executed, the sensor data fusion method described in any of the above embodiments can be implemented. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0163] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0164] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0165] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the aforementioned embodiments, those of ordinary skill in the art should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the aforementioned embodiments within the technical scope disclosed by the present invention, or can easily conceive of changes, or make equivalent replacements for some of the technical features therein. Such modifications, changes or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention.

Claims

1. A sensor data fusion method, characterized in that: Applied to an AIO architecture, the AIO architecture includes a hardware layer, a kernel space, and a user layer, and the method includes: Adding a target clock timestamp to at least one sensor data from the hardware layer through the kernel space for time synchronization, and transmitting the at least one first sensor data after time synchronization to the user layer; Reading the first sensor data and detecting a first abnormal state of the first sensor data based on the target clock timestamp and a preset length requirement for each type of the first sensor data; discarding a data packet in the first abnormal state and reporting an error report, and reporting the second sensor data; wherein the first sensor data includes a plurality of data packets, and the second sensor data includes at least one data packet in a normal state, each data packet carrying a target clock timestamp; Extracting at least one target clock timestamp from the second sensor data and performing a second abnormal state detection; discarding a data packet corresponding to the target clock timestamp in the second abnormal state and reporting an error report, and reporting third sensor data; wherein the third sensor data includes at least one data packet in a normal state, each of the data packets carrying a target clock timestamp in a normal state; Aggregate the at least one third sensor data according to the target clock timestamp and the error report, and then fuse the data according to a preset fusion rule to generate a standard interface for providing access to the target application.

2. The method according to claim 1, characterized in that The step of detecting a first abnormal state of the first sensor data according to the target clock timestamp and a preset length requirement of each type of the first sensor data includes: comparing a target clock timestamp carried in each data packet of the first sensor data of the first sensor type currently read with a target clock timestamp carried in each data packet of the first sensor data of the first sensor type previously read; A data packet whose comparison result exceeds a first preset time threshold is identified as being in a first abnormal state; wherein the first preset time threshold corresponds to the first sensor type.

3. The method according to claim 1, characterized in that The step of detecting a first abnormal state of the first sensor data according to the target clock timestamp and a preset length requirement of each type of the first sensor data further includes: comparing the length of each data packet in the first sensor data of the first sensor type currently read with a preset data length; The data packet whose comparison result does not meet the preset requirements is identified as a first abnormal state.

4. The method according to claim 1, wherein The step of extracting at least one target clock timestamp from the second sensor data and performing a second abnormal state detection includes: Extracting the target clock timestamp carried by each data packet in the second sensor data; Converting each of the target clock timestamps into a standard format timestamp; Based on the continuity of the standard format timestamps, determining whether each of the standard format timestamps has an anomaly; If so, the standard format timestamp with the jump is identified as a second abnormal state.

5. The method according to claim 4, characterized in that The step of determining whether each of the standard format timestamps has an anomaly based on the continuity of the standard format timestamps includes: Repeat the following process until all data packets collected by the current sensor are traversed: comparing the difference between the current standard format timestamp and the previous standard format timestamp with a second preset time threshold; If the difference does not exceed the second preset time threshold, the current standard format timestamp is considered to be in a normal state; If the difference exceeds the second preset time threshold, and the comparison result between the difference and the second preset time threshold satisfies the preset difference range, the second preset time threshold is adaptively adjusted so that the current standard format timestamp is in a normal state under the effect of the adjusted second preset time threshold; If the difference exceeds the second preset time threshold, and the comparison result between the difference and the second preset time threshold exceeds the preset difference range, the current standard format timestamp is identified as a second abnormal state; Set the next standard format timestamp as the new current standard format timestamp.

6. The method according to claim 1, characterized in that The step of aggregating at least one of the third sensor data according to the target clock timestamp and the error report, and then fusing the data according to a preset fusion rule to generate a standard interface for providing access to the target application includes: storing each type of the third sensor data in a time window buffer; wherein the time window buffer includes each type of historical third sensor data of a preset time period; Determining a maximum sampling frequency from sampling frequencies of various third sensor data, and performing a frame interpolation operation on each type of third sensor data other than the maximum sampling frequency according to the maximum sampling frequency to obtain at least one fourth sensor data; Determining, based on a target clock timestamp of each data packet in the fourth sensor data, data packets of the same frame with the same target clock timestamp and a sensor type corresponding to each of the data packets of the same frame; Aggregate the same-frame data packets of each sensor type at the same target clock timestamp to obtain an aggregated data packet corresponding to each target clock timestamp; determining historical third sensor data from the time window buffer according to the reported error report, and correcting the aggregated data packet having the abnormal target clock timestamp based on the historical third sensor data; fusing at least one fourth sensor data in each corrected aggregated data packet according to a preset fusion rule to obtain a fusion result corresponding to each target clock timestamp; The fusion result and the modified aggregated data packet are transmitted to a target application.

7. The method according to claim 1, characterized in that The step of adding a target clock timestamp to at least one sensor data from the hardware layer through the kernel space for time synchronization, and transmitting the at least one first sensor data after time synchronization to the user layer includes: The target clock timestamp of the current moment is read from the target clock module in the Linux system through the kernel space and added to each sensor data, and then uploaded to the user layer; wherein, the target clock timestamp includes the system clock timestamp, the navigation clock timestamp, and the Beidou satellite clock timestamp.

8. A sensor data fusion device, characterized in that: Applied to an AIO architecture, the AIO architecture includes a hardware layer, a kernel space, and a user layer, and the device includes: an acquisition module, adding a target clock timestamp to at least one sensor data from the hardware layer through the kernel space for time synchronization, and transmitting the at least one first sensor data after time synchronization to the user layer; a data reading module configured to read the first sensor data and detect a first abnormal state of the first sensor data based on the target clock timestamp and a preset length requirement for each type of the first sensor data; discard the data packet in the first abnormal state and submit an error report, and report the second sensor data; wherein the first sensor data includes multiple data packets, and the second sensor data includes at least one data packet in a normal state, each data packet carrying a target clock timestamp; A timestamp parsing module extracts at least one target clock timestamp from the second sensor data and performs a second abnormal state detection; discards a data packet corresponding to the target clock timestamp in the second abnormal state and reports an error report, and reports third sensor data; wherein the third sensor data includes at least one data packet in a normal state, each of which carries a target clock timestamp in a normal state; The data aggregation module aggregates at least one third sensor data according to the target clock timestamp and the error report, and then fuses the data according to a preset fusion rule to generate a standard interface for providing access to the target application.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A machine-readable storage medium, characterized in that The machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the steps of the method according to any one of claims 1 to 7.

Images