Multi-sensor data synchronous acquisition and fusion processing system and method

Abstract

The invention discloses a multi-sensor data synchronous acquisition and fusion processing system and method, and belongs to the technical field of marine monitoring. Unified connection between different sensors and a sensor communication module and unified transmission of multi-sensor data to an upper computer are realized by designing a unified multi-sensor communication protocol conversion interface; and then data timestamps of different sensors are aligned through a clock synchronization mechanism and a time offset correction model, and real-time fusion processing is performed on the data of the multiple sensors by adopting a dynamic fusion algorithm based on a time sequence and frequency, so that real-time fusion of different multi-sensor data is realized, the precision of multi-sensor data fusion is improved, and the accuracy of multi-sensor data fusion is improved. And the marine underwater environment monitoring effect is improved.

Description

Technical Field

[0001] The present application belongs to the field of ocean monitoring technology, and specifically relates to a system and method for synchronously collecting and fusion processing of multi-sensor data. Background Art

[0002] Marine environmental target monitoring is a basic task for the utilization and development of marine resources. In order to obtain marine environmental information more comprehensively and accurately, the commonly used method is to collect data from hydrophones, attitude sensors, depth sensors, etc. for fusion analysis. However, although the multi-sensor system in the existing technical solution introduces multiple sensors, due to the inconsistent communication protocols of the multiple sensors, there is a lack of effective fusion mechanism for the synchronization of data from different sensors, resulting in the data timestamp and frequency being out of sync, causing the data accuracy to be reduced.

[0003] Although existing methods use a central controller to uniformly schedule multiple sensors and implement data timing correction to ensure that sensors of different frequencies can work relatively synchronously, this solution faces problems of data transmission delay and frequency alignment during the timing correction and protocol conversion process. Especially in high-frequency sampled acoustic data, the accumulated delay has a great impact on the accuracy. In addition, each sensor is connected to the central controller through its own interface, which increases the burden of data transmission and timing synchronization, making it difficult to meet the needs of real-time data fusion, and thus difficult to achieve the accuracy of marine environment monitoring. Summary of the invention

[0004] In order to solve the above-mentioned problems existing in the prior art of multi-sensor data integration in ocean monitoring, such as data synchronization fusion, inconsistent communication protocols and sensor data timing correction, this application proposes a multi-sensor data synchronization acquisition and fusion processing method and system.

[0005] In a first aspect of the present application, the present application discloses a multi-sensor data synchronous acquisition, fusion and processing system, characterized in that it includes: The data acquisition module is used to collect data from multiple sensors and transmit the data to the host computer through the controller for subsequent processing; Sensor communication module, used for data transmission and communication protocol conversion of multiple sensors, ensuring that various sensors can exchange data through a unified bus; The timestamp alignment module is used to accurately align the data timestamps of different sensors through the clock synchronization mechanism and the time offset correction model; The data fusion module uses a dynamic synchronous fusion algorithm based on time series and frequency to perform real-time fusion processing on the data of multiple sensors to obtain fused data.

[0006] Further, the plurality of sensors include a gesture sensor, a depth sensor and a hydrophone.

[0007] Furthermore, the sensor communication module adopts CAN bus or Ethernet for data transmission; the CAN bus is used to transmit low-speed data, and the Ethernet is used to transmit high-speed data.

[0008] Furthermore, the data acquisition module uses multi-channel parallel acquisition of data from multiple sensors, and pre-stores the acquired data in a local cache. When the amount of data in the local cache reaches a preset value, the acquired data is transmitted to a host computer through a relay controller for subsequent processing.

[0009] Furthermore, the timestamp alignment module includes a clock synchronization module, which uses a master-slave clock synchronization protocol to periodically calibrate the clock of each sensor through a clock synchronization mechanism to maintain time consistency.

[0010] Furthermore, the timestamp alignment module includes a time offset correction module, which calculates a time offset correction value through the clock difference between sensors, and adjusts the time of each collected data according to the correction value to ensure that the data of all sensors can be aligned under the same time reference.

[0011] Furthermore, the data fusion module adopts a dynamic synchronous fusion algorithm based on time series and frequency to perform real-time fusion processing on the data of multiple sensors, and the fused data obtained includes: the data fusion module sorts and aligns the collected data of different sensors according to timestamps, and then uses a frequency alignment algorithm to fuse the high-frequency collected data and the low-frequency data in the same time series to obtain a comprehensive data set without time delay difference.

[0012] Furthermore, the system also includes a cloud computing module, which is used to upload the fusion data obtained by the data fusion module to the cloud computing platform. The cloud computing platform uses a machine learning algorithm to train the fusion algorithm based on the fusion data to optimize the fusion algorithm.

[0013] In a second aspect of the present application, the present application further provides a multi-sensor data synchronous acquisition and fusion processing method, characterized in that it includes: Step S1, using a master-slave clock synchronization protocol to calibrate the clocks inside different sensors; Step S2, collecting data from a gesture sensor, a depth sensor, and a hydrophone through multiple channels, and caching the multi-sensor data locally; Step S3, when the cached data is greater than a preset value, the cached multi-sensor data is transmitted to the host computer through a unified communication protocol; Step S4, the timestamp alignment module calculates a time offset correction value based on the clock difference of different sensors, and adjusts the time of each collected data according to the correction value to ensure that the data of all sensors can be aligned under the same time reference; Step S5, sorting and aligning the time-aligned multi-sensor data by timestamps, and then fusing the high-frequency collected data with the low-frequency data in the same time series through a frequency alignment algorithm to ensure that there is no time delay difference between the data, and obtaining a fused comprehensive data set; Furthermore, the method also includes: step S6, uploading the fusion data obtained by the data fusion module to the cloud computing platform, and the cloud computing platform uses a machine learning algorithm to train the fusion algorithm according to the fusion data to optimize the fusion algorithm.

[0014] This application designs a unified multi-sensor communication protocol conversion interface to achieve unified connection between different sensors and sensor communication modules and uniform transmission of multi-sensor data to the host computer, then aligns the data timestamps of different sensors through a clock synchronization mechanism and a time offset correction model, and uses a dynamic fusion algorithm based on timing and frequency to perform real-time fusion processing on the data of multiple sensors, thereby realizing real-time fusion of different multi-sensor data, improving the accuracy of multi-sensor data fusion, and improving the effect of marine underwater environment monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of a multi-sensor data synchronous acquisition, fusion and processing system in this application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to further clearly and completely describe the technical solutions in the embodiments of the present application. It should be noted that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0017] In order to make the invention purpose, technical scheme and advantages of this application clearer, the technical scheme of the present invention is described clearly and completely in conjunction with the accompanying drawings and specific implementation schemes. It should be pointed out that for ordinary technicians in this field, several changes and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of this application.

[0018] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings: like Figure 1As shown, in the first embodiment of the present application, a multi-sensor data synchronous acquisition and fusion processing system is provided, which is characterized by comprising: The data acquisition module is used to collect data from multiple sensors and transmit the data to the host computer through the controller for subsequent processing; Sensor communication module, used for data transmission and communication protocol conversion of multiple sensors, ensuring that various sensors can exchange data through a unified bus; The timestamp alignment module is used to accurately align the data timestamps of different sensors through the clock synchronization mechanism and the time offset correction model; The data fusion module uses a dynamic synchronous fusion algorithm based on time series and frequency to perform real-time fusion processing on the data of multiple sensors to obtain fused data.

[0019] In this embodiment, the multiple sensors include a posture sensor, a depth sensor and a hydrophone. The posture sensor is used to obtain the posture information of the target, the depth sensor is used to obtain the depth information, and the hydrophone is used to obtain the information of the seabed topography and the like.

[0020] In one embodiment of the present application, the data acquisition module includes a multi-channel synchronous acquisition device and a cache device. The multi-channel synchronous acquisition device is used to synchronously collect data from multiple different sensors. Since the data sampling frequencies of different sensors are different, different sampling frequencies need to be configured. In this embodiment, the sampling frequency of the hydrophone can be set to 5-10kHz, the attitude sensor to 1kHz, and the depth sensor to 500Hz. The collected data from different sensors with timestamps are preferentially cached locally in the cache device to avoid packet loss during data transmission. When the cached data reaches a certain preset value, the cached data is transmitted to the host computer through a unified transmission channel.

[0021] When the sensor communication module transmits the cached multi-sensor data, in order to avoid asynchrony of the communication protocol, a unified communication protocol is selected for data transmission. In this embodiment, Ethernet or CAN bus is used as the medium for data transmission. It should be noted that the present application can also select Ethernet or CAN bus according to the actual data transmission speed. Specifically, CAN bus is used to transmit low-speed data, and Ethernet is used to transmit high-speed data.

[0022] In order to ensure the consistency of data transmission, the sensor communication module in this embodiment also includes a communication protocol adapter, which is used to convert the data protocols of different sensors into a unified transmission network, thereby realizing unified transmission and communication of multiple sensors.

[0023] In this embodiment, the timestamp alignment module includes a clock synchronization module, which uses a master-slave clock synchronization protocol to periodically calibrate the clock of each sensor through a clock synchronization mechanism to maintain time consistency. Further, the timestamp alignment module includes a time offset correction module, which calculates a time offset correction value through the clock difference between sensors, and adjusts the time of each collected data according to the correction value, ensuring that the data of all sensors can be aligned under the same time reference.

[0024] In this embodiment, PTP (Precision Time Protocol) is used to synchronize the timestamps of the sensor data. By exchanging synchronization messages between the master and slave devices and recording the message sending time, the network transmission delay and the deviation of the clocks between the master and slave devices are calculated. The network transmission delay calculation method is as follows: The master clock sends a Sync message to the slave clock and records the sending timestamp t1. After the slave clock receives the message, it records the receiving timestamp as t2. The slave clock sends a Pdelay_Req message to the master clock to initiate the calculation of the reverse transmission delay and records the sending timestamp as t3. After the master clock receives the message, it records the receiving timestamp as t4. Based on these timestamps, the total round-trip delay between the master and slave devices can be calculated as (t2-t1)+(t4-t3); since the network is symmetrical, the one-way delay can be estimated as half of the total round-trip delay, that is, (t2-t1)+(t4-t3) / 2.

[0025] The clock deviation (Offset) of the slave clock relative to the master clock can be calculated by the following formula: Offset = (t2-t1)-(t2-t1)+(t4-t3) / 2. This formula actually calculates the difference between the time when the slave clock receives the Sync message and the time when the master clock sends the Sync message, minus the one-way delay.

[0026] In this embodiment, before data collection, the time clocks of different sensors may be aligned.

[0027] In this embodiment, the data fusion module sorts and aligns the collected data from different sensors according to timestamps, and then fuses the high-frequency collected data with the low-frequency data in the same time series through a frequency alignment algorithm to obtain a comprehensive data set without time delay difference.

[0028] Furthermore, the system also includes a cloud computing module, which is used to upload the fusion data obtained by the data fusion module to the cloud computing platform. The cloud computing platform uses a machine learning algorithm to train the fusion algorithm based on the fusion data to optimize the fusion algorithm.

[0029] In the second embodiment of the present application, the present application further provides a multi-sensor data synchronous acquisition and fusion processing method, which is characterized by comprising: Step S1, using a master-slave clock synchronization protocol to calibrate the clocks inside different sensors; Step S2, collecting data from a gesture sensor, a depth sensor, and a hydrophone through multiple channels, and caching the multi-sensor data locally; Step S3, when the cached data is greater than a preset value, the cached multi-sensor data is transmitted to the host computer through a unified communication protocol; Step S4, the timestamp alignment module calculates a time offset correction value based on the clock difference of different sensors, and adjusts the time of each collected data according to the correction value to ensure that the data of all sensors can be aligned under the same time reference; Step S5, sorting and aligning the time-aligned multi-sensor data by timestamps, and then fusing the high-frequency collected data with the low-frequency data in the same time series through a frequency alignment algorithm to ensure that there is no time delay difference between the data, and obtaining a fused comprehensive data set; Furthermore, the method also includes: step S6, uploading the fusion data obtained by the data fusion module to the cloud computing platform, and the cloud computing platform uses a machine learning algorithm to train the fusion algorithm according to the fusion data to optimize the fusion algorithm.

[0030] This application designs a unified multi-sensor communication protocol conversion interface to achieve unified connection between different sensors and sensor communication modules and uniform transmission of multi-sensor data to the host computer, then aligns the data timestamps of different sensors through a clock synchronization mechanism and a time offset correction model, and uses a dynamic fusion algorithm based on timing and frequency to perform real-time fusion processing on the data of multiple sensors, thereby realizing real-time fusion of different multi-sensor data, improving the accuracy of multi-sensor data fusion, and improving the effect of marine underwater environment monitoring.

Claims

1. A multi-sensor data synchronous acquisition and fusion processing system, characterized in that: include: The data acquisition module is used to collect data from multiple sensors and transmit the data to the host computer through the controller for subsequent processing; Sensor communication module, used for data transmission and communication protocol conversion of multiple sensors, ensuring that various sensors can exchange data through a unified bus; The timestamp alignment module is used to accurately align the data timestamps of different sensors through the clock synchronization mechanism and the time offset correction model; The timestamp alignment module includes a clock synchronization module, which uses a master-slave clock synchronization protocol to regularly calibrate the clock of each sensor through a clock synchronization mechanism to maintain time consistency; The data fusion module uses a dynamic synchronous fusion algorithm based on time series and frequency to perform real-time fusion processing on the data of multiple sensors to obtain fused data.

2. A multi-sensor data synchronous acquisition, fusion and processing system according to claim 1, characterized in that: The plurality of sensors include a gesture sensor, a depth sensor, and a hydrophone.

3. A multi-sensor data synchronous acquisition, fusion and processing system according to claim 1, characterized in that: The sensor communication module uses CAN bus or Ethernet for data transmission.

4. A multi-sensor data synchronous acquisition, fusion and processing system according to claim 1, characterized in that: The data acquisition module uses multi-channel parallel acquisition of data from multiple sensors, and pre-stores the acquired data in a local cache. When the amount of data in the local cache reaches a preset value, the acquired data is transmitted to the host computer through a relay controller for subsequent processing.

5. A multi-sensor data synchronous acquisition, fusion and processing system according to claim 1, characterized in that: The timestamp alignment module includes a time offset correction module, which calculates a time offset correction value through the clock difference between sensors, and adjusts the time of each collected data according to the correction value to ensure that the data of all sensors are aligned under the same time reference.

6. A multi-sensor data synchronous acquisition, fusion and processing system according to claim 1, characterized in that: The data fusion module adopts a dynamic synchronous fusion algorithm based on time series and frequency to perform real-time fusion processing on the data of multiple sensors, and the fused data obtained includes: the data fusion module sorts and aligns the collected data of different sensors according to timestamps, and then uses a frequency alignment algorithm to fuse the high-frequency collected data and the low-frequency data in the same time series to obtain a comprehensive data set without time delay difference.

7. A multi-sensor data synchronous acquisition, fusion and processing system according to claim 1, characterized in that: The system also includes a cloud computing module, which is used to upload the fusion data obtained by the data fusion module to a cloud computing platform. The cloud computing platform uses a machine learning algorithm to train the fusion algorithm based on the fusion data to optimize the fusion algorithm.

8. A method based on the multi-sensor data synchronous acquisition, fusion and processing system according to any one of claims 1 to 7, characterized in that: include: Step S1, using a master-slave clock synchronization protocol to calibrate the clocks inside different sensors; Step S2, collecting data from a gesture sensor, a depth sensor, and a hydrophone through multiple channels, and caching the multi-sensor data locally; Step S3, when the cached data is greater than a preset value, the cached multi-sensor data is transmitted to the host computer through a unified communication protocol; Step S4, the timestamp alignment module calculates a time offset correction value based on the clock difference of different sensors, and adjusts the time of each collected data according to the correction value to ensure that the data of all sensors are aligned under the same time reference; Step S5, sorting and aligning the time-aligned multi-sensor data by timestamp, and then fusing the high-frequency collected data with the low-frequency data in the same time series through a frequency alignment algorithm to ensure that there is no time delay difference between the data, and obtain a fused comprehensive data set.

9. The method according to claim 8, characterized in that The method further comprises: Step S6, uploading the fusion data obtained by the data fusion module to the cloud computing platform, and the cloud computing platform uses a machine learning algorithm to train the fusion algorithm according to the fusion data to optimize the fusion algorithm.

Images