Construction method and system of isometric sensor spatiotemporal consistency

Abstract

The embodiment of the application discloses a method and system for constructing spatio-temporal consistency of heterogeneous sensors, and relates to the technical field of sensor data processing. The method comprises the following steps: identifying the types of sensors in a sensor array, configuring the inherent parameters of each sensor, and performing time synchronization initialization; based on a static target with a known spatial coordinate, calculating an initial conversion relationship from each sensor coordinate system to a unified carrier coordinate system; based on the running process of the sensor array, optimizing the initial conversion relationship by matching dynamic characteristics in the sensor data, and obtaining a dynamic conversion relationship for real-time spatial coordinate alignment; based on a timestamp, modeling and compensating the drift of each sensor clock, and correcting the timestamp; converting the collected data to the unified carrier coordinate system and performing data fusion, and performing spatio-temporal consistency verification on the data after data fusion.

Description

Technical Field

[0001] This application relates to the technical field of sensor data processing, and in particular to a method and system for constructing spatiotemporal consistency of heterogeneous sensors. Background Technology

[0002] With the rapid development of IoT, AI, and industrial automation technologies, heterogeneous sensor systems have been widely applied in numerous fields such as intelligent transportation, environmental monitoring, industrial inspection, and healthcare. Heterogeneous sensor systems typically consist of various types of sensors, such as vision sensors, LiDAR, DIC, and strain gauge sensors. These different sensors exhibit significant differences in data dimensions, sampling frequencies, spatiotemporal references, and measurement accuracy. In the application of heterogeneous sensor systems, the spatiotemporal consistency of sensor data is a core prerequisite for ensuring the accuracy of subsequent data fusion, feature extraction, and decision analysis.

[0003] However, existing technologies suffer from the following key problems: 1. Spatial coordinate misalignment: Different sensors are installed in different locations, and even when installed on the same carrier, their coordinate origins and measurement axes are different; 2. Low time synchronization accuracy: The sampling frequencies of heterogeneous sensors differ significantly, and the internal clocks of each sensor drift, resulting in a lack of a unified time reference. Traditional synchronization methods based on Network Time Protocol (NTP) can only achieve millisecond-level accuracy, which cannot meet the microsecond-level time synchronization requirements of high-precision scenarios; 3. Poor adaptability to dynamic scenarios: In dynamic scenarios where the carrier is moving, the spatial attitude of the sensors changes in real time with the carrier's movement, and static coordinate calibration methods cannot compensate for attitude shifts during movement; at the same time, the sampling triggering timing of sensors in motion is easily affected by vibration and electromagnetic interference, further exacerbating spatiotemporal deviations; 4. Complex calibration process and low robustness: Existing calibration methods mostly rely on manual target setting or offline calibration, which is cumbersome. In complex environments, calibration parameters are prone to failure, making dynamic online calibration impossible, leading to a gradual decline in spatiotemporal consistency during long-term system operation.

[0004] To address the aforementioned issues, there is an urgent need for a solution that can achieve dynamic alignment of spatiotemporal coordinates, high-precision time synchronization, and robust data acquisition for heterogeneous sensors, in order to improve the data quality and application reliability of heterogeneous sensor systems. Summary of the Invention

[0005] This application proposes a method and system for constructing spatiotemporal consistency of heterogeneous sensors to address the shortcomings of the prior art.

[0006] According to a first aspect of the embodiments of this application, a method for constructing spatiotemporal consistency of heterogeneous sensors is provided, comprising: S1: Identify the type of each sensor in the sensor array, configure the inherent parameters of each sensor, perform time synchronization initialization, and establish a unified global reference time; S2: Based on a static target with known spatial coordinates, calculate the initial transformation relationship from each sensor coordinate system to the unified carrier coordinate system; S3: During the operation of the sensor array, the initial transformation relationship is optimized by matching the dynamic features in the data of each sensor to obtain a dynamic transformation relationship for real-time spatial coordinate alignment; S4: Configure timestamps for the data acquisition process of each sensor based on the global reference time, and correct the timestamps by modeling and compensating for the clock drift of each sensor, for time synchronization between sensors; S5: Based on the dynamic conversion relationship and the corrected timestamp, control each sensor to perform synchronous data acquisition, convert the acquired data to the unified carrier coordinate system and perform data fusion, and perform spatiotemporal consistency verification on the data after data fusion; if the degradation index in the spatiotemporal consistency verification result exceeds the set threshold, trigger and execute steps S3 to S4 for online updating of calibration parameters; otherwise, continue to control each sensor to perform synchronous data acquisition based on the current calibration parameters.

[0007] In some implementations, calculating the initial transformation relationship from each sensor coordinate system to a unified carrier coordinate system based on a static target with known spatial coordinates includes: The system controls each sensor to synchronously acquire data from targets arranged within its field of view. Each target has multiple feature points with known spatial coordinates. The spatial coordinates of the feature points in their respective sensor coordinate systems are extracted from the data collected by each sensor. Using the spatial coordinates of the feature points as a reference, the initial transformation matrix of each sensor coordinate system relative to the unified carrier coordinate system is calculated through a coordinate transformation algorithm.

[0008] In some implementations, during the operation of the sensor array-based system, the initial transformation relationship is optimized by matching dynamic features in the data from each sensor to obtain a dynamic transformation relationship for real-time spatial coordinate alignment, including: During the operation of the sensor array, dynamic features that satisfy interrelationships are extracted from the real-time data streams of each sensor. Based on the correspondence of the dynamic features in different sensor data, the initial transformation matrix is ​​iteratively optimized using a graph optimization algorithm to generate the dynamic transformation relationship that is updated in real time.

[0009] In some implementations, configuring timestamps for the data acquisition process of each sensor based on the global reference time, and correcting the timestamps by modeling and compensating for the clock drift of each sensor for time synchronization between sensors, includes: The global reference time is encapsulated into a synchronization signal and sent to each sensor, so that each sensor can embed the original timestamp associated with the synchronization signal into the data it collects; The local clock of each sensor is periodically compared with the global reference time to obtain a clock difference sequence; Based on the clock difference sequence, a clock drift model for each sensor is established using a filtering estimation algorithm; The original timestamp is compensated in real time by the clock drift model to obtain a globally synchronized corrected timestamp for time synchronization between sensors.

[0010] In some embodiments, the method further includes: Set the target acquisition frequency; Downsampling is performed on sensor data streams with acquisition frequencies higher than the target acquisition frequency, and interpolation is performed on sensor data streams with acquisition frequencies lower than the target acquisition frequency, in order to confirm the uniform alignment of all sensor data in the time dimension.

[0011] In some implementations, spatiotemporal consistency verification is performed on the data after data fusion, including: Determine whether the time deviation of each sensor data under the same time label exceeds a first time threshold; and based on the dynamic characteristics, determine whether the spatial position deviation of the same target in different sensor data exceeds a first spatial threshold; Data whose time deviation exceeds a first time threshold and / or whose spatial location deviation exceeds a first spatial threshold are marked as abnormal data; The abnormal data can be repaired by interpolation or the data at the corresponding time point can be re-acquired.

[0012] In some implementations, the degradation index includes the mean time deviation and the mean spatial deviation. When the degradation index in the result of the spatiotemporal consistency verification exceeds a set threshold, steps S3 to S4 are triggered and executed, including: If the average time deviation of the collected data exceeds the second time threshold and / or the average spatial deviation of the collected data exceeds the second spatial threshold within multiple consecutive acquisition cycles, steps S3 to S4 are triggered and executed; if not, the current calibration parameters are maintained.

[0013] In some embodiments, the method further includes: Save updated calibration parameters and historical parameter versions; If data quality deteriorates after adopting new parameters, the system will automatically revert to the previous valid parameter version.

[0014] In some implementations, the filtering estimation algorithm is a Kalman filter algorithm, and the clock drift model is a time-based linear function model.

[0015] According to a second aspect of this application, a system for constructing spatiotemporal consistency of heterogeneous sensors is provided, comprising: The clock initialization module is used to identify the type of each sensor in the sensor array, configure the inherent parameters of each sensor, perform time synchronization initialization, and establish a unified global reference time. The static spatial calibration module is used to calculate the initial transformation relationship from each sensor coordinate system to the unified carrier coordinate system based on a static target with known spatial coordinates. The dynamic space optimization module is used to optimize the initial transformation relationship by matching the dynamic features in the data of each sensor during the operation of the sensor array, so as to obtain a dynamic transformation relationship for real-time spatial coordinate alignment. The time synchronization module is used to configure timestamps for the data acquisition process of each sensor based on the global reference time, and to correct the timestamps by modeling and compensating for the drift of the clocks of each sensor, for time synchronization between sensors. The acquisition and calibration module is used to control each sensor to synchronously acquire data based on the dynamic transformation relationship and the corrected timestamp, convert the acquired data to the unified carrier coordinate system and perform data fusion, and perform spatiotemporal consistency verification on the fused data; if the degradation index in the spatiotemporal consistency verification result exceeds a set threshold, the operation of the dynamic spatial optimization module and the time synchronization module is triggered and executed for online updating of calibration parameters; otherwise, the synchronous data acquisition of each sensor continues to be controlled based on the current calibration parameters.

[0016] The beneficial effects of the method and system for constructing spatiotemporal consistency of heterogeneous sensors in this application include at least the following: This application embodiment systematically identifies and configures sensor parameters and establishes a unified global time base, providing a precise and consistent initial reference system for all subsequent spatiotemporal alignment operations. This fundamentally avoids system-level deviations caused by unknown sensor parameters or inconsistent time bases, ensuring an effective starting point for subsequent processing. This application embodiment utilizes a static target with known spatial coordinates to provide a high-precision, quantifiable initial estimate of the transformation relationship from each sensor coordinate system to a unified carrier coordinate system. This solves the problem of initial spatial misalignment caused by varying sensor installation positions and angles, laying an accurate mathematical foundation for subsequent optimization and avoiding initial value errors or convergence problems that might be introduced by relying entirely on dynamic matching. During system operation, this application embodiment continuously optimizes the initial transformation relationship by matching dynamic features in real time. This effectively compensates for changes in the relative positions between sensors caused by factors such as carrier movement, mechanical vibration, or structural deformation, achieving an improvement in spatial alignment parameters from static accuracy to dynamic accuracy, and enhancing the spatial consistency of the system in real-world working scenarios. This application's embodiments solve the inherent time asynchrony problem caused by different sampling start times and crystal oscillator frequency differences by assigning a global reference timestamp and modeling to compensate for clock drift of each sensor. This not only achieves alignment of acquisition times but also continuously corrects accumulated clock errors during operation, thus achieving high-precision and continuously stable data synchronization in the time dimension. This application's embodiments drive synchronous acquisition and fusion based on aligned spatiotemporal parameters and monitor data quality in real time through a closed-loop verification mechanism. Its core benefit lies in constructing an autonomous feedback loop including monitoring, judgment, and calibration. That is, when data is abnormal, recalibration is triggered to restore accuracy; when data is normal, the current efficient operating state is maintained, significantly improving the system's long-term robustness, adaptability, and sustainable data reliability. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for constructing spatiotemporal consistency of heterogeneous sensors according to an embodiment of this application. Figure 2 This is a schematic diagram of the hardware layout structure of an embodiment of this application; Figure 3 This is a schematic diagram of the spatial coordinate pre-alignment process according to an embodiment of this application; Figure 4 This is a schematic diagram of the timestamp anchoring process according to an embodiment of this application; Figure 5 This is a schematic diagram of the clock drift detection and compensation process according to an embodiment of this application; Figure 6 This is a schematic diagram of a system for constructing spatiotemporal consistency of heterogeneous sensors according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the various embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this disclosure to facilitate a better understanding of the disclosure. However, the technical solutions claimed in this disclosure can be implemented even without these technical details and with various variations and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of this disclosure. The various embodiments can be combined with and referenced by each other without contradiction.

[0019] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of the present application, but merely to illustrate selected embodiments of the present application. Other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are all within the scope of protection of the embodiments of the present application.

[0020] This application discloses a method and system for constructing spatiotemporal consistency of heterogeneous sensors. The method for constructing spatiotemporal consistency of heterogeneous sensors is implemented based on the system for constructing spatiotemporal consistency of heterogeneous sensors. The purpose is to overcome the shortcomings of the prior art and provide a scheme for spatiotemporal coordinate alignment and synchronous acquisition of heterogeneous sensors. This enables dynamic calibration of spatial coordinates of heterogeneous sensors, high-precision synchronization of the time dimension, and stable data acquisition in complex scenarios, thereby improving the spatiotemporal consistency of sensor data.

[0021] See attached document Figure 1 The diagram shows a flowchart of a method for constructing spatiotemporal consistency of heterogeneous sensors according to an embodiment of this application, including the following steps S1 to S5.

[0022] S1: Identify the type of each sensor in the sensor array, configure the inherent parameters of each sensor, perform time synchronization initialization, and establish a unified global reference time.

[0023] Step S1 can be understood as a step used to process the initialization and parameter configuration of the heterogeneous sensor system.

[0024] For example, before performing step S1, refer to the appendix. Figure 2 As shown, this method is first implemented based on a hardware system consisting of a main controller, a heterogeneous sensor array, a time synchronization module, and a data cache module. The main controller establishes communication with each sensor through a bus interface to complete sensor type identification, basic parameter configuration, and initial state detection.

[0025] For example, step S1 specifically includes three sub-steps: sensor type identification and parameter input, time synchronization module initialization, and initial state detection. Sensor type identification and parameter input includes: determining the sensor type based on the device identifier of each sensor read by the main controller, and inputting its inherent parameters, including sampling frequency f, measurement range, intrinsic parameter matrix, initial coordinates of installation position, and initial attitude angle. Time synchronization module initialization includes: starting the time synchronization module, which includes a GPS / BeiDou dual-mode positioning unit, a high-stability crystal oscillator, and a timestamp generator. UTC time is obtained through the GPS / BeiDou unit as the system reference time T0, and the high-stability crystal oscillator provides a high-frequency clock signal for high-precision timestamp generation. Time synchronization accuracy is calibrated to the microsecond level. Initial state detection includes: sending self-test commands to each sensor based on the main controller, verifying the stability of the communication link and the sensor's working status, removing faulty sensors, and generating an initial device list.

[0026] S2: Based on the known spatial coordinates of the static target, calculate the initial transformation relationship from the coordinate systems of each sensor to the unified carrier coordinate system.

[0027] In some implementations, the calculation of the initial transformation relationship from each sensor coordinate system to the unified carrier coordinate system based on the static target with known spatial coordinates includes: controlling each sensor to synchronously collect target data arranged within its field of view, wherein each target has multiple feature points with known spatial coordinates; extracting the spatial coordinates of the feature points in their respective sensor coordinate systems from the data collected by each sensor; and calculating the initial transformation matrix of each sensor coordinate system relative to the unified carrier coordinate system using the spatial coordinates of the feature points as a reference through a coordinate transformation algorithm.

[0028] S3: During the operation of the sensor array, the initial transformation relationship is optimized by matching the dynamic features in the data of each sensor to obtain a dynamic transformation relationship for real-time spatial coordinate alignment.

[0029] In some implementations, during the operation of the sensor array, the initial transformation relationship is optimized by matching dynamic features in the data of each sensor to obtain a dynamic transformation relationship for real-time spatial coordinate alignment. This includes: extracting interrelated dynamic features from the real-time data streams of each sensor during the operation of the sensor array; and iteratively optimizing the initial transformation matrix using a graph optimization algorithm based on the correspondence of the dynamic features in different sensor data to generate the dynamically updated dynamic transformation relationship in real time.

[0030] Steps S2 and S3 can be understood as steps for pre-aligning spatial coordinates based on multi-source features.

[0031] For example, steps S2 and S3 employ a two-stage calibration method of static target calibration and dynamic feature matching to achieve pre-alignment of spatial coordinates. Addressing the spatial coordinate differences between heterogeneous sensors, step S2 specifically includes: static target calibration and visual sensor acquisition of target images; step S3 specifically includes: lidar acquisition of target point cloud data, main controller using the carrier coordinate system as a unified reference, and dynamic feature matching optimization. In combination, static target calibration includes: placing a target (a calibration board with three-dimensional feature points) within the sensor's field of view; the world coordinates (spatial coordinates) of each feature point on the target are known; and the main controller controls each sensor to synchronously acquire target data. Visual sensor acquisition of target images includes: extracting feature point pixel coordinates using a corner detection algorithm; combining the camera's intrinsic parameter matrix and perspective projection model; and calculating the coordinates of the feature points in the camera coordinate system. The initial transformation relationship from each sensor coordinate system to the unified carrier coordinate system is also calculated. The lidar acquires target point cloud data, including: segmenting the target area using a clustering algorithm and extracting the 3D coordinates of feature points in the lidar coordinate system; the main controller uses the carrier coordinate system as a unified reference, including: calculating the initial transformation matrix T of each sensor relative to the carrier coordinate system based on the world coordinates of the target feature points and the coordinates in each sensor's coordinate system using the ICP algorithm iterating the nearest point. a The transformation matrix contains the translation vector. and rotation matrix R a Dynamic feature matching optimization includes: when the carrier is in motion, the main controller extracts dynamic features from the data of each sensor in real time and constructs a multi-source feature association graph; the initial transformation matrix T is optimized using a graph optimization algorithm. a After correction, the dynamic transformation matrix T is obtained. b This enables real-time alignment of spatial coordinates.

[0032] S4: Configure timestamps for the data acquisition process of each sensor based on the global reference time, and correct the timestamps by modeling and compensating for the clock drift of each sensor, for time synchronization between sensors.

[0033] Step S4 can be understood as a step for high-precision time synchronization based on timestamp correction.

[0034] In some implementations, the process of configuring timestamps for the data acquisition process of each sensor based on the global reference time, and correcting the timestamps by modeling and compensating for the clock drift of each sensor, is used for time synchronization between sensors. This includes: encapsulating the global reference time into a synchronization signal and sending it to each sensor, so that each sensor can embed the original timestamp associated with the synchronization signal into the acquired data; periodically comparing the local clock of each sensor with the global reference time to obtain a clock difference sequence; establishing a clock drift model for each sensor based on the clock difference sequence using a filtering estimation algorithm; and using the clock drift model to compensate the original timestamps in real time to obtain a corrected timestamp for global synchronization, which is used for time synchronization between sensors.

[0035] For example, the filtering estimation algorithm is the Kalman filter algorithm, and the clock drift model is a time-based linear function model.

[0036] Step S4 employs a combination of reference time anchoring and clock drift compensation to achieve high-precision time synchronization among heterogeneous sensors. For example, step S4 specifically includes three sub-steps: timestamp anchoring, clock drift detection and compensation, and adaptive adjustment of the sampling frequency. Timestamp anchoring includes: the time synchronization module encapsulates the reference time T0 into a time synchronization frame, which is then sent to each sensor via a hard-wired trigger signal, triggering the sensors to embed a high-precision timestamp T when acquiring data. s For sensors that do not support hard-wired triggering, the main controller sends a timestamped synchronization command via the communication bus. The sensor receives the command timestamp and uses it as a reference for the data acquisition timestamp. Clock drift detection and compensation include: the main controller periodically acquiring the difference between the internal clock time t of each sensor and the reference time T0 to construct a clock drift model. A Kalman filter algorithm is used to filter the drift data, eliminating random errors and obtaining the clock drift amount. (Where, a is the drift rate and b is the initial offset). The main controller corrects the timestamp of the sensor data in real time according to the drift model. The corrected timestamp is... The sampling frequency is adaptively adjusted, including: the main controller constructs a sampling period mapping table based on the sampling frequency differences of each sensor. For high-frequency sensors, a data downsampling strategy is adopted, and data is extracted according to the sampling period of low-frequency sensors; for low-frequency sensors, a linear interpolation method is used to fill data gaps, ensuring the consistency of data density across all sensors in the time dimension.

[0037] S5: Based on the dynamic transformation relationship and the corrected timestamp, control each sensor to collect data synchronously, convert the collected data to the unified carrier coordinate system and perform data fusion, and perform spatiotemporal consistency verification on the data after data fusion.

[0038] For example, if the degradation index in the result of the spatiotemporal consistency verification exceeds the set threshold, steps S3 to S4 are triggered and executed for online updating of calibration parameters; otherwise, the control of each sensor to perform synchronous data acquisition continues based on the current calibration parameters.

[0039] Step S5 can be understood as a step for synchronous acquisition and data verification, as well as online dynamic calibration and parameter update for spatiotemporal fusion.

[0040] In some implementations, spatiotemporal consistency verification is performed on the data after data fusion, including: determining whether the time deviation of each sensor data under the same time label exceeds a first time threshold; and based on the dynamic feature, determining whether the spatial position deviation of the same target in different sensor data exceeds a first spatial threshold; marking the data whose time deviation exceeds the first time threshold and / or the data whose spatial position deviation exceeds the first spatial threshold as abnormal data; and interpolating and repairing the abnormal data or triggering the re-acquisition of data at the corresponding time.

[0041] In some implementations, the degradation index includes the mean time deviation and the mean spatial deviation. When the degradation index in the result of the spatiotemporal consistency verification exceeds a set threshold, steps S3 to S4 are triggered and executed, including: triggering and executing steps S3 to S4 when the mean time deviation corresponding to the collected data exceeds a second time threshold and / or the mean spatial deviation corresponding to the collected data exceeds a second spatial threshold within multiple consecutive collection cycles; if not, the current calibration parameters are maintained.

[0042] In some implementations, the method further includes: setting a target acquisition frequency; downsampling sensor data streams with acquisition frequencies higher than the target acquisition frequency, and interpolating sensor data streams with acquisition frequencies lower than the target acquisition frequency, to confirm uniform alignment of all sensor data in the time dimension.

[0043] In some implementations, the method further includes: saving the updated calibration parameters and historical parameter versions; and automatically reverting to the previous valid parameter version when the data quality deteriorates after adopting the new parameters.

[0044] Step S5, based on the aligned spatiotemporal coordinates and synchronized timestamps, enables the synchronous acquisition, fusion caching, and quality verification of heterogeneous sensor data by the main controller. For example, S5 specifically includes sub-steps such as: synchronous acquisition triggering, data spatiotemporal fusion, data quality verification, time consistency verification, spatial consistency verification, and abnormal data processing. Among these, synchronous acquisition triggering includes: the main controller, based on the corrected timestamps... This generates synchronous acquisition commands to control each sensor to complete data acquisition within the allowable time error range, ensuring data time synchronization. Data spatiotemporal fusion includes: each sensor dynamically transforming the acquired data with the corrected spatiotemporal information using a dynamic transformation matrix T. b and corrected timestamp The data is uploaded to the main controller, which categorizes and stores it according to timestamp order. It then uses a spatial transformation matrix T to convert all sensor data to a unified carrier coordinate system, forming a spatiotemporally fused sensor dataset. Data quality verification includes: the main controller uses a multi-dimensional verification mechanism to verify the fused data. Temporal consistency verification includes: calculating the time deviation of data from different sensors at the same timestamp; if the deviation exceeds a threshold, it is marked as time-abnormal data. Spatial consistency verification includes: verifying the spatial position deviation of the same target in different sensor data based on a dynamic feature association graph; if the deviation exceeds a threshold, it is marked as spatial-abnormal data. Abnormal data processing includes: for abnormal data, the main controller uses interpolation to repair it; if the accuracy requirements are still not met after repair, a re-acquisition command is triggered.

[0045] In addition, to address issues such as sensor installation misalignment and equipment aging during long-term operation, the main controller monitors the spatiotemporal consistency index of sensor data in real time. When the index falls below a preset threshold, an online dynamic calibration process is initiated. That is, step S5 also includes sub-steps such as calibration triggering, dynamic calibration execution, and parameter saving and backtracking. Calibration triggering includes: the main controller calculates the average spatiotemporal deviation of N consecutive frames of data. If the average time deviation is greater than 20 μs or the average spatial deviation is greater than 10 cm, online calibration is triggered. Dynamic calibration execution includes: repeating the dynamic feature matching optimization process in step two and the clock drift model update process in step three, recalculating the dynamic transformation matrix T and the clock drift Δt(t), and updating the calibration parameters in the main controller. Parameter saving and backtracking includes: the main controller stores the updated calibration parameters in non-volatile memory, while retaining historical parameter versions. If new parameters cause a decrease in data quality, it can automatically backtrack to the previous valid parameter version.

[0046] This application embodiment systematically identifies and configures sensor parameters and establishes a unified global time base, providing a precise and consistent initial reference system for all subsequent spatiotemporal alignment operations. This fundamentally avoids system-level deviations caused by unknown sensor parameters or inconsistent time bases, ensuring an effective starting point for subsequent processing. This application embodiment utilizes a static target with known spatial coordinates to provide a high-precision, quantifiable initial estimate of the transformation relationship from each sensor coordinate system to a unified carrier coordinate system. This solves the problem of initial spatial misalignment caused by varying sensor installation positions and angles, laying an accurate mathematical foundation for subsequent optimization and avoiding initial value errors or convergence problems that might be introduced by relying entirely on dynamic matching. During system operation, this application embodiment continuously optimizes the initial transformation relationship by matching dynamic features in real time. This effectively compensates for changes in the relative positions between sensors caused by factors such as carrier movement, mechanical vibration, or structural deformation, achieving an improvement in spatial alignment parameters from static accuracy to dynamic accuracy, and enhancing the spatial consistency of the system in real-world working scenarios. This application's embodiments solve the inherent time asynchrony problem caused by different sampling start times and crystal oscillator frequency differences by assigning a global reference timestamp and modeling to compensate for clock drift of each sensor. This not only achieves alignment of acquisition times but also continuously corrects accumulated clock errors during operation, thus achieving high-precision and continuously stable data synchronization in the time dimension. This application's embodiments drive synchronous acquisition and fusion based on aligned spatiotemporal parameters and monitor data quality in real time through a closed-loop verification mechanism. Its core benefit lies in constructing an autonomous feedback loop including monitoring, judgment, and calibration. That is, when data is abnormal, recalibration is triggered to restore accuracy; when data is normal, the current efficient operating state is maintained, significantly improving the system's long-term robustness, adaptability, and sustainable data reliability.

[0047] To further explain the solutions in this application, this application embodiment conducts pure torsional performance tests on a sub-component of a key cross-section of a certain type of wind turbine blade, and provides an example of the method for constructing spatiotemporal consistency of a heterogeneous sensor. Through high-precision spatiotemporal synchronous acquisition by the heterogeneous sensor, strain distribution, tilt deformation, and three-dimensional displacement data of the sub-component under torsional conditions are obtained, providing accurate data support for the mechanical property analysis and failure mechanism research of the sub-component. This specific embodiment may include the following steps.

[0048] The initialization and parameter configuration steps for a heterogeneous sensor system include: constructing a hardware system consisting of a main controller, a heterogeneous sensor array, a time synchronization module, and a data buffer module; the main controller establishing communication with each sensor through a bus interface to complete sensor type identification, basic parameter configuration, and initial state detection. The corresponding hardware system structure principle is as follows: Figure 2As shown, the system integrates a main controller, a heterogeneous sensor array, a time synchronization module, and a data storage module, all connected via communication lines. The main controller communicates with the sensor array and each module through a bus interface. The heterogeneous sensor array involves determining the type, selection, and parameters of the sensors. Sensor installation and initial parameter input can be divided into two sub-steps: determining the type, selection, and parameters of the core heterogeneous sensor, and sensor installation and initial parameter input. The sub-step of determining the type, selection, and parameters of the core heterogeneous sensor includes: inputting strain gauges of model BE120-6AA-P150; attaching them axially along the sub-component, covering the main beam, front and rear edge bonding joints, and key areas of the core plate; sampling frequency 100Hz; measurement range ±2000µε; used to collect the axial strain response of the material; and simultaneously setting temperature compensation gauges to balance the influence of ambient temperature on the measurement results. The entered DIC sensor selection and parameters include: two high-resolution industrial cameras with a frame rate of 30Hz and a pixel count of 1920×1080, deployed on the PS and SS surfaces of the sub-component respectively, covering the entire 5m long test section, to acquire the three-dimensional displacement and strain field distributions. The entered tilt sensor selection and parameters include: arranged at 1m intervals on the leading and trailing edges of the sub-component, with a measurement range of ±15° and a sampling frequency of 50Hz, recording the torsional deformation angle of each section. The entered force and displacement sensor selection and parameters include: a spoke-type force sensor mounted on the end of the cylinder push rod, and a magnetostrictive displacement sensor deployed at the bottom of the cylinder, with a sampling frequency of 100Hz, used to collect torsional load and displacement data at the applied end. The sensor installation and initial parameter input sub-steps include: strain gauge pasting: after cleaning the filler layer on the sub-component surface, the strain gauges are fixed in the preset positions with quick-drying adhesive to ensure tight adhesion to the material surface. The strain gauge density is increased in the main beam and adhesive joint areas, with a spacing of 50mm; DIC marking arrangement: high-contrast speckle black background with white spots is sprayed on the sub-component surface, with spot diameter of 2~3mm. The speckle distribution is uniformly distributed to cover the test area to avoid occlusion; initial parameter input: the main controller uses an STM32H743 to read the identification of each sensor device through the bus interface and input the inherent parameters, including the strain gauge sensitivity coefficient, DIC sensor intrinsic parameter matrix and perspective model, tilt sensor installation attitude angle, and the initial coordinates (x0, y0, z0) of each sensor relative to the sub-component test reference point. For example, the time synchronization module initialization includes: the time synchronization module is integrated with a GPS / BeiDou dual-mode positioning unit, a 100MHz high-stability crystal oscillator, and a time stamp generator. The GPS / BeiDou dual-mode unit obtains UTC as the reference time T0, and the time synchronization accuracy reaches the microsecond level through high-stability crystal oscillator calibration. A hard-wired trigger signal is used to send synchronization commands to the strain gauge, force sensor, and displacement sensor. The DIC sensor receives the timestamped acquisition command via a bus, ensuring that all sensor data is embedded with a high-precision timestamp T from a unified reference.s For example, the initial state detection includes: the main controller sending self-test commands to each sensor to verify the stability of the communication link; the strain gauges feeding back resistance signals through the transmitter; the DIC sensor transmitting test image frames; the tilt sensor returning initial zero-position data; and generating a list of valid devices after removing faulty sensors to ensure the normal operation of the test system.

[0049] The spatiotemporal coordinate alignment steps based on multi-features are described in the appendix. Figure 3 As shown, to address the spatial coordinate differences between heterogeneous sensors, a two-stage calibration method combining static target calibration and dynamic feature matching is employed to achieve pre-alignment of spatial coordinates. The spatiotemporal coordinate alignment step based on multi-features includes: the static target calibration sub-step, which involves placing an AprilTag target with three-dimensional feature points within the field of view of the DIC sensor, where the world coordinates of the target feature points are known. The main controller controls the DIC sensor to acquire target images, extracts the pixel coordinates of the feature points using the Shi-Tomasi corner detection algorithm, and calculates the three-dimensional coordinates of the feature points in the DIC sensor coordinate system by combining the intrinsic parameter matrix and perspective projection model. The main controller uses the center of a certain section of a sub-component as the origin of a unified carrier coordinate system. Using the world coordinates of the feature points, their three-dimensional coordinates in the DIC sensor coordinate system, and the initial position data of the tilt sensor and strain gauge, the ICP algorithm is used to calculate the initial transformation matrix T of each sensor relative to the carrier coordinate system. a (including rotation matrix R) a Translation vector t a This achieves spatial coordinate pre-alignment. The spatiotemporal coordinate alignment step based on multi-features also includes a dynamic feature matching sub-step, which involves: during the specific testing process, when a sub-component undergoes torsional deformation, the main controller extracts in real-time the speckle displacement features acquired by the DIC sensor, the strain change features of the strain gauge, and the attitude change features of the tilt sensor to construct a multi-source feature association map. The initial transformation matrix T is then optimized using the g2o graph optimization algorithm. a Dynamic corrections are performed to obtain the dynamic transformation matrix T. b To compensate for the spatial positional offset of sensors caused by the deformation of the compensation sub-components, this ensures that data collected by different sensors are mapped to a unified carrier coordinate system, with a spatial alignment accuracy better than 0.1mm. The spatiotemporal coordinate alignment step based on multi-feature features also includes: a high-precision synchronization sub-step based on timestamp correction, employing a combination of reference time anchoring and clock drift compensation to achieve high-precision time synchronization of heterogeneous sensors; and adaptive adjustment of the sampling frequency to ensure uniform distribution of all sensor data along the time axis. For example, the reference time anchoring includes: the time synchronization module encapsulating the reference time T0 into a time synchronization frame, sending synchronization commands to the strain gauge, force sensor, and displacement sensor via a hard-wired trigger signal, triggering the sensors to embed a high-precision timestamp T when collecting data. sThe DIC sensor receives a timestamped synchronization command sent by the main controller via the communication bus, ensuring that all sensor data is embedded with a high-precision timestamp T of a unified reference. s .

[0050] For clock drift compensation steps, please refer to the appendix. Figure 4 As shown, this step includes: the main controller collects the difference between the internal clock time t of each sensor and the reference time T0 every 100ms, and constructs a clock drift model using the Kalman filter algorithm. (a is the drift rate, b is the initial offset), real-time correction of the timestamps of each sensor data, and the corrected timestamps. To ensure consistency of all sensor data over time, the time synchronization deviation should be ≤5μs. (See attached document.) Figure 5 As shown, the sampling frequency is adaptively adjusted, including: for the sampling frequency differences of different sensors, the main controller constructs a sampling period mapping table: the strain gauge 100Hz and the force / displacement sensor 100Hz adopt a downsampling strategy, and the data is extracted according to the sampling period of 30Hz for the DIC sensor; the tilt sensor 50Hz fills the data gap through linear interpolation, so that the data of all sensors are evenly distributed on the time axis, and the data density is uniformly 30 frames / second.

[0051] The synchronous acquisition and data verification steps for spatiotemporal fusion include: the main controller, based on the aligned spatiotemporal coordinates and synchronized timestamps, realizes synchronous acquisition, fusion buffering, and quality verification of heterogeneous sensor data. This synchronous acquisition and data verification includes: the main controller, based on the corrected timestamps... The system generates synchronous acquisition commands to control each sensor to complete data acquisition within a ±5 microsecond time window. Specifically, this includes: the DIC sensor capturing deformation images of sub-components and recording three-dimensional displacement field data; strain gauges acquiring axial strain values ​​of the material and converting them into digital signals via transmitters; tilt sensors recording the torsional angles of each cross-section; and force / displacement sensors synchronously acquiring load and displacement data. The main controller performs multi-dimensional verification of the acquired data, including: time consistency verification, calculating the time deviation of data from each sensor at the same timestamp; marking data as abnormal if the deviation exceeds 10 microseconds; spatial consistency verification, verifying the deformation consistency at the same location based on the three-dimensional displacement data from the DIC sensor and the strain data from the strain gauges; marking data as abnormal if the spatial deviation exceeds 0.5mm; and abnormal data processing, using spline interpolation to repair marked abnormal data. If the accuracy requirements are still not met after repair, a re-acquisition command is triggered to ensure the integrity of the dataset.

[0052] The online dynamic calibration and parameter update steps include: During the test, the main controller monitors the spatiotemporal consistency index of 100 consecutive frames of data in real time; if the average time deviation is greater than 20µs or the average spatial deviation is greater than 1mm, the online dynamic calibration process is automatically initiated. The dynamic feature matching optimization process is repeated, and the dynamic transformation matrix T is recalculated. b The clock drift model Δt(t) is used to update the calibration parameters in the main controller and store them in non-volatile memory. At the same time, historical parameter versions are retained. If the new parameters cause a decrease in data quality, the system will automatically backtrack to the previous valid version to ensure the stability and accuracy of data acquisition during the test.

[0053] This application's embodiments employ a combination of static target calibration and dynamic feature matching to achieve sub-centimeter-level alignment of spatial coordinates. It achieves microsecond-level time synchronization by combining reference time anchoring with Kalman filter drift compensation. Furthermore, by combining multi-dimensional data verification and online calibration mechanisms, it improves the spatiotemporal consistency of heterogeneous sensor data and the robustness of the system, solving the problems of spatial coordinate misalignment, low time synchronization accuracy, poor adaptability to dynamic scenarios, and complex and low-robust calibration processes of heterogeneous sensors.

[0054] See attached document Figure 6 The figure shows a system for constructing spatiotemporal consistency of heterogeneous sensors according to an embodiment of this application, including: a clock initialization module 610, a static spatial calibration module 620, a dynamic spatial optimization module 630, a time synchronization module 640, and an acquisition and calibration module 650.

[0055] The clock initialization module 610 is used to identify the type of each sensor in the sensor array, configure the inherent parameters of each sensor, perform time synchronization initialization, and establish a unified global reference time.

[0056] The static spatial calibration module 620 is used to calculate the initial transformation relationship from each sensor coordinate system to the unified carrier coordinate system based on a static target with known spatial coordinates.

[0057] The dynamic space optimization module 630 is used to optimize the initial transformation relationship by matching the dynamic features in the data of each sensor during the operation of the sensor array, so as to obtain a dynamic transformation relationship for real-time spatial coordinate alignment.

[0058] The time synchronization module 640 is used to configure timestamps for the data acquisition process of each sensor based on the global reference time, and to correct the timestamps by modeling and compensating for the clock drift of each sensor, for time synchronization between sensors.

[0059] The acquisition and calibration module 650 is used to control each sensor to synchronously acquire data based on the dynamic transformation relationship and the corrected timestamp, convert the acquired data to the unified carrier coordinate system and perform data fusion, and perform spatiotemporal consistency verification on the fused data; if the degradation index in the spatiotemporal consistency verification result exceeds the set threshold, the operation of the dynamic spatial optimization module and the time synchronization module is triggered and executed for online updating of calibration parameters; otherwise, the synchronous data acquisition of each sensor continues to be controlled based on the current calibration parameters.

[0060] This embodiment of the heterogeneous sensor spatiotemporal consistency construction system, by integrating hardware modules and executing the method, modularizes and automates functions such as initialization, static calibration, dynamic optimization, high-precision synchronization, and closed-loop calibration of heterogeneous sensors. It realizes a closed loop of perception, alignment, verification, and calibration throughout the entire process from initial deployment to long-term operation, thereby solidifying the ability of sub-centimeter-level spatial alignment and microsecond-level time synchronization at the hardware level, and possessing adaptive compensation functions for environmental changes and equipment aging. Ultimately, it significantly improves the spatiotemporal consistency of multi-source sensor data fusion, the robustness of long-term system operation, and the automation and reliability of the entire acquisition process.

[0061] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of this application, and this application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this application, and these modifications and improvements are also considered to represent the scope of protection of this application.

Claims

1. A method for constructing spatiotemporal consistency of heterogeneous sensors, characterized in that, include: S1: Identify the type of each sensor in the sensor array, configure the inherent parameters of each sensor, perform time synchronization initialization, and establish a unified global reference time; S2: Based on a static target with known spatial coordinates, calculate the initial transformation relationship from each sensor coordinate system to the unified carrier coordinate system; S3: During the operation of the sensor array, the initial transformation relationship is optimized by matching the dynamic features in the data of each sensor to obtain a dynamic transformation relationship for real-time spatial coordinate alignment; S4: Configure timestamps for the data acquisition process of each sensor based on the global reference time, and correct the timestamps by modeling and compensating for the drift of the clock of each sensor, for time synchronization between sensors; S5: Based on the dynamic conversion relationship and the corrected timestamp, control each sensor to perform synchronous data acquisition, convert the acquired data to the unified carrier coordinate system and perform data fusion, and perform spatiotemporal consistency verification on the data after data fusion; if the degradation index in the spatiotemporal consistency verification result exceeds the set threshold, trigger and execute steps S3 to S4 for online updating of calibration parameters; otherwise, continue to control each sensor to perform synchronous data acquisition based on the current calibration parameters.

2. The method according to claim 1, characterized in that, The calculation of the initial transformation relationship from each sensor coordinate system to the unified carrier coordinate system for the static target based on known spatial coordinates includes: The system controls each sensor to synchronously acquire data from targets arranged within its field of view. Each target has multiple feature points with known spatial coordinates. The spatial coordinates of the feature points in their respective sensor coordinate systems are extracted from the data collected by each sensor. Using the spatial coordinates of the feature points as a reference, the initial transformation matrix of each sensor coordinate system relative to the unified carrier coordinate system is calculated through a coordinate transformation algorithm.

3. The method according to claim 2, characterized in that, During the operation of the sensor array-based system, the initial transformation relationship is optimized by matching dynamic features in the data from each sensor to obtain a dynamic transformation relationship for real-time spatial coordinate alignment, including: During the operation of the sensor array, dynamic features that satisfy interrelationships are extracted from the real-time data streams of each sensor. Based on the correspondence of the dynamic features in different sensor data, the initial transformation matrix is ​​iteratively optimized using a graph optimization algorithm to generate the dynamic transformation relationship that is updated in real time.

4. The method according to claim 1, characterized in that, The process of configuring timestamps for the data acquisition process of each sensor based on the global reference time, and correcting the timestamps by modeling and compensating for the clock drift of each sensor, for time synchronization between sensors, includes: The global reference time is encapsulated into a synchronization signal and sent to each sensor, so that each sensor can embed the original timestamp associated with the synchronization signal into the data it collects; The local clock of each sensor is periodically compared with the global reference time to obtain a clock difference sequence; Based on the clock difference sequence, a clock drift model for each sensor is established using a filtering estimation algorithm; The original timestamp is compensated in real time by the clock drift model to obtain a globally synchronized corrected timestamp for time synchronization between sensors.

5. The method according to claim 4, characterized in that, The method further includes: Set the target acquisition frequency; Downsampling is performed on sensor data streams with acquisition frequencies higher than the target acquisition frequency, and interpolation is performed on sensor data streams with acquisition frequencies lower than the target acquisition frequency, in order to confirm the uniform alignment of all sensor data in the time dimension.

6. The method according to claim 1, characterized in that, Perform spatiotemporal consistency verification on the fused data, including: Determine whether the time deviation of each sensor data under the same time label exceeds a first time threshold; and based on the dynamic characteristics, determine whether the spatial position deviation of the same target in different sensor data exceeds a first spatial threshold; Data whose time deviation exceeds a first time threshold and / or whose spatial location deviation exceeds a first spatial threshold are marked as abnormal data; The abnormal data can be repaired by interpolation or the data at the corresponding time point can be re-acquired.

7. The method according to claim 1, characterized in that, The degradation index includes the mean time deviation and the mean space deviation. When the degradation index in the result of the spatiotemporal consistency verification exceeds a set threshold, steps S3 to S4 are triggered and executed, including: If the average time deviation of the collected data exceeds the second time threshold and / or the average spatial deviation of the collected data exceeds the second spatial threshold within multiple consecutive acquisition cycles, steps S3 to S4 are triggered and executed; if not, the current calibration parameters are maintained.

8. The method according to claim 1, characterized in that, The method further includes: Save updated calibration parameters and historical parameter versions; If data quality deteriorates after adopting new parameters, the system will automatically revert to the previous valid parameter version.

9. The method according to claim 4, characterized in that, The filtering estimation algorithm is the Kalman filtering algorithm, and the clock drift model is a time-based linear function model.

10. A system for constructing spatiotemporal consistency of heterogeneous sensors, characterized in that, include: The clock initialization module is used to identify the type of each sensor in the sensor array, configure the inherent parameters of each sensor, perform time synchronization initialization, and establish a unified global reference time. The static spatial calibration module is used to calculate the initial transformation relationship from each sensor coordinate system to the unified carrier coordinate system based on a static target with known spatial coordinates. The dynamic space optimization module is used to optimize the initial transformation relationship by matching the dynamic features in the data of each sensor during the operation of the sensor array, so as to obtain a dynamic transformation relationship for real-time spatial coordinate alignment. The time synchronization module is used to configure timestamps for the data acquisition process of each sensor based on the global reference time, and to correct the timestamps by modeling and compensating for the drift of the clocks of each sensor, for time synchronization between sensors. The acquisition and calibration module is used to control each sensor to synchronously acquire data based on the dynamic transformation relationship and the corrected timestamp, convert the acquired data to the unified carrier coordinate system and perform data fusion, and perform spatiotemporal consistency verification on the fused data; if the degradation index in the spatiotemporal consistency verification result exceeds a set threshold, the operation of the dynamic spatial optimization module and the time synchronization module is triggered and executed for online updating of calibration parameters; otherwise, the synchronous data acquisition of each sensor continues to be controlled based on the current calibration parameters.

Images