Multi-modal sensor time synchronization method for road monitoring

Abstract

The invention provides a multi-modal sensor time synchronization method for road monitoring, which comprises the following steps: acquiring sensor data of target road detection, and determining communication time delay; the compensation value is optimized in real time according to the actual network condition and the environment change, and an optimized data set is obtained; determining a network topology structure based on the complete optimization data set; according to the optimized network topology structure, determining a time synchronization signal sequence of each node relative to the time delay information in the network; time synchronization of all sensors for target road detection is optimized. According to the invention, by introducing strategies of dynamic environment modeling and adaptive network topology optimization, the defects of the traditional time synchronization technology are effectively overcome, and high-precision multi-modal data time synchronization can be realized in a complex dynamic environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of road monitoring, and particularly relates to a multi-modal sensor time synchronization method for road monitoring. Background Art

[0002] With the development of intelligent transportation systems (ITS) and autonomous driving technologies, the demand for the fusion of multi-modal sensor data in fields such as lane monitoring, road condition analysis, and traffic flow management is increasing day by day. Against this background, more and more sensors are being applied to road monitoring and analysis. Common sensors include vision cameras, lidar, millimeter-wave radars, infrared sensors, etc. These sensors have different functions and advantages respectively. Vision cameras can provide rich environmental details, lidar can accurately measure distances and generate high-precision three-dimensional point cloud data, millimeter-wave radars have strong robustness in bad weather, and infrared sensors perform well in night or low visibility environments. Therefore, the data fusion of multiple sensors helps to comprehensively and accurately monitor road conditions, identify traffic events, and analyze traffic flow.

[0003] However, although multi-modal sensors provide rich monitoring information, due to problems such as differences in the working principles between sensors, different data sampling frequencies, and clock drift, it is difficult to accurately align the data of these sensors in time, thereby affecting the effect of data fusion. Especially in complex road environments, the problem of data time synchronization of sensors becomes particularly prominent. Taking vision cameras and radars as an example, the sampling frequency of vision cameras is usually relatively high, dozens to hundreds of frames of images can be collected per second, while the sampling frequency of radars is usually relatively low, perhaps only a few frames per second or less. Therefore, how to effectively align the data of different modalities while ensuring high-precision data synchronization has become an urgent technical problem to be solved.

[0004] Existing solutions mainly focus on hardware clock synchronization and delay compensation. For example, the IEEE1588 Precision Time Protocol (PTP) can synchronize the clocks of each sensor to the sub-microsecond level through a hardware-level time synchronization mechanism. However, this method usually requires each sensor to have hardware devices that support this protocol and is affected by factors such as the physical distance and transmission delay between sensors, making it difficult to work effectively in a dynamically changing environment. In addition, existing delay compensation methods are often static and do not consider dynamically changing road environment factors such as traffic flow and weather changes, which makes these methods unable to adapt to the changes in delays in scenarios such as highways and complex road conditions, resulting in a reduction in synchronization accuracy.

[0005] In addition to these time delay synchronization methods, some studies have proposed methods based on data correlation or machine learning between sensors to achieve time alignment of multi-modal data. However, these methods usually rely on complex models and computations, require a large amount of training data to optimize the models, and due to the frequent overlap or errors in the timing of multi-modal data, they are often difficult to adapt to various changing environments. Especially in non-ideal states, the accumulation of errors may affect the accuracy and usability of the data.

[0006] Therefore, the existing technologies face the following core problems in the field of multi-modal sensor time synchronization:

[0007] Time delay and clock drift: The problems of time delay and clock drift of different sensors are difficult to solve through hardware-level synchronization, resulting in low time synchronization accuracy. Especially in a dynamically changing environment, efficient time delay compensation cannot be achieved.

[0008] Neglect of environmental factors: Most of the existing synchronization methods neglect the influence of external environments (such as traffic flow, weather changes, etc.) on time delay, resulting in the inability to cope with complex and dynamic road surface environments in practical applications.

[0009] Balance between synchronization efficiency and accuracy: Existing methods are difficult to find a good balance between real-time performance and synchronization accuracy. Especially in scenarios with a large number of sensors and limited bandwidth, the requirements for high-precision, multi-modal data synchronization may not be met. Summary of the Invention

[0010] The object of the present invention is to propose a multi-modal sensor time synchronization method for road monitoring. By introducing the strategies of dynamic environment modeling and adaptive network topology optimization, the disadvantages of traditional time synchronization technologies are effectively overcome, and high-precision multi-modal data time synchronization can be achieved in complex dynamic environments.

[0011] To achieve the above object, the present invention provides a multi-modal sensor time synchronization method for road monitoring, including:

[0012] Obtain sensor data for target road detection, construct a dynamic model of environmental changes based on the sensor data to predict environmental changes, predict the communication time delay caused by environmental changes according to the predicted environmental changes, generate the predicted time delay compensation, and correct the communication time delay according to the predicted time delay compensation to obtain the corrected communication time delay;

[0013] In obtaining the corrected communication time delay, design an adaptive adjustment strategy based on the predicted time delay compensation, optimize the compensation value in real time according to the actual network situation and environmental changes, and perform final synchronization and correction on all sensor data based on the optimized compensation value to obtain a complete optimized data set;

[0014] Based on the complete optimized dataset, determine the network topology structure to adapt to the dynamically changing network load, communication delay, and connection quality between nodes, and generate an optimized network topology structure;

[0015] According to the optimized network topology structure, determine the time synchronization signal sequence of the relative delay information of each node in the network, and generate an optimal synchronization signal. At the same time, by real-time monitoring the changes in the network state, dynamically adjust the time synchronization signal to generate a high-precision time synchronization signal, and adjust each data point in the complete optimized dataset according to the high-precision time synchronization signal, so that each data in the complete optimized dataset meets the global time synchronization requirements, and complete the time synchronization of all sensors for target road detection;

[0016] And / or, it further includes:

[0017] Based on the high-precision time synchronization signal, align and fuse multi-modal data from different sensors or data sources. At the same time, design an optimization algorithm based on temporal consistency constraints to fine-tune the temporal errors in the fused data to ensure that the timestamps of all modal data are consistent throughout the time range, and optimize the time synchronization of all sensors for target road detection.

[0018] In some embodiments, the obtaining of the sensor data for target road detection includes:

[0019] The sensor data is multi-modal data collected by a multi-modal device, and each data point d of the sensor data i contains observations of multiple sensors;

[0020] The building of the dynamic model of environmental change based on the sensor data to predict environmental change includes:

[0021] Use a long short-term memory network to model the temporal dependence relationship of the environment according to the sensor data, and predict future environmental changes. Through the training of the sensor data, obtain a dynamic model of environmental change, and the dynamic model of environmental change can predict environmental changes at a certain future moment or within a time window.

[0022] In some embodiments, based on predicting environmental changes at a certain future moment or within a time window, combined with the actual environmental changes and the deviation between the predicted environmental changes and the actual environmental changes, calculate the predicted delay compensation value through a non-linear mapping function, and perform delay correction on each data point according to the predicted delay compensation value to obtain a corrected dataset; wherein, the non-linear mapping function captures the complex relationship between the delay and environmental changes through a regression model or an adaptive learning method.

[0023] In some embodiments, the design of the adaptive adjustment strategy based on the predicted delay compensation includes:

[0024] For each time window t, the delay compensation value is adjusted by optimizing the following objective function :

[0025]

[0026] where d′ i is the corrected data, is to optimize the compensation value in real time according to the actual network conditions and environmental changes is the optimized data, is the optimized delay compensation value, λ 1 and λ 2 are regularization parameters used to balance the data compensation accuracy and the smoothness of the delay variation, represents the change rate of the delay compensation value to prevent excessive fluctuations in the delay compensation;

[0027] Then, the optimized delay compensation value is used to optimize the sensor data to obtain a complete optimized data set.

[0028] In some embodiments, after finally synchronizing and correcting all sensor data based on the optimized compensation value, by sampling the real-time information of the current network and combining with the dynamic model of environmental changes, the delay compensation is dynamically adjusted according to the real-time network state, and the optimal delay compensation value is calculated in real time.

[0029] In some embodiments, the optimization objective function of the network topology structure is expressed as:

[0030]

[0031] where is the objective function representing the cost after network topology optimization; Cost(i, j, T * ) represents the communication cost between node i and node j; Connectivity(i, j, T * ) represents the connection strength between node i and node j, which is 1 if there is a connection between them, otherwise 0; Load k represents the load of node k, reflecting the computing or data processing pressure of this node; Latency l represents the communication delay of node l; Distance i represents the distance between node i and its optimal communication path; λ 1 , λ 2 , λ 3 are regularization coefficients that control the trade-off between different optimization objectives;

[0032] For each moment \(t\), the network topology is adjusted by the following formula

[0033]

[0034] where is the optimized topology structure at moment \(t\); \(T\) t-1 is the topology structure of the previous moment; \(\eta\) is the adjustment step size, controlling the speed of topology adjustment; \(S\) net (\(t\)) is the current network state information, \(S\) target (\(t\)) is the target network state, given by the environmental change dynamic model; \(\Delta T\) t-1 is the topology change amount of the previous moment, used to capture the historical change trend, and \(\gamma\) is the trend correction coefficient.

[0035] In some embodiments, after the network topology is optimized, the complete optimized data set needs to adjust the path according to the new network topology structure \(T\) * to ensure that data can be transmitted with the lowest latency and the highest bandwidth, including:

[0036] For each data point its final transmission path can be expressed as:

[0037]

[0038] where is the optimized data point; represents the data point selects the best transmission path according to the optimized topology \(T\) * ; is the set of all possible paths under the topology \(T\) * ; \(Cost(j, T\) * ) is the communication cost of node \(j\) on the path, usually including latency and bandwidth, etc.;

[0039] In the long-term operation, a long-term adaptive optimization method based on deep reinforcement learning is introduced to guide the continuous optimization of the network topology through the deep reinforcement learning model:

[0040]

[0041] where \(R(t)\) is the total reward at moment \(t\); is the reward of node \(i\) under the topology ;

[0042] In some embodiments, in generating a high-precision time synchronization signal, a synchronization signal optimization algorithm based on weighted least squares is proposed to minimize the time delay difference between nodes and generate an optimal synchronization signal; where, the synchronization signal optimization algorithm is expressed as:

[0043]

[0044] Among them, is the optimization objective function for time synchronization; w ij is the weight between node i and node j, which is usually related to the communication load, distance, and topological relationship of the nodes; t i and t j are the synchronization times of node i and node j; ΔT ij is the time delay difference between node i and node j, reflecting the network delay under the current topology;

[0045] The dynamic adjustment of the time synchronization signal by real-time monitoring of the changes in the network state includes:

[0046] At each moment t, the synchronization signal update formula is:

[0047] T sync，t = T sync，t-1 + η·(S net，t - S net，t-1 )

[0048] Among them, T sync，t is the synchronization signal at moment t; T sync，t-1 is the synchronization signal at the previous moment; η is the adjustment step size, controlling the update speed of the synchronization signal; S net，t and S net，t-1 are the network state information at the current moment and the previous moment respectively;

[0049] After the time synchronization signal is generated, the optimized time synchronization signal T sync will be used to adjust each data point in the data set to meet the global time synchronization requirements, including:

[0050] The timestamp of the data point will be corrected according to the generated synchronization signal to ensure that the data of all nodes can be processed according to a unified time reference. The complete optimized data set adjustment formula is:

[0051]

[0052] Among them, is the data set after time synchronization; is the optimized data point; T sync (t i ) is the time synchronization signal of the i-th data point, correcting the timestamp of each data point.

[0053] In some embodiments, the time alignment and fusion of multimodal data from different sensors or data sources based on the high-precision time synchronization signal includes:

[0054] Align all data to a unified time reference through an interpolation method. The time synchronization signal provides time correction for each node. Using the time synchronization signal, the data of all modalities are converted to the same time scale through an interpolation algorithm;

[0055] Fuse the aligned data of each modality, and adaptively adjust the fusion weights based on the quality and signal-to-noise ratio of each modality. The data of different modalities may have different qualities and reliabilities. The fusion algorithm needs to adjust the weights according to the quality of the data to achieve the best fusion effect; the objective function of adaptive weighted fusion is as follows:

[0056]

[0057] where, D fused (t) is the fused data at time t, containing information of all modalities; D aligned，i (t) is the data of the i-th modality at time t, which has been time-aligned; w i (t) is the adaptive fusion weight of the i-th modality at time point t, and the weight w i (t) is calculated as:

[0058]

[0059] where, q i (t) is the quality evaluation function of the i-th modality at time t.

[0060] In some embodiments, the optimization objective of the optimization algorithm based on the timing consistency constraint is to minimize the time deviation of all modality data, and the specific formula is:

[0061]

[0062] where, is the timing consistency optimization objective function; D aligned，i (t) is the aligned data of the i-th modality at time t; is the ideal value of the fused data at time t.

[0063] The beneficial technical effects of the present invention are at least as follows:

[0064] (1) The present invention combines, for the first time, the dynamic data of the lane environment (such as traffic flow, weather conditions, etc.) with the estimation of sensor delay. By collecting environmental data in real time and using algorithms such as the Extended Kalman Filter (EKF) to dynamically estimate the delay change, and combining with a dynamic environment model for delay compensation. This innovation makes the delay compensation no longer static but adaptive, capable of automatically adjusting the delay compensation strategy according to environmental changes, effectively improving the synchronization accuracy.

[0065] (2) The present invention realizes the efficient transmission of synchronization signals among multiple sensors by constructing an adaptive network topology optimization mechanism based on sensor distribution and network bandwidth. In the case of limited bandwidth and wide distribution of sensors, optimizing the network topology can ensure the rapid transmission of synchronization signals, reduce signal loss or delay, and ensure the stability and efficiency of the synchronization process. This innovation further improves the reliability and accuracy of time synchronization by considering network factors during the synchronization process.

[0066] (3) Based on high-precision time synchronization, the present invention combines a multi-modal sensor data fusion method. Through precise timestamp correction and multi-modal data time alignment technology, it ensures that the data of different sensors can be accurately aligned on the time axis, thereby improving the accuracy and consistency of data fusion. The fusion of different modal data not only improves the perception accuracy but also better supports road monitoring and analysis tasks in complex environments.

[0067] (4) By introducing innovative technologies such as dynamic environment modeling, adaptive delay compensation, and network topology optimization, the present invention effectively overcomes the deficiencies in the existing technology, such as insufficient delay estimation accuracy, poor environmental adaptability, and the contradiction between synchronization efficiency and accuracy. It provides a multi-modal sensor time synchronization method applicable to complex road environments and capable of achieving high-precision time synchronization, significantly improving the overall performance of the road monitoring system. Description of the Drawings

[0068] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the following drawings without creative efforts.

[0069] Figure 1 It is a flowchart of a multi-modal sensor time synchronization method for road monitoring according to the present invention. Detailed Embodiments

[0070] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where like or similar reference numerals denote like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0071] In one or more embodiments, as Figure 1 shown, a multi-modal sensor time synchronization method for road monitoring is disclosed, including:

[0072] S1. Obtain sensor data for target road detection, construct an environmental change dynamic model based on the sensor data to predict environmental changes, predict the communication delay caused by environmental changes according to the predicted environmental changes, generate the predicted delay compensation, and correct the communication delay according to the predicted delay compensation to obtain the corrected communication delay.

[0073] In this step, the present invention will perform dynamic modeling on the environment and combine the environmental changes to predict the communication delay, further solving the problems of data synchronization and delay compensation. The core objective of this step is to predict the communication delay caused by environmental factors and optimize the data synchronization and delay calibration strategies in the subsequent steps according to the prediction results.

[0074] Further, the sensor data D = {d 1 , d 2 ,..., d n} is already multi-modal data collected by multi-modal devices such as camera arrays and lidar. Each data point d i contains observations of multiple sensors. Assuming that each d i is a vector, d i = [x i1 , x i2 ,..., x im , where m is the number of sensors, and each x ij represents the observation value of the j-th sensor in the time window i.

[0075] Further, the present invention uses a long short-term memory network (LSTM) to model the temporal dependence of the environment and predict future environmental changes. LSTM can capture the long-term dependencies in complex time series data and is suitable for describing the dynamic changes of environmental factors over time. By training D, an environmental model M env is obtained, and this model can predict environmental changes at a certain future moment or within a time window.

[0076] Specifically, the present invention trains D, minimizes the following loss function, and optimizes the parameters θ of the LSTM:

[0077]

[0078] Among them, is the environmental prediction value of the model M env at the i-th moment, and d i is the actual observed data, and λ 1 is the regularization parameter, which controls the smoothness between prediction values. The second regularization term helps to maintain the smoothness of environmental changes and avoid overfitting unstable fluctuations. After the model is trained, the present invention obtains the environmental prediction result

[0079] Furthermore, based on the output env of the environmental model M the present invention will further predict the communication delay δ t caused by environmental changes, and correct the data based on this. The delay compensation model makes the data synchronization between different sensors more accurate by adaptively adjusting the communication delay.

[0080] The present invention assumes that there is a certain relationship between the delay and environmental changes. The delay value δ t is affected not only by environmental factors, but also by sensor states (such as load, distance, etc.) and network transmission conditions. In order to model the delay, the present invention proposes a delay compensation model, which combines the deviation between the environmental prediction value and the sensor data, and calculates the delay δ t using the following formula:

[0081]

[0082] Among them, represents the deviation between the actual data and the predicted data at the i-th moment, and f is a non-linear mapping function, which is used to capture the complex relationship between the delay and environmental changes through a regression model or an adaptive learning method. δ t is the delay compensation value.

[0083] In addition, the present invention can also introduce a regularization term to avoid system instability caused by excessive adjustment:

[0084]

[0085] Among them, δ t，target is the ideal delay value, δ t，prev is the delay value at the previous moment, and λ 2 is the adjustment coefficient, which ensures the smoothness of the delay change and avoids excessive fluctuations.

[0086] Furthermore, combining the predicted delay compensation δ t , for each data point di Perform time delay correction to obtain the corrected dataset D′ = {d′ 1 , d′ 2 ,..., d′ n}. Specifically, by performing translational adjustment on the data at each moment, ensure that the sensor data can be synchronized and the errors caused by time delay are eliminated.

[0087] For the data d i of each sensor, its correction formula is:

[0088] d′ i = d i - δ t

[0089] where d′ i is the corrected data, and δ t is the compensation value predicted according to the time delay model.

[0090] S2. In obtaining the corrected communication time delay, design an adaptive adjustment strategy based on the predicted time delay compensation, optimize the compensation value in real time according to the actual network situation and environmental changes, and perform final synchronization and correction on all sensor data based on the optimized compensation value to obtain a complete optimized dataset.

[0091] The goal of this step is to further achieve adaptive time delay compensation based on the environmental dynamic modeling and time delay prediction results obtained in the previous step, and perform fine adjustment on the time delay through an optimization algorithm, so as to reduce the errors caused by communication time delay and improve the accuracy of data synchronization. The core of this step lies in dynamically adjusting the compensation value to ensure that the time delay compensation is consistent with the real-time changes of the network environment and optimize the final data transmission and fusion process.

[0092] Furthermore, due to the time-varying nature of environmental changes, the predicted time delay compensation δ t may not be statically effective in actual applications, so it needs to be dynamically adjusted. The present invention realizes more accurate time delay compensation by introducing an adaptive adjustment strategy based on the current network state and environmental model, and optimizing the compensation value δ t in real time according to the actual network situation and environmental changes.

[0093] Specifically, for each time window t, adjust the time delay compensation value by optimizing the following objective function:

[0094]

[0095] where d′ i is the data after time delay compensation in the previous step, is the optimized data, is the optimized time delay compensation value, and λ1 and λ 2 are regularization parameters used to balance the data compensation accuracy and the smoothness of the time delay variation. represents the change rate of the time delay compensation value to prevent excessive fluctuations in the time delay compensation.

[0096] This optimization problem is solved by the gradient descent method or the Bayesian optimization algorithm to minimize the data error while ensuring the smoothness of the time delay compensation.

[0097] Furthermore, to further improve the data synchronization accuracy, the present invention performs a fine adjustment of the time delay value on the compensated data set. The specific method is to use the optimized time delay compensation value to correct the data, obtaining the optimized data set D*.

[0098] For the sensor data at each moment, the present invention corrects it in the following way:

[0099]

[0100] where is the optimized data, is the finally optimized time delay compensation value. By optimizing the compensated time delay value the present invention not only eliminates the error caused by static time delay compensation, but also further fine-tunes the data to improve the data synchronization accuracy. This method ensures the self-adaptability and accuracy of the time delay compensation, not only correcting the communication delay between sensors, but also dynamically considering the environmental changes.

[0101] Furthermore, in practical applications, the network environment is constantly changing, and the time delay compensation should also be dynamically adjusted according to the real-time network state. The present invention introduces a network feedback mechanism to enable the time delay compensation strategy to be feedback-optimized according to the real-time network conditions.

[0102] Specifically, the present invention can sample the information such as the transmission rate, bandwidth, and delay of the current network, and combine it with the environmental change model M env , and calculate the optimal time delay compensation value in real time The present invention sets the following feedback function:

[0103]

[0104] where δ t is the preliminarily predicted time delay compensation value, η is the adjustment coefficient to control the speed of the feedback adjustment, NetworkStatus represents the current network state (such as bandwidth, delay, etc.), and TargetStatus represents the target network state based on the expected optimal network conditions.

[0105] Furthermore, combined with the optimized delay compensation value Perform final synchronization and correction on all sensor data to obtain a complete optimized dataset D * . After precise delay compensation, this dataset can be used for subsequent data fusion and analysis. Adaptive delay compensation and optimization steps play a crucial role throughout the patent. By precisely adjusting the delay, it optimizes the data synchronization process and ensures the delay compensation effect of multi-sensor data under environmental and network conditions. This step flexibly responds to delay changes in a dynamic network environment, provides precise data synchronization, and greatly improves the efficiency and accuracy of data processing.

[0106] S3. Based on the complete optimized dataset, determine the network topology to adapt to dynamic network load, communication delay, and connection quality between nodes, and generate an optimized network topology.

[0107] The goal of this step is based on the output of the previous step (the optimized delay compensation dataset D * ). By designing an adaptive network topology optimization algorithm, adjust the network topology to adapt to dynamic network load, communication delay, and connection quality between nodes. The optimized network topology should achieve optimal network performance globally and be able to adapt to environmental changes, enhancing the system's adaptability and data transmission efficiency.

[0108] Furthermore, for the dataset D after delay compensation * , the optimization of the network topology needs to comprehensively consider multiple factors such as the communication cost between nodes, data transmission delay, and network load. To dynamically adjust the topology in a complex environment, the present invention proposes a network topology optimization objective function for delay compensation and load adaptability, which not only considers the communication cost, delay, and load between nodes but also introduces important factors of network adaptability.

[0109] The optimization objective function can be expressed as:

[0110]

[0111] Where is the objective function, representing the cost after network topology optimization; Cost(i, j, T * ) represents the communication cost between node i and node j, usually including factors such as bandwidth, node load, and transmission delay; Connectivity(i, j, T * ) represents the connection strength between node i and node j, which is 1 if there is a connection between them, otherwise 0; Load k represents the load of node k, reflecting the computing or data processing pressure of this node; Latencyl Denotes the communication delay of node l; Distance i Denotes the distance between node i and its optimal communication path; λ 1 , λ 2 , λ 3 , where λ is the regularization coefficient that controls the trade-off between different optimization objectives. The objective function incorporates three important optimization factors: communication cost, node load, and the distance of the network topology. This optimization not only focuses on a single factor but achieves system-level balance through multi-dimensional consideration. Especially in a network environment with real-time changes, it can improve the accuracy and adaptability of topology optimization.

[0112] Furthermore, to adapt to the dynamic changes of the network state, the present invention designs an adaptive topology adjustment algorithm based on a real-time feedback mechanism. This algorithm can, at each time step t, adjust the topology structure according to the real-time state S net (t) of the network and the environmental change prediction model M env .

[0113] At each moment t, the present invention adjusts the network topology through the following formula

[0114]

[0115] where is the optimized topology structure at time t; T t-1 is the topology structure at the previous moment; η is the adjustment step size that controls the speed of topology adjustment; S net (t) is the current network state information, including node load, delay, etc.; S target (t) is the target network state given by the environmental prediction model M env ; ΔT t-1 is the topology change amount at the previous moment, used to capture the historical change trend, and γ is the trend correction coefficient. This algorithm enhances the self-adaptability of topology adjustment by combining the historical topology change trend and the real-time network state. In particular, the introduction of the ΔT t-1 term enables the topology adjustment to not only depend on the current network state but also take into account the historical evolution trend, thereby further improving the system's prediction and adaptation capabilities for future changes.

[0116] Furthermore, after the network topology optimization, the data transmission path and the connection relationship between nodes may change. Therefore, the data set D* needs to adjust the path according to the new network topology structure T* to ensure that data can be transmitted with the lowest latency and the highest bandwidth.

[0117] For each data point its final transmission path can be expressed as:

[0118]

[0119] Among them, is the optimized data point; represents the data point selected according to the optimized topology T * the best transmission path; is the topology T * all possible path sets under; Cost(j, T * ) is the communication cost of node j on the path, usually including delay and bandwidth, etc.

[0120] This method optimizes the data transmission path and combines the node connections after topology optimization to ensure that the delay and data packet loss can be minimized to the greatest extent during the data transmission process. Especially when the network load is high, a lower-cost transmission path can be dynamically selected.

[0121] Furthermore, in the long-term operation, the adaptive ability of network topology optimization is the key to ensuring the long-term efficient operation of the network. For this reason, the present invention introduces a long-term adaptive optimization method based on deep reinforcement learning to guide the continuous optimization of network topology through a reward mechanism. The present invention performs topology optimization through the following deep reinforcement learning model:

[0122]

[0123] Among them, R(t) is the total reward at time t; is the reward of node i under the topology , and the reward value is calculated according to factors such as the communication delay and bandwidth utilization of this node.

[0124] This method continuously optimizes the structure in the long-term evolution of network topology through deep reinforcement learning, enabling the network to continuously self-adjust to cope with uncertain factors such as load fluctuations and network topology changes, ensuring the stability and efficiency of the system.

[0125] The core of this step lies in adaptively adjusting the network topology structure, combining real-time network status and delay compensation data, and realizing the dynamic adaptation and long-term optimization of the network through comprehensive optimization of the objective function, historical topology change trend, and deep reinforcement learning.

[0126] S4. Based on the optimized network topology, determine the time synchronization signal sequence of the relative delay information of each node in the network, generate the optimal synchronization signal, and at the same time, by monitoring the changes in the network state in real time, dynamically adjust the time synchronization signal to generate a high-precision time synchronization signal, and adjust each data point in the complete optimized data set according to the high-precision time synchronization signal, so that each data in the complete optimized data set meets the global time synchronization requirements, and complete the time synchronization of all sensors for target road detection.

[0127] In this step, the goal of the present invention is to design a high-precision time synchronization signal generation algorithm to ensure that the time accuracy of data transmission reaches the microsecond level or even the sub-microsecond level. Through precise time synchronization, the delay deviation in network communication can be further reduced, and the overall performance of the system can be improved, especially playing a key role in real-time data transmission and dynamic scheduling.

[0128] Furthermore, the key to this step is to design a high-precision time synchronization framework based on the optimized network topology. By analyzing the communication delay and load conditions between nodes, the present invention utilizes the topology T * to construct a global time synchronization model. The goal of this model is to generate a high-precision time synchronization signal sequence based on the relative delay information of each node in the network.

[0129] Based on the optimization of the topology in the previous steps, the present invention introduces the following synchronization signal generation strategy:

[0130]

[0131] where, T sync is a high-precision time synchronization signal; Sync(·) is a synchronization function, which combines the network topology T * , the network state information S net and the optimized data set to generate the time synchronization signal of each node; where, S net includes factors such as the communication delay and bandwidth of each node, reflecting the dynamic changes of the network state; is the optimized data set, containing the data after delay compensation.

[0132] This method generates precise time synchronization signals globally by combining the network topology structure and real-time network state information, not only considering the communication delay between nodes, but also adapting to the changes in network load. The synchronization function Sync can adjust the time synchronization signal under different network states, so as to achieve high-precision delay control.

[0133] Furthermore, to further improve the accuracy of time synchronization, the present invention proposes a synchronization signal optimization algorithm based on weighted least squares (WLS). The goal of this algorithm is to generate an optimal synchronization signal by minimizing the time delay difference between nodes according to the optimized data set and the network topology T * . The objective function of the optimization algorithm is expressed as:

[0134]

[0135] where is the optimization objective function for time synchronization; w ij is the weight between node i and node j, which is usually related to the communication load, distance, and topological relationship of the nodes; t i and t j are the synchronization times of node i and node j; ΔT ij is the time delay difference between node i and node j, reflecting the network delay under the current topology; the meaning of this objective function is to minimize the time delay difference between nodes, thereby optimizing the time synchronization accuracy.

[0136] Furthermore, this method introduces the load and topological information between nodes through weighted least squares, ensuring that the optimization of the synchronization signal is more in line with the actual situation of the network state and avoiding the time delay error in traditional synchronization algorithms. In particular, the introduction of the weight w ij makes the synchronization signal not only consider the communication delay but also comprehensively consider the topological structure and load factors, enhancing the synchronization accuracy and robustness.

[0137] Furthermore, in practical applications, the state of the network is constantly changing. In particular, the load and time delay may change rapidly due to traffic fluctuations. To adapt to this dynamic environment, the present invention designs an adaptive synchronization signal update algorithm that dynamically adjusts the time synchronization signal by monitoring the changes in the network state in real time. At each moment t, the synchronization signal update formula is:

[0138] T sync，t = T sync，t-1 + η · (S net，t - S net，t-1 )

[0139] where T sync，t is the synchronization signal at moment t; T sync，t-1 is the synchronization signal at the previous moment; η is the adjustment step size, controlling the update speed of the synchronization signal; S net，t and S net，t-1 are the network state information at the current moment and the previous moment, respectively, reflecting the changes in network load, delay, etc.

[0140] This method dynamically adjusts the synchronization signal by real-time feedback of network state changes. The adjustment of the η step size makes the update of the synchronization signal smoother, avoiding drastic fluctuations in synchronization errors. In addition, the algorithm adapts more flexibly to network changes by comparing the differences between the current and past network states, improving the accuracy of time synchronization.

[0141] Further, after the time synchronization signal is generated, the optimized time synchronization signal T sync will be used to adjust the data set in each data point to meet the global time synchronization requirements. Specifically, the timestamps of the data points will be corrected according to the generated synchronization signal to ensure that the data of all nodes can be processed according to a unified time reference. The data set adjustment formula is:

[0142]

[0143] where is the data set after time synchronization; is the optimized data point; T sync (t i ) is the time synchronization signal of the i-th data point, correcting the timestamps of each data point. By correcting the data points with the time synchronization signal T sync , the data set can be processed according to a unified clock. This operation ensures that in a multi-node system, the data of all nodes can be integrated and analyzed under the same time reference, eliminating the deviation caused by different time sources.

[0144] The core of this step is to solve the problem of inconsistent node delays in the network by generating high-precision time synchronization signals, combining the data set optimized by network topology and network state information, ensuring data accuracy and consistency. The generation of time synchronization signals not only improves the overall performance of the system but also provides a reliable time reference for subsequent data fusion and real-time transmission, playing a key role especially in scenarios with extremely high synchronization accuracy requirements.

[0145] S5. Based on the high-precision time synchronization signal, align and fuse multi-modal data from different sensors or data sources, and at the same time design an optimization algorithm based on timing consistency constraints to fine-tune the timing errors in the fused data to ensure that the timestamps of all modal data are consistent throughout the time range, optimizing the time synchronization of all sensors for target road detection.

[0146] In this step, the objective of the present invention is based on the high-precision time synchronization signal T sync, multi-modal data from different sensors or data sources are time-aligned and fused. Multi-modal data usually includes images, audio, video, sensor data, etc. These data sources often have different sampling frequencies, timestamps, and data formats. Therefore, an efficient and robust time alignment and fusion method needs to be designed to ensure that they can provide unified and accurate information after being aligned on the time axis for subsequent data analysis and decision-making.

[0147] Furthermore, the time alignment of multi-modal data sources is the core of the entire fusion process. Data of different modalities may have different sampling frequencies and timestamps. Therefore, all data need to be aligned to a unified time reference through interpolation methods. The time synchronization signal T sync provides the time correction of each node. Using these signals, the present invention can convert the data of all modalities to the same time scale through an interpolation algorithm.

[0148] For each modality data D mi , interpolation is performed through the following time alignment formula:

[0149] D aligned，i (t) = Interp(D mi , T sync )

[0150] where D aligned，i (t) is the aligned data of modality i at time point t; Interp(·) represents the interpolation function, which can perform time alignment processing on data D sync according to the time synchronization signal T mi . The interpolation method selects a suitable interpolation algorithm according to the characteristics of different modalities (for example, frame interpolation may be used for video data, and linear interpolation may be used for sensor data).

[0151] This interpolation method utilizes the dual information from multi-modal data sources and synchronization signals, can efficiently align data from different frequencies and time scales, thus avoiding the possible accuracy loss problem in conventional interpolation methods and improving the accuracy of the fused data.

[0152] Furthermore, the aligned data of each modality need to be fused to generate a unified, multi-dimensional data structure. Here, the present invention proposes an adaptive weighted fusion method, which adaptively adjusts the fusion weights based on the quality and signal-to-noise ratio of each modality. Data of different modalities may have different qualities and reliabilities. The fusion algorithm needs to adjust the weights according to the quality of the data to achieve the best fusion effect.

[0153] The objective function of adaptive weighted fusion is as follows:

[0154]

[0155] Among them, D fused (t) is the fused data at time t, containing information of all modalities; D aligned，i (t) is the i-th modality data at time t, which has been time-aligned; w i (t) is the adaptive fusion weight of the i-th modality at time point t, and this weight is dynamically adjusted according to information such as the data quality and signal-to-noise ratio of modality i. The calculation method of the weight w i (t) is as follows:

[0156]

[0157] Among them, q i (t) is the quality evaluation function of the i-th modality at time t, which is usually calculated based on factors such as signal-to-noise ratio, data integrity, and sensor health status. The quality evaluation function q i (t) may be the following types of weighted functions, for example: Among them, σ i，t is the standard deviation of modality i at time t, representing the volatility of modality data; or this method improves the accuracy of data fusion by introducing an adaptive weighting mechanism to dynamically adjust the weight according to the quality of modality data. The data of different modalities may have uneven quality due to various factors (such as sensor failures, external interferences, etc.), and this weighting method can ensure that high-quality data has a greater impact on the final fusion result, and the impact of weak-quality data is suppressed, thereby improving the reliability and accuracy of the fused data.

[0158] Furthermore, there may still be small timing errors in the fused data, especially in the case of a large amount of data. To ensure the timing consistency of data of each modality, the present invention proposes an optimization algorithm based on timing consistency constraints, which fine-tunes the timing errors in the fused data to ensure that the timestamps of all modality data are consistent throughout the time range.

[0159] The optimization objective is to minimize the time deviation of all modality data, and the specific formula is:

[0160]

[0161] Among them, is the timing consistency optimization objective function; D aligned，i (t) is the aligned data of the i-th modality at time t; is the ideal value of the fused data at time t, which is the result after timing optimization.

[0162] The timing consistency optimization algorithm introduces global time constraints to finely adjust the timing errors in the fused data, ensuring a high degree of temporal consistency in the final fused result, avoiding error propagation that may be caused by timing inconsistencies, and is particularly suitable for application scenarios that require high-precision timing alignment. This step designs an innovative multimodal data alignment and fusion scheme, which can effectively align data from different modalities to a unified time reference, and on this basis, perform weighted fusion and timing consistency optimization.

[0163] In summary, the present invention proposes a multimodal sensor time synchronization method and device based on environmental dynamic modeling and adaptive network topology optimization. By introducing strategies of dynamic environmental modeling and adaptive network topology optimization, it effectively overcomes the shortcomings of traditional time synchronization technologies and can achieve high-precision multimodal data time synchronization in complex dynamic environments. The present invention effectively overcomes the problems of insufficient delay estimation accuracy, poor environmental adaptability, and the contradiction between synchronization efficiency and accuracy in the prior art by introducing innovative technologies such as dynamic environmental modeling, adaptive delay compensation, and network topology optimization, provides a multimodal sensor time synchronization method applicable to complex road environments and capable of achieving high-precision time synchronization, and significantly improves the overall performance of the road monitoring system.

[0164] The present invention can be a method, device, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for performing various aspects of the present invention.

[0165] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0166] Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

[0167] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0168] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read from, and write to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0169] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code. The computer-readable medium includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. The storage media may be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, the disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where the disk typically reproduces data magnetically, while the disc reproduces data optically with a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0170] The foregoing description of the disclosure has been provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal sensor time synchronization method for road monitoring, characterized in that: include: Acquire sensor data of target road detection, construct an environmental change dynamic model based on the sensor data to predict environmental changes, predict communication delays caused by environmental changes according to the predicted environmental changes, generate predicted delay compensation, and correct the communication delay according to the predicted delay compensation to obtain corrected communication delay; In obtaining the corrected communication delay, an adaptive adjustment strategy is designed based on the predicted delay compensation, and the compensation value is optimized in real time according to the actual network situation and environmental changes. All sensor data are finally synchronized and corrected based on the optimized compensation value to obtain a complete optimized data set. Based on the complete optimization data set, determine the network topology structure to adapt to the dynamically changing network load, communication delay and connection quality between nodes, and generate the optimized network topology structure; According to the optimized network topology, the time synchronization signal sequence of the relative delay information of each node in the network is determined, and the optimal synchronization signal is generated. At the same time, by real-time monitoring of changes in network status, the time synchronization signal is dynamically adjusted to generate a high-precision time synchronization signal. Each data point in the complete optimized data set is adjusted according to the high-precision time synchronization signal, so that each data point in the complete optimized data set meets the global time synchronization requirements, and the time synchronization of all sensors for target road detection is completed; and / or, also include: Based on high-precision time synchronization signals, multimodal data from different sensors or data sources are time-aligned and fused. At the same time, an optimization algorithm based on timing consistency constraints is designed to fine-tune the timing errors in the fused data to ensure that the timestamps of all modal data are consistent over the entire time range, thereby optimizing the time synchronization of all sensors for target road detection.

2. A multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that: The step of acquiring sensor data for target road detection includes: The sensor data is multimodal data acquired through a multimodal device, and each data point di of the sensor data contains observation values ​​of multiple sensors; The step of constructing a dynamic model of environmental changes based on the sensor data to predict environmental changes includes: According to the sensor data, a long short-term memory network is used to model the time dependency of the environment and predict future environmental changes. By training the sensor data, a dynamic model of environmental changes is obtained, which can predict environmental changes at a certain moment or time window in the future.

3. A multimodal sensor time synchronization method for road monitoring according to claim 2, characterized in that: Based on the prediction of environmental changes within a certain moment or time window in the future, combined with the actual environmental changes and the deviation between the predicted environmental changes and the actual environmental changes, a predicted delay compensation value is calculated through a nonlinear mapping function, and the delay of each data point is corrected according to the predicted delay compensation value to obtain a corrected data set; wherein the nonlinear mapping function captures the complex relationship between delay and environmental changes through a regression model or an adaptive learning method.

4. A multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that: The adaptive adjustment strategy based on the predicted delay compensation design includes: For each time window t, we optimize the following objective function To adjust the delay compensation value: Among them, d′i is the corrected data, To optimize the compensation value in real time according to the actual network conditions and environmental changes After optimization, is the optimized delay compensation value, λ1 and λ2 are regularization parameters used to balance the data compensation accuracy and the smoothness of delay change. Indicates the rate of change of the delay compensation value to prevent excessive fluctuations in the delay compensation; Reuse the optimized delay compensation value The sensor data is optimized to obtain a complete optimized data set.

5. A multimodal sensor time synchronization method for road monitoring according to claim 4, characterized in that: After all sensor data are finally synchronized and corrected based on the optimized compensation value, the delay compensation is dynamically adjusted according to the real-time network status by sampling the real-time information of the current network and combining the dynamic model of environmental changes, and the optimal delay compensation value is calculated in real time.

6. A multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that: The optimization objective function of the network topology is expressed as: in, is the objective function, which represents the cost after network topology optimization; Cost(i, j, T * ) represents the communication cost between node i and node j; Connectivity(i, j, T * ) represents the connection strength between node i and node j. If there is a connection between them, it is 1, otherwise it is 0; Load k Indicates the load of node k, reflecting the computing or data processing pressure of the node; Latency l Represents the communication delay of node l; Distance i represents the distance between node i and its optimal communication path; λ1, λ2, λ3 are regularization coefficients that control the trade-off between different optimization objectives; For each time t, the network topology is adjusted by the following formula in, is the optimized topology at time t; T t-1 is the topological structure of the previous moment; η is the adjustment step size, which controls the speed of topological adjustment; S net (t) is the current network status information, S target (t) is the target network state, which is given by the dynamic model of environmental changes; ΔT t-1 is the topological change at the previous moment, which is used to capture the historical change trend, and γ is the trend correction coefficient.

7. A multimodal sensor time synchronization method for road monitoring according to claim 6, characterized in that: After network topology optimization, the complete optimized data set needs to be based on the new network topology structure T * Path adjustments are made to ensure data is transmitted with the lowest latency and highest bandwidth, including: For each data point Its final transmission path can be expressed as: in, is the optimized data point; Represents data points According to the optimized topology T * Select the best transmission path; For topology T * The set of all possible paths; Cost(j, T * ) is the communication cost of node j on the path, which usually includes delay and bandwidth; In the long-term operation, a long-term adaptive optimization method based on deep reinforcement learning is introduced to guide the continuous optimization of network topology through the deep reinforcement learning model: Among them, R(t) is the total reward at time t; For node i in the topology The following reward.

8. A multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that: In generating high-precision time synchronization signals, a synchronization signal optimization algorithm based on weighted least squares method is proposed to minimize the delay difference between nodes and generate the optimal synchronization signal; wherein, the synchronization signal optimization algorithm is expressed as: in, is the optimization objective function of time synchronization; w ij is the weight between node i and node j, which is usually related to the communication load, distance and topological relationship of the nodes; t i and t j is the synchronization time between node i and node j; ΔT ij is the delay difference between node i and node j, reflecting the network delay under the current topology; The method of dynamically adjusting the time synchronization signal by real-time monitoring of changes in the network status includes: At each time t, the synchronization signal update formula is: T sync，t =T sync，t-1 +η·(S net，t -S net，t-1 ) Among them, T sync，t is the synchronization signal at time t; T sync，t-1 is the synchronization signal of the previous moment; η is the adjustment step size, which controls the speed of updating the synchronization signal; S net，t and S net，t-1 They are the network status information of the current moment and the previous moment respectively; After the time synchronization signal is generated, the optimized time synchronization signal T sync Will be used to adjust the dataset Each data point in the , to meet the global time synchronization requirements, including: The timestamps of the data points will be corrected based on the generated synchronization signal to ensure that the data of all nodes can be processed according to a unified time base. The complete optimized data set adjustment formula is: in, It is the dataset after time synchronization; is the optimized data point; T sync (t i ) is the time synchronization signal of the i-th data point, which corrects the timestamp of each data point.

9. A multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that: The method of time aligning and fusing multimodal data from different sensors or data sources based on a high-precision time synchronization signal includes: All data are aligned to a unified time base through interpolation methods. The time synchronization signal provides time correction for each node. Using the time synchronization signal, the data of all modes are converted to the same time scale through interpolation algorithms. The aligned modal data are fused, and the fusion weight is adaptively adjusted based on the quality and signal-to-noise ratio of each modality. Data of different modalities may have different quality and credibility. The fusion algorithm needs to adjust the weight according to the quality of the data to achieve the best fusion effect. The objective function of adaptive weighted fusion is as follows: Among them, D fused (t) is the fusion data at time t, containing information of all modalities; D aligned，i (t) is the i-th modal data at time t, which has been time aligned; w i (t) is the adaptive fusion weight of the i-th modality at time point t, and the weight w i (t) is calculated as: Among them, q i (t) is the quality assessment function of the i-th mode at time t.

10. A multi-modal sensor time synchronization method for road monitoring according to claim 9, characterized in that: The optimization goal of the optimization algorithm based on temporal consistency constraints is to minimize the time deviation of all modal data. The specific formula is: in, Optimize the objective function for temporal consistency; D aligned，i (t) is the alignment data of the i-th mode at time t; is the ideal value of the fused data at time t.

Images