A multi-modal sensor time synchronization method for road monitoring

Abstract

The application provides a multi-modal sensor time synchronization method for road monitoring, comprising: acquiring sensor data for target road detection, and determining a communication time delay; according to actual network conditions and environmental changes, optimizing a compensation value in real time to obtain an optimized data set; based on the complete optimized data set, determining a network topology structure; according to the optimized network topology structure, determining a time synchronization signal sequence of relative time delay information of each node in the network; and optimizing time synchronization of all sensors for target road detection. The application introduces a dynamic environment modeling and adaptive network topology optimization strategy, effectively overcomes the shortcomings of traditional time synchronization technology, and can realize high-precision multi-modal data time synchronization in a complex dynamic environment.

Description

A Multimodal Sensor Time Synchronization Method for Road Monitoring Technical Field

[0001] This invention belongs to the field of road monitoring technology, and in particular relates to a multimodal sensor time synchronization method for road monitoring. Background Technology

[0002] With the development of Intelligent Transportation Systems (ITS) and autonomous driving technologies, the demand for multimodal sensor data fusion in areas such as lane monitoring, road condition analysis, and traffic flow management is increasing. Against this backdrop, more and more sensors are being applied to road monitoring and analysis, including common sensors such as vision cameras, LiDAR, millimeter-wave radar, and infrared sensors. These sensors each have different functions and advantages: vision cameras can provide rich environmental details, LiDAR can accurately measure distances and generate high-precision 3D point cloud data, millimeter-wave radar has strong robustness in adverse weather conditions, and infrared sensors perform excellently at night or in low-visibility environments. Therefore, data fusion from multiple sensors helps to comprehensively and accurately monitor road conditions, identify traffic events, and analyze traffic flow.

[0003] However, despite the wealth of monitoring information provided by multimodal sensors, differences in their operating principles, data sampling frequencies, and clock drift make it difficult to precisely align the data in time, thus affecting the effectiveness of data fusion. This problem is particularly prominent in complex road environments. For example, visual cameras typically have high sampling frequencies, capturing tens to hundreds of frames per second, while radar sampling frequencies are usually lower, perhaps only a few frames per second or less. Therefore, effectively aligning the data from different modalities in time while ensuring high-precision data synchronization has become a pressing technical challenge.

[0004] Existing solutions primarily focus on hardware clock synchronization and latency compensation. For example, the IEEE 1588 Precision Time Protocol (PTP) can synchronize the clocks of various sensors to the sub-microsecond level through a hardware-level time synchronization mechanism. However, this method typically requires each sensor to have hardware that supports the protocol and is affected by factors such as physical distance between sensors and transmission delays, making it difficult to work effectively in real-time dynamic environments. Furthermore, existing latency compensation methods are often static and do not consider dynamically changing road environment factors, such as traffic flow and weather changes. This makes these methods unsuitable for latency variations in scenarios such as highways and complex road conditions, leading to reduced synchronization accuracy.

[0005] Besides these time-delay synchronization methods, some studies have proposed methods based on inter-sensor data correlation or machine learning to achieve time alignment of multimodal data. However, these methods typically rely on complex models and computations, require a large amount of training data to optimize the models, and because multimodal data often overlaps or contains errors in time series, they are often difficult to adapt to various changing environments. Especially under non-ideal conditions, the accumulation of errors may affect the accuracy and usability of the data.

[0006] Therefore, existing technologies face the following core challenges in the field of multimodal sensor time synchronization:

[0007] Latency and clock drift: The latency and clock drift problems of different sensors are difficult to solve through hardware-level synchronization, resulting in low time synchronization accuracy, especially in dynamically changing environments, where efficient latency compensation cannot be achieved.

[0008] Ignoring environmental factors: Most existing synchronization methods ignore the impact of external environment (such as traffic flow, weather changes, etc.) on time delay, making them unable to cope with complex and dynamic road environment in practical applications.

[0009] Balancing synchronization efficiency and accuracy: Existing methods struggle to find a good balance between real-time performance and synchronization accuracy, especially in scenarios with a large number of sensors and limited bandwidth, which may not meet the requirements for high-precision, multimodal data synchronization. Summary of the Invention

[0010] The purpose of this invention is to propose a multimodal sensor time synchronization method for road monitoring. By introducing dynamic environment modeling and adaptive network topology optimization strategies, it effectively overcomes the shortcomings of traditional time synchronization techniques and can achieve high-precision multimodal data time synchronization in complex dynamic environments.

[0011] To achieve the above objectives, this invention provides a multimodal sensor time synchronization method for road monitoring, comprising:

[0012] Acquire sensor data for target road detection, construct a dynamic model of environmental change based on the sensor data to predict environmental changes, predict the communication delay caused by environmental changes based on the predicted environmental changes, generate the predicted delay compensation, and correct the communication delay based on the predicted delay compensation to obtain the corrected communication delay.

[0013] In obtaining the corrected communication latency, an adaptive adjustment strategy is designed based on the predicted latency compensation. The compensation value is optimized in real time according to the actual network conditions and environmental changes. Based on the optimized compensation value, all sensor data are finally synchronized and corrected to obtain a complete optimized dataset.

[0014] Based on a complete optimization dataset, the network topology is determined to adapt to dynamically changing network load, communication latency, and connection quality between nodes, and an optimized network topology is generated.

[0015] Based on the optimized network topology, the time synchronization signal sequence of relative time delay information of each node in the network is determined, and the optimal synchronization signal is generated. At the same time, by monitoring the changes in network status in real time, the time synchronization signal is dynamically adjusted to generate a high-precision time synchronization signal. Based on the high-precision time synchronization signal, each data point in the complete optimized dataset is adjusted so that each data point in the complete optimized dataset meets the global time synchronization requirements, thus completing the time synchronization of all sensors for target road detection.

[0016] And / or, also includes:

[0017] Based on high-precision time synchronization signals, multimodal data from different sensors or data sources are time-aligned and fused. At the same time, an optimization algorithm based on time consistency constraints is designed to fine-tune the time sequence error in the fused data, ensuring that the timestamps of all modal data are consistent throughout the time range, thereby optimizing the time synchronization of all sensors for target road detection.

[0018] In some embodiments, acquiring the sensor data for target road detection includes:

[0019] Sensor data is multimodal data acquired through multimodal devices, where each data point d of the sensor data is... i It includes observations from multiple sensors;

[0020] The method of constructing a dynamic environmental change model based on the sensor data to predict environmental changes includes:

[0021] Based on sensor data, a long short-term memory network is used to model the temporal dependence of the environment and predict future environmental changes. By training the sensor data, a dynamic model of environmental change is obtained, which can predict environmental changes at a certain time or within a time window in the future.

[0022] In some embodiments, based on the predicted environmental changes at a future moment or within a time window, combined with the actual environmental changes and the deviation between the predicted and actual environmental changes, a predicted time delay compensation value is calculated through a nonlinear mapping function, and time delay correction is performed on each data point according to the predicted time delay compensation value to obtain a corrected dataset; wherein, the nonlinear mapping function captures the complex relationship between time delay and environmental changes through a regression model or adaptive learning method.

[0023] In some embodiments, the adaptive adjustment strategy based on the predicted delay compensation includes:

[0024] For each time window t, optimize the following objective function To adjust the latency compensation value:

[0025]

[0026] Where, d′ i For the corrected data, To optimize compensation values ​​in real time based on actual network conditions and environmental changes. Optimized data, The optimized delay compensation value is represented by λ1 and λ2, which are regularization parameters used to balance the accuracy of data compensation and the smoothness of delay variations. This indicates the rate of change of the delay compensation value, preventing excessive fluctuations in delay compensation.

[0027] Then use the optimized delay compensation value The sensor data is optimized to obtain a complete optimized dataset.

[0028] In some embodiments, after the final synchronization and correction of all sensor data based on the optimized compensation value, the delay compensation is dynamically adjusted according to the real-time network status by sampling the real-time information of the current network and combining it with the dynamic model of environmental changes, and the optimal delay compensation value is calculated in real time.

[0029] In some embodiments, the objective function for optimizing the network topology is expressed as:

[0030]

[0031] in, Let $Cost(i, j, T)$ be the objective function, representing the cost after network topology optimization. * The connection(i, j, T) represents the communication cost between node i and node j; * The value ) represents the connection strength between node i and node j; it is 1 if a connection exists, and 0 otherwise. k This represents the load on node k, reflecting the computational or data processing pressure on that node; Latency l Distance represents the communication delay of node l. i λ1, λ2, and λ3 represent the distance between node i and its optimal communication path; λ1, λ2, and λ3 are regularization coefficients that control the trade-offs between different optimization objectives.

[0032] For each time t, the network topology is adjusted using the following formula.

[0033]

[0034] in, The optimal topology for time t; T t-1 The topology is the structure at the previous time step; η is the adjustment step size, controlling the speed of topology adjustment; S net (t) represents the current network status information, S target (t) represents the target network state, given by the dynamic model of environmental changes; ΔT t-1 γ represents the topological change at the previous moment, used to capture historical trends, and γ is the trend correction coefficient.

[0035] In some embodiments, after network topology optimization, the complete optimization dataset needs to be determined based on the new network topology T. * Path adjustments are made to ensure data can be transmitted with minimal latency and maximum bandwidth, including:

[0036] For each data point Its final transmission path can be represented as:

[0037]

[0038] in, These are the optimized data points; Representing data points Based on the optimized topology T * Choose the best transmission path; For topology T * The set of all possible paths; Cost(j, T) * ) represents the communication cost of node j on the path, which typically includes latency and bandwidth, etc.

[0039] In long-term operation, a long-term adaptive optimization method based on deep reinforcement learning was introduced to guide the continuous optimization of the network topology through a deep reinforcement learning model:

[0040]

[0041] Where R(t) is the total reward at time t; For node i in the topology The reward is as follows.

[0042] In some embodiments, in generating a high-precision time synchronization signal, a synchronization signal optimization algorithm based on weighted least squares is proposed to minimize the time delay difference between nodes and generate the optimal synchronization signal; wherein, the synchronization signal optimization algorithm is expressed as:

[0043]

[0044] in, The objective function for time synchronization optimization; w ij The weight between node i and node j is typically related to the nodes' communication load, distance, and topology; t i and t j ΔT represents the synchronization time between node i and node j. ij The time delay difference between node i and node j reflects the network latency under the current topology;

[0045] The method of dynamically adjusting the time synchronization signal by monitoring changes in network status in real time includes:

[0046] The synchronization signal update formula at each time t is:

[0047] T sync，t =T sync，t-1 +η·(S net，t -S net，t-1 )

[0048] Among them, T sync，t The synchronization signal at time t; T sync，t-1 η is the synchronization signal from the previous moment; η is the adjustment step size, controlling the speed at which the synchronization signal is updated; S net，t and S net，t-1 These are the network status information for the current moment and the previous moment, respectively;

[0049] After the time synchronization signal is generated, the optimized time synchronization signal T sync Will be used to adjust the dataset Each data point in the dataset is made to meet global time synchronization requirements, including:

[0050] The timestamps of the data points will be corrected according to the generated synchronization signal to ensure that the data of all nodes can be processed according to a unified time base. The complete formula for optimizing the dataset adjustment is as follows:

[0051]

[0052] in, The dataset after time synchronization; For the optimized data points; T sync (t i ) is the time synchronization signal for the i-th data point, used to correct the timestamp of each data point.

[0053] In some embodiments, the step of time-aligning and fusing multimodal data from different sensors or data sources based on a high-precision time synchronization signal includes:

[0054] All data is aligned to a unified time base using interpolation methods. The time synchronization signal provides time correction for each node. Using the time synchronization signal, the data of all modes are converted to the same time scale through interpolation algorithms.

[0055] The aligned modal data are fused, and the fusion weights are adaptively adjusted based on the quality and signal-to-noise ratio of each modality. Data from different modalities may have varying qualities and reliability; the fusion algorithm needs to adjust the weights according to the data quality to achieve the best fusion effect. The objective function of adaptive weighted fusion is as follows:

[0056]

[0057] Among them, D fused (t) represents the fused data at time t, containing information from all modes; D aligned，i (t) represents the i-th modal data at time t, which has already been time-aligned; w i (t) represents the adaptive fusion weight of the i-th mode at time t, where the weight w i The calculation method for (t) is as follows:

[0058]

[0059] Where, q i (t) is the quality evaluation function of the i-th mode at time t.

[0060] In some embodiments, the optimization objective of the optimization algorithm based on temporal consistency constraints is to minimize the time deviation of all modal data, as specifically expressed by the formula:

[0061]

[0062] in, Optimize the objective function for timing consistency; D aligned，i (t) represents the alignment data for the i-th mode at time t; Let be the ideal value of the fused data at time t.

[0063] The beneficial technical effects of the present invention are at least as follows:

[0064] (1) This invention is the first to combine dynamic data of the lane environment (such as traffic flow, weather conditions, etc.) with sensor delay estimation. By collecting environmental data in real time and using algorithms such as Extended Kalman Filter (EKF) to dynamically estimate delay changes, delay compensation is performed in conjunction with a dynamic environment model. This innovation makes delay compensation no longer static, but adaptive, and can automatically adjust the delay compensation strategy according to environmental changes, effectively improving synchronization accuracy.

[0065] (2) This invention achieves efficient transmission of synchronization signals among multiple sensors by constructing an adaptive network topology optimization mechanism based on sensor distribution and network bandwidth. In situations with limited bandwidth and widespread sensor distribution, optimizing the network topology ensures rapid transmission of synchronization signals, reduces signal loss or delay, and guarantees the stability and efficiency of the synchronization process. This innovation further improves the reliability and accuracy of time synchronization by considering network factors during the synchronization process.

[0066] (3) Based on high-precision time synchronization, this invention combines a multimodal sensor data fusion method. Through precise timestamp correction and multimodal data time alignment technology, it ensures that data from different sensors can be accurately aligned on the time axis, thereby improving the accuracy and consistency of data fusion. The fusion of different modal data not only improves the perception accuracy but also better supports road monitoring and analysis tasks in complex environments.

[0067] (4) By introducing innovative technologies such as dynamic environment modeling, adaptive delay compensation and network topology optimization, this invention effectively overcomes the problems of insufficient delay estimation accuracy, poor environmental adaptability and the contradiction between synchronization efficiency and accuracy in the existing technology. It provides a multi-modal sensor time synchronization method that is suitable for complex road environments and can achieve high-precision time synchronization, which significantly improves the overall performance of the road monitoring system. Attached Figure Description

[0068] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0069] Figure 1 is a flowchart of a multimodal sensor time synchronization method for road monitoring according to the present invention. Detailed Implementation

[0070] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0071] In one or more embodiments, as shown in FIG1, a multimodal sensor time synchronization method for road monitoring is disclosed, comprising:

[0072] S1. Acquire sensor data for target road detection, construct a dynamic model of environmental change based on the sensor data to predict environmental changes, predict the communication delay caused by environmental changes based on the predicted environmental changes, generate the predicted delay compensation, and correct the communication delay based on the predicted delay compensation to obtain the corrected communication delay.

[0073] In this step, the invention dynamically models the environment and predicts communication latency based on environmental changes, further addressing data synchronization and latency compensation issues. The core objective of this step is to predict communication latency caused by environmental factors and optimize data synchronization and latency calibration strategies in subsequent steps based on the prediction results.

[0074] Furthermore, the sensor data D = {d1, d2, ..., d...} n The data is multimodal data acquired through multimodal devices such as camera arrays and lidar. Each data point d... i It includes observations from multiple sensors. Assume each d... i It is a vector, d i =[x i1 x i2 , ..., x im ], where m is the number of sensors, and each x ij This represents the observation value of the j-th sensor within time window i.

[0075] Furthermore, this invention uses a Long Short-Term Memory (LSTM) network to model the temporal dependencies of the environment and predict future environmental changes. LSTM can capture long-term dependencies in complex time-series data, making it suitable for describing the dynamic changes of environmental factors over time. An environmental model M is obtained by training D. env This model can predict environmental changes at a future point in time or within a time window.

[0076] Specifically, this invention trains D by minimizing the following loss function to optimize the parameters θ of the LSTM:

[0077]

[0078] in, Model M env For the environmental prediction value at time i, d iThe data used are actual observation data, and λ1 is a regularization parameter that controls the smoothness between predicted values. The second regularization term helps maintain the smoothness of environmental changes and avoids overfitting and unstable fluctuations. After model training, this invention obtains environmental prediction results.

[0079] Furthermore, based on the environmental model M env Output This invention will further predict the communication delay δ caused by environmental changes. t The data is then corrected based on this. The latency compensation model adaptively adjusts communication latency, making data synchronization between different sensors more accurate.

[0080] This invention assumes a certain relationship between time delay and environmental changes, with the time delay value δ. t The latency is affected not only by environmental factors but also by sensor conditions (such as load and distance) and network transmission conditions. To model the latency, this invention proposes a latency compensation model that incorporates environmental predictions. The time delay δ is calculated using the following formula to determine the deviation from the sensor data. t :

[0081]

[0082] in, δ represents the deviation between the actual data and the predicted data at time i. f is a nonlinear mapping function used to capture the complex relationship between time delay and environmental changes through regression models or adaptive learning methods. t This is the delay compensation value.

[0083] Furthermore, this invention can also introduce a regularization term to avoid system instability caused by over-adjustment:

[0084]

[0085] Where, δ t，target For the ideal time delay value, δ t，prev λ1 represents the time delay value at the previous moment, and λ2 is an adjustment coefficient to ensure smooth time delay changes and avoid excessive fluctuations.

[0086] Furthermore, combined with the predicted time delay compensation δ t For each data point d i After time delay correction, the corrected dataset D′={d′1,d′2,...,d′ n Specifically, by shifting and adjusting the data at each moment, the sensor data is ensured to be synchronized and errors caused by time delays are eliminated.

[0087] For the data d of each sensor i Its correction formula is:

[0088] d′ i =d i -δ t

[0089] Where, d′ i This is the corrected data, δ t It is the compensation value predicted based on the time delay model.

[0090] S2. In obtaining the corrected communication delay, an adaptive adjustment strategy is designed based on the predicted delay compensation. The compensation value is optimized in real time according to the actual network conditions and environmental changes. Based on the optimized compensation value, all sensor data are finally synchronized and corrected to obtain a complete optimized dataset.

[0091] The goal of this step is to further implement adaptive latency compensation based on the environmental dynamic modeling and latency prediction results obtained in the previous step. This involves fine-tuning the latency through optimization algorithms to reduce errors caused by communication latency and improve the accuracy of data synchronization. The core of this step lies in dynamically adjusting the compensation value to ensure that the latency compensation remains consistent with real-time changes in the network environment and to optimize the final data transmission and fusion process.

[0092] Furthermore, due to the time-varying nature of environmental changes, the predicted time delay compensation δ t In practical applications, this may not be statically effective, thus requiring dynamic adjustment. This invention introduces an adaptive adjustment strategy based on the current network state and environment model, optimizing the compensation value δ in real time according to actual network conditions and environmental changes. t This enables more accurate latency compensation.

[0093] Specifically, for each time window t, the delay compensation value is adjusted by optimizing the following objective function:

[0094]

[0095] Where, d′ i This is the data after time delay compensation from the previous step. For the optimized data, The optimized delay compensation value is represented by λ1 and λ2, which are regularization parameters used to balance the accuracy of data compensation and the smoothness of delay changes. This indicates the rate of change of the delay compensation value, preventing excessive fluctuations in delay compensation.

[0096] This optimization problem is solved using gradient descent or Bayesian optimization algorithms to minimize data errors while ensuring the stability of time delay compensation.

[0097] Furthermore, to further improve data synchronization accuracy, this invention performs fine-tuning of the latency value in the compensated dataset. Specifically, this is achieved by using the optimized latency compensation value... The data is corrected to obtain the optimized dataset D*.

[0098] For the sensor data at each moment, the present invention corrects it in the following way:

[0099]

[0100] in, For the optimized data, This represents the final optimized latency compensation value. The latency value is obtained after optimization and compensation. This invention not only eliminates the errors caused by static delay compensation, but also further fine-tunes the data to improve the accuracy of data synchronization. This method ensures the adaptability and accuracy of delay compensation, correcting not only the communication delay between sensors but also dynamically considering environmental changes.

[0101] Furthermore, in practical applications, the network environment is constantly changing, and latency compensation should also be dynamically adjusted according to the real-time network status. This invention introduces a network feedback mechanism, enabling the latency compensation strategy to be optimized based on real-time network conditions.

[0102] Specifically, this invention can sample information such as the current network's transmission rate, bandwidth, and latency, and combine this with an environmental change model M. env The optimal delay compensation value is calculated in real time. This invention defines the following feedback function:

[0103]

[0104] Where, δ t The initial predicted delay compensation value is η, which is the adjustment coefficient that controls the speed of feedback adjustment. NetworkStatus represents the current network status (e.g., bandwidth, latency) and TargetStatus represents the target network status, based on the expected optimal network conditions.

[0105] Furthermore, combined with the optimized delay compensation value The final synchronization and correction of all sensor data yields the complete optimized dataset D. *After precise latency compensation, this dataset can be used for subsequent data fusion and analysis. The adaptive latency compensation and optimization step plays a crucial role in the entire patent. By precisely adjusting latency, it optimizes the data synchronization process, ensuring the effectiveness of latency compensation for multi-sensor data under various environmental and network conditions. This step flexibly responds to latency changes in dynamic network environments, providing accurate data synchronization and greatly improving the efficiency and accuracy of data processing.

[0106] S3. Based on the complete optimization dataset, determine the network topology to adapt to dynamically changing network load, communication latency, and connection quality between nodes, and generate the optimized network topology.

[0107] The goal of this step is to build upon the output of the previous step (the optimized latency compensation dataset D). * This paper proposes an adaptive network topology optimization algorithm to adjust the network topology to adapt to dynamically changing network load, communication latency, and connection quality between nodes. The optimized network topology should achieve optimal network performance globally while adapting to environmental changes, thus improving the system's adaptability and data transmission efficiency.

[0108] Furthermore, regarding the time-delay compensated dataset D... * Optimizing network topology requires comprehensive consideration of multiple factors, including communication costs between nodes, data transmission latency, and network load. To dynamically adjust the topology in complex environments, this invention proposes a network topology optimization objective function with latency compensation and load adaptation. This function not only considers communication costs, latency, and load between nodes but also introduces the important factor of network adaptability.

[0109] The objective function can be expressed as:

[0110]

[0111] in, Let $Cost(i, j, T)$ be the objective function, representing the cost after network topology optimization. * The connection(i, j, T) represents the communication cost between node i and node j, which typically includes factors such as bandwidth, node load, and transmission latency; * The value ) represents the connection strength between node i and node j; it is 1 if a connection exists, and 0 otherwise. k This represents the load on node k, reflecting the computational or data processing pressure on that node; Latency l Distance represents the communication delay of node l. iλ represents the distance between node i and its optimal communication path; λ1, λ2, and λ3 are regularization coefficients that control the trade-offs between different optimization objectives. The objective function introduces three important optimization factors: communication cost, node load, and network topology distance. This optimization does not only focus on a single factor but also achieves system-level balance through multi-dimensional consideration, which can improve the accuracy and adaptability of topology optimization, especially in real-time changing network environments.

[0112] Furthermore, to adapt to dynamic changes in network conditions, this invention designs an adaptive topology adjustment algorithm based on a real-time feedback mechanism. This algorithm can adjust the network topology at each time step t based on the real-time network state S. net (t) and environmental change prediction model M env Adjust the topology.

[0113] At each time t, the present invention adjusts the network topology using the following formula.

[0114]

[0115] in, The optimal topology for time t; T t-1 The topology is the structure at the previous time step; η is the adjustment step size, controlling the speed of topology adjustment; S net (t) represents the current network status information, including node load, latency, etc.; S target (t) represents the target network state, predicted by the environmental prediction model M. env Given; ΔT t-1 ΔT represents the topology change at the previous time step, used to capture historical change trends, and γ is a trend correction coefficient. This algorithm enhances the adaptability of topology adjustments by combining historical topology change trends with real-time network conditions. Specifically, ΔT... t-1 The introduction of this item allows topology adjustments to not only depend on the current network state, but also take into account historical evolution trends, thereby further improving the system's ability to predict and adapt to future changes.

[0116] Furthermore, after network topology optimization, data transmission paths and connections between nodes may change. Therefore, dataset D* needs path adjustments based on the new network topology T* to ensure data can be transmitted with minimal latency and maximum bandwidth.

[0117] For each data point Its final transmission path can be represented as:

[0118]

[0119] in, These are the optimized data points; Representing data points Based on the optimized topology T * Choose the best transmission path; For topology T * The set of all possible paths; Cost(j, T) * ) represents the communication cost of node j on the path, which typically includes latency and bandwidth.

[0120] This method optimizes data transmission paths and combines them with node connections optimized by the topology structure to ensure that latency and data packet loss are minimized during data transmission. Especially when the network load is high, it can dynamically select lower-cost transmission paths.

[0121] Furthermore, in long-term operation, the adaptive capability of network topology optimization is crucial to ensuring the long-term efficient operation of the network. Therefore, this invention introduces a long-term adaptive optimization method based on deep reinforcement learning, using a reward mechanism to guide the continuous optimization of the network topology. This invention performs topology optimization using the following deep reinforcement learning model:

[0122]

[0123] Where R(t) is the total reward at time t; For node i in the topology The reward is calculated based on factors such as the node's communication latency and bandwidth utilization.

[0124] This method continuously optimizes the network structure through deep reinforcement learning during the long-term evolution of the network topology, enabling the network to constantly adjust itself to cope with uncertainties such as load fluctuations and network topology changes, thus ensuring the stability and efficiency of the system.

[0125] The core of this step lies in adaptively adjusting the network topology and combining real-time network status and latency compensation data. By comprehensively optimizing the objective function, historical topology change trends, and deep reinforcement learning, the network achieves dynamic adaptation and long-term optimization.

[0126] S4. Based on the optimized network topology, determine the time synchronization signal sequence of the relative time delay information of each node in the network, and generate the optimal synchronization signal. At the same time, by monitoring the changes in network status in real time, dynamically adjust the time synchronization signal to generate a high-precision time synchronization signal. Adjust each data point in the complete optimized dataset according to the high-precision time synchronization signal so that each data point in the complete optimized dataset meets the global time synchronization requirements, and complete the time synchronization of all sensors for target road detection.

[0127] In this step, the objective of this invention is to design a high-precision time synchronization signal generation algorithm to ensure that the time accuracy of data transmission reaches the microsecond or even sub-microsecond level. Precise time synchronization can further reduce latency deviations in network communication, improve the overall system performance, and play a crucial role, especially in real-time data transmission and dynamic scheduling.

[0128] Furthermore, a key aspect of this step is designing a high-precision time synchronization framework based on optimized network topology. By analyzing the communication latency and load between nodes, this invention utilizes the topology T... * A global time synchronization model was constructed. The goal of this model is to generate a high-precision time synchronization signal sequence using the relative time delay information of each node in the network.

[0129] Based on the topology optimization in the preceding steps, this invention introduces the following synchronization signal generation strategy:

[0130]

[0131] Among them, T sync It is a high-precision time synchronization signal; Sync(·) is the synchronization function, combined with the network topology T * Network status information S net and optimize datasets Generate time synchronization signals for each node; where S net Factors such as communication latency and bandwidth of each node reflect the dynamic changes in network status; It is the optimized dataset, which includes data after latency compensation.

[0132] This method generates a precise time synchronization signal globally by combining network topology and real-time network status information. It not only considers communication delays between nodes but also adapts to changes in network load. The synchronization function `Sync` can adjust the time synchronization signal under different network conditions, thereby achieving high-precision delay control.

[0133] Furthermore, to further improve the accuracy of time synchronization, this invention proposes a synchronization signal optimization algorithm based on weighted least squares (WLS). The goal of this algorithm is to optimize the synchronization signal based on the optimized dataset. and network topology T * The optimal synchronization signal is generated by minimizing the time delay difference between nodes. The objective function of the optimization algorithm is expressed as:

[0134]

[0135] in, The objective function for time synchronization optimization; w ij The weight between node i and node j is typically related to the nodes' communication load, distance, and topology; t i and t j ΔT represents the synchronization time between node i and node j. ij The time delay difference between node i and node j reflects the network latency under the current topology; the objective function is to minimize the time delay difference between nodes, thereby optimizing the time synchronization accuracy.

[0136] Furthermore, this method incorporates load and topology information between nodes through weighted least squares, ensuring that the optimization of the synchronization signal better reflects the actual network conditions and avoiding delay errors in traditional synchronization algorithms. In particular, the weight w... ij The introduction of this feature allows the synchronization signal to not only take into account communication delay, but also topology and load factors, thereby enhancing synchronization accuracy and robustness.

[0137] Furthermore, in practical applications, the network state is constantly changing, especially the load and latency, which may change rapidly due to traffic fluctuations. To adapt to this dynamic environment, this invention designs an adaptive synchronization signal update algorithm that dynamically adjusts the time synchronization signal by monitoring changes in the network state in real time. At each time t, the synchronization signal update formula is:

[0138] T sync，t =T sync，t-1 +η·(S net，t -S net，t-1 )

[0139] Among them, T sync，t The synchronization signal at time t; T sync，t-1 η is the synchronization signal from the previous moment; η is the adjustment step size, controlling the speed at which the synchronization signal is updated; S net，t and S net，t-1 These are network status information for the current moment and the previous moment, respectively, reflecting changes in network load, latency, etc.

[0140] This method dynamically adjusts the synchronization signal by providing real-time feedback on changes in the network state. Adjusting the η-step size makes the synchronization signal updates smoother, avoiding drastic fluctuations in synchronization errors. Furthermore, by comparing the differences between the current and past network states, the algorithm adapts more flexibly to network changes, improving the accuracy of time synchronization.

[0141] Furthermore, after the time synchronization signal is generated, the optimized time synchronization signal T sync Will be used to adjust the dataset Each data point in the dataset is adjusted to conform to global time synchronization requirements. Specifically, the timestamps of the data points are corrected based on the generated synchronization signal to ensure that data from all nodes can be processed according to a unified time base. The dataset adjustment formula is:

[0142]

[0143] in, The dataset after time synchronization; For the optimized data points; T sync (t i () is the time synchronization signal for the i-th data point, used to correct the timestamp of each data point. This is achieved through the time synchronization signal T. sync The data points are calibrated to ensure that the dataset is processed according to a unified clock. This operation guarantees that in a multi-node system, data from all nodes can be integrated and analyzed under the same time base, eliminating biases caused by different time sources.

[0144] The core of this step lies in generating a high-precision time synchronization signal. Combined with the optimized network topology dataset and network state information, this resolves the issue of inconsistent latency among nodes in the network, ensuring data accuracy and consistency. The generation of the time synchronization signal not only improves the overall system performance but also provides a reliable time reference for subsequent data fusion and real-time transmission, playing a crucial role, especially in scenarios requiring extremely high synchronization accuracy.

[0145] S5. Based on high-precision time synchronization signals, multimodal data from different sensors or data sources are time-aligned and fused. At the same time, an optimization algorithm based on time consistency constraints is designed to fine-tune the time sequence error in the fused data, ensuring that the timestamps of all modal data are consistent throughout the time range, and optimizing the time synchronization of all sensors for target road detection.

[0146] In this step, the objective of the present invention is to address the issue of a high-precision time synchronization signal T generated in the previous step. sync This involves time-aligning and fusing multimodal data from different sensors or data sources. Multimodal data typically includes images, audio, video, and sensor data, which often have different sampling frequencies, timestamps, and data formats. Therefore, an efficient and robust time alignment and fusion method is needed to ensure that the data, once aligned on the timeline, provides unified and accurate information for subsequent data analysis and decision-making.

[0147] Furthermore, time alignment of multimodal data sources is central to the entire fusion process. Data from different modalities may have different sampling frequencies and timestamps; therefore, interpolation methods are needed to align all data to a unified time reference. The time synchronization signal T... sync The invention provides time corrections for each node, and using these signals, it can convert the data of all modes to the same time scale through an interpolation algorithm.

[0148] For each modal data D mi Interpolation is performed using the following time alignment formula:

[0149] D aligned，i (t)=Interp(D mi T sync )

[0150] Among them, D aligned，i (t) represents the aligned data of mode i at time point t; Interp(·) represents the interpolation function, which can be used to calculate the data based on the time synchronization signal T. sync For data D mi Time alignment is performed. The interpolation method is selected based on the characteristics of different modalities (e.g., inter-frame interpolation may be used for video data, while linear interpolation may be used for sensor data).

[0151] This interpolation method utilizes dual information from multimodal data sources and synchronization signals, enabling efficient alignment of data from different frequencies and time scales. This avoids the accuracy loss that may occur in conventional interpolation methods and improves the accuracy of fused data.

[0152] Furthermore, the aligned modal data needs to be fused to generate a unified, multi-dimensional data structure. Here, this invention proposes an adaptive weighted fusion method, which adaptively adjusts the fusion weights based on the quality and signal-to-noise ratio of each modality. Data from different modalities may have varying qualities and reliability; the fusion algorithm needs to adjust the weights according to the data quality to achieve the optimal fusion effect.

[0153] The objective function for adaptive weighted fusion is as follows:

[0154]

[0155] Among them, D fused (t) represents the fused data at time t, containing information from all modes; D aligned，i (t) represents the i-th modal data at time t, which has already been time-aligned; w i(t) represents the adaptive fusion weight of the i-th mode at time t. This weight is dynamically adjusted based on information such as data quality and signal-to-noise ratio of mode i. Weight w i The calculation method for (t) is as follows:

[0156]

[0157] Where, q i (t) is the quality assessment function for the i-th mode at time t, typically calculated based on factors such as signal-to-noise ratio, data integrity, and sensor health status. The quality assessment function q i (t) could be one of the following types of weighting functions, for example: Where σ i，t Let be the standard deviation of mode i at time t, representing the volatility of the modal data; or, this method improves the accuracy of data fusion by introducing an adaptive weighting mechanism to dynamically adjust the weights according to the quality of the modal data. Data from different modalities may exhibit uneven quality due to various factors (such as sensor failure, external interference, etc.), and this weighting method ensures that high-quality data has a greater impact on the final fusion result, while the influence of weak-quality data is suppressed, thereby improving the reliability and accuracy of the fused data.

[0158] Furthermore, the fused data may still contain small temporal errors, especially with large datasets. To ensure temporal consistency across modalities, this invention proposes an optimization algorithm based on temporal consistency constraints. This algorithm fine-tunes the temporal errors in the fused data, ensuring that the timestamps of all modalities are consistent throughout the entire timeframe.

[0159] The optimization objective is to minimize the time skewness of all modal data, and the specific formula is:

[0160]

[0161] in, Optimize the objective function for timing consistency; D aligned，i (t) represents the alignment data for the i-th mode at time t; The ideal value of the fused data at time t is the result after time-series optimization.

[0162] The temporal consistency optimization algorithm introduces global time constraints to finely adjust temporal errors in the fused data, ensuring high temporal consistency in the final fusion result and avoiding error propagation that may be caused by temporal inconsistencies. This is particularly suitable for applications requiring high-precision temporal alignment. This step designs an innovative multimodal data alignment and fusion scheme that effectively aligns data from different modalities to a unified time reference, and then performs weighted fusion and temporal consistency optimization based on this reference.

[0163] In summary, this invention proposes a multimodal sensor time synchronization method and apparatus based on dynamic environmental modeling and adaptive network topology optimization. By introducing strategies of dynamic environmental modeling and adaptive network topology optimization, it effectively overcomes the shortcomings of traditional time synchronization techniques and can achieve high-precision multimodal data time synchronization in complex dynamic environments. This invention, through the introduction of innovative technologies such as dynamic environmental modeling, adaptive delay compensation, and network topology optimization, effectively overcomes the shortcomings of existing technologies, including insufficient delay estimation accuracy, poor environmental adaptability, and the contradiction between synchronization efficiency and accuracy. It provides a multimodal sensor time synchronization method suitable for complex road environments and capable of achieving high-precision time synchronization, significantly improving the overall performance of road monitoring systems.

[0164] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0165] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0166] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0167] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein can be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0168] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0169] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0170] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal sensor time synchronization method for road monitoring, characterized in that, include: Acquire sensor data for target road detection, construct a dynamic model of environmental change based on the sensor data to predict environmental changes, predict the communication delay caused by environmental changes based on the predicted environmental changes, generate the predicted delay compensation, and correct the communication delay based on the predicted delay compensation to obtain the corrected communication delay. In obtaining the corrected communication latency, an adaptive adjustment strategy is designed based on the predicted latency compensation. The compensation value is optimized in real time according to the actual network conditions and environmental changes. Based on the optimized compensation value, all sensor data are finally synchronized and corrected to obtain a complete optimized dataset. Based on a complete optimized dataset, the network topology is determined to adapt to dynamically changing network load, communication latency, and connection quality between nodes, generating an optimized network topology. According to the optimized network topology, the time synchronization signal sequence for the relative latency information of each node in the network is determined, and the optimal synchronization signal is generated. Simultaneously, by monitoring changes in network status in real time, the time synchronization signal is dynamically adjusted to generate a high-precision time synchronization signal. Each data point in the complete optimized dataset is then adjusted based on this high-precision time synchronization signal to ensure that each data point in the complete optimized dataset meets global time synchronization requirements, thus completing the time synchronization of all sensors for target road detection. Furthermore, based on the high-precision time synchronization signal, multimodal data from different sensors or data sources is time-aligned and fused. An optimization algorithm based on temporal consistency constraints is designed to fine-tune the temporal errors in the fused data, ensuring that the timestamps of all modal data are consistent throughout the entire time range, thereby optimizing the time synchronization of all sensors for target road detection.

2. The multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that, The acquisition of sensor data for target road detection includes: the sensor data is multimodal data acquired through a multimodal device, and each data point of the sensor data... The data includes observations from multiple sensors; the step of constructing a dynamic environmental change model based on the sensor data to predict environmental changes includes: using a long short-term memory network to model the temporal dependence of the environment based on the sensor data, and predicting future environmental changes; and obtaining a dynamic environmental change model by training the sensor data, wherein the dynamic environmental change model can predict environmental changes at a certain time or within a time window in the future.

3. The multimodal sensor time synchronization method for road monitoring according to claim 2, characterized in that, Based on the prediction of environmental changes at a certain future moment or within a time window, and combining the actual environmental changes and the deviation between the predicted and actual environmental changes, a predicted time delay compensation value is calculated through a nonlinear mapping function. The time delay is then corrected for each data point according to the predicted time delay compensation value, resulting in a corrected dataset. The nonlinear mapping function captures the complex relationship between time delay and environmental changes through a regression model or adaptive learning method.

4. The multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that, The adaptive adjustment strategy based on predicted delay compensation includes: for each time window By optimizing the following objective function To adjust the latency compensation value: ;in, For the corrected data, To optimize compensation values ​​in real time based on actual network conditions and environmental changes. Optimized data, This is the optimized latency compensation value. and This is a regularization parameter used to balance the accuracy of data compensation with the smoothness of latency variations. This indicates the rate of change of the delay compensation value, preventing excessive fluctuations in delay compensation; then the optimized delay compensation value is used. The sensor data is optimized to obtain a complete optimized dataset.

5. A multimodal sensor time synchronization method for road monitoring according to claim 4, characterized in that, After the final synchronization and correction of all sensor data based on the optimized compensation value, the delay compensation is dynamically adjusted according to the real-time network status by sampling the real-time information of the current network and combining it with the dynamic model of environmental changes, and the optimal delay compensation value is calculated in real time.

6. The multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that, The objective function for optimizing the network topology is expressed as: ;in, Let be the objective function, representing the cost after network topology optimization; Represents a node With nodes The communication cost between them; Represents a node With nodes The strength of the connection between them is 1 if a connection exists, and 0 otherwise; Represents a node The load reflects the computing or data processing pressure of the node; Represents a node Communication delay; Represents a node The distance between it and its optimal communication path; The regularization coefficient controls the trade-offs between different optimization objectives; for each time step... Adjust the network topology using the following formula ： ;in, For a moment Optimized topology; This represents the topology from the previous time step. To adjust the step size and control the speed of topology adjustment; This is the current network status information. The target network state is given by the dynamic model of environmental changes; This represents the topological change at the previous time step, used to capture historical trends. This is the trend correction coefficient.

7. A multimodal sensor time synchronization method for road monitoring according to claim 6, characterized in that, After network topology optimization, a complete optimization dataset needs to be prepared based on the new network topology. Perform path adjustments to ensure data can be transmitted with minimal latency and maximum bandwidth, including: for each data point Its final transmission path can be represented as: ;in, These are the optimized data points; Representing data points Based on the optimized network topology Choose the best transmission path; For network topology The set of all possible paths; For nodes on the path To reduce communication costs, a long-term adaptive optimization method based on deep reinforcement learning was introduced during long-term operation. This method guides the continuous optimization of the network topology through a deep reinforcement learning model. ;in, For a moment Total reward; For nodes In topology The reward is as follows.

8. The multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that, In generating high-precision time synchronization signals, a synchronization signal optimization algorithm based on weighted least squares is proposed to minimize the time delay difference between nodes and generate the optimal synchronization signal; wherein, the synchronization signal optimization algorithm is expressed as: ;in, The objective function for time synchronization optimization; For nodes and nodes The weights between them; and For nodes and nodes Synchronization time; For nodes and nodes The time delay difference reflects the network latency under the current topology; the dynamic adjustment of the time synchronization signal by real-time monitoring of network status changes includes: at each moment... The synchronization signal update formula is: ;in, For a moment Synchronization signal; This is the synchronization signal from the previous moment; To adjust the step size and control the speed of synchronization signal updates; and These are the network state information for the current time and the previous time, respectively; after the time synchronization signal is generated, the optimized time synchronization signal is... Will be used to adjust the dataset Each data point in the dataset is adjusted to meet global time synchronization requirements, including: the timestamps of the data points are corrected according to the generated synchronization signal to ensure that data from all nodes can be processed according to a unified time base. The complete optimization dataset adjustment formula is: ;in, The dataset after time synchronization; These are the optimized data points; For the first The time synchronization signal for each data point is used to correct the timestamp of each data point.

9. A multimodal sensor time synchronization method for road monitoring according to claim 1, characterized in that, The method, based on a high-precision time synchronization signal, performs time alignment and fusion of multimodal data from different sensors or data sources. This includes: aligning all data to a unified time base using interpolation methods; the time synchronization signal provides time correction for each node; and using the time synchronization signal, converting all modal data to the same time scale through an interpolation algorithm. The aligned modal data is then fused, with the fusion weights adaptively adjusted based on the quality and signal-to-noise ratio of each modality. Different modal data may have different qualities and reliability; the fusion algorithm needs to adjust the weights according to the data quality to achieve the best fusion effect. The objective function for adaptive weighted fusion is as follows: ;in, For time The fused data contains information on all modalities; For time The first The modal data has been time-aligned; For the first Each modality at time point Adaptive fusion weights, weights The calculation method is as follows: ;in, For the first Each mode in time The quality assessment function.

10. A multimodal sensor time synchronization method for road monitoring according to claim 9, characterized in that, The optimization objective of the optimization algorithm based on temporal consistency constraints is to minimize the time deviation of all modal data, and the specific formula is as follows: ;in, Optimize the objective function for timing consistency; For a moment First Alignment data for each modality; For a moment The ideal value for the fused data at that location.

Images