A thermal imaging and visible light data synchronization method based on dynamic queue management

Abstract

The present application relates to the technical field of data synchronization of inspection robots, and specifically discloses a thermal imaging and visible light data synchronization method based on dynamic queue management, which comprises the following steps: S1: hardware deployment and clock synchronization initialization, visible light frames and thermal imaging frames are obtained through visible light cameras and thermal imagers, and the deviation of master and slave clocks is calculated and calibrated; S2: real-time push stream and delay calculation, visible light frame delay and thermal imaging frame delay are calculated respectively, and the visible light frame delay and the thermal imaging frame delay are judged; S3: cache queue and cache queue management, thermal imaging frames are obtained; S4: frame matching of visible light frames and thermal imaging frames, comparison and matching alignment positioning are performed, and if the matching alignment is successful, a synchronous frame pair is output, and if the matching alignment fails, it is regarded as an abnormal state. Compared with the prior art, the method can significantly shorten the synchronization accuracy error time, and can effectively solve the problems of picture misplacement and target positioning drift caused by the space-time deviation of dual-mode data in industrial inspection.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to the field of image data synchronization processing technology for inspection robots. Background Technology

[0002] An inspection robot used in industry is equipped with both a visible light camera and a thermal imager to achieve comprehensive monitoring of the condition of industrial equipment. The visible light camera can capture visible features such as equipment deformation and liquid leaks, while the thermal imager can penetrate smoke and identify thermodynamic anomalies such as localized high temperatures. However, due to factors such as hardware acquisition mechanisms (mainly the frame rate difference between CMOS and infrared sensors), data transmission bandwidth (the amount of thermal imaging data is generally 3-5 times that of visible light), and network fluctuations, the two types of sensors experience delay deviations in their dual-modal data (commonly at the millisecond level). This causes problems such as target spatial position shifts and misalignment between temperature fields and visual features, severely affecting the accuracy of defect diagnosis. For example, when the inspection robot performs tasks in complex industrial environments, the inherent delay differences in hardware acquisition and data transmission between the visible light camera and the thermal imager may lead to severe spatiotemporal deviations in the dual-modal data, resulting in asynchronous image feeds. This asynchrony directly affects the accuracy of defect detection, real-time monitoring response efficiency, and the reliability of multimodal data fusion analysis in industrial scenarios.

[0003] To address the aforementioned issues, existing technologies primarily employ two approaches to achieve data alignment: hardware-level synchronization and software algorithm compensation. For instance, Chinese patent application CN202411772405 discloses a microcontroller-based multi-camera hard synchronization method and system. This method connects multiple cameras in series via a synchronization interface, coordinating acquisition timing using a master-slave clock mechanism. The master camera triggers a Sync pulse signal, and upon receiving the signal from the slave camera, adjusts its own shutter period to ensure strict physical-level synchronization. However, this method relies on hardware circuit design, requires precise wiring, and has limited scalability. It is prone to signal loss under complex electromagnetic interference in industrial environments and cannot handle software-level issues such as network transmission delays. Another example is Chinese patent application CN202311370407.9, which discloses a multi-camera time synchronization method, device, equipment, and storage medium. This method utilizes periodic dark patterns generated by screen PWM dimming as a time reference. The reference camera captures the position of the dark patterns, and other cameras compare the offset of the dark patterns to calculate the time difference, thus adjusting the shooting sequence. This method requires specific display devices, the calibration process is cumbersome, and in dynamic industrial scenarios, changes in the device screen state can cause the synchronization reference to fail, resulting in insufficient versatility.

[0004] In summary, existing technologies suffer from three major drawbacks: 1. Poor synchronization robustness: Hardware-based synchronization methods (such as the PTP clock protocol) rely on physical circuit connections, making them susceptible to vibration interference in industrial robot movement scenarios, and single-point failures can lead to system paralysis; software-based calibration methods (such as PWM dark ripple detection) are affected by ambient lighting and changes in equipment status, making it difficult to maintain long-term stable synchronization. 2. Insufficient dynamic adaptability: Traditional frame interpolation compensation algorithms rely on motion trajectory prediction models, resulting in interpolation errors when the robot turns rapidly or the equipment shakes; traditional buffer queue-based solutions do not consider the differences in data rates from multiple sources, leading to a high risk of queue overflow. 3. High resource consumption: Existing methods generally use CPU serial processing of synchronization logic, which cannot meet the real-time requirements of industrial applications, and multi-threaded concurrent processing is prone to resource contention. Summary of the Invention

[0005] The purpose of this invention is to provide a method for synchronizing thermal imaging and visible light data based on dynamic queue management, which significantly shortens the synchronization accuracy error time compared with the prior art and effectively solves the core problems such as image misalignment and target positioning drift caused by the spatiotemporal deviation of dual-modal data in industrial inspection.

[0006] To achieve the above objectives, the technical solution of the present invention is: a method for synchronizing thermal imaging and visible light data based on dynamic queue management. The data synchronization method includes the following steps: S1, hardware deployment and clock synchronization initialization, wherein the hardware deployment includes deploying a visible light camera and a thermal imager supporting the PTP protocol and connecting them to an edge computing node, and acquiring visible light frames and thermal imaging frames through the visible light camera and thermal imager; the clock synchronization initialization includes configuring the master and slave clocks of the visible light camera and thermal imager in the PTP protocol, calculating the deviation between the master and slave clocks and calibrating them; S2, real-time streaming and latency calculation, respectively calculating the end-to-end transmission time of each visible light frame and thermal imaging frame from acquisition to streaming to obtain the visible light frame latency and thermal imaging frame latency, judging the visible light frame latency and thermal imaging frame latency, and when neither exceeds a preset latency threshold, real-time streaming is performed through the RTMP protocol. The system streams dual-modal data. When one of the modalities exceeds a preset delay threshold, it proceeds to step S3. S3 involves managing the cache queue and obtaining thermal imaging frames. The cache queue stores visible light frames in the L1 video memory queue and thermal imaging frames in the L2 memory queue. The size of the L1 and L2 memory queues is dynamically adjusted. The cache queue management involves statistically analyzing the delay sequence of thermal imaging frames and updating the historical delay mean and variance using a dynamic weighted average algorithm. The resulting thermal imaging frames are then stored in the L2 memory queue, and the process proceeds to step S4. S4 involves frame matching of visible light and thermal imaging frames. A binary search is performed in the L1 video memory queue to find the visible light frame whose timestamp is closest to the thermal imaging frame timestamp obtained in step S3 in the L2 memory queue. If the matching is successful, a synchronized frame pair is obtained and output. If the matching fails, it is considered an abnormal frame matching state.

[0007] Furthermore, it also includes steps S5 and / or S6. If the matching and alignment are successful in step S4, a synchronized frame pair is obtained, and the synchronized frame pair is output to step S5. Step S5 is the cross-modal data alignment of the synchronized frame pair. The target region in the visible light image is detected in real time, the predicted coordinates of the target region are obtained and mapped to the thermal imaging image for corresponding region localization, the ROI region of the thermal imaging image is extracted, and a bilinear interpolation algorithm is used to align the resolution of the thermal imaging frame image and the visible light frame image. If the frame matching is considered to be in an abnormal state in step S4, step S6 is entered. Step S6 is the abnormal state handling of frame matching. When the matching and alignment fails, the failure timestamp is recorded, and the local counter is accumulated. The system increments the failure count and stores the failure timestamps in the time-series database of the edge node. It sets the window period and calls the sliding window algorithm to calculate the failure rate, which is calculated as the failure rate = the number of failures within the window / the total number of requests. If there are consecutive frames of matching and alignment failures within the window period where the failure rate exceeds the preset failure rate threshold and the number of failures exceeds the preset failure count threshold, it is determined that there is a continuous delay in the current thermal imaging frame. Otherwise, it is determined that the current matching and alignment failure is a sudden delay. If it is a sudden delay, the current thermal imaging frame is discarded, an alarm is triggered, and abnormal data packets are output. If it is a continuous delay, it dynamically switches to real-time streaming mode, retrieves the latest visible light frame from the visible light video stream, and pushes it in real time to output the data.

[0008] Furthermore, it also includes step S7, which proceeds after step S5. Step S7 is a synchronization error monitoring and resynchronization mechanism. It matches the Euclidean distance between the center points of the Bboxes in the images of the visible light frame and the thermal imaging frame of the synchronization frame pair, calculates the Euclidean distance spatial offset ΔP and the timestamp difference ΔT of the synchronization frame pair, and presets parameter thresholds for ΔP and ΔT. If either ΔP or ΔT is greater than the set parameter threshold, a resynchronization calibration strategy is triggered, the L1 video memory queue and the L2 memory queue are cleared, a hardware reset signal is sent to the visible light camera and the thermal imager, and the master and slave clock synchronization process of the PTP protocol is reinitialized. Otherwise, if the parameter thresholds for ΔP and ΔT are not exceeded, the synchronization frame pair data is output.

[0009] Furthermore, in step S1, the method for calculating the deviation between the master and slave clocks is as follows: the visible light camera sends a message and records the sending timestamp; the thermal imager receives the message and records the receiving timestamp; the thermal imager sends the message and the receiving timestamp back to the visible light camera; the master clock extracts the receiving timestamp and calculates the link delay, that is, the time difference between the receiving timestamp and the sending timestamp.

[0010] Furthermore, in step S3, the L1 memory queue size is set to μ_new + 3σ_new, rounded to the nearest integer frame, and the L2 memory queue size is set to several frames. The thermal imaging frame delay sequence {Δt1, Δt2, ..., Δtn} is expressed by the dynamic weighted average algorithm as: μ_new = 0.2 * Δt + 0.8 * μ_old, σ_new = 0.2 * (Δt - μ_new)² + 0.8 * σ_old, where Δt is the delay difference of the current thermal imaging frame, μ_new is the latest historical delay mean, μ_old is the original historical delay mean, σ_new is the latest delay variance, and σ_old is the original delay variance.

[0011] By adopting the above technical solutions, the beneficial effects of this invention are as follows: Firstly, this invention employs a dynamic queue management and adaptive delay compensation mechanism. Compared to the hardware-level synchronization method based on a fixed clock cycle in multi-camera hard synchronization based on microcontrollers, this invention, through a hierarchical caching architecture and real-time delay modeling, reduces synchronization errors in network fluctuation scenarios and avoids motion prediction errors caused by frame interpolation compensation. Secondly, this invention, through a cross-modal keyframe resynchronization strategy, further combines IDR keyframe forced refresh with PTP clock protocol co-optimization. Compared to the synchronization scheme based on screen PWM dimming multi-camera time synchronization that relies on dark pattern references, this solves the problem of dark pattern failure in dynamic industrial scenarios, improving synchronization robustness. Finally, this invention further proposes a target region accurate mapping and bilinear interpolation alignment technology based on a lightweight YOLOv8 model. Compared to the coarse-grained coordinate calibration method in traditional hardware-software hybrid synchronization schemes, this significantly improves target positioning accuracy, significantly reduces spatial offset between cameras, and significantly improves the reliability of industrial defect diagnosis. Through this multi-dimensional technological innovation integration, this invention achieves high-precision, low-jitter real-time synchronization of dual-modal data in complex industrial environments. Attached Figure Description

[0012] Figure 1 This invention relates to a flowchart of a method for synchronizing thermal imaging and visible light data based on dynamic queue management. Detailed Implementation

[0013] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0014] This embodiment discloses a method for synchronizing thermal imaging and visible light data based on dynamic queue management. The steps of the data synchronization method are described in detail below with reference to the accompanying drawings.

[0015] S1: Hardware deployment and clock synchronization initialization.

[0016] Hardware deployment includes a visible light camera and a thermal imager supporting the PTP protocol mounted on top of the inspection robot. Both can be connected to an edge computing node via an industrial-grade switch. During system operation, the visible light camera and thermal imager acquire visible light and thermal video streams during the movement process. The visible light video stream is separated into visible light frames, and the thermal video stream is separated into thermal image frames.

[0017] Hardware initialization: Hardware initialization includes configuring the PTP master and slave clocks. In the PTP protocol, the visible light camera is the master device and is set as the master clock, while the thermal imager is the slave device and is set as the slave clock. The visible light camera sends a message (here denoted as a Sync message) and records the sending timestamp t1; the thermal imager receives the message (i.e., the Sync message) and records the receiving timestamp t2. The thermal imager then sends a message back to the visible light camera (here denoted as a Delay_Req message) along with the receiving timestamp t2, i.e., it sends it back to the master clock. The master clock extracts the receiving timestamp t2 and calculates the link delay, i.e., the time difference between the receiving timestamp t2 and the sending timestamp t1, expressed as Δ=t2-t1. It then calibrates the slave clock's deviation, thereby reducing the global time synchronization error. Ideally, the global time synchronization error should be ≤1μs, thus completing the master-slave clock synchronization initialization.

[0018] S2: Real-time streaming and latency calculation.

[0019] When the system is working, it calculates the end-to-end transmission time of each frame from acquisition to streaming in the visible light frame of the visible light video stream and the thermal imaging frame of the thermal imaging video stream to obtain the SDK latency of the visible light frame and the SDK latency of the thermal imaging frame. It then determines the actual latency of the scene and switches different streaming modes based on the determination results.

[0020] The SDK latency of the visible light video stream is denoted as Δt_v, and its expression is Δt_v = current system time - visible light frame timestamp.

[0021] The SDK latency of the thermal imaging video stream is denoted as Δt_h, and its expression is Δt_h = current system time - thermal imaging frame timestamp.

[0022] The actual latency of the scene is determined as follows: when the SDK latency of the visible light frame and the SDK latency of the thermal imaging frame do not exceed the preset latency threshold, such as setting the latency threshold of the visible light frame SDK latency to 10ms and the latency threshold of the thermal imaging video stream SDK latency to 20ms, it can be expressed as when Δt_v ≤ 10ms and Δt_h ≤ 20ms, the current scene is determined to be a low-latency scene. Since the real-time latency is small, the latency between each frame of the visible light frame and the thermal imaging frame is almost negligible, so dual-modal data can be directly pushed and output in real time through the RTMP protocol, that is, the real-time data of the visible light frame and the thermal imaging frame is output. When the SDK delay of the visible light frame and the SDK delay of the thermal imaging frame exceed the preset delay threshold, it can be expressed as when Δt_v > 10ms or Δt_h > 20ms, the current scene is determined to be a high-delay scene. Since the real-time delay is large, the visible light frame and the thermal imaging frame need to be matched and aligned. Therefore, step S3 is entered to perform buffer queue and buffer queue management to obtain the thermal imaging frame.

[0023] S3, buffer queue and buffer queue management, to obtain thermal imaging frames.

[0024] The buffer queues are configured as follows: visible light frames are stored in the L1 video memory queue, and thermal imaging frames are stored in the L2 memory queue. The capacities of the L1 video memory queue and the L2 memory queue are set, and the queue sizes are dynamically adjusted according to the real-time status, performing matching and alignment operations. For example, the capacity of the L1 video memory queue is set to μ+3σ, where μ is the historical average latency and σ is the latency variance, and the size of the L2 memory queue is fixed at 5 frames.

[0025] The cache queue management is as follows: The SDK latency sequence {Δt1, Δt2, ..., Δtn} of the thermal imaging frames is statistically analyzed. The historical latency mean μ and latency variance σ are updated using a dynamic weighted average algorithm (newer data has higher weight, while older data has a gradually weakening influence). The expressions are: μ_new = 0.2 * Δt + 0.8 * μ_old, σ_new = 0.2 * (Δt - μ_new)² + 0.8 * σ_old, where Δt is the latency difference of the current thermal imaging frame, μ_new is the latest historical latency mean, μ_old is the original historical latency mean, σ_new is the latest latency variance, and σ_old is the original latency variance. The L1 memory queue depth is dynamically set to μ_new + 3σ_new, rounded to the nearest integer frame. After the system processes the SDK latency data of the thermal imaging frames, a thermal imaging frame is obtained and stored in the L2 memory queue. Then, step S4 is performed to match the visible light frames and the thermal imaging frames.

[0026] S4. Frame matching of visible light frames and thermal imaging frames.

[0027] When the thermal imaging frame from step S2 is stored in the L2 memory queue after step S3, a binary search is performed in the L1 video memory queue to locate the visible light frame whose timestamp is closest to the thermal imaging frame in the L2 memory queue. The location method involves comparing the difference between the visible light frame timestamp and the thermal imaging frame timestamp with a preset timestamp difference threshold for alignment. If neither exceeds the threshold, the alignment is successful, resulting in a synchronized frame pair. The synchronized frame pair is then output, and the process proceeds to step S5 for cross-modal data alignment. For example, if the preset timestamp difference threshold is 1ms, the location expression is |T_v - T_h|≤1ms, where T_v is the visible light frame timestamp and T_h is the currently arriving thermal imaging frame timestamp. If the expression is satisfied, the alignment is successful, resulting in a synchronized frame pair. The process proceeds to step S5 for cross-modal data alignment of the synchronized frame pair. If the expression is not satisfied, the alignment fails, which is considered an abnormal frame matching state, and the process proceeds to step S6 for abnormal frame matching handling.

[0028] S5, cross-modal data alignment of synchronous frame pairs.

[0029] To further pinpoint the specific location of the detected object in the thermal imaging image (image of the thermal imaging frame) and obtain accurate temperature information, this invention trains and deploys a lightweight YOLOv8 model to detect the target region in the visible light image (image of the visible light frame) in real time. It outputs the predicted coordinates of the target region (i.e., the target bounding box coordinates Bbox=(x1,y1,x2,y2)) and maps them to the thermal imaging image for corresponding region localization. Specifically, it maps the center point (u,v) of the Bbox to the thermal imaging image based on the camera calibration parameters, eliminating the difference in viewing angles between the two light sources and supporting real-time temperature correlation analysis. The ROI region corresponding to the thermal imaging image is extracted, and a bilinear interpolation algorithm (which is based on neighboring pixel weighting and can alleviate image distortion caused by resolution differences) is used to align the resolution of the thermal imaging frame image and the visible light frame image. The process then proceeds to step S7 for synchronization error monitoring and resynchronization mechanisms.

[0030] S6. Handling abnormal states in frame matching.

[0031] Abnormal state management works as follows: When matching and alignment fails, a failure timestamp is recorded, the number of failures is incremented by a local counter, and the failure timestamp is stored in the time-series database of the edge node. A window period is set (e.g., the window period is set to 10 seconds), and the sliding window algorithm is called to calculate the failure rate: failure rate = number of failures within the window / total number of requests. If there are multiple consecutive frames of matching and alignment failures within the window period where the failure rate exceeds the preset failure rate threshold and the number of failures exceeds the preset failure count threshold, it is determined that there is a continuous delay in the thermal imaging frame. Otherwise, it is determined that the current matching and alignment failure is a sudden delay. If it is a sudden delay, the current thermal imaging frame is discarded, an alarm is triggered, an abnormal log is recorded, and a data packet marked as abnormal is output. This data packet contains the latest valid visible light frame, an invalid thermal imaging frame placeholder, and a status code indicating synchronization failure. If it is a continuous delay, it dynamically switches to real-time streaming mode, retrieves the latest visible light frame from the visible light video stream, pushes it in real time, and outputs the data.

[0032] For example: if the failure rate threshold is set to 5% and the failure count threshold is 3 times, if the failure rate is ≤5% and there are ≤3 consecutive failures, the current matching failure is determined to be a sudden delay, which may be caused by occasional interference such as network jitter or temporary hardware overheating. The current thermal imaging frame is discarded, a local alarm is triggered, an abnormal log is recorded to the monitoring system, and a data packet marked as abnormal is output. This data packet contains the latest valid visible light frame, an invalid thermal imaging frame placeholder, and a status code indicating synchronization failure. If the failure rate is >5% or there are >3 consecutive failures, the current matching failure is determined to be a continuous delay, which may be caused by long-term problems such as excessive system load or hardware performance degradation. The system is dynamically switched to real-time streaming mode, the L1 and L2 queue history is cleared, GPU memory resources are released, the addition of new thermal imaging frames to the L2 memory queue is paused, the latest visible light frame is captured from the real-time visible light video stream and pushed directly, and the data is output.

[0033] S7. Synchronization error monitoring and resynchronization mechanism.

[0034] To eliminate coordinate offset issues caused by accumulated internal camera clock drift, synchronization error monitoring is performed. In step S5, the Euclidean distance between the center points of the bounding boxes in the images of the visible light frame and the thermal imaging frame of the synchronized frame pair is calculated. = In the formula, ai represents the center point of the detection target area of ​​the visible light frame image, bi represents the center point of the ROI area of ​​the thermal imaging frame image, records the Euclidean distance spatial offset ΔP of the matching synchronized frame pair, the timestamp difference ΔT of the matching synchronized frame pair, and preset parameter thresholds for ΔP and ΔT. If either ΔP or ΔT is greater than the set parameter threshold (e.g., setting the parameter threshold for ΔP to 10 pixels and the parameter threshold for ΔT to 20ms), if ΔT>20ms or ΔP>10 pixels, a resynchronization calibration strategy is triggered, the L1 video memory queue and L2 memory queue are cleared, GPU / memory resources are released, a hardware reset signal is sent to the visible light camera and thermal imager, an IDR keyframe is forcibly generated as a signal to clear the reference frame buffer, the decoding state is reinitialized, and the master and slave clock synchronization process of the PTP protocol is reinitialized to restore the system to a steady state (e.g., restoring a steady state within 10 seconds). Otherwise, if the parameter thresholds for ΔP and ΔT are not exceeded, the synchronized frame pair data is output.

[0035] The advantages of the above-mentioned technical solution of the present invention compared with the prior art are as follows:

[0036] 1. A dual-modal heterogeneous caching and adaptive latency compensation mechanism is implemented. A hierarchical storage mechanism (L1 GPU memory queue + L2 memory queue) is designed. Based on the EWMA algorithm, the depth of the L1 GPU memory queue is dynamically adjusted to the weighted sum of the historical latency mean μ and standard deviation σ, μ + 3σ. This, combined with an adaptive latency compensation algorithm, dynamically balances the differences in data flow rates between the two modes, avoiding queue overflow and resolving the data rate mismatch problem caused by network fluctuations. Compared to traditional fixed queue schemes, synchronization error is reduced to ±2ms, and resource consumption is reduced by 35%.

[0037] 2. A cross-modal keyframe resynchronization strategy is proposed, using a YOLOv8 target detection and ROI mapping method. This method aligns the resolution difference between thermal imaging and visible light using bilinear interpolation and triggers resynchronization based on Euclidean distance spatial offset verification. This solves the target position drift problem in traditional methods, achieving accurate target region extraction. When a target position offset ΔP > 10 pixels is detected, hardware-level resynchronization is triggered: clearing the buffer queue, forcibly generating IDR keyframes, and reinitializing the PTP clock. The synchronization recovery time is ≤10 seconds, improving robustness by 60%.

[0038] 3. Dynamic anomaly closed-loop management mechanism: The sliding window algorithm is used to count the matching failure rate (window period of 10 seconds) and distinguish between sudden / continuous delay scenarios: Sudden anomaly (failure rate ≤ 5%): Discard the abnormal frame and trigger a local alarm; continuous delay (failure rate > 5%): Switch to real-time mode, capture the latest visible light data for approximate matching, and ensure data continuity.

[0039] 4. Hardware-protocol co-optimization: The PTP protocol is used to achieve sub-microsecond clock synchronization. Combined with the IDR key frame forced refresh and hardware reset mechanism, the software layer synchronization accuracy bottleneck is broken through. The synchronization error can be controlled within ±2ms, which meets the industrial-grade real-time detection requirements.

[0040] In summary, this invention addresses core issues in existing technologies, such as frame loss due to network fluctuations and SDK transmission delays, target position offset, and insufficient synchronization robustness, through a multi-level caching architecture, adaptive latency compensation, and cross-modal keyframe alignment strategies. Compared to traditional frame interpolation and motion compensation schemes, this method optimizes caching strategies and latency modeling to achieve accurate synchronous output of dual-modal data while avoiding interpolation errors and the complexity of motion prediction. It solves core problems in industrial inspection such as image misalignment and target positioning drift caused by spatiotemporal deviations in dual-modal data, achieving an industrial-grade synchronization accuracy of ≤2ms error, providing high reliability for equipment defect diagnosis and high-temperature area monitoring.

[0041] The method steps of the above embodiments of this specific implementation can be adjusted by adding or reducing steps as needed in practical applications.

[0042] The above embodiments and figures are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims

1. A method for synchronizing thermal imaging and visible light data based on dynamic queue management, characterized in that, The data synchronization method includes the following steps: S1: Hardware deployment and clock synchronization initialization Hardware deployment: Deploy visible light cameras and thermal imagers supporting the PTP protocol and connect them to the edge computing node. Visible light frames and thermal image frames are acquired through the visible light cameras and thermal imagers. Clock synchronization initialization: Configure the master and slave clocks of the visible light camera and thermal imager in the PTP protocol, calculate the deviation between the master and slave clocks and calibrate them; S2: Real-time streaming and latency calculation The end-to-end transmission time from acquisition to streaming of each visible light frame and thermal imaging frame is calculated to obtain the visible light frame delay and thermal imaging frame delay. The visible light frame delay and thermal imaging frame delay are judged. When neither of them exceeds the preset delay threshold, the dual-modal data is pushed in real time through the RTMP protocol. When one of them exceeds the preset delay threshold, the process proceeds to step S3. S3, buffer queue and buffer queue management, to obtain thermal imaging frames. The cache queue stores visible light frames in the L1 video memory queue and thermal imaging frames in the L2 memory queue. It is set to dynamically adjust the queue size of the L1 video memory queue and the L2 memory queue. The cache queue management is to count the delay sequence of thermal imaging frames, obtain the queue size of the L1 video memory queue by updating the historical delay mean and delay variance through a dynamic weighted average algorithm, and store the thermal imaging frames in the L2 memory queue, and proceed to step S4. S4. Frame matching of visible light frames and thermal imaging frames. Perform comparison, matching, alignment, and positioning. In the L1 video memory queue, perform a binary search to find the visible light frame whose timestamp is closest to the thermal imaging frame in the L2 memory queue obtained in step S3. If the matching and alignment are successful, a synchronization frame pair is obtained and output. If the matching and alignment fail, it is considered an abnormal state of frame matching. In step S3, the size of the L1 memory queue is set to μ_new + 3σ_new, rounded to the nearest integer frame. The size of the L2 memory queue is set to several frames. The delay sequence of the thermal imaging frame is {Δt1, Δt2, ..., Δt...}. n The dynamic weighted average algorithm is expressed as: μ_new=0.2×Δt+0.8×μ_old, σ_new=0.2× (Δt-μ_new)²+0.8 ×σ_old, where Δt is the delay difference of the current thermal imaging frame, μ_new is the latest historical mean delay of the thermal imaging frame, μ_old is the original historical mean delay of the thermal imaging frame, σ_new is the latest delay variance of the thermal imaging frame, and σ_old is the original delay variance of the thermal imaging frame.

2. The method for synchronizing thermal imaging and visible light data based on dynamic queue management as described in claim 1, characterized in that, It also includes steps S5 and S6; Step S5 involves cross-modal data alignment of the synchronization frame pair. The target region in the visible light image is detected in real time, the predicted coordinates of the target region are obtained and mapped to the thermal imaging image for corresponding region localization, the ROI region of the thermal imaging image is extracted, and the resolution of the thermal imaging frame image and the visible light frame image is aligned using a bilinear interpolation algorithm. If the matching and alignment are successful in step S4, the synchronization frame pair is obtained and output to step S5. Step S6 is for handling abnormal frame matching states. When matching and alignment fails, the failure timestamp is recorded, the number of failures is accumulated by the local counter, and the failure timestamp is stored in the time-series database of the edge node. The window period is set, and the sliding window algorithm is called to calculate the failure rate. The failure rate = number of failures within the window / total number of requests. If there are multiple consecutive frame matching and alignment failures within the window period where the failure rate exceeds the preset failure rate threshold and the number of failures exceeds the preset failure count threshold, it is determined that there is a continuous delay in the current thermal imaging frame. Otherwise, it is determined that the current matching and alignment failure is a sudden delay. If it is a sudden delay, the current thermal imaging frame is discarded, an alarm is triggered, and abnormal data packets are output. If it is a continuous delay, it is dynamically switched to the real-time streaming mode, and the latest visible light frame is retrieved from the visible light video stream for real-time streaming. When step S4 is considered an abnormal frame matching state, step S6 is entered.

3. The method for synchronizing thermal imaging and visible light data based on dynamic queue management as described in claim 2, characterized in that, It also includes step S7; step S7 is a synchronization error monitoring and resynchronization mechanism, which matches the Euclidean distance between the center points of the Bbox in the images of the visible light frame and the thermal imaging frame of the synchronization frame pair, calculates the Euclidean distance spatial offset ΔP and the timestamp difference ΔT of the matching synchronization frame pair, presets parameter thresholds for ΔP and ΔT, if either ΔP or ΔT is greater than the set parameter threshold, triggers the resynchronization calibration strategy, clears the L1 video memory queue and the L2 memory queue, sends a hardware reset signal to the visible light camera and the thermal imager, and reinitializes the master and slave clock synchronization process of the PTP protocol, otherwise, if the parameter thresholds for ΔP and ΔT are not exceeded, the synchronization frame pair data is output; after step S5 is completed, step S7 is entered.

4. A method for synchronizing thermal imaging and visible light data based on dynamic queue management as described in claim 1, 2, or 3, characterized in that, The method for calculating the discrepancy between the master and slave clocks in step S1 is to have the visible light camera send a message and record the sending timestamp; The thermal imager receives the message and records the receiving timestamp. The thermal imager then sends the message and the receiving timestamp back to the visible light camera. The master clock extracts the receiving timestamp and calculates the link delay, which is the time difference between the receiving timestamp and the sending timestamp.

Images