Thermal imaging and visible light data synchronization method based on dynamic queue management

Abstract

The invention relates to the technical field of data synchronization of inspection robots, and particularly discloses a thermal imaging and visible light data synchronization method based on dynamic queue management, which comprises the following steps: S1, hardware deployment and clock synchronization initialization: acquiring a visible light frame and a thermal imaging frame through a visible light camera and a thermal imager, and calculating and calibrating the deviation of a master clock and a slave clock; s2, real-time flow pushing and delay calculation are carried out, visible light frame delay and thermal imaging frame delay are obtained through calculation, and the conditions of the visible light frame delay and the thermal imaging frame delay are judged; s3, caching a queue and performing cache queue management to obtain a thermal imaging frame; and S4, carrying out frame matching on the visible light frame and the thermal imaging frame, carrying out comparison, matching, alignment and positioning, obtaining a synchronous frame pair for output if matching and alignment are successful, and regarding as an abnormal state if matching and alignment are failed. Compared with the prior art, the method has the advantages that the synchronization precision error time is obviously shortened, and the problems of picture dislocation, target positioning drift and the like caused by bimodal data space-time deviation in industrial inspection can be effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data processing, in particular to the technical field of image data synchronous processing of inspection robots. BACKGROUND

[0002] An inspection robot applied in industry carries a visible light camera and a thermal imager to realize all-round monitoring of the state of industrial equipment. The visible light camera can capture visible features such as equipment appearance deformation and liquid leakage, while the thermal imager can penetrate smoke and identify local high temperature and other thermodynamic abnormalities. However, due to hardware acquisition mechanisms (mainly referring to the frame rate difference between CMOS and infrared sensors), data transmission bandwidth (generally, the amount of thermal imaging data is usually 3-5 times that of visible light), and network fluctuations, the two types of sensors cause delay deviation (commonly millisecond-level delay deviation) in dual-mode data, resulting in target spatial position deviation, temperature field and visual feature misplacement, and other problems, which seriously affect the accuracy of defect diagnosis. For example, when the inspection robot performs tasks in a complex industrial environment, due to the inherent delay difference between the visible light camera and the thermal imager in hardware acquisition, data transmission, and other links, it may cause serious spatio-temporal deviation of dual-mode data, and thus cause the problem of asynchronous pushing of pictures. This asynchrony directly affects the accuracy of defect detection, real-time monitoring response efficiency, and the reliability of multi-modal data fusion analysis in industrial scenarios.

[0003] For the above problems, the existing technology mainly realizes data alignment through hardware-level synchronization and software algorithm compensation, such as a multi-camera hardware synchronization method and system based on a microcontroller disclosed in Chinese Patent Application No. CN202411772405, which connects multiple cameras through a synchronization interface to coordinate the acquisition timing with a master-slave clock mechanism. The master camera triggers a Sync pulse signal, and the slave camera adjusts its shutter period after receiving the signal to ensure strict synchronization at the physical level. This method relies on hardware circuit design, requires accurate wiring, and has limited scalability. It is prone to signal loss under complex electromagnetic interference in industrial sites, and cannot address software layer problems such as network transmission delay. For another example, a multi-camera time synchronization method, device, equipment, and storage medium disclosed in Chinese Patent Application No. CN202311370407.9 use the periodic dark lines generated by screen PWM dimming as a time reference. The reference camera captures the dark line position, and other cameras calculate the time difference by comparing the dark line offset to adjust the shooting timing. This method requires the cooperation of specific display devices, and the calibration process is tedious. In dynamic industrial scenarios, changes in device screen state can cause the synchronization reference to fail, and the method lacks universality.

[0004] In summary, the prior art has three major core defects: 1. Poor synchronization robustness, hardware-based synchronization methods (such as PTP clock protocol) rely on physical line connection, which is easily disturbed by vibration in the industrial robot moving scene, and a single point failure will cause the system to collapse; software calibration-based methods (such as PWM dark line detection) are affected by environmental lighting and device state changes, making it difficult to maintain long-term stable synchronization. 2. Insufficient dynamic adaptability, traditional interpolation compensation algorithms require motion trajectory prediction models, which produce interpolation errors when the robot quickly turns or the device shakes; traditional queue-based schemes do not consider the rate difference of multiple data sources, and the risk of queue overflow is high. 3. Large resource consumption, existing methods generally use CPU serial processing synchronization logic, which cannot meet the real-time requirements of industrial level, and multi-thread concurrent processing can easily cause resource competition. SUMMARY

[0005] The purpose of the present application is to provide a kind of dynamic queue management-based thermal imaging and visible light data synchronization method, which can effectively solve the core problems of picture dislocation, target positioning drift and other core problems caused by the space-time deviation of double-mode data in industrial inspection, with significantly shorter synchronization accuracy error time compared to prior art.

[0006] To achieve the above object, the technical scheme of the present application is: a thermal imaging and visible light data synchronization method based on dynamic queue management, the data synchronization method steps comprising the following: S1, hardware deployment and clock synchronization initialization, the hardware deployment comprising deploying a visible light camera and a thermal imager supporting PTP protocol connected to an edge computing node, obtaining visible light frames and thermal imaging frames through the visible light camera and the thermal imager, the clock synchronization initialization comprising configuring the master and slave clocks of the visible light camera and the thermal imager in the PTP protocol, calculating the deviation of the master and slave clocks and calibrating; S2, real-time streaming and delay calculation, calculating the end-to-end transmission time consumption of each frame of visible light frames and thermal imaging frames from collection to streaming to obtain visible light frame delay and thermal imaging frame delay, judging the visible light frame delay and thermal imaging frame delay, when both do not exceed the pre-set delay threshold, real-time streaming of dual-mode data through the RTMP protocol, when one of them exceeds the pre-set delay threshold, entering step S3; S3, cache queue and cache queue management, obtaining thermal imaging frames, the cache queue is to store visible light frames into L1 video memory queue and store thermal imaging frames into L2 memory queue, setting the queue size of L1 video memory queue and L2 memory queue for dynamic adjustment, the cache queue management is to count the delay sequence of thermal imaging frames, obtaining the thermal imaging frames stored in L2 memory queue through the historical delay mean and delay variance updated by dynamic weighted average algorithm, entering step S4; S4, frame matching of visible light frames and thermal imaging frames, comparing and matching alignment positioning, performing binary search in L1 video memory queue and the closest visible light frame of the thermal imaging frame timestamp in L2 memory queue obtained in step S3, if the matching alignment is successful, obtaining a synchronous frame pair, outputting the synchronous frame pair, if the matching alignment fails, it is considered as an abnormal state of frame matching.

[0007] Furthermore, it also includes steps S5 and / or S6. If the matching alignment is successful in step S4 to obtain a synchronized frame pair, the synchronized frame pair is output and the step S5 is entered. The step S5 is cross-modal data alignment of the synchronized frame pair, real-time detection of the target area in the visible light image, obtaining the predicted coordinates of the target area and mapping the corresponding area positioning in the thermal imaging image, intercepting the ROI area of ​​the thermal imaging image, and using a bilinear interpolation algorithm to align the resolution of the thermal imaging frame image and the visible light frame image; when the step S4 is considered to be an abnormal state of frame matching, the step S6 is entered. The step S6 is the abnormal state processing of frame matching. When the matching alignment fails, the failure timestamp is recorded, and the local counter accumulates The number of failures is added and the failure timestamp is stored in the time series database of the edge node. The window period is set and the sliding window algorithm is called to calculate the failure rate. The failure rate = the number of failures in the window / the total number of requests. If there are consecutive multi-frame matching and alignment failures with a failure rate exceeding the preset failure rate threshold and the preset failure number threshold within the window period, it is determined that there is a continuous delay in the thermal imaging frame. Otherwise, it is determined that this matching and alignment failure is a burst delay. If it is a burst delay, the current thermal imaging frame is discarded, an alarm is triggered, and the abnormal data packet is output. If it is a continuous delay, it is dynamically switched to real-time streaming mode, and the latest visible light frame is taken from the visible light video stream for real-time streaming and output.

[0008] Furthermore, step S7 is included. After step S5 is completed, step S7 is entered. Step S7 is a synchronization error monitoring and resynchronization mechanism. The Euclidean distance of the center point of the Bbox in the image of the visible light frame and the thermal imaging frame of the synchronization frame pair is matched, and the Euclidean distance space offset ΔP and the timestamp difference ΔT of the synchronization frame pair are calculated. The parameter thresholds of ΔP and ΔT are preset. If one of ΔP or ΔT is greater than the set parameter threshold, the resynchronization calibration strategy is triggered, the L1 video memory queue and the L2 memory queue are cleared, and a hardware reset signal is sent to the visible light camera and the thermal imager. The master and slave clock synchronization process of the PTP protocol is reinitialized. Otherwise, if the parameter thresholds of ΔP and ΔT are not exceeded, the synchronization frame pair data is output.

[0009] Furthermore, the method for calculating the deviation between the master and slave clocks in step S1 is that the visible light camera sends a message and records the sending timestamp; the thermal imager receives the message and records the receiving timestamp, the thermal imager returns the message and the receiving timestamp to the visible light camera, and the master clock extracts the receiving timestamp and calculates the link delay, that is, the time difference between the receiving timestamp and the sending timestamp.

[0010] Further, the queue size setting method of the L1 video memory queue in the step S3 is μ_new+3σ_new, rounded to an integer frame, and the queue size setting of the L2 memory queue is a number of frames; the delay sequence of the thermal imaging frame is {Δt1, Δt2,..., Δtn}, and the expression of the dynamic weighted average algorithm is: μ_new=0.2*Δt+0.8*μ_old, σ_new=0.2*(Δt-μ_new)²+0.8*σ_old, wherein Δt is a delay difference value of the current thermal imaging frame, μ_new is the latest historical delay average, μ_old is the original historical delay average, σ_new is the latest delay variance, and σ_old is the original delay variance.

[0011] By adopting the technical scheme, the application has the beneficial effects that: compared with the hardware-level synchronization method based on the fixed clock period trigger in the multi-camera hard synchronization method based on the microcontroller, the application reduces the synchronization error in the network fluctuation scene through the hierarchical cache architecture and real-time delay modeling, and avoids the motion prediction error caused by the frame insertion compensation. Secondly, compared with the synchronization scheme based on the dark line reference in the multi-camera time synchronization method based on the screen PWM dimming, the application solves the dark line failure problem in the dynamic industrial scene through the cross-modal key frame resynchronization strategy, further combines the IDR key frame forced refresh and the PTP clock protocol cooperative optimization, and improves the synchronization robustness. Finally, compared with the coarse-grained coordinate calibration method in the traditional soft and hard hybrid synchronization scheme, the application greatly improves the target positioning accuracy, significantly reduces the spatial offset between the cameras, and significantly improves the reliability of the industrial defect diagnosis. Through the multi-dimensional technical innovation fusion, the application realizes the high-precision, low-jitter real-time synchronization of the dual-mode data in the complex industrial environment. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a flowchart of a thermal imaging and visible light data synchronization method based on dynamic queue management. DETAILED DESCRIPTION

[0013] In order to further explain the technical scheme of the application, the application will be described in detail below through specific embodiments.

[0014] The thermal imaging and visible light data synchronization method based on dynamic queue management disclosed in this embodiment is described in detail below in combination with the drawings.

[0015] S1: hardware deployment and clock synchronization initialization.

[0016] Hardware deployment: including fixing visible light camera and thermal imager supporting PTP protocol on the top of the inspection robot, both of which can be connected to the edge computing node through the industrial switch. When the system is working, the visible light camera and the thermal imager collect visible light video stream and thermal imaging video stream during the movement, and the visible light video stream is separated by frame to obtain visible light frame, and the thermal imaging video stream is separated by frame to obtain thermal imaging frame.

[0017] Hardware initialization: hardware initialization includes configuring PTP master and slave clock. In the PTP protocol, the visible light camera is set as the master clock as the master device, and the thermal imager is set as the slave clock as the slave device. The visible light camera sends a message (herein referred to as Sync message) and records the sending timestamp t1; the thermal imager receives the message (i.e. Sync message) and records the receiving timestamp t2, and the thermal imager transmits a message (herein referred to as Delay_Req message) and the receiving timestamp t2 back to the visible light camera, i.e. to the master clock. The master clock extracts the receiving timestamp t2 and calculates the link delay, i.e. the time difference between the receiving timestamp t2 and the sending timestamp t1, expressed as Δ=t2-t1, and calibrates the deviation of the slave clock, so that the synchronization error of the global time is reduced. Preferably, it is desirable to ensure that the global time synchronization error is ≤1μs, thereby completing the synchronization initialization of the master and slave clocks.

[0018] S2: real-time push stream and delay calculation.

[0019] When the system is working, the SDK delay of the visible light frame of the visible light video stream and the SDK delay of the thermal imaging frame of the thermal imaging video stream are calculated respectively, which are the end-to-end transmission time from collection to push stream of each frame in the visible light frame and the thermal imaging frame. The actual delay size of the scene is judged, and different push stream modes are switched according to the judgment result.

[0020] The SDK delay of the visible light video stream is denoted as Δt_v, and the expression is Δt_v= current system time- visible light frame timestamp.

[0021] The SDK delay of the thermal imaging video stream is denoted as Δt_h, and the expression is Δt_h= current system time- thermal imaging frame timestamp.

[0022] The judgment of the actual delay size of the scene is as follows: when the SDK delay of the visible light frame and the SDK delay of the thermal imaging frame do not exceed the pre-set delay threshold, for example, the delay threshold of the SDK delay of the visible light frame is set to 10 ms, and the delay threshold of the SDK delay of the thermal imaging video stream is set to 20 ms, that is, when Δt_v ≤ 10 ms and Δt_h ≤ 20 ms, it is determined that the current scene is a small delay scene. Since the real-time delay is small, the delay between each frame of the visible light frame and the thermal imaging frame can be ignored, so the dual-mode data can be directly pushed in real time through the RTMP protocol, and the output data, that is, the real-time visible light frame and thermal imaging frame data, is output. When the SDK delay of the visible light frame and the SDK delay of the thermal imaging frame exceed the pre-set delay threshold, that is, when Δt_v > 10 ms or Δt_h > 20 ms, it is determined that the current scene is a large delay scene. Since the real-time delay is large, the visible light frame and the thermal imaging frame need to be matched and aligned, so the step S3 of the cache queue and the cache queue management is entered to obtain the thermal imaging frame.

[0023] S3, cache queue and cache queue management, obtain the thermal imaging frame.

[0024] The cache queue is as follows: the visible light frame is stored in the L1 video memory queue, and the thermal imaging frame is stored in the L2 memory queue. The L1 video memory queue capacity and the L2 memory queue size are set, and the queue size is dynamically adjusted according to the real-time state, and the matching and alignment operation is performed. For example, the L1 video memory queue capacity is set to μ+3σ, μ is the historical delay mean, and σ is the delay variance. The L2 memory queue size is set to 5 frames.

[0025] The cache queue management is as follows: the SDK delay sequence {Δt1, Δt2,..., Δtn} of the thermal imaging frame is counted, the historical delay mean μ and the delay variance σ updated by the dynamic weighted average algorithm (the weight of the newer data is higher, and the influence of the old data gradually decreases) are updated, and the expression is: μ_new=0.2 *Δt+0.8*μ_old, σ_new=0.2* (Δt-μ_new)²+0.8 *σ_old, in the formula, Δt is the delay difference value of the current thermal imaging frame, μ_new is the latest historical delay mean, μ_old is the original historical delay mean, σ_new is the latest delay variance, and σ_old is the original delay variance. The L1 video memory queue depth is dynamically set to μ_new+3σ_new, rounded to an integer frame. When the system processes the SDK delay data of the thermal imaging frame, a thermal imaging frame is obtained and stored in the L2 memory queue, and then the step S4 of the frame matching of the visible light frame and the thermal imaging frame is entered.

[0026] S4, frame matching of the visible light frame and the thermal imaging frame.

[0027] When the thermal imaging frame of step S2 is stored in the L2 memory queue after step S3 is completed, a binary search is performed in the L1 display memory queue to locate the visible light frame closest to the time stamp of the thermal imaging frame in the L2 memory queue. The locating method is to compare and match the difference between the time stamp of the visible light frame and the time stamp of the thermal imaging frame with the pre-set time stamp difference threshold. If neither of them exceeds the time stamp difference threshold, it means that the matching and alignment are successful to obtain a synchronous frame pair. The synchronous frame pair is output, and step S5 of cross-modal data alignment is entered. For example, the pre-set time stamp difference threshold is 1 ms, and the locating expression is |T_v - T_h|≤1 ms, where T_v is the time stamp of the visible light frame, and T_h is the time stamp of the thermal imaging frame. If the locating expression is met, the matching and alignment are successful to obtain a synchronous frame pair. The synchronous frame pair is output, and step S5 of cross-modal data alignment of the synchronous frame pair is entered. If the locating expression is not met, the matching and alignment fail, which is regarded as an abnormal state of frame matching. Step S6 of abnormal state processing of frame matching is entered.

[0028] S5, cross-modal data alignment of the synchronous frame pair.

[0029] In order to further locate the specific position of the detection object in the thermal imaging image (image of the thermal imaging frame) and obtain accurate temperature information, the present application trains and deploys a lightweight YOLOv8 model to detect the target region in the visible light image (image of the visible light frame) in real time, outputs the target region prediction coordinates (i.e. target bounding box coordinates Bbox=(x1,y1,x2,y2)) and maps them to the thermal imaging image for corresponding region positioning, i.e. mapping the center point (u,v) of Bbox to the thermal imaging image according to the camera calibration parameters, eliminating the difference in the imaging angle of view of the dual light, supporting real-time temperature correlation analysis; cutting the ROI region corresponding to the thermal imaging image, using a bilinear interpolation algorithm (which is a weighted calculation based on adjacent pixels, which can alleviate the image distortion caused by the resolution difference) to align the resolutions of the thermal imaging frame image and the visible light frame image, and entering step S7 for synchronous error monitoring and re-synchronization mechanism.

[0030] S6, abnormal state processing of frame matching.

[0031] The abnormal state management is as follows: when the matching alignment fails, the failure timestamp is recorded, the local counter accumulates the failure number, and the failure timestamp is stored in the edge node time sequence database; a window period (for example, the window period is set to 10 seconds) is set, and a sliding window algorithm is called to calculate the failure rate: failure rate = failure number in the window / total request number; if there is a continuous multiple frame matching alignment failure condition with a failure rate exceeding a pre-set failure rate threshold and exceeding a pre-set failure number threshold in the window period, it is judged that there is a continuous delay of the thermal imaging frame, otherwise it is judged that the matching alignment failure is a burst delay; if it is a burst delay, the current thermal imaging frame is discarded, an alarm is triggered, an abnormal log is recorded, and an abnormal data packet is output, which includes the latest valid visible light frame, an invalid thermal imaging frame placeholder and a state code indicating synchronization failure; if it is a continuous delay, the real-time push mode is dynamically switched to, the latest visible light frame is taken from the visible light video stream for real-time push, and data is output.

[0032] For example: the failure rate threshold is set to 5%, the failure number threshold is 3 times, if the failure rate is ≤5% and the continuous failure is ≤3 times, it is judged that the matching failure is a burst delay, the cause may be network jitter / temporary hardware overheating and other incidental interference, the current thermal imaging frame is discarded, a local alarm is triggered, an abnormal log is recorded to a monitoring system, and an abnormal data packet is output, which includes the latest valid visible light frame, an invalid thermal imaging frame placeholder and a state code indicating synchronization failure; if the failure rate is >5% or the continuous failure is >3 times, it is judged that the matching failure is a continuous delay, the cause may be high system load / long-term hardware performance degradation, the real-time push mode is dynamically switched to, the L1 and L2 queue history records are emptied, the GPU memory resources are released, and the new thermal imaging frame is added to the L2 memory queue, the latest visible light frame is grabbed from the real-time visible light video stream for direct push, and data is output.

[0033] S7, synchronization error monitoring and resynchronization mechanism.

[0034] In order to eliminate the problems such as coordinate offset caused by clock drift accumulation in the camera, the synchronization error is monitored. In step S5, the Euclidean distance of the Bbox center point in the images of the visible light frame and the thermal imaging frame of the matching synchronization frame pair = Where ai represents the center point of the detection target area of ​​the visible light frame image, bi represents the center point of the ROI area of ​​the thermal imaging frame image, the Euclidean distance space offset ΔP of the matching synchronous frame pair is recorded, the timestamp difference ΔT of the matching synchronous frame pair is recorded, and the parameter thresholds of ΔP and ΔT are preset. If either ΔP or ΔT is greater than the set parameter threshold, such as setting the parameter threshold of ΔP to 10 pixels and the parameter threshold of ΔT to 20ms, if ΔT>20ms or ΔP>10 pixels, the resynchronization calibration strategy is triggered, the L1 video memory queue and the L2 memory queue are cleared, GPU / memory resources are released, and a hardware reset signal is sent to the visible light camera and thermal imager. The IDR key frame is forced to be generated as a signal to clear the reference frame buffer, reinitialize the decoding state, and reinitialize the master and slave clock synchronization process of the PTP protocol to restore the system to a steady state (for example, the steady state is restored within 10 seconds). Otherwise, if the parameter thresholds of ΔP and ΔT are not exceeded, the synchronous frame pair data is output.

[0035] The advantages of the above technical solution of the present invention compared with the prior art are as follows:

[0036] 1. Dual-mode heterogeneous caching and an adaptive delay compensation mechanism. This design utilizes a hierarchical cache architecture (L1 memory queue + L2 memory queue). Based on the EWMA algorithm, the L1 memory queue depth is dynamically adjusted to the weighted sum of the historical latency mean μ and the standard deviation σ (μ + 3σ). This, combined with the adaptive delay compensation algorithm, dynamically balances the rate differences between the two modal data flows, avoiding queue overflows and resolving data rate mismatches caused by network fluctuations. Compared to traditional fixed queue solutions, this reduces synchronization error to ±2ms, reducing resource usage by 35%.

[0037] 2. A cross-modal keyframe resynchronization strategy proposes a YOLOv8 target detection and ROI mapping method. This method uses bilinear interpolation to align the resolution difference between thermal and visible light imaging, and triggers resynchronization based on spatial offset verification using Euclidean distance. This addresses the target position drift problem inherent in traditional methods and enables precise target region extraction. When a target position offset ΔP > 10 pixels is detected, hardware-level resynchronization is triggered: the cache queue is cleared, an IDR keyframe is forcibly generated, and the PTP clock is reinitialized. Synchronization recovery time is ≤ 10 seconds, improving robustness by 60%.

[0038] 3. A dynamic anomaly closed-loop management mechanism uses a sliding window algorithm to calculate the matching failure rate (window period: 10 seconds) and distinguish between sudden and continuous delay scenarios: Sudden anomalies (failure rate ≤ 5%): abnormal frames are discarded and local alarms are triggered; continuous delays (failure rate > 5%): the system switches to real-time mode, captures the latest visible light data for approximate matching, and ensures data continuity.

[0039] 4. Hardware-protocol collaborative optimization, sub-microsecond clock synchronization is realized by adopting PTP protocol, combined with IDR key frame forced refresh and hardware reset mechanism, breaking through the synchronization accuracy bottleneck of the software layer, the synchronization error can be controlled within ±2ms, meeting the industrial real-time detection requirements.

[0040] In summary, the version of the application solves the core problems of frame loss, target position offset and insufficient synchronization robustness caused by network fluctuations and transmission delay of calling SDK in the prior art through multi-level cache architecture, adaptive delay compensation and cross-modal key frame alignment strategy. Compared with the traditional interpolation and motion compensation scheme, this method optimizes the cache strategy and delay modeling to achieve accurate synchronization output of dual-mode data while avoiding interpolation error and motion prediction complexity. It solves the core problems of picture misplacement and target positioning drift caused by the spatio-temporal deviation of dual-mode data in industrial inspection, and the synchronization accuracy reaches the industrial standard of error ≤2ms, providing high reliability for equipment defect diagnosis and high-temperature area monitoring.

[0041] The method steps of the above embodiments of the present specific embodiment can be adjusted by increasing, reducing, etc. steps according to needs in actual application.

[0042] The above embodiments and drawings do not limit the product form and style of the present application, and any appropriate changes or modifications made by those skilled in the art to the present application shall be considered as not departing from the scope of the present application.

Claims

1. A method for synchronizing thermal imaging and visible light data based on dynamic queue management, characterized in that: The data synchronization method steps include the following: S1: Hardware deployment and clock synchronization initialization, Hardware deployment: Deploy visible light cameras and thermal imagers that support the PTP protocol and connect them to edge computing nodes. Visible light frames and thermal imaging frames are collected by the visible light cameras and thermal imagers. Clock synchronization initialization: Configure the master and slave clocks of the visible light camera and thermal imager in the PTP protocol, calculate the deviation between the master and slave clocks, and calibrate them; S2: Real-time streaming and delay calculation, The end-to-end transmission time of each visible light frame and thermal imaging frame from acquisition to streaming is calculated to obtain the visible light frame delay and thermal imaging frame delay. The visible light frame delay and thermal imaging frame delay are determined. If both delays do not exceed a preset delay threshold, the dual-modal data is streamed in real time via the RTMP protocol. If either delay exceeds the preset delay threshold, the process proceeds to step S3. S3, cache queue and cache queue management, get thermal imaging frame, The cache queue stores visible light frames in the L1 memory queue and thermal imaging frames in the L2 memory queue. The queue sizes of the L1 memory queue and the L2 memory queue are dynamically adjusted. The cache queue management is to calculate the delay sequence of the thermal imaging frames, and obtain the thermal imaging frames by updating the historical delay mean and delay variance through a dynamic weighted average algorithm and storing them in the L2 memory queue, and then proceeding to step S4. S4, frame matching of visible light frame and thermal imaging frame, Perform matching and alignment. Perform a binary search in the L1 memory queue to find the visible light frame with the closest timestamp to the thermal imaging frame in the L2 memory queue obtained in step S3. If the matching and alignment are successful, a synchronized frame pair is obtained and output. If the matching and alignment fail, it is considered an abnormal state of frame matching.

2. The method for synchronizing thermal imaging and visible light data based on dynamic queue management according to claim 1, characterized in that: Also includes steps S5 and / or S6; Step S5 is cross-modal data alignment of the synchronized frame pair, detecting the target area in the visible light image in real time, obtaining the predicted coordinates of the target area and mapping them into the thermal image to locate the corresponding area, intercepting the ROI area of ​​the thermal image, and using a bilinear interpolation algorithm to align the resolutions of the thermal imaging frame image and the visible light frame image. If the matching and alignment are successful in step S4, a synchronized frame pair is obtained, and the synchronized frame pair is output and the process proceeds to step S5. The step S6 is for handling the abnormal state of frame matching. When the matching alignment fails, the failure timestamp is recorded, the local counter accumulates the number of failures, and the failure timestamp is stored in the time series database of the edge node. The window period is set, and the sliding window algorithm is called to calculate the failure rate. The failure rate = the number of failures in the window / the total number of requests. If there are consecutive multi-frame matching alignment failures with a failure rate exceeding a preset failure rate threshold and a preset failure number threshold within the window period, it is determined that there is a continuous delay in the thermal imaging frame. Otherwise, it is determined that this matching alignment failure is a burst delay. If it is a burst delay, the current thermal imaging frame is discarded, an alarm is triggered, and an abnormal data packet is output. If it is a continuous delay, it is dynamically switched to the real-time streaming mode, and the latest visible light frame is taken from the visible light video stream for real-time streaming. When the output data is considered as an abnormal state of frame matching in step S4, step S6 is entered.

3. The method for synchronizing thermal imaging and visible light data based on dynamic queue management according to claim 2, wherein: The method further includes step S7; step S7 is a synchronization error monitoring and resynchronization mechanism, which matches the Euclidean distance of the center point of the Bbox in the image of the visible light frame and the thermal imaging frame of the synchronization frame pair, calculates the Euclidean distance space offset ΔP and the timestamp difference ΔT of the synchronization frame pair, and pre-sets the parameter thresholds of ΔP and ΔT. If one of ΔP or ΔT is greater than the set parameter threshold, the resynchronization calibration strategy is triggered, the L1 video memory queue and the L2 memory queue are cleared, a hardware reset signal is sent to the visible light camera and the thermal imager, and the master and slave clock synchronization process of the PTP protocol is reinitialized. Otherwise, if the parameter thresholds of ΔP and ΔT are not exceeded, the synchronization frame pair data is output; after step S5 is completed, step S7 is entered.

4. A method for synchronizing thermal imaging and visible light data based on dynamic queue management according to claim 1, 2 or 3, characterized in that: The method for calculating the deviation between the master and slave clocks in step S1 is that the visible light camera sends a message and records the sending timestamp; The thermal imager receives the message and records the receiving timestamp. The thermal imager transmits the message and the receiving timestamp back to the visible light camera. The master clock extracts the receiving timestamp and calculates the link delay, which is the time difference between the receiving timestamp and the sending timestamp. And / or, in step S3, the queue size of the L1 video memory queue is set as μ_new+3σ_new, rounded to an integer frame, and the L2 memory queue receiving queue size is set to a number of frames; the delay sequence of the thermal imaging frame {Δt1, Δt2,..., Δtn}, the dynamic weighted average algorithm expression is: μ_new=0.2*Δt+0.8*μ_old, σ_new=0.2*(Δt-μ_new)²+0.8*σ_old, where Δt is the delay difference of the current thermal imaging frame, μ_new is the latest historical delay mean, μ_old is the original historical delay mean, σ_new is the latest delay variance, and σ_old is the original delay variance.

Images