A real-time detection and matching method for camera images and timestamps

Abstract

This invention relates to the field of computer vision and provides a real-time detection and matching method for camera images and timestamps. The method includes: inputting a camera signal into a clock synchronization system and parsing it to obtain timestamp information; performing a difference calculation between the received time of the timestamp information and the received time of the image frame to obtain a time difference; inputting the time difference into a sliding window statistical model to obtain a set of statistical parameters within the current window; performing dynamic threshold matching judgment based on the time difference and the set of statistical parameters to obtain a matching result; and performing anomaly classification and compensation processing on the image frame based on the matching result to obtain a fused image frame. This invention reduces the difficulty of data acquisition and processing for high frame rate cameras.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method for real-time detection and matching of camera images and timestamps. Background Technology

[0002] In fields such as autonomous driving, industrial inspection, and multi-sensor fusion, precise time alignment between camera images and data from other sensors is fundamental to reliable perception and decision-making. Ordinary UVC cameras or IP cameras without time synchronization capabilities lack a high-precision clock synchronization system and cannot directly output image data carrying accurate exposure time information.

[0003] Existing solutions typically fall into two categories: one relies on the system time of the acquisition host to timestamp the reception time of each frame of image. This solution requires no additional hardware, but due to the uncontrollable pipeline delay of image data from exposure to transmission to the host, the timestamp error is usually between 5ms and 50ms, and this error is difficult to measure and compensate for accurately. The other solution connects the camera's trigger signal or exposure return signal to an external high-precision clock synchronization system, and the exposure time is directly marked by hardware. The timestamp accuracy can reach microseconds or even nanoseconds. However, in this solution, the timestamp data and image data are stored independently and need to be aligned offline frame by frame after data acquisition is completed.

[0004] However, the above-mentioned solutions have obvious shortcomings in practical applications: when the camera frame rate is high or the transmission link is jittery, the timestamp sequence and the image frame sequence are prone to inconsistencies in quantity, which means that the offline alignment process requires manual frame-by-frame verification, resulting in low processing efficiency; especially when frame loss occurs, it is impossible to automatically distinguish between hardware frame loss and software frame loss, making fault location difficult and data availability significantly reduced.

[0005] Therefore, how to achieve real-time automatic matching of image frames and high-precision timestamps while ensuring hardware-level accuracy of timestamps, and how to automatically detect and recover from matching anomalies, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] In view of the shortcomings of the prior art, the purpose of the invention is to provide a real-time detection and matching method for camera images and timestamps.

[0007] This invention provides a real-time detection and matching method for camera images and timestamps, comprising: S1. Input the camera signal into the clock synchronization system and parse it to obtain the timestamp information; S2. Perform a difference calculation between the time stamp information received and the time frame received to obtain the time difference; S3. Input the time difference into the sliding window statistical model to obtain the statistical parameter group in the current window; S4. Combine the time difference with the statistical parameter group to perform dynamic threshold matching and determination to obtain the matching result; S5. Perform anomaly classification and compensation processing on the image frame based on the matching result to obtain the fused image frame.

[0008] According to the real-time detection and matching method of camera image and timestamp provided by the present invention, step S1 further includes: S11. The camera signal, including the exposure trigger signal and the exposure return signal, is connected to the clock synchronization system. The clock synchronization system samples the arrival time of the camera signal to obtain the original trigger event record. S12. Parse the original trigger event record according to the camera number field, frame sequence number field and exposure time field to obtain a structured timestamp; S13. Record the structured timestamp together with the local time of the system receiving the corresponding timestamp to obtain timestamp information with the first local receiving time.

[0009] According to the real-time detection and matching method of camera image and timestamp provided by the present invention, step S2 further includes: S21. When the camera image frame arrives at the acquisition system, record the second local reception time of the current image frame; S22. Perform an absolute difference operation between the second local reception time and the first local reception time to obtain the time difference of the current frame.

[0010] According to the real-time detection and matching method of camera image and timestamp provided by the present invention, step S3 further includes: S31. Maintain an initial sliding window queue that is first-in-first-out. Push the calculated time difference into the initial sliding window queue. When the queue is full, eliminate the earliest sample that entered the queue to obtain an updated sliding window queue. S32. The incremental calculation method is used to calculate the mean and standard deviation of each time window for all time difference samples in the updated sliding window queue, and the statistical parameter set is obtained.

[0011] According to the real-time detection and matching method of camera images and timestamps provided by the present invention, in step S31, the capacity of the sliding window queue is 30 frames.

[0012] According to the real-time detection and matching method of camera image and timestamp provided by the present invention, step S4 further includes: S41. Based on the mean and standard deviation in the statistical parameter group, construct a dynamic confidence interval to obtain the matching threshold interval for the current frame; S42. Compare the time difference with the matching threshold interval. If the time difference falls within the matching threshold interval, the match is considered successful; otherwise, the match is considered unsuccessful, and the matching result is obtained.

[0013] According to the real-time detection and matching method of camera images and timestamps provided by the present invention, in step S5, the anomaly classification processing includes the following steps: S51. Push the image frame whose matching result is a failed match and its corresponding timestamp into the abnormal cache queue to obtain the current abnormal cache status. S52. Continuously match and determine the timestamps and image frames that arrive later. When a matching success event occurs again, compare the number of abnormal timestamp frames and the number of abnormal image frames in the abnormal cache queue to obtain the abnormal type determination result. S53. Based on the anomaly type determination result, perform recovery processing on the anomaly timestamps and anomaly image frames in the anomaly cache queue to obtain anomaly-processed labeled frame data.

[0014] According to the real-time detection and matching method of camera image and timestamp provided by the present invention, step S51 further includes: S511. When the matching result is a failure, push the current image frame into the abnormal image frame queue and push the corresponding timestamp into the abnormal timestamp queue to obtain the updated abnormal image frame queue and abnormal timestamp queue. S512. When a new timestamp arrives, check whether the preceding timestamp has been matched. If it has not been matched, push the preceding timestamp into the abnormal timestamp queue and update the newly arrived timestamp to the current timestamp to obtain the updated abnormal timestamp queue.

[0015] According to the real-time detection and matching method of camera image and timestamp provided by the present invention, step S52 further includes: S521. In the state where the abnormal buffer queue is not empty, the latest arrival timestamp is continuously matched with the latest arrival image frame until a matching success event occurs, and a matching success signal is obtained again. S522. When the rematch success signal is triggered, the current frame number of the updated abnormal timestamp queue and the current frame number of the updated abnormal image frame queue are read respectively, and the numbers are compared to obtain the abnormal type determination result; the abnormal type determination result includes: the number is equal, the number of abnormal timestamp frames is greater than the number of abnormal image frames, and the number of abnormal timestamp frames is less than the number of abnormal image frames.

[0016] According to the real-time detection and matching method of camera image and timestamp provided by the present invention, step S53 further includes: When the number of abnormality types is equal, the cause of the abnormality is determined to be transmission blockage. The frames in the updated abnormal timestamp queue and the updated abnormal image frame queue are dequeued in the order of enqueueing and paired one by one to obtain an image frame sequence including multiple fused image frames with padded timestamps. When the anomaly type determination result is that the number of abnormal timestamp frames is greater than the number of abnormal image frames, the anomaly cause is determined to be image frame loss, an image frame loss alarm is issued, and the frames at the tail of the updated abnormal timestamp queue and the updated abnormal image frame queue are dequeued in sequence for pairing, resulting in an image frame sequence including multiple fused image frames with frame loss labels. When the anomaly type determination result is that the number of abnormal timestamp frames is less than the number of abnormal image frames, the cause of the anomaly is determined to be timestamp loss, triggering timestamp interpolation recovery processing to obtain an interpolated and completed timestamp sequence. The interpolated and completed timestamp sequence is then paired with the frames in the updated abnormal image frame queue in sequence to obtain an image frame sequence including multiple fused image frames with completed timestamps.

[0017] The present invention also provides a method for real-time detection and matching of camera images and timestamps, comprising: Parsing module: Used to receive camera signals and parse them to obtain timestamp information; Calculation module: used to perform a difference calculation between the time stamp information received and the time frame received to obtain the time difference; Statistics module: used to receive the time difference and perform sliding window statistics to obtain the statistical parameter group in the current window; Determination module: used to perform dynamic threshold matching determination by combining the time difference with the statistical parameter group to obtain the matching result; Classification module: used to perform anomaly classification and compensation processing on the image frame based on the matching result, to obtain the fused image frame.

[0018] This invention provides a real-time detection and matching method for camera images and timestamps. By connecting the exposure trigger signal to a high-precision clock synchronization system and parsing the structured high-precision timestamp carrying camera number, frame sequence number, and exposure time information, the exposure time corresponding to each frame image can be accurately recorded with nanosecond-level precision. This fundamentally eliminates the tens of millisecond-level uncontrollable delay error introduced by the traditional host system-based timestamp method, making the timestamp accuracy reach the level of hardware synchronization boards. This fully meets the stringent requirements for clock synchronization accuracy in application scenarios such as multi-sensor fusion for autonomous driving and SLAM positioning. Subsequently, this invention constructs a dynamic confidence interval that adapts to changes in link status by maintaining a sliding window queue for time differences and using an incremental calculation method to update the mean and standard deviation in real time. This ensures that the matching judgment remains stable even when transmission delay fluctuates, avoiding the frequent misjudgments that occur with fixed threshold methods when there is network congestion or changes in CPU load. This significantly reduces the invalid frame loss rate caused by link jitter. In addition, the introduction of an abnormal cache queue mechanism allows the system to automatically distinguish between three abnormal types—transmission blockage, image frame loss, and high-precision timestamp loss—by comparing the number of frames in the abnormal timestamp queue and the abnormal image frame queue. Sequential dequeueing and pairing, tail alignment, or interpolation recovery based on the historical frame interval mean are then performed respectively. This allows most abnormal frames to be automatically repaired and output as fused image frames with high-precision timestamps. The entire process does not require manual frame-by-frame verification, greatly reducing the complexity of data post-processing. Finally, the two independent modal data are merged into a single data source for output, significantly simplifying the data acquisition and processing flow. It can also handle the real-time synchronization task of high frame rate image streams with extremely low CPU utilization on embedded platforms. Attached Figure Description

[0019] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0020] Figure 1 This is a schematic flowchart of a real-time detection and matching method for camera images and timestamps provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the arrival time sequence of camera signals on the time axis provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a real-time detection and matching system for camera images and timestamps provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0023] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0025] The embodiments of the present invention are described below with reference to the figures.

[0026] like Figure 1 As shown, the present invention provides a real-time detection and matching method for camera images and timestamps, comprising: S1. Input the camera signal into the clock synchronization system and parse it to obtain the timestamp information.

[0027] Step S1 further includes: S11. The camera signal, including the exposure trigger signal and the exposure return signal, is connected to the clock synchronization system. The clock synchronization system samples the arrival time of the camera signal to obtain the original trigger event record.

[0028] Furthermore, each time the camera completes an exposure, it outputs a level transition signal to the outside. This signal is divided into two categories: the exposure trigger signal is a signal actively sent to the camera by the external controller to trigger the exposure, and the exposure return signal is a signal actively fed back to the outside by the camera after the exposure is completed. Both can accurately mark the actual exposure time of the camera.

[0029] In step S11, the present invention directly connects the physical line of the above signal to the hardware input port of the high-precision clock synchronization system. When the high-precision clock synchronization system detects a change in signal level, it uses its internal high-precision clock to perform hardware-level sampling of this change moment, recording the moment in nanoseconds to obtain the original trigger event record. The obtained original trigger event record is a binary data packet containing the camera identification, frame number, and exposure time.

[0030] S12. Parse the original trigger event record according to the camera number field, frame sequence number field and exposure time field to obtain a structured timestamp.

[0031] For the original trigger event record, which is a continuous binary data segment, in step S12, the present invention splits it segment by segment according to a predetermined field format. Specifically, firstly, the first 8 bits are extracted from the data packet as a camera number field, used to uniquely identify the camera that generated the exposure event in a multi-camera scenario; then, the following 16 bits are extracted as a frame sequence number field, used to mark the frame number of the exposure event in the output sequence of that camera channel. The frame sequence number increments with each exposure event and is used to detect whether the frame sequence is continuous in subsequent steps; finally, 64 bits are extracted as an exposure time field, recording the absolute time of the exposure event in nanoseconds. After the above three fields are split, the present invention integrates them into a structured data that can be directly read by subsequent steps, obtaining a structured timestamp.

[0032] S13. Record the structured timestamp together with the local time of the system receiving the corresponding timestamp to obtain timestamp information with the first local receiving time.

[0033] In step S13, the present invention transmits the structured timestamp obtained in step S12 from the high-precision clock synchronization system to the acquisition system via a serial data link. At the instant the acquisition system receives the structured timestamp data packet, the present invention simultaneously reads the current time of the local clock of the acquisition system and records this time as the first local reception time. The first local reception time does not reflect the actual exposure time of the camera, but rather the time when the structured timestamp data packet arrives at the acquisition system after transmission via the serial link. There is a fixed deviation between this time and the exposure time, caused by both link transmission delay and system processing time.

[0034] Finally, the present invention binds and stores the first local reception time with the structured timestamp to obtain timestamp information with the first local reception time, and inputs it into step S2 to perform difference calculation with the reception time of the image frame.

[0035] S2. Perform a difference operation between the time stamp information received and the image frame received to obtain the time difference.

[0036] To better understand the embodiments of the present invention, the following is in conjunction with... Figure 2 This describes the timing relationship between the timestamp data packet and the image frame data packet transmitted to the acquisition system via their respective links after the exposure return signal is triggered, as well as the physical meaning and calculation source of ΔT. Specifically, Figure 2 This diagram illustrates the arrival time sequence of three signals—camera exposure event, timestamp data packet, and image frame data packet—on the time axis. The horizontal axis represents the time axis, and the vertical axis is divided into three signal tracks: exposure return, timestamp, and image frame.

[0037] The exposure return track records the level transition signal output by the camera after each exposure action. In the figure, each square wave pulse corresponds to one exposure event, and the rising edge of the pulse is the actual exposure time of the camera. They are marked sequentially as T_01, T_02, T_03, T_04...T_ON. The interval between adjacent exposure events is the frame period of the camera.

[0038] After the exposure return signal is triggered, the high-precision clock synchronization system generates a corresponding structured timestamp data packet after internal processing delay via TS1_delay. This data packet is then transmitted to the acquisition system via the serial link of TS1_Transmit. The acquisition system records the local time of receipt of this data packet, i.e., the first local reception time TS1_RecvTime. In the diagram, the green square represents the time when the timestamp data packet arrives at the acquisition system, and the left boundary of the green square is TS1_RecvTime.

[0039] After the camera completes exposure, it synchronously begins generating image data. After image encoding and output delay (TP1_delay), the image frame data packet begins transmission via USB or IP link, with a transmission time of TP1_Transmit. The arrival time of the image frame data packet at the local acquisition system is the second local reception time (TP1_RecvTime). In the diagram, the blue square represents the arrival time of the image frame data packet at the acquisition system, and the left boundary of the blue square is TP1_RecvTime. Because the image frame data volume is much larger than the timestamp data packet, the sum of TP1_delay and TP1_Transmit is usually greater than the sum of TS1_delay and TS1_Transmit. Therefore, the blue square arrives at the acquisition system later than the green square.

[0040] Figure 2 In the diagram, ΔT is marked between TS1_RecvTime and TP1_RecvTime, which is the time interval between the left boundary of the green square and the left boundary of the blue square. ΔT is calculated as the absolute value of the difference between the second local reception time TP1_RecvTime and the first local reception time TS1_RecvTime. Since the same exposure event T_O1 is canceled out in the time difference calculation, ΔT only reflects the difference in delay between the timestamp transmission link and the image frame transmission link. Under stable link conditions, the ΔT values ​​of each frame remain relatively stable for a short period, approximately following a Gaussian distribution, which is the data basis for the sliding window statistical modeling in step S3. When the ΔT of a frame exceeds the dynamic confidence interval, it indicates that an abnormal delay occurred in the transmission of the timestamp data packet or image frame data packet for that frame.

[0041] Step S2 further includes: S21. When the camera image frame arrives at the acquisition system, record the second local reception time of the current image frame.

[0042] After the camera completes exposure, the image data is transmitted to the acquisition system via USB or IP link. Furthermore, at the instant the acquisition system receives the image frame data packet, this invention reads the current time of the acquisition system's local clock and records this time as the second local reception time.

[0043] The second local reception time and the first local reception time recorded in step S13 both come from the local clock of the same acquisition system. They have a unified time base, so the difference between them can directly reflect the difference in delay between the timestamp data packet and the image frame data packet on their respective transmission links, without introducing additional clock deviation.

[0044] S22. Perform an absolute difference operation between the second local reception time and the first local reception time to obtain the time difference of the current frame.

[0045] In step S22, the present invention subtracts the first local reception time from the second local reception time and takes the absolute value to obtain the time difference ΔT of the current frame. The physical meaning of the time difference is the difference in time between the structured timestamp data packet and the image frame data packet corresponding to the same exposure event arriving at the acquisition system via their respective transmission links. Within a short period of stable link status, both the image frame transmission delay and the timestamp transmission delay are relatively stable. The difference ΔT between the two statistically approximately follows a Gaussian distribution with a constant mean and a small standard deviation, which forms the data basis for the sliding window statistical modeling in step S3.

[0046] S3. Input the time difference into the sliding window statistical model to obtain the statistical parameter group in the current window.

[0047] Step S3 further includes: S31. Maintain an initial sliding window queue that is first-in, first-out. Push the calculated time difference into the initial sliding window queue. When the queue is full, remove the earliest sample that entered the queue to obtain an updated sliding window queue. The capacity of the sliding window queue is 30 frames.

[0048] Furthermore, this invention maintains a first-in, first-out (FIFO) sliding window queue with a fixed capacity of 30 frames to store historical time difference samples from the most recent 30 frames. Whenever a new time difference ΔT is calculated in step S22, this invention pushes it to the tail of the sliding window queue. When the number of samples stored in the queue has not yet reached 30 frames, the new sample is directly enqueued; when the queue has stored 30 frames of samples, this invention pops and discards the earliest enqueued sample at the head of the queue, and then pushes the new time difference sample to the tail of the queue, ensuring that the queue always maintains a capacity of 30 frames, resulting in an updated sliding window queue.

[0049] S32. The incremental calculation method is used to calculate the mean and standard deviation of each time window for all time difference samples in the updated sliding window queue, and the statistical parameter set is obtained.

[0050] In step S32, the present invention calculates the mean μ and standard deviation σ of all time difference samples in the updated sliding window queue obtained in step S31 using an incremental calculation method. Specifically, after each new sample is enqueued and an old sample is dequeued, the present invention only performs difference correction on the current mean μ and variance based on the value of the newly enqueued sample and the value of the eliminated sample. The difference correction is calculated by adding the difference between the new sample and the eliminated sample to the current mean and dividing by the window capacity N to obtain the updated mean μ. Then, the sample variance is updated in the same difference method, and the standard deviation σ is obtained by taking the square root of the variance. The entire update process does not require re-traversing and summing the 30 frames of samples in the queue, and the computational load is constant. The finally obtained mean μ reflects the central level of the current link delay difference, and the standard deviation σ reflects the jitter amplitude of the current link delay difference. Together, they constitute a set of statistical parameters.

[0051] S4. Combine the time difference with the statistical parameter group to perform dynamic threshold matching and obtain the matching result.

[0052] Step S4 further includes: S41. Based on the mean and standard deviation in the statistical parameter group, construct a dynamic confidence interval to obtain the matching threshold interval for the current frame.

[0053] This invention constructs a dynamic confidence interval using the mean μ obtained in step S32 as the interval center, 3 times the standard deviation σ as the interval radius, and μ-3σ as the lower boundary and μ+3σ as the upper boundary. The matching threshold interval for the current frame is obtained. The upper and lower boundaries of the matching threshold interval change synchronously with each update of the statistical parameter set. That is, when the overall link latency increases recently, the mean μ increases accordingly, and the interval shifts upward; when the link jitter increases recently, the standard deviation σ increases accordingly, and the interval width expands accordingly. Subsequently, this invention takes 3 times the standard deviation as the interval radius. Based on the normal distribution characteristics, this interval covers 99.7% of the time difference samples when the link status is normal, ensuring that the vast majority of normal frames are not misjudged as abnormal frames.

[0054] S42. Compare the time difference with the matching threshold interval. If the time difference falls within the matching threshold interval, the match is considered successful; otherwise, the match is considered unsuccessful, and the matching result is obtained.

[0055] In step S42, the present invention compares the current frame time difference ΔT obtained in step S22 with the upper and lower boundaries of the matching threshold interval obtained in step S41. When ΔT simultaneously satisfies the condition of being greater than or equal to the lower boundary μ-3σ and less than or equal to the upper boundary μ+3σ, the current frame is determined to be successfully matched; when ΔT is less than the lower boundary or greater than the upper boundary, the current frame is determined to be unmatched, and a matching result is obtained. When the matching is successful, the present invention writes the exposure time field in the structured timestamp parsed in step S12 into the attribute information of the corresponding image frame. If the timestamp field already exists in the original attributes of the image frame, it is overwritten to obtain a fused image frame carrying a high-precision timestamp; when the matching fails, the image frame and the corresponding timestamp are transferred to the anomaly classification processing flow in step S5.

[0056] S5. Perform anomaly classification and compensation processing on the image frame based on the matching result to obtain the fused image frame.

[0057] In step S5, the anomaly classification process includes the following steps: S51. Push the image frames whose matching result is a failed match and their corresponding timestamps into the exception cache queue to obtain the current exception cache status.

[0058] After outputting the matching results of the failed match in step S4, the present invention pushes the currently failed image frame and its corresponding timestamp into two independent buffer queues, namely the abnormal image frame queue and the abnormal timestamp queue. At the same time, it checks whether the preceding timestamp of the newly arrived timestamp has participated in the matching. If it has not participated, it is pushed into the abnormal timestamp queue to obtain the current abnormal buffer state reflecting the current abnormal frame backlog state.

[0059] Step S51 further includes: S511. When the matching result is a failure, push the current image frame into the abnormal image frame queue and push the corresponding timestamp into the abnormal timestamp queue to obtain the updated abnormal image frame queue and abnormal timestamp queue.

[0060] Specifically, in step S511, after outputting the matching result of the failed match in step S4, the present invention pushes the image frame from this match to the tail of the abnormal image frame queue, and pushes the corresponding timestamp from this match to the tail of the abnormal timestamp queue. The maximum capacity of the abnormal image frame queue is 5 frames, and the maximum capacity of the abnormal timestamp queue is 20 frames. Both queues use a first-in-first-out (FIFO) method to manage the data in the queues. After the pushing is completed, a new record is added to both the abnormal image frame queue and the abnormal timestamp queue, resulting in an updated abnormal image frame queue and an updated abnormal timestamp queue. Together, they constitute the complete state of the current abnormal cache, which is used in step S52 to read the frame count for comparison.

[0061] S512. When a new timestamp arrives, check whether the preceding timestamp has been matched. If it has not been matched, push the preceding timestamp into the abnormal timestamp queue and update the newly arrived timestamp to the current timestamp to obtain the updated abnormal timestamp queue.

[0062] After step S511 is completed, when a new timestamp from the high-precision clock synchronization system arrives at the acquisition system, this invention checks whether there is a matching marker on the preceding timestamp. Each time a timestamp successfully participates in a match, this invention marks that timestamp with a matching marker. If there is no matching marker on the preceding timestamp, it indicates that although the timestamp arrived at the acquisition system in step S511, it has not yet been matched with any image frame and is considered a delayed, abnormal timestamp. In this case, this invention pushes the preceding timestamp to the tail of the abnormal timestamp queue, then updates the newly arrived timestamp with the current timestamp, overwriting the original preceding timestamp record, thus obtaining an updated abnormal timestamp queue.

[0063] S52. Continuously match and determine the timestamps and image frames that arrive later. When a matching success event occurs again, compare the number of abnormal timestamp frames and the number of abnormal image frames in the abnormal cache queue to obtain the abnormal type determination result.

[0064] After step S51 is completed, the updated abnormal image frame queue and the updated abnormal timestamp queue contain several frames that have not yet been matched. The present invention then continuously repeats the matching determination process of steps S2 to S4 with the latest arriving timestamp and the latest arriving image frame until a matching success event occurs again. After a matching success event is triggered, the present invention reads the number of frames currently stored in the updated abnormal timestamp queue and the updated abnormal image frame queue, compares the two numbers, and obtains the abnormality type determination result based on the quantity relationship.

[0065] Step S52 further includes: S521. While the abnormal buffer queue is not empty, the latest arrival timestamp is continuously matched with the latest arrival image frame until a matching success event occurs, and a matching success signal is obtained again.

[0066] Furthermore, the present invention first checks whether the queue of updated abnormal image frames and the queue of updated abnormal timestamps are empty. If both are not empty, the present invention continuously substitutes the latest arriving timestamp and the latest arriving image frame into step S22 to calculate the time difference ΔT, and then substitutes this ΔT into the matching threshold range in step S41 for boundary comparison to obtain a new matching result.

[0067] The above process is continuously executed for each newly arrived timestamp and image frame until a certain pair of timestamps and image frames is successfully matched. The present invention outputs a signal indicating that the matching is successful again, triggering step S522 to read the number of frames in the abnormal cache queue for quantity comparison.

[0068] S522. When the rematch success signal is triggered, the current frame number of the updated abnormal timestamp queue and the current frame number of the updated abnormal image frame queue are read respectively, and the numbers are compared to obtain the abnormal type determination result; the abnormal type determination result includes: the number is equal, the number of abnormal timestamp frames is greater than the number of abnormal image frames, and the number of abnormal timestamp frames is less than the number of abnormal image frames.

[0069] After step S521 outputs a successful matching signal again, this invention reads the currently stored frame count in the updated abnormal timestamp queue and the currently stored frame count in the updated abnormal image frame queue, compares the two integer values, and obtains the abnormality type determination result. There are three abnormality type determination results: First, if the frame counts in both queues are equal, it indicates that neither the timestamps nor the image frames were actually lost during transmission, and the abnormality is caused by transmission congestion; second, if the frame count in the updated abnormal timestamp queue is greater than the frame count in the updated abnormal image frame queue, it indicates that the number of timestamps is greater than the number of image frames, and image frames have been lost; third, if the frame count in the updated abnormal timestamp queue is less than the frame count in the updated abnormal image frame queue, it indicates that the number of timestamps is less than the number of image frames, and timestamp data has been lost. The three determination results are then input into the corresponding recovery processing branch in step S53 for processing.

[0070] S53. Based on the anomaly type determination result, perform recovery processing on the anomaly timestamps and anomaly image frames in the anomaly cache queue to obtain anomaly-processed labeled frame data.

[0071] Step S53 further includes: When the number of abnormality types is equal, the cause of the abnormality is determined to be transmission blockage. The frames in the updated abnormal timestamp queue and the updated abnormal image frame queue are dequeued in the order of enqueueing and paired one by one to obtain an image frame sequence including multiple fused image frames with padded timestamps. When the anomaly type determination result is that the number of abnormal timestamp frames is greater than the number of abnormal image frames, the anomaly cause is determined to be image frame loss, an image frame loss alarm is issued, and the frames at the tail of the updated abnormal timestamp queue and the updated abnormal image frame queue are dequeued in sequence for pairing, resulting in an image frame sequence including multiple fused image frames with frame loss labels. When the anomaly type determination result is that the number of abnormal timestamp frames is less than the number of abnormal image frames, the cause of the anomaly is determined to be timestamp loss, triggering timestamp interpolation recovery processing to obtain an interpolated and completed timestamp sequence. The interpolated and completed timestamp sequence is then paired with the frames in the updated abnormal image frame queue in sequence to obtain an image frame sequence including multiple fused image frames with completed timestamps.

[0072] After step S522 outputs the anomaly type determination result, the present invention enters one of three recovery processing branches based on the result. Specifically, when the determination result is that the number is equal, sequential dequeue pairing processing is performed; when the determination result is that the number of abnormal timestamp frames is greater than the number of abnormal image frames, tail alignment pairing processing is performed and a frame loss alarm is issued; when the determination result is that the number of abnormal timestamp frames is less than the number of abnormal image frames, timestamp interpolation recovery processing is first performed to obtain the interpolated and completed timestamp sequence, and then sequential dequeue pairing processing is performed. All three branches aim to complete the pairing of the updated abnormal timestamp queue with the updated abnormal image frame queue, ultimately obtaining the anomaly-processed labeled frame data.

[0073] In cases where the number of data packets is equal, this invention determines the cause of the anomaly to be transmission congestion. Specifically, timestamp data packets and image frame data packets experience a brief backlog on their respective transmission links, causing the arrival time difference ΔT to exceed the matching threshold range, but no actual data is lost. This invention updates the abnormal timestamp queue and the abnormal image frame queue, dequeuing frames sequentially from the head of their respective queues according to their enqueue order. Timestamps with the same dequeue order are paired with image frames one by one, and the exposure time field from the timestamp is written into the attribute information of the corresponding image frame, resulting in an image frame sequence including multiple fused image frames with padded timestamps.

[0074] For situations where the number of abnormal timestamp frames exceeds the number of abnormal image frames, this invention determines the cause of the anomaly as image frame loss, meaning that some image frames were lost during transmission or storage, resulting in fewer frames in the abnormal image frame queue than in the abnormal timestamp queue, making one-to-one matching impossible. This invention sends an image frame loss alarm to the upper-layer system, and then sequentially dequeues the frames at the tail of the updated abnormal timestamp queue and the updated abnormal image frame queue for matching. Specifically, the timestamp at the tail of the queue is aligned with the image frame at the tail of the queue, and the exposure time field in the timestamp is written into the attribute information of the corresponding image frame, resulting in an image frame sequence including multiple fused image frames with frame loss annotations.

[0075] In cases where the number of abnormal timestamp frames is less than the number of abnormal image frames, this invention determines the cause of the anomaly to be the loss of high-precision timestamps, meaning that some timestamp data packets are lost during serial port transmission, resulting in the number of frames in the abnormal timestamp queue being less than the number of frames in the abnormal image frame queue. This invention triggers timestamp interpolation recovery processing.

[0076] Furthermore, for timestamp interpolation recovery processing, during normal matching, this invention continuously records the high-precision timestamps of the five most recent consecutive successfully matched frames, calculates the timestamp interval between adjacent frames, and averages the five frame intervals to obtain the current frame interval average. Then, for the timestamps at missing positions in the updated abnormal timestamp queue, the timestamp of the most recent successfully matched frame preceding the missing position is used as a benchmark, and the current frame interval average is added to obtain the completed interpolated timestamp. This completed timestamp is then filled into the corresponding missing position in the updated abnormal timestamp queue, resulting in an interpolated completed timestamp sequence. After interpolation completion, if the number of frames in the interpolated completed timestamp sequence matches the number of frames in the updated abnormal image frame queue, they are dequeued sequentially according to their enqueue order and paired one-to-one. The exposure time field in the timestamp is written into the attribute information of the corresponding image frame, resulting in an image frame sequence including multiple fused image frames with completed timestamps. If the number of frames is still inconsistent, the updated abnormal timestamp sequence and the updated abnormal image frame queue are re-executed for quantity comparison in step S522, entering a new round of abnormal type determination.

[0077] like Figure 3 As shown, the present invention also provides a real-time detection and matching method for camera images and timestamps, comprising: Parsing module 100: Used to receive camera signals and parse them to obtain timestamp information; Calculation module 200: used to perform a difference calculation between the time stamp information received and the time frame received to obtain the time difference; Statistics module 300: Used to receive the time difference and perform sliding window statistics to obtain the statistical parameter group in the current window; Decision module 400: used to perform dynamic threshold matching and determination by combining the time difference with the statistical parameter group to obtain the matching result; Classification module 500: Used to perform anomaly classification and compensation processing on the image frame according to the matching result, so as to obtain the fused image frame.

[0078] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0079] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the UBM ocular angle retraction recognition method based on the improved YOLO model described in the various embodiments or some parts of the embodiments.

[0080] This invention, through dynamic window statistics and abnormal queue management, compresses the timestamp error of image frames from the millisecond level of traditional methods to the accuracy level of synchronization boards (microseconds or nanoseconds), meeting the high requirements of camera clock synchronization accuracy in applications such as autonomous driving and SLAM. Furthermore, the dynamic window ensures the system maintains matching even with link latency fluctuations of ±10ms, guaranteeing data continuity in unstable transmission scenarios such as industrial vibration environments or mobile robots, and recovering 95% of lost frames through timestamp interpolation. Secondly, this invention eliminates the need for post-processing manual alignment, merging two different modalities into one, reducing processing complexity and saving 90% of post-processing time, thus accelerating the data pipeline construction and processing flow of autonomous driving multi-sensor fusion and industrial inspection systems.

[0081] The actual test results of this invention on an embedded platform (ARM) show that the single-frame matching latency is <0.1ms and the CPU utilization rate of the 50Hz image stream is <5%, enabling edge computing devices to undertake high-precision time synchronization tasks.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for real-time detection and matching of camera images and timestamps, characterized in that, include: S1. Input the camera signal into the clock synchronization system and parse it to obtain the timestamp information; S2. Perform a difference calculation between the time stamp information received and the time frame received to obtain the time difference; S3. Input the time difference into the sliding window statistical model to obtain the statistical parameter group in the current window; S4. Combine the time difference with the statistical parameter group to perform dynamic threshold matching and determination to obtain the matching result; S5. Perform anomaly classification and compensation processing on the image frame based on the matching result to obtain the fused image frame.

2. The real-time detection and matching method for camera images and timestamps according to claim 1, characterized in that, Step S1 further includes: S11. The camera signal, including the exposure trigger signal and the exposure return signal, is connected to the clock synchronization system. The clock synchronization system samples the arrival time of the camera signal to obtain the original trigger event record. S12. Parse the original trigger event record according to the camera number field, frame sequence number field and exposure time field to obtain a structured timestamp; S13. Record the structured timestamp together with the local time of the system receiving the corresponding timestamp to obtain timestamp information with the first local receiving time.

3. The real-time detection and matching method for camera images and timestamps according to claim 2, characterized in that, Step S2 further includes: S21. When the camera image frame arrives at the acquisition system, record the second local reception time of the current image frame; S22. Perform an absolute difference operation between the second local reception time and the first local reception time to obtain the time difference of the current frame.

4. The real-time detection and matching method for camera images and timestamps according to claim 1, characterized in that, Step S3 further includes: S31. Maintain an initial sliding window queue that is first-in-first-out. Push the calculated time difference into the initial sliding window queue. When the queue is full, eliminate the earliest sample that entered the queue to obtain an updated sliding window queue. S32. The incremental calculation method is used to calculate the mean and standard deviation of each time window for all time difference samples in the updated sliding window queue, and the statistical parameter set is obtained.

5. The real-time detection and matching method for camera images and timestamps according to claim 4, characterized in that, In step S31, the capacity of the sliding window queue is 30 frames.

6. The real-time detection and matching method for camera images and timestamps according to claim 1, characterized in that, Step S4 further includes: S41. Based on the mean and standard deviation in the statistical parameter group, construct a dynamic confidence interval to obtain the matching threshold interval for the current frame; S42. Compare the time difference with the matching threshold interval. If the time difference falls within the matching threshold interval, the match is considered successful; otherwise, the match is considered unsuccessful, and the matching result is obtained.

7. The real-time detection and matching method for camera images and timestamps according to claim 1, characterized in that, In step S5, the anomaly classification process includes the following steps: S51. Push the image frame whose matching result is a failed match and its corresponding timestamp into the abnormal cache queue to obtain the current abnormal cache status. S52. Continuously match and determine the timestamps and image frames that arrive later. When a matching success event occurs again, compare the number of abnormal timestamp frames and the number of abnormal image frames in the abnormal cache queue to obtain the abnormal type determination result. S53. Based on the anomaly type determination result, perform recovery processing on the anomaly timestamps and anomaly image frames in the anomaly cache queue to obtain anomaly-processed labeled frame data.

8. The real-time detection and matching method for camera images and timestamps according to claim 7, characterized in that, Step S51 further includes: S511. When the matching result is a failure, push the current image frame into the abnormal image frame queue and push the corresponding timestamp into the abnormal timestamp queue to obtain the updated abnormal image frame queue and abnormal timestamp queue. S512. When a new timestamp arrives, check whether the preceding timestamp has been matched. If it has not been matched, push the preceding timestamp into the abnormal timestamp queue and update the newly arrived timestamp to the current timestamp to obtain the updated abnormal timestamp queue.

9. The real-time detection and matching method for camera images and timestamps according to claim 8, characterized in that, Step S52 further includes: S521. In the state where the abnormal buffer queue is not empty, the latest arrival timestamp is continuously matched with the latest arrival image frame until a matching success event occurs, and a matching success signal is obtained again. S522. When the rematch success signal is triggered, the current frame number of the updated abnormal timestamp queue and the current frame number of the updated abnormal image frame queue are read respectively, and the numbers are compared to obtain the abnormal type determination result; the abnormal type determination result includes: the number is equal, the number of abnormal timestamp frames is greater than the number of abnormal image frames, and the number of abnormal timestamp frames is less than the number of abnormal image frames.

10. The real-time detection and matching method for camera images and timestamps according to claim 9, characterized in that, Step S53 further includes: When the number of abnormality types is equal, the cause of the abnormality is determined to be transmission blockage. The frames in the updated abnormal timestamp queue and the updated abnormal image frame queue are dequeued in the order of enqueueing and paired one by one to obtain an image frame sequence including multiple fused image frames with padded timestamps. When the anomaly type determination result is that the number of abnormal timestamp frames is greater than the number of abnormal image frames, the anomaly cause is determined to be image frame loss, an image frame loss alarm is issued, and the frames at the tail of the updated abnormal timestamp queue and the updated abnormal image frame queue are dequeued in sequence for pairing, resulting in an image frame sequence including multiple fused image frames with frame loss labels. When the anomaly type determination result is that the number of abnormal timestamp frames is less than the number of abnormal image frames, the cause of the anomaly is determined to be timestamp loss, triggering timestamp interpolation recovery processing to obtain an interpolated and completed timestamp sequence. The interpolated and completed timestamp sequence is then paired with the frames in the updated abnormal image frame queue in sequence to obtain an image frame sequence including multiple fused image frames with completed timestamps.

Images