Video time synchronization processing method and system based on high-precision network time service protocol

Abstract

The application discloses a video time synchronization processing method and system based on a high-precision network time service protocol, which provides time frequency traceability for an internal tame clock module and a clock synchronization module of a processing host through a second pulse signal; the internal tame clock module selects to perform clock tame operation or switches to a self-running time mode according to whether there is an external time reference; according to a time synchronization detection result of a video synchronization processing module, if it is determined that a time synchronization difference between different cameras exceeds a preset range, a hardware trigger signal is sent to the cameras through an external trigger generation module to adjust camera parameters and generate key frames; the video synchronization processing module adopts an image definition evaluation method to identify the key frames generated by the multiple cameras, and completes time axis alignment of the multiple camera video streams based on the key frames. The application solves the problems of low synchronization accuracy or cumulative error and network jitter in the traditional technology, high software post-alignment delay, and lack of feedback linkage between the camera and the processing host.

Description

Technical Field

[0001] This invention belongs to the field of time synchronization technology, specifically relating to a video time synchronization processing method and system based on a high-precision network time synchronization protocol. Background Technology

[0002] With the rapid development of technologies such as multi-camera collaborative monitoring, industrial vision inspection, and virtual reality, the time synchronization accuracy of multi-source video streams in video systems has become a key factor affecting system performance. Multi-camera video time synchronization, as a core issue in computer vision, is fundamental to key tasks such as 3D reconstruction, panoramic stitching, and target tracking; its synchronization effect directly determines the accuracy and reliability of subsequent image processing tasks.

[0003] Currently, traditional video synchronization technologies are mainly divided into two categories: hardware-triggered synchronization and software network time synchronization. However, both technologies have insurmountable bottlenecks and cannot meet the synchronization requirements of high-demand scenarios. Hardware-triggered synchronization triggers simultaneous acquisition by each camera via GPIO signals. Limited by signal transmission delay, it is difficult to achieve sub-millisecond synchronization, and the wiring complexity increases exponentially with the number of cameras, limiting its practicality in large-scale multi-camera deployments. Regarding software network time synchronization, the synchronization accuracy based on the Network Time Protocol (NTP) can only reach a certain range, failing to meet the frame-level alignment requirements of high frame rate scenarios. While the IEEE 1588 Precision Time Protocol (PTP) can theoretically achieve sub-microsecond synchronization, it faces two core problems in practical video system applications: first, the drift of the internal clock crystal oscillator of the camera leads to long-term accumulated errors, compromising long-term synchronization stability; second, network jitter during the transmission of the video stream to the processing host directly breaks the established clock synchronization relationship, causing synchronization failure.

[0004] From the perspective of technical solution completeness, existing mainstream processing solutions only focus on time synchronization at the acquisition end, failing to address the cumulative time deviations that occur during subsequent video stream encoding, decoding, and transmission. This leads to a continuous degradation of synchronization accuracy during data transfer. Furthermore, most existing video processing hosts employ software post-alignment schemes, relying on the CPU for Dynamic Time Warping (DTW) calculations. This results in high latency when processing multiple high-definition video streams, making it difficult to meet real-time requirements. In addition, differences in shooting equipment, shooting angles, and shooting environments may lead to insufficient video quality to support complex real-time computer vision processing tasks such as 3D reconstruction and panoramic stitching. Current technologies lack a systematic feedback and linkage mechanism between the camera and the processing host, making it impossible to dynamically adjust camera parameters according to image processing needs to ensure video quality and synchronization effects. This further limits the application scenarios and performance ceiling of multi-camera video systems. Summary of the Invention

[0005] To address these issues, this invention provides a video time synchronization processing method and system based on a high-precision network time synchronization protocol, solving the problems of low synchronization accuracy or cumulative errors and network jitter in traditional technologies; high software post-alignment latency; and lack of feedback linkage between the camera and the processing host.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a video time synchronization processing method based on a high-precision network time synchronization protocol, comprising the following steps:

[0007] S1: The processing host acquires satellite signals through the satellite receiving module and parses out the second pulse signal. The second pulse signal provides time and frequency traceability for the internal disciplined clock module and clock synchronization module of the processing host. The internal disciplined clock module selects to perform clock discipline operation or switch to self-running time mode based on whether there is an external time reference. The clock synchronization module uses network time synchronization technology to provide timestamps for the camera video stream.

[0008] S2: The video synchronization processing module uses a multi-level time synchronization detection mechanism to detect the time synchronization status of video streams transmitted from multiple cameras from the dimensions of network layer, protocol layer, data layer and content layer, and obtain the time synchronization detection result.

[0009] S3: Based on the time synchronization detection results, if it is determined that the time synchronization difference between different cameras exceeds the preset range, a hardware trigger signal is sent to the camera through the external trigger generation module to adjust the camera parameters and generate key frames.

[0010] S4: The video synchronization processing module uses an image sharpness evaluation method to identify the key frames generated by multiple cameras, and completes the timeline alignment of the multi-camera video streams based on the identified key frames.

[0011] As a preferred solution for video time synchronization processing based on a high-precision network timing protocol, in step S1, the satellite signals received by the satellite receiving module include GPS signals and BeiDou satellite signals, and the parsed second pulse signal is a 1PPS second pulse signal.

[0012] As a preferred solution for video time synchronization processing based on a high-precision network time synchronization protocol, in step S1, the internal disciplined clock module integrates a clock discipline algorithm. When there is an external time reference, the clock discipline algorithm is used to achieve clock discipline. When there is no external time reference, it switches to self-running time mode, and the self-running time mode has a preset timekeeping accuracy.

[0013] The clock synchronization module uses network time synchronization technologies including Network Time Protocol (NTP) and Precision Time Protocol (PTP).

[0014] As a preferred embodiment of the video time synchronization processing method based on a high-precision network time synchronization protocol, in step S2, the detection indicators, measurement methods, and judgment criteria for each level of the multi-level time synchronization detection mechanism are as follows:

[0015] The network layer uses NTP offset as the detection metric and adopts ntpq-p parsing to obtain detection data. When the NTP offset exceeds ±10ms, it is determined to be a synchronization anomaly.

[0016] The protocol layer uses the PTP clock difference as the detection indicator and obtains detection data through 1588 message analysis. When the PTP clock difference exceeds ±2ms, it is determined to be a synchronization abnormality.

[0017] The data layer uses frame timestamp difference as the detection indicator and uses hardware-triggered counting to obtain detection data. When the frame timestamp difference exceeds 1 frame interval, it is determined to be a synchronization abnormality.

[0018] The content layer uses optical flow consistency as the detection index and adopts the feature point motion analysis method to obtain detection data. When the root mean square error (RMSE) of optical flow consistency exceeds 5px, it is judged as a synchronization anomaly.

[0019] As a preferred embodiment of the video time synchronization processing method based on a high-precision network time synchronization protocol, the NTP offset is the difference between the timestamp of the camera video stream and the PTP timestamp of the video synchronization processing module.

[0020] The PTP clock difference is the difference between the timestamp of the camera video stream and the PTP timestamp of the video synchronization processing module.

[0021] When the camera uses only NTP for network time synchronization, disable the protocol layer detection program in the multi-level time synchronization detection mechanism; when the camera uses only PTP for network time synchronization, disable the network layer detection program in the multi-level time synchronization detection mechanism.

[0022] As a preferred solution for video time synchronization processing based on a high-precision network time synchronization protocol, step S3, the process of adjusting camera parameters, includes a synchronization triggering phase and a normal operation phase:

[0023] During the synchronous triggering phase, the camera focal length is finely adjusted by ±5% of the calibrated focal length to achieve a temporary improvement in sharpness, the camera aperture F value is increased by 0.5 stops to expand the depth of field, and the camera exposure time is increased by 20% to enhance the dynamic range.

[0024] During the normal operation phase, the camera's focal length, aperture, and exposure time parameters are restored to their default values.

[0025] As a preferred embodiment of the video time synchronization processing method based on a high-precision network time synchronization protocol, step S4, in which the video synchronization processing module uses an image sharpness evaluation method to identify the key frames generated by multiple cameras, includes the following specific steps:

[0026] Determine whether the video stream carries metadata. If it does, read the frame type field from the metadata. Perform image quality detection on the video frame and calculate the Laplacian variance of the video frame to characterize the image sharpness. If the frame type field of the video frame identifies it as a key frame and the sharpness corresponding to the Laplacian variance is less than a set threshold, then mark the video frame as a key frame and output a True signal.

[0027] If the frame type field of a video frame is not identified as a keyframe, or if the sharpness corresponding to the Laplacian variance reaches the set threshold, then the video frame is marked as a normal frame and a False signal is output.

[0028] As a preferred embodiment of the video time synchronization processing method based on a high-precision network time synchronization protocol, step S4, which involves aligning the timelines of the multi-camera video streams based on the identified keyframes, includes the following steps:

[0029] The video synchronization processing module detects key frames in the video streams of different cameras, records the time t1 corresponding to the first camera key frame K1 and the time t2 corresponding to the second camera key frame K2, and calculates the time difference Δt=t2-t1.

[0030] The alignment engine adjusts the video frame streams from the two cameras to time alignment based on the time difference Δt, thus eliminating time deviation.

[0031] After adjustment, the resampling module resamples the aligned video frame stream to ensure the continuity of the video frame sequence and finally outputs a time-synchronized video frame sequence.

[0032] This invention also provides a video time synchronization processing system based on a high-precision network time synchronization protocol, applied to the aforementioned video time synchronization processing method based on a high-precision network time synchronization protocol, comprising a processing host and at least two cameras; the processing host integrates a satellite receiving module, an internal disciplined clock module, a clock synchronization module, a video synchronization processing module, and an external trigger generation module;

[0033] The external trigger generation module is connected to the hardware trigger input interface of the camera through a physical interface to transmit hardware trigger signals to the camera; the camera integrates a hardware trigger input interface and a gigabit Ethernet interface, and the gigabit Ethernet interface enables real-time camera parameter control and network time synchronization between the camera and the processing host.

[0034] As a preferred solution for a video time synchronization processing system based on a high-precision network timing protocol, the satellite receiving module receives GPS / BeiDou satellite signals and parses 1PPS second pulse signals to provide a time reference for the internal disciplined clock module and clock synchronization module.

[0035] As a preferred solution for a video time synchronization processing system based on a high-precision network time synchronization protocol, the internal disciplined clock module performs clock discipline based on an external time reference, or switches to self-running time mode when there is no external reference, in order to maintain clock stability.

[0036] The clock synchronization module integrates NTP / PTP network time synchronization technology to generate and assign timestamps to the camera video stream.

[0037] The beneficial effects of this invention are as follows:

[0038] First, this invention combines hardware triggering with high-precision network timing technology. It uses the high-precision time and frequency tracing provided by the satellite receiving module, the internal disciplined clock module to solve the crystal oscillator drift problem, and the NTP / PTP dual protocol timing of the clock synchronization module to effectively overcome the shortcomings of traditional hardware triggering synchronization, which is difficult to achieve sub-millisecond precision, and software synchronization, which is easily affected by network jitter and accumulated errors. It provides a more stable and higher precision time synchronization guarantee for multi-camera video streams and meets the frame-level alignment requirements in high frame rate scenarios.

[0039] Secondly, unlike traditional solutions that only focus on synchronization at the acquisition end, this invention covers the entire process of video stream acquisition, encoding / decoding, and transmission. Through a multi-level time synchronization detection mechanism in the video synchronization processing module, it monitors the time deviation of each stage in real time to avoid the accumulation of deviations. At the same time, it abandons the software post-alignment scheme that relies on the CPU and uses hardware triggering and keyframe alignment technology to significantly reduce processing latency and meet the real-time synchronization processing requirements of multiple high-definition videos.

[0040] Third, this invention establishes a systematic feedback linkage between the camera and the processing host. When multi-level detection detects synchronization abnormalities or insufficient video quality, the processing host sends hardware signals through an external triggering module to dynamically adjust parameters such as camera focal length, aperture, and exposure time to generate key frames. At the same time, the video synchronization processing module achieves precise alignment based on image sharpness evaluation, providing high-quality and highly synchronized video data support for complex computer vision tasks such as 3D reconstruction, panoramic stitching, and target tracking.

[0041] Fourth, the host computer integrates multiple core functional modules, reducing external hardware redundancy. At the same time, hardware trigger signals are transmitted through physical interfaces, eliminating the need for complex wiring and avoiding the problem that the complexity of traditional hardware synchronous wiring increases exponentially with the number of cameras. In addition, the embedded real-time computing architecture is lower in cost than solutions that rely on high-performance CPUs, and is easier to promote and apply on a large scale in scenarios such as multi-camera collaborative monitoring and industrial vision inspection. Attached Figure Description

[0042] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0043] Figure 1 This is a schematic flowchart of a video time synchronization processing method based on a high-precision network time synchronization protocol provided in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of a multi-level time synchronization detection mechanism provided in an embodiment of the present invention;

[0045] Figure 3 This is a flowchart of camera-side parameter adjustment and processing host detection alignment provided in an embodiment of the present invention;

[0046] Figure 4 This is the keyframe detection algorithm architecture for the video stream processing host provided in the embodiments of the present invention;

[0047] Figure 5 This invention provides a timeline alignment and video output process based on keyframe detection in an embodiment of the invention.

[0048] Figure 6 This is a diagram illustrating the architecture of a video time synchronization processing system based on a high-precision network time synchronization protocol, provided in an embodiment of the present invention. Detailed Implementation

[0049] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] See Figure 1This invention provides a video time synchronization processing method based on a high-precision network time synchronization protocol, comprising the following steps:

[0051] S1: The processing host acquires satellite signals through the satellite receiving module and parses out the second pulse signal. The second pulse signal provides time and frequency traceability for the internal disciplined clock module and clock synchronization module of the processing host. The internal disciplined clock module selects to perform clock discipline operation or switch to self-running time mode based on whether there is an external time reference. The clock synchronization module uses network time synchronization technology to provide timestamps for the camera video stream.

[0052] Among them, the satellite receiving module is the core of time traceability. By receiving satellite signals such as GPS / BeiDou, it can obtain a globally unified high-precision time reference. The parsed second pulse signal can provide a stable time frequency reference for subsequent modules, avoiding time reference deviation. The internal disciplined clock module switches between disciplined and self-running modes. When there is an external time reference, it can calibrate its own clock deviation to ensure consistency with the reference time. When there is no external reference, it switches to self-running mode to maintain clock stability and solve the problem of cumulative error caused by the drift of the internal clock crystal oscillator of traditional cameras. The network time synchronization technology of the clock synchronization module marks the video stream with a unified time mark, ensuring that each frame of video captured by multiple cameras has a comparable time basis.

[0053] S2: The video synchronization processing module uses a multi-level time synchronization detection mechanism to detect the time synchronization status of video streams transmitted from multiple cameras from the dimensions of network layer, protocol layer, data layer and content layer, and obtain the time synchronization detection result.

[0054] The network layer focuses on the network transmission link between the camera and the processing host, detecting the impact of network transmission latency and jitter on time synchronization; the protocol layer encapsulates protocols for the video stream to ensure accurate timestamp transmission at the protocol level; the data layer directly deals with video frame data, judging the synchronization status at the frame level through frame timestamp differences; and the content layer starts from the characteristics of video content, verifying the synchronization effect through visual features such as optical flow consistency, avoiding the problem of timestamp synchronization but content asynchrony due to technical failures. Multi-dimensional detection can comprehensively cover synchronization risk points, ensuring accurate and reliable detection results.

[0055] S3: Based on the time synchronization detection results, if it is determined that the time synchronization difference between different cameras exceeds the preset range, a hardware trigger signal is sent to the camera through the external trigger generation module to adjust the camera parameters and generate key frames.

[0056] When a synchronization difference exceeds the limit, the hardware trigger signal can be directly applied to the camera, which is faster than software commands and can reduce correction delay. The camera parameters, such as focal length, aperture, and exposure time, are adjusted to generate keyframes. The keyframes contain richer image information and have clear time markers. Subsequently, the time alignment of video streams from multiple cameras can be quickly achieved based on the keyframes. At the same time, parameter adjustments can optimize image quality and avoid affecting the synchronization alignment accuracy due to image quality issues.

[0057] S4: The video synchronization processing module uses an image sharpness evaluation method to identify the key frames generated by multiple cameras, and completes the timeline alignment of the multi-camera video streams based on the identified key frames.

[0058] Among them, the image sharpness evaluation method can screen out key frames with qualified image quality, avoid key frame recognition errors caused by blurring, noise and other problems, and ensure the reliability of key frames used for alignment; time axis alignment is performed based on key frames. Key frames are representative in the time dimension. By calculating the time difference of key frames from different cameras, the time offset of each video stream can be quickly determined, and then the time axis can be adjusted so that multiple video streams present the corresponding picture content at the same time point, which meets the dual requirements of time-content synchronization for tasks such as 3D reconstruction and panoramic stitching.

[0059] In this embodiment, in step S1, the satellite signals received by the satellite receiving module include GPS signals and BeiDou satellite signals, and the parsed second pulse signal is a 1PPS second pulse signal.

[0060] Specifically, the satellite receiving module supports both GPS and BeiDou satellite signals, which can improve the stability and coverage of satellite signal reception and avoid time tracking interruption when a single satellite system signal is blocked or interfered with; the 1PPS second pulse signal is the carrier of high-precision time transmission. It generates a precise pulse every second, and the rising edge of the pulse corresponds to the whole second of the standard time. It can provide a nanosecond-level time reference for the internal disciplined clock module and clock synchronization module, ensuring a high-precision foundation for subsequent time synchronization.

[0061] In this embodiment, in step S1, the internal disciplined clock module integrates a clock discipline algorithm. When an external time reference exists, the clock discipline algorithm is used to achieve clock discipline. When there is no external time reference, it switches to self-running time mode, and the self-running time mode has a preset timekeeping accuracy. The network time synchronization technology used by the clock synchronization module includes Network Time Protocol (NTP) and Precision Time Protocol (PTP).

[0062] Specifically, the clock discipline algorithm continuously compares the external time reference with the module's internal clock, calculates the deviation, and dynamically calibrates it to gradually discipline the internal clock, making it consistent with the external reference and eliminating long-term accumulated errors caused by crystal oscillator drift. The preset timekeeping accuracy in the self-running mode ensures that the module can still maintain clock accuracy for a period of time when the external reference is interrupted, avoiding immediate synchronization failure. The NTP and PTP dual protocols can adapt to different scenario requirements. NTP is suitable for scenarios with slightly lower accuracy requirements and flexible deployment, while PTP is suitable for sub-millisecond high-precision synchronization scenarios. The dual protocol selection can improve system compatibility and applicability.

[0063] See Figure 2 In this embodiment, in step S2, the detection indicators, measurement methods, and judgment criteria for each level of the multi-level time synchronization detection mechanism are as follows:

[0064] The network layer uses NTP offset as the detection metric and employs the ntpq-p parsing method to obtain detection data. When the NTP offset exceeds ±10ms, it is determined to be a synchronization anomaly. Here, NTP offset refers to the difference between the NTP time at the camera end and the NTP time at the processing host, directly reflecting the degree of deviation in network layer time synchronization. ntpq-p is a commonly used query tool for the NTP protocol, which can conveniently obtain key data such as NTP offset. Setting ±10ms as the threshold is because this threshold can cover the normal jitter range of network transmission in most scenarios, and can promptly detect deviations exceeding the normal range, avoiding the accumulation of deviations that affect subsequent synchronization.

[0065] The protocol layer uses the PTP clock difference as a detection indicator, and obtains detection data through 1588 message analysis. When the PTP clock difference exceeds ±2ms, it is judged as a synchronization anomaly. The PTP clock difference is the difference between the PTP clock at the camera end and the PTP clock at the processing host, corresponding to the time synchronization status of the protocol layer. The 1588 message is the core data carrier of the PTP protocol, containing key information such as OriginTimestamp and CorrectionField. By analyzing these fields, the PTP clock difference can be accurately calculated. The ±2ms threshold setting is based on the high-precision characteristics of the PTP protocol, matching its sub-millisecond synchronization requirements, and ensuring that the synchronization accuracy of the protocol layer meets the requirements of high frame rate scenarios.

[0066] The data layer uses frame timestamp difference as the detection indicator and acquires detection data using hardware trigger counting. When the frame timestamp difference exceeds one frame interval, it is judged as a synchronization anomaly. Here, frame timestamp difference is the difference in timestamp values ​​of video frames captured by different cameras at the same time, which directly reflects the synchronization status at the video frame level. The hardware trigger counting method can accurately record the frame trigger time through hardware circuitry, avoiding the delay and error of software counting. One frame interval is the minimum time unit for video frame acquisition. Exceeding this interval means that there is a significant time misalignment in frame acquisition, which will directly affect the continuity of subsequent image processing. Therefore, using this as a threshold can ensure frame-level synchronization.

[0067] The content layer uses optical flow consistency as the detection metric and employs feature point motion analysis to acquire detection data. When the root mean square error (RMSE) of optical flow consistency exceeds 5px, it is judged as a synchronization anomaly. Optical flow consistency reflects whether the motion trajectory of the target in the image is consistent when multiple cameras capture the same target. If time synchronization is normal, the motion trajectory of the same target in images from different cameras should be consistent. Feature point motion analysis extracts feature points (such as corner points) from the image, calculates the motion vector (optical flow) of the feature points in consecutive frames, and then quantifies the difference in optical flow between different cameras using RMSE. The 5px RMSE threshold is set based on human visual perception and image processing requirements. Exceeding this threshold means that the target's motion trajectory is significantly deviated, and even if the timestamp is synchronized, the content is misaligned and needs to be corrected.

[0068] In this embodiment, the NTP offset is the difference between the timestamp of the camera video stream and the PTP timestamp of the video synchronization processing module; the PTP clock difference is the difference between the timestamp of the camera video stream and the PTP timestamp of the video synchronization processing module; when the camera only uses NTP for network time synchronization, the protocol layer detection program in the multi-level time synchronization detection mechanism is turned off; when the camera only uses PTP for network time synchronization, the network layer detection program in the multi-level time synchronization detection mechanism is turned off.

[0069] Specifically, the timestamp of the camera video stream is compared with the PTP timestamp of the processing module. The PTP timestamp of the video synchronization processing module is based on high-precision time traceability and can be used as a unified reference benchmark to ensure the accuracy of offset and clock difference calculations. The corresponding detection level is closed according to the actual time synchronization protocol used. On the one hand, this can reduce unnecessary consumption of computing resources and avoid invalid detection occupying system performance. On the other hand, it can simplify the detection process and make the detection results more focused on the current time synchronization method, thereby improving detection efficiency and accuracy.

[0070] See Figure 3 In this embodiment, step S3, the process of adjusting the camera parameters, includes a synchronization triggering phase and a normal operation phase:

[0071] During the synchronous triggering phase, the camera focal length is fine-tuned by ±5% of the calibrated focal length to achieve a temporary improvement in sharpness. The camera aperture F-stop is increased by 0.5 stops to expand the depth of field, and the camera exposure time is increased by 20% to enhance the dynamic range. The ±5% focal length fine-tuning quickly optimizes the image focus, preventing keyframe blurring due to slight defocusing. Increasing the aperture F-stop by 0.5 stops reduces the aperture diameter, expanding the depth of field and keeping more areas of the image sharp, providing richer information for feature point extraction. Increasing the exposure time by 20% improves image brightness, enhances details in dark areas, expands the dynamic range, and prevents excessive image noise due to insufficient light, ensuring that the keyframe image quality meets the requirements of subsequent alignment and image processing.

[0072] During the normal operation phase, the camera's focal length, aperture, and exposure time parameters are restored to their default values. This restoration of default parameters during normal operation is crucial because the parameters used in the synchronization trigger phase are temporarily optimized for keyframe generation. Prolonged use of these parameters could lead to overexposure and excessive depth of field affecting the subject's prominence in the normal video stream. Restoring default parameters ensures that the normally acquired video stream meets daily usage requirements, balancing the image quality of keyframe generation and regular video acquisition, and preventing long-term parameter deviations from impacting overall system performance.

[0073] See Figure 4 In this embodiment, step S4, the specific process by which the video synchronization processing module uses an image sharpness evaluation method to identify the key frames generated by multiple cameras includes:

[0074] Determine whether the video stream carries metadata. If it does, read the frame type field from the metadata. Perform image quality detection on the video frame and calculate the Laplacian variance of the video frame to characterize the image sharpness. If the frame type field of the video frame identifies it as a key frame and the sharpness corresponding to the Laplacian variance is less than a set threshold, then mark the video frame as a key frame and output a True signal.

[0075] Specifically, the frame type field in the metadata contains keyframe identifiers marked during video encoding, which can quickly filter out candidate keyframes and reduce invalid detections; Laplacian variance is a commonly used sharpness evaluation metric, and the larger the value, the sharper the image edges and the higher the sharpness. Calculating this metric can quantitatively evaluate frame image quality; setting a sharpness threshold and requiring candidate keyframes to have a sharpness less than the threshold excludes blurry candidate frames caused by improper parameter adjustment, ensuring that the marked keyframes have both temporal identification and image quality reliability, providing high-quality data for subsequent alignment.

[0076] If a video frame's frame type field is not identified as a keyframe, or if the sharpness corresponding to the Laplacian variance reaches a set threshold, then the video frame is marked as a normal frame and a False signal is output. Video frames whose frame type field is not identified as keyframes do not possess temporal representativeness of keyframes and do not need to be used for alignment; candidate frames whose sharpness reaches the set threshold may cause alignment deviations if used for alignment, so marking them as normal frames can avoid their interference with the synchronization process, ensuring that only high-quality keyframes participate in alignment and improving synchronization accuracy.

[0077] See Figure 5 In this embodiment, step S4, which involves aligning the timelines of the multi-camera video streams based on the identified keyframes, includes the following steps:

[0078] S41. The video synchronization processing module detects key frames in the video streams of different cameras respectively, records the time t1 corresponding to the first camera key frame K1 and the time t2 corresponding to the second camera key frame K2, and calculates the time difference Δt=t2-t1.

[0079] Specifically, keyframes K1 and K2 are the time anchor points of the two cameras on their respective timelines. Recording their times t1 and t2 can determine the offset relationship between the two cameras' timelines. The time difference Δt directly reflects the degree of time misalignment of the video streams from the two cameras. A positive Δt indicates that K2 lags behind K1, and a negative Δt indicates that K2 leads K1. Precise alignment is achieved by clearly defining the direction and magnitude of the offset.

[0080] S42: The alignment engine adjusts the video frame streams from the two cameras according to the time difference Δt to eliminate time deviation;

[0081] Specifically, if K2 lags behind K1, the video stream from the second camera can be read in advance or the video stream from the first camera can be delayed and buffered to ensure that the frames at the same time point in the two video streams correspond. The adjustment process is based on hardware acceleration to avoid the delay of software adjustment, ensure real-time alignment, and after eliminating time deviation, the time axis of multiple video streams can be kept consistent.

[0082] S43. After the adjustment is completed, the resampling module resamples the aligned video frame stream to ensure the continuity of the video frame sequence and finally outputs a time-synchronized video frame sequence.

[0083] Specifically, alignment adjustments may cause slight differences in video frame intervals. For example, if Δt is not an integer multiple of the frame interval, the resampling module can restore the adjusted video frame stream to a uniform frame interval through interpolation, frame extraction, and other methods, ensuring the continuity of video playback and subsequent processing. The final output time-synchronized video frame sequence can be directly used for tasks such as 3D reconstruction and panoramic stitching without additional synchronization processing, thus improving the overall efficiency of the system.

[0084] The application scenarios of the method in this embodiment of the invention are as follows:

[0085] Scenario 1: Multi-camera collaborative monitoring scenario:

[0086] In collaborative monitoring scenarios requiring multi-camera coverage, such as urban road intersections, large commercial complexes, and industrial parks, traditional synchronization technologies struggle to meet the demands for event tracing and real-time early warning due to insufficient accuracy or complex wiring. This invention addresses this by utilizing a satellite receiving module to provide 1PPS high-precision time tracing, an internal disciplined clock module to resolve crystal oscillator drift, and the PTP protocol to achieve video stream synchronization faster than 2ms. This ensures precise time alignment of video streams from different cameras, such as intersection cameras and shopping mall wide-angle cameras. In the event of traffic accidents, theft, or other incidents, multi-view videos from the same time point can be quickly retrieved, clearly reconstructing the sequence of events, such as vehicle traffic timing and personnel movement trajectories, providing accurate evidence for liability determination. Simultaneously, the multi-level detection of the video synchronization processing module monitors the synchronization status in real time, preventing synchronization failures caused by network jitter and ensuring stable 24-hour operation of the monitoring system, meeting the real-time early warning requirements for low latency and high synchronization.

[0087] Scenario 2: Industrial visual inspection scenario:

[0088] In industrial production line visual inspection scenarios such as automotive parts dimensional inspection and electronic chip pin defect screening, multiple cameras need to simultaneously acquire product images from different angles. Traditional solutions only address synchronization at the acquisition end, neglecting the accumulation of deviations in the transmission and decoding stages, which can easily lead to measurement data errors. This invention covers the entire process of video stream acquisition, encoding / decoding, and transmission synchronization. It adapts to different production line requirements through a clock synchronization module with both NTP and PTP protocols, such as NTP for general inspection and PTP for high-precision measurement. Combined with frame timestamp difference detection by the video synchronization processing module, it avoids the accumulation of deviations affecting inspection accuracy. When a synchronization anomaly is detected, an external trigger generation module can send hardware signals to adjust camera parameters, generating high-quality keyframes to ensure that images from various perspectives simultaneously capture the same product state (such as the dimensional data of a certain cross-section of a component), improving dimensional measurement accuracy. Simultaneously, an embedded real-time computing architecture replaces CPU-dependent software post-alignment, processing eight channels of 1080p video with a latency far below 30ms, meeting the real-time requirements of high-speed production line inspection and preventing production line stalls due to processing delays.

[0089] Scenario 3: Virtual Reality (VR) Panoramic Video Production Scenario:

[0090] In VR panoramic video production scenarios such as VR scenic area tours and immersive film shooting, multiple cameras need to be arranged in a ring to capture 360° images. Traditional solutions are prone to screen tearing and stuttering during panoramic stitching due to insufficient synchronization accuracy. This invention sends synchronization trigger signals to all cameras through a hardware trigger generation module. Combined with the internal timestamp accuracy of the PTP protocol, it achieves sub-millisecond synchronization, ensuring that each camera captures scene details simultaneously and avoiding "image misalignment" during stitching. At the same time, the image quality feedback mechanism can dynamically adjust camera parameters: fine-tuning the focus during synchronization triggering to improve image clarity, and restoring default parameters during normal operation to ensure video continuity and ensure consistent image quality across cameras, avoiding uneven brightness in the panoramic image due to poor image quality from some cameras; the video synchronization processing module, based on Laplacian variance keyframe recognition, can select clear keyframes for stitching, further optimizing panoramic video quality and enhancing user immersion.

[0091] Scenario 4: 3D Reconstruction Scenario

[0092] In 3D reconstruction scenarios such as digital modeling of ancient buildings and surveying and mapping of cultural relics, multiple cameras need to capture video streams from different perspectives of the object to generate accurate 3D models. Traditional solutions, due to the lack of feedback and linkage between the cameras and the processing host, are prone to model distortion due to insufficient image quality or synchronization deviations. The multi-level time synchronization detection of this invention can ensure that the video streams of each camera are synchronized in both time and content. For example, when capturing the patterns of ancient building column bases, the video from each perspective simultaneously captures the details of the patterns, avoiding misalignment of the patterns due to time deviations. When insufficient video quality is detected, the processing host sends parameter adjustment instructions to the cameras through the gigabit Ethernet interface to generate high-quality keyframes; the video synchronization processing module aligns the time axis based on the keyframes to ensure multi-perspective data matching and improve the accuracy of the 3D model; at the same time, the GPS / BeiDou dual-mode of the satellite receiving module ensures stable time traceability during outdoor surveying and mapping, avoiding synchronization interruptions caused by signal blockage, and meeting the reconstruction needs of outdoor scenes such as ancient buildings and cultural relics.

[0093] See Figure 6 This invention also provides a video time synchronization processing system based on a high-precision network time synchronization protocol, applied to the aforementioned video time synchronization processing method based on a high-precision network time synchronization protocol. The system includes a processing host and at least two cameras. The processing host integrates a satellite receiving module, an internal disciplined clock module, a clock synchronization module, a video synchronization processing module, and an external trigger generation module. Principle explanation:

[0094] The external trigger generation module is connected to the camera's hardware trigger input interface via a physical interface to transmit hardware trigger signals to the camera. The camera integrates a hardware trigger input interface and a gigabit Ethernet interface. The gigabit Ethernet interface enables real-time camera parameter control and network time synchronization between the camera and the processing host. The physical interface connection between the external trigger generation module and the camera ensures low latency and strong anti-interference capability for hardware trigger signal transmission. Compared to software commands, it can trigger camera parameter adjustments and keyframe generation faster, solving the problem of high latency correction when synchronization anomalies occur. The camera's integrated hardware trigger input interface is a prerequisite for receiving hardware trigger signals, while the gigabit Ethernet interface has high bandwidth and low latency characteristics. It can meet the requirements for fast transmission of real-time camera parameter control commands (such as focus and aperture adjustment commands) and ensure stable interaction of network time synchronization data, avoiding command lag or time synchronization data loss due to insufficient interface bandwidth, and ensuring the real-time linkage between the camera and the processing host.

[0095] In this embodiment, the satellite receiving module receives GPS / BeiDou satellite signals and parses the 1PPS second pulse signal to provide a time reference for the internal disciplined clock module and clock synchronization module. GPS / BeiDou satellite signals have global coverage and high precision. The simultaneous reception of both signals by the satellite receiving module avoids time reference interruptions caused by signal blockage or interference from a single satellite system, improving the reliability of time traceability. The rising edge of the 1PPS second pulse signal corresponds to the exact second of standard time, providing a time reference with nanosecond-level precision. The internal disciplined clock module uses this as a reference to calibrate its crystal oscillator drift, and the clock synchronization module uses it as a reference to generate a high-precision timestamp, ensuring a unified time reference for the entire system and fundamentally solving the problem of insufficient synchronization accuracy caused by time reference deviations in traditional systems.

[0096] In this embodiment, the internal disciplined clock module performs clock discipline based on an external time reference, or switches to self-running time mode when there is no external reference, to maintain clock stability. The clock synchronization module integrates NTP / PTP network time synchronization technology to generate and allocate timestamps for the camera video streams. The clock discipline function of the internal disciplined clock module continuously compares the external time reference with the module's internal clock, calculates the deviation and dynamically calibrates, gradually eliminating the long-term accumulated error caused by crystal oscillator drift, ensuring that the clock is consistent with the standard time. The self-running time mode is a backup solution when the external reference is interrupted. Its preset high timekeeping accuracy can maintain short-term clock stability and avoid immediate synchronization failure. The clock synchronization module integrates NTP / PTP dual protocols, which can be flexibly selected according to the scenario requirements. NTP is suitable for scenarios with slightly lower accuracy requirements and flexible deployment, while PTP is suitable for sub-millisecond high-precision scenarios. Dual protocol adaptation can meet the synchronization requirements of different cameras, and the generated and allocated timestamps provide a unified time identifier for each camera video stream.

[0097] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A video time synchronization processing method based on a high-precision network time synchronization protocol, characterized in that, Includes the following steps: S1: The processing host acquires satellite signals through the satellite receiving module and parses out the second pulse signal. The second pulse signal provides time and frequency traceability for the internal disciplined clock module and clock synchronization module of the processing host. The internal disciplined clock module selects to perform clock discipline operation or switch to self-running time mode based on whether there is an external time reference. The clock synchronization module uses network time synchronization technology to provide timestamps for the camera video stream. S2: The video synchronization processing module uses a multi-level time synchronization detection mechanism to detect the time synchronization status of video streams transmitted from multiple cameras from the dimensions of network layer, protocol layer, data layer and content layer, and obtain the time synchronization detection result. S3: Based on the time synchronization detection results, if it is determined that the time synchronization difference between different cameras exceeds the preset range, a hardware trigger signal is sent to the camera through the external trigger generation module to adjust the camera parameters and generate key frames. S4: The video synchronization processing module uses an image sharpness evaluation method to identify the key frames generated by multiple cameras, and completes the time axis alignment of the multi-camera video stream based on the identified key frames; In step S2, under the multi-level time synchronization detection mechanism, the detection indicators, measurement methods, and judgment criteria for each level are as follows: The network layer uses NTP offset as the detection metric and adopts ntpq-p parsing to obtain detection data. When the NTP offset exceeds ±10ms, it is determined to be a synchronization anomaly. The protocol layer uses the PTP clock difference as the detection indicator and obtains detection data through 1588 message analysis. When the PTP clock difference exceeds ±2ms, it is determined to be a synchronization abnormality. The data layer uses frame timestamp difference as the detection indicator and uses hardware-triggered counting to obtain detection data. When the frame timestamp difference exceeds 1 frame interval, it is determined to be a synchronization abnormality. The content layer uses optical flow consistency as the detection index and adopts the feature point motion analysis method to obtain detection data. When the root mean square error (RMSE) of optical flow consistency exceeds 5px, it is judged as a synchronization anomaly.

2. The video time synchronization processing method based on a high-precision network time synchronization protocol according to claim 1, characterized in that, In step S1, the satellite signals received by the satellite receiving module include GPS signals and BeiDou satellite signals, and the parsed second pulse signal is a 1PPS second pulse signal; In step S1, the internal tamed clock module integrates a clock taming algorithm. When an external time reference is available, the clock is tamed through the clock taming algorithm. When there is no external time reference, it switches to the self-running time mode, and the self-running time mode has a preset timekeeping accuracy. The clock synchronization module uses network time synchronization technologies including Network Time Protocol (NTP) and Precision Time Protocol (PTP).

3. The video time synchronization processing method based on a high-precision network time synchronization protocol according to claim 2, characterized in that, The NTP offset is the difference between the timestamp of the camera video stream and the PTP timestamp of the video synchronization processing module. The PTP clock difference is the difference between the timestamp of the camera video stream and the PTP timestamp of the video synchronization processing module. When the camera uses only NTP for network time synchronization, disable the protocol layer detection program in the multi-level time synchronization detection mechanism; when the camera uses only PTP for network time synchronization, disable the network layer detection program in the multi-level time synchronization detection mechanism.

4. The video time synchronization processing method based on a high-precision network time synchronization protocol according to claim 1, characterized in that, Step S3, the process of adjusting camera parameters includes a synchronization trigger phase and a normal operation phase: During the synchronous triggering phase, the camera focal length is finely adjusted by ±5% of the calibrated focal length to achieve a temporary improvement in sharpness, the camera aperture F value is increased by 0.5 stops to expand the depth of field, and the camera exposure time is increased by 20% to enhance the dynamic range. During the normal operation phase, the camera's focal length, aperture, and exposure time parameters are restored to their default values.

5. The video time synchronization processing method based on a high-precision network time synchronization protocol according to claim 1, characterized in that, In step S4, the specific process by which the video synchronization processing module identifies the keyframes generated by multiple cameras using an image sharpness evaluation method includes: Determine whether the video stream carries metadata. If it does, read the frame type field from the metadata. Perform image quality detection on the video frame and calculate the Laplacian variance of the video frame to characterize the image sharpness. If the frame type field of the video frame identifies it as a key frame and the sharpness corresponding to the Laplacian variance is less than a set threshold, then mark the video frame as a key frame and output a True signal. If the frame type field of a video frame is not identified as a keyframe, or if the sharpness corresponding to the Laplacian variance reaches the set threshold, then the video frame is marked as a normal frame and a False signal is output.

6. The video time synchronization processing method based on a high-precision network time synchronization protocol according to claim 1, characterized in that, In step S4, the specific process of aligning the timelines of the multi-camera video streams based on the identified keyframes includes: The video synchronization processing module detects key frames in the video streams of different cameras, records the time t1 corresponding to the first camera key frame K1 and the time t2 corresponding to the second camera key frame K2, and calculates the time difference Δt=t2-t1. The alignment engine adjusts the video frame streams from the two cameras to time alignment based on the time difference Δt, thus eliminating time deviation. After adjustment, the resampling module resamples the aligned video frame stream to ensure the continuity of the video frame sequence and finally outputs a time-synchronized video frame sequence.

7. A video time synchronization processing system based on a high-precision network time synchronization protocol, applied to the video time synchronization processing method based on a high-precision network time synchronization protocol as described in any one of claims 1-6, characterized in that, It includes a processing host and at least two cameras; the processing host integrates a satellite receiving module, an internal disciplined clock module, a clock synchronization module, a video synchronization processing module, and an external trigger generation module. The external trigger generation module is connected to the hardware trigger input interface of the camera through a physical interface to transmit hardware trigger signals to the camera; the camera integrates a hardware trigger input interface and a gigabit Ethernet interface, and the gigabit Ethernet interface enables real-time camera parameter control and network time synchronization between the camera and the processing host.

8. The video time synchronization processing system based on a high-precision network time synchronization protocol according to claim 7, characterized in that, The satellite receiving module receives GPS / BeiDou satellite signals and parses 1PPS second pulse signals to provide a time reference for the internal disciplined clock module and clock synchronization module.

9. The video time synchronization processing system based on a high-precision network time synchronization protocol according to claim 7, characterized in that, The internal disciplined clock module performs clock discipline based on an external time reference, or switches to self-running mode when there is no external reference, in order to maintain clock stability. The clock synchronization module integrates NTP / PTP network time synchronization technology to generate and assign timestamps to the camera video stream.

Images