Frame synchronization defect detection method and terminal

Abstract

The invention discloses a frame synchronization defect detection method and a terminal, and the method comprises the steps: inputting each video frame sequence into a transformation network through obtaining at least two video frame sequences, and outputting a corresponding frame embedding vector; calculating a path drift value between every two paths of video frame sequences in the at least two paths of video frame sequences according to the frame embedding vector, and determining a global path drift value based on the path drift values; and if the global path drift value is greater than a preset frame difference threshold value, determining that a frame synchronization defect exists, so that the synchronization defect among multiple paths of video streams can be detected in real time, and the reliability and stability of video transmission are ensured.

Description

Technical Field

[0001] This invention relates to the field of defect detection technology, and in particular to a frame synchronization defect detection method and terminal. Background Technology

[0002] In fields such as video streaming and cloud gaming, video streams need to be transmitted through a complete link: "server → client → CDN (Content Delivery Network) → viewer." The transmission process commonly faces problems caused by CDN, such as audio-visual asynchrony, screen tearing, and frame skipping. Existing detection technologies mostly rely on log timestamp comparison or hash value verification, which has certain limitations: firstly, most existing technologies only cover the "server..." "Client" dual-end link, for "client" First, it cannot address transmission defects at the critical "viewer-side" stage. Second, while hash value comparison can detect content distortion, it cannot effectively identify temporal drift in frame sequences. Third, existing recovery methods (such as packet retransmission and audio white noise alignment) mostly focus on the network layer or single audio-visual delay compensation, lacking the ability to visually perceive and locate drift in the global synchronization state of multiple video streams. These shortcomings lead to a deterioration in the end-user experience. For example, in cloud gaming, an operation latency exceeding 67ms can be clearly perceived as stuttering, while in esports live streaming, a delay between the viewer's screen and the player's screen easily leads to complaints, ultimately resulting in user churn and commercial losses. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a frame synchronization defect detection method and terminal that can detect synchronization defects between multiple video streams in real time, thereby ensuring the reliability and stability of video transmission.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for detecting frame synchronization defects includes the following steps: Acquire at least two video frame sequences, input each video frame sequence into the transform network, and output the corresponding frame embedding vector. The path drift value between each pair of video frame sequences in the at least two video frame sequences is calculated based on the frame embedding vector, and the global path drift value is determined based on the path drift value. If the global path drift value is greater than the preset frame difference threshold, it is determined that there is a frame synchronization defect.

[0005] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A frame synchronization defect detection terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the various steps of the frame synchronization defect detection method described above.

[0006] The beneficial effects of this invention are as follows: This invention provides a frame synchronization defect detection method and terminal. By acquiring at least two video frame sequences, it provides a data foundation for the synchronization analysis of multiple video streams. Each video frame sequence is input into a transform network, and the corresponding frame embedding vector is output, realizing efficient feature extraction and dimensionality reduction of video frames, simplifying complex data, improving analysis efficiency, and laying the foundation for subsequent path drift value calculation. The path drift value between every two video frame sequences is calculated based on the frame embedding vector, and the global path drift value is determined based on the path drift value, accurately quantifying the synchronization deviation between different video streams and comprehensively evaluating the overall video synchronization status. If the global path drift value is greater than a preset frame difference threshold, a frame synchronization defect is determined to exist, realizing fast and accurate frame synchronization problem detection and ensuring the reliability and stability of video transmission. Attached Figure Description

[0007] Figure 1 This is a flowchart of a frame synchronization defect detection method according to an embodiment of the present invention; Figure 2 This is another flowchart of a frame synchronization defect detection method according to an embodiment of the present invention; Figure 3 This is an example diagram of the multipath drift curve in an embodiment of the present invention; Figure 4 This is a schematic diagram of the rollback operation in an embodiment of the present invention; Figure 5 This is a schematic diagram of a frame synchronization defect detection terminal according to an embodiment of the present invention; Label Explanation: 1. A frame synchronization defect detection terminal; 2. Memory; 3. Processor. Detailed Implementation

[0008] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0009] The following is an explanation of the technical terms used in this invention: (1) Transform Network (T-Net): Used to process video temporal analysis tasks (such as frame synchronization detection). It converts video frame sequences into embedding vectors containing spatiotemporal information through multi-stage processing, including: using three-dimensional convolution to perform sliding window calculation on frame blocks, extracting spatiotemporal local features, and generating feature maps; compressing the feature maps through pooling operations to cool them down while retaining key information, forming a temporal feature sequence; and injecting position encoding information into the feature sequence to clarify the absolute position and relative order of each frame on the time axis, thereby outputting frame embedding vectors.

[0010] (2) Spatiotemporal Local Features: In video analysis, "spatiotemporal local features" refer to the spatial appearance information (such as object shape and texture) and temporal motion information (such as object movement and action changes) extracted simultaneously from a small segment (i.e., "frame block") of consecutive video frames. Through operations such as 3D convolution, the model can capture the joint change patterns of local regions in time and space in the video, thereby providing richer feature representations for subsequent temporal alignment and defect detection.

[0011] (3) Feature Map: A "feature map" is the output of a convolutional neural network (especially a 3D convolution). It is a multidimensional array (usually 3D or 4D), where each position (or channel) corresponds to the activation intensity of a specific visual or temporal pattern in the input data. In this paper, after performing a 3D convolution on each frame block, a feature map is generated. This map integrates the spatiotemporal information of all frames within the frame block and serves as the basis for subsequent pooling and serialization operations.

[0012] (4) Temporal feature sequence: "Temporal feature sequence" refers to the sequence formed by compressing the feature map corresponding to each frame block into a feature vector through pooling (such as global average pooling) and arranging the frame blocks in the temporal order of the original video. This sequence not only preserves the semantic features of each time segment, but also maintains the temporal order relationship between frames. It is a structured representation before being input into temporal models such as Transformer.

[0013] (5) Positional encoding information: "Positional encoding information" is a technique used to explicitly inject positional information of elements in a sequence into the model. Since models such as Transformer do not have the ability to perceive the input order, by adding position-related encoding (usually fixed encoding based on sine / cosine functions or learnable encoding) to each vector in the temporal feature sequence, the model can distinguish frames at different time steps, thereby understanding the sequential relationship and relative distance between frames, which is crucial for temporal alignment and drift detection.

[0014] (6) Path drift value: a key indicator used to quantify the timing synchronization deviation between two video streams, providing a data basis for the detection and diagnosis of synchronization problems.

[0015] (7) Dynamic time warping: A classic algorithm used to measure the similarity between two time series of different lengths or with nonlinear deformation and to find their optimal alignment path.

[0016] (8) Frame embedding vector: It is a numerical representation that maps a single frame image and its temporal context together into a low-dimensional, dense vector space through T-Net.

[0017] (9) Frame block: refers to a short time sequence of fixed length extracted from a continuous video stream, which is the basic unit for T-Net processing.

[0018] (10) Local cost matrix: In the Dynamic Time Warping (DTW) algorithm, it is a two-dimensional matrix used to quantify the frame-by-frame matching cost between two video frame sequences.

[0019] (11) Cumulative cost matrix: It is obtained by dynamic programming in the dynamic time warping (DTW) algorithm. It not only records the minimum cumulative cost from the starting point of the path to each point, but also reveals the key alignment path information through its numerical distribution: When multiple equal local minimum values ​​appear in the same row of the matrix, it indicates a multi-peak scenario - there are multiple candidate alignment paths. At this time, the global optimal solution needs to be determined by comparing the total cumulative cost of each path.

[0020] (12) Optimal alignment path: In the Dynamic Time Warping (DTW) algorithm, it is a continuous path that passes through the cumulative cost matrix and can achieve optimal matching of two sequences with the minimum total cost.

[0021] (13) End frame number of the optimal alignment path: refers to the end frame number of the target video stream (such as the viewer's end) in the optimal alignment path found by the Dynamic Time Warping (DTW) algorithm, usually denoted as j*.

[0022] In technologies such as video streaming and cloud gaming, video data needs to be transmitted through a complete link: "server → client → CDN (Content Delivery Network) → viewer." During this process, CDN nodes often become the critical link where problems occur, leading to poor user experiences such as screen tearing, frame skipping, and audio-visual asynchrony at the streaming end. Existing detection technologies mainly rely on log timestamp comparison or content verification methods based on hash values, which have certain limitations: most solutions only cover the "server..." The transmission path between "clients", for "clients" The transmission defects in the critical "viewer end" segment lack effective detection capabilities; although hash value verification can identify whether the content has been tampered with or damaged, it is difficult to capture timing drift problems caused by frame out-of-order, duplication or loss; the commonly used recovery mechanisms, such as data packet retransmission and audio white noise filling alignment, are mostly focused on network layer repair or single audio-visual delay compensation, lacking the visual perception capability of the overall synchronization status of multiple video streams, and also unable to achieve drift positioning and global synchronization across video streams.

[0023] Please refer to Figure 1 This invention provides a method for detecting frame synchronization defects, comprising the following steps: Acquire at least two video frame sequences, input each video frame sequence into the transform network, and output the corresponding frame embedding vector. The path drift value between each pair of video frame sequences in the at least two video frame sequences is calculated based on the frame embedding vector, and the global path drift value is determined based on the path drift value. If the global path drift value is greater than the preset frame difference threshold, it is determined that there is a frame synchronization defect.

[0024] As can be seen from the above description, the beneficial effects of the present invention are as follows: by acquiring at least two video frame sequences and inputting them into a transform network, and outputting corresponding frame embedding vectors, unified feature extraction of multiple video data is achieved, enhancing the representation ability of inter-frame temporal relationships and laying the foundation for subsequent path drift value calculation; the path drift value between each pair of video frame sequences is calculated based on the frame embedding vector, and the global path drift value is determined based on the path drift value, which can accurately quantify the temporal deviation between multiple video streams; by comparing the global path drift value with a preset frame difference threshold, accurate judgment of frame synchronization defects is achieved, ensuring the reliability and stability of video transmission.

[0025] Furthermore, each video frame sequence is input into the transform network, which outputs the corresponding frame embedding vector, including: Each video frame sequence is divided into frame blocks of a preset length. Three-dimensional convolution is performed on each frame block to extract spatiotemporal local features and generate a feature map. The feature map is then converted into a temporal feature sequence through pooling. Position encoding information is injected into the temporal feature sequence, and the corresponding frame embedding vector is output.

[0026] As described above, by dividing the video frame sequence into frame blocks of a preset length and performing three-dimensional convolution operations on each frame block, unified feature extraction of intra-frame visual content (spatial appearance) and inter-frame dynamic changes (temporal motion) is achieved, thereby forming spatiotemporal local features to obtain a feature map. The feature map is then converted into a temporal feature sequence through pooling operations, which improves computational efficiency while retaining key information and injects positional encoding information, enabling the model to accurately perceive the temporal relationship between frames, thereby outputting a frame embedding vector with strong temporal representation capabilities, laying the foundation for subsequent path drift value calculation.

[0027] Further, the path drift value between every two video frame sequences in the at least two video frame sequences is calculated based on the frame embedding vector, and a global path drift value is determined based on the path drift value, including: Based on the frame embedding vectors corresponding to the two video frame sequences, a similarity matrix between the two video frame sequences is calculated, and a local cost matrix is ​​calculated based on the similarity matrix and the temporal distance between the two video frame sequences. Based on the local cost matrix, a cumulative cost matrix is ​​constructed using a dynamic time warping algorithm. The optimal alignment path is determined based on the cumulative cost matrix, the endpoint frame number of the optimal alignment path is obtained, the path drift value between the two video frame sequences is calculated, and the path drift value with the largest absolute value is taken as the global path drift value. The path drift values ​​include the path drift values ​​between the server-side video frame sequence and the client-side video frame sequence, the path drift values ​​between the server-side video frame sequence and the viewer-side video frame sequence, and the path drift values ​​between the client-side video frame sequence and the viewer-side video frame sequence.

[0028] As described above, by calculating the similarity matrix between each pair of video frame sequences based on frame embedding vectors and constructing a local cost matrix in conjunction with temporal distance, the content association and temporal differences between different video streams can be effectively quantified. The dynamic time warping algorithm is used to construct a cumulative cost matrix based on the local cost matrix, which can accurately capture the optimal alignment path between the two video streams, achieving robust matching against non-uniform temporal drift. The path drift value is calculated by obtaining the endpoint frame number of the optimal alignment path, accurately reflecting the cumulative frame deviation between the two video streams. Selecting the path drift value with the largest absolute value among multiple path drift values ​​as the global path drift value allows focusing on the most severe synchronization deviation, ensuring accurate assessment of the overall synchronization status and providing a reliable basis for subsequent synchronization defect judgment.

[0029] Further, calculating the path drift value between every two video frame sequences in the at least two video frame sequences based on the frame embedding vector, and determining the global path drift value based on the path drift value, also includes: If multiple column indices in the same row of the cumulative cost matrix have the minimum value for that row, it is determined to be a multi-peak scenario. In the multi-peak scenario, the total cost of each candidate alignment path is calculated, and the path with the minimum total cost is selected as the optimal alignment path. The total cost is the sum of all cumulative costs on the corresponding path.

[0030] As described above, by detecting whether there are multiple column index values ​​that are all minimum in the same row of the cumulative cost matrix, accurate identification of multi-peak scenarios can be achieved, effectively avoiding the risk of misjudgment when multiple potential alignment paths coexist in traditional methods. On this basis, the total cost of each candidate alignment path is further calculated and the path with the minimum total cost is selected as the optimal alignment path. Even in complex multi-peak situations, the globally optimal temporal alignment scheme can be selected, significantly improving the robustness and accuracy of frame sequence drift detection. Furthermore, if the global path drift value is greater than a preset frame difference threshold, the method further includes: Extract a preset number of video data frames before and after the current moment, generate a diagnostic package based on the video data, write the diagnostic package into the detection log, and trigger a synchronization anomaly alarm.

[0031] As described above, when the global path drift value exceeds the limit, video data of a preset number of frames before and after the current moment is automatically extracted to generate a diagnostic package, which completely preserves the key information of the abnormal scene; the diagnostic package is written to the detection log and a synchronization anomaly alarm is triggered, realizing the persistent recording and real-time early warning of fault information, which significantly improves the response speed and diagnostic efficiency of frame synchronization problems.

[0032] Please refer to Figure 5 Another embodiment of the present invention provides a frame synchronization defect detection terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the frame synchronization defect detection method described above.

[0033] The frame synchronization defect detection method and terminal of the present invention are applicable to video frame synchronization detection, and can detect synchronization defects between multiple video streams in real time, ensuring the reliability and stability of video transmission. The following is a description of specific implementation methods: Please refer to Figure 1 and Figure 2 One embodiment of the present invention is as follows: A method for detecting frame synchronization defects includes the following steps: S1. Obtain at least two video frame sequences, input each video frame sequence into the transform network, and output the corresponding frame embedding vector, specifically including: Each video frame sequence is divided into frame blocks of a preset length. Three-dimensional convolution is performed on each frame block to extract spatiotemporal local features and generate a feature map. The feature map is then converted into a temporal feature sequence through pooling. Position encoding information is injected into the temporal feature sequence, and the corresponding frame embedding vector is output.

[0034] In this embodiment, three video frame sequences are set up, including a server-side video frame sequence, a client-side video frame sequence, and a viewer-side video frame sequence. Each video frame sequence is divided into frame blocks of a preset length L, forming input data of size L×C×H×W, where L defaults to 16 and can be adjusted within the range of 8 to 24, C is the number of RGB channels (3), and H and W are both 224 pixels. The frame blocks are input into a Transformer network T-Net, where 3×3×3 convolution kernels are used to perform three-dimensional convolution operations with a stride of 1 to extract spatiotemporal local features. After ReLU activation and batch normalization, a feature map with dimensions of L×64×224×224 is generated. Global average pooling is performed on the feature map to convert it into a temporal feature sequence with dimensions of L×64. The temporal feature sequence is input into a Transformer encoder (3 in series, each encoder has 8 attention heads, each head with a dimension of 16) to inject positional encoding information into the temporal feature sequence. The positional encoding calculation formula is: PE(pos,2i) = sin(pos / 10000^(2i / d)) PE(pos,2i+1) = cos(pos / 10000^(2i / d)) In the formula, pos represents the frame position (0 to L-1), i is the dimension index, d is the embedding dimension (fixed at 128), ^ is the exponentiation operator, and the final output is a frame embedding vector with dimension L×128.

[0035] S2. Calculate the path drift value between every two video frame sequences in the at least two video frame sequences according to the frame embedding vector, and determine the global path drift value based on the path drift value, specifically including S2.1-S2.2.

[0036] S2.1. Based on the frame embedding vectors corresponding to the two video frame sequences, calculate the similarity matrix between the two video frame sequences, and calculate the local cost matrix based on the similarity matrix and the temporal distance between the two video frame sequences. Based on the local cost matrix, a cumulative cost matrix is ​​constructed using a dynamic time warping algorithm. The optimal alignment path is determined based on the cumulative cost matrix, the endpoint frame number of the optimal alignment path is obtained, the path drift value between the two video frame sequences is calculated, and the path drift value with the largest absolute value is taken as the global path drift value. The path drift values ​​include the path drift values ​​between the server-side video frame sequence and the client-side video frame sequence, the path drift values ​​between the server-side video frame sequence and the viewer-side video frame sequence, and the path drift values ​​between the client-side video frame sequence and the viewer-side video frame sequence.

[0037] Please refer to Figure 3In this embodiment, path drift values ​​are calculated between three sets of video frame sequences: server-client, server-viewer, and client-viewer, to evaluate transmission link synchronization. Based on the frame embedding vectors corresponding to the two video frame sequences, a similarity matrix is ​​calculated according to a formula to accurately reflect the semantic relevance of the content of each frame between different paths. S_pq(i,j)=dot(E_p(i),E_q(j) / (||E_p(i)||2·||E_q(j)||2) In the formula, p and q both represent path indices, p,q∈{s,c,v}, s: server, c: client, v: viewer; S_pq(i,j) represents the similarity between path p and q at frames i and j, dot represents the vector dot product, E_p(i) represents the feature vector of p at frame i, E_q(j) represents the feature vector of q at frame j, ||E_p(i)||2 represents the 2-norm (Euclidean norm) of vector E_p(i), and ||E_q(j)||2 represents the 2-norm (Euclidean norm) of vector E_q(j).

[0038] The local cost matrix is ​​constructed by combining the temporal distance with a weighted formula: C(i,j)=w1·|ij|+w2·(1-S_pq(i,j)), where i and j represent DTW (Dynamic Time Warping). The grid row and column indices (frame numbers) are defined. C(i,j) represents the local cost function, and w1 and w2 both represent DTW cost weights. Optionally, w1=0.4, w2=0.6, and w1+w2=1. A cumulative cost matrix D(i,j) is constructed using the dynamic time warping algorithm. The optimal alignment path is obtained using D(i,j)=C(i,j)+min{D(i-1,j),D(i,j-1),D(i-1,j-1)}, yielding the endpoint column number j*. The path drift value Δ_pq(i)=j*-i is calculated, and the global path drift value Δ(t) = max{|Δ_sc|, |Δ_sv|, is selected from the three sets of path drift values ​​with the largest absolute value. |Δ_cv|}, where Δ_sc represents the path drift value between the server-side video frame sequence and the client-side video frame sequence, Δ_sv represents the path drift value between the server-side video frame sequence and the viewer-side video frame sequence, and Δ_cv represents the path drift value between the client-side video frame sequence and the viewer-side video frame sequence. This ensures that the most serious synchronization deviation in the transmission link can be accurately captured, providing a reliable basis for subsequent synchronization defect judgment.

[0039] S2.2 If multiple column indices in the same row of the cumulative cost matrix have values ​​that are the minimum values ​​in that row, then it is determined to be a multi-peak scenario; In the multi-peak scenario, the total cost of each candidate alignment path is calculated, and the path with the minimum total cost is selected as the optimal alignment path. The total cost is the sum of all cumulative costs on the corresponding path.

[0040] In this embodiment, when multiple column indices in the same row of the cumulative cost matrix D(i,j) have values ​​corresponding to the minimum values ​​in that row, it is determined to be a multi-peak scenario with multiple candidate alignment paths. For the detected multi-peak situation, the total cost Sum_D = ∑D(i,j) on each candidate alignment path is calculated, which is the sum of the cumulative costs at all positions traversed by the path. The candidate path with the smallest Sum_D value is selected as the optimal alignment path, and its endpoint frame number j* is recorded. The accurate path drift value is calculated using Δ_pq(i) = j* - i. This method avoids interference from similar frames and improves the reliability and accuracy of temporal drift detection.

[0041] S3. If the global path drift value is greater than the preset frame difference threshold, it is determined that there is a frame synchronization defect, and video data of a preset number of frames before and after the current time is extracted. A diagnostic package is generated based on the video data, the diagnostic package is written into the detection log, and a synchronization anomaly alarm is triggered.

[0042] Please refer to Figure 4 In this embodiment, when the global path drift value Δ(t) exceeds the preset frame difference threshold τ = 2 frames (approximately 33ms), a frame synchronization defect is determined to exist, and a rollback closed-loop mechanism is initiated: 30 frames of video data before and after the current time t (t-30 to t+30) are extracted and encapsulated in MP4 format (H.264 encoding, YUV420P pixel format). Simultaneously, a JSON log containing detailed information about the synchronization defect is generated. The route field in the JSON log automatically identifies the defective link based on the maximum absolute value of the three drift values—when |Δ_sv(t)| is at its maximum, "SV" is marked to indicate an abnormal link from the server to the viewer. Finally, the diagnostic package is written to the detection log and a synchronization anomaly alarm is triggered, providing a reliable basis for subsequent fault location and system optimization.

[0043] In this embodiment, the following specific application scenarios are also provided: In cloud gaming platform quality assurance scenarios, by continuously collecting server rendering stream Fs, client decoding stream Fc, and audience distribution stream Fv, the system can automatically detect timing defects such as rendering frame skipping and stuttering. When the client stream is detected to be 3 frames behind the server stream (Δ(t)=3), the system automatically triggers a rollback mechanism to generate a diagnostic package containing key information such as route="SC" and delta_max=3, providing quality engineers with visual evidence of faults and helping to quickly locate problems such as client decoding module timeouts.

[0044] In esports live streaming scenarios, the player operation rendering stream Fs, the on-site OB system output stream Fc, and the Twitch live streaming platform transcoding stream Fv are monitored synchronously. When the drift value Δ_sv between the player stream and the audience stream is detected to be greater than 1 frame for an extended period, an alarm is immediately sent to the broadcasting station, and the audience stream latency is controlled to within 1 frame by adjusting the CDN node buffer. At the same time, the generated rollback evidence can be used for the CDN service provider's SLA audit after the event.

[0045] In the VR / AR cloud rendering system, the three-stream extension is applied to the left-eye rendering stream Fs, the right-eye rendering stream Fc, and the pose tracking auxiliary stream Fv. By calculating the left-eye-right-eye drift value Δ_lr, a stereo vision misalignment alarm is triggered when |Δ_lr|≥2 frames (approximately 33ms) is detected, and [t-30,t+30] frames are extracted to generate an MP4+JSON packet to help engineers locate asynchronous rendering problems.

[0046] In cloud desktop and remote rendering scenarios, monitor the OpenGL instruction rendering results Fs, user terminal display stream Fc, and remote management console stream Fv of the GPU virtualization instance; when the feature similarity S_sc suddenly drops from 0.95 to 0.2, it indicates that the content is distorted. Automatically mark the time period and roll back the screen recording. Combined with MP4 video verification, it is confirmed that the screen distortion defect is caused by GPU memory overflow.

[0047] In summary, this invention provides a method for detecting frame synchronization defects. By acquiring at least two video frame sequences and inputting them into a transform network, the corresponding frame embedding vectors are output, enabling unified feature extraction from multiple video data streams. This enhances the representation of temporal relationships between frames and lays the foundation for subsequent path drift value calculation. The method calculates the path drift value between every two video frame sequences based on the frame embedding vectors, and determines the global path drift value based on these values. This accurately quantifies the temporal deviation between multiple video streams. Furthermore, in multi-peak scenarios, the optimal alignment path is obtained by calculating the total cost, improving the reliability and accuracy of frame sequence drift detection. By comparing the global path drift value with a preset frame difference threshold, accurate determination of frame synchronization defects is achieved. A diagnostic packet is automatically generated by extracting a preset number of video data frames before and after the current moment. The diagnostic packet is written to the detection log and triggers a synchronization anomaly alarm, enabling persistent recording and real-time warning of fault information. This significantly improves the response speed and diagnostic efficiency for frame synchronization problems, thereby ensuring the reliability and stability of video transmission.

[0048] According to another aspect of the invention, Figure 5 This is a schematic diagram illustrating a frame synchronization defect detection terminal according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the frame synchronization defect detection method described above.

[0049] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for detecting frame synchronization defects, characterized in that, Including the following steps: Acquire at least two video frame sequences, input each video frame sequence into the transform network, and output the corresponding frame embedding vector. The path drift value between each pair of video frame sequences in the at least two video frame sequences is calculated based on the frame embedding vector, and the global path drift value is determined based on the path drift value. If the global path drift value is greater than the preset frame difference threshold, it is determined that there is a frame synchronization defect.

2. The frame synchronization defect detection method according to claim 1, characterized in that, Each video frame sequence is input into the transform network, which outputs the corresponding frame embedding vector, including: Each video frame sequence is divided into frame blocks of a preset length. Three-dimensional convolution is performed on each frame block to extract spatiotemporal local features and generate a feature map. The feature map is then converted into a temporal feature sequence through pooling. Position encoding information is injected into the temporal feature sequence, and the corresponding frame embedding vector is output.

3. The frame synchronization defect detection method according to claim 1, characterized in that, Calculate the path drift value between every two video frame sequences in the at least two video frame sequences based on the frame embedding vector, and determine the global path drift value based on the path drift value, including: Based on the frame embedding vectors corresponding to the two video frame sequences, a similarity matrix between the two video frame sequences is calculated, and a local cost matrix is ​​calculated based on the similarity matrix and the temporal distance between the two video frame sequences. Based on the local cost matrix, a cumulative cost matrix is ​​constructed using a dynamic time warping algorithm. The optimal alignment path is determined based on the cumulative cost matrix, the endpoint frame number of the optimal alignment path is obtained, the path drift value between the two video frame sequences is calculated, and the path drift value with the largest absolute value is taken as the global path drift value. The path drift values ​​include the path drift values ​​between the server-side video frame sequence and the client-side video frame sequence, the path drift values ​​between the server-side video frame sequence and the viewer-side video frame sequence, and the path drift values ​​between the client-side video frame sequence and the viewer-side video frame sequence.

4. The frame synchronization defect detection method according to claim 3, characterized in that, Calculating the path drift value between every two video frame sequences in the at least two video frame sequences based on the frame embedding vector, and determining the global path drift value based on the path drift value, further includes: If multiple column indices in the same row of the cumulative cost matrix have the minimum value for that row, it is determined to be a multi-peak scenario. In the multi-peak scenario, the total cost of each candidate alignment path is calculated, and the path with the minimum total cost is selected as the optimal alignment path. The total cost is the sum of all cumulative costs on the corresponding path.

5. The frame synchronization defect detection method according to claim 1, characterized in that, If the global path drift value is greater than a preset frame difference threshold, the method further includes: Extract a preset number of video data frames before and after the current moment, generate a diagnostic package based on the video data, write the diagnostic package into the detection log, and trigger a synchronization anomaly alarm.

6. A frame synchronization defect detection terminal, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it performs the following steps: Acquire at least two video frame sequences, input each video frame sequence into the transform network, and output the corresponding frame embedding vector. The path drift value between each pair of video frame sequences in the at least two video frame sequences is calculated based on the frame embedding vector, and the global path drift value is determined based on the path drift value. If the global path drift value is greater than the preset frame difference threshold, it is determined that there is a frame synchronization defect.

7. A frame synchronization defect detection terminal according to claim 6, characterized in that, Each video frame sequence is input into the transform network, which outputs the corresponding frame embedding vector, including: Each video frame sequence is divided into frame blocks of a preset length. Three-dimensional convolution is performed on each frame block to extract spatiotemporal local features and generate a feature map. The feature map is then converted into a temporal feature sequence through pooling. Position encoding information is injected into the temporal feature sequence, and the corresponding frame embedding vector is output.

8. A frame synchronization defect detection terminal according to claim 6, characterized in that, Calculate the path drift value between every two video frame sequences in the at least two video frame sequences based on the frame embedding vector, and determine the global path drift value based on the path drift value, including: Based on the frame embedding vectors corresponding to the two video frame sequences, a similarity matrix between the two video frame sequences is calculated, and a local cost matrix is ​​calculated based on the similarity matrix and the temporal distance between the two video frame sequences. Based on the local cost matrix, a cumulative cost matrix is ​​constructed using a dynamic time warping algorithm. The optimal alignment path is determined based on the cumulative cost matrix, the endpoint frame number of the optimal alignment path is obtained, the path drift value between the two video frame sequences is calculated, and the path drift value with the largest absolute value is taken as the global path drift value. The path drift values ​​include the path drift values ​​between the server-side video frame sequence and the client-side video frame sequence, the path drift values ​​between the server-side video frame sequence and the viewer-side video frame sequence, and the path drift values ​​between the client-side video frame sequence and the viewer-side video frame sequence.

9. A frame synchronization defect detection terminal according to claim 8, characterized in that, Calculating the path drift value between every two video frame sequences in the at least two video frame sequences based on the frame embedding vector, and determining the global path drift value based on the path drift value, further includes: If multiple column indices in the same row of the cumulative cost matrix have the minimum value for that row, it is determined to be a multi-peak scenario. In the multi-peak scenario, the total cost of each candidate alignment path is calculated, and the path with the minimum total cost is selected as the optimal alignment path. The total cost is the sum of all cumulative costs on the corresponding path.

10. A frame synchronization defect detection terminal according to claim 6, characterized in that, If the global path drift value is greater than a preset frame difference threshold, the method further includes: Extract a preset number of video data frames before and after the current moment, generate a diagnostic package based on the video data, write the diagnostic package into the detection log, and trigger a synchronization anomaly alarm.

Images