A multi-screen projection synchronism detection method based on visual positioning

Abstract

The application discloses a multi-screen projection synchronization detection method based on visual positioning, and relates to the technical field of video image recognition. The method comprises the following steps: synchronously acquiring a video stream of a first screen, a video stream of a second screen and an audio stream; determining a picture delay between the first screen and the second screen by a feature matching method based on the video stream of the first screen and the video stream of the second screen; determining an audio-picture delay corresponding to each screen by a cross-correlation calculation method based on the audio stream, the video stream of the first screen and the video stream of the second screen; and determining a synchronization detection result of multi-screen projection based on the picture delay and the audio-picture delay. The application realizes the automatic detection of the synchronization performance of a multi-screen system such as a vehicle machine.

Description

Technical Field

[0001] This application relates to the field of video image recognition technology, and in particular to a method for detecting the synchronization of multi-screen projection based on visual positioning. Background Technology

[0002] With the increasing popularity of smart cockpits and multi-screen interactive systems, in-vehicle infotainment systems often employ a display scheme where the main screen and secondary screens collaboratively project the same content. Existing technologies for detecting display anomalies in in-vehicle systems primarily focus on judging image faults within a single screen, such as identifying black screens or stuttering by comparing differences between adjacent frames. However, these methods can only monitor the display status of a single screen and cannot quantitatively detect the synchronization delay between multiple screens, nor can they effectively evaluate the synchronization relationship between audio and video. This results in an inability to comprehensively assess system synchronization performance in multi-screen projection scenarios, limiting the automated detection and optimization of the in-vehicle system's image and audio-visual synchronization quality. Therefore, a vision-based multi-screen projection synchronization detection method is urgently needed to address the aforementioned technical problems. Summary of the Invention

[0003] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. This summary section is not intended to limit the key and essential technical features of the claimed technical solutions, nor is it intended to define the scope of protection of the claimed technical solutions.

[0004] In a first aspect, this application provides a method for detecting multi-screen projection synchronization based on visual positioning, including: Simultaneously acquire the video stream from the first screen, the video stream from the second screen, and the audio stream; Based on the video streams of the first screen and the second screen, the image delay between the first screen and the second screen is determined by a feature matching method. Based on the audio stream, the video stream of the first screen, and the video stream of the second screen, the audio-visual delay corresponding to each screen is determined by a cross-correlation calculation method. Based on the image delay and the audio-visual delay, the synchronization detection result of multi-screen projection is determined.

[0005] In some implementations, determining the image delay between the first screen and the second screen using a feature matching method based on the video streams of the first screen and the second screen includes: Based on the video streams of the first screen and the second screen, the first and second video frames corresponding to the content are determined by content search. Feature extraction operations are performed on the first video frame and the second video frame respectively to generate a first feature descriptor subset and a second feature descriptor subset; Perform feature matching operations on the first feature description subset and the second feature description subset to determine feature matching point pairs; Perform a false match elimination operation on the feature matching point pairs to determine the valid matching point pairs; Based on the timestamps associated with each of the valid matching points, the single-frame delay is calculated; The frame delay between the first screen and the second screen is determined based on the delay of multiple single frames.

[0006] In some implementations, the step of performing a false match removal operation on the feature matching point pairs to determine valid matching point pairs includes: The feature matching point pairs are processed based on a preset consistency estimation algorithm to determine candidate matching point pairs that satisfy geometric consistency. When the number of candidate matching point pairs is greater than a preset number threshold and the distribution of the candidate matching point pairs within the screen area meets a preset spatial distribution condition, the candidate matching point is determined to be a valid matching point pair.

[0007] In some implementations, determining the frame delay between the first screen and the second screen based on multiple single-frame frame delays includes: Based on the video stream of the first screen and / or the video stream of the second screen, determine the video scene type, wherein the video scene type includes a static scene or a dynamic scene; When the video scene type is the static scene, the median delay is calculated based on the single-frame image delay of multiple consecutive frames, and the median delay is determined as the image delay; or, When the video scene type is the dynamic scene, the same moving target in the video stream of the first screen and the video stream of the second screen is tracked based on the optical flow method to determine the target tracking delay; Based on the target tracking delay, the delays of the multiple single-frame images are verified to determine the image delay.

[0008] In some implementations, determining the audio-visual delay for each screen based on the audio stream, the video stream from the first screen, and the video stream from the second screen using a cross-correlation calculation method includes: The audio stream is processed to extract audio feature sequences; Visual features are extracted from the video streams of the first screen and the second screen respectively to generate a first visual feature sequence and a second visual feature sequence; The audio feature sequence is cross-correlated with the first visual feature sequence and the second visual feature sequence respectively to determine the global audio-visual delay for each screen. Based on the video stream from the first screen, the video stream from the second screen, and the audio stream, determine the event-level audio-visual delay; Based on the global audio-visual delay and the event-level audio-visual delay, the audio-visual delay corresponding to each screen is determined.

[0009] In some implementations, the step of performing cross-correlation calculations on the audio feature sequence with the first visual feature sequence and the second visual feature sequence respectively to determine the global audio-visual delay corresponding to each screen includes: A first cross-correlation calculation is performed on the audio feature sequence and the first visual feature sequence to generate a first cross-correlation function; The first relative time offset is determined based on the peak position of the first cross-correlation function; Based on the first relative time offset, the global audio-visual delay corresponding to the first screen is determined; Perform a second cross-correlation calculation on the audio feature sequence and the second visual feature sequence to generate a second cross-correlation function; The second relative time offset is determined based on the peak position of the second cross-correlation function; Based on the second relative time offset, the global audio-visual delay corresponding to the second screen is determined.

[0010] In some implementations, determining the event-level audio-visual delay based on the video stream from the first screen, the video stream from the second screen, and the audio stream includes: Based on preset acoustic features, audio event points and corresponding audio timestamps are identified from the audio stream; Based on preset image change features, a first visual event point and a first visual timestamp corresponding to the first visual event point are identified from the video stream of the first screen. Based on the audio timestamp and the first visual timestamp associated with the same logical event, calculate the event-level audio-visual delay corresponding to the first screen; Based on preset image change features, identify second visual event points and the second visual timestamps corresponding to the second visual event points from the video stream of the second screen; Based on the audio timestamp and the second visual timestamp associated with the same logical event, the event-level audio-visual delay corresponding to the second screen is calculated.

[0011] In some implementations, determining the audio-visual delay corresponding to each screen based on the global audio-visual delay and the event-level audio-visual delay includes: Based on multiple event-level audio-visual delays corresponding to the same screen, calculate event-level delay statistics; The deviation between the event-level latency statistics and the global audio-visual latency of the corresponding screen is calculated to obtain the deviation result; When the deviation result is less than a preset deviation threshold, the global audio-visual delay is determined as the audio-visual delay of the corresponding screen; or, when the deviation result is greater than or equal to the preset deviation threshold, the event-level delay statistics value is determined as the audio-visual delay of the corresponding screen.

[0012] In some implementations, determining the synchronization detection result of multi-screen projection based on the image delay and the audio-visual delay includes: The maximum synchronization delay is determined based on the screen delay, the audio-visual delay corresponding to the first screen, and the audio-visual delay corresponding to the second screen. The synchronization detection result is determined based on the preset delay threshold and the maximum synchronization delay.

[0013] In some embodiments, prior to the simultaneous acquisition of the video stream from the first screen, the video stream from the second screen, and the audio stream, the method further includes: Based on network time protocols or precision time protocols, the system clocks of video and audio acquisition devices are synchronized and calibrated.

[0014] Secondly, this application proposes a multi-screen projection synchronization detection device based on visual positioning, comprising: The media data acquisition unit is used to simultaneously acquire the video stream from the first screen, the video stream from the second screen, and the audio stream; The screen delay calculation unit is used to determine the screen delay between the first screen and the second screen based on the video stream of the first screen and the video stream of the second screen by using a feature matching method. The audio-visual delay calculation unit is used to determine the audio-visual delay corresponding to each screen based on the audio stream, the video stream of the first screen and the video stream of the second screen by using a cross-correlation calculation method. The media synchronization detection unit is used to determine the synchronization detection result of multi-screen projection based on the image delay and the audio-visual delay.

[0015] In summary, the vision-based multi-screen projection synchronization detection method provided in this application ensures a unified time reference for the multi-source data upon which subsequent analysis is based by synchronously acquiring the video streams of the first screen, the second screen, and the audio stream, thus providing a prerequisite for time difference calculation. Based on the video streams of the first and second screens, the image delay is determined using a feature matching method, directly quantifying the time difference in image content updates between the two display terminals, overcoming the limitation of existing technologies that can only perform single-screen anomaly detection. Based on the audio stream and the video stream of each screen, the respective audio-visual delay is determined using a cross-correlation calculation method, enabling the evaluation of the synchronization relationship between the same audio source and the images of different display terminals. The total synchronization error is determined based on the combined image delay and audio-visual delay, reflecting the overall synchronization performance of the multi-screen projection system in both image coordination and audio-visual coordination dimensions, providing an objective indicator for system optimization and quality evaluation. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit this specification. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A schematic flowchart of a multi-screen projection synchronization detection method based on visual positioning is provided in an embodiment of this application; Figure 2 This is a schematic diagram of a multi-screen projection synchronization detection device based on visual positioning, provided in an embodiment of this application. Detailed Implementation

[0017] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0018] The technical solution provided in this application is mainly applied to automated testing scenarios for the synchronization performance of smart terminal devices equipped with multi-screen display systems. Examples include detecting the synchronization of images between the driver's instrument panel, central control information screen, and passenger entertainment screen in an in-vehicle infotainment system, or verifying the synchronization quality when multiple display devices project the same multimedia content. This method achieves objective measurement of the system output signal through non-contact visual and acoustic acquisition, making it suitable for various stages such as performance debugging during product development, quality control during final inspection on the production line, and fault diagnosis during after-sales maintenance. It provides a means of testing the synchronization accuracy of multi-screen collaborative systems.

[0019] The first screen and the second screen in this application refer to two independent display devices or two display areas within the same device that require synchronization detection. In specific embodiments of this application, they typically correspond to the main display screen and the secondary display screen of an in-vehicle infotainment system, which are expected to maintain a high degree of timing consistency when receiving and rendering content from the same signal source.

[0020] The video stream and audio stream of this application refer to a continuous media data sequence containing timestamp information, synchronously acquired through an external camera and microphone. The video stream records the changes in visual display content between the first and second screens, while the audio stream records synchronous sound signals acquired from the device's audio output interface or speakers.

[0021] The screen delay in this application refers to the time difference between when the same visual content or event is displayed on the first screen and the second screen. Its value is calculated by comparing the timestamps of the corresponding image features in the two video streams. It is an indicator for measuring the synchronization performance of multi-screen images.

[0022] The audio-visual delay in this application refers to the time difference between the occurrence of the audio signal feature and the occurrence of the corresponding visual feature in the video signal of a specific screen during the same logical event (such as a prompt sound and the corresponding icon highlight). This application requires calculating the audio stream and the video streams of the first and second screens separately to obtain two independent audio-visual delay values, which are used to evaluate the synchronization relationship between the audio and each screen image.

[0023] The feature matching method of this application refers to a series of computer vision algorithms that extract key points with discriminative power and their descriptors from two frames of images and establish the correspondence between feature points in the two frames of images by comparing the similarity of the descriptors. For example, the ORB (Oriented FAST and Rotated BRIEF) algorithm aims to find image frames depicting the same content in two screen video streams.

[0024] The cross-correlation calculation method in this application is a signal processing technique used to measure the similarity between two signal sequences at different time offsets. In this application, the method is applied to calculate the cross-correlation function between an audio feature sequence and a visual feature sequence extracted from a video stream. By finding the time offset corresponding to the peak of the function, the overall time difference between the audio and the video, i.e., the global audio-visual delay, is estimated.

[0025] The synchronization detection results of this application are structured outputs generated based on the analysis results of screen delay and audio-visual delay. The results include at least the determination of the system synchronization status (such as synchronization or desynchronization), and may further include specific delay values, statistical indicators, trend analysis and decision information for triggering alarms, providing a basis for evaluating system performance and guiding optimization.

[0026] Please see Figure 1 The above is a flowchart illustrating a multi-screen projection synchronization detection method based on visual positioning, provided in an embodiment of this application. Specifically, it may include: S110, Simultaneously acquire the video stream of the first screen, the video stream of the second screen, and the audio stream; For example, this step involves deploying time-synchronized video and audio acquisition devices to record the output signals of the device under test during operation. The acquisition targets include the video stream from the first screen, the video stream from the second screen, and the unified audio stream output by the device. These data streams are all associated with precise timestamps during acquisition, ensuring that all video frames and audio sampling points have a unified time reference, thereby ensuring the alignment of the signals processed by subsequent time-difference-based image delay analysis and audio-visual delay analysis at the time origin.

[0027] S120. Based on the video streams of the first screen and the second screen, determine the image delay between the first screen and the second screen using a feature matching method. For example, this step analyzes the video streams of the first and second screens and uses feature matching technology to determine the image delay between them. This step involves identifying and associating image frames depicting the same visual content from the two video streams, and calculating the time difference in the display of the image content on the two screens by comparing the timestamps associated with these corresponding frames. This process enables a quantitative evaluation of the image synchronization performance within a multi-screen system, solving the problem that existing single-screen detection technologies cannot measure the timing consistency of collaborative display between screens, and providing image-dimensional delay data for judging the overall synchronization status of the system.

[0028] S130. Based on the audio stream, the video stream of the first screen, and the video stream of the second screen, the audio-visual delay corresponding to each screen is determined by the cross-correlation calculation method. For example, this step analyzes the synchronization of an audio stream acquired from the same audio output channel with the video streams of the first and second screens, respectively. Since the audio signal is a unified system output, but each screen's display may have independent rendering and transmission delays, it is necessary to evaluate the timing relationship between the audio and each screen's display separately. Using a cross-correlation calculation method, the audio feature sequence is compared with the visual feature sequences extracted from each screen's video stream, calculating the similarity between the two at different time offsets. This determines the time offset that maximizes the similarity, which represents the overall delay between the audio and the corresponding screen display, thus achieving a quantitative evaluation of the synchronization performance between the unified audio source and each independent display terminal.

[0029] S140. Based on image delay and audio-visual delay, determine the synchronization detection results of multi-screen projection.

[0030] For example, this step evaluates the overall synchronization performance of a multi-screen projection system by combining two quantitative indicators: screen latency and audio-visual latency. Specifically, it analyzes the screen latency and the corresponding audio-visual latency for each screen according to preset rules, selecting the largest value as the maximum synchronization latency representing the worst-case synchronization condition of the system. This maximum synchronization latency is compared with a pre-set latency threshold representing the upper limit of acceptable synchronization error, and a conclusion is drawn based on the comparison result regarding whether the system is synchronized. The screen latency data, the two audio-visual latency data, the calculated maximum synchronization latency, the judgment conclusion based on the threshold comparison, and necessary alarm information are collectively packaged to generate a synchronization detection result report, realizing an automated detection process from multi-source data acquisition to synchronization performance evaluation.

[0031] In summary, this application's embodiments establish a data foundation for timing analysis by synchronously acquiring multi-screen video and audio streams and assigning precise timestamps. Then, feature matching technology is used to identify and associate image frames describing the same visual content from the two video streams. By comparing timestamps, the screen latency between screens is directly quantified, filling the technical problem that existing single-screen detection technologies cannot assess the timing consistency of multi-screen collaborative displays. Furthermore, for the output audio signal, cross-correlation analysis is performed with the video stream of each screen to calculate the audio-visual latency between the audio and the images on each independent screen, achieving a quantitative assessment of the audio-visual synchronization relationship. Finally, by fusing the image latency and audio-visual latency data and making judgments based on preset thresholds, a detection result and structured report reflecting the system's synchronization performance in both image collaboration and audio-visual coordination dimensions are generated. The method proposed in this application's embodiments expands the detection scope from internal anomalies of a single screen to the timing consistency between multi-screen systems. Cross-validation using multi-source data improves the robustness of automated detection, providing a technical means for performance debugging of multi-screen projection systems.

[0032] In some instances, the process of synchronously acquiring the video stream from the first screen, the video stream from the second screen, and the audio stream also includes: Based on network time protocols or precision time protocols, the system clocks of video and audio acquisition devices are synchronized and calibrated.

[0033] For example, before synchronously acquiring the video stream of the first screen, the video stream of the second screen, and the audio stream, a system initialization and calibration operation is first performed to eliminate system errors introduced by hardware acquisition and lay the foundation for timing analysis. This operation specifically includes three steps: hardware deployment, camera calibration, and time synchronization.

[0034] In the hardware deployment phase, a high-frame-rate camera is fixed in a position that allows for simultaneous unobstructed recording of both the first and second screens. An audio acquisition device is connected to capture the unified audio output of the device under test (DUT), ensuring the DUT is in multi-screen projection mode. In the camera calibration phase, a pre-set checkerboard calibration board is used to perform distortion correction and spatial mapping calculations, establishing a one-to-one coordinate transformation relationship between each pixel in the camera's imaging plane and its actual physical location on the screen, thereby eliminating the impact of lens distortion on image positioning. In the time synchronization phase, based on a network time protocol or a precise time protocol, the system clocks of the video acquisition device, audio acquisition device, and the DUT are synchronized to the same time base. This ensures that the timestamps associated with each subsequently acquired video frame and audio data segment are based on a unified start time and timing unit, achieving alignment of multi-source heterogeneous data in the time dimension.

[0035] It should be noted that the device under test (DUT) refers to the core host or signal source device of the entire multi-screen display system under test. Specifically, in the context of this application using an in-vehicle infotainment system as an example, the DUT is the in-vehicle host (or in-vehicle infotainment system host). It is the source responsible for generating and outputting the audio and video signals to be tested: it runs applications or plays media content internally, generating video and graphic signals that need to be displayed synchronously (sent to the first and second screens respectively) and corresponding audio signals (output through a unified audio channel). Therefore, the target and objective of the testing method in this application is to evaluate the synchronization performance between images and between audio and video caused by the internal system or output links of the DUT when processing and outputting multiple signals.

[0036] In summary, the embodiments of this application provide physical conditions for data acquisition through hardware deployment, ensuring that visual and acoustic signals are completely captured; the camera calibration solves the problem of image geometric distortion caused by lens optical characteristics, so that the screen coordinates extracted from the video stream truly reflect the physical space of the display terminal, providing input for image feature-based matching; and the precise time synchronization protocol unifies the clock source of all acquisition devices, ensuring that the timestamps on which subsequent calculations of screen delay and audio-visual delay depend are comparable, avoiding measurement errors caused by clock drift between devices.

[0037] In some instances, it also includes: In the multi-source data acquisition step, to achieve quantitative analysis of the synchronization of the multi-screen projection system, it is first necessary to synchronously acquire media data sources with a unified time reference. This process is accomplished by deploying high frame rate cameras and audio acquisition devices that have undergone time synchronization calibration. The cameras simultaneously record the display content of the first and second screens at a fixed acquisition frame rate of no less than 60 frames per second, ensuring the precise timing of image changes.

[0038] During video acquisition, each frame of the image is associated with a precise timestamp originating from the camera's hardware clock or a synchronized system clock. The video stream containing timestamp metadata is stored in an uncompressed or lightly compressed format, or frame-by-frame is saved as an image sequence that retains complete information, laying the data foundation for time-stamp-based frame-level comparison.

[0039] In terms of audio acquisition, the audio acquisition device synchronously captures the sound signal emitted by the audio output interface or speaker of the device under test. Its sampling rate is set to match the video frame rate to ensure that an integer number of audio samples are contained within each video frame period, achieving temporal alignment between the audio stream and the video stream. The acquired audio data is stored in WAV or PCM raw format, and each audio data segment is associated with its acquisition timestamp, forming a multi-source media data set that is synchronized with the two video streams in the temporal dimension and has a resolvable temporal relationship.

[0040] In some instances, based on the video streams of the first and second screens, the image delay between the first and second screens is determined using feature matching methods, including: Based on the video streams from the first screen and the second screen, the first and second video frames corresponding to the content are determined by content search. Feature extraction operations are performed on the first video frame and the second video frame respectively to generate a first feature descriptor subset and a second feature descriptor subset; Perform feature matching operations on the first feature descriptor subset and the second feature descriptor subset to determine feature matching point pairs; Perform a false match elimination operation on feature matching point pairs to determine valid matching point pairs; Calculate the single-frame delay based on the timestamps associated with each valid matching point pair. The frame delay between the first screen and the second screen is determined based on the delay of multiple single frames.

[0041] For example, based on the video streams of the first and second screens, content search is used to determine the first and second video frames corresponding to the content. This step aims to locate image frame pairs depicting the same visual content from two independently acquired video streams. Specifically, since the first and second screens are expected to display synchronized content, their video streams will contain a large number of highly similar frames. By performing temporal content comparison and analysis on the two video streams, such as calculating the differences between consecutive frames or performing inter-frame similarity matching, key frames in the first screen video stream that show the first appearance or significant change of a specific piece of content are identified, and the corresponding frames in the second screen video stream that are most similar to this content are searched. This pair of frames is then determined as the basis for subsequent feature matching and delay calculation, i.e., the first and second video frames corresponding to the content, logically linking the two asynchronous video streams in terms of content.

[0042] Feature extraction is performed on the first and second video frames respectively, generating a first feature descriptor set and a second feature descriptor set. The purpose of this step is to transform the image content into a computable and discriminative mathematical representation. Feature extraction typically involves detecting keypoints in the image (such as corners and edge intersections) and calculating a feature vector (i.e., a descriptor) for each keypoint, describing the surrounding local image information. For example, using the ORB algorithm, the keypoint locations are first detected using the oFAST algorithm, and then a binary string descriptor is generated for each keypoint using the rBRIEF algorithm. This operation is performed on the first video frame to obtain all its keypoints and the corresponding first feature descriptor set; the same operation is performed on the second video frame to obtain the second feature descriptor set. These descriptors possess rotation invariance and a certain degree of noise resistance, and can characterize the essential features of the image.

[0043] Feature matching is performed on the first and second feature descriptor subsets to determine feature matching point pairs. This step establishes the correspondence between key points in the first and second video frames. By calculating the distance between each pair of descriptors in the first and second feature descriptor subsets (e.g., Hamming distance for binary descriptors), the nearest or nearest neighbor distance ratio (NNR) corresponding key point in the second video frame can be found for each key point in the first video frame, thus initially forming a series of feature matching point pairs. Each matching point pair consists of a two-dimensional pixel coordinate point in the first video frame and a two-dimensional pixel coordinate point in the second video frame that is considered to correspond to it. This step establishes a potential correspondence between the two frames at the pixel level, but these relationships may contain erroneous matches due to image noise, repetitive textures, or viewpoint differences.

[0044] False matches are removed from feature matching point pairs to determine valid matching point pairs. Since the initial matching results contain false matches, direct use would lead to computational errors; therefore, filtering is necessary. This operation filters outliers based on geometric consistency constraints. For example, a random sampling consensus algorithm is used. This algorithm randomly selects the smallest subset from the matching point pairs to estimate the geometric transformation model (such as the homography matrix), then calculates the fitting error between all matching point pairs and the model, marking point pairs with errors less than a preset threshold as inliers (i.e., candidate correct matches). This process is iterated until the largest group of inliers that satisfies model consistency is retained. Furthermore, the retained inliers need further filtering. Only when their number exceeds a preset threshold to ensure statistical significance, and the spatial distribution of these points within the screen display area meets a preset dispersion condition (e.g., not all clustered in a local area), are they determined as valid matching point pairs.

[0045] The single-frame latency is calculated based on the timestamps associated with each valid matching point pair. During the data acquisition phase, each video frame is associated with a precise timestamp. For each valid matching point pair, the timestamps of its first and second video frames are obtained. Since the matching point pair represents the same visual content, the difference between these two timestamps reflects the instantaneous time difference of the content displayed on the two screens. Calculating this difference yields the instantaneous latency observation based on this pair of matching points. Typically, for a video frame to be analyzed, a statistical measure (such as the median or average) is taken from the multiple instantaneous latency values ​​calculated for all valid matching point pairs, which is used as the single-frame latency represented by that pair of video frames. This value quantifies the display lag (or lead) time of the second screen relative to the first screen at a specific moment or for a specific scene content.

[0046] The image delay between the first and second screens is determined based on multiple single-frame image delays. Delay measurements of individual frame pairs may be affected by transient interference or specific content characteristics, lacking overall representativeness. Therefore, it is necessary to calculate multiple single-frame image delay values ​​based on multiple pairs of video frames selected consecutively or by sampling over a period of time. These values ​​are then statistically analyzed to obtain an overall image delay estimate. The specific strategy depends on the characteristics of the video content. For example, for static scenes, the single-frame delay of consecutive frames may fluctuate less, and its average or median can be directly calculated as the final image delay; for dynamic scenes, it may be necessary to combine other methods such as optical flow tracing for supplementary verification and fusion. The finally determined image delay is a comprehensive statistical indicator that characterizes the statistically significant average time offset of the content displayed on the second screen relative to the first screen during the testing period.

[0047] In summary, this application's embodiments semantically associate two asynchronous video streams through content search, providing a comparison object for synchronization analysis. Feature extraction and descriptor generation transform visual content into feature representations, overcoming the sensitivity to lighting and noise inherent in direct pixel comparison. Feature matching establishes preliminary pixel-level correspondences, providing a basis for time difference calculation. Mismatch elimination based on geometric consistency and validity determination based on quantity and spatial distribution improve the reliability of matching results, filtering out noise and erroneous data and ensuring data quality for subsequent delay calculations. Sampling of instantaneous synchronization states is achieved by calculating single-frame delay based on the timestamps of valid matching points. Determining the overall image delay by integrating multiple single-frame delays ensures the stability and representativeness of the evaluation results, accurately reflecting the overall level of image synchronization performance between the two screens.

[0048] In some instances, a false match removal operation is performed on feature matching point pairs to determine valid matching point pairs, including: Based on the preset consistency estimation algorithm, the feature matching point pairs are processed to determine the candidate matching point pairs that satisfy geometric consistency; When the number of candidate matching point pairs is greater than a preset threshold and the distribution of candidate matching point pairs within the screen area meets the preset spatial distribution conditions, the candidate matching point pairs are determined to be valid matching point pairs.

[0049] For example, based on the feature matching point pairs determined by the preliminary feature matching operation, this step performs a mismatch elimination operation to determine valid matching point pairs. The feature matching point pairs are processed based on a preset consistency estimation algorithm. This algorithm estimates a geometric transformation model describing the spatial correspondence between two images by randomly selecting a minimum subset from all feature matching point pairs, and calculates the fitting error between all matching point pairs and this model. Point pairs with fitting errors less than a preset error threshold are determined as inliers satisfying geometric consistency, i.e., candidate matching point pairs. This process is iteratively executed, ultimately retaining the geometric model corresponding to the maximum number of inliers and all inliers under that model, thereby filtering out mismatched point pairs caused by image noise, local texture duplication, or incorrect correspondence, resulting in a set of geometrically consistent candidate matching point pairs.

[0050] The validity of candidate matching point pairs is determined. The determination criteria include: the number of candidate matching point pairs must be greater than a preset threshold, which ensures the matching results are statistically significant and avoids random matches due to insufficient matching points being accepted; simultaneously, the spatial distribution of candidate matching point pairs within the screen display area must meet preset spatial distribution conditions, for example, requiring these point pairs to be distributed across multiple sub-regions of the screen, rather than all clustered in a single local area (such as a fixed status bar icon). Only when a candidate matching point pair simultaneously meets both the above quantity and spatial distribution conditions is it ultimately determined as a valid matching point pair for subsequent delayed calculations.

[0051] The technical solution adopted in this step is based on the following principle: the initial feature matching result must contain outliers, i.e., incorrect matching pairs. The core principle of consistency estimation algorithms (such as RANSAC) is to utilize the prior knowledge that correct matching points (inliers) should conform to the same implicit geometric constraint (such as homography transformation). Through the iterative process of random sampling and model verification, the geometric model is robustly estimated and inliers that conform to the model are selected, thereby resisting the interference of mismatches. However, satisfying geometric consistency alone is not enough to guarantee matching quality sufficient to support reliable delay calculation. Setting a quantity threshold is based on statistical considerations. Too few matching point pairs, even if geometrically consistent, may originate from chance or feature-deficient areas, and the calculated delay value has a large variance and weak representativeness. Introducing spatial distribution conditions aims to ensure that the matching captures global changes that represent the main content of the screen, rather than local static or periodically flashing elements on the screen (such as constantly lit indicator lights), preventing the synchronization of local elements from being misjudged as the synchronization of the entire screen content. Therefore, combining geometric consistency checks, quantity sufficiency checks, and spatial distribution uniformity checks constitutes the matching judgment logic.

[0052] In summary, this application's embodiments filter out geometrically inconsistent mismatched point pairs through a preset consistency estimation algorithm, reducing the interference of erroneous data on latency calculation. By screening candidate matching point pairs using a preset quantity threshold, it ensures that the observation samples used for latency calculation are statistically significant, avoiding the inclusion of individual unreliable latency values ​​due to scarce feature points or poor matching quality, thus enhancing the representativeness of the final screen latency index. By screening candidate matching point pairs using preset spatial distribution conditions, it effectively distinguishes between global screen content changes and local fixed element updates, preventing the misidentification of unchanging icons in a small area of ​​the screen as evidence of two-screen synchronization, ensuring that the calculated latency truly reflects the synchronization of the main screen content perceptible to the user. The combined effect of these judgment conditions ensures that the determined valid matching point pairs are both correct and representative, laying a data foundation for calculating single-frame screen latency based on timestamps.

[0053] In some instances, the frame delay between the first screen and the second screen is determined based on multiple single-frame frame delays, including: Based on the video stream from the first screen and / or the video stream from the second screen, determine the video scene type, where the video scene type includes a static scene or a dynamic scene; When the video scene type is a static scene, the median delay is calculated based on the single-frame image delay of multiple consecutive frames, and the median delay is determined as the image delay; or, When the video scene type is a dynamic scene, the same moving target in the video stream of the first screen and the video stream of the second screen is tracked based on the optical flow method to determine the target tracking delay; Based on the target tracking delay, the delay of multiple single frames is verified to determine the frame delay.

[0054] For example, pattern recognition of video content can be performed by analyzing the degree of pixel change, the coherence and amplitude of feature point movement between consecutive frames in a video sequence, or by using a pre-trained lightweight classification model. For instance, if the positions of feature points extracted from multiple consecutive frames remain largely unchanged, or if the differences between frames are mainly due to noise, the scene is classified as static; if there are significant and continuously moving groups of feature points or object outlines, the scene is classified as dynamic. The core of this step lies in classifying the content based on its temporal and spatial characteristics, providing a decision-making basis for delay calculation methods adapted to different content characteristics.

[0055] When the video scene is determined to be a static scene, a strategy based on calculating the median of the single-frame delay across multiple consecutive frames is used to determine the final frame delay. In static scenes, since the main content of the image remains unchanged, the single-frame delays calculated based on feature matching should theoretically be close to a stable value. However, fluctuations may occur due to image acquisition noise, slight changes in illumination, or random errors in feature matching itself. Therefore, multiple pairs of valid video frames are continuously acquired over a period of time (e.g., corresponding to several seconds of video duration), and the single-frame delay of each pair is calculated to obtain a sequence of delay values. The median of this sequence is taken as the final frame delay because the median statistic is insensitive to outliers (i.e., extremely large or small delay values ​​caused by accidental mismatches or other interference), effectively filtering out these atypical fluctuations and obtaining a more representative quantitative result of the frame delay level for the static scene, reflecting a more realistic and stable synchronization level.

[0056] When the video scene is determined to be a dynamic scene, the same moving target in the video streams of the first and second screens is tracked using optical flow to determine the target tracking delay. This target tracking delay is then used to verify the delay of multiple single frames, thus determining the final frame delay. In dynamic scenes, there are fast-moving or large-scale moving objects in the frame. Traditional methods based on sparse feature point matching may have few or transient feature points in the motion region, leading to insufficient or unreliable calculation samples for single-frame frame delay.

[0057] Therefore, optical flow is introduced as a supplementary verification method in this step. First, in the video streams of the first and second screens, the same moving target (such as a moving vehicle icon in the video) is selected, and the pixel position of the target in each frame is continuously tracked using optical flow. Optical flow estimates the motion vector by calculating the brightness change of pixel blocks between adjacent frames, thus enabling accurate tracking of the target's trajectory. The timestamps of the moving target first appearing in a specific region of interest in the first screen video stream and the timestamps of its appearance in the corresponding region in the second screen video stream are recorded. The difference between the two timestamps is the target tracking delay.

[0058] Meanwhile, the feature matching method continues to run in parallel to provide delay information for the background or relatively static areas. The delays of multiple single frames obtained based on feature matching are compared and cross-validated with the target tracking delay. If the results are consistent, the confidence level is increased; if the feature matching results are scattered or unreliable due to scene complexity, the target tracking delay becomes the primary criterion. This step, through multi-algorithm fusion and verification mechanisms, ensures accurate and reliable frame delays even in complex scenes with rapidly changing content.

[0059] In summary, this application's embodiments automatically identify video scene types, enabling the detection method to achieve scene adaptability and overcoming the limitations of a single algorithm in specific scenarios. Secondly, for static scenes, median statistics are used to effectively filter out random matching errors and occasional outliers caused by the invariance of image content, improving the accuracy of latency measurement in common static applications such as navigation interfaces and desktops. Furthermore, for dynamic scenes, optical flow is introduced for target tracking, compensating for the potential failure of feature point matching in fast-moving areas. A cross-validation mechanism enhances the robustness of latency analysis under complex dynamic content. Finally, this scheme ensures high-confidence quantification results of image synchronization latency under various real-world in-vehicle display content, providing a data foundation for performance evaluation of multi-screen projection systems.

[0060] In some instances, the audio-visual delay for each screen is determined using cross-correlation calculations based on the audio stream, the video stream from the first screen, and the video stream from the second screen, including: Process the audio stream to extract audio feature sequences; Visual features are extracted from the video streams of the first screen and the second screen respectively to generate a first visual feature sequence and a second visual feature sequence; The audio feature sequence is cross-correlated with the first visual feature sequence and the second visual feature sequence to determine the global audio-visual delay for each screen. Determine event-level audio-visual latency based on the video stream from the first screen, the video stream from the second screen, and the audio stream. Based on global audio-visual latency and event-level audio-visual latency, the audio-visual latency corresponding to each screen is determined.

[0061] For example, based on the acquired audio stream, preprocessing operations are first performed, such as applying spectral subtraction or wavelet transform for noise reduction and removing silent segments to enhance the proportion of effective audio signal. Subsequently, the processed continuous audio signal is framed according to time intervals matching the video capture frame rate, ensuring a temporal correspondence between each audio frame and a video frame. For each audio frame, its Mel-frequency cepstral coefficients (MFCC) feature is calculated, which simulates the frequency perception characteristics of the human auditory system, thereby generating a temporally ordered audio feature sequence. Each feature vector in this sequence represents the spectral content of the audio signal within the corresponding time period, and because the framing is aligned with the video timestamp, it lays the foundation for correlation analysis with the visual feature sequence.

[0062] Visual features reflecting the degree of change in image content are extracted from the video streams of the first and second screens, respectively. For each frame, the visual feature can be the intensity of brightness change across the entire frame, the energy value calculated through inter-frame differencing, or a statistical measure of pixel values ​​within a specific region of interest. This feature extraction operation is performed independently and frame-by-frame for consecutive video frames of each screen, generating two time-aligned sequences: a first visual feature sequence and a second visual feature sequence. These sequences numerically quantify the dynamic changes of each screen's image over time, transforming the originally heterogeneous audio and visual signals into isomorphic, time-varying one-dimensional feature signals, thus providing a comparable data format for calculating the time offset between the two.

[0063] The audio feature sequence is cross-correlated with both the first and second visual feature sequences. Cross-correlation is a signal processing method used to measure the similarity between two sequences at different relative time offsets. Specifically, the audio feature sequence is used as a reference signal, slided along the time axis, and a similarity metric is calculated point-by-point with the visual feature sequence of each screen to obtain the cross-correlation function. This function is analyzed to find the sliding offset corresponding to its maximum value. This offset physically represents the amount by which one of the sequences needs to be moved along the time axis to achieve the best matching state between the audio and visual feature sequences; this amount is the estimated overall time delay of the audio relative to the screen image, i.e., the global audio-visual delay.

[0064] Based on the video streams from the first and second screens, as well as the audio stream, event-level audio-visual latency is determined. This process aims to identify and utilize discrete synchronization points within the content for high-precision measurement. In the video stream, key visual events such as interface pop-ups, button highlighting, or navigation arrow transitions are identified by detecting abrupt changes in pixel intensity, the appearance of specific shapes, or transitions in motion trajectories, and their timestamps are recorded. In the audio stream, logically related audio events are detected synchronously, such as specific prompts, the start of voice-over keywords, or rhythmic changes in background music, and their timestamps are recorded. By matching visual and audio events that logically belong to the same causal relationship, the difference between their timestamps is directly calculated to obtain the event-level audio-visual latency observation. This method provides an instantaneous measurement of synchronization performance by capturing discrete time points.

[0065] Based on the calculated global audio-visual delay and multiple event-level audio-visual delays, the final audio-visual delay for each screen is determined. Specifically, statistical analysis is performed on multiple event-level delays corresponding to the same screen, for example, calculating the median as the event-level delay statistical value. Subsequently, the deviation between this event-level delay statistical value and the global audio-visual delay is calculated. This deviation is compared with a preset deviation threshold that characterizes the allowable fluctuation range. If the deviation is less than the threshold, it indicates that the global delay based on overall signal statistics is consistent with the result measured based on discrete event points, and the global delay is stable and reliable; therefore, it is determined as the final audio-visual delay for that screen. If the deviation is greater than or equal to the threshold, it suggests that the global delay may be affected by non-stationary signal interference or that there is a calculation error. In this case, the event-level measurement result is considered to better reflect the true synchronization state; therefore, the event-level delay statistical value is determined as the final audio-visual delay. This step, through a cross-validation mechanism, achieves the evaluation and correction of the preliminary calculation results.

[0066] In summary, the embodiments of this application effectively improve the accuracy and robustness of audio-visual synchronization detection through feature extraction, multi-level delay calculation, and cross-validation. By preprocessing the audio stream and extracting MFCC feature sequences, the original sound wave signal is transformed into a feature representation that conforms to the characteristics of human auditory perception, enhancing the discriminative power of audio event representation. Visual feature sequences are extracted from the video stream, abstracting changes in image content into a numerical sequence that evolves over time, achieving unification and temporal alignment of the two heterogeneous modal signals (audio and video) at the feature level, enabling the calculation of their temporal correlation. By calculating the global audio-visual delay using a cross-correlation function, this method can perform statistical analysis on long-term signals, obtaining stable estimates that reflect the average synchronization level, overcoming the randomness that may exist when relying solely on single-point measurements.

[0067] Furthermore, by introducing event-level audio-visual delay measurement and subtracting timestamps from synchronization points in the content, a high-precision, instantaneous evaluation of synchronization performance is achieved. This method is less affected by the overall signal quality and can effectively capture instantaneous anomalies. By establishing a deviation comparison and decision-making mechanism between global delay and event-level delay, the overall statistical results are cross-validated with precise local measurements. When the two are consistent, the result is confirmed as reliable; when they are inconsistent, the more reliable local measurement is preferred. This ensures that the final output audio-visual delay value maintains high accuracy regardless of whether the signal is stable or interfered with, providing technical assurance for evaluating the audio-visual synchronization quality of in-vehicle infotainment systems.

[0068] In some instances, the audio feature sequence is cross-correlated with a first visual feature sequence and a second visual feature sequence to determine the global audio-visual delay for each screen, including: A first cross-correlation calculation is performed on the audio feature sequence and the first visual feature sequence to generate a first cross-correlation function; The first relative time offset is determined based on the peak position of the first cross-correlation function; Based on the first relative time offset, determine the global audio-visual delay corresponding to the first screen; Perform a second cross-correlation calculation on the audio feature sequence and the second visual feature sequence to generate a second cross-correlation function; The second relative time offset is determined based on the peak position of the second cross-correlation function; Based on the second relative time offset, the global audio-visual delay corresponding to the second screen is determined.

[0069] For example, when calculating the global audio-visual delay for each screen, a first cross-correlation calculation is first performed on the audio feature sequence and the first visual feature sequence. This calculation generates a first cross-correlation function describing the degree of matching between the two at different relative time offsets by sliding the audio feature sequence along the time axis and calculating the similarity metric with the first visual feature sequence point by point. Analyzing this function, the sliding offset corresponding to its global maximum value is found. This offset physically represents the amount of time adjustment required to achieve the optimal matching state between the audio feature sequence and the first visual feature sequence, thereby determining the first relative time offset. Based on this first relative time offset, its value is determined as the global audio-visual delay for the first screen. This delay value quantifies the overall, average time difference between the audio stream and the first screen image.

[0070] Subsequently, a second cross-correlation calculation is performed on the same audio feature sequence and the second visual feature sequence. The second cross-correlation function is generated using the same process. The second relative time offset is determined by locating its peak position. Finally, this offset is determined as the global audio-visual delay corresponding to the second screen, thereby completing the separate evaluation of the overall synchronization relationship between the sound source and the screen of each independent display terminal.

[0071] The principle behind the above steps lies in using the mathematical properties of the cross-correlation function to estimate the time delay between two signal sequences. The cross-correlation function quantitatively describes the change in the similarity of the waveforms or characteristic patterns of two sequences when one sequence is shifted relative to the other in time. When two sequences are content-synchronized, their corresponding characteristic patterns are aligned on the time axis. In this case, shifting one sequence to completely overlap with the other should maximize the similarity, resulting in a zero shift. However, when there is a systematic transmission or processing delay, the characteristic patterns of one sequence will experience an overall time shift relative to the other. In this case, by calculating the cross-correlation function and finding its peak value, the shift required to align the two sequences can be found; this shift is the time delay.

[0072] In this application, the audio feature sequence represents the temporal evolution of the sound signal, and the visual feature sequence represents the temporal evolution of changes in the content of a specific screen. Calculating the cross-correlation between the two and determining the peak offset essentially involves finding the time difference that optimally aligns the sound event and the visual event in a statistical sense, thus achieving a robust estimate of the audio-visual synchronization delay. Performing this operation on the first and second screens separately means that the audio decoding and playback paths are unified, but the graphics rendering and display links for each screen are independent. Therefore, different delay values ​​may exist between the audio and the visual content on different screens.

[0073] In summary, this application's embodiments, by performing cross-correlation calculations between the audio feature sequences and the independent visual feature sequences of each screen, can identify and quantify the independent time delays between the audio stream and the images on the first and second screens. This aligns with the reality in multi-screen systems where the audio source is unified but the processing paths of each display terminal may be heterogeneous, avoiding evaluation errors caused by confusing the delays of different screens. The method of peak localization of the cross-correlation function, through global statistical analysis of the feature sequences over the entire test period, yields delay results that exhibit stability in smoothing noise and resisting transient interference, reflecting the overall synchronization level of audio and image on a macroscopic time scale.

[0074] In some instances, event-level audio-visual latency is determined based on the video stream from the first screen, the video stream from the second screen, and the audio stream, including: Based on preset acoustic features, audio event points and corresponding audio timestamps are identified from the audio stream; Based on preset image change features, identify the first visual event point and the first visual timestamp corresponding to the first visual event point from the video stream of the first screen; Based on the audio timestamp and the first visual timestamp associated with the same logical event, calculate the event-level audio-visual delay corresponding to the first screen; Based on preset image change features, identify second visual event points and corresponding second visual timestamps from the video stream of the second screen; Based on the audio timestamp and the second visual timestamp associated with the same logical event, calculate the event-level audio-visual delay corresponding to the second screen.

[0075] For example, in the step of determining the event-level audio-visual delay, the principle is to use discrete synchronization points in the media content that occur naturally or are pre-designed and have a clear causal relationship, and to achieve instantaneous measurement of audio-visual synchronization by directly comparing the timestamps of the audio event and the visual event.

[0076] This method first identifies audio event points and their corresponding audio timestamps from the audio stream based on preset acoustic features. Second, it identifies visual event points and their corresponding visual timestamps from the video streams of the first and second screens, respectively, based on preset image change features. Finally, by associating audio and visual timestamps belonging to the same logical event and calculating the difference, the event-level audio-visual delays for the first and second screens are obtained, respectively. This process does not rely on the statistical characteristics of the overall signal but focuses on key synchronization moments at the content level, providing discrete and precise observation samples for evaluating synchronization performance.

[0077] Audio event identification is achieved by analyzing the time-frequency characteristics of the audio signal. Based on preset acoustic features, such as energy abrupt changes in high-frequency components, peak intensity within a specific frequency band, or significant changes in the Mel-frequency cepstral coefficient sequence, the system continuously scans and extracts features from the audio stream. When the extracted feature value exceeds a preset detection threshold, an audio event is determined to have occurred, such as the start of a prompt tone, the burst of a voice keyword, or a strong beat in background music. The acquisition time corresponding to the event is recorded, generating an audio timestamp. The key to this step lies in the selection of preset acoustic features and the setting of thresholds, which together determine the type of sound changes that the system is sensitive to, thus ensuring that the captured audio events logically correspond to changes in the visuals; for example, the start of a navigation command voice corresponds to the appearance of an arrow icon on the map interface.

[0078] Visual event point identification is accomplished by analyzing abrupt changes in pixel values ​​or image features within the video sequence. Based on preset image change features, such as the average change in pixel brightness or color across the entire screen or a specific region of interest, significant differences in edge detection results, or the appearance of specific interface elements (such as pop-ups or highlighted buttons) identified by object detection algorithms, the video streams of the first and second screens are analyzed independently. When the calculated change feature value exceeds a preset visual detection threshold, a visual event point is determined to have occurred in the video stream of that screen. The acquisition time of the video frame corresponding to the event point is recorded, generating a first visual timestamp and a second visual timestamp, respectively. The preset image change features are designed to capture user-perceptible and significant screen state transitions, ensuring that the identified visual events have clear temporal significance and can be logically paired with the aforementioned audio events.

[0079] In summary, this application's embodiments achieve point-to-point high-precision measurement of audio-visual synchronization by directly capturing and associating logically strongly correlated discrete audio and visual events and calculating their timestamp differences. This method is less affected by the overall signal stability. Since event-level latency originates from the synchronization points of the content itself, it provides a reference benchmark for verifying the global audio-visual latency obtained based on cross-correlation calculations. If the two match, the credibility of the overall conclusion is enhanced; if there is a significant deviation, it indicates that the applicability of the global algorithm needs further examination or that the system has transient anomalies. This method can effectively identify and quantify transient synchronization phenomena that may occur in the system at specific moments, such as an abnormally large delay between a certain prompt tone and its corresponding icon display. Such transient problems may be smoothed out by macroscopic statistical methods, thus providing more comprehensive and detailed system synchronization performance results.

[0080] In some instances, the audio-visual latency for each screen is determined based on global audio-visual latency and event-level audio-visual latency, including: Calculate event-level latency statistics based on multiple event-level audio-visual delays corresponding to the same screen. Calculate the deviation between the event-level latency statistics and the corresponding screen's global audio-visual latency to obtain the deviation result; When the deviation result is less than the preset deviation threshold, the global audio-visual delay is determined as the audio-visual delay of the corresponding screen; or, when the deviation result is greater than or equal to the preset deviation threshold, the event-level delay statistics value is determined as the audio-visual delay of the corresponding screen.

[0081] For example, the process of determining the final audio-visual delay for each screen involves a computational flow based on cross-validation and adaptive decision-making using global audio-visual delay and event-level audio-visual delay. The principle is that global audio-visual delay is obtained through cross-correlation analysis of long-term audio and visual feature sequences, reflecting the overall average synchronization trend, but may be insensitive to signal non-stationarity or local transient anomalies. Event-level audio-visual delay, on the other hand, is obtained by directly subtracting the timestamps of discrete, paired audio and visual events, providing multiple instantaneous synchronization observation samples, but it is highly dependent on the accuracy of event detection, and individual observations may contain random errors. Therefore, this application combines the two methods, using the statistical value of event-level delay to verify the reliability of global delay, and designs a decision logic based on a preset deviation threshold, aiming to integrate the advantages of both methods to determine a more accurate and robust audio-visual delay.

[0082] Based on multiple event-level audio-visual delays corresponding to the same screen, event-level delay statistics are calculated. During testing, multiple pairs of audio and visual events are identified and calculated, resulting in multiple event-level audio-visual delay observations for that screen. These observations constitute a sample set. To obtain a stable and representative central tendency estimate and eliminate potential random errors in individual observations (e.g., due to slight biases in event detection or transient environmental interference), statistical analysis of this set is required. Typically, the median or mean of this set is calculated as the event-level delay statistics. The median is insensitive to outliers and better reflects the synchronization level of most events. This step transforms discrete point measurements into aggregated, statistically meaningful reference values.

[0083] The deviation between the event-level delay statistics and the corresponding global audio-visual delay of the screen is calculated to obtain the deviation result. The deviation refers to the absolute value of the difference between the event-level delay statistics and the global audio-visual delay value. The purpose of calculating this deviation is to quantify the degree of consistency between the delay estimate based on overall signal statistics and the delay estimate based on discrete event point measurements. A smaller deviation indicates that the conclusions obtained by the two methods are similar, meaning that the synchronization state is relatively stable during the test period and both measurement methods are reliable; a larger deviation suggests that the results of the two methods differ, which may stem from signal non-stationarity, specific interference with the global calculation method, or systematic biases in event detection.

[0084] The final audio-visual delay is determined by comparing the results with a preset deviation threshold. The preset deviation threshold is a pre-defined threshold used to determine whether two delay estimates can be considered consistent. When the calculated deviation result is less than the preset threshold, the event-level measurement is considered consistent with the global calculation result. In this case, the global audio-visual delay obtained through long-term signal analysis is considered stable and reliable, and is therefore determined as the final audio-visual delay for the corresponding screen. Conversely, when the deviation result is greater than or equal to the preset deviation threshold, a significant inconsistency is considered between the two results. In this case, since event-level delay is directly measured based on the synchronization point of the content itself, it is generally considered more direct and resistant to interference in reflecting the true instantaneous synchronization relationship. Therefore, the event-level delay statistics are selected as the final audio-visual delay for the screen. This decision mechanism achieves the adaptability of the method, enabling it to adopt a more stable global estimate in most cases and switch to a more reliable local event measurement result when anomalies or discrepancies occur.

[0085] In summary, this application's embodiments effectively smooth out potential random errors in single-point measurements by calculating event-level delay statistics and statistically analyzing multiple discrete high-precision observations. This yields a robust estimate that represents the typical level of event synchronization within the test period, enhancing the statistical stability of instantaneous synchronization state assessment. Based on a decision logic with a preset deviation threshold, when the results of the two methods are consistent, the global delay, based on a large amount of statistical data and possessing strong noise resistance, is prioritized, ensuring measurement stability under stable signals. When significant discrepancies occur, the event-level statistics based on content logic and direct measurement are adopted, effectively avoiding potential misjudgments by the global algorithm when the signal is non-stationary or subject to specific interference. This ensures that audio-visual delays can be output under various complex test scenarios, enhancing the applicability of the entire detection method.

[0086] In some instances, the synchronization detection results of multi-screen projection are determined based on image latency and audio-visual latency, including: The maximum synchronization delay is determined based on the screen latency, the audio-visual latency corresponding to the first screen, and the audio-visual latency corresponding to the second screen. The synchronization detection result is determined based on the preset delay threshold and the maximum synchronization delay.

[0087] For example, the maximum value among the three calculated values—the image latency between the first and second screens, the audio-visual latency between the audio stream and the image on the first screen, and the audio-visual latency between the audio stream and the image on the second screen—is selected as the maximum synchronization latency. The underlying principle of this step is that the overall synchronization performance and user experience of a multi-screen projection system are limited by the worst-performing dimension among all synchronization dimensions. Whether it's image desynchronization between two screens or audio desynchronization with the image on either screen, it will cause a perceptible degradation in the user experience. Therefore, comparing the image latency with the two independent audio-visual latencys and taking the maximum value allows for a comprehensive capture of the system's synchronization error under the worst-case scenario. This maximum synchronization latency represents the time magnitude of the most serious synchronization problem that the system needs to address during the current test period.

[0088] After determining the maximum synchronization delay, a synchronization detection result is generated based on its comparison with a pre-defined delay threshold. The pre-defined delay threshold is a technical parameter defined in advance based on the synchronization performance requirements of the target application scenario, industry standards, or empirical data; it clarifies the upper limit of the acceptable synchronization error for the system. The calculated maximum synchronization delay is directly compared with this pre-defined delay threshold: if the maximum synchronization delay is less than or equal to the pre-defined delay threshold, the system's synchronization performance is deemed to meet the requirements; conversely, if the maximum synchronization delay is greater than the pre-defined delay threshold, the system is deemed to have a synchronization anomaly.

[0089] In some instances, it also includes: Based on a preset update cycle, the feature extraction model used in the feature matching method is updated. Based on historical detection data, the preset delay threshold is dynamically adjusted. Based on hardware acceleration technology or edge computing technology, the processing of synchronization detection methods is optimized.

[0090] For example, to continuously improve the accuracy and adaptability of the synchronization detection system, this method introduces a model update mechanism. This mechanism is triggered based on a preset update cycle, which can be set according to operational experience after actual deployment, such as a fixed time interval or the cumulative completion of a certain number of detection tasks as the trigger condition. When the update condition is met, the system evaluates the current feature extraction model used for image feature matching and replaces it with a more advanced algorithm model, such as replacing the original ORB algorithm with the deep learning-based SuperPoint feature extractor and SuperGlue matcher. This update operation includes acquiring the new model, testing its accuracy and speed on a validation dataset, and integrating it into the existing processing flow. By regularly updating the feature extraction model, the system can continuously utilize the latest advancements in computer vision, thereby improving the success rate and accuracy of feature matching in complex scenes such as complex textures, drastic lighting changes, or extreme viewpoints, and enhancing the robustness and accuracy of image latency calculation.

[0091] To further ensure the rationality and scenario adaptability of the synchronization judgment conclusions, this method implements a dynamic adjustment function for the preset latency threshold. This function operates based on a large amount of detection data accumulated during the system's historical operation. The system collects and stores the screen latency and audio-visual latency results obtained from each detection, forming a historical database. By statistically analyzing this historical data, the actual distribution pattern of latency values ​​under different application scenarios can be obtained. For example, the average latency is lower and fluctuates less when playing high-definition videos, while the average latency is higher and may have peak values ​​when running complex 3D navigation. Based on these statistical patterns, the system can automatically or semi-automatically correct the initially set fixed latency threshold. For example, the threshold can be lowered for game scenarios to meet their stringent real-time requirements, or the threshold can be appropriately relaxed for certain display scenarios to avoid unnecessary false alarms. This dynamic threshold adjustment allows the synchronization evaluation criteria to align with the actual performance of the tested system and the quality requirements of different functional modules, achieving intelligent and differentiated performance evaluation.

[0092] To meet the efficiency requirements of automated testing, this method integrates performance optimization operations to accelerate the entire synchronization detection process. This optimization is implemented along two main technical paths: first, applying hardware acceleration technology, leveraging the parallel computing capabilities of graphics processing units (GPUs) or dedicated AI computing chips to accelerate core computationally intensive tasks such as feature extraction, feature matching, and cross-correlation calculations at the hardware level. For example, optimizing relevant functions in the OpenCV library using the CUDA programming model; second, employing edge computing technology, lightweighting some or all detection algorithm modules and deploying them closer to the data source on edge computing devices, such as directly integrating them into test industrial control computers or optimized embedded devices, reducing data transmission latency and cloud processing overhead. Through these performance optimization operations, the overall processing time from data acquisition to the generation of detection results can be shortened, transforming potentially time-consuming offline analysis into real-time online detection. This meets the needs of applications with higher timeliness requirements, such as rapid production line testing and real-time system monitoring, thus improving the practicality and deployability of this detection method.

[0093] It should be noted that the terms "first screen," "second screen," etc., used in this application are merely exemplary names for ease of description, intended to refer to any two of the multiple independent display terminals requiring synchronization detection, and not to limit the number of display terminals to which this method applies. The core of this method lies in processing the timing relationship between video streams from multiple independent display sources and a unified audio stream; its technical solution does not depend on the specific number of screens being two. In practical applications, when more than two screens need to be included in synchronization detection, this method can synchronously acquire video streams from more screens accordingly, and calculate the image delay between any two screens based on the same principle, as well as the audio-visual delay between the unified audio stream and the video stream of each newly added screen, thereby extending the detection range to the entire multi-screen system and achieving automated evaluation of the synchronization performance between any number of screens.

[0094] Please see Figure 2 The diagram below illustrates a multi-screen projection synchronization detection device based on visual positioning, as provided in an embodiment of this application. The device includes: Media data acquisition unit 21 is used to simultaneously acquire the video stream of the first screen, the video stream of the second screen, and the audio stream; The screen delay calculation unit 22 is used to determine the screen delay between the first screen and the second screen based on the video stream of the first screen and the video stream of the second screen by using a feature matching method. The audio-visual delay calculation unit 23 is used to determine the audio-visual delay corresponding to each screen based on the audio stream, the video stream of the first screen and the video stream of the second screen through a cross-correlation calculation method. The media synchronization detection unit 24 is used to determine the synchronization detection result of multi-screen projection based on the image delay and audio-visual delay.

[0095] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0096] Although preferred embodiments have been described in this specification, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications that fall outside the scope of this specification.

[0097] Obviously, those skilled in the art can make various modifications to this specification without departing from its spirit and scope. Therefore, this specification also intends to include any modifications that fall within the scope of the claims and their equivalents.

Claims

1. A method for detecting multi-screen projection synchronization based on visual positioning, characterized in that, include: Simultaneously acquire the video stream from the first screen, the video stream from the second screen, and the audio stream; Based on the video streams of the first screen and the second screen, the image delay between the first screen and the second screen is determined by a feature matching method. Based on the audio stream, the video stream of the first screen, and the video stream of the second screen, the audio-visual delay corresponding to each screen is determined by a cross-correlation calculation method. Based on the image delay and the audio-visual delay, the synchronization detection result of multi-screen projection is determined.

2. The method according to claim 1, characterized in that, The determination of the image delay between the first screen and the second screen based on the video streams of the first screen and the second screen using a feature matching method includes: Based on the video streams of the first screen and the second screen, the first and second video frames corresponding to the content are determined by content search. Feature extraction operations are performed on the first video frame and the second video frame respectively to generate a first feature descriptor subset and a second feature descriptor subset; Perform feature matching operations on the first feature description subset and the second feature description subset to determine feature matching point pairs; Perform a false match elimination operation on the feature matching point pairs to determine the valid matching point pairs; Based on the timestamps associated with each of the valid matching points, the single-frame delay is calculated; The frame delay between the first screen and the second screen is determined based on the delay of multiple single frames.

3. The method according to claim 2, characterized in that, The step of performing a false match removal operation on the feature matching point pairs to determine valid matching point pairs includes: The feature matching point pairs are processed based on a preset consistency estimation algorithm to determine candidate matching point pairs that satisfy geometric consistency. When the number of candidate matching point pairs is greater than a preset number threshold and the distribution of the candidate matching point pairs within the screen area meets a preset spatial distribution condition, the candidate matching point is determined to be a valid matching point pair.

4. The method according to claim 2, characterized in that, Determining the frame delay between the first screen and the second screen based on multiple single-frame frame delays includes: Based on the video stream of the first screen and / or the video stream of the second screen, determine the video scene type, wherein the video scene type includes a static scene or a dynamic scene; When the video scene type is the static scene, the median delay is calculated based on the single-frame image delay of multiple consecutive frames, and the median delay is determined as the image delay; or, When the video scene type is the dynamic scene, the same moving target in the video stream of the first screen and the video stream of the second screen is tracked based on the optical flow method to determine the target tracking delay; Based on the target tracking delay, the delays of the multiple single-frame images are verified to determine the image delay.

5. The method according to claim 1, characterized in that, The step of determining the audio-visual delay for each screen based on the audio stream, the video stream of the first screen, and the video stream of the second screen using a cross-correlation calculation method includes: The audio stream is processed to extract audio feature sequences; Visual features are extracted from the video streams of the first screen and the second screen respectively to generate a first visual feature sequence and a second visual feature sequence; The audio feature sequence is cross-correlated with the first visual feature sequence and the second visual feature sequence respectively to determine the global audio-visual delay for each screen. Based on the video stream from the first screen, the video stream from the second screen, and the audio stream, determine the event-level audio-visual delay; Based on the global audio-visual delay and the event-level audio-visual delay, the audio-visual delay corresponding to each screen is determined.

6. The method according to claim 5, characterized in that, The step of performing cross-correlation calculations between the audio feature sequence and the first visual feature sequence and the second visual feature sequence, respectively, to determine the global audio-visual delay corresponding to each screen includes: A first cross-correlation calculation is performed on the audio feature sequence and the first visual feature sequence to generate a first cross-correlation function; The first relative time offset is determined based on the peak position of the first cross-correlation function; Based on the first relative time offset, the global audio-visual delay corresponding to the first screen is determined; Perform a second cross-correlation calculation on the audio feature sequence and the second visual feature sequence to generate a second cross-correlation function; The second relative time offset is determined based on the peak position of the second cross-correlation function; Based on the second relative time offset, the global audio-visual delay corresponding to the second screen is determined.

7. The method according to claim 6, characterized in that, The determination of event-level audio-visual latency based on the video stream from the first screen, the video stream from the second screen, and the audio stream includes: Based on preset acoustic features, audio event points and corresponding audio timestamps are identified from the audio stream; Based on preset image change features, a first visual event point and a first visual timestamp corresponding to the first visual event point are identified from the video stream of the first screen. Based on the audio timestamp and the first visual timestamp associated with the same logical event, calculate the event-level audio-visual delay corresponding to the first screen; Based on preset image change features, identify second visual event points and the second visual timestamps corresponding to the second visual event points from the video stream of the second screen; Based on the audio timestamp and the second visual timestamp associated with the same logical event, the event-level audio-visual delay corresponding to the second screen is calculated.

8. The method according to claim 7, characterized in that, The step of determining the audio-visual delay corresponding to each screen based on the global audio-visual delay and the event-level audio-visual delay includes: Based on multiple event-level audio-visual delays corresponding to the same screen, calculate event-level delay statistics; The deviation between the event-level latency statistics and the global audio-visual latency of the corresponding screen is calculated to obtain the deviation result; When the deviation result is less than a preset deviation threshold, the global audio-visual delay is determined as the audio-visual delay of the corresponding screen; or, when the deviation result is greater than or equal to the preset deviation threshold, the event-level delay statistics value is determined as the audio-visual delay of the corresponding screen.

9. The method according to claim 1, characterized in that, The determination of the synchronization detection result of multi-screen projection based on the image delay and the audio-visual delay includes: The maximum synchronization delay is determined based on the screen delay, the audio-visual delay corresponding to the first screen, and the audio-visual delay corresponding to the second screen. The synchronization detection result is determined based on the preset delay threshold and the maximum synchronization delay.

10. The method according to claim 1, characterized in that, Before synchronously acquiring the video stream from the first screen, the video stream from the second screen, and the audio stream, the method further includes: Based on network time protocols or precision time protocols, the system clocks of video and audio acquisition devices are synchronized and calibrated.

Images