An ultra-high-definition audio and video synchronization system based on PTP network synchronization

Abstract

The present invention discloses an ultra-high-definition audio and video synchronization system based on PTP network synchronization. The system at least comprises: a PTP clock reference source, which continuously sends out PTP network protocol data packets; a clock synchronization module, which synchronizes a local clock with the PTP clock reference source based on the PTP network protocol data packets; an audio and video splitting module, which extracts video data and audio data from a media input signal; a video synchronization module, which synchronizes the extracted video data to the PTP clock reference source via the PTP network; an audio synchronization module, which synchronizes the extracted audio data to the PTP clock reference source via the PTP network; and an audio and video combining module, which combines the video data and audio data synchronized to the PTP clock reference source into one picture frame to obtain a media output signal. The present invention has the advantages of low complexity, high stability and no mandatory media signal interface type, and solves the problems of high deployment difficulty, high cost burden and incomplete media signal interface support in existing ultra-high-definition audio and video synchronization systems.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video synchronization, and in particular to an ultra-high-definition audio and video synchronization system based on PTP network synchronization. Background Art

[0002] In existing technologies, media signals originate from a variety of devices, such as set-top boxes, satellite decoders, and webcams. Due to differences in device hardware, media signals between these devices are inevitably out of sync. Many existing ultra-high-definition audio and video usage scenarios require synchronization between multiple media signals. For example, the ultra-high-definition audio and video connected to multiple ultra-high-definition displays in a stadium must be synchronized; otherwise, the audience's audio-visual experience will be unbearable.

[0003] Existing ultra-high-definition audio and video synchronization systems typically utilize Genlock to synchronize media signals such as SDI interfaces. However, this inevitably increases system deployment complexity and costs. Deploying Genlock can be particularly challenging in larger environments, especially for equipment routing. Furthermore, Genlock's support for interfaces like HDMI and DP is limited, creating deployment difficulties, high costs, and incomplete support for media signal interfaces. These issues plague existing ultra-high-definition audio and video synchronization systems. Summary of the Invention

[0004] In response to the above-mentioned problems of high deployment difficulty, high cost burden and incomplete support for media signal interfaces in ultra-high-definition audio and video synchronization systems, the present invention provides an ultra-high-definition audio and video synchronization system based on PTP network synchronization, which has the advantages of low complexity, high stability and no mandatory media signal interface type, and effectively solves the problems of high deployment difficulty, high cost burden and incomplete support for media signal interfaces in existing ultra-high-definition audio and video synchronization systems.

[0005] In order to solve the above technical problems, the specific solutions provided by the present invention are as follows:

[0006] An ultra-high-definition audio and video synchronization system based on PTP network synchronization, the system at least comprising:

[0007] PTP clock reference source, continuously sending PTP network protocol data packets;

[0008] A clock synchronization module, which synchronizes a local clock with the PTP clock reference source based on the PTP network protocol data packet;

[0009] Audio and video splitting module, extracting video data and audio data from media input signals;

[0010] A video synchronization module synchronizes the extracted video data to the PTP clock reference source via a PTP network;

[0011] An audio synchronization module synchronizes the extracted audio data to the PTP clock reference source via a PTP network;

[0012] The audio and video combination module combines the video data and the audio data synchronized to the PTP clock reference source into one picture frame to obtain a media output signal.

[0013] In some implementations, synchronizing a local clock with the PTP clock reference source based on the PTP network protocol data packet includes:

[0014] Based on the PTP network protocol data packet, obtaining a clock difference between a local clock and a PTP clock reference source, and calculating a clock frequency difference;

[0015] Applying the calculated clock frequency difference to the local clock to synchronize the local clock with the PTP clock reference source;

[0016] When the PTP clock reference source continuously initiates PTP network protocol packets, the local clock will maintain sub-microsecond clock synchronization accuracy with the PTP clock reference source clock.

[0017] In some implementations, extracting video data and audio data from a media input signal includes:

[0018] The video data in the media input signal exists in the effective video part of the entire picture frame. The line and field positioning information in the picture frame data is used to locate the effective data segment of the video data and extract the video data.

[0019] The audio data in the media input signal exists in the blanking portion of the entire picture frame. The audio data packet header information encapsulated in the blanking portion is used to locate and extract the audio data.

[0020] The audio synchronization strategy based on PTP network synchronization is different from the video synchronization strategy. Therefore, it is necessary to obtain independent audio and video data through audio and video splitting.

[0021] In some implementations, synchronizing the extracted video data to the PTP clock reference source via a PTP network includes:

[0022] The video synchronization module synchronizes the extracted video data to the local clock of the clock synchronization module, and synchronizes the local clock in the clock synchronization module to the PTP clock reference source;

[0023] Synchronize video data to the PTP clock reference source within sub-microsecond error.

[0024] In some implementations, combining the video data and the audio data synchronized to the PTP clock reference source into one picture frame to obtain a media output signal includes:

[0025] Encapsulating the video data synchronized to the PTP clock reference source in an effective video portion of a picture frame, and encapsulating the audio data synchronized to the PTP clock reference source in a blanking portion of the picture frame, to obtain a media output signal;

[0026] After the video data and audio data are synchronized to the PTP clock reference source through the video synchronization module and the audio synchronization module respectively, the video data and the audio data need to be recombined into one picture frame.

[0027] In some embodiments, the extracted video data is synchronized to the PTP clock reference source via a PTP network, and the synchronization process includes a video data caching process and a video data playback process;

[0028] The specific workflow of the video data caching process is as follows:

[0029] After waiting for the first video data of a new frame to arrive, record the current local clock value;

[0030] Calculating a playback clock value according to the current local clock value;

[0031] Determine whether it is a low-latency output application scenario, and write the video data into the corresponding frame buffer area based on the judgment result;

[0032] The specific workflow of the video data playback process is as follows:

[0033] After waiting for the local clock to reach the playback clock value of the playback time, check whether there is video data at the current playback time in the cache. If so, play the video data at the current playback time in the buffer; if not, play the video data at the previous playback time in the cache.

[0034] In some embodiments, determining whether the application scenario is low-latency output and writing the video data into the corresponding frame buffer area according to the determination result includes:

[0035] If the application scenario is low-latency output, determine whether the currently calculated playback clock value is consistent with the playback clock value calculated when the previous frame arrived. If so, write the currently received video data to the area cached by the previous frame. If not, write the currently received video data to a new frame cache area.

[0036] If the application scenario is not low-latency output, the currently received video data will be written to a new frame buffer area regardless of whether the currently calculated playback clock value is consistent with the playback clock value calculated when the previous frame arrived.

[0037] There are differences in the processing mechanisms for low-latency output application scenarios and non-low-latency output application scenarios. After the current frame data in the above two application scenarios is cached, the process returns to the beginning and waits for the first video data of the next frame to arrive.

[0038] In some embodiments, during the playback of the video data, the discarding or repetition of the video data is controlled by the playback clock value at the playback moment, so that the video data after the PTP network synchronization is synchronized;

[0039] After continuous PTP network synchronization of video data, there may be two situations where there are too many or too few frames during long-term operation. By controlling the discarding or duplication of video data, the video data synchronization after PTP network synchronization can be achieved.

[0040] In some implementations, the audio synchronization module includes:

[0041] A sampling rate clock generation module, used to generate an audio sampling rate clock synchronized with a local clock;

[0042] an audio sampling rate conversion module, configured to synchronize unsynchronized audio data to the audio sampling rate clock generated by the sampling rate clock generation module;

[0043] After synchronization, the audio data and PTP clock reference source will maintain synchronization accuracy at the sub-microsecond level.

[0044] In some implementations, the synchronization reference sources of the video data and audio data are both PTP clock reference sources, and the synchronization accuracy is sub-microsecond, so the video data and audio data are synchronized.

[0045] The present invention provides an ultra-high-definition audio and video synchronization system based on PTP network synchronization. PTP is applied to the ultra-high-definition audio and video synchronization system, and the sub-microsecond clock synchronization accuracy of the PTP network protocol is utilized to synchronize asynchronous media signals of different devices to the same clock reference, thereby realizing ultra-high-definition audio and video synchronization. The deployment of the ultra-high-definition audio and video synchronization system based on PTP network synchronization only requires a network connection, and has the advantages of low complexity, high stability, and no mandatory media signal interface type. It effectively solves the problems of high deployment difficulty, high cost burden, and incomplete media signal interface support of existing ultra-high-definition audio and video synchronization systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1This is a structural block diagram of an ultra-high-definition audio and video synchronization system based on PTP network synchronization provided in an embodiment of the present invention;

[0047] Figure 2 This is a diagram of usage scenarios of multiple ultra-high-definition audio and video synchronization systems based on PTP network synchronization provided in an embodiment of the present invention;

[0048] Figure 3 Flowchart of a video data caching process provided in an embodiment of the present invention;

[0049] Figure 4 Flowchart of a video data playback process provided in an embodiment of the present invention;

[0050] Figure 5 This is a structural block diagram of the audio synchronization module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are also within the scope of protection of the present invention.

[0053] For example, an ultra-high-definition audio and video synchronization system based on PTP network synchronization includes at least: a PTP clock reference source, which continuously sends PTP network protocol data packets; a clock synchronization module, which synchronizes the local clock with the PTP clock reference source based on the PTP network protocol data packets; an audio and video splitting module, which extracts video data and audio data from the media input signal; a video synchronization module, which synchronizes the extracted video data to the PTP clock reference source through the PTP network; an audio synchronization module, which synchronizes the extracted audio data to the PTP clock reference source through the PTP network; and an audio and video combination module, which combines the video data and the audio data synchronized to the PTP clock reference source into a picture frame to obtain a media output signal.

[0054] This embodiment provides an ultra-high-definition audio and video synchronization system based on PTP network synchronization, which applies PTP to the ultra-high-definition audio and video synchronization system. It utilizes the sub-microsecond clock synchronization accuracy of the PTP network protocol to synchronize asynchronous media signals of different devices to the same clock reference, thereby achieving ultra-high-definition audio and video synchronization. The ultra-high-definition audio and video synchronization system based on PTP network synchronization only requires a network connection for deployment, and has the advantages of low complexity, high stability, and no mandatory media signal interface type. It effectively solves the problems of high deployment difficulty, high cost burden, and incomplete media signal interface support in existing ultra-high-definition audio and video synchronization systems.

[0055] Example 1:

[0056] like Figure 1 and Figure 2 As shown, an ultra-high-definition audio and video synchronization system based on PTP network synchronization includes at least a PTP clock reference source, a clock synchronization module, an audio and video splitting module, a video synchronization module, an audio synchronization module and an audio and video combination module.

[0057] The PTP clock reference source continuously sends PTP network protocol packets, ensuring sub-microsecond synchronization between the local clock and the PTP clock reference source. The PTP clock reference source, serving as the system's clock reference, is typically a PTP host capable of exchanging PTP network protocol packets. The PTP clock reference source relies on a stable global clock, such as GPS. Although multiple PTP clock reference sources may exist in an environment, they are synchronized because they rely on the same stable global clock.

[0058] The clock synchronization module synchronizes the local clock with the PTP clock reference source based on the PTP network protocol data packet.

[0059] The clock synchronization module processes PTP network protocol packets and synchronizes the local clock to a PTP reference clock source based on the PTP network. Given the PTP network protocol's sub-microsecond clock synchronization accuracy, the clock synchronization module can synchronize the local clock to within sub-microseconds of the PTP reference clock source.

[0060] The audio and video splitting module extracts video data and audio data from the media input signal.

[0061] The audio synchronization strategy based on PTP network synchronization is different from the video synchronization strategy. Therefore, it is necessary to obtain independent audio data and video data through audio and video splitting.

[0062] The video synchronization module synchronizes the extracted video data to the PTP clock reference source through the PTP network.

[0063] The independent video data obtained by the audio and video splitting module is extracted from the media input signal. Its rate is synchronized with the media input signal and synchronized to the PTP clock reference source within sub-microsecond error through the PTP network by the video synchronization module.

[0064] The audio synchronization module synchronizes the extracted audio data to the PTP clock reference source through the PTP network.

[0065] The independent audio data obtained by the audio and video splitting module is extracted from the media input signal. Its rate is synchronized with the media input signal and synchronized to the PTP clock reference source within sub-microsecond error through the PTP network by the audio synchronization module.

[0066] The audio and video combination module combines the video data and audio data synchronized to the PTP clock reference source into one picture frame to obtain the media output signal.

[0067] The audio synchronization strategy based on PTP network synchronization is different from the video synchronization strategy. Therefore, it is necessary to use the audio and video splitting module to split the video data and audio data from the media input signal. After the video data and audio data are synchronized to the PTP clock reference source through the video synchronization module and audio synchronization module respectively, the video data and audio data need to be recombined into a picture frame to obtain the media output signal.

[0068] This example provides an ultra-high-definition audio and video synchronization system based on PTP network synchronization, such as Figure 2 As shown, in the usage scenario of multiple media signals, multiple ultra-high-definition audio and video synchronization systems based on PTP network synchronization can be integrated. Taking advantage of the sub-microsecond clock synchronization accuracy of the PTP network protocol, multiple ultra-high-definition audio and video synchronization systems based on PTP network synchronization synchronize multiple asynchronous media input signals to the same PTP clock reference source, thereby achieving synchronization of multiple asynchronous media signals. The ultra-high-definition audio and video synchronization system based on PTP network synchronization has low complexity and high stability, and does not force the media signal interface type. It effectively solves the problems of high deployment difficulty, high cost burden, and incomplete media signal interface support of existing ultra-high-definition audio and video synchronization systems.

[0069] Example 2:

[0070] In this example, synchronizing a local clock with the PTP clock reference source based on the PTP network protocol data packet includes:

[0071] Based on the PTP network protocol data packet, a clock difference between the local clock and the PTP clock reference source is obtained, and a clock frequency difference is calculated.

[0072] The specific calculation process is to divide the clock difference by the time interval between the two PTP clock synchronization processes. The resulting value is the clock frequency difference between the local clock and the PTP clock reference source in seconds. For example, if the time interval between two PTP clock synchronization processes is 0.125 seconds, the local clock change value and the PTP clock reference source clock change value are obtained from the PTP network protocol packets during these two PTP clock synchronization processes. The clock difference is then subtracted from each other, and the time interval between the two clock differences is 0.125 seconds. Dividing the clock difference by 0.125 seconds yields the clock frequency difference between the local clock and the PTP clock reference source in seconds. Due to network jitter and computational errors, filtering algorithms are typically added to optimize the calculated clock frequency difference.

[0073] The calculated clock frequency difference is applied to the local clock to synchronize the local clock with the PTP clock reference source.

[0074] When the PTP clock reference source continuously initiates PTP network protocol packets, the local clock will maintain sub-microsecond clock synchronization accuracy with the PTP clock reference source. Furthermore, due to the uncertainty of network jitter, a filtering algorithm is typically used when calculating the clock frequency difference between the local clock and the PTP clock reference source. In this embodiment of the present invention, there are no restrictions on the method used to obtain and parse the PTP network protocol packets initiated by the PTP clock reference source and then calculate the clock frequency difference between the local clock and the PTP clock reference source.

[0075] Example 3:

[0076] In some implementations, extracting video data and audio data from a media input signal includes:

[0077] The video data in the media input signal exists in the effective video part of the entire picture frame. The line and field positioning information in the picture frame data is used to locate the effective data segment of the video data and extract the video data.

[0078] The audio data in the media input signal exists in the blanking portion of the entire picture frame. The audio data packet header information encapsulated in the blanking portion is used to locate and extract the audio data.

[0079] The audio synchronization strategy for PTP network synchronization differs from the video synchronization strategy, so audio and video splitting is required to obtain independent audio and video data. Typically, the audio data in the media input signal exists in the blanking portion of the entire picture frame, while the video data exists in the active video portion of the entire picture frame. The audio data packet header information encapsulated in the blanking portion can be used to locate and extract the audio data; the line-field positioning information in the picture frame data can be used to locate the active data segment of the video data and extract the video data, thereby obtaining independent audio and video data.

[0080] For media input interface chips that use HDMI to MIPI, for example, the chip will internally split the audio and video data and transmit them through independent interfaces. The audio and video data are received through different interfaces to obtain independent audio and video data. In the embodiments of the present invention, there are no restrictions on the method used to split and obtain the audio and video data in the media signal input.

[0081] Example 4:

[0082] In this example, synchronizing the extracted video data to the PTP clock reference source via the PTP network includes:

[0083] The video synchronization module synchronizes the extracted video data to the local clock of the clock synchronization module, and synchronizes it to the PTP clock reference source through the local clock in the clock synchronization module; achieving video data synchronization to the PTP clock reference source within a sub-microsecond error.

[0084] Video data is extracted from the media input signal and its rate is synchronized with the media input signal. The video synchronization module synchronizes the video data to the local clock in the clock synchronization module. The local clock in the clock synchronization module is synchronized to the PTP clock reference source within sub-microsecond error, thereby achieving video data synchronization to the PTP clock reference source within sub-microsecond error.

[0085] Similarly, synchronizing the extracted audio data to the PTP clock reference source through the PTP network includes:

[0086] The audio synchronization module synchronizes the extracted audio data to the local clock of the clock synchronization module, and synchronizes it to the PTP clock reference source through the local clock in the clock synchronization module; achieving audio data synchronization to the PTP clock reference source within a sub-microsecond error.

[0087] Similar to video data, audio data is extracted from the media input signal and its rate is synchronized with the media input signal. The audio synchronization module synchronizes the audio data to the local clock in the clock synchronization module. The local clock in the clock synchronization module is synchronized to the PTP clock reference source within sub-microsecond error, thereby achieving audio data synchronization to the PTP clock reference source within sub-microsecond error.

[0088] The step of combining the video data and the audio data synchronized to the PTP clock reference source into one picture frame to obtain a media output signal includes:

[0089] Encapsulating the video data synchronized to the PTP clock reference source in an effective video portion of a picture frame, and encapsulating the audio data synchronized to the PTP clock reference source in a blanking portion of the picture frame, to obtain a media output signal;

[0090] Since the audio synchronization strategy and video synchronization strategy based on PTP network synchronization in the embodiment of the present invention are different, it is necessary to split the audio data and video data from the media input signal during the splitting process of the audio and video splitting module. After the video data and audio data are synchronized to the PTP clock reference source through the video synchronization module and the audio synchronization module respectively, the video data and audio data need to be recombined into one picture frame.

[0091] By encapsulating audio data in the blanking portion of a picture frame and video data in the active video portion of a picture frame, a media output signal can be obtained. For a media output interface chip that uses a MIPI to HDMI converter, for example, the chip can obtain audio data and video data through independent audio and video interfaces, respectively, and complete the audio and video combination operation within the chip to obtain a media output signal. In the embodiments of the present invention, there are no restrictions on the method used to combine audio data and video data into a media output signal.

[0092] Embodiment 5:

[0093] like Figure 3 and Figure 4 As shown, in some embodiments, the extracted video data is synchronized to the PTP clock reference source through the PTP network, and the synchronization process includes a video data caching process and a video data playback process;

[0094] The specific workflow of the video data caching process is as follows:

[0095] After waiting for the first video data of a new frame to arrive, record the current local clock value;

[0096] Calculating a playback clock value according to the current local clock value;

[0097] Determine whether it is a low-latency output application scenario, and write the video data to the corresponding frame buffer area based on the judgment result.

[0098] Video data is stored in a buffered area in frames. The system first waits for the arrival of the first video data of a frame. Typically, the first video data of a frame is the first pixel in the first row. At this point, the current local clock value is recorded and then the playback clock value is found based on the local clock value. After PTP network synchronization, strict requirements are imposed on the playback of video frame data. The inter-frame interval for video frame data playback must meet the inter-frame interval calculated by the local clock after PTP network synchronization. For example, for 2160P video with a 25Hz frame rate, the inter-frame interval is a fixed 40ms calculated by the local clock after PTP network synchronization. Therefore, based on this fixed interval and the fixed local clock base value, the playback clock value at the start of each frame can be calculated. The local clock base value is related to the PTP clock reference source and typically starts at 0.

[0099] Furthermore, for systems that wish to control video playback timing through playback delay, this can be achieved by adding a fixed delay to the calculated playback clock value for each frame's start time. Based on the local clock value recorded when the first video data arrives, the most recent playback clock value is calculated and used as the start time for that frame of video data.

[0100] The specific workflow of the video data playback process is as follows:

[0101] After waiting for the local clock to reach the playback clock value of the playback time, check whether there is video data at the current playback time in the cache. If so, play the video data at the current playback time in the buffer; if not, play the video data at the previous playback time in the cache.

[0102] As mentioned above, the playback clock value at the start of each frame's playback can be calculated. After the local clock reaches the playback clock value at the playback time, the system checks whether the video data for the current playback time exists in the cache. If the video data for the current playback time exists in the cache, the video data in the buffer corresponding to the current playback time is played. If the video data for the current playback time does not exist in the cache, the video data in the cache corresponding to the previous playback time is played. The previous playback time usually refers to the previous playback time, that is, the video data of the previous frame is played.

[0103] In some embodiments, determining whether the application scenario is low-latency output and writing the video data into the corresponding frame buffer area according to the determination result includes:

[0104] If the application scenario is low-latency output, determine whether the currently calculated playback clock value is consistent with the playback clock value calculated when the previous frame arrived. If so, write the currently received video data to the area cached by the previous frame. If not, write the currently received video data to a new frame cache area.

[0105] If the application scenario is not low-latency output, the currently received video data will be written to a new frame buffer area regardless of whether the currently calculated playback clock value is consistent with the playback clock value calculated when the previous frame arrived.

[0106] There are differences in the processing mechanisms for low-latency output application scenarios and non-low-latency output application scenarios. After the current frame data in the above two application scenarios is cached, the process returns to the beginning and waits for the first video data of the next frame to arrive.

[0107] The playback clock value is the time at which the currently arriving video frame will be played, which determines when the currently arriving frame will be played. In low-latency output scenarios, frames with consistent playback times are cached in the same area, allowing later frames to be played more promptly, thus achieving low latency. However, this process may result in two frames of video data appearing for a single frame during playback, such as the upper half of the screen containing the older frame and the lower half containing the newer frame. When multiple frames of video data are written to the same cache area simultaneously, issues such as "tearing" are inevitable. This "tearing" issue is infrequent and is typically related to the difference between the pre-synchronization and post-synchronization video frame rates. In non-low-latency scenarios, video data for different frames is cached in different areas, with no overlap between frames, thus unaffecting the playback image.

[0108] In some embodiments, during the playback of the video data, the discarding or repetition of the video data is controlled by the playback clock value at the playback moment, so that the video data after the PTP network synchronization is synchronized;

[0109] After continuous PTP network synchronization of video data, there may be two situations where there are too many or too few frames during long-term operation. By controlling the discarding or duplication of video data, the video data synchronization after PTP network synchronization can be achieved.

[0110] The video data rate before synchronization is not completely equal to the video data rate after synchronization. In the embodiment of the present invention, after the video data is continuously synchronized with the PTP network, two situations may occur during long-term operation: "many frames" or "few frames". "Many frames" means that the arrival rate of the video data before synchronization is faster than the playback rate of the video data after synchronization, and the video data gradually accumulates, and one more frame of video data gradually appears. In the embodiment of the present invention, it is manifested as the playback clock values calculated after the arrival of the video data of the two frames are consistent. In the above-mentioned video data playback workflow, the video data in the cache that matches the playback clock value of the current playback moment will be searched during playback. Even if the playback moments of the two frames of video data are consistent, since the playback will not be performed after the playback moment has passed, only one of the two frames of video data will be played, and the other frame will not be played, which is equivalent to "discarding" the frame of video data. "Fewer frames" means that the arrival rate of the video data before synchronization is slower than the playback rate of the video data after synchronization, and the video data belonging to the current playback moment does not exist in the cache. In this case, because new video data has not yet arrived but the playback time has arrived, only the video data in the cache belonging to the old playback time can be played, usually the video data from the previous playback time, that is, the previous frame of video data, which is equivalent to "repeating" the previous frame of video data. The playback clock value at the playback time controls the "discarding" of video data and the "repeating" of frame video data to achieve video data rate synchronization after PTP network synchronization.

[0111] Under the sub-microsecond clock synchronization accuracy of the PTP network protocol, the local clock and the PTP clock reference source maintain sub-microsecond synchronization accuracy. The playback clock value at the time of video data playback is obtained by calculation. When the local clock reaches the playback time, the video data is played out. The played video data and the PTP clock reference source will maintain sub-microsecond synchronization accuracy, realizing video data output after PTP network synchronization.

[0112] Example 6:

[0113] like Figure 5 As shown, in this example, the audio synchronization module includes:

[0114] The sampling rate clock generation module is used to generate an audio sampling rate clock synchronized with the local clock.

[0115] During the continuous operation of the above-mentioned clock synchronization module, the local clock will always maintain sub-microsecond clock synchronization accuracy with the PTP clock reference source. Therefore, the audio sampling rate clock generated also maintains sub-microsecond clock synchronization accuracy with the PTP clock reference source. The generated audio sampling rate clock does not have to be consistent with the input audio data sampling rate. For example, if the input is 48kHz sampling rate audio, if the user wants to obtain synchronized 44.1kHz audio, the sampling rate clock generation module can be controlled to generate a 44.1kHz sampling rate clock. The method for generating the sampling rate clock can be to create a local clock counter, and generate a sampling rate clock when the count reaches a certain value. The sampling rate clock can also be generated based on the local clock using a clock generation chip. In the embodiment of the present invention, there is no restriction on which method is used to generate the audio sampling rate clock.

[0116] The audio sampling rate conversion module is used to synchronize the unsynchronized audio data to the audio sampling rate clock generated by the sampling rate clock generation module; the synchronized audio data and the PTP clock reference source will maintain synchronization accuracy at the sub-microsecond level.

[0117] The unsynchronized audio data is extracted through the audio and video splitting module, and its audio sampling rate will not be exactly the same as the audio sampling rate clock generated by the sampling rate clock generation module. The audio sampling rate conversion module will involve the sampling rate conversion of the audio data. The sampling rate conversion of audio data can be easily achieved by using algorithms such as FIR filters. If there is no requirement for the continuity of the audio data, simply "discarding" or "repeating" audio data samples can also achieve sampling rate conversion. In this embodiment of the present invention, there is no restriction on the method used to achieve the sampling rate conversion of audio data.

[0118] Specifically, the synchronization reference sources for the video data and audio data are both PTP clock reference sources, with sub-microsecond synchronization accuracy, and the video data and audio data are synchronized. This is embodied in that the video synchronization module enables the video data to maintain sub-microsecond synchronization accuracy with the PTP clock reference source, and the audio synchronization module enables the audio data to maintain sub-microsecond synchronization accuracy with the PTP clock reference source. The synchronization reference sources for the video data and audio data are both PTP clock reference sources with sub-microsecond synchronization accuracy, and the audio data and video data are synchronized.

[0119] The present invention provides an ultra-high-definition audio and video synchronization system based on PTP network synchronization. PTP is applied to the ultra-high-definition audio and video synchronization system, and the sub-microsecond clock synchronization accuracy of the PTP network protocol is utilized to synchronize asynchronous media signals of different devices to the same clock reference, thereby realizing ultra-high-definition audio and video synchronization. The deployment of the ultra-high-definition audio and video synchronization system based on PTP network synchronization only requires a network connection, and has the advantages of low complexity, high stability, and no mandatory media signal interface type. It effectively solves the problems of high deployment difficulty, high cost burden, and incomplete media signal interface support of existing ultra-high-definition audio and video synchronization systems.

[0120] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0121] It should be understood that the present invention is not limited to the above description and that various modifications and changes can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0122] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. An ultra-high-definition audio and video synchronization system based on PTP network synchronization, characterized in that: The system comprises at least: PTP clock reference source, continuously sending PTP network protocol data packets; A clock synchronization module, which synchronizes a local clock with the PTP clock reference source based on the PTP network protocol data packet; Audio and video splitting module, extracting video data and audio data from media input signals; A video synchronization module synchronizes the extracted video data to the PTP clock reference source via a PTP network; An audio synchronization module synchronizes the extracted audio data to the PTP clock reference source via a PTP network; An audio and video combining module combines the video data and the audio data synchronized to the PTP clock reference source into one picture frame to obtain a media output signal; The extracted video data is synchronized to the PTP clock reference source via a PTP network, wherein the synchronization process includes a video data caching process and a video data playback process; The specific workflow of the video data caching process is as follows: After waiting for the first video data of a new frame to arrive, record the current local clock value; Calculating a playback clock value according to the current local clock value; Determine whether it is a low-latency output application scenario, and write the video data to the corresponding frame buffer area based on the judgment result; The specific workflow of the video data playback process is as follows: After waiting for the local clock to reach the playback clock value of the playback time, check whether there is video data at the current playback time in the cache. If so, play the video data at the current playback time in the cache; if not, play the video data at the previous playback time in the cache.

2. The ultra-high-definition audio and video synchronization system based on PTP network synchronization according to claim 1, characterized in that The step of synchronizing a local clock with the PTP clock reference source based on the PTP network protocol data packet includes: Based on the PTP network protocol data packet, obtaining a clock difference between a local clock and a PTP clock reference source, and calculating a clock frequency difference; The calculated clock frequency difference is applied to the local clock to synchronize the local clock with the PTP clock reference source.

3. The ultra-high-definition audio and video synchronization system based on PTP network synchronization according to claim 1 is characterized in that, The extracting of video data and audio data from the media input signal includes: The video data in the media input signal exists in the effective video part of the entire picture frame. The line and field positioning information in the picture frame data is used to locate the effective data segment of the video data and extract the video data. The audio data in the media input signal exists in the blanking portion of the entire picture frame. The audio data packet header information encapsulated in the blanking portion is used to locate and extract the audio data.

4. The ultra-high-definition audio and video synchronization system based on PTP network synchronization according to claim 1, characterized in that: The step of synchronizing the extracted video data to the PTP clock reference source via the PTP network includes: The video synchronization module synchronizes the extracted video data to the local clock of the clock synchronization module, and synchronizes the local clock in the clock synchronization module to the PTP clock reference source.

5. The ultra-high-definition audio and video synchronization system based on PTP network synchronization according to claim 1 is characterized in that, Combining the video data and the audio data synchronized to the PTP clock reference source into one picture frame to obtain a media output signal includes: The video data synchronized to the PTP clock reference source is encapsulated in the effective video portion of the picture frame, and the audio data synchronized to the PTP clock reference source is encapsulated in the blanking portion of the picture frame to obtain a media output signal.

6. The ultra-high-definition audio and video synchronization system based on PTP network synchronization according to claim 1 is characterized in that, The determining whether the application scenario is low-latency output and writing the video data into the corresponding frame buffer area according to the determination result includes: If the application scenario is low-latency output, determine whether the currently calculated playback clock value is consistent with the playback clock value calculated when the previous frame arrived. If so, write the currently received video data to the area cached by the previous frame. If not, write the currently received video data to a new frame cache area. If it is a non-low-latency output application scenario, regardless of whether the currently calculated playback clock value is consistent with the playback clock value calculated when the previous frame arrived, the currently arriving video data will be written to a new frame buffer area.

7. The ultra-high-definition audio and video synchronization system based on PTP network synchronization according to claim 1 is characterized in that, During the video data playback process, the discarding or repetition of the video data is controlled by the playback clock value at the playback moment, so that the video data after the PTP network synchronization is synchronized.

8. The ultra-high-definition audio and video synchronization system based on PTP network synchronization according to claim 1 is characterized in that: The audio synchronization module includes: A sampling rate clock generation module, used to generate an audio sampling rate clock synchronized with a local clock; The audio sampling rate conversion module is used to synchronize the unsynchronized audio data to the audio sampling rate clock generated by the sampling rate clock generation module.

9. The ultra-high-definition audio and video synchronization system based on PTP network synchronization according to any one of claims 1 to 8, characterized in that: The synchronization reference sources of the video data and audio data are both PTP clock reference sources, and the synchronization accuracy is sub-microsecond level.

Images