Multi-source Synchronous Detection Method, Device and System

Abstract

The present application provides a multi-source synchronization detection method, device and system, which relates to the technical field of video transmission and display control. The method can be applied to an encoder and includes: receiving a segmented video source with frame number counting; encoding the segmented video source and marking an encoded frame number in each frame of the received segmented video source; sending the encoded segmented video source to a decoder, so that the decoder decodes the segmented video source and marks a decoded frame number in each frame of the segmented video source, and detects whether the original video source is synchronized according to the frame number counting, the encoded frame number and the decoded frame number. By using the method provided in the embodiments of the present application, the detection of the reasons for video asynchronization can be realized, and the efficiency and accuracy of multi-source video source synchronization detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of video transmission and display control, and more particularly, to a multi-source synchronization detection method, apparatus, and system. Background Art

[0002] Compared with picture data, the biggest difference in video data is that it includes timing data. When transmitting video data, in view of the drawback of limited clock frequency in a general clock system, a new timing system, called a source synchronous timing system, has been designed. Its greatest advantage is that it increases the speed of the bus, and theoretically the signal transmission can be unaffected by transmission delay. However, in video source synchronization technology, due to reasons such as the video source itself, network packet loss or congestion, there will be occasional display out-of-sync problems, which will cause the splicing screen to tear. However, since the process of video source synchronization is in a black box state, there is currently no suitable way to detect the reasons for video out-of-sync. Summary of the Invention

[0003] Based on this, the purpose of the embodiments of the present application is to provide a multi-source synchronization detection method, apparatus, and system to achieve the detection of the reasons for video out-of-sync, and to improve the efficiency and accuracy of multi-source video source synchronization detection.

[0004] In a first aspect, an embodiment of the present application provides a multi-source synchronization detection method applied to an encoder, including:

[0005] Receiving a segmented video source with a frame number count;

[0006] Encoding the segmented video source and marking an encoded frame number in each frame of the received segmented video source;

[0007] Sending the encoded segmented video source to a decoder, so that the decoder decodes the segmented video source and marks a decoded frame number in each frame of the segmented video source, and detects whether the original video source is synchronized according to the frame number count, the encoded frame number, and the decoded frame number.

[0008] In the above implementation process, the process of video source synchronization is detected based on a test video with a frame number count. By adding an encoding mark and a decoding mark during the encoding process and the decoding process of video source synchronization, the process of multi-source video source synchronization is detected based on the frame number count, the encoded frame number, and the decoded frame number in each segmented video source when displaying the video source. The embodiments of the present application analyze the source synchronization process in a black box state based on the frame number count, the encoded frame number, and the decoded frame number, so as to be able to detect the reasons for video out-of-sync and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0009] Optionally, encoding the segmented video source and marking the encoded frame number in each frame of the received segmented video source includes:

[0010] Collecting frame number counting pictures from the segmented video source and performing source synchronization on the frame number counting pictures;

[0011] Based on the frame number after source synchronization and modulo operation, superimposing the encoded frame number on each frame number counting picture, and encoding the frame number counting picture to obtain an encoded bitstream;

[0012] Calculating a timestamp according to the frame number in the encoded bitstream and packing and sending it to the decoder.

[0013] In the above implementation process, superimposing the encoded frame number on the segmented video source during the encoding process of the segmented video source can determine whether the video source is out of sync due to encoding based on the encoded frame number when displaying the video source, thereby improving the detection efficiency of multi-source synchronization.

[0014] Optionally, performing source synchronization on the frame number counting picture includes:

[0015] When the encoder is the main encoder, when a frame of image is collected, incrementing the frame number and sending the frame number to a preset multicast address in a multicast manner;

[0016] When the encoder is a slave encoder, when a frame of image is collected, incrementing the frame number and obtaining the main encoder frame number from the preset multicast address. When the own frame number is inconsistent with the main encoder frame number, correcting the own frame number to the main encoder frame number, and each encoder collects images in different segmented video sources.

[0017] In the above implementation process, performing source synchronization on the frame number counting picture in the segmented video source to characterize the timing of the frame number counting picture, and synchronously transmitting the picture data and the source synchronization clock signal to ensure that the transmission times of the two signals are exactly the same. As long as the timing at the encoder sending end is correct, it can ensure that the correct timing is received, thereby improving the accuracy of the transmitted data.

[0018] In a second aspect, an embodiment of the present application provides a multi-source synchronization detection method applied to a decoder, including:

[0019] Receiving segmented video sources sent from multiple encoders respectively, where the segmented video sources carry frame numbers and each frame is marked with an encoded frame number;

[0020] Decode the segmented video source, and mark the decoded frame number in each frame of the segmented video source, so as to detect whether the original video source is synchronized according to the frame number count, the encoded frame number, and the decoded frame number.

[0021] In the above implementation process, the process of video source synchronization is detected based on a test video with frame number counting. By adding encoding marks and decoding marks during the encoding process and decoding process of video source synchronization, the process of multi-source video source synchronization is detected based on the frame number count, encoded frame number, and decoded frame number in each segmented video source when displaying the video source. The embodiments of the present application analyze the source synchronization process in the black box state based on the frame number count, encoded frame number, and decoded frame number, so as to be able to detect the reasons for video desynchronization, and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0022] Optionally, the marking the decoded frame number in each frame of the segmented video source includes:

[0023] Overlay the decoded frame number on each frame number count picture according to the timestamp in the segmented video source;

[0024] Send the segmented video source with the decoded frame number overlaid to the display.

[0025] In the above implementation process, overlaying the decoded frame number in the segmented video source during the decoding process of the segmented video source can determine whether the video source is desynchronized due to decoding based on the decoded frame number when displaying the video source, thereby improving the detection efficiency of multi-source synchronization.

[0026] In a third aspect, an embodiment of the present application provides a multi-source synchronization detection method applied to a multi-source synchronization detection system. The multi-source synchronization detection system includes a plurality of encoders and at least one decoder, and each encoder is connected to the decoder. The method includes:

[0027] Receive a segmented video source based on the encoder, and the segmented video source carries a frame number count;

[0028] Encode the segmented video source based on the encoder, mark the encoded frame number in each frame of the received segmented video source, and send the encoded segmented video source to the decoder;

[0029] Decode the segmented video source based on the decoder, and mark the decoded frame number in each frame of the segmented video source;

[0030] Detect whether the original video source is synchronized according to the frame number count, the encoded frame number, and the decoded frame number.

[0031] In the above implementation process, the process of video source synchronization is detected based on a test video with frame number counting. By adding encoding markers and decoding markers during the encoding process and decoding process of video source synchronization, the process of multi-source video source synchronization is detected based on the frame number counting, encoded frame numbers, and decoded frame numbers in each segmented video source when displaying the video source. The embodiments of the present application analyze the source synchronization process in the black box state based on the frame number counting, encoded frame numbers, and decoded frame numbers, so as to be able to detect the reasons for video desynchronization and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0032] Optionally, before the encoder receives the segmented video source, the method includes:

[0033] Segment the original video source to obtain a plurality of the segmented video sources, where one segmented video source corresponds to one encoder, to control the encoder to receive the segmented video source corresponding to itself.

[0034] Optionally, after detecting whether the original video source is synchronized according to the frame number counting, the encoded frame numbers, and the decoded frame numbers, the method further includes:

[0035] When the frame number counting of each segmented video source is the same in the decoded original video source, determine that the original video source is a synchronized video;

[0036] When the frame number counting of each segmented video source is different, determine the reason for the desynchronization of the original video source based on the encoded frame numbers, the decoded frame numbers, and the difference between the encoded frame numbers and the decoded frame numbers of the segmented video source.

[0037] In the above implementation process, it is possible to quickly determine the reasons for video desynchronization based on the frame number counting, encoded frame numbers, and decoded frame numbers, and quickly determine the specific link where the desynchronization problem occurs. The source synchronization process in the black box state is analyzed based on the frame number counting, encoded frame numbers, and decoded frame numbers, so as to be able to detect the reasons for video desynchronization and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0038] Fourthly, the embodiments of the present application further provide a multi-source synchronization detection device, and the device may include:

[0039] A receiving module, configured to receive the segmented video source based on an encoder, and the segmented video source carries frame number counting;

[0040] An encoding module, configured to encode the segmented video source based on the encoder, mark the encoded frame numbers in each frame of the received segmented video source, and send the encoded segmented video source to the decoder;

[0041] A decoding module, configured to decode the segmented video source based on the decoder, and mark a decoded frame number in each frame of the segmented video source;

[0042] A detection module, configured to detect whether the original video source is synchronized according to the frame number count, the encoded frame number, and the decoded frame number.

[0043] In the above implementation process, the process of video source synchronization is detected based on a test video with frame number counting. By adding encoding marks and decoding marks during the encoding process and decoding process of video source synchronization, the process of multi-source video source synchronization is detected based on the frame number count, the encoded frame number, and the decoded frame number in each segmented video source when displaying the video source. The embodiments of the present application analyze the source synchronization process in a black box state based on the frame number count, the encoded frame number, and the decoded frame number, so as to be able to detect the reasons for video desynchronization, and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0044] Optionally, the multi-source synchronization detection device may further include:

[0045] A segmentation module, configured to segment the original video source to obtain a plurality of the segmented video sources, and one of the segmented video sources corresponds to one encoder, so as to control the encoder to receive the segmented video source corresponding to itself.

[0046] Optionally, the multi-source synchronization detection device may further include:

[0047] A judgment module, configured to determine that the original video source is a synchronized video when the frame number counts of each of the segmented video sources in the decoded original video source are the same; and when the frame number counts of each of the segmented video sources are different, determine the reason for the desynchronization of the original video source based on the encoded frame number, the decoded frame number, and the difference between the encoded frame number and the decoded frame number of the segmented video source.

[0048] In the above implementation process, the reason for video desynchronization can be quickly determined based on the frame number count, the encoded frame number, and the decoded frame number, and the specific link where the desynchronization problem occurs can be quickly judged. The source synchronization process in a black box state is analyzed based on the frame number count, the encoded frame number, and the decoded frame number, so as to be able to detect the reasons for video desynchronization, and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0049] In a fifth aspect, the embodiments of the present application further provide a multi-source synchronization detection system. The multi-source synchronization detection system may include a plurality of encoders and at least one decoder. When receiving a program instruction, the encoder and the decoder are configured to execute the steps in the multi-source synchronization detection method applied to the multi-source synchronization detection system as described above. Description of the Drawings

[0050] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following accompanying drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related accompanying drawings can also be obtained based on these drawings.

[0051] Figure 1 Schematic diagram of the steps of the multi-source synchronization detection method applied to an encoder provided by an embodiment of the present application;

[0052] Figure 2 Schematic diagram of the steps of superimposing coding marks on a segmented video source provided by an embodiment of the present application;

[0053] Figure 3 Schematic diagram of the steps of the source synchronization process provided by an embodiment of the present application;

[0054] Figure 4 Schematic diagram of the steps of the multi-source synchronization detection method applied to a decoder provided by an embodiment of the present application;

[0055] Figure 5 Schematic diagram of the steps of editing decoding frame numbers in a segmented video source provided by an embodiment of the present application;

[0056] Figure 6 Schematic diagram of the steps of the multi-source synchronization detection method applied to a multi-source synchronization detection system provided by an embodiment of the present application;

[0057] Figure 7 Schematic diagram of the steps of determining the reasons for video desynchronization provided by an embodiment of the present application;

[0058] Figure 8 Schematic diagram of the multi-source synchronization detection device provided by an embodiment of the present application. Detailed implementation manners

[0059] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, may be implemented using a dedicated hardware-based system that performs the specified functions or actions, or may be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present invention may be integrated together to form a separate part, or each module may exist separately, or two or more modules may be integrated to form a separate part.

[0060] During the research process, the applicant found that in the existing video source synchronization technology, due to the video source itself, network packet loss or blockage, etc., occasional display asynchrony problems may occur, resulting in tearing of the video splicing screen. Since the problem occurs for a short time and the video source synchronization process is in a black box state, it is impossible to determine the cause of the video asynchrony.

[0061] Based on this, the embodiment of the present application provides a multi-source synchronization detection method, which is based on the encoder and decoder to perform frame number superposition processing on the test video source, and judge the cause of video source synchronization by the frame number and the frame number of the test video source. Figure 1 , Figure 1 This is a schematic diagram of the steps of the multi-source synchronous detection method applied to an encoder provided in an embodiment of the present application. The multi-source synchronous detection process applied to the encoder may include the following steps:

[0062] In step S11 , a segmented video source is received, wherein the segmented video source carries a frame number count.

[0063] Among them, the split video source is a video that displays part of the original video content after the original video passes through the video splitter. Frame number counts can be added to each split video source when the video splitter splits the original video, or frame number counts can be pre-added to each part of the original video before the video splitter splits the original video.

[0064] In step S12, the segmented video source is encoded, and an encoded frame number is marked in each frame of the received segmented video source.

[0065] In step S13, the encoded segmented video source is sent to a decoder, so that the decoder decodes the segmented video source, and a decoded frame number is marked in each frame of the segmented video source, to detect whether the original video source is synchronized according to the frame number count, the encoded frame number, and the decoded frame number.

[0066] Among them, after the encoder and the decoder mark the encoded frame number and the decoded frame number for the segmented video source, each segmented video source contains three values: the frame number count, the encoded frame number, and the decoded frame number. The encoded frame number and the decoded frame number can be placed in the upper left corner and the upper right corner of the image frame respectively. By comparing whether the frame number count values of each segmented video source are equal, and comparing the encoded frame number and the decoded frame number and the differences between the encoded frame numbers and the decoded frame numbers of each segmented video source, the reason for video desynchronization can be determined.

[0067] It can be seen that in the embodiment of the present application, the process of video source synchronization is detected based on a test video with a frame number count. By adding an encoding mark and a decoding mark during the encoding process and the decoding process of video source synchronization, the process of multi-source video source synchronization is detected based on the frame number count, the encoded frame number, and the decoded frame number in each segmented video source when the video source is displayed. In the embodiment of the present application, the source synchronization process in the black box state is analyzed based on the frame number count, the encoded frame number, and the decoded frame number, so as to be able to detect the reason for video desynchronization, and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0068] In an optional embodiment, for step S12 of encoding the segmented video source and marking an encoded frame number in each frame of the received segmented video source, the embodiment of the present application provides an implementation method for superimposing an encoding mark on the segmented video source during the encoding process. Please refer to Figure 2 , Figure 2 which is a schematic diagram of the steps for superimposing an encoding mark on the segmented video source provided by the embodiment of the present application. The steps for superimposing an encoding mark on the segmented video source may include the following:

[0069] In step S21, a frame number count picture is collected from the segmented video source, and source synchronization is performed on the frame number count picture.

[0070] Among them, the frame number counting picture can be a YUV picture. YUV is a color encoding method, which is divided into three components. "Y" represents luminance, that is, the grayscale value; while "U" and "V" represent chrominance, which is used to describe the influence on color and saturation, and specify the color of pixels. YUV separates the luminance information (Y) from the color information (UV). Without UV information, a complete image can still be displayed, but it is black and white. Such a design enables YUV not to require three independent video signals to be transmitted simultaneously like the RGB data format. Transmitting in the YUV format only requires extremely little bandwidth.

[0071] In step S22, based on the frame number after source synchronization and the modulo operation, the encoded frame number is superimposed on each of the frame number counting pictures, and the frame number counting pictures are encoded to obtain an encoded bitstream.

[0072] Exemplarily, the modulo operation can be a modulo 60 operation (Fsync % 60). By superimposing the corresponding frame number YUV small pictures (numbered 1 - 60) onto the video screen. For example, if the currently source-synchronized frame number Fsync = 1122, the value obtained through the Fsync % 60 operation is 42. The YUV picture with the number 42 is superimposed on the video screen, and the frame number keeps increasing. The superimposed pictures can cycle and increase with a period of 60.

[0073] In step S23, a timestamp is calculated according to the frame number in the encoded bitstream and is packed and sent to the decoder.

[0074] Exemplarily, the timestamp can be calculated based on the current encoding frame rate and the frame number obtained from the bitstream, timestamp = frame number * 90000 / frame rate. Among them, 90000 is the clock frequency corresponding to the timestamp according to the RFC3984 / RFC7798 protocol (the definition of RTP payload being h.264 / h.265) document.

[0075] It can be seen that in the process of encoding the segmented video source in the embodiment of the present application, the encoded frame number is superimposed on the segmented video source, which can determine whether the video source is out of sync due to encoding based on the encoded frame number when displaying the video source, thereby improving the detection efficiency of multi-source synchronization.

[0076] In an optional embodiment, for the part of source synchronization of the frame number counting pictures in step S21, the embodiment of the present application provides an implementation manner of source synchronization for the frame number counting pictures. Please refer to Figure 3 , Figure 3 which is a schematic diagram of the steps of the source synchronization process provided by the embodiment of the present application. The steps of source synchronization for the frame number counting pictures may include:

[0077] In step S31, when the encoder is the main encoder, when a frame of image is captured, the frame number is incremented and the frame number is sent to a preset multicast address in a multicast manner.

[0078] In step S32, when the encoder is a slave encoder, when a frame of image is captured, the frame number is incremented, and the main encoder frame number is obtained from the preset multicast address. When the own frame number is inconsistent with the main encoder frame number, the own frame number is corrected to the main encoder frame number, and each encoder captures images in different segmented video sources.

[0079] Specifically, multiple encoders can encode the original video source. After the original video source is segmented, multiple segmented video sources are obtained. One encoder encodes one segmented video source. The encoders can be grouped and the main encoder and slave encoders are determined. The main encoder can be a preset encoder or randomly selected from multiple encoders based on a random algorithm. Connections are established between multiple encoders to form a multicast network.

[0080] In addition, as an optional implementation manner, a frame interval can be set. The main encoder sends the current encoded frame number to the preset multicast address after each frame interval. The slave encoder obtains the main encoder frame number from the preset multicast address after each frame interval. When the own frame number is inconsistent, the current frame number is corrected to the main encoder frame number, and the frame numbers of the previous frame are sequentially corrected based on the current frame until the frame number of the previous frame interval is corrected.

[0081] It can be seen that the embodiment of the present application synchronizes the source of the frame number counting pictures in the segmented video source to characterize the timing of the frame number counting pictures. By synchronously transmitting the picture data and the source synchronization clock signal, it is ensured that the transmission times of the two signals are exactly the same. As long as the timing at the encoder sending end is correct, it can be ensured that the completely correct timing is received at the receiving end, thereby improving the accuracy of the transmitted data.

[0082] Based on the same inventive concept, the embodiment of the present application further provides a multi-source synchronization detection method applied to a decoder. Please refer to Figure 4 , Figure 4 which is a schematic diagram of the steps of the multi-source synchronization detection method applied to the decoder provided by the embodiment of the present application. The multi-source synchronization detection method applied to the decoder can include the following steps:

[0083] In step S41, segmented video sources sent from multiple encoders are respectively received. The segmented video sources carry frame number counting and each frame is marked with an encoded frame number.

[0084] In step S42, the segmented video source is decoded, and a decoded frame number is marked in each frame of the segmented video source, so as to detect whether the original video source is synchronized according to the frame number count, the encoded frame number, and the decoded frame number.

[0085] Exemplarily, one decoder can be used to receive the data sent by all the encoders, or multiple decoders can be used to decode the data. Each decoder corresponds to one or more encoders. The segmented video source data sent by the decoders is synchronously spliced by multiple encoders and sent to the display for display.

[0086] It can be seen that in the embodiment of the present application, the process of video source synchronization is detected based on the test video with frame number counting. By adding an encoding mark and a decoding mark during the encoding process and the decoding process of video source synchronization, the process of multi-source video source synchronization is detected based on the frame number count, the encoded frame number, and the decoded frame number in each segmented video source when the video source is displayed. The embodiment of the present application analyzes the source synchronization process in the black box state based on the frame number count, the encoded frame number, and the decoded frame number, so as to be able to detect the reasons for video desynchronization and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0087] In an optional embodiment, for step S41 of marking the decoded frame number in each frame of the segmented video source, the embodiment of the present application provides an implementation manner of marking the decoded frame number in the segmented video source. Please refer to Figure 5 , Figure 5 which is a schematic diagram of the steps of editing the decoded frame number in the segmented video source provided by the embodiment of the present application. The steps of editing the decoded frame number in the segmented video source may include:

[0088] In step S51, the decoded frame number is superimposed on each frame number count picture according to the time stamp in the segmented video source.

[0089] In step S52, the segmented video source after superimposing the decoded frame number is sent to the display.

[0090] Among them, the calculation method of superimposing the decoded frame number can be specifically referred to the description in the above steps S21-S23, and can be calculated inversely according to the formula of the time stamp, which will not be elaborated here.

[0091] It can be seen that in the embodiment of the present application, the decoded frame number is superimposed on the segmented video source during the decoding process of the segmented video source, so as to be able to determine whether the video source is desynchronized due to decoding based on the decoded frame number when the video source is displayed, thereby improving the detection efficiency of multi-source synchronization.

[0092] Based on the same inventive concept, an embodiment of the present application further provides a multi-source synchronization detection method applied to a multi-source synchronization detection system. The multi-source synchronization detection system includes a plurality of encoders and at least one decoder, and each encoder is connected to the decoder. Please refer to Figure 6 , Figure 6 which is a schematic diagram of the steps of the multi-source synchronization detection method applied to the multi-source synchronization detection system provided by the embodiment of the present application. The multi-source synchronization detection method applied to the multi-source synchronization detection system may include the following steps:

[0093] In step S61, based on the encoder, a segmented video source is received, and the segmented video source carries frame number counting.

[0094] In step S62, based on the encoder, the segmented video source is encoded, an encoded frame number is marked in each frame of the received segmented video source, and the encoded segmented video source is sent to the decoder.

[0095] In step S63, based on the decoder, the segmented video source is decoded, and a decoded frame number is marked in each frame of the segmented video source.

[0096] In step S64, according to the frame number counting, the encoded frame number, and the decoded frame number, it is detected whether the original video source is synchronized.

[0097] The embodiment of the present application is described by using two encoders and one decoder for multi-source synchronization detection. Taking the test video as 1080P and 60 frames per second as an example, the original video source is segmented into two left and right segmented video sources, and the two segmented video sources are respectively encoded by two encoders. After encoding, the data is sent to the decoder for decoding and then sent to the splicing display for display. The steps of encoding, decoding, and superimposing the encoded frame number and the decoded frame number can be specifically referred to the corresponding implementation manners above, and will not be elaborated here.

[0098] The video signal source plays the same video picture with frame number counting and a frame rate of 60fps. The end display shows the video picture that has been encoded and decoded by the multi-source synchronization detection system and the encoded frame number and the decoded frame number are superimposed. Among them, the actual frame numbers in the video picture after encoding and decoding can be represented by Fn_L’ and Fn_R’, the encoded frame numbers can be represented by Fe_L and Fe_R, and the decoded frame numbers can be represented by Fd_L and Fd_R. The links where out-of-sync occurs during the multi-source synchronization process can be determined by comparing Fn_L’, Fe_L, Fe_L, Fn_R’, Fe_R, and Fe_R.

[0099] In the actual application process, the number of encoders and decoders can be selected according to the video resolution. The video splicer can be used to splice the segmented video sources first and then send them to the display for display, or the segmented video sources can be directly sent to the tiled display for display.

[0100] It can be seen that in the embodiment of the present application, the process of video source synchronization is detected based on the test video with frame number counting. By adding encoding marks and decoding marks in the encoding process and decoding process of video source synchronization, the process of multi-source video source synchronization is detected based on the frame number counting, encoding frame numbers, and decoding frame numbers in each segmented video source when displaying the video source. In the embodiment of the present application, the source synchronization process in the black box state is analyzed based on the frame number counting, encoding frame numbers, and decoding frame numbers, so as to be able to detect the reasons for video desynchronization and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0101] In an alternative embodiment, before step S61, the method provided in the embodiment of the present application may further include:

[0102] Segment the original video source to obtain a plurality of the segmented video sources, and one segmented video source corresponds to one encoder to control the encoder to receive the segmented video source corresponding to itself.

[0103] Among them, the number of segmented video sources can be specifically selected according to the video resolution. When the video resolution is high, the number of segmented video sources can be increased. When the video resolution is low, the original video source can be selected to be divided into only two segmented video sources.

[0104] In an alternative embodiment, after detecting whether the original video source is synchronized according to the frame number counting, the encoding frame numbers, and the decoding frame numbers in step S64, the embodiment of the present application provides an implementation manner for determining the reasons for video desynchronization according to the frame number counting, encoding frame numbers, and decoding frame numbers. Please refer to Figure 7 , Figure 7 which is the schematic diagram of the steps for determining the reasons for video desynchronization provided by the embodiment of the present application. The steps for determining the reasons for video desynchronization can be as follows:

[0105] In step S71, when the frame number counting of each segmented video source in the decoded original video source is the same, it is determined that the original video source is a synchronized video.

[0106] In step S72, when the frame number counting of each segmented video source is different, the reasons for the desynchronization of the original video source are determined based on the encoding frame numbers, decoding frame numbers, and the differences between the encoding frame numbers and the decoding frame numbers of the segmented video sources.

[0107] Exemplarily, when Fn_L’ and Fn_R’ are equal, it can be determined that the video is synchronized; otherwise, it is asynchronous.

[0108] When the video playback is asynchronous, the values of Fe_L and Fe_R are judged. If Fe_L and Fe_R are equal, it can be determined that the cause of the asynchronous problem appears in the video source or the encoder superposition coding frame number link. Therefore, the specific link causing the problem can be further located by checking whether the output of the video source itself is synchronized and viewing the relevant logs of the encoder.

[0109] If Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L’ is equal to the difference between Fe_R and Fn_R’, it can be determined that the cause of the asynchronous problem appears in the network transmission or decoder processing link. The asynchronous problem caused by network packet loss, congestion, or the decoder's untimely processing of the video stream can be determined by network packet capture and viewing the relevant logs of the decoder.

[0110] If Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L’ is not equal to the difference between Fe_R and Fn_R’, the frame numbers (Fd_L, Fd_R) marked by the decoder and the difference between them and the frame numbers marked by the encoder can be judged. If Fd_L and Fd_R are not equal and the difference between Fd_L and Fe_L is equal to the difference between Fd_R and Fe_R, it can be determined that the cause of the asynchronous problem appears in the video source or the encoder superposition coding frame number link. The specific link causing the problem can be further located by checking whether the output of the video source itself is synchronized and viewing the relevant logs of the encoder.

[0111] It can be seen that the embodiments of the present application can quickly determine the cause of video asynchronization based on frame number counting, coding frame numbers, and decoding frame numbers, and can quickly judge the specific link where the asynchronous problem occurs. The source synchronization process in the black box state is analyzed based on frame number counting, coding frame numbers, and decoding frame numbers, so as to realize the detection of the cause of video asynchronization, and improve the efficiency and accuracy of multi-source video source synchronization detection.

[0112] Based on the same inventive concept, the embodiments of the present application also provide a multi-source synchronization detection device 80. Please refer to Figure 8 , Figure 8 which is a schematic diagram of the multi-source synchronization detection device provided by the embodiments of the present application. The multi-source synchronization detection device 80 may include:

[0113] A receiving module 81, configured to receive a segmented video source based on an encoder, and the segmented video source carries frame number counting.

[0114] An encoding module 82 is configured to encode the segmented video source based on the encoder, mark an encoding frame number in each frame of the received segmented video source, and send the encoded segmented video source to the decoder.

[0115] A decoding module 83 is configured to decode the segmented video source based on the decoder, and mark a decoding frame number in each frame of the segmented video source.

[0116] A detection module 84 is configured to detect whether the original video source is synchronized according to the frame number count, the encoding frame number, and the decoding frame number.

[0117] Optionally, the multi-source synchronization detection device 80 may further include:

[0118] A segmentation module is configured to segment the original video source to obtain a plurality of segmented video sources, and one encoder corresponds to one segmented video source, so as to control the encoder to receive the segmented video source corresponding to itself.

[0119] Optionally, the multi-source synchronization detection device 80 may further include:

[0120] A judgment module is configured to, in the decoded original video source, when the frame number counts of each segmented video source are the same, determine that the original video source is a synchronized video; and when the frame number counts of each segmented video source are different, determine the reason for the non-synchronization of the original video source based on the encoding frame number, the decoding frame number, and the difference between the encoding frame number and the decoding frame number of the segmented video source.

[0121] Based on the same inventive concept, an embodiment of the present application further provides a multi-source synchronization detection system. The multi-source synchronization detection system may include a plurality of encoders and at least one decoder. When receiving a program instruction, the encoder and the decoder are configured to execute the steps in the above multi-source synchronization detection method applied to the multi-source synchronization detection system.

[0122] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point, the displayed or discussed coupling or direct coupling or communication connection between each other may be through some communication interfaces. The indirect coupling or communication connection of the devices or units may be in an electrical, mechanical or other form.

[0123] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] Furthermore, the functional modules in each embodiment of the present application may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0125] Alternatively, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part.

[0126] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).

[0127] In this document, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not expressly listed, or elements that are inherent to such process, method, article, or device. Without further limitation, the elements defined by the statement "including..." do not preclude the existence of additional identical elements in the process, method, article, or device that includes the said elements.

[0128] The above are only the embodiments of the present application and are not used to limit the protection scope of the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-source synchronous detection method, characterized in that Applied to an encoder, including: Receiving a segmented video source with frame number counting; Encoding the segmented video source and marking the encoded frame number in each frame of the received segmented video source; Sending the encoded segmented video source to a decoder so that the decoder decodes the segmented video source and marks the decoded frame number in each frame of the segmented video source to detect whether the original video source is synchronized according to the frame number counting, the encoded frame number, and the decoded frame number; When the frame number count of each of the segmented video sources is different, determining the reason for the asynchronization of the original video source based on the encoded frame number, the decoded frame number of the segmented video source, and the difference between the encoded frame number and the decoded frame number; When Fe_L and Fe_R are equal, determining that the reason for the asynchronization problem appears in the video source or the encoder's link of superimposing the encoded frame number; When Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L' is equal to the difference between Fe_R and Fn_R', determining that the reason for the asynchronization problem appears in the network transmission or the decoder processing link; When Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L' is not equal to the difference between Fe_R and Fn_R', and the difference between Fd_L and Fe_L is equal to the difference between Fd_R and Fe_R, determining that the reason for the asynchronization problem appears in the video source or the encoder's link of superimposing the encoded frame number; Wherein, Fn_L' and Fn_R' represent the actual frame numbers in the video picture after encoding and decoding, Fe_L and Fe_R represent the encoded frame numbers, and Fd_L and Fd_R represent the decoded frame numbers.

2. The method according to claim 1, characterized in that, The encoding the segmented video source and marking the encoded frame number in each frame of the received segmented video source includes: Collecting frame number counting pictures from the segmented video source and performing source synchronization on the frame number counting pictures; Superimposing the encoded frame number on each of the frame number counting pictures based on the source-synchronized frame number and modulo operation, and encoding the frame number counting pictures to obtain an encoded bitstream; Calculating a timestamp according to the frame number in the encoded bitstream and packing and sending it to the decoder.

3. The method according to claim 2, wherein The performing source synchronization on the frame number counting pictures includes: When the encoder is the main encoder, when a frame of image is collected, incrementing the frame number and sending the frame number to a preset multicast address in a multicast manner; When the encoder is a slave encoder, when a frame of image is collected, incrementing the frame number and obtaining the main encoder frame number from the preset multicast address. When the local frame number is inconsistent with the main encoder frame number, correcting the local frame number to the main encoder frame number. Each encoder collects images in different segmented video sources.

4. A multi-source synchronous detection method, characterized in that, Applied to a decoder, including: Receiving segmented video sources sent from multiple encoders respectively, the segmented video sources with frame number counting and each frame marked with an encoded frame number; Decode the segmented video source, and mark the decoded frame number in each frame of the segmented video source to detect whether the original video source is synchronized according to the frame number count, the encoded frame number, and the decoded frame number; When the frame number counts of each of the segmented video sources are different, determine the reason for the asynchronization of the original video source based on the encoded frame number, the decoded frame number, and the difference between the encoded frame number and the decoded frame number of the segmented video source; When Fe_L and Fe_R are equal, determine that the reason for the asynchronization problem appears in the video source or the encoder's superimposing of the encoded frame number; When Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L' is equal to the difference between Fe_R and Fn_R', determine that the reason for the asynchronization problem appears in the network transmission or the decoder processing; When Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L' is not equal to the difference between Fe_R and Fn_R', and the difference between Fd_L and Fe_L is equal to the difference between Fd_R and Fe_R, determine that the reason for the asynchronization problem appears in the video source or the encoder's superimposing of the encoded frame number; Wherein, Fn_L' and Fn_R' represent the actual frame numbers in the video pictures after encoding and decoding, Fe_L and Fe_R represent the encoded frame numbers, and Fd_L and Fd_R represent the decoded frame numbers.

5. The method according to claim 4, wherein The marking of the decoded frame number in each frame of the segmented video source includes: Superimpose the decoded frame number onto each frame number count picture according to the time stamp in the segmented video source; Send the segmented video source after superimposing the decoded frame number to the display.

6. A multi-source synchronous detection method, characterized in that, Applied to a multi-source synchronization detection system, the multi-source synchronization detection system includes multiple encoders and at least one decoder, and each encoder is connected to the decoder. The method includes: Receive a segmented video source based on the encoder, and the segmented video source has a frame number count; Encode the segmented video source based on the encoder, mark the encoded frame number in each frame of the received segmented video source, and send the encoded segmented video source to the decoder; Decode the segmented video source based on the decoder, and mark the decoded frame number in each frame of the segmented video source; Detect whether the original video source is synchronized according to the frame number count, the encoded frame number, and the decoded frame number; When the frame number counts of each of the segmented video sources are different, determine the reason for the asynchronization of the original video source based on the encoded frame number, the decoded frame number, and the difference between the encoded frame number and the decoded frame number of the segmented video source; When Fe_L and Fe_R are equal, determine that the reason for the asynchronization problem appears in the video source or the encoder's superimposing of the encoded frame number; When Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L' is equal to the difference between Fe_R and Fn_R', determine that the reason for the asynchronization problem appears in the network transmission or the decoder processing; When Fe_L and Fe_R are not equal, the difference between Fe_L and Fn_L’ is not equal to the difference between Fe_R and Fn_R’, and the difference between Fd_L and Fe_L is equal to the difference between Fd_R and Fe_R, it is determined that the cause of the out-of-sync problem appears in the video source or the encoder overlay encoding frame number link; Among them, Fn_L’ and Fn_R’ represent the actual frame numbers in the video picture after encoding and decoding, Fe_L and Fe_R represent the encoding frame numbers, and Fd_L and Fd_R represent the decoding frame numbers.

7. The method according to claim 6, wherein Before the encoder receives the segmented video source, the method includes: Segment the original video source to obtain a plurality of segmented video sources, where one segmented video source corresponds to one encoder, so as to control the encoder to receive the segmented video source corresponding to itself.

8. The method according to claim 6, characterized in that, After detecting whether the original video source is synchronized according to the frame number count, the encoding frame number and the decoding frame number, the method further includes: In the decoded original video source, when the frame number counts of each segmented video source are the same, it is determined that the original video source is a synchronized video; When the frame number counts of each segmented video source are different, based on the encoding frame number, the decoding frame number of the segmented video source, and the difference between the encoding frame number and the decoding frame number, determine the reason for the out-of-sync of the original video source.

9. A multi-source synchronous detection device, characterized in that, It includes: A receiving module, configured to receive a segmented video source based on an encoder, where the segmented video source carries a frame number count; An encoding module, configured to encode the segmented video source based on the encoder, mark the encoding frame number in each frame of the received segmented video source, and send the encoded segmented video source to the decoder; A decoding module, configured to decode the segmented video source based on the decoder, and mark the decoding frame number in each frame of the segmented video source; A detection module, configured to detect whether the original video source is synchronized according to the frame number count, the encoding frame number, and the decoding frame number; A judgment module, configured to determine that the original video source is a synchronized video when the frame number counts of each segmented video source are the same in the decoded original video source; And when the frame number counts of each of the segmented video sources are different, determining the reason for the asynchronization of the original video source based on the encoded frame number, the decoded frame number, and the difference between the encoded frame number and the decoded frame number of the segmented video source; wherein, when the frame number counts of each of the segmented video sources are different, determining the reason for the asynchronization of the original video source based on the encoded frame number, the decoded frame number, and the difference between the encoded frame number and the decoded frame number of the segmented video source includes: when Fe_L and Fe_R are equal, determining that the reason for the asynchronization problem appears in the video source or the encoder superposition encoded frame number link; when Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L' is equal to the difference between Fe_R and Fn_R', determining that the reason for the asynchronization problem appears in the network transmission or the decoder processing link; when Fe_L and Fe_R are not equal, and the difference between Fe_L and Fn_L' is not equal to the difference between Fe_R and Fn_R', and the difference between Fd_L and Fe_L is equal to the difference between Fd_R and Fe_R, determining that the reason for the asynchronization problem appears in the video source or the encoder superposition encoded frame number link; wherein, Fn_L' and Fn_R' represent the actual frame numbers in the video picture after encoding and decoding, Fe_L and Fe_R represent the encoded frame numbers, and Fd_L and Fd_R represent the decoded frame numbers.

10. A multi-source synchronous detection system, characterized in that, Including a plurality of encoders and at least one decoder, the encoder and the decoder are configured to execute the steps in the method according to any one of claims 6-8 when receiving a program instruction.

Images