Distributed video synchronous display control method and computer readable storage medium

Abstract

The invention discloses a distributed video synchronous display control method and a computer readable storage medium, and the method comprises the steps: a drive layer of each slave adds a to-be-displayed picture into a frame buffer queue, and reports a display timestamp of the to-be-displayed picture to a host, the display timestamp of the to-be-displayed picture is determined according to the video stream ID corresponding to the to-be-displayed picture and the display timestamp of the video frame; when the host detects that each slave reports the same display timestamp, the display time of the to-be-displayed picture corresponding to the same display timestamp is determined according to the same display timestamp, and the display time is broadcasted to each slave; and according to the display time, the driving layer of each slave obtains a to-be-displayed picture corresponding to the same display timestamp from the frame buffer queue and sends the to-be-displayed picture to a display for display. According to the invention, zero-frame-difference synchronous playing display of the distributed playing terminal can be realized.

Description

Technical Field

[0001] This invention relates to the field of video synchronization display technology, and in particular to a distributed video synchronization display control method and a computer-readable storage medium. Background Technology

[0002] Current mainstream software / hardware video encoding and decoding technologies are limited to supporting only 8K resolution video file encoding, decoding, and display; they cannot support 16K and higher resolution video files. For scenarios requiring 16K and higher resolution displays, a distributed terminal synchronous playback and splicing display approach is typically used. The distributed display effect requires that the images from each terminal can be spliced ​​together at any given time to form a complete 16K or higher resolution image, with zero frame difference between video playback terminals and no sense of misalignment. Achieving this technical specification requires two conditions: 1. The refresh frequency and phase of each terminal display must be consistent, meaning the horizontal and vertical signals must be synchronized. This requirement can be met by using the PTP protocol (IEEE 1588 protocol) to achieve clock synchronization, with the clock error of each terminal within 1 microsecond; furthermore, by dynamically adjusting the phase-locked loop, microsecond-level synchronized refresh can be achieved for each terminal.

[0003] 2. Controlling the simultaneous display of the same frame image segment on each distributed video playback terminal. This is typically achieved by comparing the display time (PTS) of the decoded video frame with the synchronized system clock within the application. However, after synchronization in the application, the displayed image still needs to be processed by the application layer, system layer, and driver layer. Each of these processes can potentially lead to video frame desynchronization. Therefore, this solution cannot guarantee that video frames on each terminal will display with zero frame difference at any given time. For example, if a terminal's video segment decoding is complex, leading to excessive system load and a drop in frame rate, it may be unable to output a synchronized display with other terminals.

[0004] Therefore, how to achieve synchronous playback and splicing display with zero frame difference from multiple distributed playback terminals has become an urgent problem to be solved.

[0005] Chinese patent publication CN111601180A discloses a PTS-based distributed splicing large-screen video stream cluster synchronous display system, comprising: a multicast receiving module, a TS stream unpacking module, a video decoding module, a synchronization control module, and a display module. The multicast receiving module, TS stream unpacking module, and video decoding module each operate independently on the system's master node and each child node. The multicast receiving module receives multicast data; the TS stream unpacking module extracts the video stream from the composite signal stream and splices it into a single frame of video data, which is then input into the video decoding module; the video decoding module decodes the spliced ​​frame of video data to obtain the timestamp of the video stream; the synchronization control module controls the child node to operate synchronously with the master node; and the display module converts the timestamp of the decoded video stream and sends it to the display screen. However, multicast is implemented based on the UDP protocol, which carries the risk of packet loss and cannot strictly guarantee that all terminals receive the synchronization message. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a distributed video synchronous display control method and a computer-readable storage medium, which can realize synchronous playback and display with zero frame difference in distributed playback terminals.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a distributed video synchronous display control method, comprising: Each slave device's driver layer adds the image to be displayed to the frame buffer queue and reports the display timestamp of the image to be displayed to the host. The display timestamp of the image to be displayed is determined based on the video stream ID and video frame display timestamp corresponding to the image to be displayed. When the host detects that all slave devices have reported the same display timestamp, it determines the display time of the screen to be displayed corresponding to the same display timestamp based on the same display timestamp, and broadcasts the display time to all slave devices. Each slave device's driver layer retrieves the image to be displayed corresponding to the same display timestamp from the frame buffer queue and sends it to the display for display.

[0008] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0009] The beneficial effects of this invention are as follows: By setting the display timestamp of the screen to be displayed according to the display timestamp (PTS) of the video frame corresponding to the screen to be displayed and the video stream ID corresponding to the video frame, the screen to be displayed also has PTS information. This PTS information is then transmitted to the graphics card driver layer, enabling frame buffering and frame synchronization operations at the very end of the display process. By reporting the display timestamp of the rendered screen to be displayed to the host, the host confirms that all slave devices have completed rendering and are ready for display, and notifies all slave devices to display the screen at the same time, thereby achieving distributed multi-terminal absolute zero frame difference synchronous display. Attached Figure Description

[0010] Figure 1 This is a flowchart of a distributed video synchronization display control method according to the present invention; Figure 2 This is a flowchart of the method according to Embodiment 1 of the present invention. Detailed Implementation

[0011] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0012] Glossary: Surface: Can be understood as a pixel buffer for a single layer. In Android, all displayable content (application interface, status bar, wallpaper, pop-ups) corresponds to a Surface.

[0013] SurfaceFlinger: Combines pixel data from multiple Surfaces into a single complete screen frame according to rules such as Z-order (hierarchical order), transparency, and cropping area.

[0014] Framebuffer: This is a standardized abstraction of display hardware (such as LCD and OLED screens) in Linux / Android, and it is the final destination for all output. Framebuffer acts as a proxy for the screen's video memory; any pixel data that is intended to be displayed must ultimately be written to the buffer it manages before it can be seen by the screen.

[0015] PTS (Presentation Time Stamp): The core timestamp mechanism in video streaming formats used to control audio and video synchronization, primarily indicating the display sequence of video frames. That is, the offset time of a video frame relative to the start playback time of the video stream.

[0016] Please refer to Figure 1 A distributed video synchronization display control method, comprising: Each slave device's driver layer adds the image to be displayed to the frame buffer queue and reports the display timestamp of the image to be displayed to the host. The display timestamp of the image to be displayed is determined based on the video stream ID and the display timestamp of the video frame corresponding to the image to be displayed. When the host detects that all slave devices have reported the same display timestamp, it determines the display time of the screen to be displayed corresponding to the same display timestamp based on the same display timestamp, and broadcasts the display time to all slave devices. Each slave device's driver layer retrieves the image to be displayed corresponding to the same display timestamp from the frame buffer queue based on the display time and sends it to the display for display.

[0017] As can be seen from the above description, the beneficial effects of the present invention are: it can achieve synchronous playback and display with zero frame difference in distributed playback terminals. When multiple terminals are spliced ​​together, the images of each terminal are captured simultaneously at any time using a camera with a shutter speed of 1 / 1000 second, and there is no misalignment in the spliced ​​image; when multiple terminals play the same video, each terminal displays the same frame at any time.

[0018] Furthermore, before the driver layer of each slave device adds the image to be displayed to the frame buffer queue and reports the display timestamp of the image to be displayed to the host, it further includes: Each slave device's application layer renders each video frame in the video to be played, frame by frame. Generate the corresponding display screen based on the rendered video frames, and set the display timestamp of the corresponding display screen based on the display timestamp of the video frames and the video stream ID corresponding to the video frames; The screen to be displayed is sent to the operating system layer; After receiving the image to be displayed, the operating system layer sends it to the driver layer. As described above, the PTS information is passed to the graphics card driver layer, and the PTS information is also included in the GPU output image. Furthermore, the frames to be displayed after GPU rendering are cached in video memory, so high CPU or GPU load has little impact on the display of the current frame, ensuring that images with set display times are displayed and output on time.

[0019] Furthermore, the application layer sets the task level of the screen to be displayed to the highest priority, and after the operating system layer receives the screen to be displayed, it prioritizes sending the screen to be displayed to the driver layer.

[0020] Furthermore, before the application layer of each slave device renders each video frame in the video to be played frame by frame, the process further includes: Set a unique identifier for the video to be played, where the unique identifier is the video stream ID; The video to be played, or the segmented video to be played, is distributed to each slave device.

[0021] Furthermore, it also includes: If it is detected that the number of images to be displayed in the frame buffer queue of a slave device exceeds a preset ratio of the length of the frame buffer queue, then the images to be discarded are identified in the frame buffer queue of the slave device and removed from the frame buffer queue of the slave device. The host notifies other slave devices to discard the image based on the display timestamp of the image to be discarded.

[0022] As described above, when it is detected that the frame buffer queues of some slave devices are about to overflow, frames are dropped. The principle of dropping every other frame can be followed to determine the video frames to be dropped in order to ensure the continuity of the picture.

[0023] Furthermore, the length of the frame buffer queue is 10-15 frames.

[0024] As described above, in the past, the FrameBuffer cached 2-3 frames of images to be displayed (such as double-buffering or triple-buffering display mechanisms). This invention increases the length of the FrameBuffer queue, for example, by increasing the FrameBuffer to cache up to 15 frames, so that the system has more redundant time to set the display time of the images to be displayed.

[0025] Furthermore, when the host detects that all slave devices have reported the same display timestamp, it determines the display time of the screen to be displayed corresponding to the same display timestamp based on the same display timestamp, and broadcasts the display time to all slave devices, including: If the host detects that all slave devices have reported the same display timestamp, it determines whether the screen to be displayed corresponding to the same display timestamp is the first screen to be displayed. If so, the host determines the display time of the screen to be displayed corresponding to the same display timestamp based on the current time and the preset delay threshold, and takes the display time of the first screen to be displayed as the starting display time; If not, then the display time of the screen to be displayed corresponding to the same display timestamp is determined based on the start display time and the display timestamp of the screen to be displayed corresponding to the same display timestamp; The host broadcasts the display command for the screen to be displayed corresponding to the same display timestamp to each slave device. The display command includes the display timestamp and the display time.

[0026] As described above, the offset (i.e., the delay threshold) is used to adjust the screen refresh time. All screens are displayed with a delay, giving the system enough time to process frame synchronization.

[0027] Furthermore, after the driver layer of each slave device retrieves the image to be displayed corresponding to the same display timestamp from the frame buffer queue according to the display time and sends it to the display for display, it further includes: Remove the image to be displayed corresponding to the same display timestamp from the frame buffer queue.

[0028] As described above, by promptly removing images that have already been sent to the display, the frame buffer queue can be guaranteed to have space to store new rendered images to be displayed.

[0029] Furthermore, before the driver layer of each slave device retrieves the image to be displayed corresponding to the same display timestamp from the frame buffer queue according to the display time and sends it to the display for display, it further includes: After each slave device's driver layer receives the display time of the screen to be displayed corresponding to the same display timestamp, it determines whether the screen to be displayed corresponding to the same display timestamp can be displayed on time. If not, the screen to be displayed corresponding to the same display timestamp is discarded, and the host notifies other slaves to discard the screen to be displayed corresponding to the same display timestamp. If each slave device can display the screen to be displayed corresponding to the same display timestamp on time, then the driver layer of each slave device will execute the step of retrieving the screen to be displayed corresponding to the same display timestamp from the frame buffer queue according to the display time and sending it to the display for display.

[0030] As described above, since some slave devices may not receive the display command in time due to excessive load or network jitter, further judgment is made on whether the screen to be displayed can be displayed on time to ensure display synchronization.

[0031] Furthermore, after receiving the display time of the screen to be displayed corresponding to the same display timestamp, the driver layer of each slave device determines whether the screen to be displayed corresponding to the same display timestamp can be displayed on time, including: When a slave device's driver layer receives the display time of the screen to be displayed corresponding to the same display timestamp, it determines whether the display time of the screen to be displayed corresponding to the same display timestamp exceeds the sum of the current time and a preset first time threshold. If so, it determines that the slave device can display the screen to be displayed corresponding to the same display timestamp on time; otherwise, it determines that the slave device cannot display the screen to be displayed corresponding to the same display timestamp on time.

[0032] Furthermore, after the driver layer of each slave device adds the image to be displayed to the frame buffer queue and reports the display timestamp of the image to be displayed to the host, it further includes: The host caches the display timestamps reported by each slave device into a buffer. If the number of uncollected display timestamps in the buffer exceeds a preset number, then the display timestamps to be discarded are determined from the uncollected display timestamps, and each slave device is notified to discard the display screen corresponding to the display timestamp to be discarded.

[0033] As described above, the host coordinates and notifies the fast-rendering slaves to discard already rendered frames to adapt to the playback speed of the slow-rendering terminal, so that the lightly loaded terminal can adapt to the speed of the heavily loaded terminal.

[0034] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0035] Example 1 Please refer to Figure 2 The first embodiment of the present invention is: a distributed video synchronous display control method based on frame buffer, which can be applied to the scenario of distributed synchronous playback display of multiple terminals. After setting the horizontal and vertical signals of each terminal to be synchronously refreshed, the video decoding and display of each terminal can be synchronously displayed with absolute 0 frame difference at any time, without any sense of misalignment.

[0036] Generally, displaying a video file on a monitor involves the following steps: (1) Decode the video file to obtain the original video frames; (2) Process the original video frames and render them onto the Surface layer; (3) Multiple Surface layers are mixed and superimposed under the drive of vertical synchronization signal to form the final display screen, and the display screen is sent to FrameBuffer. (4) Take a frame from the FrameBuffer and send it to the display.

[0037] Steps (1) and (2) are implemented at the application layer, while steps (3) and (4) are implemented at the system layer.

[0038] Because the load on each display terminal is different, the application layer synchronous display control will generate asynchronous delay errors during the system CPU resource scheduling process. The specific value is affected by the terminal load, and in low-power embedded devices, it will generally only reach the millisecond level. Therefore, in this embodiment, by processing the frame buffer in the final stage of screen display, it is ensured that the screen sent to the display by each terminal is the same PTS (Presentation Time Stamp), thereby achieving absolute zero frame difference synchronous display of multiple terminal screens.

[0039] Specifically, such as Figure 2As shown, the method in this embodiment includes the following steps: S1: Set a unique identifier for the video to be played. In this embodiment, a video stream ID is set.

[0040] S2: Distribute the video to be played or the segmented video to be played to each slave device.

[0041] In some embodiments, each terminal may play the same video.

[0042] In other embodiments, each terminal can also stitch together and display high-resolution video. In this case, the video file to be played can be pre-segmented, and the video segments can be cached locally on each terminal. For example, a 16K video can be split into 4×4 4K files, played on 16 terminals, with each terminal playing one 4K file, and the 16 displays stitched together to form a 16K display. Subsequently, the host notifies each terminal to synchronously play the specified video file segments via network protocols, including playback control protocols such as pause, fast forward, rewind, and replay. Using the segmented video synchronous playback method can effectively reduce network load and achieve microsecond-level synchronous control.

[0043] Among them, the PTS of the video segments displayed on all terminals must be fixed and the video frames of each segment must be consistent; otherwise, seamless splicing cannot be achieved during display.

[0044] S3: The application layer of each slave device generates a corresponding display screen based on the rendered video frame, sets the display timestamp of the corresponding display screen based on the display timestamp of the video frame and the video stream ID corresponding to the video frame, and sends the display screen to the driver layer through the operating system layer.

[0045] Specifically, the application layer of each slave device renders each video frame in the video to be played frame by frame. If a slave device detects that a video frame has been rendered, it generates the display screen corresponding to that video frame, and sets the display timestamp of the corresponding display screen according to the display timestamp of the video frame and the video stream ID of the video to be played to which the video frame belongs.

[0046] In practical applications, multiple Surfaces can be pre-created for video playback, picture-in-picture, HDMI input, 3D real-time rendering, and other layer overlays. Video frames are rendered into one Surface layer, and other display content that needs to be displayed simultaneously with the video frame is rendered into other Surface layers. Then, multiple Surface layers are overlaid to create a single display screen, which is the display screen corresponding to the video frame.

[0047] In this invention, multiple layers of images are displayed with zero frame difference, and the maximum time delay of each frame is 160µs. Existing technologies generally display video as a single layer, making it impossible to achieve high-precision zero-frame-difference display by overlaying multiple layers of images.

[0048] Then, the application layer sets a priority display tag for the corresponding screen to be displayed, that is, sets the task level of the screen to be displayed to the highest priority, and sends the screen to be displayed to the operating system layer. After receiving the screen to be displayed, the operating system layer recognizes that the screen to be displayed needs frame synchronization based on the tag, and prioritizes sending the screen to be displayed with the display timestamp to the graphics card driver layer.

[0049] In practical applications, a custom video player can be created and the operating system interface can be modified to pass the PTS of each video frame to the system layer, ensuring that the screen to be displayed contains a corresponding display timestamp.

[0050] S4: The driver layer of each slave device adds the screen to be displayed to the frame buffer queue and reports the display timestamp of the screen to be displayed to the host.

[0051] Previously, the frame buffer cached 2-3 frames of the image to be displayed (such as double-buffered or triple-buffered display mechanisms). This embodiment increases the length of the frame buffer queue, for example, by increasing the frame buffer to cache up to 15 frames, so that the system has more redundant time to set the display time of the image to be displayed.

[0052] In this embodiment, the frame buffer is implemented in the graphics card driver. It is a complete frame of the image to be displayed that is rendered and output by the GPU in the video memory. The graphics card can directly refresh the final image displayed on the monitor through the horizontal and vertical sync signals.

[0053] In some embodiments, one terminal is selected as the master terminal and the other terminals as slave terminals. The master terminal itself also performs rendering and display control operations. In other embodiments, the server can be used as the master terminal and the terminal devices (such as set-top boxes) connected to each display individually can be used as slave terminals. In a preferred embodiment, gigabit LAN communication is used between the master and slave terminals to reduce the delay caused by network jitter in confirming frame-by-frame display between the master and slave terminals.

[0054] In practical applications, each terminal (slave) connects to the server (master) immediately after startup. Each terminal maintains a list of frames to be displayed (i.e., a frame buffer queue), the length of which is defined based on the video memory size, such as 15 frames. After accepting the connection, the server maintains a list of terminals and the display status of each terminal's display list. Once a terminal detects that a frame has been rendered, it adds that frame to the list of frames to be displayed and immediately reports the frame's PTS (Presentation Time Schedule) to the server.

[0055] S5: When the host detects that all slave devices have reported the same display timestamp, it determines the display time of the screen to be displayed corresponding to the same display timestamp and the preset delay threshold, and broadcasts the display command of the screen to be displayed corresponding to the same display timestamp to each slave device.

[0056] Specifically, the host receives the display timestamps of the images to be displayed from each slave device. If it detects that all slave devices have sent the same display timestamp, it means that the images to be displayed corresponding to that timestamp have been collected. The host then determines the display time of that image and broadcasts the display command corresponding to that image to each slave device. The display command includes the display time. The display time is the playback time of the image to be displayed.

[0057] Specifically, if the video frame corresponding to the image to be displayed is the first video frame in the video to be played (which can be determined based on the display timestamp of the image to be displayed), then the display time of the image to be displayed is set to the current time (i.e., the time when the display timestamps of the image to be displayed have been collected) plus a preset delay threshold, and the display time of the image to be displayed is used as the starting display time. If the video frame corresponding to the image to be displayed is not the first video frame in the video to be played, then the display time of the image to be displayed is set to the starting display time plus the PTS of the image to be displayed.

[0058] In this embodiment, the frame buffer queue is 15 frames long, meaning 15 buffers are created in the video memory, capable of buffering a maximum of 15 rendered video frames. Normally, after rendering, the image is immediately switched to the monitor for refresh upon display refresh (vertical sync signal, horizontal sync signal, Vsync (vertical sync) signal). However, in this embodiment, considering that each frame needs to report its rendering completion status to the server (host), and the server also needs to notify each terminal of the display time, all images to be displayed are delayed. Assuming a frame rate of 60fps, meaning refresh and display are performed at 60fps, the frame display interval is approximately 16.7ms, so the delay threshold is 16.7 × 15 = 250ms. Because all frames are delayed, the system has sufficient time to handle frame synchronization. In other words, the screen will be black for 250ms after playback begins because no image is refreshed.

[0059] S6: After each slave device receives the display command, it determines whether the video frame to be displayed corresponding to the same display timestamp can be displayed on time. If yes, then proceed to step S7; otherwise, proceed to step S8.

[0060] Specifically, when a slave device receives a display command, it determines whether the display time of the image to be displayed corresponding to the same display timestamp exceeds the current time (i.e., the system time when the display command is received) plus a preset first time threshold. The first time threshold can be determined based on the time it takes for the driver layer to retrieve the image from the frame buffer queue and send it to the display for display. If the time exceeds this threshold, the image to be displayed is considered too late to be displayed, and it is discarded. Simultaneously, the master device notifies other slave devices to discard the image to be displayed as well. If the time does not exceed this threshold, the image to be displayed is determined to be ready for timely display. In other words, the image to be displayed will only be displayed if all slave devices can display it on time.

[0061] S7: The driver layer of each slave device retrieves the display screen corresponding to the same display timestamp from the frame buffer queue according to the display time of the display screen corresponding to the same display timestamp and sends it to the display for display.

[0062] Specifically, after receiving the display command, each slave device sends the screen to be displayed corresponding to the same display timestamp to the display for display according to the display time of the screen to be displayed corresponding to the same display timestamp, and removes the screen to be displayed corresponding to the same display timestamp from the frame buffer queue.

[0063] In this embodiment, the display control program runs at the system layer and can obtain the FrameBuffer state and the video stream ID and PTS data transmitted from the application layer. The terminal reports the frame state to the server frame by frame, and the server is responsible for coordinating whether these frames to be displayed can be directly displayed. The timing of video display needs to adapt the video frame rate to the display refresh. If the video content is 30fps, the display will display the same frame content for two consecutive frames, and remove the frame after two frame refreshes are completed. Since the display refresh time has already completed PTP time synchronization and phase-locked loop adjustment, the screen refresh time is a fixed time, while which frame content is refreshed can be dynamically adjusted.

[0064] In practical applications, a customizable video player can send various playback control states, such as play and pause, to the display processing process to determine whether to clear the FrameBuffer and re-coordinate the display of new video frames. Furthermore, the core of the system's display processing process can be locked, and its process priority increased, further reducing the risk of CPU scheduling impact on the display processing process during synchronous display processing.

[0065] S8: Discard the screen to be displayed corresponding to the same display timestamp, and notify other slave devices to discard the screen to be displayed corresponding to the same display timestamp through the host, that is, remove the screen to be displayed corresponding to the same display timestamp from the frame buffer queue of each slave device.

[0066] Furthermore, due to varying loads across terminals, some terminals may experience slow decoding output or slow reporting of display timestamps due to excessive load or network jitter, failing to refresh the display at the preset frame rate, such as 60fps. This results in the host having too many incomplete display timestamps (i.e., only some of the display timestamps reported by the slaves). The host can then select a portion of these incomplete display timestamps as discardable display timestamps and notify each slave to discard the corresponding display image, that is, remove the display image corresponding to the discardable display timestamp from the slave's frame buffer queue.

[0067] In other words, regardless of the decoding speed of each terminal, the host ultimately coordinates and notifies the terminals with faster rendering (normal terminals) to discard already rendered frames and not display them, in order to adapt to the playback speed of terminals with slower rendering.

[0068] For example, some terminals may fail to refresh the display at a preset frame rate, such as 60fps. Suppose a terminal continuously refreshes at 45fps for a period of time. If most terminals render 60 frames per second, but this terminal only renders 45, then frames 46 through 60 cannot be rendered synchronously. In this case, if there are more than eight missing display timestamps, frame dropping can be considered. Earlier, missing frames can be discarded first. Alternatively, to ensure display continuity, a frame-by-frame dropping strategy can be implemented, coordinating the speed of less heavily loaded terminals to match the speed of more heavily loaded terminals.

[0069] In special circumstances, such as network jitter, when the host receives the frame ready status, the FrameBuffer storage space is insufficient. That is, there is no buffer space to store the screen to be displayed after the host confirms it. In this case, the host notifies the slave to catch up with the display progress by displaying the frame every other frame starting from the specified display timestamp until the FrameBuffer has some spare space.

[0070] Furthermore, if it is detected that the frame buffer queue of a slave device is about to overflow (this can be detected by the master device or by the slave device itself and reporting the detection result to the master), for example, if the frame buffer queue has already buffered 14 frames (the length of the frame buffer queue is 15 frames), then frame dropping is performed. In this embodiment, frame dropping is performed according to the principle of dropping every other frame to ensure the continuity of the picture. For example, the 2nd, 4th, and 6th frames in the current frame buffer queue are removed first, and other slave devices are notified to drop these frames as well.

[0071] In this embodiment, the PTS information is passed to the graphics card driver layer, and the PTS information is also included in the GPU output screen. In addition, in this embodiment, the frame to be displayed after the GPU has completed rendering is cached in the video memory, so high CPU or GPU load has little impact on the display of the current frame. After each terminal completes time synchronization within 1µs using the PTP protocol, the frame to be displayed can be displayed and output according to the time.

[0072] Frame synchronization control uses the TCP communication protocol. To achieve the strict requirement of zero frame difference display, the TCP protocol is more reliable in communication. The protocol ensures that each frame of video data on the terminal can only be displayed after confirmation by the host. All terminals display together, and discarded frames are discarded together.

[0073] Example 2 This embodiment is a computer-readable storage medium corresponding to the above embodiments, on which a computer program is stored. When the program is executed by a processor, it implements the various steps of the distributed video synchronous display control method in the above embodiments and can achieve the same technical effect, which will not be repeated here.

[0074] In summary, the distributed video synchronous display control method and computer-readable storage medium provided by this invention first control the display refresh to be synchronous, and then control the final display buffer to refresh the same content of the image to be displayed at the same time, thereby achieving zero frame difference display. This enables multi-terminal splicing display of images exceeding 16K resolution without any tearing to the human eye. Even when shooting with a camera at a shutter speed of 1 / 1000 second and checking the image frame by frame, there is no frame misalignment or image tearing. Furthermore, when multiple terminals play the same video, each terminal displays the same frame at any given time. By directly modifying the graphics card driver, frame buffering and frame synchronization operations are completed in the graphics card at the very end of the display process, thus achieving microsecond-level synchronous display.

[0075] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A distributed video synchronization display control method, characterized in that, include: Each slave device's driver layer adds the image to be displayed to the frame buffer queue and reports the display timestamp of the image to be displayed to the host. The display timestamp of the image to be displayed is determined based on the video stream ID and the display timestamp of the video frame corresponding to the image to be displayed. When the host detects that all slave devices have reported the same display timestamp, it determines the display time of the screen to be displayed corresponding to the same display timestamp based on the same display timestamp, and broadcasts the display time to all slave devices. Each slave device's driver layer retrieves the image to be displayed corresponding to the same display timestamp from the frame buffer queue based on the display time and sends it to the display for display.

2. The distributed video synchronization display control method according to claim 1, characterized in that, Before the driver layer of each slave device adds the image to be displayed to the frame buffer queue and reports the display timestamp of the image to be displayed to the host, it further includes: Each slave device's application layer renders each video frame in the video to be played, frame by frame. Generate the corresponding display screen based on the rendered video frames, and set the display timestamp of the corresponding display screen based on the display timestamp of the video frames and the video stream ID corresponding to the video frames; The screen to be displayed is sent to the operating system layer; After receiving the screen to be displayed, the operating system layer sends the screen to the driver layer.

3. The distributed video synchronization display control method according to claim 2, characterized in that, The application layer sets the task level of the screen to be displayed to the highest priority. After receiving the screen to be displayed, the operating system layer prioritizes sending the screen to be displayed to the driver layer.

4. The distributed video synchronization display control method according to claim 2, characterized in that, Before the application layer of each slave device renders each video frame of the video to be played frame by frame, it further includes: Set a unique identifier for the video to be played, where the unique identifier is the video stream ID; The video to be played, or the segmented video to be played, is distributed to each slave device.

5. The distributed video synchronization display control method according to claim 1, characterized in that, Also includes: If it is detected that the number of images to be displayed in the frame buffer queue of a slave device exceeds a preset ratio of the length of the frame buffer queue, then the images to be discarded are identified in the frame buffer queue of the slave device and removed from the frame buffer queue of the slave device. The host notifies other slave devices to discard the image based on the display timestamp of the image to be discarded.

6. The distributed video synchronization display control method according to any one of claims 1-5, characterized in that, The length of the frame buffer queue is 10-15 frames.

7. The distributed video synchronization display control method according to claim 1, characterized in that, When the host detects that all slave devices have reported the same display timestamp, it determines the display time of the screen to be displayed corresponding to the same display timestamp based on the same display timestamp, and broadcasts the display time to all slave devices, including: If the host detects that all slave devices have reported the same display timestamp, it determines whether the screen to be displayed corresponding to the same display timestamp is the first screen to be displayed. If so, the host determines the display time of the screen to be displayed corresponding to the same display timestamp based on the current time and the preset delay threshold, and takes the display time of the first screen to be displayed as the starting display time; If not, then the display time of the screen to be displayed corresponding to the same display timestamp is determined based on the start display time and the display timestamp of the screen to be displayed corresponding to the same display timestamp; The host broadcasts the display command for the screen to be displayed corresponding to the same display timestamp to each slave device, and the display command includes the display time.

8. The distributed video synchronization display control method according to claim 1, characterized in that, Before the driver layer of each slave device retrieves the image to be displayed corresponding to the same display timestamp from the frame buffer queue and sends it to the display for display, it further includes: After each slave device's driver layer receives the display time of the screen to be displayed corresponding to the same display timestamp, it determines whether the screen to be displayed corresponding to the same display timestamp can be displayed on time. If not, the screen to be displayed corresponding to the same display timestamp is discarded, and the host notifies other slaves to discard the screen to be displayed corresponding to the same display timestamp. If each slave device can display the screen to be displayed corresponding to the same display timestamp on time, then the driver layer of each slave device will execute the step of retrieving the screen to be displayed corresponding to the same display timestamp from the frame buffer queue according to the display time and sending it to the display for display.

9. The distributed video synchronization display control method according to claim 8, characterized in that, After receiving the display time of the screen to be displayed corresponding to the same display timestamp, the driver layer of each slave device determines whether the screen to be displayed corresponding to the same display timestamp can be displayed on time, including: When a slave device's driver layer receives the display time of the screen to be displayed corresponding to the same display timestamp, it determines whether the display time of the screen to be displayed corresponding to the same display timestamp exceeds the sum of the current time and a preset first time threshold. If so, it determines that the slave device can display the screen to be displayed corresponding to the same display timestamp on time; otherwise, it determines that the slave device cannot display the screen to be displayed corresponding to the same display timestamp on time.

10. The distributed video synchronization display control method according to claim 1, characterized in that, After the driver layer of each slave device adds the image to be displayed to the frame buffer queue and reports the display timestamp of the image to be displayed to the host, it further includes: The host caches the display timestamps reported by each slave device into a buffer. If the number of uncollected display timestamps in the buffer exceeds a preset number, then the display timestamps to be discarded are determined from the uncollected display timestamps, and each slave device is notified to discard the display screen corresponding to the display timestamp to be discarded.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images