Multi-screen playing method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a multi-screen playing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring video source data of a target video, and demultiplexing the video source data to obtain video coding data; determining a video timestamp of each video frame in the target video according to the playing speed in the video coding data; packaging the video coding data to obtain a real-time transmission data packet, the real-time transmission data packet at least comprising the video coding data and each video timestamp; and sending the real-time transmission data packet to a receiving end device, so that the receiving end device decodes the video coding data to obtain video decoding data, and performs picture rendering on the video decoding data based on the video timestamps to synchronously play video pictures of the target video with the sending end device based on the video timestamps. By adopting the method, the playing time delay of picture synchronous playing on different screen interfaces can be reduced.

Description

Technical Field

[0001] This application relates to the field of media playback technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for multi-screen playback. Background Technology

[0002] With the development of technology, multi-screen synchronization has become widely used in daily life. Multi-screen synchronization refers to the transmission, parsing, and display of the same media content on different media playback devices, so that the media content to be displayed is synchronized.

[0003] Currently, it's possible to record the screen or layer displaying media files, encode the recorded video data, and transmit it to other devices for synchronized multi-screen playback. However, the process of recording, encoding, and pushing video data not only requires a large amount of data transmission, but also suffers from high latency due to the inability to synchronize recording and encoding / decoding between different devices. This leads to a lack of synchronization between different screen interfaces. Therefore, reducing playback latency for synchronized playback across different screen interfaces in multi-screen scenarios is a pressing issue that needs to be addressed. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, storage medium, and computer program product for multi-screen playback that reduces playback latency when synchronizing video playback on different screen interfaces in a multi-screen playback scenario, in order to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for multi-screen playback. The method is applied to a sending device, where the sending device and the receiving device have a communication connection within the same local area network. The method includes:

[0006] Obtain the video source data of the target video and demultiplex the video source data to obtain the video encoded data;

[0007] The playback speed in the video encoding data is used to determine the video timestamp of each video frame in the target video.

[0008] The video encoded data is encapsulated to obtain real-time transmission data packets, which include at least the video encoded data and the timestamps of each video.

[0009] Real-time data packets are sent to the receiving device so that the receiving device can decode the video encoded data to obtain video decoded data, and render the video decoded data based on each video timestamp so that the video frame of the target video can be played synchronously with the sending device based on each video timestamp.

[0010] Secondly, this application also provides a multi-screen playback device. The device includes:

[0011] The data processing module is used to acquire the video source data of the target video and demultiplex the video source data to obtain the video encoded data;

[0012] The time determination module is used to determine the video timestamp of each video frame in the target video by using the playback speed in the video encoding data;

[0013] The encapsulation processing module is used to encapsulate the video encoded data to obtain real-time transmission data packets. The real-time transmission data packets include at least the video encoded data and the timestamps of each video.

[0014] The multi-screen synchronous playback module is used to send real-time transmission data packets to receiving devices within the same local area network, so that the receiving devices can decode the video encoding data to obtain video decoding data, and render the video decoding data based on each video timestamp, so as to synchronously play the video frame of the target video based on each video timestamp.

[0015] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0016] Obtain the video source data of the target video and demultiplex the video source data to obtain the video encoded data;

[0017] The playback speed in the video encoding data is used to determine the video timestamp of each video frame in the target video.

[0018] The video encoded data is encapsulated to obtain real-time transmission data packets, which include at least the video encoded data and the timestamps of each video.

[0019] Real-time data packets are sent to the receiving device so that the receiving device can decode the video encoded data to obtain video decoded data, and render the video decoded data based on each video timestamp so that the video frame of the target video can be played synchronously with the sending device based on each video timestamp.

[0020] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0021] Obtain the video source data of the target video and demultiplex the video source data to obtain the video encoded data;

[0022] The playback speed in the video encoding data is used to determine the video timestamp of each video frame in the target video.

[0023] The video encoded data is encapsulated to obtain real-time transmission data packets, which include at least the video encoded data and the timestamps of each video.

[0024] Real-time data packets are sent to the receiving device so that the receiving device can decode the video encoded data to obtain video decoded data, and render the video decoded data based on each video timestamp so that the video frame of the target video can be played synchronously with the sending device based on each video timestamp.

[0025] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0026] Obtain the video source data of the target video and demultiplex the video source data to obtain the video encoded data;

[0027] The playback speed in the video encoding data is used to determine the video timestamp of each video frame in the target video.

[0028] The video encoded data is encapsulated to obtain real-time transmission data packets, which include at least the video encoded data and the timestamps of each video.

[0029] Real-time data packets are sent to the receiving device so that the receiving device can decode the video encoded data to obtain video decoded data, and render the video decoded data based on each video timestamp so that the video frame of the target video can be played synchronously with the sending device based on each video timestamp.

[0030] The aforementioned multi-screen playback method, apparatus, computer equipment, storage medium, and computer program product acquire the video source data of the target video and demultiplex the video source data to obtain video encoded data, thereby avoiding processing delays caused by reading and recording the decoded video source data. Then, by using the playback speed in the video encoded data, the video timestamp of each video frame in the target video is determined, and the video encoded data is encapsulated to obtain a real-time transmission data packet. The real-time transmission data packet includes at least the video encoded data and each video timestamp. Thus, encapsulation processing avoids encoding delays caused by secondary encoding before decoding the video source data. This process sends real-time data packets to the receiving device, enabling the receiving device to decode the video encoded data and render the video based on each video timestamp. This allows the receiving device to synchronously play the target video frame with the sending device based on each video timestamp. By demultiplexing the obtained video encoded data to determine the video timestamp, the video decoded data can ensure that the corresponding video frames are played at the same timestamp when the target video frame is played on different devices' displays, thereby reducing the playback latency when playing the video synchronously on different screen interfaces. Attached Figure Description

[0031] Figure 1 This is an application environment diagram of a multi-screen playback method in one embodiment;

[0032] Figure 2 This is a schematic diagram illustrating multi-screen playback in a vehicle-mounted simultaneous playback scenario, as shown in one embodiment.

[0033] Figure 3 This is a flowchart illustrating a multi-screen playback method in one embodiment;

[0034] Figure 4 This is a timing flowchart illustrating the establishment of a communication connection for multi-screen playback between a sending device and a receiving device in one embodiment.

[0035] Figure 5 This is a schematic diagram of the timing process for multi-screen playback with synchronized video and audio in one embodiment;

[0036] Figure 6 This is a schematic diagram of the playback status acquisition process in one embodiment;

[0037] Figure 7 This is a flowchart illustrating a multi-screen playback method in another embodiment;

[0038] Figure 8 This is a schematic diagram of the timing process for encapsulation adjustment when the control parameter is a pause playback parameter in one embodiment;

[0039] Figure 9 This is a timing flowchart illustrating the encapsulation adjustment process when the control parameter is a playback parameter in one embodiment.

[0040] Figure 10 This is a schematic diagram of the timing process for encapsulation adjustment when the control parameter is an adjustment speed or a positioning playback parameter in one embodiment.

[0041] Figure 11 This is a complete flowchart of a multi-screen playback method in one embodiment;

[0042] Figure 12 This is a structural block diagram of a multi-screen playback device in one embodiment;

[0043] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] This application provides a method for multi-screen playback that can reduce playback latency when synchronizing video playback on different screen interfaces. The multi-screen playback method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0046] The multi-screen playback method provided in this application can be applied to various screen-sharing scenarios, including but not limited to in-vehicle screen-sharing scenarios, indoor screen-sharing scenarios, and office screen-sharing scenarios that require screen-sharing. The terminal 102 includes a sending end device and a receiving end device that are connected in communication within the same local area network. The sending end device and the receiving end device can be the same terminal device or different devices; this is not limited here. Specifically, the multi-screen playback method provided in this application is applied to the sending end device. That is, the sending end device obtains the video source data of the target video, demultiplexes the video source data to obtain video encoded data, and then determines the video timestamp of each video frame in the target video based on the playback speed in the video encoded data. The sending end device then encapsulates the video encoded data to obtain a real-time transmission data packet, which includes at least the video encoded data and each video timestamp. Based on this, the sending end device sends the real-time transmission data packet to the receiving end device. Therefore, the receiving device can decode the received video encoded data to obtain video decoded data. The receiving device then renders the video decoded data based on each video timestamp, so that the target video can be played synchronously with the sending device based on the same video timestamp. Since the video timestamp is determined by demultiplexing the video encoded data, when the target video is played on different devices' displays, the corresponding video frames can be displayed at the same timestamp, thus reducing playback latency when playing the target video synchronously on different screens.

[0047] Since the multi-screen playback method provided in this application embodiment specifically requires synchronized playback of images on the display interfaces of multiple different terminal devices, the technical terms involved in simultaneous screen playback are explained below:

[0048] 1. Sender. In multi-screen synchronous playback, the sender acts as the server for the Real-Time Streaming Protocol (RTSP). It is responsible for downloading video data, playing audio and video, and receiving and processing requests from other devices, as well as encapsulating and sending Real-Time Transport Protocol (RTP) data. While RTP is used to transmit audio and video data, RTSP is used to control the transmission of streaming media.

[0049] 2. Receiver. The receiver, in multi-screen synchronous playback, acts as an RTSP client, sending requests to the server (i.e., the sender), receiving data returned by the server, and playing the content based on that data.

[0050] 3. LAN Device Discovery. LAN device discovery is a technology that enables terminal devices within the same Local Area Network (LAN) to identify, discover, and communicate with each other. This allows users to locate other terminal devices within the LAN, such as printers and network storage devices, thereby enabling functions such as file sharing and printing.

[0051] 4. Multicast Domain Name System (mDNS). mDNS is a network protocol for device discovery and name resolution within a local area network (LAN). mDNS allows end devices to discover the Internet Protocol Addresses (IP Addresses) and services of other devices through broadcast queries, even without a central Domain Name System (DNS) server. Therefore, mDNS is commonly used in home and small business networks to facilitate mutual discovery and communication between end devices.

[0052] 5. Miracast. Miracast is a wireless display technology that allows a user to wirelessly project the content displayed on one device onto another. This enables the user to watch videos, play games, or give presentations on another device without the need for any physical cables.

[0053] 6. RTSP. RTSP is a network protocol used to control multimedia sessions, such as the transmission of audio or video streams. It allows clients to pause, play, fast forward, and perform other operations, enabling real-time playback and on-demand functions.

[0054] 7. RTP. RTP is a transport protocol designed for real-time applications, used to transmit audio and video data over the Internet. It provides timestamps and sequence numbers so that the receiver can reassemble the data and synchronize audio and video.

[0055] 8. Demultiplexing. Demultiplexing refers to the process of separating a composite media file (such as MP4, MKV, etc.) containing multiple audio, video, and other data streams into individual audio, video, and data streams. The purpose of demultiplexing is to enable the separate processing of audio and video data during playback, editing, or transcoding, so as to perform operations such as decoding, synchronization, and editing.

[0056] 9. Decoding. Decoding refers to the process of restoring audio and video data (usually in encoded form) to its original form. The decoding process includes steps such as decompression, synchronization, etc., in order to recover the original audio and video data for use during playback, editing, or transcoding.

[0057] 10. Reference Frame. A reference frame is a concept in video compression coding, typically an I-frame or P-frame. They serve as a reference for predictive coding, effectively reducing data redundancy. Reference frames help the decoder recover the image content of other frames (such as P-frames and B-frames).

[0058] 11. Group of Pictures (GOP). A GOP is a group of consecutive video frames, starting with an I-frame followed by several P-frames and B-frames. The I-frame is a complete image frame, while the P-frames and B-frames are predictive coded frames. The GOP structure enables efficient video compression, reducing storage and transmission costs.

[0059] To make it easier to understand, let's take the scenario of screen mirroring in a car as an example, such as... Figure 2 As shown, in Figure 2 In Figure (A), when the target video is played on the display interface 202 of the sending device, the terminal user can initiate a screen-sharing playback trigger operation on the screen-sharing control 2021 of the display interface 202. Through this screen-sharing playback trigger operation, the multi-screen playback method provided in this embodiment is executed, enabling... Figure 2 In Figure (B), when the target video is played on the display interface 202 of the sending device, the video frame of the target video can also be played synchronously on the display interface 204 of the receiving device.

[0060] The following examples illustrate this in detail: In one example, as... Figure 3 As shown, a method for multi-screen playback is provided. Figure 1 Terminal 102 includes a sending end device and a receiving end device that have a communication connection within the same local area network, therefore, this method can be applied to... Figure 1 Taking the sending device of terminal 102 as an example, this method can also be applied to a system including terminal 102 and server 104, that is, requiring the interaction of the sending device, the receiving device, and server 104. In this embodiment, the method includes the following steps:

[0061] Step 302: Obtain the video source data of the target video and demultiplex the video source data to obtain the video encoded data.

[0062] The target video is the video that needs to be played on multiple screens. Therefore, the target video can be a complete video or a segment of a complete video. For example, the target video could be the first episode of a series, 90% of the first episode, or even the first minute of the first episode; there is no limitation on the target video. Since video playback requires displaying both visual and audio content, the video source data must include at least video encoded data and audio encoded data. Secondly, as described above, demultiplexing refers to the process of separating a composite media file containing multiple audio, video, and other data streams into independent audio, video, and data streams. Since video source data includes at least video encoded data and audio encoded data, demultiplexing in this embodiment involves separating the video source data into independent video encoded data and audio encoded data.

[0063] Specifically, the sending device first determines the target video, meaning it first plays the complete video. Then, the sending device initiates a screen mirroring trigger operation for the target object. This screen mirroring trigger operation determines the segment of the complete video that needs to be mirrored as the target video. For example, if the complete video is one hour long, and the target object initiates a screen mirroring trigger operation at the 10-minute mark, then the target object needs to mirror the 50 minutes of video after the 10-minute mark. In this case, the latter 50 minutes of the complete video are the target video. Alternatively, considering the amount of data to be processed, the first minute of the latter 50 minutes of the complete video can be used as the target video, meaning the segment between the 10th and 11th minutes of the complete video is used as the target video. It's understandable that in practical applications, the target object can also initiate a screen mirroring trigger operation at the start of playback, in which case the target video is the complete video being played by the sending device.

[0064] Based on this, the sending device obtains the video source data of the target video. The sending device can obtain the video source data of the target video through communication with a server, which can be a Content Delivery Network (CDN). Alternatively, the sending device can pre-cache the data locally and then retrieve the video source data of the target video from the local cache when data retrieval is needed. The method of obtaining the video source data is not limited here.

[0065] Furthermore, the sending device demultiplexes the video source data to obtain video encoded data. As explained above, since video playback requires displaying both visual and audio content, the video source data includes at least video encoded data and audio encoded data. To allow for separate processing of video and audio encoded data during playback, editing, and transcoding operations, the sending device separates the video source data into video encoded data and audio encoded data, resulting in independent video encoded data. The video encoded data includes multiple encoded video frames and the playback speed between each frame. The playback speed between each encoded video frame is consistent with the playback speed of the target video; without playback speed adjustment, the playback speed is always 1.

[0066] It is understandable that since the sending and receiving devices have a communication connection within the same local area network (LAN), i.e., multi-screen playback is being performed, it is necessary to establish a communication connection between the sending and receiving devices within the same LAN. Essentially, what needs to be established is a communication connection for multi-screen playback between the sending and receiving devices within the same LAN, in order to achieve synchronous playback service discovery between the sending and receiving devices. Therefore, the following describes how to establish a communication connection for multi-screen playback: In an optional embodiment, the multi-screen playback method further includes: starting a real-time streaming protocol service within the LAN and registering a synchronous playback port, the synchronous playback port being used to provide synchronous playback services to playback devices within the same LAN; sending a communication transmission message to the multicast address port based on the multicast domain name system, the communication transmission message including the playback port information of the synchronous playback port; when the receiving device detects the playback port information in the multicast address port through the multicast domain name system, the receiving device sends a real-time streaming protocol message to the synchronous playback port indicated by the playback port information, thereby establishing a communication connection between the sending and receiving devices.

[0067] The synchronous playback port is used to provide synchronous playback services to playback devices within the same local area network. In other words, it's necessary to provide synchronous playback services between the sending and receiving devices based on the synchronous playback port. To ensure that the existing communication connection is for the synchronous playback service—that is, the receiving device specifically needs to connect to the synchronous playback port registered by the sending device—the communication transmission message includes the playback port information of the synchronous playback port. Furthermore, in this embodiment, the communication connection established between the sending and receiving devices is specifically a communication connection for multi-screen playback.

[0068] Specifically, the sending device starts a Real-time Streaming Protocol (RTSP) service within its local area network (LAN), and then registers a synchronization port based on mDNS. The sending device then provides synchronization playback services to other playback devices within the same LAN through this port. After registering the synchronization port, the sending device becomes an RTSP server, meaning it can download data, play audio and video, and receive and process requests from the receiving end. Furthermore, the sending device needs to monitor the synchronization port in real time to ensure it can detect requests that trigger simultaneous playback operations.

[0069] Furthermore, the sending device sends communication transmission messages including playback port information to the multicast address port based on the multicast domain name system. In other words, the multicast address port records the playback port information of the synchronous playback port. Based on this, the receiving device monitors the multicast address port in real time through the multicast domain name system. When the receiving device detects the playback port information in the multicast address port through the multicast domain name system, it can determine that the sending device can provide synchronous playback services to other playback devices within the same local area network through the synchronous playback port. The receiving device then sends a Real-Time Streaming Protocol (RTSP) message to the synchronous playback port indicated by the playback port information. The RTSP message carries the receiving device's communication port information (such as IP address and port). The synchronous playback port then processes the RTSP message, establishing a communication connection between the sending and receiving devices through the receiving device indicated by the communication port information in the RTSP message, thus ensuring a communication connection between the sending and receiving devices within the same local area network.

[0070] To facilitate understanding of how a communication connection is established between the sending and receiving devices for multi-screen playback, such as Figure 4 As shown, the sending device first starts the Real-Time Streaming Protocol (RTP) service within the local area network (LAN) and then registers the synchronous playback port. The sending device then sends communication transmission messages to the multicast address port based on the Multicast Domain Name System (DNS). At this time, the receiving device detects the multicast address port in real time through the DNS. Upon detecting the playback port information in the multicast address port through the DNS, the receiving device sends RTP messages to the synchronous playback port indicated by the playback port information. At this point, a communication connection for multi-screen playback is established between the sending and receiving devices.

[0071] It is understood that in practical applications, the aforementioned data interaction process can be performed using the User Datagram Protocol (UDP) or based on the Transmission Control Protocol (TCP). Secondly, this embodiment provides terminal device discovery and connection based on mDNS; however, mDNS is only an open-source implementation, and custom protocols can be customized to achieve the purpose of terminal device discovery and connection according to actual needs. Therefore, it should not be construed as a limitation on the embodiments of this application.

[0072] Step 304: Determine the video timestamp of each video frame in the target video by using the playback speed in the video encoding data.

[0073] As described above, the video encoding data includes multiple encoded video frames and the playback speed between each encoded video frame. The playback speed between each encoded video frame is consistent with the playback speed of the target video; without playback speed adjustment, the playback speed is 1. The video encoding data also includes the number of frames played, which is the same as the number of frames played in the target video. For example, if the target video is 24 frames per second (fps), then 24 video frames can be played every second, resulting in a playback frame rate of 24 fps. Secondly, the video timestamp is the timestamp of each video frame played. For example, given video frames A1, A2, and A3, the timestamp of video frame A1 is x year x month x day x hour x minute x second x millisecond, the timestamp of video frame A2 is x year x month x day x hour x minute x second y millisecond, and the timestamp of video frame A3 is x year x month x day x hour x minute x second z millisecond.

[0074] Specifically, the sending device first determines the playback speed and the number of frames in the video encoding data. It then determines the number of video frames played per second based on the playback speed and frame count. Next, it determines the timestamp at which each video frame in the target video will be played, based on the target video's duration, thus establishing the individual video timestamp for each frame. In other words, based on the current timestamp, playback speed, and frame count, the sending device determines which timestamp the receiving device will use to play each video frame after receiving the data. This involves considering the initial timestamp of each video frame when playing the target video from the current timestamp, as well as the data transmission latency between the sending and receiving devices, the data encapsulation processing time of the sending device, and the decoding time of the receiving device. Adding this latency to the initial timestamp of each video frame yields its individual video timestamp.

[0075] For example, assuming a target video duration of 2 seconds, a playback speed of 1, and a frame rate of 10fps in the video encoding data, it can be seen that 10 video frames can be played per second, and playing a 2-second target video requires playing 20 video frames. If the current timestamp is 12:00:00:00 milliseconds, and the target video is played directly using the current timestamp, then the initial timestamp of video frame A1 in the target video is 12:00:00:00 milliseconds, the initial timestamp of video frame A2 is 12:00:00:06 milliseconds, the initial timestamp of video frame A3 is 12:00:00:12 milliseconds, the initial timestamp of video frame A4 is 12:00:00:18 milliseconds, and the initial timestamp of video frame A5 is... The initial timestamp is 12:00:00.24 milliseconds. The initial timestamp of video frame A6 is 12:00:00.30 milliseconds, the initial timestamp of video frame A7 is 12:00:00.36 milliseconds, the initial timestamp of video frame A8 is 12:00:00.42 milliseconds, the initial timestamp of video frame A9 is 12:00:00.48 milliseconds, and the initial timestamp of video frame A10 is 12:00:00.54 milliseconds. Subsequent timestamps are similar and will not be repeated.

[0076] Considering the data transmission delay between the sending and receiving devices, the data encapsulation processing time of the sending device, and the decoding time of the receiving device, if the delay time is determined to be 0.5 seconds (i.e., 30 milliseconds), then the video timestamp of video frame A1 is 12:00:00:30 milliseconds, the video timestamp of video frame A2 is 12:00:00:36 milliseconds, the video timestamp of video frame A3 is 12:00:00:42 milliseconds, and so on. The initial timestamp of video frame A10 is 12:00:01:24 milliseconds, and subsequent timestamps are similar and will not be elaborated further.

[0077] Step 306: Encapsulate the video encoded data to obtain a real-time transmission data packet. The real-time transmission data packet includes at least the video encoded data and the timestamps of each video.

[0078] The real-time transmission data packet includes at least video encoded data and video timestamps. Specifically, the real-time transmission data includes video encoded data used for decoding to obtain video frames, and video timestamps indicating the playback of each video frame. The aforementioned video encoded data is specifically raw H.264 data, and the real-time transmission data packet is an RTP packet. Specifically, the sending device encapsulates the video encoded data and the determined video timestamps into a real-time transmission data packet.

[0079] In this embodiment, the sending device does not encapsulate all video data into real-time transmission data packets for transmission at once. Instead, it divides the video played by the sending device into multiple video segments according to data transmission requirements and data interaction latency. These video segments are then sequentially identified as the target video as defined in the previous embodiment for subsequent steps. For example, if the video played by the sending device is 1 minute long, it can be divided into 30 equal video segments, each 2 seconds long. Before decoding the video encoding data corresponding to the first 2-second video segment (target video), the sending device performs the steps described in this embodiment and encapsulates the video encoding data corresponding to the video segment (target video) to obtain the corresponding real-time transmission data packet. After this processing, similar processing is performed on the second and third 2-second video segments.

[0080] It is understandable that steps 304 and 306 must be performed before the sending device decodes the video encoded data. Furthermore, the division of the aforementioned video segments and the duration of each segment need to be determined based on actual scenario requirements and data interaction latency; therefore, no restrictions are imposed here.

[0081] Step 308: The real-time transmission data packet is sent to the receiving device so that the receiving device can decode the video encoded data to obtain video decoded data, and render the video decoded data based on each video timestamp so that the video frame of the target video can be played synchronously with the sending device based on each video timestamp.

[0082] Specifically, the sending device sends real-time data packets to the receiving device. The receiving device extracts the video encoded data and each video timestamp pair from the real-time data packets, then decodes the video encoded data to obtain video decoded data, and renders the video based on each video timestamp. In other words, the receiving device plays the video frames in the video decoded data according to the video timestamps. Similarly, after the sending device sends the real-time data packets to the receiving device, it also decodes the video encoded data to obtain video decoded data, and renders the video based on each video timestamp. This means that the sending device then plays the video frames in the video decoded data according to the video timestamps. Since the sending and receiving devices play the video frames using the same video timestamps, this ensures that the target video is played synchronously by both devices.

[0083] Since the target video includes not only video playback but also video audio playback, after satisfying the synchronous playback of video playback on different playback devices through the aforementioned embodiments, the synchronization of video playback and video audio on the sending device should also be considered. This is described below: In an optional embodiment, the multi-screen playback method further includes: determining an independent clock for the audio timestamp of each audio frame in the target video and the clock of each video timestamp; and performing audio-visual synchronization processing on the video decoding data and audio decoding data based on the independent clock and each video timestamp.

[0084] There are three methods for setting the master clock when performing audio-visual synchronization: one is to use the time corresponding to the audio frame as the master clock, another is to use the time corresponding to the video frame as the master clock, and the third is to use an external clock independent of the audio and video as the master clock. In this embodiment, the master clock set is the aforementioned independent clock. Specifically, the transmitting device determines the clock, independent of the audio timestamp of each audio frame and the video timestamps in the target video, as the independent clock. In other words, it uses an independent external clock as the master clock for playback; for example, if one second has passed, then the independent clock is [the time specified in the original text].

[0085] Based on this, the transmitting device performs audio-visual synchronization processing on the video and audio decoded data using independent clocks and video timestamps. That is, the transmitting device plays the video decoded data using each video timestamp to ensure synchronization with the receiving device. Furthermore, the independent clocks determine the independent video timestamps of the decoded video data and the independent audio timestamps of the decoded audio data. Within the same target video, the independent video timestamps and audio timestamps must correspond. Therefore, by matching the independent video timestamps of video frames to each other, and by establishing the correspondence between video and audio timestamps, the matching and adjustment between the video and audio timestamps of each video frame can be completed, thus achieving audio-visual synchronization. For example, video frame A1 has a video timestamp of 12:00:00:30 milliseconds, and video frame A1 corresponds to audio frame B1. Video frame A1 has a video timestamp of C1, and audio frame B1 has an audio timestamp of D1. Since the video timestamp C1 corresponds to the video timestamp "12:00:00:30 milliseconds", when video frame A1 is played at the video timestamp "12:00:00:30 milliseconds", audio frame B1 will inevitably be played. At this time, audio-visual synchronization can be achieved by determining the specific time when the audio timestamp D1 should be played using the video timestamp "12:00:00:30 milliseconds".

[0086] The following describes the process of multi-screen playback that considers video and audio synchronization, such as... Figure 5As shown, the sending end downloads the video source data of the target video through a downloader. The sending end then demultiplexes the video source data using a demultiplexer to obtain video encoded data and audio encoded data, at which point the decoding and rendering process begins. Before decoding the data at the sending end, the playback speed is used to determine the video timestamp of each video frame. Then, the video encoded data is encapsulated into real-time transmission data packets. The decoder at the sending end decodes the video encoded data and audio encoded data to obtain video decoded data and audio decoded data. The video decoded data fills the buffer, and based on independent clocks and each video timestamp, audio-visual synchronization processing is performed on the video decoded data and audio decoded data. Furthermore, independent rendering of the video decoded data is performed based on each video timestamp to ensure video synchronization.

[0087] Meanwhile, the sending end transmits real-time data packets through the local real-time transport protocol service, that is, it transmits the real-time data packets to the local real-time transport protocol service, and then sends the real-time data packets to the receiving end device through the local real-time transport protocol service. The receiving end device decodes the video encoded data to obtain video decoded data, and renders the video decoded data based on each video timestamp, so as to synchronize the video frame of the target video with the sending end device based on each video timestamp. Through the above process, reference frames are not lost, and decoding failure at the receiving end is not caused. Moreover, there is no secondary encoding process. While ensuring video frame synchronization and audio-visual synchronization, low-latency synchronous playback is guaranteed.

[0088] Since the target object may need to obtain the playback status in practical applications, the following describes how, in an optional embodiment, the multi-screen playback method further includes: in response to a status acquisition instruction, acquiring the status field indicated by the status acquisition instruction; reading the playback status indicated by the status field; and feeding back the playback status to the receiving device.

[0089] The specific status acquisition instruction is the GET_PARAMETER instruction, which carries a status field indicating the desired playback status. This status field could be the playback percentage, playback rate, or the target video's title. Specifically, the sending device responds to the status acquisition instruction by retrieving the status field indicated by the instruction. This instruction can be initiated by the target device from the sending device or by the target device from the receiving device to the sending device. Based on this, the sending device reads the playback status indicated by the status field and sends the playback status back to the receiving device. For example, if the status field is the target video's title, the sending device would read the target video's title as "XYZ1" and send the title "XYZ1" back to the receiving device. The following describes the playback status acquisition process, using the receiving device sending a status acquisition instruction as an example. Figure 6 As shown, the receiving device sends a status acquisition command to the sending device. The sending device responds to the status acquisition command by acquiring the status field indicated by the status acquisition command and reading the playback status indicated by the status field. Then, the sending device feeds back the playback status to the receiving device.

[0090] It should be understood that the corresponding examples in the embodiments of this application are used to understand this solution, but should not be construed as specific limitations on this solution.

[0091] In the aforementioned multi-screen playback method, the video source data is demultiplexed to obtain video encoded data, thus avoiding processing latency caused by reading and recording the decoded video source data. Furthermore, encapsulation processing is performed before decoding the video source data to avoid encoding latency caused by secondary encoding. The video timestamp is determined by the video encoded data obtained through demultiplexing. When the target video is played on different devices' displays, the corresponding video frames can be played and displayed at the same timestamp, thereby reducing playback latency for synchronized playback on different screens.

[0092] In one embodiment, such as Figure 7 As shown, other methods for multi-screen playback include:

[0093] Step 702: In response to the control operation command, obtain the control parameters in the control operation command.

[0094] The control operation commands are used to control video playback, specifically to control functions such as pausing, playing, adjusting playback speed, fast forwarding, and rewinding. Therefore, in one optional embodiment, when the control operation command is a pause operation, the control parameter is the pause playback parameter. When the control operation command is a play operation, the control parameter is the play parameter. When the control operation command is an adjust playback speed operation, the control parameter is the speed adjustment parameter. When the control operation command is a positional playback operation, the control parameter is the positional playback parameter; the aforementioned positional playback operation is equivalent to video fast forwarding and rewinding.

[0095] Specifically, the sending device responds to the control operation command and extracts the control parameters from it. That is, when the target object performs a control operation on video playback, it generates a corresponding control operation command. This command can be triggered directly by the target object performing a control operation while the sending device is playing video; in other words, the sending device can directly generate the corresponding control operation command and respond to it. Alternatively, the control operation command can be triggered directly by the target object performing a control operation on the receiving device while the receiving device is playing video; in other words, the receiving device generates the corresponding control operation command and sends it to the sending device, causing the sending device to respond to the control operation command.

[0096] Step 704: Based on the control parameters, the video encoded data is encapsulated and adjusted to adjust the real-time transmission data packets.

[0097] Specifically, the transmitting device performs encapsulation and adjustment processing on the video encoded data based on control parameters, adjusting the real-time transmitted data packets. This encapsulation and adjustment processing can include pausing encapsulation, encapsulating new video encoded data, or encapsulating based on a new video timestamp. Furthermore, as seen in the aforementioned embodiments, the control parameters can include pause playback parameters, playback parameters, speed adjustment, and positioning playback parameters. It is understood that in practical applications, the control parameters can be set, changed, and expanded based on actual conditions; that is, it can support not only playback, pause, speed adjustment, fast forward, and rewind operations, but also other operations, which are not limited here.

[0098] The following section describes how to encapsulate and adjust control parameters for pause playback, playback, speed adjustment, and positional playback:

[0099] In one optional embodiment, the control parameter is a pause playback parameter. Based on this, the video encoded data is encapsulated and adjusted according to the control parameter to adjust the real-time transmission data packet, including: determining paused video encoded data for paused playback based on the pause playback parameter, pausing the encapsulation processing of the paused video encoded data, and pausing the generation of real-time transmission data packets. Based on this, the multi-screen playback method further includes: pausing the transmission of real-time transmission data packets to the receiving device.

[0100] The encapsulation of real-time transmission data packets and the pushing of these data packets are closely related to the video playback process of the sending device. Therefore, since the sending device is the main device for video playback, it needs to communicate with the player on the receiving device when handling video playback and video pause.

[0101] Specifically, since the control parameter is a pause playback parameter, meaning the video being played by the sending device needs to be paused, the sending device needs to determine the pause video encoding data based on the pause playback parameter. For example, if the video being played by the sending device is 10 minutes long, and the target object triggers a control operation command at the first minute of synchronous playback with the receiving device, indicating a pause playback operation, then the video data corresponding to the unplayed 9 minutes of the 10-minute video can be determined as the pause video encoding data. Furthermore, considering the amount of data processed, the video data corresponding to the first minute of the unplayed 9 minutes can also be determined as the pause video encoding data. In other words, the video data corresponding to the segment between the first and second minutes of the video played by the sending device can be determined as the pause video encoding data.

[0102] Furthermore, since the control operation command instructs a pause playback operation, the sending device needs to pause the encapsulation processing of the paused video encoding data and the generation of real-time transmission data packets. The sending device also needs to pause sending real-time transmission data packets to the receiving device, thus preventing the generation of real-time transmission data packets so that the receiving device can continue video playback. In this embodiment, the sending device does not encapsulate all video data into real-time transmission data packets for transmission at once. Instead, it divides the video being played by the sending device into multiple video segments according to data transmission requirements and latency. These video segments are then sequentially identified as the target video as defined in the aforementioned embodiment for subsequent steps.

[0103] To facilitate understanding of the encapsulation and adjustment process when the control parameter is used for pause playback, we will use the example of the receiving device sending a control operation command as an example. Figure 8As shown, the receiving device sends a control operation command to the sending device to instruct it to pause playback. At this time, the sending device can extract the pause playback parameters and determine the video encoding data that needs to be paused. Then, the sending device's player pauses video playback and stops the encapsulation of the paused video encoding data to prevent it from becoming real-time transmission data packets, thus pausing the streaming. Simultaneously, the transmission of real-time transmission data packets to the receiving device is also paused. To ensure the receiving device confirms that the sending device has responded to the control operation command, the sending device also sends an acknowledgment command to the receiving device, thereby achieving the purpose of pausing playback.

[0104] In one optional embodiment, the control parameters are playback parameters. Based on this, the video encoded data is encapsulated and adjusted according to the control parameters to adjust the real-time transmission data packet, including: determining the replay video encoded data for replay based on the playback parameters, encapsulating the replay video encoded data to obtain the replay real-time transmission data packet; the replay real-time transmission data packet includes at least the replay video encoded data and the replay video timestamps of the corresponding replay video frames.

[0105] Specifically, since the control parameters are playback parameters, it is necessary to re-stream the video played by the sending device. At this time, the sending device needs to determine the replay video encoding data based on the playback parameters. For example, as shown in the previous example, when the sending device and the receiving device play the video synchronously to the 1st minute, the target object pauses the video. So, the pause occurs at the 1st minute. When the target object needs to play the video again, the playback parameters can be used to determine that the replay should start at the 1st minute. At this time, the video data corresponding to the video segment between the 1st and 2nd minute of the video played by the sending device can be determined as the replay video encoding data.

[0106] Based on this, similar to the aforementioned embodiments, the playback speed in the replay video encoded data is used to determine the replay video timestamps of the corresponding replay video frames. The replay video encoded data is then encapsulated to obtain a replay real-time transmission data packet. Similar to the aforementioned embodiments, this replay real-time transmission data packet includes at least the replay video encoded data and the replay video timestamps of the corresponding replay video frames. Further details are omitted here.

[0107] The following explanation uses the sending of control operation commands by the receiving device as an example. Figure 9As shown, the receiving device sends a control operation command to the sending device to instruct it to perform a playback operation. At this time, the sending device can extract playback parameters and determine the replay video encoding data that needs to be replayed. Then, it restarts playback through the player on the sending device and starts pushing the stream, which means re-determining the replay video timestamp and encapsulating the replay video encoding data to obtain the replay real-time transmission data packet. Then, it sends the replay real-time transmission data packet to the receiving device and sends a confirmation command for the control operation command to the receiving device, so that the receiving device can decode the replay video encoding data to obtain the replay video decoded data. Based on each replay video timestamp, it renders the replay video decoded data to synchronize the playback of the replay video with the sending device based on each replay video timestamp, thereby completing synchronous replay.

[0108] In one optional embodiment, the control parameter is the adjustment speed. Based on this, the control parameter encapsulates and adjusts the video encoded data to adjust the real-time transmission data packet, including: determining the adjustment timestamps of the encoded data to be adjusted corresponding to the encoded data to be adjusted in the video encoded data based on the adjustment speed; encapsulating the encoded data to be adjusted to obtain the adjusted real-time transmission data packet; the adjusted real-time transmission data packet includes at least the encoded data to be adjusted and each adjustment timestamp.

[0109] The adjustment factor can be greater than or less than 1. When the adjustment factor is greater than 1, playback is essentially accelerated; when it is less than 1, playback is essentially decelerated. Specifically, since the control parameter is the adjustment speed, meaning the video being played needs to be accelerated or decelerated, the client device first determines the encoded data to be adjusted. This involves using the timestamp of the control operation command as the start timestamp, extracting a video segment of a preset duration (e.g., 2 seconds in the previous example) from the playing video, and defining this preset duration segment as the encoded data to be adjusted. Then, based on the adjustment speed, the timestamps of the corresponding encoded data to be adjusted are determined. For example, based on the previous example, if the target video is 2 seconds long and the video encoded data has 10 frames per second, and the adjustment factor is 2 (i.e., the playback speed is adjusted from 1 to 2), the playback speed changes from 10 frames per second to 20 frames per second. The video timestamp of video frame A1 is 12:00:00:30 milliseconds, and after adjustment, the adjusted timestamp of video frame A1 remains 12:00:00:30 milliseconds. The video timestamp of video frame A2 is 12:00:00:36 (30+6) milliseconds, and after adjustment, the adjusted timestamp of video frame A2 is 12:00:00:33 (30+6 / 2) milliseconds. Similarly, the video timestamp of video frame A3 is 12:00:00:42 (30+12) milliseconds, and after adjustment, the adjusted timestamp of video frame A3 is 12:00:00:36 (30+12 / 2) milliseconds.

[0110] Conversely, if the adjustment factor is 0.5, meaning the playback speed is adjusted from 1 to 0.5, the required playback speed per second decreases from 10 frames to 5 frames. The timestamp of video frame A2 is 12:00:00:36 (30+6) milliseconds; after adjustment, the timestamp of video frame A2 is 12:00:00:42 (30+6*2) milliseconds. Similarly, the timestamp of video frame A3 is 12:00:00:42 (30+12) milliseconds; after adjustment, the timestamp of video frame A3 is 12:00:00:54 (30+12*2) milliseconds.

[0111] In one optional embodiment, the control parameter is a positioning playback parameter. Based on this, the video encoded data is encapsulated and adjusted according to the control parameter to adjust the real-time transmission data packet, including: a positioning video frame indicated by the positioning playback parameter, and positioning video encoded data corresponding to the positioning video frame; the positioning video encoded data is encapsulated to obtain a positioning real-time transmission data packet; the positioning real-time transmission data packet includes at least the positioning video encoded data and the positioning video timestamps of the positioning video frames corresponding to the positioning video encoded data.

[0112] Specifically, since the control parameter is a positioning playback parameter, meaning it's necessary to fast-forward or rewind the video played by the sending device to a specific position, the sending device needs the positioning video frame indicated by the positioning playback parameter, and the corresponding positioning video encoding data. The aforementioned positioning video frame is the video frame that needs to be fast-forwarded or rewound. For example, in the previous example, video frames A1 to A10, positioning playback is performed while playing video frame A1; if fast-forwarding reaches video frame A10, then the positioning video frame is video frame A10. Alternatively, positioning playback is performed while playing video frame A0; if rewinding reaches video frame A3, then the positioning video frame is video frame A3.

[0113] Based on this, similar to the embodiments described above, a video segment of a preset duration (such as 2 seconds in the previous example) is extracted from the played video based on the positioning video frame. This preset duration video segment is determined as the positioning video encoding data. Then, in a manner similar to the embodiments described above, the positioning video timestamps corresponding to the positioning video frames are determined by the playback speed in the positioning video encoding data. The positioning video encoding data is then encapsulated to obtain the positioning real-time transmission data packet. Similar to the embodiments described above, the positioning real-time transmission data packet at this time includes at least the positioning video encoding data and the positioning video timestamps corresponding to the positioning video frames. Further details are omitted here.

[0114] The following explanation uses the sending of control operation commands by the receiving device as an example. Figure 10 As shown, taking location-based playback as an example, the receiving device sends a control operation command to the sending device to instruct it to perform a location-based playback operation. At this time, the sending device can extract the location-based playback parameters and determine the location-based video encoding data to be played. Then, it sets the parameters through the player on the sending device and starts streaming, which means re-determining the location-based video timestamp and encapsulating the location-based video encoding data to obtain a location-based real-time transmission data packet. This data packet is then sent to the receiving device, along with a confirmation command for the control operation command. This allows the receiving device to decode the location-based video encoding data to obtain location-based video decoding data. Based on each location-based video timestamp, it renders the decoded data to synchronize the playback with the sending device's video based on each timestamp, thus completing synchronized playback after fast-forwarding or rewinding. Similarly, when the control parameter is to adjust the playback speed, it is necessary to determine the encoding data to be adjusted and, based on the control parameter (i.e., the adjustment speed), re-determine the adjustment timestamp. Through similar subsequent steps, synchronized playback after adjusting the playback speed is completed.

[0115] It is understood that in this embodiment, control operation commands are transmitted via control parameters using the RTSP protocol. However, in practical applications, additional data transmission paths can be established to complete the transmission and processing of control operation commands and control parameters. Furthermore, the examples in this application are for understanding the solution and should not be construed as specific limitations of the solution.

[0116] In this embodiment, the video source data is encapsulated before decoding to avoid encoding delay caused by secondary encoding. The control parameters in the control operation initiated by the target object are determined, and the data that needs to be encapsulated is adjusted by the control parameters. The timestamp is also re-determined to ensure the accuracy of the timestamp determination and avoid playback synchronization delay caused by the control operation of the target object. Furthermore, the corresponding video frames are played and displayed using accurate and reliable timestamps, thereby reducing the playback delay of synchronized playback on different screen interfaces.

[0117] Based on the detailed description of the foregoing embodiments, the complete flow of the multi-screen playback method in the embodiments of this application will be described below. In one embodiment, such as Figure 11 As shown, a method for multi-screen playback is provided. Figure 1 Terminal 102 includes a sending end device and a receiving end device that have a communication connection within the same local area network, therefore, this method can be applied to... Figure 1 Taking the sending device of terminal 102 as an example, this method can also be applied to a system including terminal 102 and server 104, that is, requiring the interaction of the sending device, the receiving device, and server 104. In this embodiment, the method includes the following steps:

[0118] Step 1101: Start the Real-Time Streaming Protocol service within the local area network and register the synchronous playback port.

[0119] Step 1102: Send a communication transmission message to the multicast address port based on the multicast domain name system. The communication transmission message includes the playback port information of the synchronous playback port.

[0120] Step 1103: When the receiving device detects the playback port information in the multicast address port through the multicast domain name system, the receiving device sends a real-time streaming protocol message to the synchronous playback port indicated by the playback port information, thereby establishing a communication connection between the sending device and the receiving device.

[0121] Step 1104: Obtain the video source data of the target video and demultiplex the video source data to obtain the video encoded data.

[0122] Step 1105: Determine the video timestamp of each video frame in the target video by using the playback speed in the video encoding data.

[0123] Step 1106: Encapsulate the video encoded data to obtain real-time transmission data packets.

[0124] Step 1107: The real-time transmission data packet is sent to the receiving device so that the receiving device can decode the video encoded data to obtain video decoded data, and render the video decoded data based on each video timestamp so that the video frame of the target video can be played synchronously with the sending device based on each video timestamp.

[0125] Step 1108: Determine an independent clock, which is independent of the audio timestamp of each audio frame in the target video and the clock of each video timestamp.

[0126] Step 1109: Based on independent clocks and video timestamps, perform audio-visual synchronization processing on the video decoding data and audio decoding data.

[0127] Step 1110: In response to the control operation command, obtain the control parameters in the control operation command.

[0128] Step 1111: Based on the control parameters, the video encoded data is encapsulated and adjusted to adjust the real-time transmission data packets.

[0129] Step 1112: In response to the status acquisition command, acquire the status field indicated by the status acquisition command; read the playback status indicated by the status field and feed back the playback status to the receiving device.

[0130] It should be understood that the specific implementation methods of steps 1101 to 1112 are similar to those of the aforementioned embodiments, and will not be repeated here.

[0131] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0132] Based on the same inventive concept, this application also provides a multi-screen playback device for implementing the multi-screen playback method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more multi-screen playback device embodiments provided below can be found in the limitations of the multi-screen playback method described above, and will not be repeated here.

[0133] In one embodiment, such as Figure 12 As shown, a multi-screen playback device is provided, comprising: a data processing module 1202, a time determination module 1204, a packaging processing module 1206, and a multi-screen synchronous playback module 1208, wherein:

[0134] The data processing module 1202 is used to acquire the video source data of the target video and demultiplex the video source data to obtain the video encoded data;

[0135] The time determination module 1204 is used to determine the video timestamp of each video frame in the target video by using the playback speed in the video encoding data;

[0136] The encapsulation processing module 1206 is used to encapsulate the video encoded data to obtain real-time transmission data packets. The real-time transmission data packets include at least the video encoded data and each video timestamp.

[0137] The multi-screen synchronous playback module 1208 is used to send real-time transmission data packets to receiving devices located in the same local area network, so that the receiving devices can decode the video encoding data to obtain video decoding data, and render the video decoding data based on each video timestamp, so as to synchronously play the video screen of the target video based on each video timestamp.

[0138] In one embodiment, the time determination module is further configured to determine an independent clock, independent of the audio timestamp of each audio frame in the target video and the clock of each video timestamp.

[0139] The multi-screen synchronous playback module is also used to perform audio-visual synchronization processing on video decoding data and audio decoding data based on independent clocks and timestamps of each video.

[0140] In one embodiment, the encapsulation processing module is further configured to, in response to a control operation command, acquire control parameters from the control operation command; and perform encapsulation and adjustment processing on the video encoded data based on the control parameters to adjust the real-time transmission data packets.

[0141] In one embodiment, the apparatus further includes a pause transmission module;

[0142] When the control operation command is to pause playback, the control parameter is the pause playback parameter;

[0143] The encapsulation processing module is specifically used to determine the paused video encoding data based on the pause playback parameters, and to pause the encapsulation processing of the paused video encoding data and the generation of real-time transmission data packets.

[0144] The pause sending module is used to pause the transmission of real-time data packets to the receiving device.

[0145] In one embodiment, when the control operation instruction is a playback operation, the control parameter is a playback parameter;

[0146] The encapsulation processing module is specifically used to determine the replay video encoding data based on playback parameters, encapsulate the replay video encoding data, and obtain the replay real-time transmission data packet. The replay real-time transmission data packet includes at least the replay video encoding data and the replay video timestamps of the corresponding replay video frames.

[0147] In one embodiment, when the control operation command is a speed-up playback operation, the control parameter is to adjust the speed.

[0148] The encapsulation processing module is specifically used to determine the adjustment timestamps of the corresponding encoded data to be adjusted in the video encoded data based on the adjustment speed; to encapsulate the encoded data to be adjusted to obtain the adjustment real-time transmission data packet; the adjustment real-time transmission data packet includes at least the encoded data to be adjusted and each adjustment timestamp.

[0149] In one embodiment, when the control operation instruction is a positioning playback operation, the control parameter is the positioning playback parameter;

[0150] The encapsulation processing module is specifically used to encapsulate the positioning video frame indicated by the positioning playback parameters and the positioning video encoded data corresponding to the positioning video frame; to obtain the positioning real-time transmission data packet; the positioning real-time transmission data packet includes at least the positioning video encoded data and the positioning video timestamps of the positioning video frames corresponding to the positioning video encoded data.

[0151] In one embodiment, the device further includes a playback status acquisition module;

[0152] The playback status acquisition module is used to respond to a status acquisition command, acquire the status field indicated by the status acquisition command, read the playback status indicated by the status field, and feed back the playback status to the receiving device.

[0153] In one embodiment, the device further includes a communication connection module;

[0154] The communication connection module is used to initiate the real-time streaming protocol service within the local area network (LAN) and register a synchronous playback port. The synchronous playback port is used to provide synchronous playback services to playback devices within the same LAN. It sends communication transmission messages to the multicast address port based on the multicast domain name system. These messages include playback port information for the synchronous playback port. When the receiving device detects the playback port information in the multicast address port through the multicast domain name system, it sends a real-time streaming protocol message to the synchronous playback port indicated by the playback port information, thus establishing a communication connection between the sending and receiving devices.

[0155] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a multi-screen playback method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0156] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0158] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0159] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0160] It should be noted that the multi-screen playback (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0161] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based multi-screen playback logic devices, etc., and are not limited to these.

[0162] The technical features in the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for multi-screen playback, characterized in that, The method is applied to a transmitting device, wherein the transmitting device and the receiving device have a communication connection within the same local area network, and the method includes: The video source data of the target video is obtained, and the video source data is demultiplexed to obtain the video encoded data; The playback speed in the video encoding data is used to determine the video timestamp of each video frame in the target video. The video encoded data is encapsulated to obtain a real-time transmission data packet, which includes at least the video encoded data and each of the video timestamps. The real-time transmission data packet is sent to the receiving device, so that the receiving device can decode the video encoded data to obtain video decoded data, and render the video decoded data based on each video timestamp, so as to synchronously play the video frame of the target video with the sending device based on each video timestamp.

2. The method according to claim 1, characterized in that, The method includes: The audio timestamps of each audio frame in the target video and the clocks of each video timestamp are determined as independent clocks; Based on the independent clock and each of the video timestamps, the video decoding data and audio decoding data are processed for audio-visual synchronization.

3. The method according to claim 1, characterized in that, The method further includes: In response to a control operation command, the control parameters in the control operation command are obtained; The video encoded data is encapsulated and adjusted based on the control parameters to adjust the real-time transmission data packets.

4. The method according to claim 3, characterized in that, When the control operation command is a pause playback operation, the control parameter is a pause playback parameter; The process of encapsulating and adjusting the video encoded data based on the control parameters, and adjusting the real-time transmission data packets, includes: Based on the pause playback parameters, determine the pause video encoding data for pause playback, pause the encapsulation process of the pause video encoding data, and pause the generation of real-time transmission data packets; The method further includes: Pause the transmission of the real-time data packets to the receiving device.

5. The method according to claim 3, characterized in that, When the control operation command is a playback operation, the control parameter is a playback parameter; The process of encapsulating and adjusting the video encoded data based on the control parameters, and adjusting the real-time transmission data packets, includes: Based on the playback parameters, the replay video encoding data is determined, and the replay video encoding data is encapsulated to obtain a replay real-time transmission data packet; the replay real-time transmission data packet includes at least the replay video encoding data and the replay video timestamps of the corresponding replay video frames.

6. The method according to claim 3, characterized in that, When the control operation command is a speed-up playback operation, the control parameter is to adjust the speed. The process of encapsulating and adjusting the video encoded data based on the control parameters, and adjusting the real-time transmission data packets, includes: Based on the adjusted speed, determine the adjustment timestamps of the corresponding encoded data to be adjusted in the video encoded data; The encoded data to be adjusted is encapsulated to obtain an adjusted real-time transmission data packet; the adjusted real-time transmission data packet includes at least the encoded data to be adjusted and each of the adjusted timestamps.

7. The method according to claim 3, characterized in that, When the control operation command is a positioning playback operation, the control parameter is a positioning playback parameter; The process of encapsulating and adjusting the video encoded data based on the control parameters, and adjusting the real-time transmission data packets, includes: Based on the positioning playback parameters indicated by the positioning video frame, and the positioning video encoding data corresponding to the positioning video frame; The location video encoding data is encapsulated to obtain a location real-time transmission data packet; the location real-time transmission data packet includes at least the location video encoding data and the location video timestamps of the location video frames corresponding to the location video encoding data.

8. The method according to claim 1, characterized in that, The method further includes: In response to a status acquisition command, the status field indicated by the status acquisition command is acquired; Read the playback status indicated by the status field and feed the playback status back to the receiving end.

9. The method according to claim 1, characterized in that, The method includes: The Real-Time Streaming Protocol service is started within the local area network, and a synchronous playback port is registered. The synchronous playback port is used to provide synchronous playback services to playback devices within the same local area network. Based on the multicast domain name system, a communication transmission message is sent to the multicast address port, and the communication transmission message includes the playback port information of the synchronous playback port; When the receiving device detects the playback port information in the multicast address port through the multicast domain name system, it sends a real-time streaming protocol message to the synchronous playback port indicated by the playback port information, thereby establishing a communication connection between the sending device and the receiving device.

10. A multi-screen playback device, characterized in that, The device includes: The data processing module is used to acquire the video source data of the target video and demultiplex the video source data to obtain video encoded data; The time determination module is used to determine the video timestamp of each video frame in the target video based on the playback speed in the video encoding data; The encapsulation processing module is used to encapsulate the video encoded data to obtain a real-time transmission data packet, wherein the real-time transmission data packet includes at least the video encoded data and each of the video timestamps; The multi-screen synchronous playback module is used to send the real-time transmission data packet to the receiving end device in the same local area network, so that the receiving end device can decode the video encoding data to obtain video decoding data, and render the video decoding data based on each video timestamp, so as to synchronously play the video screen of the target video based on each video timestamp.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 9.

Images