Video processing methods, video playback methods and devices, equipment and media

Abstract

This disclosure provides a video processing method, video playback method, apparatus, device, and medium, relating to the field of computer technology, and particularly to the field of video processing and video playback technology. The implementation scheme includes: acquiring multiple video files to be played on a client; determining multiple sets of video frame data corresponding to each of the multiple video files by performing decoding operations; establishing a point-to-point connection between the server and the client; responding to a first instruction to play a first video file from the multiple video files, pushing the corresponding first video frame data of the first video file to the client using the initial media track of the point-to-point connection; and responding to a second instruction to play a second video file from the multiple video files, while maintaining the point-to-point connection, calling a track replacement interface to replace the initial media track with a new media track associated with the corresponding second video frame data of the second video file, so as to push the second video frame data to the client.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of video processing and video playback technology, specifically to video processing methods, video playback methods and apparatus, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] With the development of internet technology and multimedia applications, real-time video generation and interactive scenarios are becoming increasingly common. In these scenarios, users often need to switch frequently between multiple video sources. Traditional video playback technologies typically load multiple video files directly on the user's client or transmit video data through conventional streaming media protocols. However, the video source switching operation under traditional video playback methods is relatively complex, cannot meet the low-latency requirements of real-time interactive scenarios, and also consumes a lot of hardware resources, making it unsuitable for devices with limited hardware performance, such as mobile terminals.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides video processing methods for use on a server, video playback methods and apparatus for use on a client, electronic devices, computer-readable storage media, and computer program products.

[0005] According to one aspect of this disclosure, a video processing method applied to a server is provided, comprising: acquiring multiple video files to be played on a client; determining multiple sets of video frame data corresponding to the multiple video files by performing a decoding operation based on the multiple video files; establishing a peer-to-peer connection between the server and the client; in response to receiving a first instruction to play a first video file among the multiple video files, pushing the first video frame data corresponding to the first video file to the client using an initial media track of the peer-to-peer connection; and in response to receiving a second instruction to play a second video file among the multiple video files, while maintaining the peer-to-peer connection, calling a track replacement interface to replace the initial media track with a new media track associated with the second video frame data corresponding to the second video file, so as to push the second video frame data to the client.

[0006] According to one aspect of this disclosure, a video playback method applied to a client is provided, comprising: receiving first video frame data from the server based on a peer-to-peer connection between the server and the client, wherein the first video frame data is obtained by the server based on a first video file and is pushed by the server using an initial media track of the peer-to-peer connection; performing a rendering playback operation based on the first video frame data; transmitting the second instruction to the server in response to receiving a second instruction to play a second video file; receiving second video frame data from the server, wherein the second video frame data is obtained by the server based on a second video file and is pushed by the server to the client by calling a track replacement interface to replace the initial media track with a new media track associated with the second video frame data while maintaining the peer-to-peer connection; and performing a rendering playback operation based on the second video frame data.

[0007] According to one aspect of this disclosure, a video processing apparatus for a server is provided, comprising: an acquisition unit configured to acquire multiple video files to be played on a client; a decoding unit configured to determine multiple sets of video frame data corresponding to the multiple video files by performing a decoding operation based on the multiple video files; a connection unit configured to establish a peer-to-peer connection between the server and the client; and a push unit configured to: in response to receiving a first instruction to play a first video file among the multiple video files, push the first video frame data corresponding to the first video file to the client using an initial media track of the peer-to-peer connection; and in response to receiving a second instruction to play a second video file among the multiple video files, while maintaining the peer-to-peer connection, call a track replacement interface to replace the initial media track with a new media track associated with the second video frame data corresponding to the second video file, so as to push the second video frame data to the client.

[0008] According to one aspect of this disclosure, a video playback device for a client is provided, comprising: a receiving unit configured to receive first video frame data from a server based on a peer-to-peer connection between the server and the client, wherein the first video frame data is obtained by the server based on a first video file and is pushed by the server using an initial media track of the peer-to-peer connection; a playback unit configured to perform a rendering playback operation based on the first video frame data; and a transmission unit configured to transmit a second instruction to the server in response to receiving a second instruction to play a second video file, wherein the receiving unit is further configured to receive second video frame data from the server, wherein the second video frame data is obtained by the server based on a second video file and is pushed by the server to the client by calling a track replacement interface to replace the initial media track with a new media track associated with the second video frame data while maintaining the peer-to-peer connection, and wherein the playback unit is further configured to perform a rendering playback operation based on the second video frame data.

[0009] According to one aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the video processing method or video playback method described above.

[0010] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores computer instructions, wherein the computer instructions are used to cause the computer to perform the video processing method or video playback method described above.

[0011] According to one aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, is capable of implementing the above-described video processing method or video playback method.

[0012] According to one or more embodiments of this disclosure, video playback efficiency and smoothness can be improved.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram of an exemplary system in which various methods described herein may be implemented, according to exemplary embodiments of the present disclosure;

[0016] Figure 2 A flowchart is shown of a video processing method applied to a server according to an exemplary embodiment of the present disclosure; Figure 3 A flowchart illustrating a video playback method applied to a client according to an exemplary embodiment of the present disclosure is shown; Figure 4 An interactive flowchart of a video playback method according to an exemplary embodiment of the present disclosure is shown; Figure 5 A structural block diagram of a video processing apparatus applied to a server according to an exemplary embodiment of the present disclosure is shown; Figure 6 A structural block diagram of a video playback device applied to a client according to an exemplary embodiment of the present disclosure is shown; Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0017] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0019] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0020] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0021] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0022] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of video processing methods or video playback methods.

[0023] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0024] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0025] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to view videos or initiate video playback commands. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0026] Client devices 101, 102, 103, 104, 105, and / or 106 may include various categories of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various categories and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0027] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0028] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0029] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0030] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0031] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0032] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different categories. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0033] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be different categories of databases, such as key-value stores, object stores, or regular stores supported by a file system.

[0034] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0035] With the development of internet technology and multimedia applications, real-time video generation and interactive scenarios are becoming increasingly common. In this scenario, users often need to switch frequently between multiple video sources. Traditional video playback technologies typically load multiple video files directly on the user's client or transmit video data through conventional streaming media protocols.

[0036] In existing video streaming and switching technologies, the video source is typically switched by directly changing the front-end video element. This requires the front-end to reload the video file, such as re-establishing a network connection and re-buffering or loading a new video file, leading to noticeable playback interruptions or black screens, which cannot meet the low-latency requirements of real-time interactive scenarios. Furthermore, this method consumes a significant amount of memory and network bandwidth on the client device, making it unsuitable for devices with limited hardware performance, such as mobile terminals. Another approach is to use streaming media technologies based on the Hypertext Transfer Protocol (HTTP) to transmit the video stream, such as real-time streaming or dynamic adaptive streaming, which essentially divides the video into several slices for progressive loading and transmission. However, conventional streaming media protocols have high switching latency and are limited by the browser's underlying mechanisms, making it difficult to achieve precise time synchronization and frame-level control between multiple video sources.

[0037] Based on this, this disclosure provides video processing and playback methods for both the server and client sides. This scheme pre-decodes video files in batches on the server backend, and then directly pushes the decoded video frame data to the frontend based on the real-time peer-to-peer connection established between the server and client. When a video source switching command is received, the transmitting media track is switched directly through the underlying interface while maintaining an uninterrupted real-time peer-to-peer connection, thereby achieving rapid video stream switching. This mechanism shifts the computationally intensive video decoding task from the client to the server, saving the overhead of reloading video files and rebuilding network connections on the frontend, reducing client resource consumption and video switching latency.

[0038] Figure 2 A flowchart of a video processing method 200 applied to a server according to an exemplary embodiment of the present disclosure is shown. Figure 2 As shown, method 200 includes: Step S201: Obtain multiple video files to be played on the client; Step S202: By performing decoding operations based on multiple video files, determine multiple sets of video frame data corresponding to each of the multiple video files; Step S203: Establish a point-to-point connection between the server and the client; Step S204: In response to receiving a first instruction to play the first video file among multiple video files, push the corresponding first video frame data of the first video file to the client using the initial media track of the point-to-point connection; and Step S205: In response to receiving a second instruction to play a second video file among multiple video files, while maintaining a point-to-point connection, the track replacement interface is called to replace the initial media track with a new media track associated with the second video file's corresponding second video frame data, so as to push the second video frame data to the client.

[0039] By applying the above method 200, video files can be pre-decoded in batches on the server backend. Then, based on the real-time peer-to-peer connection established between the server and client, the decoded video frame data is directly pushed to the frontend. When a video source switching command is received, the media track being transmitted is switched directly by calling the track replacement interface while maintaining an uninterrupted real-time peer-to-peer connection, thus achieving rapid switching of video content. Server-side decoding of video files saves on client-side decoding buffer latency, reducing frontend stuttering. By executing multiple video decoding tasks on the server, video source switching can be achieved more efficiently and conveniently. The dynamic replacement mechanism of media tracks in the peer-to-peer connection saves on latency caused by network reconnection and video source loading during traditional video stream switching, reducing client-side computing power consumption and network bandwidth usage, and improving the stability of video switching and the smoothness of video playback.

[0040] The aforementioned method 200 can be applied to various interactive scenarios that require high real-time video switching and smooth visual continuity. For example, in virtual reality (VR) street view roaming or panoramic navigation, when a user turns their head or clicks on directional arrows, the system needs to quickly switch to a new perspective of high-definition real-scene video; in interactive games based on dynamic images, the user's real-time choices at key plot points will trigger different branching plot scenes; in educational demonstrations, multi-camera sports broadcasts, and other scenarios, users typically need to switch frequently between close-ups of lecturers, courseware screens, or multiple camera angles.

[0041] In step S201, the multiple video files to be played can be pre-stored on the server or be a video sequence output by a generative model in the background. These multiple video files can be digital files in MP4, MKV, or AVI formats, for example. In different specific application scenarios, the content corresponding to the multiple video files has different focuses. For example, in a panoramic navigation scenario, different video files correspond to continuous live-action recordings of movement from the same intersection in different directions; in multi-camera educational recordings or sports broadcasts, different video files correspond to content from different camera angles; and in interactive applications based on virtual reality, different video files correspond to different plot developments or story branches triggered by different user actions.

[0042] In some examples, the decoding operation in step S202 is implemented by calling a video codec module deployed on the server. The video file before decoding is a compressed video file, and the decoding operation restores the original video footage and audio tracks. The decoded video frame data is image pixel data with the video container format and compression algorithm removed, such as the original image data in RGB or YUV color space, which can then be directly rendered pixel-level to obtain displayable content. In one example, the decoding operation is implemented using the FFmpeg tool. The FFmpeg tool performs batch decoding of MP4 format video files in a Node.js backend environment, de-frames them, and converts them into a sequence of original RGB24 format video frame data.

[0043] In some examples, the operations described in steps S201 and S202 are completed before the actual video playback request is received. In one example, the server preloads multiple candidate video sources in parallel. During the initialization phase for each specific video source, the server pre-starts and completes the aforementioned decoding operations and preloads the decoded raw video frame data into the memory frame buffer allocated for each video source. This ensures that the relevant video frame data is in a ready-to-use state, enabling faster response to the client's video frame push requests, reducing video loading and switching latency, saving decoding computational overhead, and improving video playback efficiency.

[0044] In some examples, a peer-to-peer connection is an end-to-end, low-latency data transmission channel established between the server and the client. In a peer-to-peer connection, the server and client can exchange video data in real time without relying on a third-party relay server for caching or forwarding. In one example, the peer-to-peer connection between the server and client is implemented using Web Real-Time Communications (WebRTC) technology. WebRTC allows web applications or websites to establish peer-to-peer connections between browsers without the need for intermediaries or additional plugins. For example, the server can use the @koush / wrtc library resources and leverage its non-standard RTCVideoSource API to manage video frame data. Simultaneously, the client can create a corresponding WebRTCClient instance and establish a communication interface with the server through a TCP-based WebSocket channel. After a successful handshake, both parties obtain a channel for pushing video frame data and can further establish a channel for transmitting control commands and feedback information using WebRTC DataChannel technology, achieving low-latency bidirectional data transmission.

[0045] Besides WebRTC, peer-to-peer connections can also be built using other underlying network protocols. For example, a peer-to-peer connection can be established based on the User Datagram Protocol (UDP). In this example, the server can directly split the decoded video frame data into data packets with sequence numbers and error correction codes, and then push them to the client via the UDP link. After receiving the data packets, the client can reassemble them into complete frame data according to the sequence numbers for rendering, thus achieving efficient video data transmission.

[0046] In some examples, the first instruction can take various forms, such as a structured text message, a JSON data block in a specific format, or binary control code. The first instruction can be automatically generated or triggered based on interactive actions performed by the client user on an interactive interface. For example, when method 200 is applied to a virtual reality application, the system can automatically trigger the generation of a first instruction requesting the loading of an initial viewpoint video when a user logs into the application. As another example, when method 200 is applied to a browser player interface, when a user actively clicks the play button on the player interface, the front-end application logic can assemble a first instruction for the first video file corresponding to the play button based on this physical click event and send it to the server.

[0047] In some examples, the initial media track in step S204 is a basic logical data stream channel allocated to both the server and the client after a successful point-to-point connection is established. The media track is responsible for carrying video frame data and transmitting it from the sending end (server) to the receiving end (client). In one example, when the first video frame data is pushed using the initial media track, the push program configured on the server will continuously acquire the decoded first video frame data corresponding to the first video file according to a set rhythm and inject it into the sending queue of the initial media track. In another example, if the initial decoding result of the video file by the server does not match the requirements of the media track or the client's playback format requirements, the server can perform a light format conversion operation on the video frame data before injecting it into the media track for push. For example, after acquiring the video frame data to be pushed, the server can use a color space algorithm to convert the original RGB24 format frame data into the corresponding YUV format frame data of the WebRTC standard (e.g., I420 format), and then push the format-converted frame data to the client by calling push methods such as onFrame.

[0048] In some examples, the second instruction for the second video file is a control signal triggered by the user while watching the video, or it can be a control signal triggered by the system based on the user's video viewing progress and plot development. For example, when using panoramic navigation, a user can click the direction button to determine the next view. As another example, a user can also select plot branches by manipulating plot options when watching interactive videos, or select different camera angles when watching educational videos or sports broadcasts. Based on this, the system can generate a second instruction to request video switching based on user actions or the user's video viewing progress and send it to the server.

[0049] In some examples, the new media track in step S205 is a logically re-established or newly allocated data stream channel within the existing peer-to-peer connection between the server and client, while the connection remains open, to transmit new video content. The track replacement interface is a functional interface provided by the underlying network communication protocol (such as the WebRTC standard) that allows the server to dynamically change the source of media data transmission. While maintaining the peer-to-peer connection, the server only needs to call the track replacement interface to replace the video frame data to be transmitted with the replacement track, while the state information of the peer-to-peer connection (such as network socket, datagram encryption status, handshake credentials, etc.) remains unchanged, thus maintaining the communication state and improving the efficiency and convenience of video source switching.

[0050] In one example, the server can use the `replaceTrack()` method from the WebRTC API specification to implement the video source switching operation in step S205, thereby enabling the underlying replacement of the video track without re-establishing a network connection. When the server receives the second instruction to switch the video, it directly calls the `replaceTrack()` tool to replace the initial track associated with the current transmitter with a new track mapping the new video frame data. By maintaining the communication state between the server and the client, the efficiency and convenience of video source switching are improved. In some examples, to further optimize the user experience, the server can also apply the waitForCurrentVideoComplete control method before performing the track replacement action to switch the video source, to wait for the currently pushed video segment or keyframe group to finish playing, to avoid video flickering or playback interruption, thereby optimizing the video transition effect and improving the user experience.

[0051] According to some embodiments, method 200 further includes: caching multiple sets of video frame data into multiple memory buffers respectively, wherein step S204, which uses the initial media track of the point-to-point connection to push the first video frame data corresponding to the first video file to the client, includes: mapping the initial media track of the point-to-point connection to the first memory buffer corresponding to the first video file; and using the initial media track to push the first video frame data extracted from the first memory buffer, wherein the new media track is used to push the second video frame data extracted from the second memory buffer corresponding to the second video file.

[0052] Therefore, a memory buffer can be used to temporarily store the decoded video frame data, decoupling the static video decoding result from the dynamic data transmission channel. In this case, the server can perform more efficient addressing and switching operations between multiple independent video sources, thereby responding more efficiently to interactive commands indicating video switching and improving video switching efficiency.

[0053] A memory buffer is a temporary data storage block allocated in the server's runtime memory (e.g., physical RAM) to store video frame data that may be accessed frequently. After the pre-decoding operation described above is completed, the server can write pixel data into the pre-allocated memory block according to the natural playback sequence of the video. In one example, the memory buffers corresponding to different video files can be maintained based on unique identifiers. For example, a unique identifier can be assigned to each video file and used as the index key of a hash table or memory address dictionary, thereby enabling independent maintenance of memory buffer space for multiple video sources to support efficient switching of video sources.

[0054] In one example, when the server needs to send specific video content using a media track in a peer-to-peer connection, it can locate the starting address pointer of the memory buffer corresponding to the target video based on the unique identifier of the target video, and then pass the memory address information to the sending component of the media track, so that the sending component in the network connection can automatically extract the video frame data in the memory buffer based on the address pointer and transmit it through the media track.

[0055] According to some embodiments, step S204, pushing the first video frame data extracted from the first memory buffer using the initial media track, includes: periodically extracting single-frame data from the first video frame data in the first memory buffer according to the target playback frame rate and pushing it using the initial media track. By periodically extracting and pushing video frame data frame by frame according to the target playback frame rate, the frame rate of video push and playback can be controlled more precisely, improving the stability of video playback. At the same time, the timing of video switching can be controlled more precisely during video switching, improving the smoothness of video switching.

[0056] In one example, the server can periodically extract and push video frame data based on a customized mechanism. For instance, if the system's target video playback frame rate is 30 frames per second (30fps), the server can determine that the ideal time interval between two data pushes is approximately 33.3 milliseconds. Based on this, a timer can be set to extract the decoded video frame data at certain time intervals and push it continuously and stably, thereby improving the stability of video playback.

[0057] In one example, the server can also use other mechanisms to extract and push video frame data, such as monitoring the amount of local cached data on the client to determine whether new video frame data needs to be pushed, so as to achieve dynamic adaptive utilization of available network bandwidth and improve the utilization rate of data transmission resources.

[0058] According to some embodiments, method 200 further includes: creating multiple video indexes corresponding to the multiple video files respectively; and pushing the multiple video indexes to a client, so that the client user can initiate playback commands for any of the multiple video files based on the multiple video indexes. Thus, operation commands for video files can be transmitted based on video indexes. By providing video index information to the client user, the client user can interact based on the video indexes, improving the convenience of interaction and the flexibility of video playback control.

[0059] In some examples, multiple video indexes correspond to unique identifiers assigned by the system to multiple video files, or may further include descriptive information and metadata (such as video duration information) for the video files. Upon receiving multiple video indexes, the client can instantiate them into intuitive interactive interface elements, allowing users to initiate playback commands based on these indexes. For example, the client can generate a window containing multiple video thumbnail cards based on the video indexes. When a user clicks the play button corresponding to a video thumbnail card, the client's interactive interface event listener captures the user's action information and generates a command message, which is then sent to the server through the established peer-to-peer connection channel to instruct the server to switch video sources.

[0060] According to one aspect of this disclosure, a video playback method applied to a client is also provided. Figure 3 A flowchart of a video playback method 300 applied to a client according to an exemplary embodiment of the present disclosure is shown. Figure 3 As shown, method 300 includes: Step S301: Based on the peer-to-peer connection between the server and the client, receive the first video frame data from the server. The first video frame data is obtained by the server based on the decoding of the first video file, and the first video frame data is pushed by the server using the initial media track of the peer-to-peer connection. Step S302: Perform rendering and playback operations based on the first video frame data; Step S303: In response to receiving a second instruction to play the second video file, transmit the second instruction to the server; Step S304: Receive second video frame data from the server, wherein the second video frame data is obtained by the server based on the second video file, and the second video frame data is pushed to the client by the server while maintaining a point-to-point connection by calling the track replacement interface to replace the initial media track with a new media track associated with the second video frame data; and Step S305: Perform rendering and playback operations based on the second video frame data.

[0061] By applying the method described above 300, server-side decoding of video files can save latency caused by client-side decoding buffering, reducing front-end stuttering. Simultaneously, by applying a dynamic media track replacement mechanism in point-to-point connections, latency caused by network reconnection and video source loading during traditional video stream switching is reduced. This allows the client to complete video playback under low load (only pixel-level rendering of already decoded video frame data is required). This implementation reduces the hardware computing power requirements of the client during video playback, enabling low-configuration client devices to support higher-definition or smoother multi-channel video playback or switching, thereby improving the user experience.

[0062] In step S301, the client receives video frame data based on the peer-to-peer connection between the client and the server described above. The client triggers data reception, decapsulation, and other operations by continuously listening for incoming data packets in the media track of the peer-to-peer connection. For example, a WebRTCClient communication component can be deployed in the client, and the transceiver's media transmission direction attribute can be set to "recvonly" (receive only) by calling the addTransceiver method in the API to establish a dedicated connection for unidirectional reception of server video data, avoiding unnecessary uplink resource consumption and improving data transmission efficiency.

[0063] In step S302, the rendering and playback operation based on video frame data can be implemented using the graphics processing hardware module on the client device. For example, after receiving video data, the client can bind it to the HTML5 video tag element in the webpage to call the browser's media playback engine, and then use the hardware acceleration mechanism of the local device to render and play the video data, thereby obtaining playable video and audio data.

[0064] According to some embodiments, method 300 further includes: receiving multiple video indexes corresponding to multiple video files respectively; and displaying an interactive interface including the multiple video indexes, wherein the second instruction is determined based on the user's interactive operation on the corresponding video index of the second video file. Thus, operation instructions for video files can be transmitted based on video indexes. By providing video index information to client users, client users can interact based on video indexes, improving the convenience of interaction and the flexibility of video playback control.

[0065] In some examples, multiple video indexes correspond to unique identifiers assigned by the system to multiple video files, or may further include descriptive information and metadata (such as video duration information) for the video files. Upon receiving multiple video indexes, the client can instantiate them into intuitive interactive interface elements, allowing users to initiate playback commands based on these indexes. For example, the client can generate a window containing multiple video thumbnail cards based on the video indexes. When a user clicks the play button corresponding to a video thumbnail card, the client's interactive interface event listener captures the user's action information and generates a command message, which is then sent to the server through the established peer-to-peer connection channel to instruct the server to switch video sources.

[0066] According to some embodiments, method 300 further includes: displaying the configuration status of multiple video files on an interactive interface, the configuration status indicating the connection status between the server and the client and / or the processing status of the server on the multiple video files. This allows the system configuration and processing status of the backend to be displayed in real time on the client's interactive interface, enabling users to intuitively monitor whether the video resources are in a ready state, thereby improving the user experience.

[0067] In one example, the connection status between the server and the client characterizes the connectivity of the point-to-point communication link, indicating whether it is currently possible to switch video sources to receive video data pushed by the server. The server's processing status for multiple video files characterizes the decoding progress or memory buffer information of the multiple video files, indicating whether each video source can be quickly switched and played.

[0068] Figure 4 An interactive flowchart of a video playback method 400 according to an exemplary embodiment of this disclosure is shown. Figure 4 As shown, method 400 is implemented jointly by the server and the client, and specifically includes the following steps: Step S401: Establish a peer-to-peer connection. The client and server establish a peer-to-peer connection based on the WebRTC protocol. In this step, the client creates a WebRTCClient instance, establishes a signaling connection with the server via WebSocket, and sets the receiver direction to receive only to build a low-latency media transmission channel.

[0069] Step S402: Preload multiple video files: The server starts a parallel preloading mechanism in the background to establish a video index mapping relationship. Specifically, the server uses FFmpeg to perform batch decoding operations on multiple candidate video sources (such as MP4 files), converting the videos into raw frame data in RGB24 format, and pre-filling them into the independent memory buffers of each video source to ensure that the data is in a ready state for immediate push.

[0070] Step S403: Pushing the first video frame data: In response to the initial playback command, the server periodically extracts the first video frame data from the corresponding first memory buffer. Before pushing, the server periodically converts the raw frame data into I420 pixel format conforming to the WebRTC standard based on a preset frame rate (e.g., 30fps) and pushes it to the client through the initialized initial media track.

[0071] Step S404: Rendering and Playing the First Video: The client receives the video stream pushed by the server by listening to the ontrack event. The received first video frame data is directly bound to the HTML5 video element, utilizing the browser's hardware acceleration mechanism to achieve efficient rendering and real-time playback.

[0072] Step S405: Identify video switching instructions: The client monitors the user's interactive behavior in real time through an interactive interface. For example, when the user initiates a switching request through the "forward / rewind" button or text input command, the client identifies the playback intention for a specific second video file and generates a video switching instruction.

[0073] Step S406: Transmit video switching command: The client transmits the generated switching command to the server in real time through the WebRTC data channel. This command is sent through the established point-to-point link, ensuring extremely low latency and high real-time performance of the command transmission.

[0074] Step S407: Call the track replacement interface: After receiving the video switching command, the server calls the underlying track replacement interface (such as the replaceTrack() method) while maintaining an uninterrupted point-to-point connection. Through this dynamic change of logical mapping, the server switches the video source associated with the initial media track from the current buffer to the target buffer corresponding to the second video file.

[0075] Step S408: Pushing the second video frame data: Based on the new track mapping relationship, the server starts to extract the second video frame data from the second memory buffer and continuously pushes it.

[0076] Step S409: Render and play the second video: The client playback component can directly receive and render the new video stream without re-establishing a connection or reloading the file, thereby achieving low-latency video switching and improving the user experience.

[0077] According to one aspect of this disclosure, a video processing apparatus for use on a server is also provided. Figure 5 A structural block diagram of a video processing apparatus 500 applied to a server according to an exemplary embodiment of the present disclosure is shown. Figure 5 As shown, the device 500 includes: Acquisition unit 501 is configured to acquire multiple video files to be played on the client. Decoding unit 502 is configured to determine multiple sets of video frame data corresponding to the multiple video files by performing decoding operations based on the multiple video files; Connection unit 503 is configured to establish a point-to-point connection between the server and the client; and The push unit 504 is configured to: in response to receiving a first instruction to play a first video file among multiple video files, push the corresponding first video frame data of the first video file to the client using an initial media track with a point-to-point connection; and in response to receiving a second instruction to play a second video file among multiple video files, while maintaining the point-to-point connection, call the track replacement interface to replace the initial media track with a new media track associated with the corresponding second video frame data of the second video file, so as to push the second video frame data to the client.

[0078] According to some embodiments, the apparatus 500 further includes: a caching unit configured to cache multiple sets of video frame data into multiple memory buffers respectively; and a push unit 504 configured to: map an initial media track connected in a point-to-point manner to a first memory buffer corresponding to a first video file; and push first video frame data extracted from the first memory buffer using the initial media track, and a new media track is used to push second video frame data extracted from a second memory buffer corresponding to a second video file.

[0079] According to some embodiments, the push unit 504 is configured to periodically extract single-frame data from the first video frame data from the first memory buffer according to the target playback frame rate and push it using the initial media track.

[0080] According to some embodiments, the apparatus 500 further includes: a creation unit configured to create multiple video indexes corresponding to multiple video files respectively, and a push unit 504 further configured to push the multiple video indexes to a client, so that a user of the client can initiate a playback command for any one of the multiple video files based on the multiple video indexes.

[0081] According to one aspect of this disclosure, a video playback device for use on a client is also provided. Figure 6 A structural block diagram of a video playback device 600 applied to a client according to an exemplary embodiment of the present disclosure is shown. Figure 6 As shown, the device 600 includes: The receiving unit 601 is configured to receive first video frame data from the server based on a point-to-point connection between the server and the client. The first video frame data is obtained by the server based on the decoding of the first video file, and the first video frame data is pushed by the server using the initial media track of the point-to-point connection. Playback unit 602 is configured to perform rendering and playback operations based on the first video frame data; and The transmission unit 603 is configured to transmit the second instruction to the server in response to receiving a second instruction to play the second video file. The receiving unit 601 is further configured to receive second video frame data from the server. This second video frame data is obtained by the server based on a second video file, and is pushed to the client by the server while maintaining a point-to-point connection by calling a track replacement interface to replace the initial media track with a new media track associated with the second video frame data. Furthermore, the playback unit 602 is configured to perform rendering and playback operations based on the second video frame data.

[0082] According to some embodiments, the receiving unit 601 is further configured to receive multiple video indexes corresponding to multiple video files respectively, and the device 600 further includes a display unit configured to display an interactive interface including multiple video indexes, wherein the second instruction is determined based on the user's interactive operation on the corresponding video index of the second video file.

[0083] According to some embodiments, the display unit is also configured to display the configuration status of multiple video files on an interactive interface, the configuration status indicating the connection status between the server and the client and / or the processing status of the server on the multiple video files.

[0084] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0085] According to one aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the video processing method or video playback method described above.

[0086] According to one aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform the video processing method or video playback method described above.

[0087] According to one aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above-described video processing method or video playback method.

[0088] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0089] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0090] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to device 700. Input unit 706 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, a hard disk and an optical disk. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0091] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as video processing methods or video playback methods. For example, in some embodiments, the video processing method or video playback method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the video processing method or video playback method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform a video processing method or a video playback method by any other suitable means (e.g., by means of firmware).

[0092] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0096] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0097] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0098] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0099] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A video processing method applied to a server, comprising: Retrieve multiple video files to be played on the client; By performing decoding operations based on the multiple video files, multiple sets of video frame data corresponding to the multiple video files are determined; Establish a peer-to-peer connection between the server and the client; In response to receiving a first instruction to play the first video file of the plurality of video files, the first video frame data of the first video file is pushed to the client using the initial media track of the point-to-point connection; as well as In response to receiving a second instruction to play the second video file among the plurality of video files, while maintaining the point-to-point connection, the track replacement interface is invoked to replace the initial media track with a new media track associated with the corresponding second video frame data of the second video file, so as to push the second video frame data to the client.

2. The method of claim 1, further comprising: The multiple sets of video frame data are cached into multiple memory buffers respectively. The step of pushing the first video frame data corresponding to the first video file to the client using the initial media track of the point-to-point connection includes: Map the initial media track of the point-to-point connection to the corresponding first memory buffer of the first video file; and The first video frame data extracted from the first memory buffer is pushed using the initial media track. Furthermore, the new media track is used to push the second video frame data extracted from the corresponding second memory buffer of the second video file.

3. The method as described in claim 2, wherein, The step of pushing the first video frame data extracted from the first memory buffer using the initial media track includes: Based on the target playback frame rate, single-frame data is periodically extracted from the first video frame data in the first memory buffer and pushed using the initial media track.

4. The method according to any one of claims 1-3, further comprising: Create multiple video indexes corresponding to the multiple video files respectively; as well as The multiple video indexes are pushed to the client so that the client's user can initiate a playback command for any of the multiple video files based on the multiple video indexes.

5. A video playback method applied to a client, comprising: Based on the peer-to-peer connection between the server and the client, the first video frame data is received from the server, wherein the first video frame data is obtained by the server based on the decoding of the first video file, and the first video frame data is pushed by the server using the initial media track of the peer-to-peer connection. Rendering and playback operations are performed based on the first video frame data; In response to receiving a second instruction to play the second video file, The second instruction is transmitted to the server. The server receives second video frame data, wherein the second video frame data is obtained by the server based on the second video file, and the second video frame data is pushed to the client by the server, while maintaining the peer-to-peer connection, by calling the track replacement interface to replace the initial media track with a new media track associated with the second video frame data; and Rendering and playback operations are performed based on the second video frame data.

6. The method of claim 5, further comprising: Receive multiple video indexes corresponding to the multiple video files respectively; as well as Display an interactive interface including the plurality of video indexes, wherein the second instruction is determined based on the user's interactive operation on the corresponding video index of the second video file.

7. The method of claim 6, further comprising: The interactive interface displays the configuration status of the multiple video files, which indicates the connection status between the server and the client and / or the processing status of the server on the multiple video files.

8. A video processing apparatus for use on a server, comprising: The first acquisition unit is configured to acquire multiple video files to be played on the client. The first determining unit is configured to determine multiple sets of video frame data corresponding to the multiple video files by performing a decoding operation based on the multiple video files; The connection unit is configured to establish a point-to-point connection between the server and the client; as well as The push unit is configured as follows: In response to receiving a first instruction to play the first video file of the plurality of video files, the first video frame data of the first video file is pushed to the client using the initial media track of the point-to-point connection; as well as In response to receiving a second instruction to play the second video file among the plurality of video files, while maintaining the point-to-point connection, the track replacement interface is invoked to replace the initial media track with a new media track associated with the corresponding second video frame data of the second video file, so as to push the second video frame data to the client.

9. A video playback device for a client, comprising: The receiving unit is configured to receive first video frame data from the server based on a peer-to-peer connection between the server and the client, wherein the first video frame data is obtained by the server based on the decoding of a first video file, and the first video frame data is pushed by the server using the initial media track of the peer-to-peer connection. The playback unit is configured to perform rendering and playback operations based on the first video frame data; and The transmission unit is configured to transmit the second instruction to the server in response to receiving a second instruction to play the second video file. The receiving unit is further configured to receive second video frame data from the server. The second video frame data is obtained by the server based on the second video file and is pushed to the client by the server while maintaining the peer-to-peer connection by calling a track replacement interface to replace the initial media track with a new media track associated with the second video frame data. Furthermore, the playback unit is configured to perform rendering and playback operations based on the second video frame data.

10. An electronic device, comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

12. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-7.

Images