Audio and video playing synchronization method and system, terminal equipment and storage medium

Abstract

The invention is applicable to the technical field of intelligent terminals, and provides an audio and video playing synchronization method and system, terminal equipment and a storage medium, the method is applied to a client, and the method comprises the following steps: when it is detected that an audio and video file is inserted into a collaboration platform, initializing a local player primitive, and displaying the local player primitive in a canvas mode on a user interface; when a selection operation initiated for the local player primitive is detected, loading and displaying a play control component on the local player primitive; when a play control instruction sent by a user through a play control component is detected, displaying a local player primitive in a document object model mode on a user interface; the playing control instruction is synchronously submitted to the server, and the server forwards the playing control instruction to other clients participating in the collaboration platform, so that audio and video playing synchronization is realized. According to the application, each client participating in the collaboration platform can be ensured to smoothly and efficiently perform audio and video synchronous playing.

Description

Technical Field

[0001] The present application relates to the field of smart terminal technology, and in particular to a method, system, terminal device and storage medium for synchronizing audio and video playback. Background Art

[0002] The collaborative mode is an important technical form that supports real-time online collaboration among multiple people, and can realize the synchronous sharing and interaction of resources such as documents and editing content. The traditional collaborative mode mainly realizes the synchronization of basic functions through direct communication between clients or forwarding by a central server. For example, collaborative documents can support multiple people to edit text content at the same time, and collaborative editing tools can realize real-time updates of code snippets. The collaborative mode can also be further extended to interactive whiteboard functions. Taking the collaborative whiteboard as an example, it is an electronic whiteboard that supports multiple people to operate online at the same time. Multiple users can connect to the same collaborative whiteboard through the network, share and edit the content on the whiteboard in real time, thereby realizing remote collaborative communication. In modern remote collaboration and online communication scenarios, collaborative whiteboards play an important role.

[0003] However, traditional collaborative platforms have deficiencies in handling the synchronization of audio and video file playback and cannot guarantee the smoothness of collaboration. How to ensure that all clients participating in the collaborative platform can smoothly and efficiently play audio and video synchronously and promote the smoothness of collaboration is a technical issue that needs to be considered at present. Summary of the Invention

[0004] The embodiments of the present application provide a method, system, terminal device and storage medium for synchronizing audio and video playback, which can ensure that each client participating in the collaborative mode can smoothly and efficiently perform audio and video synchronous playback, thereby promoting the smoothness of collaboration.

[0005] In a first aspect, an embodiment of the present application provides an audio and video playback synchronization method, applied to a client, comprising:

[0006] When detecting that an audio or video file is inserted into the collaborative platform, initializing a local player element, and displaying the local player element in a canvas mode on the user interface of the client;

[0007] When a selection operation initiated for the local player element is detected, a playback control component is loaded and displayed on the local player element;

[0008] When a playback control instruction sent by the user through the playback control component is detected, the local player graphic element is displayed in the document object model mode on the user interface;

[0009] If the client is in the collaborative follow-up mode, the playback control instruction is synchronously submitted to the server, so that the server forwards the playback control instruction to other clients participating in the collaborative platform to achieve audio and video playback synchronization.

[0010] In a possible implementation of the first aspect, displaying the local player graphic element in the user interface in a document object model mode includes:

[0011] Determining a rendering area in a document object model mode according to a position of the local player primitive in the canvas mode;

[0012] An interactive media player component is dynamically loaded in the rendering area.

[0013] In a possible implementation of the first aspect, the method further includes:

[0014] If the client is in the collaborative follow-up mode, it receives the synchronized playback control instruction forwarded by the server in real time;

[0015] Based on the to-be-synchronized playback control instruction, the playback status and playback progress of the local player graphic element are updated.

[0016] In a possible implementation of the first aspect, the method further includes:

[0017] Receiving a calibration instruction broadcast by the server based on a heartbeat mechanism;

[0018] According to the calibration instruction, the playing state and playing progress of the local player graphic element are adjusted.

[0019] In a possible implementation of the first aspect, the method further includes:

[0020] Maintaining an operation instruction queue locally on the client, wherein the operation instruction queue is used to cache playback control instructions to be executed according to a reception time before the inserted audio and video file is loaded;

[0021] After the audio and video files are loaded, the buffered playback control instructions are executed in queue order.

[0022] In a possible implementation of the first aspect, when the player primitive is in the document object model mode, the method further includes:

[0023] Monitoring and processing scaling instructions and / or movement instructions for the player primitives;

[0024] Monitor and process user drawing instructions on the collaborative platform.

[0025] In a second aspect, an embodiment of the present application provides an audio and video playback synchronization system, including: a client and a server, wherein:

[0026] The client is configured to, upon detecting the insertion of an audio or video file into the collaborative platform, initialize a local player element and display the local player element in canvas mode on the user interface of the client; upon detecting a selection operation initiated on the local player element, load and display a playback control component on the local player element; upon detecting a playback control instruction sent by a user through the playback control component, display the local player element in document object model mode on the user interface; and if the client is in collaborative follow-up mode, synchronously submit the playback control instruction to the server;

[0027] The server is used to forward the playback control instruction to other clients participating in the collaborative platform.

[0028] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the audio and video playback synchronization method as described in the first aspect above is implemented.

[0029] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio and video playback synchronization method as described in the first aspect above is implemented.

[0030] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a terminal device, enables the terminal device to execute the audio and video playback synchronization method as described in the first aspect above.

[0031] In an embodiment of the present application, when the client detects that an audio or video file is inserted into the collaborative platform, in the initial stage, the player element is inserted in the form of a picture in canvas mode (i.e., canvas mode), which effectively reduces the interface rendering pressure and avoids starting the complex audio and video playback engine and related resource loading when not in use. It can effectively reduce the system resource usage and improve the system's response speed and overall performance. When the user selects the player element, the display playback control component is activated again. By delaying the loading of the player function module, interface redundancy is avoided while optimizing client performance. When the user triggers the playback control instruction, it switches from canvas mode to document object model mode (i.e., dom mode) and loads the player function to ensure smooth playback. For clients in collaborative follow-up mode, the playback control instruction is synchronously forwarded to other clients participating in the collaborative platform through the server, so that each client participating in the collaborative platform can smoothly and efficiently perform audio and video synchronous playback, thereby promoting the smoothness of collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 This is a system architecture diagram of the audio and video playback synchronization system provided by an embodiment of the present application;

[0034] Figure 2 This is a schematic diagram of the architecture of a client in the audio and video playback synchronization system provided by an embodiment of the present application;

[0035] Figure 3 This is a flowchart of the implementation of the audio and video playback synchronization method provided in the embodiment of the present application;

[0036] Figure 4 This is a specific implementation flow chart of displaying local player graphics elements in DOM mode in the audio and video playback synchronization method provided in an embodiment of the present application;

[0037] Figure 5 This is a specific implementation flowchart of synchronizing the audio and video playback status and playback progress of other clients in the audio and video playback synchronization method provided in an embodiment of the present application;

[0038] Figure 6 This is a specific implementation flowchart of calibrating the playback status and playback progress of the local player element in the audio and video playback synchronization method provided in the embodiment of the present application;

[0039] Figure 7It is a schematic diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0041] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0042] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0043] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0044] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0045] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0046] The audio and video playback synchronization method provided in the embodiments of the present application is applicable to various types of terminal devices that can participate in the collaborative mode and support audio and video playback. The terminal devices may specifically include mobile phones, tablet computers, wearable devices, laptop computers, ultra-mobile personal computers (UMPCs), desktop computers, and interactive large screens. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0047] Figure 1 A system architecture diagram of an audio and video playback synchronization system provided in an embodiment of the present application is shown, which is detailed as follows: For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0048] Reference Figure 1 The audio and video playback synchronization system includes a client 1 and a server 2, wherein:

[0049] Client 1 is the interface and tool for users to interact with the collaborative platform. There can be multiple clients 1, which are used to detect, receive and execute the user's playback control instructions. At the same time, client 1 is also responsible for processing and displaying the data returned by server 2, so that users can intuitively see the results of collaborative work. Server 2 is the background support of the collaborative platform. Server 2 is used to forward playback control instructions to multiple clients 1 participating in the collaborative platform. After receiving the playback control instructions from client 1, server 2 processes them according to the rules of the collaborative platform, and then feeds back the processing results to the corresponding client 1 to ensure that the data and operation status of all clients 1 remain consistent. The functions of the collaborative platform are realized through the collaboration of client 1 and server 2. The collaborative platform can be a collaborative whiteboard, or it can be other collaborative tools with a canvas interface and support for audio and video insertion, such as a remote conferencing system and an online education platform.

[0050] In online education scenarios, synchronized audio and video playback between teachers and students helps students keep up with the teacher's teaching pace and enhance learning outcomes. In remote conference scenarios, synchronized playback ensures that all participants can watch and discuss the same audio and video materials at the same time, improving communication efficiency, avoiding information bias caused by inconsistent playback, and making meeting discussions more targeted and efficient. For team project discussions, members can simultaneously watch project-related demonstration videos, analyze audio and other materials, discuss and make decisions based on the same progress, promote smooth team collaboration, and improve project advancement efficiency.

[0051] In the embodiment of the present application, the audio and video playback synchronization system can use Bluetooth technology, Wi-Fi technology, or 3G / 4G / 5G technology to establish a wireless connection between the server 2 and multiple clients 1. Alternatively, it can use serial port technology or USB interface technology to establish a wired connection between the server 2 and multiple clients 1. Clients 1 include but are not limited to mobile phones, tablets, wearable devices, laptops, desktop computers, interactive large screens, and other terminal devices with communication capabilities. Multiple clients 1 can connect to the server 2 via wired or wireless means.

[0052] In the collaborative platform scenario of the embodiment of the present application, the client 1 can be set to personal mode and collaborative mode, and the collaborative mode includes collaborative follow mode and collaborative non-follow mode. In personal mode, the client 1 only serves the operations of a single user. The user can insert audio and video files, control playback, and perform other operations on the terminal device. All operations are only effective on the local client and will not interact with other clients. For example, a user can load a teaching video on the terminal device alone, freely pause, fast forward, and rewind, draw on the terminal device, add annotations, etc. These operations will not affect other users, and will not receive synchronization information from other clients.

[0053] When client 1 is in collaborative follow-up mode, it needs to maintain a high degree of synchronization with other clients participating in the collaborative platform. The user's local client audio and video playback control commands, such as play, pause, and adjust progress, will be immediately submitted to server 2. Server 2 then forwards these playback control commands to other clients, ensuring that the audio and video playback status and playback progress of all clients in collaborative follow-up mode are consistent. For example, in a multi-person online training, the training instructor performs video playback operations on his or her client, and the trainee clients in collaborative follow-up mode will perform the same operations synchronously, ensuring that everyone sees exactly the same video content and progress, facilitating collective learning and discussion.

[0054] In collaborative non-follow mode, although Client 1 participates in the collaborative platform environment, the user has a certain degree of autonomy in controlling audio and video playback. Users can perform audio and video playback control operations, and these operations will not be automatically synchronized to other clients. In collaborative non-follow mode, users can still see some basic collaborative operations of other clients on the platform, such as drawing content and adding text, but each Client 1 can independently control the audio and video playback status according to its own needs. For example, in a creative discussion meeting, participants can jointly draw mind maps and share ideas on the collaborative platform, while each person decides the progress of audio and video playback to meet the differences in people's understanding and viewing speed of content.

[0055] As a possible implementation of the present application, the server 2 is further configured to broadcast calibration instructions to the clients participating in the collaborative platform based on a heartbeat mechanism. The client 1 is further configured to adjust the playback status and playback progress of the local player element according to the calibration instructions.

[0056] The heartbeat mechanism is a strategy used to ensure a valid connection between server 2 and client 1. Server 2 periodically sends heartbeat signals to client 1, and client 1 responds with a confirmation message. This allows server 2 to know in real time whether the client is online. If server 2 does not receive a response from client 1 within a certain period of time, it can be determined that client 1 is offline.

[0057] In the actual collaborative platform scenario, due to factors such as network delay and device performance differences, the audio and video playback progress of different clients may gradually deviate. For example, some clients play a little faster, while others play a little slower. In an embodiment of the present application, server 2, based on the heartbeat mechanism, broadcasts calibration instructions to all clients 1 participating in the collaborative platform at a specific time (for example, at a preset time interval). By broadcasting the calibration instructions, each client 1 is allowed to use server 2 as a benchmark to adjust the playback status (such as play, pause, fast forward, fast rewind, etc.) and playback progress of the client's local audio and video, thereby ensuring that the audio and video playback of all clients remains synchronized, avoiding confusion in collaborative work due to inconsistent playback progress. For example, in a multi-person online meeting, everyone watches a teaching video together. Through the calibration instructions broadcast by the server based on the heartbeat mechanism, the client of each participant can maintain consistency in video playback, so that everyone can be based on the same playback progress when communicating and discussing video content, greatly improving the efficiency and experience of collaborative work.

[0058] As a possible implementation of the present application, server 2 is further configured to provide the playback status and progress of player elements to clients newly joining the collaborative platform. After a new client joins the collaborative platform, if the new client is in collaborative follow-up mode, the playback status and progress of the player elements are synchronized from server 2.

[0059] As a possible implementation of the present application, the client 1 also maintains an operation instruction queue locally, which is used to cache the playback control instructions to be executed according to the receiving time before the audio and video files inserted into the collaborative platform are loaded; when the audio and video files are loaded, the cached playback control instructions are executed in queue order.

[0060] In one possible implementation, Figure 2As shown, client 1 includes a canvas engine and a rendering user interface engine. The insertion and playback of audio and video files locally on client 1 are jointly completed by the canvas engine and the rendering user interface engine. The canvas engine is responsible for the generation of player graphics, the storage of information, and the drawing of graphics element canvases. The rendering user interface engine is responsible for receiving information from the canvas engine and rendering the canvas into a document object model element at a specific moment. Among them, the rendering user interface engine contains two classes: the document object model container management class and the player class. The document object model container management class is responsible for receiving messages from the canvas engine and executing the creation, destruction, and hiding of the player document object model according to the messages. After the player is created, the message is forwarded to the corresponding player component to update the view state of the player component. The player class is used to create various types of players, such as iframe, html video, html audio, etc.

[0061] In one possible implementation, server 2 assists client 1 in achieving synchronization of audio and video playback on multiple clients. Server 2 includes a collaborative service module, which includes a canvas engine for generating player graphics elements, storing information, and drawing graphics element canvases on server 2. The collaborative service module runs independently on the server and serves as a benchmark for all clients. It forwards and synchronizes the audio and video files and playback control instructions of each client participating in the collaborative version to ensure the consistency of the canvas content seen by each client.

[0062] In one possible implementation, a special primitive is used with two drawing modes: canvas mode and document object model (DOM) mode. In the inactive state, the primitive is drawn on the canvas as an image. In the active state, the user interface rendering engine generates a corresponding DOM element based on the received message content and displays it on top of the canvas. Rendering DOM elements only in specific states significantly reduces platform canvas performance when processing other operations, improving the user experience.

[0063] In one possible implementation, the player primitive is initialized through the canvas engine. To reduce the burden of interface rendering, the player primitive is in canvas mode by default. At this time, the player primitive is only inserted into the platform canvas as a picture. The content source of the picture is the real document object model element of the audio and video when the audio and video file is first inserted. The real document object model element of the audio and video is rendered under the platform canvas document object model mode, and the thumbnail generated at this time is submitted to the canvas engine for storage. The thumbnail can be directly used to insert the canvas during subsequent rendering without the need for re-rendering. When the client 1 is in the collaborative mode following state, the playback control instruction detected by the client 1 will be submitted to the server 2, that is, the drawing mode of the player primitive on the server 2 will be updated to the document object model mode, and then forwarded to other clients by the server 2.

[0064] In one possible implementation, when the client detects that an audio or video file is inserted into the collaborative platform, in the initial stage, the player element is inserted in the form of a picture in canvas mode to avoid starting a complex audio and video playback engine and loading related resources when not in use, which can effectively reduce the occupation of system resources and improve the response speed and overall performance of the system. When the user selects the player element, the display playback control component is activated and the player function module is loaded with delay to avoid interface redundancy while optimizing client performance. When the user triggers the playback control instruction, the canvas mode is switched to the document object model mode and the player function is loaded to ensure smooth playback. For clients in collaborative follow-up mode, the playback control instruction is synchronously forwarded to other clients participating in the collaborative platform through the server, so that each client participating in the collaborative platform can smoothly and efficiently play audio and video synchronously, thereby promoting the fluency of collaboration.

[0065] In one possible implementation, Figure 3 The implementation process of the audio and video playback synchronization method provided by the embodiment of the present application is shown. The execution end of the method process is Figure 1 The method comprises steps S301 to S304. The specific implementation principle of each step is as follows:

[0066] Step S301: when it is detected that an audio or video file is inserted into the collaborative platform, a local player element is initialized and displayed in a canvas mode on the user interface of the client.

[0067] The client continuously monitors platform operation events, detecting the insertion of audio and video files into the collaborative platform in real time. Once a new audio or video file is added to the collaborative platform environment, it immediately initializes the local player element. The local player element refers to the player element local to the client and is the core element that supports audio and video playback. Initializing the local player element essentially initializes the relevant audio and video playback parameters, including memory space, initial volume, and playback mode. It also involves allocating system resources to the local player element.

[0068] Canvas mode is a specific presentation method characterized by providing a relatively simple and unified display space. In Canvas mode, the local player element is simply inserted into the platform canvas as an image and displayed as a static element on the client user interface, acting as a placeholder for audio and video on the canvas. In Canvas mode, the player element display is not overly complex, which not only reduces initial interface rendering pressure, but also avoids distracting users from the overall platform content.

[0069] In an embodiment of the present application, when the client detects that an audio or video file is inserted into the collaborative platform, it initializes the local player element and builds the basic operating environment of the player element to prepare for subsequent audio and video playback. At the same time, the local player element is inserted into the platform canvas in the form of a picture to reduce the interface rendering pressure and avoid starting the complex audio and video playback engine and related resource loading when not in use. It can effectively reduce the occupation of system resources in the initial stage and improve the response speed and overall performance of the system.

[0070] Step S302: When a selection operation initiated on the local player element is detected, a playback control component is loaded and displayed on the local player element.

[0071] Before the client detects that the local player element has been selected, the local player element is rendered as a static element. After the client detects that the local player element has been selected, the playback control component is loaded and displayed. The playback control component is used to control audio and video playback.

[0072] The playback control component is a collection of functional buttons or interface elements used to control the playback status of audio and video. The playback control components include buttons such as play, pause, fast forward, fast rewind, volume adjustment, and a progress bar. When a selection operation is detected for the local player element, the selection of the local player element triggers the update of its selected state isActive to true. At the same time, the client calls these playback control components from the resource library and loads them into the interface of the local player element for display. For example, the play button is displayed on top of the local player element image.

[0073] In one possible implementation, the user selects a local player element by mouse click, touch operation, etc., and the client detects whether a selection operation has occurred on the local player element by detecting mouse click events and touch events on the area where the player element is located.

[0074] In one possible implementation, the selection of the local player icon can be automatically triggered by the system. Upon detecting a specific selection trigger event, the client automatically selects the local player icon and displays the playback control components. For example, when an audio or video file is loaded, the system automatically selects the player icon and displays a play button.

[0075] The loading process of the playback control component involves the layout and rendering of the component. In the embodiment of the present application, the playback control component is loaded and displayed only when it is detected that the user has an operation requirement. This not only reduces the pressure of component layout and rendering and alleviates bandwidth pressure, but also through this on-demand loading method, the client can allocate system resources more reasonably, avoid resource waste, and ensure the stability and smoothness of the entire collaborative platform system during operation. At the same time, it also avoids the playback control component occupying too much interface space when the player element is not selected, which can effectively maintain the simplicity of the user interface and improve the user experience.

[0076] Step S303: When a playback control instruction sent by the user through the playback control component is detected, the local player graphic element is displayed in the user interface in a document object model mode.

[0077] In an embodiment of the present application, a user initiates a playback control command through the playback control component. The types of playback control commands include, but are not limited to, play commands, pause commands, fast-forward commands, fast-rewind commands, and volume increase and decrease commands. When the client captures the playback control command, it identifies and parses the type of the playback control command. For example, when the user clicks the "Play" button, the client recognizes and parses it as a command to start playback; when the user clicks the "Fast Forward" button, the client recognizes and parses it as a command to speed up playback.

[0078] The Document Object Model (DOM) is a tree-like structure used to represent and manipulate HTML and XML documents. When the client detects playback control commands, it switches the display mode of the local player element from canvas mode to DOM mode. In DOM mode, the player element is activated as a DOM element, and its attributes and styles can be dynamically manipulated and modified using scripting languages like JavaScript.

[0079] Compared to canvas mode, the Document Object Model (DOM) mode offers richer interactive capabilities and allows developers precise control over the layout and style of elements. In DOM mode, clients can precisely lay out the position, size, color, transparency, and other attributes of player elements based on design requirements, helping to achieve a more aesthetically pleasing player display that better matches the overall user interface style. Precise layout control also ensures the proper positioning and spacing between player elements and other elements, avoiding overlap or layout clutter.

[0080] In Document Object Model mode, the local player primitive can respond to playback control commands in real time, interact with the user, and promptly update its status and displayed content. For example, when the user clicks the "Pause" button, the player primitive can immediately stop playback and update the playback progress bar; when the user adjusts the volume, the player primitive can adjust the audio output volume in real time. This real-time response and dynamic update capability provides users with timely feedback during operation, enhancing the smoothness and realism of operation.

[0081] As a possible implementation of this application, Figure 4 A specific implementation process of displaying the local player graphics element in the document object model mode on the user interface in the audio and video playback synchronization method provided by an embodiment of the present application is shown, and is detailed as follows:

[0082] A1: Determine the rendering area in the document object model mode according to the position of the local player element in the canvas mode, wherein the position of the local player element in the canvas mode is determined according to the interface layout rules of the canvas mode.

[0083] In collaborative platform scenarios, the local player element is initially displayed in the client user interface in canvas mode. Canvas mode has its own specific coordinate system and layout rules, and the local player element occupies a specific position in canvas mode. When the local player element switches to Document Object Model mode for display, the system determines its corresponding rendering area in Document Object Model mode based on the position of the local player element in canvas mode. The rendering area is used to define the rendering range, allowing the system to render only that area without having to fully render the entire interface. Ensure that the position of the player element is coherent and consistent in different modes.

[0084] Exemplarily, by recording the upper left corner coordinates, width and height of the player element in canvas mode, combined with the interface layout rules of the document object model mode, the rectangular area occupied by the player element in the document object model mode is calculated, and the rectangular area is determined as the rendering area.

[0085] A2: Dynamically load an interactive media player component in the rendering area. The media player component is responsible for the actual playback of media content such as audio and video files, including decoding audio and video data, rendering the video screen to the corresponding rendering area, and outputting audio.

[0086] The rendering area defines a specific display space for the media player component. When the media content of the media player component changes, such as playback progress update or screen switching, only the relevant content within the rendering area needs to be updated, reducing unnecessary rendering overhead, improving rendering efficiency, and saving system resources.

[0087] In an embodiment of the present application, once the rendering area of a local player primitive is determined in the Document Object Model (DOM) mode, the system dynamically loads interactive media playback components into that rendering area. Dynamic loading means that the media playback components are not all loaded upon initialization. Instead, based on actual needs, only when presentation is required, will the media playback components associated with the audio and video files be loaded into the corresponding rendering area in the DOM mode of the local player primitive. Interactive functions of the media playback components, such as play, pause, fast forward, rewind, and volume adjustment, are implemented through the playback control components of the DOM mode.

[0088] Step S304: If the client is in the collaborative follow-up mode, the playback control instruction is synchronously submitted to the server, so that the server forwards the playback control instruction to other clients participating in the collaborative platform to achieve audio and video playback synchronization.

[0089] The client can be set to personal mode, collaborative follow mode, or collaborative non-follow mode. The client can determine which mode it is currently in by determining the mode status flag or the user's configuration information. The client can switch operating modes by receiving system configuration information update instructions or user mode switching instructions.

[0090] When the client is in collaborative follow mode and detects the playback control instruction sent by the user through the playback control component, the client will encapsulate and process the playback control instruction, and then synchronously submit the encapsulated instruction information to the server. The submitted instruction information usually includes the instruction type (such as play, pause, fast forward, fast rewind, etc.), the timestamp of the instruction, and related parameters (such as the duration of fast forward, etc.) to ensure that the server can accurately understand and process the instruction. After the server receives the instruction information submitted by the client, it will forward the instruction information to other clients in real time based on the list of clients participating in the collaborative platform maintained by itself. Other clients refer to clients in collaborative follow mode. After receiving the playback control instruction forwarded by the server, other clients will parse the instruction and perform corresponding operations, thereby achieving audio and video playback synchronization with the client that initiated the instruction.

[0091] For example, when the first client in the collaborative follow mode detects a "pause" command initiated by the user, it submits the "pause" command to the server, and the server synchronously forwards the "pause" command to the second client and the third client participating in the collaborative platform and also in the collaborative follow mode. After receiving the "pause" command, the second client and the third client will also pause the audio and video being played, ensuring that the playback status of all clients in the collaborative follow mode remains consistent.

[0092] As a possible implementation of this application, Figure 5 A specific implementation process of synchronizing the audio and video playback status and playback progress of other clients in the audio and video playback synchronization method provided in an embodiment of the present application is shown, and is detailed as follows:

[0093] B1: If the client is in the collaborative follow-up mode, it receives in real time the to-be-synchronized play control instruction forwarded by the server. The to-be-synchronized play control instruction is a play control instruction submitted to the server by other clients.

[0094] B2: Based on the to-be-synchronized-play control instruction, the play status and play progress of the local player graphic element are updated.

[0095] The local player element is the carrier of audio and video playback. After the client receives the control instruction for synchronized playback, it needs to parse the control instruction to determine the instruction type and playback progress value, and then call the relevant interface or method of the player element to update its playback status (such as starting playback, pausing playback) and playback progress (such as jumping to a specified time point). It should be noted that when updating the playback status and progress, the client needs to ensure that the display of the player element is synchronized with the actual playback status. For example, when the playback status changes from pause to play, the interface display of the player element (such as the change of the play button icon) should also be updated accordingly to provide accurate feedback to the user.

[0096] In an embodiment of the present application, the client updates the playback status and progress of the local player element by receiving the playback control instructions to be synchronized forwarded by the server in real time, ensuring the synchronization of the client's playback with other collaborative users. Regardless of the playback control operations performed by other clients, the client in collaborative follow-up mode can adjust its own playback status in a timely manner to achieve audio and video playback consistency for all clients. By accurately updating the playback status and progress of the local player element and synchronously displaying it on the player element, the user is provided with clear and accurate playback information, without the need for the user to manually adjust the playback progress, which improves the user experience and makes collaborative operations smoother and more natural.

[0097] As a possible implementation of this application, Figure 6 A specific implementation process for calibrating the playback status and playback progress of a local player element in the audio and video playback synchronization method provided in an embodiment of the present application is shown, and is detailed as follows:

[0098] C1: Receive the calibration instruction broadcast by the server based on the heartbeat mechanism.

[0099] The heartbeat mechanism ensures the stability and real-time nature of the connection between the server and client. The server broadcasts calibration instructions based on the heartbeat mechanism, enabling frequent and timely exchange of calibration instructions between the server and client. Even in the face of network fluctuations, calibration instructions can be delivered to the client in a relatively short period of time, ensuring the system's real-time requirements for playback status and progress adjustments. This not only confirms the connection between the client and server, ensuring that all clients receive the same calibration instructions at the same time, but also enables playback progress calibration, ensuring consistent and synchronized information transmission.

[0100] C2: According to the calibration instruction, adjust the playing state and playing progress of the local player graphic element.

[0101] The calibration command includes relevant information for adjusting the playback status and progress of the local player element, such as the target playback time point, playback speed adjustment parameters, playback status switching flag, etc. Based on the calibration command, the client adjusts the playback status and playback progress of the local player element.

[0102] In one possible implementation, the playback status and progress of the local player element are calibrated based on the calibration instruction, and a smooth transition to the target progress is achieved through an interpolation algorithm. For example, the current playback progress of the local player element is obtained, and the current playback progress is numerically compared with the progress in the calibration instruction. If there is a difference between the two, the playback progress of the local player element is adjusted, thereby achieving synchronization of playback progress and status.

[0103] In one possible implementation, a synchronization buffer threshold is set, and the synchronization buffer interval is the maximum synchronization deviation value allowed. The synchronization buffer interval can be customized according to device performance, system configuration, and user needs. The current playback progress is compared with the progress in the calibration instruction. If the difference between the two does not exceed the synchronization buffer threshold, it is determined to be normal synchronization. If the difference between the two is greater than the synchronization buffer threshold, the playback progress of the local player element needs to be adjusted. For example, the synchronization buffer interval is set to 5s, which means that the playback progress difference within 5s is considered a normal deviation. If the difference between the current playback progress of the local player element and the progress in the calibration instruction is greater than 5s, the playback progress of the local player element is adjusted.

[0104] When multiple clients are playing the same audio or video file simultaneously, the playback progress of each client may deviate due to factors such as network latency and device performance. Calibration instructions provide a unified playback progress standard for all clients, enabling each client to precisely adjust the playback status and progress of its local player elements based on the calibration instructions. This ensures consistent playback progress across multiple clients and synchronizes audio and video playback. This effectively avoids playback asynchrony caused by factors such as network latency and device performance differences, and improves the consistency and fluidity of the collaborative experience.

[0105] As a possible implementation of the present application, an operation instruction queue is maintained locally on the client, and the operation instruction queue is used to cache the playback control instructions to be executed according to the receiving time before the inserted audio and video file is loaded; when the audio and video file is loaded, the cached playback control instructions are executed in queue order.

[0106] In the embodiment of the present application, to avoid timing conflicts, before the client executes the cached playback control instructions in queue order, the client merges the playback control instructions of the same type or conflicting ones in the operation instruction queue and deletes redundant instructions. By merging instructions, the number of execution times of the instructions is reduced, and the playback speed is accelerated.

[0107] For playback control instructions of the same type, they can be directly merged into one playback control instruction based on the instruction parameters. For example, before the audio and video are loaded, the user issues multiple fast-forward instructions in a short period of time. The length of each fast-forward is different. The system will accumulate the lengths of these fast-forward instructions cached in the operation instruction queue and merge them into one fast-forward instruction. After the multiple fast-forward instructions are merged, the player element only needs to perform one fast-forward operation. For example, the user first issues a command to fast-forward 10 seconds, and then issues a command to fast-forward 20 seconds. The result is a command to fast-forward 30 seconds.

[0108] Conflicting playback control commands are merged according to certain rules. For example, the command with the later command reception timestamp is used for merging. For example, if the cached operation command queue includes a play command and a pause command, if a play command is received first and then a pause command is received, the merged command becomes the pause command.

[0109] In an embodiment of the present application, the client can merge and process the same type or conflicting instructions by maintaining an operation instruction queue, thereby reducing the number of instruction executions, avoiding operational confusion, and reducing the processing burden of the system, saving processing time and system resources. Correspondingly, it also reduces the amount of data transmitted over the network, reduces the pressure on network transmission, improves the efficiency of data transmission, and helps maintain the stability and smoothness of the collaborative platform in a network environment.

[0110] As a possible implementation of the present application, when multiple player elements exist simultaneously on the collaborative platform, in order to reduce resource consumption, only one player element of the same type will be in the playing state. For example, a first player element and a second player element exist simultaneously on the collaborative platform, and the first player element is in the playing state. When the client detects the play control instruction of the second player element, the play state of the first player element will be updated to pause, and the document object model mode will be switched to the canvas mode, and the first player element will return to the image presentation form. This process is also applicable to the scenario where multiple collaborators click on different players to play in the collaborative mode follow state. In the scenario where there are multiple player elements, the user can only see that one player is in the playing state.

[0111] As a possible implementation of the present application, when the player element is in the document object model mode, a scaling instruction and / or a moving instruction for the player element is monitored and processed, and a drawing instruction of a user on the collaborative platform is monitored and processed.

[0112] When the player element is in the document object model mode, the user can move and scale the player element, and can also perform drawing operations on the collaborative platform while playing audio and video.

[0113] In a possible implementation, the user's drawing instructions on the layer where the player primitive is located are monitored, and the drawing content is rendered in layers to avoid interaction conflicts with the player primitive.

[0114] In an embodiment of the present application, users are allowed to scale, move, and draw player elements while playing audio and video on a collaborative platform, which increases the interaction mode and flexibility between users and the platform, improves user participation and convenience of operation, and thus enhances the overall user experience.

[0115] As can be seen from the above, in an embodiment of the present application, when the client detects that an audio or video file is inserted into the collaborative platform, the local player element is initialized and displayed in canvas mode on the client's user interface, which can reduce the initial rendering burden and facilitate the client to quickly complete the initial loading. When a selection operation initiated for the local player element is detected, a playback control component is loaded and displayed on the local player element to provide the user with a playback control entrance, so that the user can adjust the audio and video playback status and progress. When a playback control instruction sent by the user through the playback control component is detected, the local player element is displayed in document object model mode on the user interface to ensure playback smoothness. When the client is in collaborative follow-up mode, the playback control instruction is synchronously submitted to the server, and the instruction distribution is centrally managed by the server, so that each client participating in the collaborative platform can smoothly and efficiently perform audio and video synchronous playback, thereby promoting the fluency of collaboration.

[0116] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0117] It should be noted that the information interaction, execution process, etc. between the above-mentioned systems / modules are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0118] The embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the following Figures 3 to 6 The steps of any audio and video playback synchronization method are represented.

[0119] The embodiment of the present application further provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, Figures 3 to 6 The steps of any audio and video playback synchronization method are represented.

[0120] The embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the following Figures 3 to 6 The steps of any audio and video playback synchronization method are represented.

[0121] Figure 7 This is a schematic diagram of a terminal device provided by an embodiment of the present application. Figure 7As shown, the terminal device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned embodiments of the audio and video playback synchronization method are implemented, for example Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of the modules / units in the above-mentioned system embodiments are realized, for example, Figure 2 The functions of each module component are shown.

[0122] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to implement the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the terminal device 7.

[0123] The terminal device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that Figure 7 It is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 7 may also include input and output devices, network access devices, buses, etc.

[0124] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0125] The memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. The memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 7. Furthermore, the memory 71 may include both an internal storage unit of the terminal device 7 and an external storage device. The memory 71 is used to store the computer program and other programs and data required by the terminal device. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0126] It should be noted that the information interaction, execution process, etc. between the above-mentioned systems / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0127] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or system capable of carrying the computer program code to the system / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable medium cannot be an electric carrier signal or a telecommunication signal.

[0129] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0130] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for synchronizing audio and video playback, characterized in that: Applied to the client, including: When detecting that an audio or video file is inserted into the collaborative platform, initializing a local player element, and displaying the local player element in a canvas mode on the user interface of the client; When a selection operation initiated for the local player element is detected, a playback control component is loaded and displayed on the local player element; When a playback control instruction sent by the user through the playback control component is detected, the local player graphic element is displayed in the document object model mode on the user interface; If the client is in the collaborative follow-up mode, the playback control instruction is synchronously submitted to the server, so that the server forwards the playback control instruction to other clients participating in the collaborative platform to achieve audio and video playback synchronization.

2. The method according to claim 1, characterized in that The displaying of the local player graphic element in the user interface in a document object model mode includes: Determining a rendering area in a document object model mode according to a position of the local player primitive in the canvas mode; An interactive media player component is dynamically loaded in the rendering area.

3. The method according to claim 1, characterized in that The method further comprises: If the client is in the collaborative follow-up mode, it receives the synchronized playback control instruction forwarded by the server in real time; Based on the to-be-synchronized playback control instruction, the playback status and playback progress of the local player graphic element are updated.

4. The method according to claim 1, wherein The method further comprises: Receiving a playback calibration instruction broadcast by the server based on a heartbeat mechanism; According to the calibration instruction, the playing state and playing progress of the local player graphic element are adjusted.

5. The method according to claim 1, wherein The method further comprises: Maintaining an operation instruction queue locally on the client, wherein the operation instruction queue is used to cache playback control instructions to be executed according to a reception time before the inserted audio and video file is loaded; After the audio and video files are loaded, the buffered playback control instructions are executed in queue order.

6. The method according to claim 1, characterized in that When the player element is in the document object model mode, the method further includes: Monitoring and processing scaling instructions and / or movement instructions for the player primitives; Monitor and process user drawing instructions on the collaborative platform.

7. An audio and video playback synchronization system, characterized in that: The audio and video playback synchronization control system includes a client and a server, wherein: The client is configured to, upon detecting the insertion of an audio or video file into the collaborative platform, initialize a local player element and display the local player element in canvas mode on the user interface of the client; upon detecting a selection operation initiated on the local player element, load and display a playback control component on the local player element; upon detecting a playback control instruction sent by a user through the playback control component, display the local player element in document object model mode on the user interface; and if the client is in collaborative follow-up mode, synchronously submit the playback control instruction to the server; The server is used to forward the playback control instruction to other clients participating in the collaborative platform to achieve audio and video playback synchronization.

8. The system according to claim 7, characterized in that The server is further configured to broadcast calibration instructions to clients participating in the collaborative platform based on a heartbeat mechanism; The client is further configured to adjust the playback status and playback progress of the local player graphic element according to the calibration instruction.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the audio and video playback synchronization method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the audio and video playback synchronization method according to any one of claims 1 to 6 is implemented.

Images