Method and device for synchronously playing multimedia on multiple terminals and multimedia server

Abstract

The invention discloses a method for synchronously playing multimedia at multiple terminals, which comprises the following steps: a multimedia server receives an operation instruction sent by first terminal equipment, and the first terminal equipment generates the operation instruction and sends the operation instruction to the multimedia server after receiving an operation executed by a user on a multimedia user interface; the multimedia server processes the operation instruction and sends the processed operation instruction to a predetermined playing device; the multimedia server receives an operation execution result returned by the playing device; and the multimedia server detects the currently registered terminal equipment and synchronizes the operation execution result to all the detected registered terminal equipment, so that all the registered terminal equipment realizes the operation execution result consistent with that of the playing equipment. The invention also discloses a computer device and a multimedia server. According to the invention, synchronous playing of the multimedia content in each device can be realized.

Description

Technical Field

[0001] This invention relates to the field of multimedia playback technology, and in particular to a method, apparatus, program product, and multimedia server for synchronous playback of multimedia on multiple terminals. Background Technology

[0002] With the advancement and continuous development of network technology, network bandwidth has gradually increased, and data transmission speeds have significantly improved, enabling the rapid transmission of multimedia data between different terminals. For example, with the widespread adoption of various smart terminals such as smartphones, tablets, and smart TVs, multi-terminal, multi-media playback synchronization technology has been widely applied. Users expect to be able to seamlessly switch between watching video, audio, and other multimedia content on different devices while maintaining synchronized playback progress.

[0003] In one existing technology, playback on multiple terminals is controlled via a server. For example, the player receives time information transmitted from multiple display terminals, calibrates the time information of each terminal based on the system time, and ensures that the calibrated time of each terminal is consistent. Then, based on the system time, it determines the push time for the video to be played to each terminal, and further determines the download time for each terminal to download and play the video, thus achieving synchronized playback. This technical solution is suitable for scenarios with high requirements for synchronized playback on multiple terminals, such as large-scale video conferencing systems and multi-terminal synchronized learning on online education platforms, where there are many terminals and centralized management is required. However, this technical solution is highly dependent on server performance and network stability; if the server malfunctions or there is significant network latency, the synchronization effect will be affected.

[0004] Another prior art discloses a master-slave terminal mode technical solution. First, a reference timestamp is marked on the source content to ensure that the terminal can accurately reconstruct the reference timestamp for each frame of audio and video data after receiving the data. One master terminal and multiple slave terminals are set up. After the master terminal starts playback, it reports the current playback progress and status information to the synchronization server. When a slave terminal starts playback, it first retrieves the current playback progress and status information of the master terminal from the synchronization server, and simultaneously retrieves the media stream. Then, it starts playback according to the data offset set to synchronize with the master terminal. This technical solution is commonly used in scenarios such as home multimedia systems and small conference rooms, where one device acts as the master device to control playback, and other devices act as slave devices to follow the master device's playback. In this technical solution, the performance and stability of the master terminal have a significant impact on the entire synchronized playback process; if the master terminal malfunctions, the playback of the slave terminals will also be affected. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for synchronous playback of multimedia on multiple terminals, which enables synchronous playback of multimedia content on multiple terminals, including user terminals and playback device terminals.

[0006] According to a first aspect of the present invention, a method for synchronous playback of multimedia on multiple terminals is provided, comprising: The multimedia server receives an operation instruction sent to a first terminal device that has been registered with it, wherein the first terminal device generates an operation instruction and sends it to the multimedia server after receiving an operation performed by a user on the multimedia user interface. The multimedia server processes the received operation instructions and then sends the processed operation instructions to a predetermined playback device. The multimedia server receives the operation execution result returned by the playback device after performing the corresponding operation according to the received operation instruction; The multimedia server detects currently registered terminal devices and synchronizes the operation execution results to all detected registered terminal devices, so that all registered terminal devices achieve the same operation execution results as the playback device.

[0007] In a further embodiment, the multimedia server processes the received operation instruction and sends the processed operation instruction to a predetermined playback device, including: if the operation instruction sent by the first terminal device is a control instruction, the multimedia server forwards the control instruction to the playback device; and if the operation instruction sent by the first terminal device is a streaming media data operation instruction, the multimedia server generates a streaming media data operation instruction containing a multimedia file URL address and sends the streaming media data operation instruction to the playback device.

[0008] In an optional embodiment, the method further includes the following steps: when the operation instruction is a request to play a video, the multimedia server sends a video playback instruction containing a video URL address to the playback device; when the playback device retrieves video data from the playback device based on the video URL address received in the video playback instruction, the multimedia server pushes the video data to the playback device for playback; and the multimedia server receives a start-up instruction sent by the playback device after video playback, the start-up instruction containing a multimedia ID and video playback position information; the multimedia server sends the start-up instruction to all registered terminal devices; after receiving a request for a specific video URL address from a registered terminal device based on the multimedia ID, the multimedia server sends the video URL address corresponding to the multimedia ID to the terminal device, so that the terminal device retrieves video data from the multimedia server based on the video URL address and synchronizes multimedia playback according to the start-up instruction.

[0009] In an optional embodiment, the method further includes the following steps: when the operation instruction is to request audio playback, the multimedia server sends an audio playback instruction containing an audio or video URL address to the playback device; when the playback device retrieves audio or video data from the playback device based on the audio or video URL address received in the audio playback instruction, the multimedia server pushes the audio or video data to the playback device, wherein the playback device suspends playback after receiving the audio or video data under the audio playback instruction; when the playback device requests a lyrics file, the multimedia server sends a lyrics file to the playback device; the multimedia server receives a start playback instruction sent by the playback device, wherein the playback device cancels the suspension and starts playing the audio or video after obtaining the lyrics file, and renders the lyrics into the playing audio or video, wherein the start playback instruction contains a multimedia ID and audio or video playback position information; The multimedia server sends the start-up command to all registered terminal devices. After receiving a request from a registered terminal device for a specific audio or video URL address and lyrics file URL address based on the multimedia ID, the multimedia server sends the audio or video URL address and lyrics file URL address corresponding to the multimedia ID to the terminal device. This allows the terminal device to obtain the corresponding audio or video and lyrics from the multimedia server based on the audio or video URL address and lyrics file URL address, and to synchronize the playback of the audio or video and the rendering of the lyrics according to the start-up command.

[0010] In an optional embodiment, the method further includes the multimedia server requesting playback progress from the playback device when it receives a request for playback progress from a newly registered terminal device during multimedia playback; after receiving the playback progress instruction sent by the playback device, the multimedia server sends the playback progress instruction to the newly registered terminal device so that the newly registered terminal device and other registered terminal devices can play multimedia content synchronously, wherein the playback progress instruction includes a multimedia ID and multimedia playback position information.

[0011] In a further embodiment, the newly registered terminal device requests a specific multimedia file URL address from the multimedia server based on the multimedia ID. The multimedia server returns the multimedia file URL address to the newly registered terminal device, thereby enabling the newly registered terminal device to obtain multimedia content from the multimedia server and achieve synchronized playback with other terminal devices and playback devices according to playback progress instructions.

[0012] In an optional embodiment, the first terminal device and the playback device are configured in a first registration group. Upon receiving a registration request from a new terminal device, the multimedia server determines whether to add the new terminal device to the first registration group. The multimedia server synchronizes the execution result of the operation instruction to all registered terminal devices, including synchronizing the execution result of the operation instruction to terminal devices in the same registration group. In an optional embodiment, all registered terminal devices include two or more terminal devices, and the two or more terminal devices include the first terminal device.

[0013] In optional embodiments, the terminal device includes a mobile phone and a tablet computer, the multimedia user interface includes the user interface of a karaoke program; and / or, the playback device includes a speaker, a television, a smart home screen, a touch screen speaker, or a vehicle-mounted system.

[0014] According to a second aspect of the present invention, a multimedia server is provided, comprising: a processor, a memory, and a computer program stored in the memory, the processor being configured to perform the method steps of the embodiments described above by executing the computer program.

[0015] According to a third aspect of the present invention, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method steps of the embodiments described above.

[0016] According to a fourth aspect of the present invention, a method for synchronous playback of multimedia on multiple terminals is provided, comprising: a playback device receiving a multimedia playback instruction sent by a multimedia server; wherein the multimedia server generates a playback instruction containing a multimedia file URL address when a first terminal device requests playback of specific multimedia; the playback device obtains a corresponding multimedia data stream from the multimedia server according to the multimedia file URL address for playback; after the playback device starts playing the multimedia, it sends a start-up instruction to the multimedia server, so that the multimedia server forwards the start-up instruction to all terminal devices registered with it; the start-up instruction contains a multimedia ID and multimedia content playback position information, such that all terminal devices registered with it obtain specific multimedia data from the multimedia server according to the multimedia ID and synchronously play multimedia content consistent with that of the playback device according to the multimedia content playback position information.

[0017] In an optional embodiment, the method further includes the following steps: when the multimedia playback instruction is an audio playback instruction, the playback device obtains audio or video data from the multimedia server according to the audio or video URL address in the audio playback instruction; after receiving the audio or video data, the playback device suspends playback and requests a lyrics file from the multimedia server; after receiving the lyrics file sent by the multimedia server, the playback device cancels the suspension, starts playing the audio or video, and renders the lyrics into the playing audio or video, wherein the rendering of the lyrics is bound together with the audio or video playback; and the playback device sends a start playback instruction to the multimedia server.

[0018] In an optional embodiment, the method further includes the playback device sending a playback progress instruction to the multimedia server after receiving a request for playback progress from the multimedia server during multimedia playback, so that the multimedia server synchronizes the received playback progress instruction to the newly registered terminal device, wherein the multimedia server requests playback progress from the playback device after receiving a request from the newly registered terminal device for the playback progress of the multimedia being played.

[0019] In the embodiment of the method for synchronous playback of multimedia on multiple terminals described above in this invention, the method further includes requesting playback progress from the playback device when the multimedia server receives a request for playback progress from the first terminal device. The multimedia server receives a playback progress instruction from the playback device and sends the playback progress instruction to the first terminal device, so that the first terminal device can correct the playback progress of the multimedia based on the time difference determined by the first local time when the multimedia content starts playing and the second local time when the playback progress instruction is received, according to a synchronization correction algorithm.

[0020] In an optional embodiment, all registered terminal devices include two or more terminal devices, and the two or more terminal devices include the first terminal device.

[0021] In optional embodiments, the terminal device includes a mobile phone and a tablet computer, the multimedia user interface includes the user interface of a karaoke program; and / or, the playback device includes a speaker, a television, a smart home screen, a touch screen speaker, or a vehicle-mounted system.

[0022] According to a fifth aspect of the present invention, a computer device is provided, including a processor, a memory, and a computer program stored in the memory, the computer device being capable of being disposed in a playback device, the processor being configured to perform the method steps of the embodiments described above by executing the computer program.

[0023] According to a sixth aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the method steps of the embodiments described above.

[0024] In this embodiment of the invention, the operation results performed by the playback device are synchronized to one or more terminal devices held by users through a multimedia server, thereby realizing the synchronization of the playback of multimedia content such as audio, video, and lyrics on the playback device and the playback on the terminal devices held by users. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in some embodiments described in the implementation description of this application are briefly introduced below. Those skilled in the art can also draw other drawings based on the description in the specification, and can obtain other drawings based on the specification and these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of a system architecture for synchronous playback of multimedia on multiple terminals according to an embodiment of the present invention.

[0027] Figure 2 This is a schematic flowchart of a method for synchronous playback of multimedia on multiple terminals according to an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of the operation instruction flow for synchronous playback of multimedia on multiple terminals according to an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram of the operation instruction flow for synchronously playing video on multiple terminals according to an embodiment of the present invention.

[0030] Figure 5This is a schematic diagram of the operation instruction flow for synchronously playing audio, video and lyrics on multiple terminals according to an embodiment of the present invention.

[0031] Figure 6 This is a schematic flowchart of a method for synchronous playback of multimedia on multiple terminals according to another embodiment of the present invention.

[0032] Figure 7 This is a schematic diagram of the synchronization correction process for multimedia playback on multiple terminals according to an embodiment of the present invention.

[0033] Figure 8 This is a schematic diagram of the structure of a multimedia server according to an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be further described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that the embodiments described in the specification are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments described in this application without creative effort are within the scope of protection of the present invention.

[0035] In this patent specification, the terms "comprising" and "including" mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is "connected" to another element, the element can be directly connected to the other element, or it can mean that the element and the other element are interconnected through an intermediate element. The term "connection" as used herein can include wireless connections.

[0036] Those skilled in the art will understand that the audio and video playback-related data involved in this application require user permission or consent when the embodiments of this invention are applied to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0037] In this invention, multimedia, as a carrier of information storage, dissemination, and representation, refers to the integration of multiple media, generally including various media forms such as text information, sound information, graphic information, and image information, and may include video, audio, and lyrics text.

[0038] This invention describes a technical solution for enabling synchronized playback of multimedia content between a playback device and multiple user terminal devices in a multi-user scenario. For the sake of brevity and clarity, the following example uses two terminal devices, one playback device, and one multimedia server to illustrate the technical solution. Figure 1 The diagram illustrates a system architecture for simultaneous multimedia playback across multiple terminals. The system includes a first terminal device 10, a second terminal device 20, a playback device 30, and a multimedia server 40. The first terminal device 10 and the second terminal device 20 each communicate directly with the multimedia server 40, which in turn communicates with the playback device 30. The playback device 30 has a built-in processor module, storage module, and wireless communication module (e.g., a Wi-Fi module), enabling it to connect to the Internet and access online music, online videos, and other content. In this embodiment, the two terminal devices can be a mobile phone and a tablet computer, or two mobile phones or two tablet computers. Those skilled in the art will understand that describing three or more terminal devices is also applicable to this invention. In this embodiment, the playback device can be a speaker, television, smart home screen, touchscreen speaker, or in-vehicle infotainment system. For example, if the playback device is a device with a display screen, it can play audio and video files. If the playback device is a speaker, the user can play music through the speaker and view the corresponding video interface through a mobile phone or tablet computer.

[0039] Figure 2 A schematic flowchart illustrating a method for synchronous playback of multimedia on multiple terminals according to an embodiment of the present invention is shown. Figure 2 As shown, the method includes S201, whereby a multimedia server receives an operation instruction sent to a first terminal device that has been registered thereto, wherein the first terminal device generates the operation instruction and sends it to the multimedia server after receiving an operation performed by a user on a multimedia user interface.

[0040] The method further includes S202, whereby the multimedia server processes the received operation instruction and sends the operation instruction to a predetermined playback device.

[0041] S203, the multimedia server receives the operation execution result returned by the playback device after performing the corresponding operation according to the received operation instruction.

[0042] S204, the multimedia server detects the currently registered terminal devices and synchronizes the operation execution result to all detected registered terminal devices so that all registered terminal devices achieve consistent operation execution results with the playback device.

[0043] In one embodiment, the multimedia server detects that the first terminal device and the second terminal device have registered with it at the current time, and synchronizes the execution result of the operation instruction to the first terminal device and the second terminal device based on the identity information of the first terminal device and the second terminal device. In another embodiment, the multimedia server detects that the first terminal device, the second terminal device, and the third terminal device have registered with it, and then synchronizes the execution result of the operation instruction to the first terminal device, the second terminal device, and the third terminal device. Specifically, the information of each terminal device is stored in the memory of the multimedia server. In the method of this embodiment, it is also possible to add other new terminal devices when existing terminal devices are already playing multimedia synchronously, and the newly added terminal devices play multimedia synchronously with the existing terminal devices.

[0044] The method for synchronous playback of multimedia on multiple terminals proposed in this invention can be applied to scenarios such as video-on-demand, multimedia teaching, and song request / singing. Operation instructions include control instructions and streaming media data operation instructions. If the operation instruction sent by the first terminal device 10 is a control instruction, the multimedia server 40 forwards the control instruction to the playback device 30. If the operation instruction sent by the first terminal device 10 is a streaming media data operation instruction, the multimedia server 40 adds the multimedia file URL address to the operation instruction and then sends the operation instruction to the playback device 30.

[0045] Because song request and singing applications involve frequent interactions and require high synchronization, the following diagram illustrates the operation command flow for synchronized playback of multimedia across multiple terminals, using the song request and singing application scenario as an example. In this scenario, operation commands include switching songs, increasing volume, decreasing volume, muting, unmute, pausing, adding songs, pinning songs, adding and pinning songs, and deleting songs. In this embodiment, these commands that do not require the transmission of streaming media data are classified as control commands. Operation commands also include playing audio, playing video, switching original vocals, and switching backing vocals. In this embodiment, these commands that require the transmission of streaming media data are classified as streaming media data operation commands.

[0046] In a karaoke scenario, a first user picks up a first terminal device and enters the karaoke user interface through a karaoke program. A second user picks up a second terminal device and also enters the karaoke user interface through a karaoke program. In this specification, the karaoke program refers to customized client software, including karaoke software, karaoke apps, and karaoke mini-programs. Specifically, the first terminal device enters the karaoke program user interface after receiving an operation from the first user. The second terminal device enters the karaoke program interface after receiving an operation from the second user. In this case, both the first and second terminal devices have completed registration with the multimedia server. Figure 3This is a schematic diagram of the operation instruction flow for synchronous playback of multimedia on multiple terminals according to an embodiment of the present invention. Figure 3 As shown, methods for synchronous playback of multimedia on multiple terminals include: S301, after receiving the operation performed by the first user on the user interface of the song selection program, the first terminal device sends operation instruction A to the multimedia server. S302, the multimedia server forwards operation instruction A to the designated playback device; S303, The playback device executes operation instruction A received from the multimedia server; S304, return the execution result of operation instruction A to the multimedia server; S305, the multimedia server synchronizes the result of the operation to the first terminal device and the second terminal device; S306, the first terminal device and the second terminal device realize the operation execution result of operation instruction A. In an optional embodiment, the first terminal device and the second terminal device can display the operation execution result of operation instruction A on the karaoke program user interface.

[0047] Taking operation instruction A as an example of adding a song, the first terminal device 10 sends an add song instruction to the multimedia server 40. After receiving the add song instruction from the first terminal device 10, the multimedia server 40 processes it accordingly and then forwards the processed add song instruction to the playback device 30. In an optional embodiment, this processing may involve the multimedia server converting the add song instruction into a communication method with the playback device. The playback device 30 adds the song X indicated in the add song instruction to the song list and returns the operation execution result of adding song X to the song list to the multimedia server 40. The multimedia server 40 synchronizes this operation execution result to the first terminal device 10 and the second terminal device 20, so that song X is present in the song list of the song selection program user interface of the first terminal device 10 and the song X is also present in the song list of the song selection program user interface of the second terminal device 20. Figure 3 In the example, the second terminal device 20 registered with the multimedia server 40 before the first terminal device 10 sent the operation instruction A. In other embodiments, the second terminal device 20 may also register with the multimedia server 40 at any time after the first terminal device 10 sends the operation instruction A. After receiving the operation execution result, the multimedia server 40 will also synchronize the operation execution result to the second terminal device 20, so that the song X will also be in the song list of the song selection program user interface of the second terminal device 20.

[0048] In the embodiments described above, the format of the operation instruction is {"uid": "xxx", "cmd": 0", "data": "xxxxx"}. Here, uid represents the user ID; cmd represents the command; and data represents the relevant data.

[0049] exist Figure 3 In the schematic diagram of the operation instruction flow shown, based on the SOCKET communication protocol, the first terminal device 10 sends the operation instruction to the multimedia server 40, the multimedia server sends the operation instruction to the playback device 30, the playback device 30 executes the operation instruction and returns the result to the multimedia server 40, and then the multimedia server 40 synchronizes the result to each terminal device.

[0050] The following is a flowchart illustrating the operation instructions for synchronously playing multimedia content across multiple terminals, using streaming media data operation instructions as an example. Application scenario: A first user selects a song using a first terminal device. The first user's playlist changes, and the playback device plays the video of the song selected by the first user. At this time, the first user via the first terminal device 10 and the second user via the second terminal device 20 can both see the song video synchronized with the playback device 30 in real time. For example... Figure 4 As shown, when the first terminal device 10 and the second terminal device 20 have registered with the multimedia server 40, the method for synchronously playing the requested song video on multiple terminals includes: S401, the first terminal device 10 requests the multimedia server 40 to obtain the song list.

[0051] The method further includes S402, where the multimedia server 40 returns a song list to the first terminal device 10.

[0052] S403, after the first user selects a song to play from the song list displayed in the user interface of the song selection program of the first terminal device 10, the first terminal device 10 requests the multimedia server to play video A.

[0053] S404, after the first terminal device 10 sends a request to play video A, it suspends the song selection program and waits for playback to start.

[0054] S405, after receiving the instruction to play video A, the multimedia server 40 processes it accordingly, generates a video playback instruction containing the video URL address, and sends it to the playback device 30. In an optional embodiment, the video playback instruction format is: {"song_id": 1000001, "song_name": "xxxxxx", "singer_name":"xxxxxx", "video_url": "xxxxxx", "audio_url":"xxxxxx", "vip": 0,"song_track": 0, "acc_track": 1, "mac": "xxxxxx"}. Wherein, song_id represents the song ID; song_name represents the song name; singer_name represents the singer's name; video_url represents the video URL link; audio_url represents the audio URL link; vip represents membership status; song_track represents the original vocal track; acc_track represents the backing track; and mac represents the MAC address.

[0055] S406, the playback device 30 obtains video data from the multimedia server 40 according to the received video URL address.

[0056] S407, After the playback device 30 completes the decoding of the video data, it begins to play the video.

[0057] S408, the playback device 30 sends a start-up command to the multimedia server 40. In one embodiment, the start-up command includes information such as video playback position information and song ID.

[0058] S409, the multimedia server 40 forwards the received start-up command to the first terminal device 10 and the second terminal device 20.

[0059] S410, the first terminal device 10 and the second terminal device 20 request a specific video URL address from the multimedia server 40 based on the song ID information in the playback start command. For example, the playback start command contains song ID information, and the first and second terminal devices request the video URL address corresponding to the song ID based on the song ID information. In this embodiment of the invention, depending on the application scenario, the multimedia ID can be a song ID, video ID, audio ID, etc.

[0060] S411, the multimedia server 40 sends the video URL address to the first terminal device 10 and the second terminal device 20.

[0061] S412, the first terminal device 10 and the second terminal device 20 obtain video data from the multimedia server 40 according to the video URL address and perform local decoding.

[0062] S413, the first terminal device 10 and the second terminal device 20 synchronously play the video according to the video playback position information in the playback start instruction forwarded by the multimedia server 40. The video playback position information can be identified by a positive number determined based on the total video playback duration and the current playback progress. For example, if the total duration of the video being played is 5 minutes (300,000 milliseconds), and the current playback progress is at the 5th millisecond, 50th millisecond, or 100th millisecond, then 5, 50, or 100 can be used to identify the video playback position information.

[0063] In an optional embodiment of the present invention, during the synchronous video playback of the first terminal device 10, the second terminal device 20, and the playback device 30, if a third terminal device registers with the multimedia server 40, the newly registered third terminal device will request the multimedia server 40 to obtain the song list, the currently playing song video, and the playback progress of the requested song video. The multimedia server 40 then returns the song list and the URL address of the currently playing song video to the third terminal device. Further, the multimedia server 40 will also send a playback progress request to the playback device 30. Upon receiving the playback progress request, the playback device 30 sends a playback progress instruction for the currently playing video to the multimedia server 40. This video playback progress instruction includes information such as the video playback position. Then, the multimedia server 40 forwards the received playback progress instruction to the third terminal device, which retrieves the video data from the multimedia server 40 based on the received song video URL address and performs local decoding. Finally, based on the video playback position information in the playback progress instruction, the third terminal device synchronizes the video playback with the first terminal device 10 and the second terminal device 20.

[0064] In this embodiment of the invention, after the current song video A finishes playing, the playback device 30 will proceed to play the next song video B according to the song order in the song list. At this time, the playback device 30 will send the start instruction for the next song video B to the multimedia server 40. Then, the multimedia server 40 will forward the start instruction to the first terminal device 10, the second terminal device 20, and the third terminal device. Thus, when the next song video B begins playback, the first terminal device 10, the second terminal device 20, and the third terminal device will still play the song video B synchronously with the playback device 30. The steps or program instructions in the method for achieving synchronized video playback across multiple terminals in this embodiment are performed under the SOCKET and HTTP communication protocols.

[0065] The following describes an embodiment of synchronized playback of audio, video, and lyrics across multiple terminals. Application scenario: A first user selects a song via a first terminal device. The first user's playlist changes, and the playback device plays the audio and video of the song selected by the first user. Both the first user (via the first terminal device 10) and a second user (via the second terminal device 10) can see not only the song's audio and video played synchronously with the playback device, but also the lyrics played synchronously with the playback device. Figure 5 This is a schematic diagram of the operation instruction flow for synchronously playing audio, video, and lyrics on multiple terminals according to an embodiment of the present invention. The synchronization of audio, video, and lyrics playback in this embodiment is also performed under the SOCKET and HTTP communication protocols. When the first terminal device 10 and the second terminal device 20 have registered with the multimedia server, the method for synchronously playing multimedia content on multiple terminals includes: S501, the first terminal device 10 requests a song list from the multimedia server 40.

[0066] The method further includes S502, where the multimedia server 40 returns a song list to the first terminal device 10.

[0067] S503, after the first terminal device 10 receives the operation of the first user selecting a song to play from the song list displayed in the song selection program user interface, it requests the multimedia server 40 to play the audio.

[0068] S504, after the first terminal device 10 requests to play audio, it suspends the song selection program and waits for playback to begin.

[0069] S505, after receiving the request to play an audio command, the multimedia server 40 sends the audio playback command, which includes the audio / video URL address, to the playback device 30. In an optional embodiment, the format of the audio playback command is: {"song_id": 1000001, "song_name": "xxxxxx", "singer_name": "xxxxxx", "url":"xxxxxx","vip": 0, "song_track": 0, "acc_track": 1, "mac":"xxxxxx"}. Wherein, song_id represents the song ID; song_name represents the song name; singer_name represents the singer's name; url represents the audio / video address link; vip represents the membership status; song_track represents the original vocal track; acc_track represents the backing track; and mac represents the MAC address.

[0070] S506, the playback device 30 obtains audio / video data from the multimedia server 40 based on the received audio / video URL and decodes it.

[0071] S507, playback device 30 suspends playback after completing the decoding of audio / video data.

[0072] S508, playback device 30 requests lyrics file from multimedia server 40.

[0073] S509, the multimedia server 40 returns the lyrics file data to the playback device 30. In an optional embodiment, the lyrics file is obtained by the playback device 30 from the multimedia server 40 via the HTTP communication protocol. The lyrics file instruction format is: {"song_id": "xxxxxx", "lyric_url": "xxxxxx"}. `song_id` represents the song ID; `lyric_url` represents the lyrics file link.

[0074] S510, after receiving the lyrics data, the playback device 30 begins drawing the lyrics and cancels the suspension, the audio / video begins playing, and the lyrics are drawn on the screen. In this embodiment of the invention, the drawing of lyrics is bound together with the playback of audio / video.

[0075] S511, playback device 30 sends a start playback command to multimedia server 40.

[0076] S512, the multimedia server 40 forwards the received start-up command to the first terminal device 10 and the second terminal device 20. In one embodiment, the start-up command includes information such as audio / video playback location information and song ID.

[0077] S513, the first terminal device 10 and the second terminal device 20 request specific audio / video URL addresses and lyrics URL addresses from the multimedia server 40 based on the song ID information in the start-up instruction.

[0078] S514, the multimedia server 40 sends the audio / video URL address and the lyrics URL address to the first terminal device 10 and the second terminal device 20.

[0079] S515, the first terminal device 10 and the second terminal device 20 obtain audio / video data and lyrics data from the multimedia server 40 according to the audio / video URL address and the lyrics URL address, and perform local decoding.

[0080] S516, the first terminal device 10 and the second terminal device 20 synchronize the playback of audio / video and the drawing of lyrics according to the audio / video playback position information in the start-up command. In this embodiment of the invention, since the drawing of lyrics is bound to the playback of audio / video, the drawing of lyrics is synchronized as long as the audio / video playback is synchronized.

[0081] In an optional embodiment of the present invention, during the synchronous playback of a song / video by the first terminal device 10, the second terminal device 20, and the playback device 30, if a third terminal device registers with the multimedia server 40, the newly registered third terminal device will request a song list, the currently playing audio, and the playback progress of the audio from the multimedia server 40. The multimedia server 40 then returns the song list, the URL address of the currently playing song's audio / video, and the URL address of the lyrics to the third terminal device. Further, the multimedia server 40 also sends a playback progress request to the playback device 30. Upon receiving the playback progress request, the playback device 30 sends a playback progress instruction for the currently playing audio / video to the multimedia server 40. This playback progress instruction includes information such as the audio / video playback position. Then, the multimedia server 40 forwards the received playback progress instruction to the third terminal device. The third terminal device obtains audio / video data from the multimedia server 40 based on the received audio / video URL address and performs local decoding, and obtains the lyrics file from the multimedia server 40 based on the lyrics URL. Then, based on the audio / video playback position information in the playback progress instruction, it achieves synchronous video playback and synchronized lyrics rendering with the first terminal device 10 and the second terminal device 20.

[0082] According to another aspect of the present invention, a method for synchronous playback of multimedia on multiple terminals is also provided. Figure 6 This is a schematic flowchart of a method for synchronous playback of multimedia on multiple terminals according to another embodiment of the present invention, as shown below. Figure 6 As shown, the method includes: S601, the playback device receives a multimedia playback instruction sent by a multimedia server. The multimedia playback instruction includes the URL address of the multimedia file. Specifically, when a first terminal device that has registered with the multimedia server requests the multimedia server to play specific multimedia, the multimedia server generates a multimedia playback instruction containing the URL address of the multimedia file.

[0083] The method further includes S602, whereby the playback device obtains the corresponding multimedia data stream from the multimedia server based on the multimedia file URL address for playback.

[0084] S603, after the playback device starts playing multimedia, it sends a start command containing a multimedia ID and multimedia content playback position information to the multimedia server, so that the multimedia server sends the start command to all registered terminal devices. In this way, all registered terminal devices can obtain the corresponding multimedia URL address from the multimedia server based on the multimedia ID in the start command, and then obtain specific multimedia data from the multimedia server based on the multimedia URL address, and synchronously play multimedia content consistent with the playback device according to the multimedia content playback position information in the start command.

[0085] In an optional embodiment, the method for synchronous playback of multimedia on multiple terminals further includes correction and calibration of the synchronous playback of multimedia. (See reference...) Figure 7 The method for synchronous playback of multimedia on multiple terminals according to the present invention further includes: S701, the playback device 30 sends a start-up command to the multimedia server 40.

[0086] The method also includes S702, where the multimedia server 40 forwards the start-up command to the first terminal device 10.

[0087] S703, after the first terminal device 10 obtains the corresponding video from the multimedia server 40, it starts playing the video according to the multimedia playback position information in the start-up instruction; at this time, the first terminal device 10 will record the device local time t1 when the video starts playing.

[0088] S704, the first terminal device 10 requests the playback progress again from the multimedia server 40.

[0089] S705, the multimedia server 40 forwards a request for playback progress to the playback device 30 again.

[0090] S706, the playback device 30 returns a playback progress instruction to the multimedia server 40. This playback progress instruction includes the multimedia ID and video playback position information.

[0091] S707, the multimedia server 40 forwards the playback progress instruction to the first terminal device 10.

[0092] S708, after receiving the playback progress instruction, the first terminal device 10 adjusts the video playback progress according to the synchronization correction algorithm. At this time, the first terminal device 10 records the device's local time t2 when the playback progress instruction is received. Furthermore, the first terminal device 10 uses the time difference between t2 and t1 to adjust the video playback progress, for example, by adding this time difference to the playback progress.

[0093] In this embodiment of the invention, if the lyrics and video are out of sync with the lyrics and video played by the playback device during the playback of multimedia content on the terminal device, the playback progress of the terminal device can be corrected by the correction method. It can also correct the delay caused by the instruction forwarding process, thereby ensuring that the video playback of the first terminal device, the second terminal device and the playback device is synchronized, and improving the user experience.

[0094] In this embodiment of the invention, for a newly registered third terminal device, the third terminal device requests the current playback progress of the multimedia from the multimedia server 40 upon registration, and plays content synchronized with the first terminal device 10, the second terminal device 20, and the playback device 30 after receiving a playback progress instruction from the playback device 30. At this time, the third terminal device records its local time t1 when it starts playing the multimedia content. Afterwards, the third terminal device can request the playback progress from the multimedia server 40 again, and records its local time t2 after receiving another playback progress instruction from the playback device 30. Furthermore, the third terminal device adjusts the multimedia playback progress using the time difference between t2 and t1, for example, by adding this time difference to the playback progress.

[0095] In one embodiment, a multimedia server 40 is provided, including a first receiving module 41 for receiving operation instructions sent to a first registered terminal device; a processing module 42 for processing the received operation instructions; a sending module 43 for sending the processed operation instructions to a predetermined playback device; a second receiving module 44 for receiving the operation execution result returned by the playback device after performing the corresponding operation according to the received operation instructions; and a detection module 45 for detecting currently registered terminal devices. The processing module is further used to synchronize the operation execution result to all detected registered terminal devices, so that all registered terminal devices present the same operation execution result as the playback device in the multimedia user interface. In one embodiment, the registration information, exit messages, etc., of multimedia users, terminal devices, and playback devices are stored in the memory of the multimedia server 40. In another embodiment, the multimedia server further includes a database 46 for recording the registration information, exit information, and deregistration information of multimedia users and terminal devices. The database is also used to record the registration information, exit information, and deregistration information of playback devices. Depending on the design requirements, the multimedia server can serve one terminal user and terminal device, or it can serve tens of thousands, tens of millions, or hundreds of millions of terminal users and terminal devices. Based on the end-user's usage scenario, end-users within the same scenario can be grouped together and recorded in a database. For example, in a break activity with a dozen or so people planning to sing for two hours, the playback device is a television, and the terminal devices are mobile phones. Each person can register with the multimedia server through a karaoke application on their phone or by scanning a QR code in the application. The database will then record the user information and terminal information of all dozen or so people in the same group. The television, as the playback terminal, can be pre-set in this group in the database, or any user can set the television as the playback device via their mobile phone, and the television's registration information will be recorded in the database.

[0096] The multimedia server also includes a storage module 47 for storing various multimedia content, such as video, audio, and lyrics files. The path to the multimedia file is the URL of the multimedia file on the server. The multimedia server can stream video, audio, lyrics files, etc., to playback devices and terminal devices for playback.

[0097] In this embodiment of the invention, the processing module is further configured to transmit multimedia content to the playback device according to the multimedia content acquisition request of the playback device. The processing module is also configured to forward the playback start instruction sent by the playback device to all registered terminal devices. The processing module is further configured to forward the playback progress instruction sent by the playback device to newly registered terminal devices. Optionally, the processing module may also simultaneously forward the playback progress instruction sent by the playback device to all registered terminal devices. The modules in the aforementioned multimedia server can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0098] In one embodiment, a multimedia server is provided, including a processor, a memory, and a computer program stored on the memory, the processor being configured to implement the method steps of the embodiments described above by executing the computer program.

[0099] In one embodiment, a computer device for use in a playback device is provided, the computer device including a processor, memory, input / output interface, and communication interface. The computer device may be a circuit board module including integrated circuits, and can be installed within the playback device. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external devices (such as a multimedia server) via a network connection. When the computer program is executed by the processor, it implements the method steps of the embodiments described above, thereby enabling synchronous playback of multimedia on multiple terminals.

[0100] In one embodiment, a computer program product is provided, including a computer program that can be configured at a multimedia server and executed by the processor of the multimedia server to implement corresponding method steps, thereby enabling synchronous playback of multimedia on multiple terminals together with playback devices and terminal devices.

[0101] In one embodiment, a computer program product is provided, including a computer program that can be configured at a playback device and executed by a processor at the playback device to implement corresponding method steps, thereby enabling synchronous playback of multimedia on multiple terminals together with a multimedia server and terminal devices.

[0102] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer. The computer program, when executed, may include the processes described in the embodiments of the methods above, stored in a readable storage medium. Any references to memory, databases, or other media used in the embodiments provided in this application may include at least one of non-volatile memory and volatile memory. The embodiments provided in this application involve... The database may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0103] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0104] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for synchronous playback of multimedia on multiple terminals, comprising: The multimedia server receives an operation instruction sent to a first terminal device that has been registered with it, wherein the first terminal device generates an operation instruction and sends it to the multimedia server after receiving an operation performed by a user on the multimedia user interface. The multimedia server processes the received operation instructions and sends the processed operation instructions to a predetermined playback device. The multimedia server receives the operation execution result returned by the playback device after performing the corresponding operation according to the received operation instruction; The multimedia server detects currently registered terminal devices and synchronizes the operation execution results to all detected registered terminal devices, so that all registered terminal devices achieve the same operation execution results as the playback device.

2. The method for synchronous playback of multimedia on multiple terminals according to claim 1, characterized in that, The multimedia server processes the received operation instructions and sends the processed operation instructions to a predetermined playback device, including: If the operation instruction sent by the first terminal device is a control instruction, the multimedia server forwards the control instruction to the playback device; If the operation instruction sent by the first terminal device is a streaming media data operation instruction, the multimedia server generates a streaming media data operation instruction containing the URL address of the multimedia file and sends the streaming media data operation instruction to the playback device.

3. The method for synchronous playback of multimedia on multiple terminals according to claim 2, characterized in that, The method further includes: When the operation instruction is to request video playback, the multimedia server sends a video playback instruction containing the video URL address to the playback device. When the playback device retrieves video data from the playback device based on the video URL address received in the video playback instruction, the multimedia server pushes the video data to the playback device for playback. The multimedia server receives a start-up command sent by the playback device after video playback, wherein the start-up command includes a multimedia ID and video playback position information; The multimedia server sends the start-up command to all registered terminal devices. After receiving a request for a specific video URL address from a registered terminal device based on the multimedia ID, the multimedia server sends the video URL address corresponding to the multimedia ID to the terminal device, so that the terminal device can obtain video data from the multimedia server based on the video URL address and synchronize video playback according to the start instruction.

4. The method for synchronous playback of multimedia on multiple terminals according to claim 2, characterized in that, The method further includes: When the operation instruction is to request audio playback, the multimedia server will send an audio playback instruction containing the audio or video URL address to the playback device; When the playback device requests audio or video data from the playback device based on the audio or video URL address received in the audio playback command, the multimedia server pushes the audio or video data to the playback device. The playback device suspends playback after receiving the audio or video data under the audio playback command. When the playback device requests a lyrics file, the multimedia server sends the lyrics file to the playback device. The multimedia server receives a start-up command sent by the playback device, wherein the playback device cancels the suspension and starts playing audio or video after obtaining the lyrics file, and draws the lyrics into the playing audio or video. The start-up command includes a multimedia ID and audio or video playback position information. The multimedia server sends the start-up command to all registered terminal devices. After receiving a request from a registered terminal device for a specific audio or video URL address and lyrics file URL address based on the multimedia ID, the multimedia server sends the audio or video URL address and lyrics file URL address corresponding to the multimedia ID to the terminal device. This allows the terminal device to retrieve the corresponding audio or video and lyrics from the multimedia server based on the audio or video URL address and lyrics file URL address, and to synchronize the playback of the audio or video and the rendering of the lyrics according to the start-up command.

5. The method for synchronous playback of multimedia on multiple terminals according to claim 3 or 4, characterized in that, The method further includes: When the multimedia server receives a request for playback progress from a newly registered terminal device during multimedia playback, it requests playback progress from the playback device. After receiving the playback progress instruction sent by the playback device, the multimedia server sends the playback progress instruction to the newly registered terminal device so that the newly registered terminal device and other registered terminal devices can play multimedia content synchronously. The playback progress instruction includes a multimedia ID and multimedia location playback information.

6. A method for synchronous playback of multimedia on multiple terminals, comprising: The playback device receives a multimedia playback instruction sent by a multimedia server, wherein the multimedia server generates a multimedia playback instruction containing the URL address of the multimedia file when the first terminal device requests it to play specific multimedia. The playback device obtains the corresponding multimedia data stream from the multimedia server according to the multimedia file URL address for playback; After the playback device starts playing multimedia, it sends a start-up command to the multimedia server, so that the multimedia server forwards the start-up command to all registered terminal devices. The start-up command includes a multimedia ID and multimedia content playback location information, so that all registered terminal devices can obtain specific multimedia data from the multimedia server according to the multimedia ID and synchronously play multimedia content consistent with that of the playback device according to the multimedia content playback location information.

7. The method for synchronous playback of multimedia on multiple terminals according to claim 6, characterized in that, The method further includes: When the multimedia playback instruction is an audio playback instruction, the playback device obtains audio or video data from the multimedia server according to the audio or video URL address in the audio playback instruction. After receiving the audio or video data, the playback device suspends playback and requests lyrics files from the multimedia server. After receiving the lyrics file sent by the multimedia server, the playback device cancels the suspension, starts playing the audio or video, and draws the lyrics into the playing audio or video, wherein the lyrics drawing is bound together with the audio or video playback; The playback device sends the start playback command to the multimedia server.

8. The method for synchronous playback of multimedia on multiple terminals according to claim 6 or 7, characterized in that, The method further includes: After receiving a request for playback progress from the multimedia server during multimedia playback, the playback device sends a playback progress instruction to the multimedia server so that the multimedia server can synchronize the received playback progress instruction to the newly registered terminal device. The multimedia server requests playback progress from the playback device after receiving a request from the newly registered terminal device for the playback progress of the multimedia being played.

9. The method for synchronous playback of multimedia on multiple terminals according to claim 1 or 6, characterized in that, All registered terminal devices include two or more terminal devices, and the two or more terminal devices include the first terminal device.

10. The method for synchronous playback of multimedia on multiple terminals according to claim 9, characterized in that, The terminal devices include mobile phones and tablet computers, and the multimedia user interface includes the user interface of a karaoke program; and / or, the playback devices include speakers, televisions, smart home screens, touch screen speakers, or in-vehicle systems.

11. The method for synchronous playback of multimedia on multiple terminals according to any one of claims 3, 4, 6 and 7, the method further comprising: When the multimedia server receives a request for playback progress from the first terminal device, it requests playback progress from the playback device. The multimedia server receives a playback progress instruction from the playback device and sends the playback progress instruction to the first terminal device, so that the first terminal device can correct the playback progress of the multimedia based on the time difference determined by the first local time when the multimedia content starts playing and the second local time when the playback progress instruction is received, according to a synchronization correction algorithm.

12. A multimedia server, comprising a processor, a memory, and a computer program stored in the memory, characterized in that, The processor is configured to implement the steps of the method according to any one of claims 1-5 by executing the computer program.

13. A computer device comprising a processor, a memory, and a computer program stored in the memory, the computer device being capable of being installed in a playback device, characterized in that, The processor is configured to implement the steps of the method according to any one of claims 6-8 by executing the computer program.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

15. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 6 to 8.

Images