Method, apparatus, device, medium and program product for playing content

Abstract

Embodiments of the present disclosure relate to a method, device, equipment, medium and program product for playing content. The method comprises receiving, by a first client, a target media stream for live streaming from a second client through a media stream channel, a media frame of the target media stream comprising metadata. The method further comprises determining a message identifier in the metadata in response to the metadata indicating that the media frame is to be synchronously displayed with a corresponding message. The method further comprises determining whether a target message having the message identifier exists in a message queue of a message channel for the live streaming. The method further comprises synchronously displaying the media frame and the target message in response to the target message existing in the message queue. Through the method, after receiving the media frame for synchronous display, the matching of the message identifier in the metadata of the media frame and the message identifier in the message queue makes the target message in the message queue be synchronously displayed with the media frame, improving user experience.

Description

Technical Field

[0001] The embodiments disclosed herein generally relate to the field of live streaming, and specifically to methods, apparatuses, devices, media, and program products for playing content. Background Technology

[0002] Currently, in the live streaming context, an increasing amount of content is being discovered and presented to a wide audience, including content promoting knowledge, entertaining the public, and showcasing competitive events. During live streams, hosts typically interact with other hosts or viewers, who can participate by entering text or sending gifts. As the live stream gains popularity, the number of participants also increases, leading to more frequent interactive actions and placing greater pressure on the data transmission between the live stream and the viewers.

[0003] With the rise of the live streaming industry, more and more people are paying attention to this field and joining the industry, resulting in an explosive growth in the number of live streamers. As the number of streamers increases dramatically, interactive features between streamers and viewers have emerged. Unlike traditional simple dialogues, streamers can now interact with viewers through various interactive operations and scenarios, greatly enhancing the viewing experience and increasing the frequency of interaction between streamers and viewers. While this brings more buzz to the live stream, it also presents greater challenges to the overall experience of the live stream content. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, medium, and program product for playing content.

[0005] According to a first aspect of this disclosure, a method for playing content is provided. The method includes receiving, via a media streaming channel, a target media stream for live streaming from a second client, wherein media frames of the target media stream include metadata. The method further includes determining a message identifier in the metadata in response to a metadata indication that media frames and corresponding messages are displayed synchronously. The method further includes determining whether a target message with the message identifier exists in a message queue of the message channel for live streaming. The method further includes synchronously displaying media frames and the target message in response to the presence of a target message in the message queue.

[0006] In a second aspect of this disclosure, a method for playing content is provided. The method includes receiving, at a second client, operations for synchronously displaying media frames and a target message from an original media stream for live streaming. The method further includes generating metadata for the media frames based on the operations, the metadata including a message identifier for the target message. The method also includes generating a target media stream by inserting the metadata into the media frames. The method further includes providing the target media stream to a first client via a media streaming channel, wherein the first client synchronously displays the target message and the media frames based on determining that a target message with a message identifier exists in a message queue of a live streaming message channel, the target message being sent to the first client via the live streaming message channel.

[0007] In a third aspect of this disclosure, an apparatus for playing content is provided. The apparatus includes a target media stream receiving module configured to receive a target media stream for live streaming from a second client via a media stream channel at a first client, wherein media frames of the target media stream include metadata; a message identifier determining module configured to determine a message identifier in the metadata in response to a metadata indication that media frames and corresponding messages are displayed synchronously; a target message existence determining module configured to determine whether a target message with a message identifier exists in the message queue of the message channel for live streaming; and a media frame and target message synchronous display module configured to synchronously display media frames and target messages in response to the existence of a target message in the message queue.

[0008] In a fourth aspect of this disclosure, an apparatus for playing content is provided. The apparatus includes an operation receiving module for media frames and a target message, configured to receive, on a second client, an operation for synchronously displaying media frames and a target message in an original media stream for live streaming; a metadata generation module, configured to generate metadata for the media frames based on the operation, the metadata including a message identifier for the target message; a target media stream generation module, configured to generate a target media stream by inserting metadata into the media frames; and a target media stream providing module, configured to provide the target media stream to a first client via a media stream channel, wherein the first client synchronously displays the target message and the media frames based on determining that a target message with a message identifier exists in a message queue of a message channel for live streaming, the target message being sent to the first client via the message channel for live streaming.

[0009] In a fifth aspect of this disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the methods according to the first and second aspects of this disclosure.

[0010] In a sixth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods according to the first and second aspects of this disclosure.

[0011] In a seventh aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the methods according to the first and second aspects of this disclosure.

[0012] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0014] Figure 1 The illustration shows a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;

[0015] Figure 2 The illustration shows a schematic diagram of an example method 200 for playing content according to some embodiments of the present disclosure;

[0016] Figure 3 The illustration shows a schematic diagram of an example method 300 for playing content according to some embodiments of the present disclosure;

[0017] Figure 4 The illustration shows a schematic diagram of an example of generating a target media stream according to some embodiments of the present disclosure;

[0018] Figure 5 The illustration shows an example of synchronously displaying media frames and target messages according to some embodiments of the present disclosure;

[0019] Figure 6 The illustration shows a schematic block diagram of an apparatus for playing content according to some embodiments of the present disclosure;

[0020] Figure 7 The illustration shows a schematic block diagram of an apparatus for playing content according to some embodiments of the present disclosure;

[0021] Figure 8 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated.

[0022] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0024] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0025] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0030] During content playback, for example in a live streaming scenario, information is typically presented in two forms. First, there's the stream of the host's video feed, i.e., the content being broadcast. In addition, a large amount of other information is generated during the live stream (gift messages, public chat messages, administrator messages, notifications of the host pausing the stream, e-commerce promotional cards, etc.). Interaction between the host and viewers is facilitated through "live stream messages." These two types of information reach the user's device through two different information channels. Due to the different transmission methods, there may be a time lag between these two types of information. This time lag can negatively impact the user experience when watching the live stream, especially if the live stream is implemented in a live streaming room format.

[0031] In the traditional approach described above, the video stream information in the live streaming room is retrieved by the player from the content delivery network via a specific protocol, while the messages in the live streaming room are obtained through the room's message channel (such as a WebSocket long connection). There is a time difference between these two. For example, if the end-to-end video stream latency is 5 seconds, while the live streaming message latency is 2 seconds, then the time difference is 3 seconds, resulting in a loss of user experience. When the live stream content is e-commerce sales, when the host prepares to add a new product link and shouts "3, 2, 1, add link!", and executes the link-adding operation on the host's end, because this type of operation is transmitted much faster than video streams, and because live streaming video and audio streams have a certain latency, viewers will only see and hear the host's shout after the product link has already been added by the merchant. This leads to a desynchronized audio-visual experience for users, reducing their willingness to continue watching and significantly diminishing their viewing experience.

[0032] To address at least the aforementioned and other potential problems, embodiments of this disclosure propose a method for playing content. In this method, at a first client, a target media stream for live streaming is first received from a second client via a media streaming channel. The media frames of the target media stream include metadata. Then, when it is determined that the metadata indicates that the media frame is to be displayed synchronously with a corresponding message, a message identifier in the metadata is determined. Next, the first client determines whether a target message with the message identifier exists in the message queue of the message channel for live streaming. Finally, if the target message exists in the message queue, the media frame and the target message are displayed synchronously. This method allows the target message to be displayed synchronously with the media frame, preventing the target message from being displayed on the first client before the media frame arrives, significantly improving the user's live streaming experience.

[0033] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1The illustration shows an example environment in which the devices and / or methods of embodiments of this disclosure can be implemented. In environment 100, taking a live streaming scenario as an example, client 122 is the viewer end, which may also be referred to as the first client for ease of description, and client 102 is the broadcaster end, which may also be referred to as the second client for ease of description. The original media stream 104 is the data stream displayed on the broadcaster end, and the original media stream 104 may be a video stream and / or an audio stream.

[0034] It is understood that this application is only described in conjunction with relevant embodiments and is not intended to limit the scope of protection of this application. Any technical solutions in other applications that fall within the scope of protection of this application should be protected.

[0035] exist Figure 1 In environment 100, at client 102, being the live streaming client, the original media stream can be obtained for display during the live stream. For example, if the streamer is live streaming, the original media stream could be the streamer's video captured by the camera device in client 102. For live stream 114, the streamer can also interact with viewers via messages. For example, interactions between the streamer and viewers in the live stream room of live stream 114, such as chatting in the public chat, can be accomplished through "live stream messages." Other messages could include sending gifts, administrator messages, the streamer pausing the live stream, e-commerce explanation cards, etc. For some messages, the message and the corresponding media frame in the media stream need to be displayed simultaneously on the viewer's client. In this case, client 102 can receive operation 110 to synchronously display media frame 106 and target message 112 in the original media stream 104 on the viewer's end. For example, in a scenario involving a product link, if the streamer says "link" and clicks the "link" button during the live stream, this click operation can trigger the synchronous display of media frame 106 and target message 112 on the viewer's end. After the click operation, a request for the operation can be sent to the server. The server then generates target message 112, returns the identifier of the target message to client 102, and sends the target message to client 122.

[0036] In some embodiments, media frame 106 may be one of multiple media frames in the original media stream 104. Target message 112 may be a specific message for the current live streaming scenario. For example, if the current live streaming scenario is an e-commerce demonstration, then target message 112 may be product listing information.

[0037] Next, after determining the operation 110 for the target message 112, metadata 108 for the media frame 106 displayed synchronously with the target message 112 can be generated. The metadata 108 includes the message identifier 126 of the target message 112.

[0038] Metadata 108 is determined based on message identifier 126 of target message 112. In some embodiments, metadata 108 is also determined based on scene identifier and timestamp of the scene corresponding to media frame 106. Additionally, scene identifier and timestamp are generated by client 102 after receiving operation 110. Subsequently, client 102 inserts the generated metadata 108 into media frame 106 to generate target media stream 116.

[0039] In some embodiments, metadata 108 is in the form of Supplemental Enhancement Information (SEI). Additionally, media frame 106 is the latest media frame among a plurality of media frames in the original media stream 104, and corresponds temporally to the target message.

[0040] Finally, the target media stream 116 is provided to the client 122 through the media stream channel 120. The client 122 can then obtain metadata from the media frames in the target media stream. The client 122 further determines whether the media frame has a synchronization message based on whether the metadata indicates that the media frame is to be displayed synchronously with the corresponding message. In one example, the data format of the metadata can be used to determine whether the metadata indicates that the media frame is to be displayed synchronously with the corresponding message. If the metadata is not in a predetermined format, it indicates that the media frame does not have a synchronization message. In this case, no further processing is performed, and the media frame is displayed directly. If the metadata is in a predetermined format, it can be determined that the media frame has a message that can be displayed synchronously. In another example, a flag bit can be set for the metadata, and this flag bit can be used to determine whether the metadata indicates that the media frame is to be displayed synchronously with the corresponding message. For example, if the flag bit is set to 0, it indicates that the metadata does not indicate that the media frame is to be displayed synchronously with the corresponding message; if the flag bit is set to 1, it indicates that the metadata indicates that the media frame is to be displayed synchronously with the corresponding message. The above examples are only for describing this disclosure and are not intended to specifically limit this disclosure.

[0041] Then, client 122 checks message queue 124 of message channel 118 for live stream 114 to see if target message 112 with message identifier 126 exists, in order to synchronously display target message 112 and media frame 106. If target message 112 is not yet in the message queue, it indicates that the streaming data arrived before the message, and metadata is stored in the metadata queue for the media stream, waiting for target message 112 to arrive for display, while the media frame needs to be displayed directly. If target message 112 is in the message queue, it indicates that the target message arrived faster. At this time, media frame 106 and target message 112 are synchronously displayed 128. Target message 112 is sent to client 122 through message channel 118 for live stream 114.

[0042] Message channel 118 and media stream channel 120 are used to transmit different types of data. In some embodiments, the message transmission rate in message channel 118 is faster than the media stream transmission rate in media stream channel 120. Additionally, message queue 124 stores multiple messages from message channel 118, all of which are messages generated by client 102 that need to be displayed in synchronization with video frames.

[0043] This method generates metadata corresponding to media frames in the original media stream based on operations performed on the target message, and then inserts the metadata into the media frames. The processed media frames and the target message are then transmitted from the second client to the first client via the media stream channel and message channel, respectively. By matching the message identifier in the media frame metadata with the message identifier in the message queue, the target message can be displayed synchronously with the media frame, preventing the target message from being displayed on the first client before the media frame arrives, thus significantly improving the user's live streaming experience.

[0044] The above combination Figure 1 The following is a schematic diagram illustrating an example environment in which some embodiments of this disclosure may be implemented, in conjunction with... Figure 2 The illustration depicts an example method 200 for playing content according to some embodiments of the present disclosure. This example method can be... Figure 1 It can be executed on client 122 or any suitable computing device.

[0045] like Figure 2 As shown, in example method 200, at block 202, a first client receives a target media stream for live streaming from a second client via a media streaming channel, wherein the media frames of the target media stream include metadata. For example, client 122 receives a target media stream 116 for live streaming from client 102 via a media streaming channel. For example, client 122 obtains the target media stream 116 from client 102 via a content delivery network, at which point client 122 pulls the target media stream 116 from the content delivery network. In some embodiments, the target media stream includes multiple media frames, and the media frames among the multiple media frames may include metadata, which may have different formats.

[0046] Next, at box 204, in response to the metadata indicating that the media frame and its corresponding message are displayed synchronously, the message identifier in the metadata is determined. For example, after receiving a media frame, client 122 needs to parse its metadata. If the metadata is in a predefined format, it indicates that the data frame has a message that is presented with it. For metadata in this predefined format, it includes at least the message identifier of the message presented with it. Therefore, after client 122 determines that the metadata is in a predefined format, it can obtain the message identifier 126 in the metadata.

[0047] In some embodiments, the data type of the metadata can be a supplemental enhancement information type, and the metadata can have different data formats, with a predetermined format indicating that the media frame has a corresponding message to be displayed synchronously. In one example, the predetermined format is that the metadata consists of three parts: the first part is the identifier of the message to be displayed synchronously, the second part is the identifier of the scene in which the live broadcast takes place, such as the identifier of the scene in which the live broadcast room is located, and the third part is the timestamp of message creation. Additionally, the predetermined format is MessageSEIMeta. The message identifier in the metadata is further determined only after it has been determined that the metadata is in the predetermined data format. For example, the MessageSEIMeta format is as follows:

[0048] {

[0049] “message_id”:xxxx / / Message ID

[0050] "scene":"xxx", / / Scene identifier

[0051] "timestamp":xxx / / Timestamp when the information was created

[0052] }

[0053] In some embodiments, when it is determined that the metadata does not indicate that the media frame is displayed synchronously with the corresponding message, it can be determined that the media frame corresponding to the metadata can be displayed directly, and the content of the media frame can be displayed directly.

[0054] Then, at box 206, it is determined whether a target message with the aforementioned message identifier exists in the message queue of the message channel for live streaming. After obtaining the video frames to be synchronized and displayed, it is also necessary to further determine whether the corresponding target message has reached client 122 based on the message identifier in the metadata.

[0055] Therefore, after determining that the metadata indicates the media frame and the corresponding message are displayed synchronously and that the message identifier in the metadata is determined, client 122 further determines whether a target message with a message identifier exists in the message queue of the message channel for live streaming. The message queue stores multiple reference messages from the live streaming room, each with a corresponding message identifier, and the message identifier is consistent with the message identifier in the metadata of the corresponding media frame.

[0056] In some embodiments, the message identifier in the metadata is matched with the reference identifier of the reference message in the message queue. If the message identifier in the metadata matches the reference identifier of the reference message, the reference message is identified as the target message. At this point, the media frame corresponding to the metadata and the target message are successfully matched.

[0057] Finally, at box 208, in response to the presence of the target message in the message queue, the media frame and the target message are displayed synchronously. The presence of the target message in the message queue indicates that the target message arrived at the first client before the video frame.

[0058] Once the existence of the target message in the message queue is confirmed, it indicates that the message identifier of the target message in the message queue and the message identifier in the metadata have successfully matched. At this point, the media frame corresponding to the metadata and the target message are synchronously displayed on the viewer's end. For example, the target message may include information that can be used to present the target item, such as displaying an item for sale in a live stream. The media frame or media stream can be used to prompt the user to present the target item, allowing the user to view the target item presented by the target message. In this way, it is ensured that if the target message arrives first, it waits for the corresponding video frame to arrive before being displayed synchronously.

[0059] In some embodiments, when the target message is not present in the message queue, the metadata is stored in the metadata queue, and then the content of the media frame corresponding to the metadata is displayed directly.

[0060] In some embodiments, the target message described above is the first message. The first client will also receive a second message for the live broadcast from the second client through a message channel. The process of receiving the second message is described below. The process of receiving the first message is similar to the process of receiving the second message described below, and will not be described in detail again.

[0061] Upon receiving the second message, client 122 determines whether the second message is a predefined type message. If the second message is a predefined type message, it indicates that the message has a corresponding video frame to be displayed synchronously. For example, when the second message is a predefined type LiveSEIMessage, it indicates that the second message has a corresponding video frame to be displayed synchronously. Then, client 122 further determines an identifier for the second message and processes the second message using the identifier.

[0062] For example, after determining the predefined type of the second message, it is determined whether the metadata queue contains an identifier for the second message. The metadata queue stores metadata received via the media streaming channel in a predefined format. If the metadata queue contains the identifier for the second message, it indicates that the video frame arrived before the message, and the content of the second message is displayed directly. Additionally, after displaying the content of the second message, the metadata corresponding to the identifier for the second message is removed from the metadata queue.

[0063] In some embodiments, the metadata in the metadata queue does not include an identifier for the second message, indicating that the message arrived at client 122 earlier than the media frame. In this case, the waiting time for the second message is further determined, which is the average latency of the media frames in the target media stream (e.g., 500 milliseconds). The second message is then placed in the message queue.

[0064] If the second message has not been processed by the timeout period in the message queue, it means the average latency period has been exceeded. At this point, the second message is removed from the message queue, and its content is displayed.

[0065] This method processes media frames in the original media stream, determines the metadata corresponding to the operation on the target message, and then inserts the metadata into the media frame. The processed media frame and the target message are then transmitted to the first client via the media stream channel and message channel, respectively, through the second client. Matching the message identifier in the media frame metadata with the message identifier in the message queue, or matching the message identifier with the message identifier in the metadata queue, ensures that the target message is displayed synchronously with the media frame. This prevents the target message from being displayed on the first client before the media frame arrives, significantly improving the user's live streaming experience.

[0066] The above combination Figure 2 A schematic diagram of an example method 200 for playing content according to some disclosed embodiments is described below. Figure 3 The illustration depicts an example method 300 for playing content according to some embodiments of the present disclosure. This example method can be... Figure 1It can be executed on client 102 or any suitable device.

[0067] like Figure 3 As shown, in example method 300, at block 302, the second client receives an operation for synchronously displaying media frames and target messages from the original media stream used for live streaming. Client 102 performs live streaming, for example, in a live streaming room, to provide media streams and messages to client 122 for users at client 122 to watch or interact with. During the live stream on client 102, there are operations that require synchronously displaying video frames from the media stream with their corresponding target messages on client 122.

[0068] In some embodiments, the original media stream is a media stream displayed at client 102. The original media stream includes multiple media frames. In one example, the original media stream may include a video stream, such as a video stream acquired through a camera at client 102. In another example, the original media stream may also include both video and audio streams.

[0069] In some embodiments, the target message can be a message specific to the currently live-streamed business scenario, which needs to be displayed synchronously with media frames in the displayed media stream. For example, if the currently live-streamed business scenario is a talent show and the host is singing, the target message could be a prompt message to the users in the live stream indicating when the host should switch songs; or it could be item information displayed when linked to an item.

[0070] In some embodiments, the operation may be a click or long-press operation on a target control or information in the client 102. It may also be any other operation capable of controlling the target message.

[0071] Next, at box 304, based on the operation, metadata for the media frame is generated, including the message identifier of the target message. For example, client 102 can generate metadata including the message identifier of target message 112. This media frame is a media frame in the media stream that corresponds to the target message in time at the time the operation is performed. For example, during an uplink operation, this media frame could be the media frame corresponding to when the uplink button is clicked to generate target information, such as a media frame containing the anchor saying "uplink".

[0072] In some embodiments, metadata generation can be achieved by obtaining the identifier of the target message displayed synchronously with the media frame. In other embodiments, the scene identifier and the timestamp corresponding to the media frame can be further obtained, and then metadata can be generated based on the scene identifier, the timestamp, and the identifier of the target message.

[0073] Then, at box 306, the target media stream is generated by inserting the metadata into the media frame. To enable synchronized display of the media frame and the target message, the metadata needs to be inserted into the media frame to generate a media frame including the metadata. This media frame is then combined with other media frames from the original video stream to form the target media stream.

[0074] Finally, at box 308, the target media stream is provided to the first client through a media stream channel, wherein the first client synchronously displays the target message and the media frame based on determining that a target message with the message identifier exists in the message queue of the message channel for the live broadcast, and the target message is sent to the first client through the message channel for the live broadcast.

[0075] In some embodiments, the media streaming channel and the messaging channel are implemented through different servers. For example, the media streaming channel is implemented through a content delivery network, while the messaging channel is implemented through a server corresponding to the live streaming application. Additionally, after the live streaming-related server receives operations on media frames and target messages, the server generates an identifier for the target message. The server then sends the target message to a first client, and the identifier for the target message is sent to a second client for generating metadata.

[0076] This method processes media frames in the original media stream, determines the metadata corresponding to the operation on the target message, and then inserts the metadata into the media frames. The processed media frames and the target message are then transmitted from the second client to the first client via the media stream channel and message channel, respectively. This ensures that the target message is displayed synchronously with the media frames, preventing the target message from being displayed on the first client before the media frames arrive, thus significantly improving the user's live streaming experience.

[0077] The above combination Figure 3 A schematic diagram of an example method 300 for playing content according to some disclosed embodiments is described below. Figure 4 A schematic diagram illustrating examples of generating a target media stream according to some embodiments of the present disclosure.

[0078] like Figure 4 As shown in Example 400, the target message is an interactive message. In box 402, the host initiates a flash sale. In box 404, the server successfully calls the interactive (flash sale) interface and returns data. During this process, the server returns a message identifier for the target message to the host.

[0079] At box 406, client 102 determines whether the streaming screen contains MessageSEIMeta metadata. This can be determined based on the type of the flash sale operation or the type of the returned message. If MessageSEIMeta metadata is not required, the streaming screen is sent directly at box 410.

[0080] If MessageSEIMeta metadata is required, insert the SEI data (MessageSEIMeta metadata) into the streaming frame at box 408. In one example, the SEI data is inserted into the media frame corresponding to the streaming frame. Additionally, the media frame is the media frame that corresponds to the target message in time among multiple media frames in the streaming frame.

[0081] The above combination Figure 4 A schematic diagram illustrating an example of generating a target media stream according to some disclosed embodiments is shown below. Figure 5 A schematic diagram illustrating an example of synchronously displaying media frames and target messages according to some embodiments of the present disclosure.

[0082] like Figure 5 As shown, in Example 500, in conjunction with the previous Example 400, it describes the operations performed on the client side for the viewer. Client 122 pulls the stream from the media streaming channel at box 502, and at box 508, receives a message through the live message channel 504, such as the first or second message described earlier.

[0083] After client 122 pulls the stream, at box 506, the SEI data in the player is parsed, and the data format of the metadata is parsed to determine whether its type is MessageSEIMeta at box 510. If the data format of the metadata is not MessageSEIMeta, the media frames in the target media stream are displayed directly.

[0084] If the metadata's data format is MessageSEIMeta, at box 514, a matching message is searched in the message waiting queue. Based on the message identifier in the metadata, a reference message with the same message identifier is searched in the message waiting queue. If a matching reference message is found in the message waiting queue, at box 524, the message is dispatched directly, and the media frame corresponding to the metadata and the target message are displayed synchronously.

[0085] If no matching reference message is found in the message waiting queue, the corresponding metadata is stored. At box 520, the SEI information is written to the StreamMessageSEI cache queue for subsequent matching.

[0086] After client 122 receives the message, at box 512, it further determines whether the message type is LiveSEIMessage. LiveSEIMessage is a predefined type used for settlement messages. Typically, the message contains a field specific to LiveSEIMessage; if it is this type, the field is marked as 1; otherwise, it can be marked as Null. If the message type is determined not to be LiveSEIMessage, indicating it is a non-SEI message, the message is dispatched directly.

[0087] If the message is determined to be of type LiveSEIMessage, meaning the message type is a predefined type, then at box 516, a direct search is performed in the StreamMessageSEI cache queue (metadata queue) to check if there is metadata containing the identifier of this message. If metadata containing the identifier of the second message is found, the message is dispatched directly at box 524.

[0088] If no metadata including the message identifier is matched, at box 518, the average video latency is obtained, and the consumption timeout is set. For example, if the obtained average video latency is 500 milliseconds, the consumption timeout (waiting time for the message) can be set to the average video latency. Additionally, the waiting time for the message can also be set to 1 second, which is not limited herein.

[0089] After setting the waiting time for the second message, the message (second message) is placed in the message waiting queue at box 522. When the waiting time for the second message in the waiting queue exceeds the preset waiting time, it is determined that the second message has timed out. At this time, the second message is dispatched and displayed on the viewer's end.

[0090] like Figure 6 As shown, the device 600 includes a target media stream receiving module 602, configured to receive a target media stream for live streaming from a second client through a media stream channel at a first client, wherein the media frames of the target media stream include metadata; a message identifier determination module 604, configured to determine a message identifier in the metadata in response to a metadata indication that the media frame and the corresponding message are displayed synchronously; a target message existence determination module 606, configured to determine whether a target message with a message identifier exists in the message queue of the message channel for live streaming; and a media frame and target message synchronous display module 608, configured to synchronously display the media frame and the target message in response to the existence of a target message in the message queue.

[0091] In some embodiments, the apparatus 600 further includes: a media frame display determination module, configured to determine that a media frame can be directly displayed in response to metadata not indicating that the media frame and the corresponding message are displayed synchronously; and a media frame content display module, configured to display the content of the media frame.

[0092] In some embodiments, the target message existence determination module 606 includes: a message identifier and reference identifier matching module, configured to match a message identifier with a reference identifier of a reference message in a message queue; and a target message determination module, configured to determine the reference message as the target message in response to the matching of the message identifier and the reference identifier.

[0093] In some embodiments, the apparatus 600 further includes: a metadata storage module configured to store metadata in a metadata queue in response to the absence of a target message in the message queue; and a media frame content display module configured to display the content of the media frame.

[0094] In some embodiments, the target message is a first message, and the apparatus 600 further includes: a second message receiving module configured to receive a second message for live streaming via a message channel; a second message predetermined type determining module configured to determine whether the second message is a message of a predetermined type, wherein the predetermined type indicates that the message is displayed synchronously with the corresponding media frame; and a second message processing module configured to process the second message based on the identifier of the second message in response to the second message being a message of a predetermined type.

[0095] In some embodiments, the second message processing module includes: an identifier determination module configured to determine whether the metadata in the metadata queue includes an identifier of the second message; a second message content display module configured to display the content of the second message in response to the metadata in the metadata queue including the identifier of the second message; and a metadata removal module configured to remove the metadata from the metadata queue.

[0096] In some embodiments, the second message processing module further includes: a waiting duration determination module, configured to determine a waiting duration for the second message in response to an identifier in the metadata queue that the metadata does not include the second message, the waiting duration being the average latency of media frames in the target media stream; and a message placement module, configured to place the second message into the message queue.

[0097] In some embodiments, the second message processing module further includes: a second message removal module configured to remove the second message from the message queue in response to the expiration of the waiting time for the second message in the message queue; and a second message content display module configured to display the content of the second message.

[0098] In some embodiments, a target message is used to present a target item, and a media frame is used to prompt the presentation of the target item.

[0099] like Figure 7 As shown, the apparatus 700 includes a media frame and target message operation receiving module 702, configured to receive operations from a second client for synchronously displaying media frames and target messages in an original media stream for live streaming; a metadata generation module 704, configured to generate metadata for the media frames based on the operations, the metadata including a message identifier for the target message; a target media stream generation module 706, configured to generate a target media stream by inserting metadata into the media frames; and a target media stream providing module 708, configured to provide the target media stream to a first client through a media stream channel, wherein the first client synchronously displays the target message and media frames based on determining that a target message with a message identifier exists in the message queue of the live streaming message channel, the target message being sent to the first client through the live streaming message channel.

[0100] In some embodiments, the metadata generation module 704 includes: a target message identifier acquisition module, configured to acquire the identifier of the target message to be displayed synchronously with the media frame; a scene identifier and timestamp determination module, configured to determine the scene identifier of the scene corresponding to the media frame and the timestamp corresponding to the media frame; and a metadata generation module, configured to generate metadata based on the target message identifier, scene identifier and timestamp.

[0101] Figure 8 A schematic block diagram of an example device 800 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The second client 102 and the first client 122 can be implemented using device 800. As shown, device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 can also store various programs and data required for the operation of device 800. CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0102] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0103] The various processes and handling described above, such as methods 200, 300, and examples 400 to 500, can be executed by processing unit 801. For example, in some embodiments, methods 200, 300, and examples 400 to 500 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of example methods 200, 300, and examples 400 to 500 described above can be performed.

[0104] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0105] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0106] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0107] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0108] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for playing content, comprising: The first client receives a target media stream for live streaming from the second client via a media streaming channel, wherein the media frames of the target media stream include metadata; In response to the metadata instructing the media frame to be displayed synchronously with the corresponding message, the message identifier in the metadata is determined; Determine whether a target message with the message identifier exists in the message queue of the message channel for the live broadcast; as well as In response to the presence of the target message in the message queue, the media frame and the target message are displayed synchronously.

2. The method according to claim 1, further comprising: In response to the fact that the metadata does not indicate that the media frame is displayed synchronously with the corresponding message, it is determined that the media frame can be displayed directly; as well as Display the content of the media frame.

3. The method according to claim 1, wherein determining whether a target message with the message identifier exists in the message queue of the message channel for the live broadcast includes: Match the message identifier with the reference identifier of the reference message in the message queue; as well as In response to the message identifier matching the reference identifier, the reference message is identified as the target message.

4. The method according to claim 1, further comprising: In response to the absence of the target message in the message queue, the metadata is stored in the metadata queue; as well as Display the content of the media frame.

5. The method according to claim 4, wherein the target message is a first message, further comprising: Receive a second message for the live broadcast through the message channel; Determine whether the second message is a message of a predetermined type, wherein the predetermined type indicates that the message is displayed synchronously with the corresponding media frame; as well as In response to the second message being a message of the predetermined type, the second message is processed based on its identifier.

6. The method of claim 5, wherein processing the second message based on the identifier of the second message includes: Determine whether the metadata in the metadata queue includes the identifier of the second message; In response to the metadata in the metadata queue including the identifier of the second message, the content of the second message is displayed; as well as Remove the metadata from the metadata queue.

7. The method of claim 6, wherein processing the second message based on the identifier of the second message further comprises: In response to the fact that the metadata in the metadata queue does not include the identifier of the second message, a waiting time for the second message is determined, the waiting time being the average latency of media frames in the target media stream; and The second message is placed into the message queue.

8. The method of claim 7, wherein processing the second message based on the identifier of the second message further comprises: In response to the expiration of the waiting time for the second message in the message queue, the second message is removed from the message queue; as well as Display the content of the second message.

9. The method according to claim 1, wherein the target message is used to present the target item, and the media frame is used to prompt the presentation of the target item.

10. A method for playing content, comprising: The second client receives operations for synchronously displaying media frames and target messages in the original media stream used for live streaming; Based on the operation, metadata of the media frame is generated, the metadata including the message identifier of the target message; The target media stream is generated by inserting the metadata into the media frame; as well as The target media stream is provided to a first client through a media stream channel, wherein the first client synchronously displays the target message and the media frame based on determining that a target message with the message identifier exists in the message queue of the message channel for the live broadcast, and the target message is sent to the first client through the message channel for the live broadcast.

11. The method of claim 10, wherein generating the metadata of the media frame comprises: Obtain the identifier of the target message to be displayed synchronously with the media frame; Determine the scene identifier and the timestamp corresponding to the media frame; as well as The metadata is generated based on the identifier of the target message, the scene identifier, and the timestamp.

12. An apparatus for playing content, comprising: The target media stream receiving module is configured to receive a target media stream for live streaming from a second client via a media stream channel on a first client, wherein the media frames of the target media stream include metadata; The message identifier determination module is configured to determine the message identifier in the metadata in response to the metadata indicating that the media frame and the corresponding message are displayed synchronously. The target message existence determination module is configured to determine whether a target message with the message identifier exists in the message queue of the message channel for the live broadcast. as well as The media frame and target message synchronization display module is configured to synchronously display the media frame and the target message in response to the presence of the target message in the message queue.

13. An apparatus for playing content, comprising: The media frame and target message operation receiving module is configured to receive operations from the second client for synchronous display of media frames and target messages in the original media stream used for live streaming; A metadata generation module is configured to generate metadata for the media frame based on the operation, the metadata including the message identifier of the target message; The target media stream generation module is configured to generate the target media stream by inserting the metadata into the media frame; as well as A target media stream providing module is configured to provide the target media stream to a first client through a media stream channel, wherein the first client synchronously displays the target message and the media frame based on determining that a target message with the message identifier exists in the message queue of the message channel for the live broadcast, and the target message is sent to the first client through the message channel for the live broadcast.

14. An electronic device comprising: At least one processor; as well as A storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1-11.

15. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1-11 when executed by a processor.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images