Methods, apparatuses, devices, media, and products for processing media streams

Abstract

Embodiments of the present disclosure relate to methods, apparatuses, devices, media and products for processing a media stream. The method comprises receiving a media stream for live streaming via a first channel. The method further comprises determining whether the media stream has lost a predetermined number of information items within a predetermined period, a target information item in the predetermined number of information items comprising supplemental enhancement information for a target media frame in the media stream. The method further comprises, in response to the media stream having lost the predetermined number of information items within the predetermined period, obtaining a target backup information item corresponding to the target media frame, the target backup information item being received via a second channel and comprising the supplemental enhancement information. The method further comprises controlling display of the media stream based on the target backup information item and the media stream. By this method, the use of multiple transmission channels to transmit information items comprising supplemental enhancement information and backup information items increases the stability and reliability of live streaming, improving the user experience.

Description

Technical Field

[0001] The embodiments disclosed herein generally relate to the field of live streaming, and specifically to methods, apparatuses, devices, media, and products for processing media streams. Background Technology

[0002] Currently, in the live streaming context, an increasing amount of content is being discovered and presented to a wide audience, including content promoting knowledge, entertaining the public, and showcasing competitive events. During live streams, hosts typically interact with other hosts or viewers, who can participate by typing text or sending virtual gifts. As the live stream gains popularity, the number of participants also increases, leading to more frequent interactions and a gradual increase in the stream's load.

[0003] With the rise of the live streaming industry, more and more people are paying attention to this field and joining the industry, resulting in an explosive growth in the number of live streamers. As the number of viewers in each streamer's room increases dramatically, the number of viewers also grows, varying from streamer to streamer. Therefore, the pressure on different streamers varies. With the continuous increase in viewers and equipment performance reaching its limits, improving the stability and reliability of live streams presents even more stringent challenges. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, medium, and product for processing media streams.

[0005] According to a first aspect of this disclosure, a method for processing a media stream is provided. The method includes receiving a media stream for live streaming via a first channel. The method further includes determining whether a predetermined number of information items have been lost in the media stream within a predetermined time period, wherein target information items among the predetermined number of information items include supplementary enhancement information for a target media frame in the media stream. The method further includes, in response to the media stream losing the predetermined number of information items within the predetermined time period, acquiring a target backup information item corresponding to the target media frame, the target backup information item being received via a second channel and including the supplementary enhancement information. The method further includes controlling the display of the media stream based on the target backup information item and the media stream.

[0006] In a second aspect of this disclosure, a method for processing a media stream is provided. The method includes determining a target information item and a target timestamp for a target media frame in a media stream used for live streaming, the target information item including supplementary enhancement information for the target media frame. The method also includes determining a target backup information item for the target information item, the target backup information item including the supplementary enhancement information. The method further includes generating an additional message based on the target backup information item and the target timestamp. The method also includes transmitting the media stream via a first channel and transmitting the additional message via a second channel for controlling the display of the media stream.

[0007] In a third aspect of this disclosure, an apparatus for processing a media stream is provided. The apparatus includes a media stream receiving module configured to receive a media stream for live streaming via a first channel; a predetermined number of information item loss determination module configured to determine whether the media stream has lost a predetermined number of information items within a predetermined time period, wherein target information items among the predetermined number of information items include supplementary enhancement information for a target media frame in the media stream; a target backup information item acquisition module configured to acquire a target backup information item corresponding to the target media frame in response to the media stream losing the predetermined number of information items within the predetermined time period, wherein the target backup information item is received via a second channel and includes supplementary enhancement information; and a media stream display control module configured to control the display of the media stream based on the target backup information item and the media stream.

[0008] In a fourth aspect of this disclosure, an apparatus for processing a media stream is provided. The apparatus includes a target information item and target timestamp determination module configured to determine a target information item and a target timestamp of a target media frame in a media stream for live streaming, the target information item including supplementary enhancement information for the target media frame; a target backup information item determination module configured to determine a target backup information item for the target information item, the target backup information item including supplementary enhancement information; an additional message generation module configured to generate an additional message based on the target backup information item and the target timestamp; and a media stream and additional message transmission module configured to transmit the media stream via a first channel and transmit the additional message via a second channel for controlling the display of the media stream.

[0009] In a fifth aspect of this disclosure, an electronic device is provided, including at least one processor; and a storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to a first or second aspect of this disclosure.

[0010] In a sixth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method according to a first or second aspect of this disclosure.

[0011] In a seventh aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the method according to a first or second aspect of this disclosure.

[0012] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0014] Figure 1 The illustration shows a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;

[0015] Figure 2 The illustration shows a schematic diagram of an example method for processing a media stream according to some embodiments of the present disclosure;

[0016] Figure 3 The illustration shows a schematic diagram of another example method for processing a media stream according to some embodiments of the present disclosure;

[0017] Figure 4 The illustration shows a schematic diagram of an example flowchart for processing a media stream according to some embodiments of the present disclosure;

[0018] Figure 5 The illustration shows a schematic diagram of an example of detecting information items according to some embodiments of the present disclosure;

[0019] Figure 6 The illustration shows a schematic diagram of an example of determining a target backup information item according to some embodiments of the present disclosure;

[0020] Figure 7 The illustration shows another example of determining a target backup information item according to some embodiments of the present disclosure;

[0021] Figure 8 The illustration shows a schematic diagram of example code for determining a target reference timestamp according to some embodiments of the present disclosure;

[0022] Figure 9 The illustration shows a schematic block diagram of an apparatus for processing media streams according to some embodiments of the present disclosure;

[0023] Figure 10 The illustration shows a schematic block diagram of an apparatus for processing media streams according to some embodiments of the present disclosure;

[0024] Figure 11 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated.

[0025] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0026] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0027] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0028] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0029] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0030] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0032] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0033] In media streaming, consider a scenario where a broadcaster is live. When a user is watching the broadcast, in addition to receiving the video stream from the broadcaster, they also receive Supplemental Enhancement Information (SEI) from the video stream. SEI is defined within the video stream and provides a way to add information to it. This SEI can include information specific to the live stream, such as the broadcaster's nickname, identifier, room number, and live stream layout. However, SEI may be lost during video stream transmission due to various factors, thus failing to reach the viewer or playback device.

[0034] In the traditional solutions described above, supplementary enhancement information is particularly important during live streaming, especially in the media stream transmission link. This is because it relies on supplementary enhancement information in the video stream to achieve certain functions or effects. For live streaming, many business scenarios depend on supplementary enhancement information for data and stream alignment. Examples include the layout and "health bar" display in player-versus-player (PK) battles, layout styles and letter information in multi-player scenes, lyrics and Musical Instrument Digital Interface (MIDI) progress in singing scenes, interactive components, progress bars, and program schedules in program scenes, and dual-screen layouts in game live streams. All of these require obtaining supplementary enhancement information from the video stream.

[0035] However, in actual video stream transmission, supplementary enhancement information may be lost due to factors such as the Content Delivery Network's (CDN) transcoding implementation, frame dropping strategies, and video format compatibility issues. In such cases, the stability of the data transmission link cannot be guaranteed, leading to availability problems caused by the limitations of the supplementary enhancement information and a significant decline in the user experience during live streaming.

[0036] To address at least the aforementioned and other potential problems, embodiments of this disclosure propose a method for processing media streams. In this method, a media stream for live streaming is first received via a first channel. Then, after receiving the media stream, it is determined whether a predetermined number of information items have been lost within a predetermined time period. The predetermined number of information items includes target information items that include supplementary enhancement information for a target media frame in the media stream. Then, after determining that the media stream has lost the predetermined number of information items within the predetermined time period, a target backup information item corresponding to the target media frame is obtained. This target backup information item is received via a second channel and includes supplementary enhancement information. Finally, the display of the media stream is controlled using the target backup information item and the media stream. This method allows the use of multiple transmission channels to transmit information items for live streaming and provides backup information items, increasing the stability and reliability of the live stream. In cases of poor network conditions or information item loss, the transmitted backup information items can be used to improve the stability of the live stream, thereby improving the viewer experience.

[0037] It is understood that this disclosure is only described in conjunction with relevant embodiments and is not intended to limit the scope of protection of this disclosure. Any technical solutions in other disclosures that fall within the scope of protection of this disclosure should be protected.

[0038] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings. Figure 1 The illustration shows an example environment in which the devices and / or methods of embodiments of this disclosure can be implemented. In environment 100, taking a live streaming scenario as an example, computing device 102 can be used to provide a media stream for live streaming to client 122. In one example, the media stream is a media stream for game live streaming. In another example, the media stream can be a media stream for a multi-person online scenario, such as a singing scenario or a live competition scenario. Client 122 can be a device on the viewer side for watching the live stream. Computing device 102 and client 122 can communicate via a first channel 118 and a second channel 120. The transmission method of the first channel is different from that of the second channel; for example, the first channel includes a content delivery network, and the second channel includes a business server for live streaming, a partially open-source data transmission platform, etc.

[0039] Examples of computing device 102 and client 122 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0040] like Figure 1As shown, taking a live streaming scenario as an example, computing device 102 can generate a media stream 104 for client 122. The client 122 used by the viewer can first receive the media stream 104 for live streaming from computing device 102 via the first channel 118. In one example, computing device 102 can be a broadcaster. In another example, computing device 102 can also be a Real-Time Communication (RTC) server for live streaming services. The broadcaster or RTC server can transmit the media stream 104 for live streaming to client 122 via the first channel 118.

[0041] In one example, media stream 104 may include a video stream for live streaming. In another example, media stream 104 may include an audio stream for live streaming. In yet another example, media stream 104 may also include both video and audio streams for live streaming. Additionally, media stream 104 may also include information such as the broadcaster's identity identifier, a timestamp corresponding to the live stream, and a room identifier for the live stream room.

[0042] To ensure the display of the media stream, the computing device 102 can generate corresponding supplementary enhancement information for the media frames in the media stream 104 and add the supplementary enhancement information as information items to the media stream. Therefore, the media stream 104 can include multiple media frames and multiple information items. For example, a corresponding information item can be set for each media frame in the media stream. The client 122 can perform media stream reception 124, and after receiving the media stream, it can further perform information item detection 126. When performing information item detection 126, the client 122 can determine whether the media stream 104 has lost a predetermined number of information items within a predetermined time period. In addition to including media frames and information items for media frames, the media stream 104 can also include a timestamp for each media frame, such as a presentation timestamp (PTS) for each media frame. It is understood that all media frames in the media stream 104 include a corresponding timestamp.

[0043] If media stream 104 does not lose a predetermined number of information items within a predetermined time period, the media stream is presented using the information items obtained from media stream 104. However, if it is determined that media stream 104 has lost a predetermined number of information items within a predetermined time period, client 122 can further obtain backup information items corresponding to media frames within that time period. For example, if the predetermined number of information items includes target information item 108, then target backup information item 106 corresponding to target media frame 112 is obtained. Target backup information item 106 is transmitted to client 122 via second channel 120 and includes supplementary enhancement information for target media frame 112. The supplementary enhancement information in target backup information item 106 is the same as the supplementary enhancement information 110 in target information item 108; therefore, target backup information item 106 is a backup of target information item 108.

[0044] When the computing device 102 transmits the target backup information item to the client 122, the computing device 102 may use the target timestamp 114 and the target backup information item 106 to generate an additional message 116. The additional message 116 may be sent to the client 122 via the second channel 122.

[0045] Finally, after the additional message 116 is sent to the client 122 via the second channel 122, a mapping 128 for backup information items and reference timestamps can be generated using the backup information items and reference timestamps of other received media frames. This mapping 128 can be a list for storing backup information items and reference timestamps. For example, the target backup information item 106 described above is retrieved from this mapping using the target timestamp corresponding to the target media frame. Next, the received media stream 104 and the target backup information item 106 can be used to control the display of the media stream 104 on the client 122.

[0046] This method allows the use of multiple transmission channels to transmit information items for live streaming and provides backup information items, thereby increasing the stability and reliability of the live stream. Even if information items are lost, the transmitted backup information items can be used to improve the stability of the live stream, thus improving the viewer's experience.

[0047] The above combination Figure 1 The following is a schematic diagram illustrating an example environment in which some embodiments of this disclosure may be implemented, in conjunction with... Figure 2 The illustration depicts an example method 200 for processing a media stream according to some embodiments of the present disclosure. This example method can be... Figure 1 It can be executed on client 122 or any suitable computing device.

[0048] like Figure 2As shown, in example method 200, at block 202, client 122 receives a media stream for live streaming via first channel 118. Viewers watching the live stream can use the client to pull the media stream. For example, first channel 118 could be a media streaming channel formed by a content delivery network.

[0049] Media stream 104 may include various data streams for live streaming, such as video streams and audio streams. Additionally, before pushing media stream 104 via the first channel 118, computing device 102 may add information items for media frames to the media stream, such as information items storing supplementary enhancement information.

[0050] Next, at box 204, client 122 determines whether a predetermined number of information items have been lost from the media stream within a predetermined time period. The target information items among these predetermined number of information items include supplementary enhancement information for the target media frames in the media stream. For the received media stream, to ensure that the media stream is presented to the user in a better way, it is necessary to further detect whether any information items inserted into the media stream are lost.

[0051] In some embodiments, when the client 122 receives the media stream 104 via the first channel 118, it can also receive backup information items for the media stream 104 via the second channel 120. To determine whether a predetermined number of information items have been lost within a predetermined time period, a counting operation can be performed on the received backup information items and detected information items in the media stream 104 within the predetermined time period to determine a first number of backup information items received for the media stream 104 within the predetermined time period and a second number of information items detected in the media stream 104 within the predetermined time period. It should be noted that the predetermined time period is set by the user, and the user can change the length of the predetermined time period according to their own needs.

[0052] Subsequently, client 122 can determine whether media stream 104 has lost a predetermined number of information items 106 within a predetermined time period based on the magnitude of the first and second counts obtained from the counting. In one example, when it is determined that the first count is greater than a first threshold number and the second count is equal to a second threshold number, it can be determined that media stream 104 has lost a predetermined number of information items 106 within the predetermined time period. For example, if the first threshold number is 3 and the second threshold number is 0, it indicates that three backup information items were received within the predetermined time period, but no information item was received, and it can be determined that three information items were lost within the predetermined time period. In another example, when it is determined that the first count is greater than or equal to the first threshold number and the second count is less than the second threshold number, it can be determined that media stream 104 has lost a predetermined number of information items 106 within the predetermined time period. The first and second counts can be set according to user needs. It should be noted that the rules for determining the loss of a predetermined number of information items 106 can be defined by the user, and this application does not impose any restrictions on this.

[0053] If the media stream does not lose a predetermined number of information items within the predetermined time period, the information from the information items obtained from the media stream can be used to display the media stream to the audience of client 122.

[0054] Then, at box 206, in response to the media stream losing a predetermined number of information items within a predetermined time period, client 122 obtains the target backup information items corresponding to the target media frame, the target backup information items being received via the second channel and including supplementary enhancement information.

[0055] In some embodiments, client 122 may receive an additional message 116 generated by computing device 102. The additional message 116 may be sent to client 122 via second channel 120. The additional message 116 includes reference timestamps and backup information items for media frames in media stream 104. Upon receiving the additional message 116, client 122 stores the paired reference timestamps and backup information items in a mapping for backup information items and reference timestamps within client 122. For example, the obtained reference timestamps and backup information items may be stored in a list within the client used for storing reference timestamps and backup information items.

[0056] Once it is determined that the media stream 104 has lost a predetermined number of information items 106 within a predetermined time period, it is necessary to further obtain the target timestamp 114 corresponding to the target media frame 112 in the media stream 104 at this time. During this process, the client 122 will obtain a stored mapping for backup information items and reference timestamps. This mapping stores a set of backup information items and a set of reference timestamps, and the backup information items in the set of backup information items and the timestamps in the set of reference timestamps correspond one-to-one. Additionally, the set of backup information items and the set of reference timestamps stored in the mapping arrive at the client 122 earlier than the media stream 104.

[0057] Client 122 can first determine the media frames in the media stream 104 received within a predetermined time period, and further obtain the metadata of the media frames. Then, it can obtain the timestamp of the media frame from the metadata. For example, if the media frames in the media stream 104 received within the predetermined time period include the target media frame 112, the target timestamp 114 can be obtained. Subsequently, the determined target timestamp 114 and the mapping are used to determine the target backup information item 106 corresponding to the target timestamp 114 from a set of backup information items.

[0058] When retrieving target backup information item 106, client 122 needs to look up a set of reference timestamps to determine the target reference timestamp corresponding to target timestamp 114. Subsequently, client 122 uses the retrieved target reference timestamp to determine the target backup information item 106 corresponding to the target reference timestamp from the mapping between a set of backup information items and a set of reference timestamps.

[0059] Since there is a certain delay in data transmission, there is also a certain difference between the timestamp corresponding to the media frame in the media stream 104 and the reference timestamp in a set of reference timestamps. Therefore, when it is determined that the difference between the target timestamp 114 and the reference timestamp in a set of reference timestamps is less than the threshold difference, such as less than 50ms, the reference timestamp can be determined as the target timestamp 114.

[0060] In some embodiments, the client 122 may also determine a reference timestamp corresponding to the media stream 104 when it is initially received, such as the timestamp at which the media stream was first received. Then, the client 122 records the elapsed time since receiving the media stream and generates an updated timestamp based on a standard timestamp interval. After determining that a predetermined number of information items have been lost in the media stream within a predetermined time period, a mapping and updated timestamps for a set of backup information items and a set of reference timestamps can be further obtained, and the target backup information item 106 corresponding to the updated timestamp can be obtained from the mapping based on the updated timestamp.

[0061] In some embodiments, when no corresponding target backup information item is found in the mapping relationship between a set of backup information items and a set of reference timestamps using the target timestamp, the updated timestamp can be further used to obtain the target backup information item 106 corresponding to the updated timestamp from the mapping. The above examples are only used to describe this disclosure and are not intended to specifically limit this disclosure.

[0062] Finally, at box 208, client 122 controls the display of the media stream based on the target backup information item and the media stream.

[0063] Once a predetermined number of information items 106 within a predetermined time period are determined to be lost, the client 122 can use the obtained target backup information item 106 and target timestamp 114 to adjust the media stream 104 so that the media stream 104 can be displayed more stably on the viewer's end, thereby improving the reliability of the live broadcast.

[0064] This method allows the use of multiple transmission channels to transmit information items for live streaming and provides backup information items to increase the stability and reliability of the live stream. In the event of information item loss due to poor network conditions, the transmitted backup information items can still be used to continue the live stream, thereby improving the stability of the live stream and greatly enhancing the live stream experience and the viewer experience.

[0065] The above combination Figure 2 A schematic diagram of an example method 200 for processing media streams according to some disclosed embodiments is described below. Figure 3 The illustration depicts an example method 300 for processing a media stream according to some embodiments of the present disclosure. This example method 300 can be provided by... Figure 1 The computing device 102 in the middle is used to execute.

[0066] like Figure 3 As shown, in example method 300, at block 302, computing device 102 determines a target information item and a target timestamp for a target media frame in a media stream used for live streaming. The target information item includes supplemental enhancement information for the target media frame. Additionally, the target media frame can be any frame in the video stream.

[0067] In some embodiments, when the computing device 102 is used for live streaming, it can generate a corresponding information item for each media frame in the media stream to be generated. This information item includes supplementary enhancement information for that media frame. The computing device 102 can then add this supplementary enhancement information to the video stream for delivery to the viewer.

[0068] Media stream 104 is determined based on the original media stream. In one example, the original media stream includes a video stream used for live streaming. In another example, the original media stream includes an audio stream used for live streaming. Additionally, media stream 104 can be generated by capturing, merging, or performing other operations on the original media stream from the broadcaster and / or from other users (such as guests participating in the live stream).

[0069] Next, at box 304, computing device 102 determines a target backup information item for the target information item, the target backup information item including supplementary enhancement information. Since the media stream 104 may lose information items during transmission through the first channel 118, computing device 102 also determines a target backup information item 106 for the target information item 108, and the target backup information item 106 also includes the supplementary enhancement information from the target information item 108.

[0070] Then, at box 306, computing device 102 generates an additional message based on the target backup information item and the target timestamp. In order to send the target backup information item and the target timestamp to client 122, computing device 102 places the target backup information item and the target timestamp into an additional message that can be transmitted to client 122 via the service server.

[0071] Additionally, in order to reduce the number of additional messages transmitted, computing device 102 may place backup information items and target timestamps for a period of time into additional message 116.

[0072] Finally, at box 308, computing device 102 sends the media stream via the first channel and additional messages via the second channel for controlling the display of the media stream. After generating the media stream and additional messages, computing device 102 sends them to the client through different channels.

[0073] For example, computing device 102 uses a first channel comprised of a content delivery network to send a media stream to client 122. Computing device 102 can also send generated supplementary messages 116 to client 122 via a second channel 120, implemented through a server of the live streaming application. Client 122 can determine the timestamp and corresponding backup information item from the supplementary messages according to pre-set rules. Client 122 can then utilize the above-mentioned reference... Figure 2 The described method controls the display of media stream 104 on client 122.

[0074] This method utilizes multiple transmission channels to transmit information items for live streaming and provides backup information items, thereby increasing the stability and reliability of the live stream. Therefore, in cases where poor network conditions lead to information item loss, the transmitted backup information items can be used to improve live stream stability, significantly enhancing the live stream experience and the viewer experience.

[0075] The above combination Figure 3 A schematic diagram of an example method 300 for processing media streams according to some disclosed embodiments is described below. Figure 4 A schematic diagram illustrating an example of a flowchart for processing a media stream according to some embodiments of the present disclosure.

[0076] like Figure 4 As shown in Example 400, taking a live streaming scenario as an example, the business layer includes the broadcaster side 402 and the viewer side 444. The framework layer includes supplementary enhancement information management 406 for the broadcaster side 402 and supplementary enhancement information 448 for the viewer side 444. It should be noted that the information item may only include supplementary enhancement information. The following example uses supplementary enhancement information SEI as the information item, the Content Delivery Network (CDN) as the first channel, and the business server as the second channel for related explanation.

[0077] First, at position 404, the broadcaster side 402 sends the SEI that needs to be backed up to the Supplemental Enhancement Information Manager 406, and the Backup Supplemental Enhancement Information Generator 408 generates the backup SEI. Then, at position 420, the SEI field is added to the Media Stream Acquisition, Merging, and Push Module 422. Furthermore, at position 418, after generating the merged media stream, the Media Stream Acquisition, Merging, and Push Module 422 provides the latest Presentation Timestamp (PTS) to the Supplemental Enhancement Information Manager 406 so that the Backup Supplemental Enhancement Information Generator 408 can generate a PTS and backup SEI pair.

[0078] The media stream acquisition, merging, and pushing module 422 can acquire the original media stream from the individual broadcast 410, client merging 412, and RTC client merging 414. Then, the acquired original media stream is acquired and merged at the media stream acquisition, merging, and pushing module 422 to generate a processed media stream. After obtaining the original media stream, it needs to be sent to the media stream acquisition, merging, and pushing module 422 at point 416.

[0079] The media stream acquisition, merging and pushing module 422 can also push the processed media stream 426 and transmit it to the player 464 at the viewer's end via CDN 438.

[0080] In some embodiments, besides merging the media stream at the broadcaster's side, merging can also be performed without going through the broadcaster's side. In this case, the media stream can be merged at the server side at 424, and the media stream, along with the corresponding PTS and backup SEI pairs, can be generated via RTC server merging at 434. Subsequently, the merged result is pushed to CDN at 436, thereby transmitting the media stream to the viewer via CDN 428. The media stream includes SEIs and PTSs for video frames.

[0081] After the media stream is received at the viewer's end, the PTS will be updated at position 460. The frequency of PTS updates is set by the user. The CDN SEI will also be updated at position 462, which is the SEI obtained from the media stream.

[0082] Subsequently, a CDN SEI validity check is performed at point 458, which is used to determine whether a predetermined number of information items have been lost within a predetermined time period, that is, whether a predetermined number of CDN SEIs have been lost within a predetermined time period.

[0083] If it is determined that no CDN SEIs of the predetermined number are lost within the predetermined time period, then SEI 450 is directly dispatched, and SEIs are sent at 446 and transmitted together to the audience side 444 of the service layer.

[0084] Additionally, the client obtains the backup SEI and PTS at 440, then receives backup supplementary enhancement information at 454, and forms a backup supplementary enhancement information list at 452, which stores the mapping between backup supplementary enhancement information and PTS. If it is determined that a predetermined number of CDN SEIs have been lost within a predetermined time period, the target backup SEI corresponding to the target timestamp is determined using the backup supplementary enhancement information list at 456, and the target backup SEI is sent at 446 for use on the audience side at the business layer at 444.

[0085] The backup SEI is generated on the broadcaster's side. After generating the backup SEI and the corresponding PTS, the PTS and backup SEI can be sent on the client at 428, and then sent to the audience via the business server 430.

[0086] In some embodiments, backup SEI and PTS can also be sent from the results of RTC server merge at 432 using server merge.

[0087] This method allows for the use of multiple transmission channels to transmit information items for live streaming and provides backup information items, increasing the stability and reliability of live streaming and greatly enhancing the viewer experience.

[0088] The above combination Figure 4Describes an example schematic diagram of a flowchart for processing a media stream according to some disclosed embodiments. The following description is made in conjunction with Figure 5 describes an example schematic diagram for detecting information items according to some embodiments of the present disclosure.

[0089] As shown in Figure 5 , in example 500, in combination with the foregoing example 400, when determining whether a predetermined number of information items are lost within a predetermined time period, this can be implemented by using an alternative supplementary enhancement information list <PTS, backup SEI> 502 and a player 514.

[0090] Within the predetermined time period, the player 514 updates the PTS at 510 and updates the CDN SEI at 512, so as to update the timestamp and supplementary enhancement information in the media stream, and can also obtain the backup SEI and PTS in the backup supplementary enhancement information list.

[0091] Subsequently, the pre-set rules in the example code 506 are used to determine whether a predetermined number of supplementary enhancement information are lost within the predetermined time period. For example, when it is determined according to the example code that the number of supplementary enhancement information in the media stream within the predetermined time period from pts_0 to pts_n is 0 and the number of alternative supplementary enhancement information is greater than 3, it can be determined that the predetermined number of supplementary enhancement information are lost within the predetermined time period. In this case, the alternative SEI is used at 504.

[0092] If the pre-set rule in the example code is not satisfied, it can be determined that the predetermined number of supplementary enhancement information are not lost within the predetermined time period. In this case, the CND SEI is used at 508.

[0093] The above description is given in conjunction with Figure 5 describes an example schematic diagram for detecting information items according to some disclosed embodiments. The following description is made in conjunction with Figure 6 describes an example schematic diagram for determining a target backup information item according to some embodiments of the present disclosure.

[0094] As shown in Figure 6 , in example 600, in combination with the foregoing example 400 and example 500, a method for determining target backup supplementary enhancement information is as follows: after client-side merging, individual streaming 602 and server-side merging 604, the client sends PTS and backup SEI at 606, and the server-side merging sends PTS and backup SEI to the service server 608 at 610. Then at 612, the viewer client receives the backup SEI and PTS, wherein the backup SEI and PTS can be received via an open source data transmission platform or Instant Messaging (IM).

[0095] Subsequently, the player 616 updates the SEI and PTS at 618, and performs a judgment operation at 622 to check whether the CDNSEI is valid, that is, to determine whether a predetermined number of supplementary enhancement information has been lost within a predetermined time period. If it is determined that a predetermined number of supplementary enhancement information has been lost within the predetermined time period, at 620, the target timestamp in the media stream is used to match a set of reference timestamps in the backup SEI list 614. When the match is successful, the backup SEI is returned at 626, and the backup SEI is sent to multiple services, such as program 630, screen sharing 632, multi-user service 634, and / or other services 636, using SEI monitoring 628.

[0096] When it is determined that no predetermined number of supplementary enhancement information has been lost within the predetermined time period, at 624, the CDN SEI is returned, and then the SEI is sent to multiple services using SEI monitoring 628, such as programs 630, screen sharing 632, multi-person services 634 and / or other 636.

[0097] The above combination Figure 6 A schematic diagram illustrating an example of determining a target backup information item for processing a media stream, according to some disclosed embodiments, is described below. Figure 7 A schematic diagram illustrating another example of determining a target backup information item according to some embodiments of the present disclosure.

[0098] like Figure 7 As shown in Example 700, combined with the previous Example 600, the validity of the CDN SEI can be detected, i.e., whether a predetermined number of supplementary enhancement information has been lost within a predetermined time period. When the CDN SEI is detected as invalid, a backup SEI allocation operation is performed at 706. At this time, the PTS clock is updated at 702 to determine the backup SEI corresponding to the updated timestamp from the backup SEI list 704, and the backup SEI is returned. The updated PTS clock is determined using a base timestamp, the duration of the current time from the base timestamp, and the standard time intervals between timestamps. For example, how many standard time intervals have elapsed within that duration, and then the number of standard intervals can be used as an index, or the duration corresponding to multiple standard time intervals can be added to the base timestamp to obtain the currently updated timestamp. When it is determined that no predetermined number of supplementary enhancement information has been lost within the predetermined time period, the CDN SEI is returned directly.

[0099] It should be noted that the two methods for determining the target backup supplemental enhancement information in Example 600 and Example 700 are different, and users can choose either method to determine the target backup supplemental enhancement information.

[0100] Additionally, you can first choose the method in Example 600 to determine the target backup supplemental enhancement information. If the method in Example 600 cannot determine the target backup supplemental enhancement information, you can choose the method in Example 700 to determine the target backup supplemental enhancement information.

[0101] The above combination Figure 7 A schematic diagram illustrating another example of determining a target backup information item according to some disclosed embodiments is described below. Figure 8 A schematic diagram illustrating example code for determining a target reference timestamp according to some embodiments of the present disclosure.

[0102] like Figure 8 As shown, the PTS of the source stream are [0, 40, 80, 120, 160, 200], the PTS of the transcoded stream after CDN (reduced by 160 frames) is [0, 50, 100, 150, 160, 200], and the PTS in the backup list received by the user side is [0, 40, 80, 120, 160, 200]. As shown in Example 800, in Example Code 802, the timestamps of the transcoded streams presented in the CDN within the predetermined time period are cdn_pts_0, cdn_pts_50, cdn_pts_100, cdn_pts_150, and cdn_pts_200, respectively, while the timestamps in the backup supplementary enhancement information are backup_pts_0, backup_pts_40, backup_pts_80, backup_pts_120, backup_pts_160, and backup_pts_200, respectively.

[0103] This can cause a mismatch between the timestamps in the CDN and the timestamps in the backup and supplementary enhancement information, leading to problems with the live stream. In this case, according to example code 802, the first `backup_pts` value greater than `cdn_pts` can be retrieved. The difference between `backup_pts` and `cdn_pts` is used to determine whether to consume `backup_pts`. If the difference is less than a threshold, such as 50, then `backup_pts` is consumed normally; otherwise, it is not consumed.

[0104] like Figure 9As shown, the device 900 includes a media stream receiving module 902, configured to receive a media stream for live streaming via a first channel; a predetermined number of information item loss determination module 904, configured to determine whether the media stream has lost a predetermined number of information items within a predetermined time period, wherein the target information items in the predetermined number of information items include supplementary enhancement information for a target media frame in the media stream; a target backup information item acquisition module 906, configured to acquire a target backup information item corresponding to the target media frame in response to the media stream losing a predetermined number of information items within a predetermined time period, wherein the target backup information item is received via a second channel and includes supplementary enhancement information; and a media stream display control module 908, configured to control the display of the media stream based on the target backup information item and the media stream.

[0105] In some embodiments, the predetermined number of information item loss determination module 904 includes: a first number and a second number counting module, configured to count a first number of backup information items for the media stream received via a second channel within a predetermined time period and a second number of information items detected in the media stream within the predetermined time period; and a predetermined number of information item loss determination module, configured to determine whether the media stream has lost a predetermined number of information items within the predetermined time period based on the first number and the second number.

[0106] In some embodiments, the predetermined number of information item loss determination module includes: a first number and a second number determination module, configured to determine whether the first number is greater than a first threshold number and whether the second number is equal to a second threshold number; and a predetermined number of information item loss determination module, configured to determine that the media stream has lost a predetermined number of information items within a predetermined time period in response to the first number being greater than the first threshold number and the second number being equal to the second threshold number.

[0107] In some embodiments, the target backup information item acquisition module 906 includes: a target timestamp acquisition module, configured to acquire a target timestamp corresponding to a target media frame in response to the media stream losing a predetermined number of information items within a predetermined time period; a mapping determination module, configured to determine a mapping for a set of backup information items and a set of reference timestamps; and a target backup information item determination module, configured to determine a target backup information item corresponding to the target timestamp from a set of backup information items based on the target timestamp and the mapping.

[0108] In some embodiments, the target timestamp acquisition module includes: a target media frame determination module configured to determine a target media frame in a media stream received within a predetermined time period; and a target timestamp acquisition module configured to acquire the target timestamp of the target media frame from the metadata of the target media frame.

[0109] In some embodiments, the target backup information item determination module includes: a target reference timestamp lookup module, configured to look up a target reference timestamp corresponding to the target timestamp in a set of reference timestamps; and a target information item determination module, configured to determine the backup information item corresponding to the target reference timestamp in the mapping as the target information item.

[0110] In some embodiments, the target reference timestamp lookup module includes: a target timestamp and reference timestamp difference determination module, configured to determine the difference between the target timestamp and a reference timestamp in a set of reference timestamps; and a target reference timestamp determination module, configured to determine the reference timestamp as the target reference timestamp in response to the difference being less than a threshold difference.

[0111] In some embodiments, the apparatus 900 further includes: an additional message receiving module configured to receive additional messages for a media stream via a second channel, the additional messages including reference timestamps and backup information items for media frames in the media stream; and a mapping generation module configured to generate a mapping based on the reference timestamps and backup information items.

[0112] In some embodiments, the target backup information item acquisition module 906 includes: a base timestamp determination module configured to determine a base timestamp corresponding to a media stream; an updated timestamp generation module configured to generate an updated timestamp based on the base timestamp and the elapsed duration; a mapping determination module configured to determine a mapping for a set of backup information items and a set of reference timestamps; and a target backup information item acquisition module configured to acquire the target backup information item corresponding to the updated timestamp from the mapping based on the updated timestamp.

[0113] In some embodiments, the first channel includes a content delivery network and the second channel includes a service server for live streaming.

[0114] like Figure 10 As shown, the device 1000 includes a target information item and target timestamp determination module 1002, configured to determine the target information item and target timestamp of a target media frame in a media stream used for live streaming, wherein the target information item includes supplementary enhancement information for the target media frame; a target backup information item determination module 1004, configured to determine a target backup information item for the target information item, wherein the target backup information item includes supplementary enhancement information; an additional message generation module 1006, configured to generate an additional message based on the target backup information item and the target timestamp; and a media stream and additional message sending module 1008, configured to send the media stream via a first channel and send the additional message via a second channel for controlling the display of the media stream.

[0115] In some embodiments, the apparatus 1000 further includes: an original media stream acquisition module configured to acquire a set of original media streams; and a media stream generation module configured to generate a media stream based on the set of original media streams.

[0116] In some embodiments, the first channel includes a content delivery network and the second channel includes a service server for live streaming.

[0117] In some embodiments, the method is performed on a client or a real-time communication server.

[0118] Figure 11 A schematic block diagram of an example device 1100 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 102 and client 122 can be implemented using device 1100. As shown, device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 1102 or loaded from storage unit 1108 into random access memory (RAM) 1103. Various programs and data required for the operation of device 1100 can also be stored in RAM 1103. CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0119] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The various processes and handling described above, such as methods 200, 300, and examples 400 to 800, can be executed by processing unit 1101. For example, in some embodiments, methods 200, 300, and examples 400 to 800 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by CPU 1101, one or more actions of example methods 200, 300, and 400 to 800 described above can be performed.

[0121] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0122] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0123] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0124] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0125] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0126] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0127] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0129] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing media streams, comprising: Receive media streams for live streaming via the first channel; Determine whether the media stream has lost a predetermined number of information items within a predetermined time period, wherein the target information items among the predetermined number of information items include supplementary enhancement information for the target media frame in the media stream; In response to the media stream losing the predetermined number of information items within the predetermined time period, a target backup information item corresponding to the target media frame is obtained, the target backup information item being received via the second channel and including the supplementary enhancement information; as well as The display of the media stream is controlled based on the target backup information item and the media stream.

2. The method according to claim 1, wherein determining whether the media stream has lost a predetermined number of information items within a predetermined time period includes: The first number of backup information items for the media stream received via the second channel during the predetermined time period and the second number of information items detected in the media stream during the predetermined time period are counted. as well as Based on the first number and the second number, it is determined whether the media stream lost a predetermined number of information items within a predetermined time period.

3. The method according to claim 2, wherein determining whether the media stream lost a predetermined number of information items within a predetermined time period based on the first number and the second number includes: Determine whether the first number is greater than a first threshold number and whether the second number is equal to a second threshold number; as well as In response to the first number being greater than the first threshold number and the second number being equal to the second threshold number, it is determined that the media stream has lost a predetermined number of information items within a predetermined time period.

4. The method according to claim 1, wherein obtaining the target backup information item corresponding to the target media frame includes: In response to the media stream losing the predetermined number of information items within the predetermined time period, a target timestamp corresponding to the target media frame is obtained; Determine the mapping between a set of backup information items and a set of reference timestamps; as well as Based on the target timestamp and the mapping, the target backup information item corresponding to the target timestamp is determined from the set of backup information items.

5. The method according to claim 4, wherein obtaining the target timestamp corresponding to the target media frame includes: Identify the target media frame in the media stream received within the predetermined time period; as well as Obtain the target timestamp of the target media frame from the metadata of the target media frame.

6. The method of claim 4, wherein determining the target backup information item corresponding to the target timestamp from the set of backup information items comprises: Find the target reference timestamp that corresponds to the target timestamp in the set of reference timestamps; as well as The backup information item in the mapping that corresponds to the target reference timestamp is determined as the target backup information item.

7. The method according to claim 6, wherein finding the target reference timestamp corresponding to the target timestamp in the set of reference timestamps comprises: Determine the difference between the target timestamp and the reference timestamp in the set of reference timestamps; as well as In response to the difference being less than a threshold difference, the reference timestamp is determined as the target reference timestamp.

8. The method according to claim 4, further comprising: Additional messages for the media stream are received via the second channel, the additional messages including reference timestamps and backup information items for media frames in the media stream; as well as The mapping is generated based on the reference timestamp and the backup information item.

9. The method according to claim 1, wherein obtaining the target backup information item corresponding to the target media frame includes: Determine the base timestamp corresponding to the media stream; An updated timestamp is generated based on the base timestamp and the elapsed time. Determine the mapping between a set of backup information items and a set of reference timestamps; as well as Based on the updated timestamp, the target backup information item corresponding to the updated timestamp is obtained from the mapping.

10. The method of claim 1, wherein the first channel comprises a content delivery network and the second channel comprises a service server for the live stream.

11. A method for processing a media stream, comprising: Determine the target information item and target timestamp of the target media frame in the media stream used for live streaming, wherein the target information item includes supplementary enhancement information for the target media frame; Determine a target backup information item for the target information item, wherein the target backup information item includes the supplementary enhancement information; Generate an additional message based on the target backup information item and the target timestamp; as well as The media stream is sent via the first channel and the additional message is sent via the second channel to control the display of the media stream.

12. The method of claim 11, further comprising: Acquire a set of raw media streams; as well as The media stream is generated based on the set of original media streams.

13. The method of claim 11, wherein the first channel comprises a content delivery network and the second channel comprises a service server for the live stream.

14. The method of claim 11, wherein the method is performed at a client or a real-time communication server.

15. An apparatus for processing a media stream, comprising: The media stream receiving module is configured to receive media streams for live streaming via the first channel; A predetermined number of information item loss determination module is configured to determine whether the media stream has lost a predetermined number of information items within a predetermined time period, wherein the target information items in the predetermined number of information items include supplementary enhancement information for the target media frame in the media stream; The target backup information item acquisition module is configured to acquire a target backup information item corresponding to the target media frame in response to the media stream losing the predetermined number of information items within the predetermined time period. The target backup information item is received via a second channel and includes the supplementary enhancement information. as well as The media stream display control module is configured to control the display of the media stream based on the target backup information item and the media stream.

16. An apparatus for processing a media stream, comprising: The target information item and target timestamp determination module is configured to determine the target information item and target timestamp of a target media frame in a media stream used for live streaming, wherein the target information item includes supplementary enhancement information for the target media frame; A target backup information item determination module is configured to determine a target backup information item for the target information item, wherein the target backup information item includes the supplementary enhancement information; The additional message generation module is configured to generate an additional message based on the target backup information item and the target timestamp. as well as The media stream and additional message sending module is configured to send the media stream via a first channel and the additional messages via a second channel for controlling the display of the media stream.

17. An electronic device comprising: At least one processor; as well as A storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1-14.

18. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the method according to any one of claims 1-14 when executed by a processor.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-14.

Images