Media stream processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a media stream processing method and device, equipment, a storage medium and a program product, and relates to the technical field of video processing, the method comprises the following steps: receiving a first request sent by first client equipment, the first request being used for requesting media stream data; under the condition that the stream pulling mode selected by the first client device is related to a first protocol, in response to the first request, performing protocol conversion and packaging on first media stream data sent by a second client device based on a second protocol according to the first protocol to obtain second media stream data; and sending the second media stream data to the first client device. According to the scheme, the transmission delay is reduced as much as possible on the premise of controlling the cost, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of streaming media technology, and in particular to a media stream processing method, device, equipment, storage medium and program product. Background Art

[0002] Live broadcast distribution technology plays an important role in the field of streaming media. Common existing live broadcast distribution technologies include: standard live broadcast, low-latency live broadcast and web real-time communication (Web Real Time Communication, Web RTC). However, the latency of standard live broadcast technology is relatively high and does not meet the current user requirements for real-time live broadcast. Although low-latency live broadcast has a significant improvement in latency, it is difficult to achieve a good interactive effect for strong interactive scenarios, and the support for packet loss and freezes in weak network environments is not good enough. The construction and maintenance costs of Web RTC are high. Therefore, how to reduce the transmission latency as much as possible while controlling costs is a problem currently faced. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a media stream processing method, apparatus, device, storage medium and program product, so as to solve the current problem that it is impossible to reduce the transmission delay as much as possible under the premise of controlling costs.

[0004] In a first aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a media stream processing method, which is applied to a server device, including:

[0005] Receiving a first request sent by a first client device, where the first request is used to request media stream data;

[0006] In a case where the stream pulling mode selected by the first client device is related to the first protocol, in response to the first request, performing protocol conversion and encapsulation on the first media stream data sent by the second client device based on the second protocol according to the first protocol to obtain second media stream data;

[0007] The second media stream data is sent to the first client device.

[0008] Optionally, the method further comprises:

[0009] receiving a signaling negotiation request sent by the first client device, where the signaling negotiation request carries a session description protocol SDP description;

[0010] Send a signaling negotiation response to the first client device, where the signaling negotiation response carries a response SDP description corresponding to the SDP description, and the response SDP description includes switched connection establishment ICE information, and the ICE information is used to establish a connection for transmitting media streams between the server device and the first client device.

[0011] Optionally, in response to the first request, perform protocol conversion and encapsulation on the first media stream data transmitted by the second client device using a second protocol according to a first protocol to obtain second media stream data, including:

[0012] Query the SDP description of the first client device;

[0013] When the SDP description of the first client device is queried, perform protocol conversion and encapsulation on the first media stream data transmitted by the second client device using a second protocol according to a first protocol to obtain second media stream data.

[0014] Optionally, the method further includes:

[0015] Call a signaling storage interface to store the SDP description and the response SDP description in a shared signaling memory.

[0016] Optionally, perform protocol conversion and encapsulation on the first media stream data sent by the second client device based on a second protocol according to a first protocol to obtain second media stream data, including:

[0017] Decapsulate the video data packets in the first media stream data, and encapsulate the decapsulated non-image data and network abstraction layer unit NALU into a first data packet related to the first protocol;

[0018] Decapsulate the audio data packets in the first media stream, transcode the decapsulated advanced audio coding AAC data stream, and encapsulate the transcoded data packets into a second data packet related to the first protocol;

[0019] Wherein, the second media stream data includes the first data packet and the second data packet.

[0020] Optionally, the method further includes:

[0021] Receive a first notification message sent by the first client device, where the first notification message carries first indication information, and the first indication information is used to indicate second media stream data not received by the first client device;

[0022] According to the first indication information, obtain the unreceived second media stream data in the buffer of the server device;

[0023] Encapsulate the unreceived second media stream data into a retransmission data packet, where the retransmission data packet includes an identifier of an original transmission data packet of the unreceived second media stream data;

[0024] Send the retransmission data packet to the first client device.

[0025] Optionally, the method further includes:

[0026] Receive a Real-Time Transport Control Protocol (RTCP) Extended Report (XR) data packet periodically sent by the first client device, where the RTCP XR data packet includes a first timestamp when the first client device sends the RTCP XR data packet;

[0027] Send an updated RTCP XR data packet to the first client device, where the updated RTCP XR data packet includes a processing delay of the server device for the RTCP XR data packet, and the processing delay is used to determine the timing for the first client device to send the first notification message.

[0028] Optionally, the method further includes:

[0029] Receive a second notification message sent by the first client device, where the second notification message carries second indication information, and the second indication information is used to indicate the unreceived second media stream data of the first client device;

[0030] Determine a packet loss rate according to the second indication information;

[0031] When the packet loss rate is greater than or equal to a packet loss rate threshold, obtain target second media stream data in a buffer of the server device, where the target second media stream data carries key information of a video;

[0032] Send the target second media stream data to the first client device.

[0033] In a second aspect, to achieve the above object, an embodiment of the present application provides a media stream processing method, which is applied to a first client device and includes:

[0034] Send a first request to a server device, where the first request is used to request media stream data;

[0035] Receive second media stream data sent by the server device, where the second media stream data is media stream data obtained by the server device through protocol conversion and encapsulation of first media stream data according to a first protocol, and the first media stream data is media stream data sent by a second client device to the server device based on a second protocol;

[0036] Decode the second media stream data and play the video corresponding to the decoding result.

[0037] Optionally, the method further includes:

[0038] Send a signaling negotiation request to the server device, where the signaling negotiation request carries an SDP description;

[0039] Receive a signaling negotiation response sent by the server device, where the signaling negotiation response carries a response SDP description corresponding to the SDP description, and the response SDP description includes ICE information, and the ICE information is used to establish a connection for transmitting media streams between the server device and the first client device.

[0040] Optionally, the method further includes:

[0041] In the case where the second media stream data is not received within a first duration, send a first notification message to the server device, where the first notification message carries first indication information, and the first indication information is used to indicate the second media stream data not received by the first client device;

[0042] Receive a retransmission data packet sent by the server device, where the retransmission data packet includes an identifier of an initial transmission data packet of the second media stream data not received;

[0043] Decode the second media stream data in the retransmission data packet and play the video corresponding to the decoding result.

[0044] Optionally, the method further includes:

[0045] In the case where the second media stream data is not received within a first duration, send a second notification message to the server device, where the second notification message carries second indication information, and the second indication information is used to indicate the second media stream data not received by the first client device;

[0046] Receive the target second media stream data sent by the server device; where the target second media stream data carries key information of the video.

[0047] Optionally, the method further includes:

[0048] Periodically send RTCP XR data packets to the server device, where the RTCP XR data packets include a first timestamp when the first client device sends the RTCP XR data packets;

[0049] Receive the updated RTCP XR packet sent by the server device, where the updated RTCP XR packet includes the processing delay of the RTCP XR packet by the server device;

[0050] Determine the first duration according to the second timestamp when the first client device receives the updated RTCP XR packet, the first timestamp, and the processing delay.

[0051] Optionally, determining the first duration according to the second timestamp when the first client device receives the updated RTCP XR packet, the first timestamp, and the processing delay includes:

[0052] Calculate the difference between the second timestamp and the sum of the first timestamp and the processing delay to obtain the round-trip delay RTT of the RTCP XR packet;

[0053] Calculate the first duration according to the RTTs of multiple adjacent RTCP XR packets.

[0054] In a third aspect, to achieve the above object, an embodiment of the present application provides a media stream processing device, which is applied to a server device and includes:

[0055] A first receiving module, configured to receive a first request sent by a first client device, where the first request is used to request media stream data;

[0056] A processing module, configured to, when the pulling mode selected by the first client device is related to a first protocol, in response to the first request, perform protocol conversion and encapsulation on the first media stream data sent by a second client device based on a second protocol according to the first protocol to obtain second media stream data;

[0057] A first sending module, configured to send the second media stream data to the first client device.

[0058] In a fourth aspect, to achieve the above object, an embodiment of the present application provides a media stream processing device, which is applied to a first client device and includes:

[0059] A first sending module, configured to send a first request to a server device, where the first request is used to request media stream data;

[0060] A first receiving module, configured to receive the second media stream data sent by the server device, where the second media stream data is the media stream data obtained by the server device performing protocol conversion and encapsulation on the first media stream data according to the first protocol, and the first media stream data is the media stream data sent by the second client device to the server device based on the second protocol;

[0061] A first decoding module, configured to decode the second media stream data and play a video corresponding to the decoding result.

[0062] In a fifth aspect, to achieve the above object, an embodiment of the present application provides an electronic device, including: a transceiver, a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, it implements the media stream processing method as described in the first aspect, or implements the media stream processing method as described in the second aspect.

[0063] In a sixth aspect, to achieve the above object, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the media stream processing method as described in the first aspect, or implements the media stream processing method as described in the second aspect.

[0064] In a seventh aspect, to achieve the above object, an embodiment of the present application provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the media stream processing method as described in the first aspect, or implement the media stream processing method as described in the second aspect.

[0065] The above technical solutions of the present application have at least the following beneficial effects:

[0066] In the media stream processing method according to the embodiment of the present application, first, a server device receives a first request sent by a first client device, where the first request is used to request media stream data; second, when the pulling method selected by the first client device is related to a first protocol, in response to the first request, according to the first protocol, protocol conversion and encapsulation are performed on first media stream data sent by a second client device based on a second protocol to obtain second media stream data; third, the second media stream data is sent to the first client device. In this way, protocol conversion and encapsulation are performed on the received first media stream data based on the pulling method selected by the first client device, so as to convert the received first media stream data into second media stream data adapted to the pulling method selected by the first client, and the second media stream data is sent to the first client device. In this way, it is possible to achieve lower-latency transmission in the last mile while keeping the pushing method at the pushing end unchanged, so as to reduce the transmission latency as much as possible on the premise of controlling costs. Description of the Drawings

[0067] Figure 1 It is a schematic diagram of an existing low-latency live broadcast link;

[0068] Figure 2It is a schematic diagram of the process of existing SDP negotiation;

[0069] Figure 3 It is one of the schematic diagrams of the media stream processing method according to the embodiment of the present application;

[0070] Figure 4 It is another schematic diagram of the media stream processing method according to the embodiment of the present application;

[0071] Figure 5 It is a schematic diagram of the SDP negotiation process in the embodiment of the present application;

[0072] Figure 6 It is a schematic diagram of RTC signaling sharing in the embodiment of the present application;

[0073] Figure 7 It is a schematic diagram of the process of protocol conversion and encapsulation of media stream data in the embodiment of the present application;

[0074] Figure 8 It is a schematic diagram of the interaction process between the client and the server in the embodiment of the present application;

[0075] Figure 9 It is a schematic diagram of the data structure of the retransmitted data packet in the embodiment of the present application;

[0076] Figure 10 It is a schematic diagram of the RTT sliding window in the embodiment of the present application;

[0077] Figure 11 It is another schematic diagram of the media stream processing method according to the embodiment of the present application;

[0078] Figure 12 It is one of the schematic diagrams of the structure of the media stream processing device according to the embodiment of the present application;

[0079] Figure 13 It is another schematic diagram of the structure of the media stream processing device according to the embodiment of the present application;

[0080] Figure 14 It is a schematic diagram of the structure of the electronic device according to the embodiment of the present application. Detailed implementation manners

[0081] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0082] The terms "first", "second", etc. in the description and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the description and claims means at least one of the connected objects, and the character " / ", generally indicates an "or" relationship between the associated objects before and after.

[0083] Next, the technologies related to the embodiments of this application will be described first.

[0084] Standard live broadcast: The commonly used protocol is HLS (Hypertext Transfer Protocol Live Streaming). HLS is a streaming media transmission protocol based on the Hypertext Transfer Protocol (HTTP) and is widely used to deliver live video to mobile devices and desktop browsers on the Internet. HLS cuts the video into small fragments, and each small fragment is an independent file to achieve better compatibility and adaptive streaming media transmission. The transmission delay is about 10 - 20s. Among them, the standard live broadcast has a relatively high delay and does not meet the current users' requirements for the real-time nature of live broadcasts, especially in scenarios involving live interactions, such as live streaming with goods, remote meetings, game live broadcasts, etc.

[0085] Low-latency live broadcast: Low-latency live broadcast means reducing the transmission delay so that the audience can receive the content with the shortest possible delay. In low-latency live broadcasts, two common protocols are: Real Time Messaging Protocol (RTMP) / HTTP-FLV protocol, and the transmission delay is about 3 - 5s. Specifically, the low-latency live broadcast content delivery network (Content Delivery Network, CDN) mainly includes functions such as live video pushing, live video pulling, and streaming media processing, providing powerful ultra-high-definition video pushing / pulling capabilities based on the RTMP / HTTP-FLV protocol, and can meet the basic requirements of large-traffic and low-latency distribution. Among them, compared with the standard live broadcast, the low-latency live broadcast has a significant improvement in delay, but for strong interaction scenarios, it is difficult to achieve better interaction effects. At the same time, the low-latency live broadcast does not support packet loss and freezing well in weak network environments.

[0086] Specifically, the existing low-latency live broadcast link is as Figure 1As shown in the figure, the push end pushes the audio and video data stream to the central or edge push node of the low-latency live CDN through the RTMP protocol. The audio and video data stream undergoes streaming media processing such as transcoding, and then is transmitted to the pull end through the edge pull node of the CDN using the RTMP or HTTP-FLV protocol. The latency of the entire link is about 2-3 seconds, including the RTMP push latency, the CDN transmission latency, the player buffer and decoding latency, etc. Both RTMP and HTTP-FLV are transmitted based on the Transmission Control Protocol (TCP). Although multiple optimizations have been made in the CDN transmission link, problems such as data backlog, packet loss, and out-of-order still exist under poor network conditions. Therefore, the ability to resist weak networks is relatively weak.

[0087] Web Real Time Communication (WebRTC) is a real-time communication technology mainly used to achieve real-time audio and video communication in browsers that support WebRTC. WebRTC supports point-to-point live broadcasts and is suitable for scenarios that require low latency and real-time interaction. The transmission latency is about 300ms. Among them, although Web real-time communication has great advantages in terms of latency and weak network resistance, its construction and maintenance costs are relatively high, which has become an important factor hindering its wide application.

[0088] Based on the above analysis, how to reduce the transmission latency as much as possible and better resist the weak network environment under the premise of controlling costs is a technical problem that needs to be solved. In view of the above problems, the embodiments of the present application provide a media stream processing method, device, equipment, storage medium, and program product based on the existing low-latency live link.

[0089] Next, the implementation process of the embodiments of the present application will be described in conjunction with specific examples.

[0090] The embodiments of the present application provide a media stream processing method, which is applied to a server device. Here, the server device can be a server or a server cluster, etc. As Figure 3 shown, the method includes:

[0091] Step 301, receive a first request sent by a first client device, where the first request is used to request media stream data.

[0092] Here, the first client device can be a terminal device, such as a terminal watching a live video in a live broadcast scenario.

[0093] Step 302, when the pulling stream mode selected by the first client device is related to the first protocol, in response to the first request, perform protocol conversion and encapsulation on the first media stream data sent by the second client device based on the second protocol according to the first protocol, to obtain second media stream data.

[0094] Here, the transmission of media stream data through the first protocol has a lower delay than the transmission of media stream data through the second protocol; for example, the first protocol is the Real Time Communication (RTC) protocol, and the second protocol is the RTMP protocol. The second client device is a device that generates media stream data, such as the host device in a live broadcast scenario. That is to say, when the second client device pushes the first media stream data to the server device through the RTMP protocol, and the second client device expects the server device to send the media stream data through the RTC protocol, the server device can perform protocol conversion and encapsulation on the received first media stream data to obtain second media stream data that can be sent through the RTC protocol, so as to realize the distribution of the last mile of media stream data through the RTC protocol with lower delay.

[0095] Step 303, send the second media stream data to the first client device.

[0096] In the media stream processing method of the embodiment of the present application, first, the server device receives a first request sent by the first client device, and the first request is used to request media stream data; secondly, when the pulling stream mode selected by the first client device is related to the first protocol, in response to the first request, perform protocol conversion and encapsulation on the first media stream data sent by the second client device based on the second protocol according to the first protocol, to obtain second media stream data; finally, the server device sends the second media stream data to the first client device. In this way, on the basis of the existing low-delay live broadcast link, transmission protocol conversion can be added on the server device to realize the distribution of the last mile of media stream data using a protocol with lower delay, so as to obtain stronger distribution capabilities and lower transmission delays without increasing costs.

[0097] In addition, after step 301, the method further includes:

[0098] When the pulling stream mode selected by the first client device is related to the second protocol, in response to the first request, perform streaming media processing on the first media stream data to obtain third media stream data;

[0099] Transmit the third media stream data to the first client device based on the second protocol or a protocol related to the second protocol.

[0100] Here, the protocol related to the second protocol is, for example, the HTTP-FLV protocol.

[0101] That is to say, the server device can perform different operations based on the pull stream mode selected by the first client device. If the first client device selects the conventional pull stream mode, the server device can perform conventional streaming media processing on the received first media stream data, and send the processed third media stream data to the first client device through the second protocol or a protocol related to the second protocol (RTMP / HTTP-FLV). If the first client device selects the pull stream mode related to the first protocol, the server device can perform protocol conversion and encapsulation on the received first media stream data, and send the obtained second media stream data to the first client device through the first protocol (RTC protocol). In this way, the server device not only has the existing low-latency live pull stream ability, but also has the RTC protocol negotiation and repackaging ability. Thus, as Figure 4 shown, on the basis of the original low-latency live link, the protocol conversion and encapsulation functions are added, that is: the push-pull stream link (RTC push-pull stream link) involved in the embodiments of the present application is similar to the low-latency live CDN. The push stream end still maintains the push stream through the second protocol (RTMP protocol). The difference is that the edge pull stream node can not only provide the low-latency live pull stream ability based on the second protocol or a protocol related to the second protocol (RTMP protocol or HTTP-FLV protocol), but also provide the first protocol (RTC protocol) negotiation and repackaging ability, realizing the distribution using the RTC protocol in the last mile, and finally achieving stronger RTC distribution ability, lower transmission latency and better jitter control without increasing the cost, greatly improving the user experience.

[0102] Further, as an optional implementation manner, the method further includes:

[0103] Receiving a signaling negotiation request sent by the first client device, where the signaling negotiation request carries a Session Description Protocol (SDP) description. Specifically, this step can be executed before step 301.

[0104] Here, it should be noted that the SDP description in the above steps usually includes: session information, session time, media type, media format, port number, transport protocol, transport bit rate, Internet Connectivity Establishment (ICE) information, etc. For example, an example of the SDP description is:

[0105]

[0106] c = IN IP4 0.0.0.0 / / Connection information of the audio stream, specifying the IP address as 0.0.0.0, indicating an unknown address.

[0107] Send a signaling negotiation response to the first client device, where the signaling negotiation response carries a response SDP description corresponding to the SDP description. Among them, the response SDP description includes Interactive Connectivity Establishment (ICE) information, and the ICE information is used to establish a connection for transmitting media streams between the server device and the first client device. In this step, the ICE information may include, for example, ICE candidate addresses.

[0108] As a specific example of the above optional implementation method, as Figure 5 shown, the following steps are included:

[0109] The RTC client creates a local SDP description;

[0110] The RTC client sends the SDP description to the RTC instance - signaling negotiation module;

[0111] The RTC instance - signaling negotiation module receives and creates a response SDP description;

[0112] The RTC instance - signaling negotiation module sends the response SDP description to the RTC client;

[0113] The RTC client receives the response SDP description and obtains the ICE candidate addresses;

[0114] So far, the signaling negotiation is completed. Subsequently, the RTC client and the server device will establish a connection according to the ICE candidate addresses in the SDP description for transmitting media streams.

[0115] Here, it should be noted that the above RTC client is equivalent to the aforementioned first client device, and the RTC instance - signaling negotiation module is equivalent to the module of the aforementioned server device.

[0116] Here, it should also be noted that in the low - latency live CDN, in order to achieve the characteristics of high availability, high disaster tolerance, and large capacity, all distribution nodes adopt a cluster containerization and distributed deployment mode. For the RTC distribution to be compatible with the existing deployment mode, a containerization deployment solution is also adopted in the edge stream - pulling nodes. On each host, in addition to the RTMP distribution instance, multiple RTC distribution instances can be deployed simultaneously, and each instance supports two functions: RTC signaling interaction (SDP) and media data distribution (RTP). Load balancing is achieved among multiple hosts through the Linux Virtual Server (LVS). In this way, while improving resource utilization, the RTC distribution capacity is also greatly enhanced.

[0117] However, when the server device (especially the edge pulling flow node) adopts the cluster containerization deployment solution, there will be a situation where the host for signaling negotiation is inconsistent with the host for media stream data distribution. For example, Figure 2 As shown in the figure, if the client first requests SDP interaction and accesses the RTC instance A1 on host A, after the signaling interaction is completed, the client obtains the ICE candidate address of the cluster media server from the SDP as the Virtual Internet Protocol (VIP) address. The backend servers corresponding to the VIP are host A and host B. When the client requests media data again, it may fall to the RTC instance B2 on host B. Since there is no SDP information of the client on the RTC instance B2, this will cause the RTC pulling flow to fail.

[0118] For the above situation where the landing points of the two requests for RTC pulling flow by the client are inconsistent, another optional implementation manner of this application includes:

[0119] Call the signaling storage interface to store the SDP description and the replied SDP description into the shared signaling memory.

[0120] That is to say, as Figure 6 shown in the figure, after each RTC instance completes the signaling interaction, it calls the signaling storage store interface to store the signaling of both parties into the shared signaling memory. When the media data request of the same user falls to the RTC instance on other hosts, this instance will first call the load interface of the shared signaling memory to query whether the user signaling already exists. If it exists, it will continue to provide the pulling flow service; otherwise, the user pulling flow fails. In this way, the problem of inconsistent landing points of RTC pulling flow in the cluster containerization deployment mode is solved. The signaling and media requests can be completely separated, and no matter which host and which RTC instance the request falls to, the server device can handle it normally. In this way, the compatibility problem between RTC signaling negotiation and the original cluster containerization deployment mode is solved, and at the same time, the RTC distribution ability is also greatly improved.

[0121] Among them, the signaling information structure in the shared signaling memory is as follows:

[0122] RtcUserConfig

[0123] {

[0124] string localSDP,

[0125] string remoteSDP,

[0126] string req,

[0127] int cid,

[0128] }

[0129] Among them, cid represents the user session ID, which is the unique identity of the user. req is the user request information. localSDP represents the server-side SDP, and remoteSDP represents the user-side SDP.

[0130] Based on the above optional implementation, step 302 includes:

[0131] Query the SDP description of the first client device. Here, when the server device is deployed in a cluster, this step is for the RTP instance that receives the signaling negotiation request to query whether there is a user signaling / SDP description of the first client device locally. If not, call the store interface of the shared signaling memory to query whether there is a user signaling / SDP description of the first client device in the shared signaling memory.

[0132] When the SDP description of the first client device is queried, perform protocol conversion and encapsulation on the first media stream data transmitted by the second client device using the second protocol according to the first protocol to obtain the second media stream data. That is to say, after determining that the signaling negotiation between the first client device and the server device is completed, the server device can provide a pull stream service for the first client device.

[0133] As a specific implementation, step 302 includes:

[0134] Demultiplex the video data packets in the first media stream data, and encapsulate the demultiplexed non-image data and Network Abstraction Layer Unit (NALU) into a first data packet related to the first protocol. Here, the non-image data includes, for example, Sequence Parameter Set (SPS) or Picture Parameter Set (PPS), etc.; the first data packet related to the first protocol is, for example, an RTP data packet; as Figure 7 shown, this step is: The server device demultiplexes the parsed video data packet (video) to obtain video data. Then, it encapsulates the SPS / PPS, Supplemental Enhancement Information (SEI), and NALU in the video data into an RTP data packet (the first data packet).

[0135] Demultiplex the audio data packets in the first media stream, transcode the demultiplexed Advanced Audio Coding (AAC) data stream, and encapsulate the transcoded data packets into second data packets related to the first protocol; as Figure 7 shown, this step is as follows: The server device demultiplexes the obtained audio data packets (video) to obtain AAC RAW, then transcodes AAC RAW into a lossy audio coding format (opus), and finally encapsulates it into RTP data packets (second data packets);

[0136] Among them, the second media stream data includes the first data packet and the second data packet. Here, as Figure 7 shown, after encapsulating the first data packet and the second data packet, write the first data packet and the second data packet into the User Datagram Protocol (UDP) service (Server) so that the UDP Server can provide a pulling stream service for the first client device later and send the first data packet and the second data packet to the first client device.

[0137] In addition, B-frames are usually used in video compression. B-frames further improve the compression efficiency by performing bidirectional motion estimation on the previous and subsequent frames. Introducing B-frames can reduce the bit rate and save bandwidth, but it also increases the decoding time. In the embodiments of the present application, in order to obtain lower latency, CDN providers will filter out B-frames when distributing RTC streams. The RTC distribution service in the embodiments of the present application adds the forwarding of hevc / h264 B-frame data during transcapsulation (in Figure 7 the video data processing link, all NALUs corresponding to B-frames will be parsed out for subsequent encapsulation and forwarding of RTP data packets). Among them, whether to forward B-frames depends on the pushing stream end (the second client device). If the source stream includes B-frames, the B-frames will be forwarded to the RTC pulling stream end (the first client device), otherwise, they will not be forwarded. This strategy allows for flexible configuration under different requirements, supporting both scenarios that require lower latency and allowing users to choose whether to use B-frames according to network conditions and device performance.

[0138] Here, it should be noted that, as mentioned above, the existing low-latency live streaming does not support packet loss and stuttering well in a weak network environment. Therefore, in the embodiments of the present application, it is proposed that the first client device can have a pulling stream method related to the first protocol (RTC protocol). At this time, the first client device needs to integrate the corresponding Software Development Kit (SDK), so that the first client device can simply and quickly implement the corresponding pulling stream. Specifically, after the first client device selects the pulling stream method related to the first protocol, the corresponding media stream data is transmitted to the first client device through the UDP protocol. Among them, the advantage of UDP transmission is that no connection needs to be established and the time consumption is small, but the disadvantage is that the transmission is unreliable and packet loss is likely to occur, resulting in a frozen or stuttering video during playback. In view of this situation, the embodiments of the present application further provide an optional implementation method. After step 303, the method further includes:

[0139] (1) Receiving a first notification message sent by the first client device, where the first notification message carries first indication information for indicating second media stream data not received by the first client device. Here, the first client device may send the first notification message to the server device when it has not received the second media stream data sent by the server device within a first time period.

[0140] (2) Obtaining the unreceived second media stream data in the buffer of the server device according to the first indication information.

[0141] (3) Encapsulating the unreceived second media stream data into a retransmission data packet, where the retransmission data packet includes an identifier of an initial transmission data packet of the unreceived second media stream data.

[0142] Here, it should be noted that since the retransmission data packet and other original RTP data packets (other initial transmission data packets) are sent mixed together, there is a situation where the first client device cannot distinguish whether the received data packet is a Negative-Acknowledgement (NACK) retransmission data packet or an original RTP data packet, resulting in the packet loss rate feedback by the first client device being lower than the actual value. Among them, when the packet loss rate decreases, the retransmission data packet also decreases, but the stuttering problem still exists. Therefore, in order to enable the first client device to distinguish between the original RTP data packet and the retransmission data packet, as Figure 9 shown, 2 bytes are added before the payload of the retransmission data packet in this step (the RTP sequence number occupies at most 2 bytes, and the represented range is 0 to 2 16-1), which is used to record the sequence number of the original RTP packet. Among them, the RTP header field carries the sequence number of the retransmitted packet (PT = 97), and the original RTP number field carries the sequence number of the original packet corresponding to the retransmitted packet (PT = 96). In this way, the first client device can accurately identify and process the retransmitted packet, which helps to improve the adaptability of the system to the network condition, reduce unnecessary retransmissions, and improve the transmission efficiency.

[0143] (4) Send the retransmitted packet to the first client device.

[0144] Next, in combination with Figure 8 , the implementation process of the above optional implementation manner will be described:

[0145] First, the server (corresponding to the aforementioned server device) sends an RTP packet with a sequence number of PT = 96 to the client (corresponding to the aforementioned first client device);

[0146] Second, when the client does not receive the RTP packet with a sequence number of PT = 96, the client sends a Real-Time Control Protocol (RTCP) Feedback NACK (corresponding to the aforementioned notification message) to the server;

[0147] Third, the server sends a retransmitted packet (RTP PT = 97 RTX) to the client, where the sequence number of the retransmitted packet is PT = 97.

[0148] That is to say, in order to solve the problem of packet loss, the above optional implementation manner adopts a NACK-based retransmission scheme. The first client device notifies the server device of the unreceived packet through NACK. After receiving the NACK notification, the server device finds the lost packet from the buffer, encapsulates it into an RTX packet (retransmitted packet) and sends it to the first client device together with the original RTP. Among them, in order to enable the first client device to distinguish between the original RTP packet and the retransmitted packet, 2 bytes (used to record the sequence number of the original packet) are added before the payload of the retransmitted packet in the above optional implementation manner. In this way, the first client device can more accurately identify and process the retransmitted packet, which helps to improve the adaptability of the system to the network condition, reduce unnecessary retransmissions, and improve the transmission efficiency.

[0149] In addition, if the network latency is large, resulting in slow data packet transmission, the first client device may mistakenly think that a packet is lost and frequently send NACK retransmission messages (the aforementioned first notification messages) to the server device. The server device retransmits a large number of data packets, which will also increase network congestion. To avoid this situation, in the embodiments of the present application, the first client device sends a notification message to the server device only when it does not receive the second media stream data within the first duration. Among them, to calculate the first duration, the embodiments of the present application further provide an optional implementation method, including the following steps:

[0150] Receive the RTCP extended report XR (Extended Report, XR) data packets periodically sent by the first client device. The RTCP XR data packets include the first timestamp when the first client device sends the RTCP XR data packets;

[0151] Send the updated RTCP XR data packets to the first client device. Among them, the updated RTCP XR data packets include the processing delay of the server device for the RTCP XR data packets, and the processing delay is used to determine the timing for the first client device to send the first notification message.

[0152] In addition, it should be noted that in a weak network environment, packet loss retransmission sometimes cannot completely solve the problem of packet loss and freezing. If key audio and video information is lost, it will cause the first frame to appear slowly. To address the problems of unsmooth playback and slow startup caused by high packet loss rate scenarios, the embodiments of the present application propose a retransmission scheme. An optional implementation method corresponding to this scheme includes the following steps:

[0153] Receive the second notification message sent by the first client device. The second notification message carries second indication information, and the second indication information is used to indicate the second media stream data that the first client device has not received;

[0154] Determine the packet loss rate according to the second indication information;

[0155] In the case where the packet loss rate is greater than or equal to the packet loss rate threshold, obtain target second media stream data in the buffer of the server device. The target second media stream data carries the key information of the video;

[0156] Send the target second media stream data to the first client device.

[0157] In the above optional implementation manner of the present application, when it is determined that the packet loss rate reaches the packet loss rate threshold, redundant transmission is performed on key information (such as dtls handshake packets, pps / sps information). Specifically, when the packet loss rate exceeds the packet loss rate threshold (such as 40%), active 3 (configurable) times retransmission can be performed to ensure that key information is not lost. In this way, it is possible to avoid the situation where the first frame appears slowly due to the loss of key audio and video information.

[0158] An embodiment of the present application further provides a media stream processing method, which is applied to a first client device, such as Figure 11 As shown, the method includes:

[0159] Step 1101, send a first request to the server device, where the first request is used to request media stream data.

[0160] Specifically, the above step is: the first client device requests the media stream data of the second client device from the server device. Taking the live broadcast scenario as an example, the first client device is the terminal device of the live broadcast audience, and the second client device is the device of the live broadcast anchor.

[0161] Step 1102, receive the second media stream data sent by the server device, where the second media stream data is the media stream data obtained by the server device through protocol conversion and encapsulation of the first media stream data according to the first protocol, and the first media stream data is the media stream data sent by the second client device to the server device based on the second protocol.

[0162] In the above step, the first protocol is, for example, the RTC protocol, and the second protocol is, for example, the RTMP protocol.

[0163] Step 1103, decode the second media stream data, and play the video corresponding to the decoding result.

[0164] In the media stream processing method of the embodiment of the present application, first, the first client device sends a first request to the server device, where the first request is used to request media stream data; secondly, receive the second media stream data sent by the server device, where the second media stream data is the media stream data obtained by the server device through protocol conversion and encapsulation of the first media stream data according to the first protocol, and the first media stream data is the media stream data sent by the second client device to the server device based on the second protocol; finally, decode the second media stream data, and play the video corresponding to the decoding result. In this way, it is realized that the media stream data is transmitted using the first protocol with lower latency in the last mile of the low-latency live broadcast link, so that a lower transmission latency can be obtained without increasing the cost.

[0165] In addition, after step 1101, the method further includes:

[0166] When the pulling stream mode selected by the first client device is related to the second protocol, receiving third media stream data transmitted by the server device to the first client device based on the second protocol or a protocol related to the second protocol, where the third media stream data is media stream data obtained by the server device performing streaming processing on the first media stream data.

[0167] Further, as an optional implementation manner, the method further includes:

[0168] Sending a signaling negotiation request to the server device, where the signaling negotiation request carries an SDP description; specifically, this step may be executed before step 1101; among them, the SDP description in this step generally includes: session information, session time, media type, media format, port number, transport protocol, transport bit rate, ICE information, etc.;

[0169] Receiving a signaling negotiation response sent by the server device, where the signaling negotiation response carries a response SDP description corresponding to the SDP description, and the response SDP description includes ICE information, and the ICE information is used to establish a connection for transmitting media streams between the server device and the first client device. Here, the ICE information may include ICE candidate addresses.

[0170] Here, it should be noted that the above optional implementation manner may specifically be implemented after the first client device selects a pulling stream mode related to the first protocol according to factors such as network conditions, user device performance, current playback quality, and user interaction requirements. When selecting the pulling stream mode, for example, if the user is currently watching a live e-commerce broadcast and has high requirements for interactivity and real-time performance, the client can automatically select RTC access (a pulling stream mode related to the first protocol); if the network quality on the user side is poor, the client can also preferentially select RTC access (a pulling stream mode related to the first protocol); if the user is watching a live performance broadcast and does not have very high requirements for interactivity and real-time performance, RTMP / HTTP-FLV access (a pulling stream mode related to the second protocol or a protocol related to the second protocol) can be preferentially selected. In this way, the client automatically selects different access modes according to different scenarios, and the user has no perception in terms of experience, realizing a smooth switch from low-latency live broadcast to RTC live broadcast. In this way, both the pushing and transcoding capabilities of the low-latency live broadcast CDN are reused, and the CDN's powerful edge distribution capabilities are also available, which can support tens of millions of RTC users to be online at the same time, meeting requirements such as low cost, low latency, large capacity, and high compatibility.

[0171] Further, as an optional implementation manner, the method further includes:

[0172] In the case that the second media stream data is not received within the first duration, send a first notification message to the server device, where the first notification message carries first indication information for indicating the second media stream data not received by the first client device;

[0173] Receive a retransmission data packet sent by the server device, where the retransmission data packet includes an identifier of an original transmission data packet of the unreceived second media stream data; here, including the identifier of the original transmission data packet corresponding to the retransmission data packet in the retransmission data packet can avoid the situation where the first client device cannot distinguish whether the received data packet is a NACK retransmission data packet or an original RTP data packet, thereby avoiding the problem of reduced retransmission data packets and jitter caused by the packet loss rate feedback by the first client device being lower than the actual value;

[0174] Decode the second media stream data in the retransmission data packet and play the video corresponding to the decoding result.

[0175] In the above optional implementation manner, a NACK-based retransmission scheme is adopted. The first client device notifies the server device of the unreceived data packets through NACK. After receiving the NACK notification, the server device finds the lost data packets from the buffer, encapsulates them into RTX packets (retransmission data packets) and sends them to the first client device together with the original RTP. Among them, in order to enable the first client device to distinguish between the original RTP data packets and the retransmission data packets, 2 bytes are added before the payload of the retransmission data packet (for recording the original packet sequence number) in the above optional implementation manner, so that the first client device can more accurately identify and process the retransmission packets, which helps to improve the adaptability of the system to the network condition, reduce unnecessary retransmissions, and improve the transmission efficiency.

[0176] In the above optional implementation manner, it can avoid the situation where due to large network latency, the data packet transmission is slow, and the first client device mistakenly thinks that there are packet losses and will frequently send NACK retransmission messages (the aforementioned first notification message) to the server device, and the server device retransmits a large number of data packets, increasing network congestion.

[0177] Further, as an optional implementation manner, the method further includes:

[0178] In the case that the second media stream data is not received within the first duration, send a second notification message to the server device, where the second notification message carries second indication information for indicating the second media stream data not received by the first client device;

[0179] Receive the target second media stream data sent by the server device; where the target second media stream data carries key information of the video.

[0180] In the above optional implementation, the problem that packet loss retransmission sometimes cannot completely solve the packet loss and lag problem in a weak network environment is solved. If key audio and video information is lost, it will cause the problem of slow start of the first frame, avoiding the problems of unsmooth playback and slow start caused by high packet loss rate scenarios.

[0181] To determine the aforementioned first duration, an optional implementation is further provided in an embodiment of the present application, including the following steps:

[0182] Periodically send RTCP XR packets to the server device, where the RTCP XR packets include a first timestamp when the first client device sends the RTCP XR packets;

[0183] Receive the updated RTCP XR packets sent by the server device, where the updated RTCP XR packets include the processing delay of the server device for the RTCP XR packets;

[0184] Determine the first duration according to the second timestamp when the first client device receives the updated RTCP XR packets, the first timestamp, and the processing delay.

[0185] As a specific implementation, determining the first duration according to the second timestamp when the first client device receives the updated RTCP XR packets, the first timestamp, and the processing delay includes:

[0186] Calculate the difference between the second timestamp and the sum of the first timestamp and the processing delay to obtain the round-trip delay RTT of the RTCP XR packets;

[0187] Calculate the first duration according to the RTT of multiple adjacent RTCP XR packets.

[0188] A specific example of the above optional implementation and the above specific implementation is as follows:

[0189] First, the client device periodically sends RTCP XR packets (XR packets containing specific XR report blocks) to the server device; among them, the RTCP XR packets include a timestamp t1 (first timestamp) when the first client device sends;

[0190] Secondly, the server device receives the RTCP XR data packet sent by the first client device and records the timestamp t2 of the reception. After the server device processes the RTCP XR data packet, it records the timestamp as t3. Then the processing delay of this XR packet at the server is △t=t3-t2. The server writes △t to another field of the report block in the RTCP XR packet (updated RTCP XR).

[0191] Again, the server device sends the updated RTCP XR data packet to the first client device;

[0192] Afterwards, the first client device receives the updated RTPC XR data packet sent by the server device and records the receiving timestamp as t4 (second timestamp), and parses the original sending time t1 and the server processing delay △t in the RTCP XR data packet. Then, the round-trip delay RTT of this RTCP XR packet on the network = t4-t1-△t (all times are synchronized);

[0193] Finally, the first client device calculates the sliding average RTT (first duration). In order to reduce instantaneous fluctuations and improve the stability of the measurement, the first client device uses a sliding average method to calculate the first duration so as to reduce the sensitivity to instantaneous changes. Figure 10 As shown, by maintaining a sliding window of size N, storing the calculated values of the most recent N (default 10, configurable) RTTs (RTT calculation method is as described in the previous steps), the sliding average RTT calculation formula is as follows:

[0194]

[0195] Among them, i is the sequence number of the sliding window, RTTi is the round-trip delay of the i-th packet, and Wi is the weight value corresponding to the delay (the RTT weight closest to the current time defaults to 1.0, and decreases by 0.1 (configurable) in sequence, for example, 1.0, 0.9, 0.8, ...).

[0196] The media stream processing method according to the embodiments of the present application, on the one hand, utilizes the nodes and networks of the existing low-latency live CDN system. By performing transmission protocol conversion at the live edge nodes and using RTC distribution for the last mile, it fully reuses the resources of the existing system. While having the powerful distribution ability of the CDN, it maximally utilizes the existing infrastructure. New users can either choose the original RTMP / HTTP-FLV access method or the pure RTC access method. This not only enriches the basic functions of the low-latency distribution system but also provides users with more autonomous and flexible choices. On the other hand, based on its own container cloud platform, a signaling separation scheme is adopted to achieve clustered and distributed deployment, with high availability and high disaster tolerance, and can handle instantaneous and massive connection requests. Even in the event of a machine room-level failure, high availability can be ensured. On the other hand, the RTC distribution technology is used for the last mile, and the end-to-end distribution delay is shortened to about 1 s. By real-time sensing the network bandwidth and packet loss situation, smooth playback can still be achieved under 40% packet loss, significantly improving the user's viewing experience.

[0197] An embodiment of the present application further provides a media stream processing device, which is applied to a server device, such as Figure 12 shown, the device includes:

[0198] A first receiving module 1201, configured to receive a first request sent by a first client device, where the first request is used to request media stream data;

[0199] A processing module 1202, configured to, when the pulling stream method selected by the first client device is related to a first protocol, in response to the first request, perform protocol conversion and encapsulation on first media stream data sent by a second client device based on a second protocol according to the first protocol, to obtain second media stream data;

[0200] A first sending module 1203, configured to send the second media stream data to the first client device.

[0201] Further, the device further includes:

[0202] A second receiving module, configured to receive a signaling negotiation request sent by the first client device, where the signaling negotiation request carries a Session Description Protocol (SDP) description;

[0203] A second sending module, configured to send a signaling negotiation response to the first client device, where the signaling negotiation response carries a response SDP description corresponding to the SDP description, and the response SDP description includes Interactive Connectivity Establishment (ICE) information, and the ICE information is used to establish a connection for transmitting media streams between the server device and the first client device.

[0204] Optionally, the processing module 1202 is specifically configured to:

[0205] Query the SDP description of the first client device;

[0206] When the SDP description of the first client device is queried, perform protocol conversion and encapsulation on the first media stream data transmitted by the second client device according to the first protocol to obtain second media stream data.

[0207] Furthermore, the device further includes:

[0208] A calling module, configured to call a signaling storage interface to store the SDP description and the response SDP description into a shared signaling memory.

[0209] Furthermore, the processing module 1202 includes:

[0210] A first decapsulation sub-module, configured to decapsulate video data packets in the first media stream data, and encapsulate the decapsulated non-image data and network abstraction layer unit NALU into a first data packet related to the first protocol;

[0211] A second decapsulation sub-module, configured to decapsulate audio data packets in the first media stream, transcode the decapsulated advanced audio coding AAC data stream, and encapsulate the transcoded data packet into a second data packet related to the first protocol;

[0212] Wherein, the second media stream data includes the first data packet and the second data packet.

[0213] Optionally, the device further includes:

[0214] A third receiving module, configured to receive a first notification message sent by the first client device, where the first notification message carries first indication information for indicating second media stream data not received by the first client device;

[0215] A first obtaining module, configured to obtain the unreceived second media stream data in the buffer of the server device according to the first indication information;

[0216] An encapsulation module, configured to encapsulate the unreceived second media stream data into a retransmission data packet, where the retransmission data packet includes an identifier of an initial transmission data packet of the unreceived second media stream data;

[0217] A third sending module, configured to send the retransmission data packet to the first client device.

[0218] Optionally, the device further includes:

[0219] A fourth receiving module, configured to receive Real - Time Transport Control Protocol (RTCP) Extended Reports (XR) data packets periodically sent by the first client device, where the RTCP XR data packets include a first timestamp when the first client device sends the RTCP XR data packets;

[0220] A fourth sending module, configured to send updated RTCP XR data packets to the first client device, where the updated RTCP XR data packets include a processing delay of the server device for the RTCP XR data packets, and the processing delay is used to determine the timing for the first client device to send the first notification message.

[0221] Optionally, the apparatus further includes:

[0222] A fifth receiving module, configured to receive a second notification message sent by the first client device, where the second notification message carries second indication information, and the second indication information is used to indicate second media stream data not received by the first client device;

[0223] A determination module, configured to determine a packet loss rate according to the second indication information;

[0224] A second obtaining module, configured to obtain target second media stream data in the buffer of the server device when the packet loss rate is greater than or equal to a packet loss rate threshold, where the target second media stream data carries key information of a video;

[0225] A fifth sending module, configured to send the target second media stream data to the first client device.

[0226] It should be noted here that the media stream processing apparatus provided in the embodiments of the present application can implement all the method steps implemented by the media stream processing method embodiments applied to the server device, and can achieve the same technical effects. Here, the same parts and beneficial effects as those in the method embodiments will not be specifically described in this embodiment.

[0227] Embodiments of the present application further provide a media stream processing apparatus, which is applied to a first client device, as Figure 13 shown. The apparatus includes:

[0228] A first sending module 1301, configured to send a first request to a server device, where the first request is used to request media stream data;

[0229] The first receiving module 1302 is configured to receive the second media stream data sent by the server device, where the second media stream data is the media stream data obtained by the server device through protocol conversion and encapsulation of the first media stream data according to a first protocol, and the first media stream data is the media stream data sent by a second client device to the server device based on a second protocol;

[0230] The first decoding module 1303 is configured to decode the second media stream data and play the video corresponding to the decoding result.

[0231] Optionally, the apparatus further includes:

[0232] A second sending module, configured to send a signaling negotiation request to the server device, where the signaling negotiation request carries an SDP description;

[0233] A second receiving module, configured to receive a signaling negotiation response sent by the server device, where the signaling negotiation response carries a responding SDP description corresponding to the SDP description, and the responding SDP description includes ICE information, and the ICE information is used to establish a connection for transmitting media streams between the server device and the first client device.

[0234] Optionally, the apparatus further includes:

[0235] A third sending module, configured to send a first notification message to the server device when the second media stream data is not received within a first duration, where the first notification message carries first indication information, and the first indication information is used to indicate the second media stream data not received by the first client device;

[0236] A third receiving module, configured to receive a retransmission data packet sent by the server device, where the retransmission data packet includes an identifier of an initial transmission data packet of the second media stream data not received;

[0237] A second decoding module, configured to decode the second media stream data in the retransmission data packet and play the video corresponding to the decoding result.

[0238] Optionally, the apparatus further includes:

[0239] A fourth sending module, configured to send a second notification message to the server device when the second media stream data is not received within a first duration, where the second notification message carries second indication information, and the second indication information is used to indicate the second media stream data not received by the first client device;

[0240] A fourth receiving module, configured to receive the target second media stream data sent by the server device; wherein, the target second media stream data carries key information of the video.

[0241] Optionally, the apparatus further includes:

[0242] A fifth sending module, configured to periodically send RTCP XR packets to the server device, where the RTCP XR packets include a first timestamp when the first client device sends the RTCP XR packets;

[0243] A fifth receiving module, configured to receive the updated RTCP XR packets sent by the server device, where the updated RTCP XR packets include a processing delay of the server device for the RTCP XR packets;

[0244] A determining module, configured to determine the first duration according to a second timestamp when the first client device receives the updated RTCP XR packets, the first timestamp, and the processing delay.

[0245] Optionally, the determining module includes:

[0246] A first calculating sub-module, configured to calculate a difference between the second timestamp and the sum of the first timestamp and the processing delay to obtain a round-trip delay RTT of the RTCP XR packets;

[0247] A second calculating sub-module, configured to calculate the first duration according to the RTTs of multiple adjacent RTCP XR packets.

[0248] It should be noted here that the above media stream processing apparatus provided in the embodiments of the present application can implement all the method steps implemented in the embodiments of the media stream processing method applied to the first client device, and can achieve the same technical effects. The same parts and beneficial effects as those in the method embodiments are not specifically described in this embodiment.

[0249] An embodiment of the present application further provides an electronic device, including a transceiver 1410, a processor 1400, a memory 1420, and a program stored on the memory 1420 and executable on the processor 1400; wherein, when the processor 1400 executes the program, it implements the media stream processing method applied to the server device or the first client device as described above.

[0250] The transceiver 1410 is configured to receive and send data under the control of the processor 1400.

[0251] Wherein, in Figure 14Among them, the bus architecture may include any number of interconnected buses and bridges, specifically, various circuits of one or more processors represented by processor 1400 and memory represented by memory 1420 are linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, etc., which are well known in the art, so they will not be further described herein. The bus interface provides an interface. The transceiver 1410 may be multiple components, that is, including a transmitter and a receiver, and provides a unit for communicating with various other devices on the transmission medium. For different devices, the processor 1400 is responsible for managing the bus architecture and general processing, and the memory 1420 may store data used by the processor 1400 when executing operations.

[0252] The embodiments of the present application also provide a readable storage medium, on which a program is stored. When the program is executed by a processor, it implements each process of the advertising placement method embodiments applied to the server device or the first client device as described above, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here. Among them, the readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0253] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for executing the methods described in each embodiment of the present application.

[0254] Therefore, the embodiments of the present application also provide a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the steps in the advertising placement method applied to the server device or the first client device as described above, and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0255] The above is the preferred embodiment of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle described in the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A media stream processing method, characterized in that, Applied to a server device, including: Receiving a first request sent by a first client device, where the first request is used to request media stream data; When the pulling method selected by the first client device is related to a first protocol, in response to the first request, according to the first protocol, performing protocol conversion and encapsulation on first media stream data sent by a second client device based on a second protocol to obtain second media stream data; Sending the second media stream data to the first client device.

2. The method according to claim 1, wherein The method further includes: Receiving a signaling negotiation request sent by the first client device, where the signaling negotiation request carries a Session Description Protocol (SDP) description; Sending a signaling negotiation response to the first client device, where the signaling negotiation response carries a response SDP description corresponding to the SDP description, and the response SDP description includes Interactive Connectivity Establishment (ICE) information, and the ICE information is used to establish a connection for transmitting media stream between the server device and the first client device.

3. The method according to claim 2, wherein In response to the first request, according to the first protocol, performing protocol conversion and encapsulation on first media stream data transmitted by a second client device using a second protocol to obtain second media stream data, including: Querying the SDP description of the first client device; When the SDP description of the first client device is queried, according to the first protocol, performing protocol conversion and encapsulation on first media stream data transmitted by a second client device using a second protocol to obtain second media stream data.

4. The method according to claim 1 or 3, characterized in that, According to the first protocol, performing protocol conversion and encapsulation on first media stream data sent by a second client device based on a second protocol to obtain second media stream data, including: Decapsulating video data packets in the first media stream data, and encapsulating the decapsulated non-image data and Network Abstraction Layer Unit (NALU) into first data packets related to the first protocol; Decapsulating audio data packets in the first media stream, transcoding the decapsulated Advanced Audio Coding (AAC) data stream, and encapsulating the transcoded data packets into second data packets related to the first protocol; Wherein, the second media stream data includes the first data packets and the second data packets.

5. The method according to claim 1, wherein The method further includes: Receiving a first notification message sent by the first client device, where the first notification message carries first indication information, and the first indication information is used to indicate second media stream data not received by the first client device; According to the first indication information, obtaining the un-received second media stream data in the buffer of the server device; Encapsulating the un-received second media stream data into a retransmission data packet, where the retransmission data packet includes an identifier of an initial transmission data packet of the un-received second media stream data; Sending the retransmission data packet to the first client device.

6. A media stream processing method, characterized in that, Applied to a first client device, including: Sending a first request to a server device, where the first request is used to request media stream data; Receive the second media stream data sent by the server device, where the second media stream data is the media stream data obtained by the server device through protocol conversion and encapsulation of the first media stream data according to the first protocol, and the first media stream data is the media stream data sent by the second client device to the server device based on the second protocol; Decode the second media stream data and play the video corresponding to the decoding result.

7. The method according to claim 6, characterized in that, The method further includes: Send a signaling negotiation request to the server device, where the signaling negotiation request carries an SDP description; Receive the signaling negotiation response sent by the server device, where the signaling negotiation response carries a response SDP description corresponding to the SDP description, and the response SDP description includes ICE information, and the ICE information is used to establish a connection for transmitting media streams between the server device and the first client device.

8. The method according to claim 6, characterized in that The method further includes: In the case that the second media stream data is not received within the first duration, send a first notification message to the server device, where the first notification message carries first indication information, and the first indication information is used to indicate the second media stream data not received by the first client device; Receive the retransmission data packet sent by the server device, where the retransmission data packet includes the identifier of the initial transmission data packet of the second media stream data not received; Decode the second media stream data in the retransmission data packet and play the video corresponding to the decoding result.

9. An electronic device, characterized in that, Includes: A transceiver, a processor, a memory, and a computer program stored on the memory and executable on the processor, where when the computer program is executed by the processor, it implements the media stream processing method according to any one of claims 1 to 5, or implements the media stream processing method according to any one of claims 6 to 8.

10. A readable storage medium, characterized in that A computer program is stored on the readable storage medium, and when the computer program is executed by the processor, it implements the media stream processing method according to any one of claims 1 to 5, or implements the media stream processing method according to any one of claims 6 to 8.

Images