A method for implementing live broadcast, edge node, electronic device and storage medium

Abstract

The embodiments of the present application relate to the field of live broadcast technology, and in particular to a method for implementing live broadcast, an edge node, an electronic device, and a storage medium. The method for implementing live broadcast, applied to an edge node, includes: receiving video frames sent by a central node; starting from the video frames of type I frame, caching the received video frames; after receiving a content request from a client, sending the cached video frames to the client based on the RTC protocol, so that the client can present the received video frames to the user; after sending the cached video frames to the client, continue to receive the video frames sent by the central node and send the currently received video frames to the client based on the RTC protocol. This reduces the live broadcast delay without affecting the user experience, so as to meet the real-time requirements of RTC.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of live broadcast technology, and in particular to a method for implementing live broadcast, an edge node, an electronic device, and a storage medium. Background Art

[0002] Live streaming is the transmission of audio and video content of a performance over the internet in real time. As live video becomes more popular, live streaming has become an important part of many businesses and marketing strategies. Live streaming can be used for everything from live events (matches), providing customer service, and holding webinars.

[0003] Live streaming usually involves two parts: push streaming and pull streaming. Push streaming refers to pushing the audio and video stream containing the live content to the server, and pull streaming refers to the user pulling the audio and video stream from the server.

[0004] When pulling the stream, the Content Delivery Network (CDN) is usually used for acceleration to improve the transmission efficiency of the audio and video streams, that is, the commonly used live broadcasts based on the Real Time Messaging Protocol (RTMP), the live broadcasts based on the Hyper Text Transfer Protocol-Flash Video (HTTP-FLV), the live broadcasts based on the Hyper Text Transfer Protocol Live Streaming (HLS), and other ordinary live broadcasts.

[0005] With the development of real-time communication (RTC), in order to further improve the transmission efficiency of audio and video streams, people gradually began to consider applying RTC to live broadcast, that is, RTC-based live broadcast. Among them, RTC-based live broadcast is usually based on RTC, and RTC-related protocols are configured on edge nodes, so that edge nodes can convert the audio and video frames of ordinary live broadcasts sent by the central node received into RTC audio and video frames, and then send the cached audio and video frames to the client based on the RTC protocol.

[0006] However, when implementing RTC-based live broadcast, how to meet the real-time requirements of RTC without affecting the user experience remains to be solved. Summary of the invention

[0007] The embodiments of the present application provide a live broadcast implementation method, an edge node, an electronic device, and a storage medium, which are conducive to reducing the live broadcast delay without affecting the user experience to meet the real-time requirements of RTC.

[0008] Reduce the first screen time of live content presented by the client in RTC-based live broadcast.

[0009] According to some embodiments of the present application, on the one hand, an embodiment of the present application provides a method for implementing live broadcast, which is applied to an edge node, and the method includes: receiving video frames sent by a central node; starting from the video frames of type I frame, caching the received video frames; after receiving a content request from the client, sending the cached video frames to the client based on the RTC protocol, so that the client can present the received video frames to the user.

[0010] According to some embodiments of the present application, on the one hand, the embodiments of the present application also provide an edge node, including: a receiving module, used to receive video frames sent by a central node; a caching module, used to cache the received video frames starting from the video frames of type I frame; a sending module, used to send the cached video frames to the client after receiving a content request from the client, so that the client can present the received video frames to the user.

[0011] According to some embodiments of the present application, on the one hand, an embodiment of the present application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the live broadcast implementation method as described above.

[0012] According to some embodiments of the present application, on the one hand, the embodiments of the present application further provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the live broadcast implementation method as described above.

[0013] The technical solution provided by the embodiments of the present application has at least the following advantages:

[0014] After receiving the video frame sent by the central node, the video is cached starting from the video frame of type I frame, so the starting video frame is I frame. The I frame is a frame that can be decoded independently, that is, after the edge node sends the cached video frame to the client based on the RTC protocol, the starting video frame can be directly decoded and played by the client, without waiting for the I frame to assist in the decoding and playback of the starting video frame. After that, the video frame sent by the central node continues to be received and the currently received video frame is sent to the client based on the RTC protocol, that is, the video frame is sent to the client while receiving the video frame sent by the central node, so that the client can decode and play the currently received video frame based on the video frame sent to the client before, and realize the real-time playback of the live screen, which meets the real-time requirements of RTC, and will not be unable to decode and affect the user experience due to the lack of the previous video frame of type I frame in the received video frame. It is achieved to reduce the live broadcast delay without affecting the user experience to meet the real-time requirements of RTC. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0016] Figure 1 It is a schematic diagram of the structure of the network where the edge node is located provided in an embodiment of the present application;

[0017] Figure 2 is a flow chart of a method for implementing live broadcasting provided in an embodiment of the present application;

[0018] Figure 3 It is a flowchart of a method for implementing live broadcasting provided in an embodiment of the present application, including a step of clearing a video frame cached before a k-th video frame of type I frame;

[0019] Figure 4 It is a flowchart of a method for implementing live broadcasting provided in an embodiment of the present application, including a step of determining whether the number of currently cached video frames has reached a preset value;

[0020] Figure 5 It is a flowchart of a method for implementing live broadcasting including a step of caching audio frames provided in an embodiment of the present application;

[0021] Figure 6 It is an interactive flow chart of a method for implementing live broadcasting provided in an embodiment of the present application;

[0022] Figure 7 is a schematic diagram of the structure of an edge node provided in an embodiment of the present application;

[0023] Figure 8 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0024] As described in the background art, when implementing RTC-based live broadcast, satisfying the real-time requirements of RTC without affecting user experience remains to be solved.

[0025] After analysis, it was found that the reason for the above problem is that in order to make the client streaming and playing more smoothly, live broadcasts based on RTMP, HTT-FLV, HLS, etc. generally cache multiple image groups (Group Of Picture, GOP) in the central node or edge node, so that when the client pulls the stream, multiple GOPs are sent to the client at one time. When implementing live broadcast based on RTC, if multiple GOPs are still cached according to the aforementioned ordinary live broadcast solution, the client's picture delay time will be too long, the real-time performance will be poor, and the high real-time requirements of RTC cannot be met. Therefore, in order to meet the real-time requirements of RTC, the edge node cannot cache multiple GOPs. However, if the edge node sends the data frame to the client as soon as it receives the data frame, the problem of too long first screen time on the client will occur, which will seriously reduce the user experience.

[0026] Among them, after further analysis, it was found that the reason why the first screen time process occurs when the edge node receives the data frame and sends the data frame to the client is:

[0027] The timing of the client's request for live broadcast is uncertain, so the type of the starting video frame received by the edge node from the central node can be any of I frame, P frame and B frame. Regardless of whether the starting audio and video frame type in the received video frame is I frame, P frame or B frame, the edge node will send the received video frame to the client based on the RTC protocol.

[0028] However, if the type of the starting audio and video frame is a P frame or a B frame, since the P frame and the B frame cannot be decoded independently, it will be necessary to wait for the subsequent audio and video frame of type I frame before the video frame can be decoded and played. In other words, the first screen time will be very long.

[0029] Moreover, the probability of B-frame and P-frame in audio and video frames is significantly higher than that of I-frame, so the type of the starting audio and video frame is most likely to be P-frame or B-frame. That is, when live broadcast is implemented, the above-mentioned long first screen time and low user experience are likely to occur.

[0030] In addition, it is impossible to limit the time when users join the live broadcast, otherwise it will seriously affect the user experience.

[0031] To solve the above problems, an embodiment of the present application provides a live broadcast implementation method, which is applied to an edge node, and the method includes: receiving video frames sent by a central node; starting from video frames of type I frame, caching the received video frames; after receiving a content request from a client, sending the cached video frames to the client, so that the client can present the received video frames to the user; after sending the cached video frames to the client, continuing to receive the video frames sent by the central node and sending the currently received video frames to the client based on the RTC protocol.

[0032] The technical solution provided by the embodiment of the present application is that in the live broadcast scenario based on RTC, when the edge node caches the video frame, it no longer starts from the starting video frame sent by the central node, but starts from the received I frame, that is, the P frame and B frame before the I frame will not be cached, therefore, the starting video frame cached by the edge node and sent to the client based on the RTC protocol is an I frame, so based on the characteristic that the I frame can be independently decoded, the client can directly decode and play the starting video frame, therefore, when implementing the live broadcast based on RTC, there will be no problem that the starting video frame received by the client cannot be directly decoded, thereby avoiding the problem that the first screen time of the client presenting the live content is too long due to the inability to decode the starting video frame, and will not affect the user experience. Furthermore, after this, continue to receive the video frame sent by the central node and send the currently received video frame to the client based on the RTC protocol, that is, the edge node receives the video frame sent by the central node while sending the video frame to the client, so that the client can decode and play the currently received video frame based on the previously received video frame of the type of I frame, realize the real-time playback of the live screen, and meet the real-time requirements of RTC. It reduces the live broadcast delay without affecting the user experience, thus meeting the real-time requirements of RTC.

[0033] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented.

[0034] The division of the following embodiments is for the convenience of description and shall not constitute any limitation on the specific implementation of the present application. The embodiments may be combined with each other and referenced to each other without contradiction.

[0035] On the one hand, the embodiment of the present application provides a method for implementing live broadcasting, which is applied to an edge node. The edge node is in a state such as Figure 1The network shown in the figure includes a central node and other edge nodes on the basis of the edge nodes, and each edge node also corresponds to at least one client. In the process of live streaming, the request for audio and video is issued by the client and first responded to by the edge node. If the edge node does not cache the audio and video frames, the edge node requests the central node; as for the audio and video stream, it starts from the central node, is cached at the edge node where the corresponding client has a live broadcast demand, and is finally pushed to one or more clients by the edge node. Among them, there can be at least one intermediate node between the central node and some edge nodes.

[0036] The process of implementing live broadcast is as follows Figure 2 As shown, at least the following steps are included:

[0037] Step 201: Receive a video frame sent by a central node.

[0038] This embodiment does not limit the format of the video frame, the receiving method of the video frame, etc. The video frame can be in any format, and one video frame can be received at a time, or multiple video frames can be received at a time.

[0039] It should be noted that an edge node corresponds to at least one client. When it corresponds to multiple clients, users of at least two of the multiple clients may have the need to watch the same live broadcast, and the timing for users who have the need to watch the same live broadcast to watch the live broadcast through the client may be the same or different. In other words, the video frame received by the edge node may be the video frame returned by the central node based on the edge node's first request for a video frame of a live content. The video frame received by the edge node may also be the video frame that the central node continues to send to the edge node after returning the video frame based on the edge node's first request for a video frame of a live content. This embodiment does not limit this.

[0040] It should also be noted that the central node sending video frames to the edge nodes can be implemented by a live broadcast solution based on RTMP, and the protocol adopted can be the Transmission Control Protocol (TCP). Among them, the live broadcast solution based on RTMP can also include the RTMP protocol and multiple variants related to the RTMP protocol, such as RTMPS, RTMPE, etc. The central node sending video frames to the edge nodes can also be implemented by a live broadcast solution based on HTTP-FLV, which adopts the HTTP protocol. Of course, the central node sending video frames to the edge nodes can also be implemented by a live broadcast solution based on HLS, etc., which will not be repeated here.

[0041] Step 202, starting from the video frames of the I frame type, the received video frames are buffered.

[0042] There are three types of video frames: I-frames, P-frames, and B-frames. Among them, I-frames are intra-coded frames and can be decoded independently; P-frames are forward reference frames and cannot be decoded independently; P-frames are bidirectional reference frames and cannot be decoded independently either.

[0043] In this embodiment, caching starts from the I frame, so the first video frame cached is the I frame, so that after the client receives the video frame, the first video frame to be decoded and played is also the I frame, that is, the starting video frame is the I frame, wherein the starting video frame is the earliest video frame corresponding to the time stamp among the video frames received by the client. The I frame can be decoded independently, that is, the starting video frame can be decoded independently and can be quickly displayed to the user. In particular, compared with P frames and B frames, there is no need to refer to other video frames for decoding, the decoding and playback efficiency is higher, the first screen time of the live broadcast is reduced, and the user experience is improved.

[0044] In some embodiments, starting from a video frame of type I frame, the received video frames are cached, which can be achieved in the following manner: detecting whether the type of the currently received video frame is an I frame; in the case where it is detected that the type of the currently received video frame is an I frame, caching the currently received video frame and caching video frames received after the currently received video frame; in the case where it is detected that the type of the currently received video frame is not an I frame, discarding the currently received video frame and detecting whether the type of the next received video frame is an I frame.

[0045] In this way, before receiving a video frame of type I frame, each time a video frame is received, a detection decision is made as to whether the video frame is to be discarded or cached, so that unnecessary video frames can be discarded in time, thereby reducing the storage space occupied by the received video frames.

[0046] In some embodiments, starting from a video frame of type I, caching the received video frames can also be implemented in the following manner: detecting whether the type of the currently received video frame is an I frame; in the case where it is detected that the type of the currently received video frame is an I frame, caching the currently received video frame and the video frames whose timestamps are after the currently received video frame in the received video frames, and discarding the received video frames whose timestamps are before the currently received video frame, and then continuing to receive video frames and caching the video frames that continue to be received; in the case where it is detected that the type of the currently received video frame is not an I frame, temporarily storing the currently received video frame and confirming whether the type of the next received video frame is an I frame. For example, when video frames A, B, C, and D are received in sequence, assuming that due to network jitter and other conditions, the type of video frame D is an I frame, but the timestamp of video frame C is later than that of video frame D, then based on the solution of the current embodiment, when caching video frame D of type I frame, since the timestamp of video frame C is later than that of video frame D, video frame C will also be cached instead of discarding video frame C. Subsequently, video frame E and the like are continuously received, and the received video frames continue to be cached.

[0047] That is to say, after receiving a video frame of type I, a decision is made based on the timestamp of the video frame of type I to decide whether to discard the currently received video frames. In this way, even if network jitters occur, which may cause the video frame sending order of the central node and the video frame receiving order of the edge node to be inconsistent, the video frames whose timestamps are after the video frame of type I and whose reception timing at the edge node is before the video frame of type I can be cached, thereby reducing the loss of video frames whose timestamps are after the video frame of type I, that is, reducing the loss of valid video frames. Among them, the video frames whose timestamps are before the video frames of type I are invalid video frames because they cannot be decoded.

[0048] Of course, in some embodiments, it is also possible to detect whether the type of the currently received video frame is an I frame; in the case where it is detected that the type of the currently received video frame is an I frame, the currently received video frame and the video frame with a timestamp after the currently received video frame are cached and the received video frame with a timestamp before the currently received video frame is discarded; in the case where it is detected that the type of the currently received video frame is not an I frame, the currently received video frame is temporarily stored and the type of the next received video frame is detected to be an I frame. That is, on the basis of the previous embodiment, all data frames received after receiving a video frame of type I frame are no longer cached, but whether to discard or cache is determined according to the timestamp, so that all video frames with a timestamp after the video frame of type I frame can be cached, and all video frames with a timestamp before the video frame of type I frame can be discarded, while reducing the loss of valid video frames, the utilization rate of storage space is improved by reducing the cache of invalid video frames.

[0049] As described in step 201, the video frame received by the edge node may be a video frame returned by the central node based on the edge node's first request for a video frame of a live content. The video frame received by the edge node may also be a video frame that the central node continues to send to the edge node after the video frame has been returned by the central node based on the edge node's first request for a video frame of a live content.

[0050] It should be noted that, in the second case mentioned above, the video frames sent by the central node are video frames returned by the central node to the edge node after at least one client corresponding to the edge node has requested a certain live content and is playing the live content, and another (some) client also requests the live content. At this time, there are two types of clients that need the edge node to send video frames, including clients that are playing live broadcasts and clients waiting to play live broadcasts.

[0051] For the client that is playing the live broadcast, it continuously receives video frames. Even if the starting video frame currently cached by the edge node is a P frame or a B frame, since an I frame has been sent to the edge node before, the P frame or B frame can refer to the received I frame when decoding, without waiting for the next I frame. In other words, the video frames that need to be sent to the client that is playing the live broadcast can be cached at the edge node in a way that the edge node caches a video frame as it receives one video frame. In this way, the video frames sent to the client that is playing the live broadcast will be more comprehensive, without video frame discarding, and the live content presented to the user is information lossless.

[0052] For the client waiting to play the live broadcast, its decoding and playback needs to start from the I frame. Therefore, the video frames received by the client waiting to play the live broadcast need to start from the I frame. That is to say, these video frames are cached at the edge node starting from the received video frames of type I frame.

[0053] The video frames before the video frames of type I frame are delayed longer and cannot be decoded. Therefore, not caching or discarding the video frames before the video frames of type I frame has the advantage of avoiding the inability to decode the received video frames and is also beneficial to reducing the delay of the live broadcast picture.

[0054] It can be seen from this that when the video frame sent by the central node is a video frame returned to the edge node by the central node after at least one client corresponding to the edge node has requested a certain live content and is playing the live content, and another (some) client also requests the live content, the edge node can provide different video frame caching methods for the client that is playing the live broadcast and the client waiting to play the live broadcast, respectively, to provide a better user experience.

[0055] It is understandable that the edge node sends the cached video frames to the client until the client is ready to receive the edge node's video frames, but the timing is uncertain. Therefore, before sending to the client waiting to play the live broadcast, the number of video frames cached by the edge node is uncertain. At this time, the number of video frames cached by the edge node can also be limited so that the video frames cached by the edge node are always the latest video frames. In this way, the video frames sent by the edge node to the client waiting to play the live broadcast are always the latest video frames, which is conducive to reducing the delay of the client waiting to play the live broadcast decoding and playing the picture, and improving the user experience.

[0056] In some embodiments, Figure 3 As shown, starting from the video frame of type I, the received video frames are cached, which can also be implemented in the following manner: after receiving the k-th video frame of type I, the video frames cached before the k-th video frame of type I are cleared, and the received video frames are cached starting from the k-th video frame of type I. In this case, k is a positive integer.

[0057] In some examples, k is not greater than N1, where N1 is the number of I frames contained in the live content from the time the client requests the live content from the edge node to the end of the live broadcast. That is, the edge node continues to wait for the opportunity to send video frames to the client until the live broadcast ends, and keeps caching video frames for the client, so that as long as possible is reserved for the arrival of the opportunity to send video frames to the client, without the client having to initiate a request to the edge node again, thereby improving the user experience.

[0058] In some examples, k is not greater than N2, and N2 is a preset value. That is, if the edge node has not yet received N2 video frames of type I and the timing for sending the video frame to the client has not arrived, the data can be no longer cached, so that the resources of the edge node can be used to implement other tasks, saving resources and improving the user experience brought by other tasks.

[0059] In some examples, k is not greater than N3, where N3 is the number of video frames of type I included in the live content after the client requests the live content from the edge node and before other clients requesting the same live content finish watching. That is, when the edge node caches video frames for other clients requesting the same live content, it can cache video frames for the client based on the received video frames, making full use of resources.

[0060] That is to say, every time a video frame of type I frame is received, the previously cached video frame is cleared. Therefore, the edge node caches only the video in the current latest group of pictures (Group of picture, GOP), and the video sent to the client is also the video in the current latest group of pictures (Group of picture, GOP).

[0061] Of course, in other embodiments, after receiving the k-th video frame of type I, the video frames cached before the k-1-th video frame of type I are cleared, and the received video frames are cached starting from the k-th video frame of type I. That is, the edge node caches the videos in the two latest image groups GOP. In this case, k is an integer greater than 1. This will not be described in detail here.

[0062] In this way, the number of video frames cached by the edge node is limited in units of GOP, and the video frames of two adjacent GOPs are connected, so that the edge node caches at least one video frame in at least one GOP, ensuring that when the opportunity to send a video frame to the client arrives at any time, there is at least one video frame cached in the edge node that can be sent to the client, without the client waiting, which is conducive to reducing the first screen time.

[0063] In some embodiments, starting from a video frame of type I frame, the received video frames are cached, which can also be achieved in the following manner: caching the first video frame of type I frame; detecting whether the number of currently cached video frames has reached a preset value; when the number of currently cached video frames has reached the preset value, clearing the currently cached video frames, and caching the received video frames starting from the next video frame of type I frame; when the number of currently cached video frames has not reached the preset value, caching the currently received video frames.

[0064] In order to facilitate those skilled in the art to better understand the above embodiments, Figure 4 Let's give an example. Figure 4 As shown, starting from the video frame of the I frame type, the received video frame is cached, and the following steps may also be included:

[0065] Step 2021: After receiving the i-th video frame of type I frame, cache the received i-th video frame of type I frame, where i is a positive integer.

[0066] It should be noted that the value range of i here is roughly the same as the aforementioned parameter values, and will not be described in detail here.

[0067] Step 2022, receiving the next video frame sent by the central node.

[0068] Step 2023, detect whether the number of currently cached video frames has reached a preset value; if so, execute step 2024, if not, execute step 2025.

[0069] In this embodiment, the preset value is the maximum number of cacheable video frames set according to demand.

[0070] Step 2024, clear the currently cached video frames and continue to receive the video frames sent by the central node.

[0071] In some examples, the preset value may be smaller than the number of video frames contained in a GOP. Since caching starts from the video frames of type I frame, that is, caching starts from the first video frame of the GOP, when the number of cached video frames reaches the preset value, the currently cached video frames are cleared, which means that the edge node only caches the preset number of video frames for each image group, further reducing the number of cached video frames and saving storage space.

[0072] Step 2025: Cache the currently received video frame.

[0073] Accordingly, after step 2024 and step 2025, the following should be included respectively:

[0074] Step 2026, detecting whether a content request sent by the client is currently received.

[0075] Step 2027, detecting whether a content request sent by the client is currently received.

[0076] If a content request sent by the client is received, step 203 is executed, otherwise, the video frame should continue to be received and / or cached. Figure 4, I will not go into details here.

[0077] That is to say, the number of video frames cached by the edge node is limited in units of video frames, reducing the storage space occupied by the cached video frames. At the same time, the number of video frames with a preset value is more fixed than the number of video frames contained in the preset number of GOPs. Therefore, the configurations such as the length of the cache queue are more fixed, reducing the content of autonomous settings, which is conducive to reducing the occurrence of errors.

[0078] Of course, the above is only a specific example. In other embodiments, the specific method of caching can be limited according to other requirements, as long as the starting video frame for caching and sending to the client is an I frame.

[0079] Step 203: after receiving the content request from the client, the cached video frames are sent to the client based on the RTC protocol, so that the client can present the received video frames to the user.

[0080] In this embodiment, the content request of the client is used to indicate that the time for the edge node to send the video frame to the client has arrived. When the edge node receives the content request sent by the client, if the edge node determines that the time for sending the cached video frame to the client has arrived, it will trigger the action of sending the cached video frame to the client.

[0081] It should be noted that, according to different requirements, the content request may be different information.

[0082] In some embodiments, the content request is a Session Traversal Utilities for Network Address Translation (STUN) request to establish a STUN data channel between the client and the edge node, so that the edge node can send the video frame to the client through the established STUN data channel. That is, the edge node receives the STUN data packet, and after receiving the STUN data packet, it triggers the action of sending the cached video frame to the client based on the RTC protocol according to the STUN data packet, so as to send the cached video frame to the client based on the RTC protocol. At this time, a private SDK (Software Development Kit) is used between the edge node and the client for non-encrypted video frame transmission. Thereby, by reducing the encryption processing time, the response time after receiving the content request is reduced, which is conducive to further reducing the first screen time.

[0083] In some embodiments, the content request can also be a Datagram Transport Level Security (DTLS) handshake request to perform a DTLS handshake between the edge node and the client. That is, the edge node receives the DTLS handshake request sent by the client; according to the DTLS handshake request, the DTLS handshake is performed with the client. Therefore, after the DTLS handshake is completed, the cached video frames are encrypted; and then the encrypted video frames are sent to the client based on the RTC protocol. In this way, the subsequent edge node uses an encrypted transmission method when sending video frames to the client, which improves the transmission security of the video frames. Among them, before performing the DTLS handshake, the edge node and the client also establish a STUN data transmission channel through a STUN data packet to provide a transmission channel for the video frame.

[0084] That is to say, after receiving the STUN request, the edge node needs to wait for receiving the DTLS handshake request before sending the cached video frame to the client.

[0085] Of course, the above are only specific examples of content requests. In other embodiments, other requests may also be defined as content requests, which will not be described in detail here.

[0086] Regarding the sending of video frames, in some embodiments, the cached video frames can be sent to the client at one time based on the RTC protocol, so that the client can continue to decode and play after receiving the video frames, and there will be no idle time between the decoding and playback of two video frames, affecting the user experience.

[0087] It should be noted that the edge node is configured with a protocol conversion function to support the function of sending the cached video frames to the client based on the RTC protocol, so that before sending the video frames, the edge node converts the audio and video streams received from the central node based on the RTMP protocol transmission into an RTC stream and sends it to the user's client, thereby transmitting video data with the client based on the RTC protocol and further improving the transmission efficiency of the audio and video streams.

[0088] Specifically, the conversion to RTC stream is implemented as follows: the edge node performs Real-time Transport Protocol (RTP) segmentation on the video frames taken out from the cache queue frame by frame, and then sends them to the client, thereby sending the video frames to the client in an RTC manner with a smaller delay. I will not go into details here.

[0089] It should also be noted that the edge node sends video frames to the client based on the RTC mechanism, which is different from the aforementioned central node sending video frames to the edge node. And because the transmission efficiency of RTC is higher, the transmission efficiency of video frames is improved. It can be seen that the live broadcast solution based on RTC is conducive to further reducing the delay of live broadcast and improving user experience.

[0090] Step 204: After sending the cached video frame to the client, continue to receive the video frame sent by the central node and send the currently received video frame to the client based on the RTC protocol.

[0091] That is to say, after the edge node sends the video frame to the client for the first time, it will transmit the video frame by receiving the video frame sent by the central node while sending the video frame to the client, so that the live broadcast screen can be played in real time on the client, meeting the real-time requirements of RTC, and ensuring that the client can decode and play subsequent video frames based on the previously received video frames without affecting the user experience.

[0092] It is understandable that live broadcast content usually includes not only video frames but also audio frames, thereby providing live broadcast services to users visually and aurally.

[0093] In some embodiments, since video frames contain more information than audio frames, and users respond faster and more strongly to images than to sounds, the edge node can cache and send video frames first before sending live content to the client for the first time, so that the limited bandwidth of the edge node can be fully used to send video frames to the client, which is conducive to using limited resources to provide users with a relatively better experience. Especially when the video frame contains a large amount of information and occupies a large amount of bandwidth, and the available bandwidth of the edge node is limited, sending video frames and audio frames at the same time is likely to cause jamming, and sending video frames first reduces the jamming of video frame playback.

[0094] Of course, in some embodiments, both video frames and audio frames may be cached, and whether to send the video frames to the user at the same time may also be determined based on the bandwidth resources of the edge node before sending and the bandwidth size estimated to be occupied by the cached video frames and / or audio frames.

[0095] Based on this, in some embodiments, Figure 5 As shown, the live broadcast implementation method also includes the following contents:

[0096] While receiving the video frame sent by the central node, the audio frame sent by the central node is also received.

[0097] Starting from the video frame of the I frame type, while caching the received video frames, the received audio frames are also cached starting from the audio frames corresponding to the video frames of the I frame type.

[0098] While sending the cached video frames to the client based on the RTC protocol, the cached audio frames are also sent to the client based on the RTC protocol.

[0099] After sending the cached video frames and audio frames to the client, it continues to receive the video frames and audio frames sent by the central node, and sends the currently received audio frames and video frames to the client based on the RTC protocol.

[0100] In some cases, the audio frames and the video frames may correspond to each other based on the timestamps, such as the audio frames and the video frames having the same timestamps or having the smallest timestamp difference correspond to each other.

[0101] That is, the audio frames and the video frames are cached starting from a corresponding frame, which is conducive to ensuring the synchronization between the cache and the audio frames and the video frames sent to the client based on the RTC protocol, thereby improving the user experience.

[0102] Of course, the audio frame may also be determined whether to be cached according to actual conditions and sent to the client based on the RTC protocol.

[0103] In some embodiments, after receiving the audio frame and the video frame at the same time, the method for implementing live broadcasting further includes: determining whether to cache the audio frame according to the bit rate of the video frame. Accordingly, when it is determined to cache the audio frame, the received video frame is cached starting from the video frame of the type I frame, and the received audio frame is also cached starting from the audio frame corresponding to the I frame. When it is determined not to cache the audio frame, there is no need to cache the audio frame, and the audio frame will not be sent to the client.

[0104] In some embodiments, it is also possible to determine whether to cache audio frames based on the total bit rate of the audio frames and the video frames, or to determine whether to cache audio frames based on the idle bandwidth of the edge node, or to determine whether to cache audio frames based on the idle bandwidth of the edge node and the bit rate of the video frames, etc.

[0105] Of course, the above instructions on whether to cache audio and whether to cache audio frames are only for the case where a client is about to open a live broadcast on the first screen. After the first screen, in order to provide users with a better experience, there is no need to determine whether to cache video frames. Instead, audio frames are cached while caching video frames, and video frames and audio frames are sent to the client based on the RTC protocol at the same time. I will not go into details here.

[0106] It should be noted that the audio frame can be cached by searching for the audio frame corresponding to the cached video frame after the video frame is cached; or the corresponding audio frame can be cached after the first video frame is cached, and then the subsequent received audio frames can be cached starting from this without referring to the cache of the video frame. Among them, when the edge node caches the audio frame without referring to the cache of the video frame, the caching of the audio frame is independent, and its implementation process is roughly the same as the aforementioned caching process of the video frame. The main difference is that one caches the audio frame and the other caches the video frame. The rest of the content will not be repeated here.

[0107] In order to facilitate those skilled in the art to better understand the implementation method of live broadcast described in the above embodiment, the following will be described by taking the scenario where all clients corresponding to a certain edge node request a certain live broadcast content for the first time as an example. Figure 6 As shown, the method for implementing live broadcast includes:

[0108] Step 601: The client sends a Session Description Protocol (SDP) request to the edge node. The SDP request is used to indicate the client's live streaming demand.

[0109] Step 602: The edge node parses the SDP request to determine whether the node supports the audio and video encoding format requested by the client.

[0110] Step 603: If the node supports the audio and video encoding format requested by the client, a stream pull request is sent to the central node.

[0111] Step 604: The central node returns a stream pull response to the stream pull request to the edge node, where the stream pull request includes a video frame and an audio frame.

[0112] Step 605, when the edge node determines to cache the audio frames according to the bit rate of the video frames, it starts caching the received video frames from the first video frame of type I frame and the audio frames corresponding to the first video frame of type I frame.

[0113] Step 606: The client sends a STUN request to the edge node.

[0114] Step 607: The edge node sends a response to the STUN request to the client.

[0115] Step 608: The edge node performs RTP fragmentation on the cached audio and video frames frame by frame.

[0116] Step 609: The edge node sends the segmented audio and video frames to the client based on the RTC protocol, so that the client can decode and play the audio and video frames and present them to the user.

[0117] After step 609, the process also includes: the edge node receives the audio and video frames sent by the central node, and performs RTP segmentation on the currently received audio and video frames and sends them to the client ( Figure 6 ), and will not be described in detail here.

[0118] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0119] On the other hand, the embodiment of the present application also provides an edge node, such as Figure 7 As shown, including:

[0120] The receiving module 701 is used to receive the video frames sent by the central node.

[0121] The cache module 702 is used to cache the received video frames starting from the video frames of the I frame type.

[0122] The sending module 703 is used to send the cached video frames to the client based on the RTC protocol after receiving the content request from the client, so that the client can present the received video frames to the user.

[0123] After sending the cached video frames to the client, the forwarding module 704 continues to receive the video frames sent by the central node and sends the currently received video frames to the client based on the RTC protocol.

[0124] It is not difficult to find that this embodiment is a node embodiment corresponding to the method embodiment, and this embodiment can be implemented in conjunction with the method embodiment. The relevant technical details mentioned in the method embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the method embodiment.

[0125] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0126] Another aspect of the present application embodiment further provides an electronic device, such as Figure 8 As shown, it includes: at least one processor 801; and a memory 802 that is communicatively connected to the at least one processor 801; wherein the memory 802 stores instructions that can be executed by the at least one processor 801, and the instructions are executed by the at least one processor 801 so that the at least one processor 801 can execute the method described in any of the above method embodiments.

[0127] The memory 802 and the processor 801 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 801 and the memory 802 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 801 is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor 801.

[0128] The processor 801 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 802 can be used to store data used by the processor 801 when performing operations.

[0129] Another aspect of the present application embodiment further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above method embodiment is implemented.

[0130] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0131] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for implementing live broadcast, characterized in that: Applied to an edge node, the method comprises: Receive video frames sent by the central node, and at the same time, receive audio frames sent by the central node; Determining whether to cache the audio frame according to the bit rate of the video frame; Starting from the video frame of type I frame, the received video frame is cached, and at the same time, when it is determined to cache the audio frame, the received audio frame is cached starting from the audio frame corresponding to the video frame of type I frame; After receiving a content request from a client, the cached video frame is sent to the client based on an RTC protocol, so that the client can present the received video frame to the user, and the cached audio frame is sent to the client based on the RTC protocol; After sending the cached video frames to the client, the central node continues to receive the video frames sent by the central node and sends the currently received video frames to the client based on the RTC protocol.

2. The method for implementing live broadcast according to claim 1, characterized in that: The step of caching the received video frames starting from the video frames of the I frame type includes: After receiving the kth video frame of type I frame, the video frames cached before the kth video frame of type I frame are cleared, and the received video frames are cached starting from the kth video frame of type I frame.

3. The method for implementing live broadcast according to claim 1 or 2, characterized in that: The step of caching the received video frames starting from the video frames of the I frame type includes: Cache the first video frame of type I frame; Detecting whether the number of the video frames currently cached has reached a preset value; When the number of the currently cached video frames has reached the preset value, the currently cached video frames are cleared, and after receiving the next video frame of the I frame type, the received video frames are cached; When the number of the currently cached video frames does not reach the preset value, the currently received video frames are cached.

4. The method for implementing live broadcast according to claim 1 or 2, characterized in that: Before sending the cached video frame to the client based on the RTC protocol, the method further includes: Receive a DTLS handshake request sent by the client; Performing a DTLS handshake with the client according to the DTLS handshake request; The sending the cached video frame to the client includes: After completing the DTLS handshake, encrypting the cached video frames; The encrypted video frame is sent to the client based on the RTC protocol.

5. The method for implementing live broadcast according to claim 1 or 2, characterized in that: Before sending the cached video frame to the client based on the RTC protocol, the method further includes: Receive a STUN data packet sent by the client; The sending the cached video frame to the client based on the RTC protocol includes: An action of sending the cached video frame to the client based on the RTC protocol is triggered according to the STUN data packet, so as to send the cached video frame to the client based on the RTC protocol.

6. The method for implementing live broadcast according to claim 1 or 2, characterized in that: Before sending the cached video frame to the client based on the RTC protocol, the method further includes: The cached video frames are sent to the client at one time based on the RTC protocol.

7. An edge node, characterized in that: include: A receiving module, used to receive the video frame sent by the central node, and at the same time, receive the audio frame sent by the central node; A cache module, used to determine whether to cache the audio frame according to the bit rate of the video frame; Starting from the video frame of type I frame, the received video frame is cached, and at the same time, when it is determined to cache the audio frame, the received audio frame is cached starting from the audio frame corresponding to the video frame of type I frame; A sending module, configured to send the cached video frames to the client based on the RTC protocol after receiving a content request from the client, so that the client can present the received video frames to the user, and at the same time, send the cached audio frames to the client based on the RTC protocol; The forwarding module is used to continue receiving the video frames sent by the central node after sending the cached video frames to the client and send the currently received video frames to the client based on the RTC protocol.

8. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the live broadcast implementation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the live broadcast implementation method as described in any one of claims 1 to 6 is implemented.

Images