Method and apparatus for streaming of audio and video, electronic device and medium

Abstract

The present disclosure provides a streaming method for audio and video, applied to a streaming sending device, the method comprising: obtaining initial network state information, and configuring target live protocol header information unit, wherein the initial network state information comprises a measure of initial network performance of a transmission network used for transmitting audio and video; selecting a target frame packaging type that is adapted to the initial network state information from a plurality of frame packaging types based on the initial network state information; packaging the audio and video stream to be transmitted into one or more first target live protocol media information units based on at least the target frame packaging type, each of the one or more first target live protocol media information units comprising a field for identifying the frame packaging type of the first target live protocol media information unit; and transmitting the target live protocol header information unit to a streaming receiving device, and streaming the one or more first target live protocol media information units to the streaming receiving device.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, specifically to a method and apparatus for audio and video streaming, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] With the continuous development of internet technology and the advancement of streaming media technology, live streaming is attracting increasing attention from users. Currently, the mainstream streaming media protocols include RTMP (Real Time Messaging Protocol) and HLS (HTTP Live Streaming).

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a method and apparatus for streaming audio and video, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a method for streaming audio and video is provided, applied to a streaming transmission device. The method includes: acquiring initial network state information and configuring a target live streaming protocol header information unit, wherein the initial network state information includes a measure of the initial network performance of the transmission network used for transmitting audio and video; selecting a target frame encapsulation type that matches the initial network state information from a plurality of frame encapsulation types based on the initial network state information; encapsulating the audio and video stream to be transmitted into one or more first target live streaming protocol media information units, at least based on the target frame encapsulation type, each of the one or more first target live streaming protocol media information units including a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and transmitting the target live streaming protocol header information unit to a streaming transmission receiving device and streaming one or more first target live streaming protocol media information units to the streaming transmission receiving device.

[0006] According to another aspect of this disclosure, a method for streaming audio and video is also provided, applied to a streaming receiving device. The method includes: providing initial network state information, wherein the initial network state information includes a measure of the initial network performance of the transmission network used for transmitting audio and video; the initial network state information is used by the streaming sending device to select a target frame encapsulation type adapted to the initial network state information from a plurality of frame encapsulation types; and encapsulating the audio and video stream to be transmitted into one or more first target live streaming protocol media information units, at least based on the target frame encapsulation type; each of the one or more first target live streaming protocol media information units includes a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and receiving a target live streaming protocol header information unit transmitted by the streaming sending device and one or more first target live streaming protocol media information units streamed by the streaming sending device.

[0007] According to another aspect of this disclosure, an audio / video streaming apparatus is also provided, the apparatus comprising: an initialization module for acquiring initial network state information and configuring target live streaming protocol header information units, wherein the initial network state information includes a measure of the initial network performance of the transmission network used for transmitting audio / video; a frame encapsulation selection module for selecting a target frame encapsulation type adapted to the initial network state information from a plurality of frame encapsulation types based on the initial network state information; a frame encapsulation execution module for encapsulating the audio / video stream to be transmitted into one or more first target live streaming protocol media information units, at least based on the target frame encapsulation type, each of the one or more first target live streaming protocol media information units including a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and a transmission module for transmitting the target live streaming protocol header information units to a streaming receiving device and streaming one or more first target live streaming protocol media information units to the streaming receiving device.

[0008] According to another aspect of this disclosure, an audio / video streaming apparatus is also provided, the apparatus comprising:

[0009] A monitoring module is used to provide initial network status information, wherein the initial network status information includes a measure of the initial network performance of the transmission network used for transmitting audio and video. The initial network status information is used by the streaming transmission device to select a target frame encapsulation type that matches the initial network status information from multiple frame encapsulation types, and to encapsulate the audio and video stream to be transmitted into one or more first target live streaming protocol media information units based at least on the target frame encapsulation type. Each of the one or more first target live streaming protocol media information units includes a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit. A receiving module is used to receive a target live streaming protocol header information unit transmitted by the streaming transmission device and one or more first target live streaming protocol media information units streamed by the streaming transmission device.

[0010] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory communicatively connected to the at least one processor, wherein the at least one memory stores a computer program that, when executed by the at least one processor, implements the above-described method for streaming audio and video.

[0011] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-described method for streaming audio and video when executed by a processor.

[0012] According to another aspect of this disclosure, a computer program product is also provided, including a computer program, wherein the computer program, when executed by a processor, implements the above-described method for streaming audio and video.

[0013] According to one or more embodiments of this disclosure, audio and video data can be encapsulated in a manner that is less than one frame. The server can then select the frame encapsulation type of the audio and video data according to the real-time network conditions. When the network conditions are poor, a complete frame of audio and video can be cut into partial audio and video frames for encapsulation and streaming, thereby effectively reducing startup latency.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0017] Figure 2 A flowchart of a method for streaming audio and video according to an embodiment of the present disclosure is shown;

[0018] Figure 3 A schematic block diagram of a target live streaming protocol according to an embodiment of the present disclosure is shown;

[0019] Figure 4 A schematic block diagram of a multi-level server architecture according to embodiments of the present disclosure is shown;

[0020] Figure 5 A schematic signaling diagram between a streaming media transmitting device and a streaming media receiving device according to an embodiment of the present disclosure is shown;

[0021] Figure 6 A flowchart is shown of a method for streaming a second target live streaming protocol media information unit according to an embodiment of the present disclosure;

[0022] Figure 7 A schematic signaling diagram between a streaming media transmitting device and a streaming media receiving device according to an embodiment of the present disclosure is shown;

[0023] Figure 8 A flowchart of a method for streaming audio and video according to an embodiment of the present disclosure is shown;

[0024] Figure 9 A structural block diagram of an audio / video streaming apparatus according to an embodiment of the present disclosure is shown;

[0025] Figure 10 A structural block diagram of an audio / video streaming apparatus according to an embodiment of the present disclosure is shown; and

[0026] Figure 11 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0027] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0028] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0029] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0030] With the continuous development of internet technology and the advancement of streaming media technology, live streaming has attracted increasing attention from users. Currently, the mainstream streaming media protocols include RTMP (Real Time Messaging Protocol) and HLS (HTTP Live Streaming). However, both of these mainstream protocols have different problems. On the one hand, due to a lack of maintenance, the RTMP standard is no longer updated, resulting in poor compatibility and insufficient support for current live streaming needs, failing to support existing and future encoders. On the other hand, because HLS is based on short HTTP connections, it leads to a very high QPS (Queries per second), consuming significant computing power. Furthermore, HLS requires three media segments to start playback, resulting in relatively high startup latency. Therefore, there is an urgent need for a live streaming protocol with low startup latency and good compatibility to meet users' demands for an improved live streaming experience.

[0031] In view of this, embodiments of the present disclosure provide a method for streaming audio and video, which can be applied to a streaming transmission device. The method selects a target frame encapsulation type from multiple frame encapsulation types that matches the initial network state information based on acquired initial network state information, and encapsulates the audio and video stream to be transmitted into one or more first target live streaming protocol media information units according to the target frame encapsulation type, so as to stream it to a streaming receiving device. This method allows audio and video data to be encapsulated in a manner that is less than one frame. The server can therefore select the frame encapsulation type of the audio and video data according to the real-time network conditions. When network conditions are poor, a complete frame of audio and video can be cut into partial audio and video frames for encapsulation and streaming, thereby effectively reducing startup latency at the client end and improving the live streaming playback experience.

[0032] Before describing the specific embodiments of this disclosure, the following terms will first be explained.

[0033] Live stream: Live stream refers to the transmission of audio and video data in a live streaming scenario, which can be transmitted to the audience as a stable and continuous streaming media over the network.

[0034] Live streaming pull: Live streaming pull refers to the process of pulling a live stream from a user-specified origin server through a live streaming cloud platform.

[0035] Origin pull: Origin pull refers to the process where a user initiates a request to a CDN (Content Delivery Network), and when the CDN does not have the required content, it needs to send a request to the content provider to pull the live content.

[0036] CDN: CDN stands for Content Delivery Network, as described above.

[0037] CDN service provider: A CDN service provider can refer to a service provider that provides and / or maintains a content delivery network.

[0038] Live Streaming Origin Site: A live streaming origin site can refer to a site used by CDN to pull back live streams and return live content.

[0039] HLS: HLS refers to a streaming media protocol proposed by Apple, which can be used for live streaming and video-on-demand.

[0040] RTMP: RTMP is a streaming media protocol proposed by Adobe, commonly used for live streaming.

[0041] P2P: P2P can refer to a content sharing technology that is typically used between users for the purpose of network acceleration and bandwidth saving.

[0042] QPS: QPS can refer to queries per second or requests per second, and is generally used to indicate the frequency of user access to the server.

[0043] HTTP: HTTP stands for Hypertext Transfer Protocol, which is generally used for browser access.

[0044] RTT: RTT (round-trip time) refers to the time it takes for a network request to travel from its origin to its destination and back to its origin. It usually does not include the processing time at the receiving end.

[0045] TCP: TCP (Transmission Control Protocol) is the underlying transmission protocol of the Internet, the cornerstone of current Internet transmission, and a commonly used transmission protocol in computer programming. It is usually implemented in the operating system and can be directly called by ordinary developers.

[0046] CHUNK: CHUNK (chunk) refers to a container unit composed of multiple frames. Under the existing fMP4 protocol, this unit is the smallest unit that the player decapsulates.

[0047] FRAGMENT: A FRAGMENT (fragment file) is an audio / video segment composed of multiple CHUNKs, usually corresponding to a single file. This unit is the smallest audio / video unit in HLS transmission.

[0048] In this disclosure, the terms target live streaming protocol, audio and video transmission protocol, streaming media protocol, live streaming protocol, etc., can be used interchangeably.

[0049] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0050] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 System 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. Client devices 101, 102, 103, 104, 105, and 106 can be configured to execute one or more applications. It should be understood that, although... Figure 1 Only six client devices are described, but this disclosure can support any number of client devices.

[0051] exist Figure 1 In the illustrated system, server 120 may include one or more components to implement one or more operations performed thereon. For example, in embodiments of this disclosure, these one or more operations may include one or more steps of the above-described method for streaming audio and video. These components may include software components, hardware components, or combinations thereof executable by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may sequentially use one or more client applications to interact with server 120 to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0052] Users can initiate communication with server 120 using client devices 101, 102, 103, 104, 105, and / or 106. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to users through this interface.

[0053] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0054] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, network 110 can be a local area network (LAN), Ethernet, wide area network (WAN), Internet, virtual network, virtual private network (VPN), intranet, extranet, infrared network, wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0055] Server 120 may include one or more general-purpose computers, special-purpose computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0056] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0057] In some implementations, server 120 may include one or more applications to analyze and merge data received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display the data via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0058] In some implementations, server 120 can be a server for a distributed system or a server incorporating blockchain technology. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0059] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store data structures such as those representing multiple frame encapsulation types. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120 or may be located remotely to server 120 and accessed via a network. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0060] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0061] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0062] Figure 2 A flowchart of a method 200 for streaming audio and video according to an embodiment of the present disclosure is shown. According to some embodiments of the present disclosure, method 200 can be applied to a streaming transmission device. As used herein, the term streaming transmission device refers to an entity that streams streaming media content (which can be obtained through methods such as live streaming or origin streaming as described above) to a downstream entity (e.g., a secondary server, a final server relative to the parent server corresponding to the streaming transmission device, a streaming transmission receiving device such as a client, etc.). Correspondingly, the term streaming transmission receiving device refers to an entity that receives streamed streaming media content from an upstream entity (e.g., a streaming transmission device), the received streamed streaming media content which may be decoded at a final server or at the streaming transmission receiving device, without any limitation herein.

[0063] like Figure 2 As shown, method 200 may include: step S210, obtaining initial network state information and configuring target live streaming protocol header information units, wherein the initial network state information includes a measure of the initial network performance of the transmission network used for transmitting audio and video; step S220, selecting a target frame encapsulation type that matches the initial network state information from multiple frame encapsulation types based on the initial network state information; step S230, encapsulating the audio and video stream to be transmitted into one or more first target live streaming protocol media information units, at least based on the target frame encapsulation type, each of the one or more first target live streaming protocol media information units including a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and step S240, transmitting the target live streaming protocol header information unit to the streaming receiving device and streaming one or more first target live streaming protocol media information units to the streaming receiving device.

[0064] In some examples, initial network state information can be obtained by the streaming device from the streaming receiving device. In other examples, the initial network state information can also be obtained by the streaming device from a third party other than the streaming receiving device (e.g., a network operator, the maintainer of the transmission network for transmitting audio and video, etc.).

[0065] According to some examples, initial network performance metrics may include indicators such as bandwidth, latency, bandwidth-delay product, throughput, packet loss rate, etc., used to characterize network performance during the initialization period of streaming (e.g., a period of time across the initialization process, or at a certain moment(s) during the initialization period).

[0066] Based on some examples, the mapping between initial network state information (e.g., a metric of initial network performance) and the appropriate target frame encapsulation type can be stored in a database. In a streaming transmission device, for example as a reference... Figure 1 In the case of the described server 120, the database can be database 130 as described above. In the example, the database can take the form of a Redis (Remote Dictionary Server) database. Redis is a key-value based database. In this example, the key can represent initial network state information (e.g., a metric of initial network performance), and the value can represent the target frame encapsulation type that matches the initial network state information (e.g., a metric of initial network performance). When storing the mapping relationship based on this database, high read and write speeds can be achieved because the data is stored in memory. Furthermore, the Redis database supports data persistence and data type diversification, which facilitates the backup of the stored mapping relationship between the initial network state information and the target frame encapsulation type and meets the need to retrieve another target frame encapsulation type that matches the updated network state information (e.g., a metric of current network performance) when the network state changes, as will be described below. Of course, this disclosure does not limit the specific implementation of the database, and those skilled in the art can use any other suitable type of database as needed.

[0067] According to some embodiments of this disclosure, the target live streaming protocol is the BMT (BvcMediaTransport) live streaming protocol.

[0068] Based on some examples, the basic information unit of a target live streaming protocol can take the form of an mp4 box. The following exemplary code illustrates an exemplary definition of the basic information unit of a target live streaming protocol.

[0069]

[0070] The exemplary code above illustrates the data structure of a basic information unit in the target live streaming protocol. According to some examples, all information units in the target live streaming protocol share this common data structure. It can be seen that the first 4 bytes within the basic information unit represent the overall size of the information unit, and the next 4 bytes represent the type name of the information unit (generally unique). When size equals 1, it indicates that the information unit is extremely large; that is, 4 bytes cannot represent the information unit, and an additional 8 bytes are used to represent it. When size equals 0, it indicates the end of the file. If boxtype equals uuid, an additional 16 bytes represent the extended type. It will be understood that the interpretation of the above exemplary code can be referenced to existing audio and video transmission protocols (e.g., MP4 or fMP4 protocols), which will not be elaborated upon in this disclosure.

[0071] It will be understood that the exemplary code described above is merely exemplary and not intended to impose any limitations. Those skilled in the art can make any suitable additions, deletions, modifications, substitutions, etc., to the exemplary code described above after learning of the concept of this disclosure.

[0072] Based on some examples, the target live streaming protocol header information unit can be constructed based on the target live streaming protocol's basic information units. The exemplary code below illustrates an exemplary definition of a target live streaming protocol header information unit (e.g., the BHIN (BvcHeader Information) box).

[0073]

[0074] According to some examples, the Target Live Streaming Protocol (DLP) header information element may appear at the very beginning of the live stream. It can be used to identify the version number and the starting number, and / or to determine whether the live stream is to be streamed using the target DLP. In the case where the target DLP is a BMT live streaming protocol, the DLP header information element can indicate whether the live stream is a BMT stream.

[0075] It will be understood that the exemplary code described above is merely exemplary and not intended to impose any limitations. Those skilled in the art can make any suitable additions, deletions, modifications, substitutions, etc., to the exemplary code described above after learning of the concept of this disclosure.

[0076] Based on some examples, the target frame encapsulation type can be several complete frames, a single complete frame, a partial frame (e.g., 1 / 2, 1 / 3, 1 / 4, etc. of a complete frame), or a combination of complete and partial frames (e.g., one or more complete frames plus partial frames).

[0077] Therefore, with the help of method 200, audio and video data can be encapsulated in a manner that is less than one frame. The server can then select the frame encapsulation type of audio and video data according to the real-time network conditions. When the network conditions are poor, a complete frame of audio and video can be cut into partial audio and video frames for encapsulation and streaming, which can effectively reduce the startup latency at the client end and improve the live streaming experience.

[0078] Figure 3 A schematic block diagram of a target live streaming protocol 300 according to an embodiment of the present disclosure is shown.

[0079] According to some examples, the target live streaming protocol 300 includes a target live streaming protocol header information unit (shown as BHIN) and one or more target live streaming protocol media information units (shown as BMIF, which may be, for example, a first target live streaming protocol media information unit). According to some examples, the target live streaming protocol 300 also includes information units for storing audio and video metadata files (e.g., shown as ftyp and moov). For an interpretation of ftyp and moov (which may be implemented as ftyp box and moovbox, respectively), please refer to the relevant content of existing audio and video transmission protocols (e.g., MP4 or fMP4 protocols, etc.), which will not be elaborated here.

[0080] According to some embodiments of this disclosure, the target live streaming protocol media information unit can support frame assembly operations, that is, encapsulate several complete frames, one complete frame, partial frames (e.g., 1 / 2, 1 / 3, 1 / 4, etc. of a complete frame), or a combination of complete frames and partial frames (e.g., one or more complete frames plus partial frames), to distinguish it from the implementation in existing streaming media protocols that encapsulates (several) complete frames with CHUNK as the smallest encapsulation unit.

[0081] What will be understood is... Figure 3 The number of target live streaming protocol media information units (shown as BMIF) shown is merely illustrative.

[0082] Based on some examples, a set of target live streaming protocol media information units can be composed of chunks, and several chunks can be composed of fragments.

[0083] The following exemplary code illustrates an exemplary definition of a target live streaming protocol media information unit (e.g., a BMIF box).

[0084]

[0085]

[0086] The field used to identify the frame encapsulation type of the target live streaming protocol media information unit (e.g., the first target live streaming protocol media information unit), as described above, can be defined using frame_type.

[0087] Based on some examples, a target live streaming protocol media information unit (MLI) can be the primary carrier for transmitting audio and video content. Here, the track id identifies which track the MPI belongs to, and the frame_type identifies the frame encapsulation type. For both transmission and playback operations, this frame encapsulation type can be used for operations such as frame dropping or buffering. The chunk_flag indicates whether the MPI is the starting point of a chunk, while the fragment_flag indicates whether it is the beginning of a file segment.

[0088] According to some embodiments of this disclosure, each of one or more first target live streaming protocol media information units further includes: a segmentation flag for identifying whether the first target live streaming protocol media information unit is the starting point of an audio / video segment; and a segmentation file flag for identifying whether the first target live streaming protocol media information unit is the starting point of an audio / video segment file, wherein multiple audio / video segments constitute an audio / video segment file.

[0089] According to some embodiments of this disclosure, the target live streaming protocol media information unit (e.g., the first target live streaming protocol media information unit) is a structure that primarily stores audio and video data, and it supports various flexible framing operations (e.g., allowing half-frame packetization). In some examples, the timestamp of the target live streaming protocol media information unit (e.g., the first target live streaming protocol media information unit) can be in the form of an offset, thereby effectively reducing the size of the information unit and reducing bandwidth consumption.

[0090] It will be understood that the exemplary code described above is merely exemplary and not intended to impose any limitations. Those skilled in the art can make any suitable additions, deletions, modifications, substitutions, etc., to the exemplary code described above after learning of the concept of this disclosure.

[0091] Table 1 below shows a performance comparison between the BMT live streaming protocol and existing mainstream live streaming protocols.

[0092]

[0093]

[0094] Table 1

[0095] The BMT live streaming protocol outperforms existing mainstream live streaming protocols (such as RTMP and CMAF) in terms of performance metrics, including encapsulation overhead and bandwidth consumption at the data encapsulation layer. Specifically, as shown in the table, BMT significantly outperforms RTMP and CMAF protocols in these two performance metrics. Furthermore, although the gap between BMT and FLV (Flash Video) protocols in these two performance metrics is smaller than the gap between BMT and RTMP and CMAF protocols, considering that BMT is compatible with the existing fMP4 standard, thus supporting conversion between multiple protocols and exhibiting strong scalability, the BMT protocol disclosed in this publication still significantly outperforms the FLV protocol.

[0096] Figure 4 A schematic block diagram of a multi-level server architecture 400 according to an embodiment of the present disclosure is shown. As shown, the multi-level server architecture 400 may include a parent server, a secondary server, and a final server. Existing mainstream live streaming protocols require the secondary server to wait for a complete frame to be fully buffered before distributing the streaming media to the final server. That is, when distributing at the secondary server, it is necessary to wait until a complete frame is buffered before distributing to the final server. However, due to the fact that the target live streaming protocol of the present disclosure determines the frame encapsulation type of the target live streaming protocol media information unit (e.g., BMIF) based on network conditions (as opposed to using chunked CHUNKs as the smallest decapsulation unit, the target live streaming protocol media information unit can be part of a complete frame), it is not necessary to wait to receive a complete frame before distributing to downstream entities, thereby effectively reducing startup latency. For example, assuming that originally two target live streaming protocol media information units constitute a complete frame, the secondary server can send only one target live streaming protocol media information unit to the final server first, thereby reducing startup latency at the final server.

[0097] What will be understood is... Figure 4 The multi-level server structure 400 shown is merely illustrative and may include more or fewer intermediate servers between the parent server and the final server. It will also be understood that... Figure 4 The number of target live streaming protocol media information units shown is merely illustrative, and this disclosure does not impose any limitations on it.

[0098] Figure 5 A schematic signaling diagram 500 is shown between a streaming media transmitting device and a streaming media receiving device according to an embodiment of the present disclosure.

[0099] As shown in the figure, client 501 sends a Target Live Protocol Request Information Element (BRIF 510, described below) to server 502. Next, in response to receiving the Target Live Protocol Request Information Element, server 502 sends a Target Live Protocol Acknowledgment Information Element (BAIF 520, described below) to client 501. Immediately following this, server 502 transmits a Target Live Protocol Header Information Element 510 (BHIN 530) and one or more Target Live Protocol Media Information Elements (BMIF 540) to the client.

[0100] What will be understood is... Figure 5 The server 502 shown can be a streaming transmission sending device as described above, and the client 501 can be a streaming transmission receiving device as described above.

[0101] Based on some examples, the target live streaming protocol of this disclosure (e.g., BMT live streaming protocol) can support streaming over HTTP (i.e., BMT over HTTP). For example, pushing a stream can be based on POST or PUT operations under the HTTP protocol, while pulling a stream can be based on GET operations under the HTTP protocol. Both of these methods can carry business information through HTTP parameters, and the transmitted content is similar to that described above. Figure 3 The schematic diagrams of the target live streaming protocol are basically the same, that is, no additional information units are required.

[0102] Based on some examples, the target live streaming protocol of this disclosure (e.g., BMT live streaming protocol) can support streaming over TCP (i.e., BMT over TCP). Since no relevant service information is defined on the TCP protocol, additional information units are needed to define the push or pull streaming method and the associated service information.

[0103] The following exemplary code illustrates the Target Live Streaming Protocol Request Information Unit and the Target Live Streaming Protocol Request Information Unit (i.e., reference to...) Figure 5 Exemplary definitions of BRIF 510 and BAIF 520 as depicted.

[0104] An exemplary definition of BRIF is:

[0105]

[0106] The BRIF box is used for interaction between the client and the server. It is sent from the client to the server and includes fields such as version number, request mode, random number, space name, stream name, expected timestamp, and burl. The above exemplary code is readily understood by those skilled in the art, and will not be elaborated further in this disclosure.

[0107] An exemplary definition of BAIF is:

[0108]

[0109]

[0110] The BRIF box is used for interaction between the client and the server. The server sends this box when responding to the client's BRIF request. `code` represents the status code of this interaction; when `code` equals 1, it indicates that the server requires the client to re-request the new address. `Session_id` is the request / response for a field in the BRIF box; the server verifies this field and returns it to prevent invalid data collisions. The above exemplary code is readily understood by those skilled in the art, and will not be elaborated further in this disclosure.

[0111] According to some embodiments of this disclosure, the method 200 may further include: receiving a target live streaming protocol request information unit from a streaming receiving device, the target live streaming protocol request unit including a request mode field for identifying a request mode; in response to receiving the target live streaming protocol request information unit and the request mode field indicating that the request mode is push streaming, transmitting a target live streaming protocol confirmation information unit to the streaming receiving device, wherein the target live streaming protocol confirmation information unit is transmitted by the streaming sending device before transmitting the target live streaming protocol header information unit.

[0112] According to some embodiments of this disclosure, when a streaming sending device receives a live stream request from a streaming receiving device (e.g., a request for the streaming sending device to push a stream to it), it can send the audio and video data of the live stream together with the response to the request from the streaming receiving device to the streaming receiving device, thereby effectively reducing the number of handshakes between the two parties, thereby reducing startup latency and improving user experience.

[0113] According to some embodiments of this disclosure, the method 200 may further include: transmitting a target live streaming protocol request information unit to a streaming receiving device, the target live streaming protocol request unit including a request mode field for identifying a request mode, and the request mode being pull streaming; receiving a target live streaming protocol confirmation information unit from the streaming receiving device, wherein the target live streaming protocol confirmation information unit is received by the streaming sending device before transmitting the target live streaming protocol header information unit.

[0114] According to some embodiments of this disclosure, when a streaming receiving device receives a streaming request from a streaming sending device, it indicates that the streaming sending device expects the streaming receiving device to actively pull the stream from it. Therefore, when the streaming receiving device sends a confirmation signal (e.g., a target live streaming protocol confirmation information unit) to the streaming sending device, the streaming receiving device can immediately start receiving the live stream (including header information units and media information units) immediately after sending the confirmation signal, thereby effectively reducing the number of handshakes between the two parties, reducing startup latency, and improving user experience.

[0115] Figure 6 A flowchart of a method 600 for streaming a second target live streaming protocol media information unit according to an embodiment of the present disclosure is shown. As shown, method 600 may include: step S610, continuously acquiring current network status information at preset time intervals during streaming one or more first target live streaming protocol media information units to a streaming receiving device, wherein the current network status information includes a metric of the current network performance of the transmission network used for transmitting audio and video; step S620, in response to a difference between the metric indicated by the current network status information and the metric indicated by the initial network status exceeding a predetermined threshold, selecting a second target frame encapsulation type adapted to the current network status information from a plurality of frame encapsulation types, the second target frame encapsulation type being different from the first target frame encapsulation type; step S630, encapsulating the audio and video stream to be transmitted into one or more second target live streaming protocol media information units, each of the one or more second target live streaming protocol media information units including a field for identifying the frame encapsulation type of the second target live streaming protocol media information unit; and step S640, streaming one or more second target live streaming protocol media information units to a streaming receiving device.

[0116] According to some embodiments of this disclosure, method 600 allows the server (e.g., a streaming transmission device, etc.) to select the size of the encapsulated data based on real-time network conditions. When the network conditions are poor, larger audio and video frames can be cut into smaller audio and video frames for transmission, thereby achieving the elimination of the adverse effects of network jitter to a certain extent.

[0117] According to some embodiments of this disclosure, the above method 200 or 600 may additionally include: in response to the completion of streaming, transmitting a target live streaming protocol completion information unit to a streaming receiving device, the target live streaming protocol completion information unit indicating the end of the streaming process. This can reduce unnecessary waiting time at downstream entities (e.g., end-level servers or streaming receiving devices).

[0118] Figure 7 A schematic signaling diagram 700 between a streaming media transmitting device and a streaming media receiving device according to an embodiment of the present disclosure is shown.

[0119] As shown in the figure, after server 702 sends the target live streaming protocol header information element (shown as BHIN 730) to client 701, server 702 also sends information elements containing audio and video metadata files (e.g., shown as ftyp 732 and moov 734) to client 701. Immediately following this, server 702 streams one or more target live streaming protocol media information elements (shown as BMIF 740) to client 701.

[0120] What will be understood is... Figure 7 The server 702 shown can be a streaming transmission sending device as described above, and the client 701 can be a streaming transmission receiving device as described above.

[0121] Figure 8 A flowchart of a method 800 for streaming audio and video according to an embodiment of the present disclosure is shown. According to some embodiments of the present disclosure, method 800 can be applied to a streaming receiving device. Figure 8 As shown, method 800 may include: step S810, providing initial network state information, wherein the initial network state information includes a measure of the initial network performance of the transmission network used for transmitting audio and video, the initial network state information being used by the streaming transmission sending device to select a target frame encapsulation type that matches the initial network state information from a plurality of frame encapsulation types, and to encapsulate the audio and video stream to be transmitted into one or more first target live streaming protocol media information units based at least on the target frame encapsulation type, each of the one or more first target live streaming protocol media information units including a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and step S820, receiving a target live streaming protocol header information unit transmitted by the streaming transmission sending device and one or more first target live streaming protocol media information units streamed by the streaming transmission sending device.

[0122] For a description of the steps in Method 800, please refer to the section above. Figure 2 The content of the method 200 applied to the streaming transmission device is not disclosed and will not be elaborated here.

[0123] Therefore, with the help of method 800, audio and video data can be encapsulated in less than a frame. The server can then select the frame encapsulation type of the audio and video data according to the real-time network conditions. When the network conditions are poor, a complete frame of audio and video can be cut into partial audio and video frames for encapsulation. The streaming receiving device can thus receive live streams encapsulated based on flexible target frame encapsulation types, thereby effectively reducing startup latency at the client end and improving the live streaming playback experience.

[0124] According to some embodiments of this disclosure, each of one or more first target live streaming protocol media information units further includes: a segmentation flag for identifying whether the first target live streaming protocol media information unit is the starting point of an audio / video segment; and a segmentation file flag for identifying whether the first target live streaming protocol media information unit is the starting point of an audio / video segment file, wherein multiple audio / video segments constitute an audio / video segment file.

[0125] According to some embodiments of this disclosure, the target live streaming protocol media information unit (e.g., the first target live streaming protocol media information unit) is a structure that primarily stores audio and video data, and it supports various flexible framing operations (e.g., allowing half-frame packetization). In some examples, the timestamp of the target live streaming protocol media information unit (e.g., the first target live streaming protocol media information unit) can be in the form of an offset, thereby effectively reducing the size of the information unit and reducing bandwidth consumption.

[0126] According to some embodiments of this disclosure, the method 800 described above may further include: transmitting a target live streaming protocol request information unit to a streaming transmission sending device, the target live streaming protocol request unit including a request mode field for identifying a request mode, and the request mode being push streaming; receiving a target live streaming protocol confirmation information unit from the streaming transmission sending device, wherein the target live streaming protocol confirmation information unit is received by the streaming transmission receiving device before receiving the target live streaming protocol header information unit.

[0127] According to some embodiments of this disclosure, after a streaming sending device receives a live stream request from a streaming receiving device (e.g., a request for the streaming sending device to push a stream to it), the streaming receiving device can receive the latter's response to the request from the streaming sending device, along with the audio and video data of the live stream, thereby effectively reducing the number of handshakes between the two parties, thereby reducing startup latency and improving user experience.

[0128] According to some embodiments of this disclosure, the method 800 described above may further include: receiving a target live streaming protocol request information unit from a streaming transmission sending device, the target live streaming protocol request unit including a request mode field for identifying a request mode; in response to receiving the target live streaming protocol request information unit and the request mode field indicating that the request mode is pull streaming, transmitting a target live streaming protocol confirmation information unit to the streaming transmission sending device, wherein the target live streaming protocol confirmation information unit is transmitted by the streaming transmission receiving device before receiving the target live streaming protocol header information unit.

[0129] According to some embodiments of this disclosure, when a streaming receiving device receives a streaming request from a streaming sending device, it indicates that the streaming sending device expects the streaming receiving device to actively pull the stream from it. Therefore, when the streaming receiving device sends a confirmation signal (e.g., a target live streaming protocol confirmation information unit) to the streaming sending device, the streaming receiving device can immediately start receiving the live stream (including header information units and media information units) immediately after sending the confirmation signal, thereby effectively reducing the number of handshakes between the two parties, reducing startup latency, and improving user experience.

[0130] According to some embodiments of this disclosure, the method 800 may further include: during the receipt of one or more first target live streaming protocol media information units streamed by the streaming transmission device, continuously providing current network status information at preset time intervals, wherein the current network status information includes a metric of the current network performance of the transmission network used for transmitting audio and video, the current network status information being used by the streaming transmission device in response to a metric indicated by the current network information differing from a metric indicated by the initial network status by more than a predetermined threshold, selecting a second target frame encapsulation type from a plurality of frame encapsulation types that is compatible with the current network status information, the second target frame encapsulation type being different from the first target frame encapsulation type, and encapsulating the audio and video stream to be transmitted into one or more second target live streaming protocol media information units based at least on the second target frame encapsulation type, each of the one or more second target live streaming protocol media information units including a field for identifying the frame encapsulation type of the second target live streaming protocol media information unit; and receiving one or more second target live streaming protocol media information units streamed by the streaming transmission device.

[0131] Therefore, according to some embodiments of this disclosure, the server (e.g., a streaming transmission device) can select the size of the encapsulated data based on the real-time network conditions. When the network conditions are poor, larger audio and video frames can be cut into smaller audio and video frames for transmission, thereby eliminating the adverse effects of network jitter to a certain extent.

[0132] According to some embodiments of this disclosure, the method 800 described above may additionally include: upon completion of streaming, receiving a target live streaming protocol completion information unit from the streaming sending device, the target live streaming protocol completion information unit indicating the end of the streaming process. This can reduce unnecessary waiting time at downstream entities (e.g., end-level servers or streaming receiving devices).

[0133] Figure 9A structural block diagram of an audio / video streaming apparatus 900 according to an embodiment of the present disclosure is shown. As shown, the apparatus 900 may include: an initialization module 910, configured to acquire initial network state information and configure target live streaming protocol header information units, wherein the initial network state information includes a measure of the initial network performance of the transmission network used for transmitting audio / video; a frame encapsulation selection module 920, configured to select a target frame encapsulation type that matches the initial network state information from a plurality of frame encapsulation types based on the initial network state information; a frame encapsulation execution module 930, configured to encapsulate the audio / video stream to be transmitted into one or more first target live streaming protocol media information units, at least based on the target frame encapsulation type, each of the one or more first target live streaming protocol media information units including a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and a transmission module 940, configured to transmit the target live streaming protocol header information units to a streaming receiving device and stream one or more first target live streaming protocol media information units to the streaming receiving device.

[0134] According to some embodiments of this disclosure, the above-described device 900 can allow audio and video data to be encapsulated in a manner that is less than one frame. The server can then select the frame encapsulation type of the audio and video data according to the real-time network conditions. When the network conditions are poor, a complete frame of audio and video can be cut into partial audio and video frames for encapsulation and streaming, thereby effectively reducing the startup latency at the client end and improving the live streaming experience.

[0135] According to some embodiments of this disclosure, the apparatus 900 may additionally include: a monitoring module 950, configured to continuously acquire current network status information at preset time intervals during the streaming of one or more first target live streaming protocol media information units to a streaming receiving device, wherein the current network status information includes a metric of the current network performance of the transmission network used for transmitting audio and video, wherein the frame encapsulation selection module is further configured to select a second target frame encapsulation type adapted to the current network status information from a plurality of frame encapsulation types in response to a difference between the metric indicated by the current network status information and the metric indicated by the initial network status exceeding a predetermined threshold, the second target frame encapsulation type being different from the first target frame encapsulation type, wherein the frame encapsulation execution module is further configured to encapsulate the audio and video stream to be transmitted into one or more second target live streaming protocol media information units, at least based on the second target frame encapsulation type, each of the one or more second target live streaming protocol media information units including a field for identifying the frame encapsulation type of the second target live streaming protocol media information unit, and wherein the transmission module is further configured to stream one or more second target live streaming protocol media information units to the streaming receiving device.

[0136] Figure 10A structural block diagram of an audio / video streaming apparatus 1000 according to an embodiment of the present disclosure is shown. As shown, the apparatus 1000 may include: a monitoring module 1010, configured to provide initial network status information, wherein the initial network status information includes a measure of the initial network performance of the transmission network used for transmitting audio / video, the initial network status information being used by the streaming transmission device to select a target frame encapsulation type adapted to the initial network status information from a plurality of frame encapsulation types, and to encapsulate the audio / video stream to be transmitted into one or more first target live streaming protocol media information units, at least based on the target frame encapsulation type, each of the one or more first target live streaming protocol media information units including a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and a receiving module 1020, configured to receive a target live streaming protocol header information unit transmitted by the streaming transmission device and one or more first target live streaming protocol media information units streamed by the streaming transmission device.

[0137] According to some embodiments of this disclosure, the apparatus 1000 can allow audio and video data to be encapsulated in a manner that is less than one frame. The server can therefore select the frame encapsulation type of the audio and video data based on real-time network conditions. When network conditions are poor, a complete frame of audio and video can be segmented into partial audio and video frames for encapsulation. The streaming receiving device can thus receive live streams encapsulated based on flexible target frame encapsulation types, thereby effectively reducing startup latency at the client end and improving the live streaming playback experience.

[0138] According to some embodiments of this disclosure, the monitoring module is further configured to continuously provide current network status information at preset time intervals during the reception of one or more first target live streaming protocol media information units streamed by the streaming transmission device. The current network status information includes a metric of the current network performance of the transmission network used for transmitting audio and video. The current network status information is used by the streaming transmission device in response to a metric indicated by the current network information differing from a metric indicated by the initial network status by more than a predetermined threshold. The second target frame encapsulation type is selected from a plurality of frame encapsulation types to be compatible with the current network status information. The second target frame encapsulation type is different from the first target frame encapsulation type, and the audio and video stream to be transmitted is encapsulated into one or more second target live streaming protocol media information units based at least on the second target frame encapsulation type. Each of the one or more second target live streaming protocol media information units includes a field for identifying the frame encapsulation type of the second target live streaming protocol media information unit. The receiving module is further configured to receive one or more second target live streaming protocol media information units streamed by the streaming transmission device.

[0139] It should be understood that Figure 9-10 The various modules of the devices 900 and 1000 shown can be connected to the reference. Figure 2and Figure 8 The steps in methods 200 and 800 described correspond to each other. Therefore, the operations, features, and advantages described above for methods 200 and 800 are equally applicable to apparatuses 900-1000 and their constituent units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0140] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and at least one memory communicatively connected to the at least one processor; wherein the at least one memory stores a computer program that, when executed by the at least one processor, implements the above-described method for streaming audio and video.

[0141] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-described method for audio and video streaming when executed by a processor.

[0142] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the above-described method for streaming audio and video.

[0143] See Figure 11 The present invention describes a structural block diagram of an electronic device 1100 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0144] like Figure 11 As shown, the electronic device 1100 may include at least one processor 1110, working memory 1120, input unit 1140, display unit 1150, speaker 1160, storage unit 1170, communication unit 1180 and other output units 1190 that are capable of communicating with each other via system bus 1130.

[0145] Processor 1110 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 1110 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Processor 1110 may be configured to acquire and execute computer-readable instructions stored in working memory 1120, storage unit 1170, or other computer-readable media, such as program code of operating system 1120a, program code of application program 1120b, etc.

[0146] Working memory 1120 and storage cell 1170 are examples of computer-readable storage media for storing instructions executed by processor 1110 to perform the various functions described above. Working memory 1120 may include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, storage cell 1170 may include hard disk drives, solid-state drives, removable media including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Working memory 1120 and storage cell 1170 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 1110 as a specific machine configured to perform the operations and functions described in the examples herein.

[0147] Input unit 1160 can be any type of device capable of inputting information to electronic device 1100. Input unit 1160 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 1160 can be any type of device capable of presenting information, and may include, but is not limited to, display unit 1150, speaker 1160, and other output units 1190. Other output units 1190 may include, but are not limited to, video / audio output terminals, vibrators, and / or printers. Communication unit 1180 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth. TM Equipment, 802.11 equipment, Wi-Fi equipment, WiMAX equipment, cellular communication equipment and / or the like.

[0148] The application program 1120b in working register 1120 can be loaded to execute the various methods and processes described above, for example... Figure 2 Steps S210-S240 in the process Figure 6 Steps S610-S640 in the process Figure 8 Steps S810-S820 in the above process. For example, in some embodiments, methods 200, 600, and / or 800 described above may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1170. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 1100 via storage unit 1170 and / or communication unit 1180. When the computer program is loaded and executed by processor 1110, one or more steps of methods 200, 600, and / or 800 described above may be performed. Alternatively, in other embodiments, processor 1110 may be configured to perform methods 200, 600, and / or 800 by any other suitable means (e.g., by means of firmware).

[0149] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0151] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0152] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0153] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0154] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0155] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0156] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for streaming audio and video, applied to a streaming transmission device, the method comprising: Obtain initial network status information and configure the target live streaming protocol header information unit, wherein the initial network status information includes a measure of the initial network performance of the transmission network used to transmit the audio and video; Based on the initial network state information, a first target frame encapsulation type that is compatible with the initial network state information is selected from multiple frame encapsulation types, wherein the first target frame encapsulation type allows audio and video data to be encapsulated in a manner that is less than one frame. At least based on the first target frame encapsulation type, the audio and video streams to be transmitted are encapsulated into one or more first target live streaming protocol media information units, each of the one or more first target live streaming protocol media information units including a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and The target live streaming protocol header information unit is transmitted to the streaming receiving device, and one or more first target live streaming protocol media information units are streamed to the streaming receiving device.

2. The method according to claim 1, wherein, Each of the one or more first target live streaming protocol media information units further includes: A segmentation flag used to identify whether the first target live streaming protocol media information unit is the starting point of an audio / video segment; and A segment file flag used to identify whether the first target live streaming protocol media information unit is the starting point of an audio / video segment file. Multiple audio and video blocks are combined to form an audio and video segment file.

3. The method according to claim 1 or 2, further comprising: The target live streaming protocol request information unit is received from the streaming receiving device, wherein the target live streaming protocol request information unit includes a request mode field for identifying the request mode; In response to receiving the target live streaming protocol request information unit and the request mode field indicating that the request mode is push streaming, a target live streaming protocol confirmation information unit is transmitted to the stream transmission receiving device. The target live streaming protocol confirmation information unit is transmitted by the streaming transmission device before transmitting the target live streaming protocol header information unit.

4. The method according to claim 1 or 2, further comprising: The target live streaming protocol request information unit is transmitted to the streaming receiving device. The target live streaming protocol request information unit includes a request mode field for identifying the request mode, and the request mode is pull streaming. Receive target live streaming protocol confirmation information unit from the streaming receiving device. The target live streaming protocol confirmation information unit is received by the streaming transmission device before transmitting the target live streaming protocol header information unit.

5. The method according to claim 1 or 2, further comprising: During the streaming of one or more first target live streaming protocol media information units to the streaming receiving device, current network status information is continuously acquired at preset time intervals, wherein the current network status information includes a measure of the current network performance of the transmission network used for transmitting the audio and video. In response to a metric indicated by the current network state information differing from the metric indicated by the initial network state by more than a predetermined threshold, a second target frame encapsulation type adapted to the current network state information is selected from the plurality of frame encapsulation types. The second target frame encapsulation type is different from the first target frame encapsulation type. At least based on the second target frame encapsulation type, the audio and video streams to be transmitted are encapsulated into one or more second target live streaming protocol media information units, each of the one or more second target live streaming protocol media information units including a field for identifying the frame encapsulation type of the second target live streaming protocol media information unit; and The one or more second target live streaming protocol media information units are streamed to the streaming receiving device.

6. The method according to claim 1 or 2, further comprising: In response to the completion of the streaming, a target live streaming protocol completion information unit is transmitted to the streaming receiving device, the target live streaming protocol completion information unit indicating the end of the streaming process.

7. A method for streaming audio and video, applied to a streaming receiving device, the method comprising: The system provides initial network state information, which includes a metric of the initial network performance of the transmission network used to transmit the audio and video. This initial network state information is used by the streaming transmission device to select a first target frame encapsulation type from a plurality of frame encapsulation types that matches the initial network state information, and to encapsulate the audio and video stream to be transmitted into one or more first target live streaming protocol media information units (MLIMs) based at least on the first target frame encapsulation type. Each of the one or more first target live streaming protocol media information units includes a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit, wherein the first target frame encapsulation type allows encapsulation of audio and video data in a manner that is less than one frame. Receives a target live streaming protocol header information unit transmitted by the streaming transmission device and one or more first target live streaming protocol media information units streamed by the streaming transmission device.

8. The method according to claim 7, wherein, Each of the one or more first target live streaming protocol media information units further includes: A segmentation flag used to identify whether the first target live streaming protocol media information unit is the starting point of an audio / video segment; and A segment file flag used to identify whether the first target live streaming protocol media information unit is the starting point of an audio / video segment file. Multiple audio and video blocks are combined to form an audio and video segment file.

9. The method according to claim 7 or 8, further comprising: The target live streaming protocol request information unit is transmitted to the streaming transmission sending device. The target live streaming protocol request information unit includes a request mode field for identifying the request mode, and the request mode is push streaming. Receive target live streaming protocol confirmation information unit from the streaming transmission device. The target live streaming protocol confirmation information unit is received by the streaming receiving device before receiving the target live streaming protocol header information unit.

10. The method according to claim 7 or 8, further comprising: The target live streaming protocol request information unit is received from the streaming transmission sending device, wherein the target live streaming protocol request information unit includes a request mode field for identifying the request mode; In response to receiving the target live streaming protocol request information unit and the request mode field indicating that the request mode is pull streaming, a target live streaming protocol confirmation information unit is transmitted to the stream transmission sending device. The target live streaming protocol confirmation information unit is transmitted by the streaming receiving device before receiving the target live streaming protocol header information unit.

11. The method according to claim 7 or 8, further comprising: During the reception of one or more first target live streaming protocol media information units (MLIMs) streamed by the streaming transmission device, current network status information is continuously provided at preset time intervals. The current network status information includes a metric of the current network performance of the transmission network used for transmitting the audio and video. The current network status information is used by the streaming transmission device in response to a metric indicated by the current network status information differing from a metric indicated by the initial network status by more than a predetermined threshold. In this response, the streaming transmission device selects a second target frame encapsulation type from a plurality of frame encapsulation types that matches the current network status information. The second target frame encapsulation type is different from the first target frame encapsulation type, and the audio and video stream to be transmitted is encapsulated into one or more second target live streaming protocol media information units, at least based on the second target frame encapsulation type. Each of the one or more second target live streaming protocol media information units includes a field identifying the frame encapsulation type of the second target live streaming protocol media information unit. Receive one or more second target live streaming protocol media information units streamed by the streaming transmission device.

12. The method according to claim 7 or 8, further comprising: When the streaming is completed, a target live streaming protocol completion information unit is received from the streaming sending device, the target live streaming protocol completion information unit indicating the end of the streaming process.

13. An audio / video streaming apparatus, the apparatus comprising: An initialization module is used to obtain initial network status information and configure the target live streaming protocol header information unit, wherein the initial network status information includes a measure of the initial network performance of the transmission network used to transmit the audio and video. The frame encapsulation selection module is used to select a first target frame encapsulation type that matches the initial network state information from multiple frame encapsulation types based on the initial network state information, wherein the first target frame encapsulation type allows audio and video data to be encapsulated in a manner that is less than one frame. A frame encapsulation execution module is configured to encapsulate the audio and video stream to be transmitted into one or more first target live streaming protocol media information units, based at least on the first target frame encapsulation type, wherein each of the one or more first target live streaming protocol media information units includes a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit; and The transmission module is used to transmit the target live streaming protocol header information unit to the streaming receiving device, and to stream one or more first target live streaming protocol media information units to the streaming receiving device.

14. The apparatus of claim 13, further comprising: The monitoring module is configured to continuously acquire current network status information at preset time intervals during the streaming of one or more first target live streaming protocol media information units to the streaming receiving device, wherein the current network status information includes a measure of the current network performance of the transmission network used for transmitting the audio and video. The frame encapsulation selection module is further configured to, in response to a metric indicated by the current network state information differing from the metric indicated by the initial network state information by more than a predetermined threshold, select a second target frame encapsulation type from the plurality of frame encapsulation types that is compatible with the current network state information, wherein the second target frame encapsulation type is different from the first target frame encapsulation type. The frame encapsulation execution module is further configured to encapsulate the audio and video stream to be transmitted into one or more second target live streaming protocol media information units, based at least on the second target frame encapsulation type. Each of the one or more second target live streaming protocol media information units includes a field for identifying the frame encapsulation type of the second target live streaming protocol media information unit. The transmission module is further configured to stream one or more second target live streaming protocol media information units to the streaming receiving device.

15. An audio / video streaming apparatus, the apparatus comprising: A monitoring module is configured to provide initial network status information, wherein the initial network status information includes a measure of the initial network performance of the transmission network used to transmit the audio and video. The initial network status information is used by the streaming transmission device to select a first target frame encapsulation type adapted to the initial network status information from a plurality of frame encapsulation types, and to encapsulate the audio and video stream to be transmitted into one or more first target live streaming protocol media information units (MLIMs) based at least on the first target frame encapsulation type. Each of the one or more first target live streaming protocol media information units includes a field for identifying the frame encapsulation type of the first target live streaming protocol media information unit, wherein the first target frame encapsulation type allows encapsulation of audio and video data in a manner less than one frame. The receiving module is configured to receive a target live streaming protocol header information unit transmitted by the streaming transmission sending device and one or more first target live streaming protocol media information units streamed by the streaming transmission sending device.

16. The apparatus according to claim 15, in, The monitoring module is further configured to continuously provide current network status information at preset time intervals during the reception of one or more first target live streaming protocol media information units streamed by the streaming transmission device. The current network status information includes a metric of the current network performance of the transmission network used to transmit the audio and video. The current network status information is used by the streaming transmission device in response to a difference between the metric indicated by the current network status information and the metric indicated by the initial network status exceeding a predetermined threshold. In this response, the streaming transmission device selects a second target frame encapsulation type from the plurality of frame encapsulation types that matches the current network status information. The second target frame encapsulation type is different from the first target frame encapsulation type, and the audio and video stream to be transmitted is encapsulated into one or more second target live streaming protocol media information units, at least based on the second target frame encapsulation type. Each of the one or more second target live streaming protocol media information units includes a field for identifying the frame encapsulation type of the second target live streaming protocol media information unit. The receiving module is further configured to receive one or more second target live streaming protocol media information units streamed by the streaming transmission sending device.

17. An electronic device comprising: At least one processor; as well as At least one memory communicatively connected to the at least one processor The at least one memory stores a computer program that, when executed by the at least one processor, implements the method according to any one of claims 1-12.

18. A non-transitory computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the method according to any one of claims 1-12.

19. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method according to any one of claims 1-12.

Images