Live streaming media data processing method, system, device and computer equipment

Abstract

The present application relates to a method, system, device, computer equipment, storage medium and computer program product for processing live streaming media data. The method includes: caching live streaming media source data, slicing the cached live streaming media source data according to a preset slicing granularity in response to the streaming media duration of the cached live streaming media source data reaching a target caching duration, and obtaining a plurality of independently coded streaming media segments; encoding the streaming media segments obtained by slicing in parallel through at least two encoding servers whose overall encoding rate is not less than the frame rate of the live streaming media source data, and obtaining the respective coded video data of each encoding server; merging the respective coded video data of each encoding server, and obtaining the live streaming media coded data for sending to the live streaming client. The use of this method can ensure the real-time nature of the live broadcast.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a live streaming media data processing method, system, device, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, live broadcasting has become more and more diverse. Live broadcasting through video can publish content such as product displays, related meetings, background introductions, program evaluations, online surveys, dialogue interviews, online training, and game entertainment on the Internet. The Internet's intuitiveness, speed, good presentation, rich content, and strong interactivity can be used to enhance the effectiveness of live broadcasting. With the development of audio and video technology, the amount of data in streaming media such as audio and video in live broadcasting has increased dramatically, and the computing power requirements for real-time encoding have become increasingly high. During live broadcasting, playback freezes are prone to occur, affecting the real-time nature of live broadcasting. Summary of the invention

[0003] Based on this, it is necessary to provide a live streaming media data processing method, system, device, computer equipment, computer-readable storage medium and computer program product that can ensure the real-time performance of live broadcast in response to the above technical problems.

[0004] In a first aspect, the present application provides a method for processing live streaming media data. The method comprises:

[0005] caching live streaming source data, and in response to the streaming duration of the cached live streaming source data reaching a target caching duration, slicing the cached live streaming source data according to a preset slicing granularity to obtain a plurality of streaming segments that can be independently encoded; the target caching duration is positively correlated with the preset slicing granularity;

[0006] The streaming media segments obtained by slicing are encoded in parallel by at least two encoding servers to obtain encoded video data of each encoding server; the overall encoding rate of the at least two encoding servers is not less than the frame rate of the live streaming media source data;

[0007] The encoded video data of each encoding server are combined to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

[0008] In a second aspect, the present application provides a live streaming media data processing system. The system includes: a slicing server, a merging server, and an encoding server cluster including at least two encoding servers;

[0009] The slicing server is used to cache live streaming media data, and in response to the streaming media duration of the cached live streaming media data reaching the target caching duration, the cached live streaming media data is sliced ​​according to a preset slicing granularity to obtain a plurality of streaming media segments that can be independently encoded; the target caching duration is positively correlated with the preset slicing granularity and is positively correlated with the number of encoding servers in the encoding server cluster;

[0010] The encoding server cluster is used to encode the streaming media segments obtained by slicing in parallel through at least two encoding servers to obtain the encoded video data of each encoding server;

[0011] The merging server is used to merge the encoded video data of each encoding server to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

[0012] In a third aspect, the present application also provides a live streaming media data processing device. The device comprises:

[0013] A live streaming slicing module is used to cache live streaming source data, and in response to the streaming duration of the cached live streaming source data reaching the target caching duration, the cached live streaming source data is sliced ​​according to a preset slicing granularity to obtain a plurality of streaming segments that can be independently encoded; the target caching duration is positively correlated with the preset slicing granularity;

[0014] The slice parallel encoding module is used to encode the streaming media segments obtained by slicing in parallel through at least two encoding servers to obtain the encoded video data of each encoding server; the overall encoding rate of the at least two encoding servers is not less than the frame rate of the live streaming media source data;

[0015] The coding data merging module is used to merge the coded video data of each coding server to obtain live streaming media coded data; the live streaming media coded data is used to send to the live client.

[0016] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0017] caching live streaming source data, and in response to the streaming duration of the cached live streaming source data reaching a target caching duration, slicing the cached live streaming source data according to a preset slicing granularity to obtain a plurality of streaming segments that can be independently encoded; the target caching duration is positively correlated with the preset slicing granularity;

[0018] The streaming media segments obtained by slicing are encoded in parallel by at least two encoding servers to obtain encoded video data of each encoding server; the overall encoding rate of the at least two encoding servers is not less than the frame rate of the live streaming media source data;

[0019] The encoded video data of each encoding server are merged to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

[0020] In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0021] caching live streaming source data, and in response to the streaming duration of the cached live streaming source data reaching a target caching duration, slicing the cached live streaming source data according to a preset slicing granularity to obtain a plurality of streaming segments that can be independently encoded; the target caching duration is positively correlated with the preset slicing granularity;

[0022] The streaming media segments obtained by slicing are encoded in parallel by at least two encoding servers to obtain encoded video data of each encoding server; the overall encoding rate of the at least two encoding servers is not less than the frame rate of the live streaming media source data;

[0023] The encoded video data of each encoding server are merged to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

[0024] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0025] caching live streaming source data, and in response to the streaming duration of the cached live streaming source data reaching a target caching duration, slicing the cached live streaming source data according to a preset slicing granularity to obtain a plurality of streaming segments that can be independently encoded; the target caching duration is positively correlated with the preset slicing granularity;

[0026] The streaming media segments obtained by slicing are encoded in parallel by at least two encoding servers to obtain encoded video data of each encoding server; the overall encoding rate of the at least two encoding servers is not less than the frame rate of the live streaming media source data;

[0027] The encoded video data of each encoding server are merged to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

[0028] The live streaming media data processing method, system, device, computer equipment, storage medium and computer program product described above, when the streaming media duration of the cached live streaming media source data reaches the target cache duration, triggers slicing of the cached live streaming media source data according to a preset slicing granularity to obtain multiple streaming media segments that can be independently encoded, and performs parallel encoding of the streaming media segments obtained by slicing through at least two encoding servers whose overall encoding rate is not less than the frame rate of the live streaming media source data, and merges the encoded video data obtained by each encoding server to obtain live streaming media encoded data for sending to the live streaming client. In the processing of live streaming media data, the streaming media duration of the cached live streaming media source data reaches the target cache duration, and the overall encoding rate of at least two encoding servers is not less than the frame rate of the live streaming media source data, which can ensure that the processing efficiency of the parallel encoding of the streaming media segments obtained by slicing through at least two encoding servers matches the frame rate of the live streaming media source data, avoiding jamming due to encoding processing efficiency problems during the live broadcast process, thereby ensuring the real-time nature of the live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A diagram of an application environment of a live streaming media data processing method in one embodiment;

[0030] Figure 2 A flowchart of a method for processing live streaming media data in one embodiment;

[0031] Figure 3 A schematic diagram of a parallel encoding process in one embodiment;

[0032] Figure 4 is a schematic diagram of video resolution comparison in one embodiment;

[0033] Figure 5 A schematic diagram of a flow chart of audio and video processing in one embodiment;

[0034] Figure 6 A schematic diagram of a video encoding process in one embodiment;

[0035] Figure 7 is a schematic diagram of coding tree unit segmentation in one embodiment;

[0036] Figure 8 A flowchart of a live streaming media data processing method according to another embodiment;

[0037] Fig. 9 A schematic diagram of an interface for ultra-high-definition video live broadcast in one embodiment;

[0038] Fig.10 A structural block diagram of a live streaming media data processing system in one embodiment;

[0039] Fig.11 A structural block diagram of a live streaming media data processing device in one embodiment;

[0040] Fig.12 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0042] The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Specifically, the live streaming media data processing method provided in the embodiments of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal includes a first terminal 1021 and a second terminal 1022, the first terminal 1021 communicates with the slice server 104 through the network, the second terminal 1022 communicates with the merge server 108 through the network, and the encoding server 106 communicates with the slice server 104 and the merge server 108 through the network. The data storage system can store data that needs to be processed by the corresponding server. The data storage system can be integrated on the corresponding server, or it can be placed on the cloud or other servers. The anchor user can collect live streaming media data through the first terminal 1021, and send the collected live streaming media data to the slicing server 104. The slicing server 104 caches the live streaming media source data sent by the first terminal 1021. When the streaming media duration of the cached live streaming media source data reaches the target cache duration, it triggers slicing of the cached live streaming media source data according to the preset slicing granularity to obtain multiple streaming media segments that can be independently encoded, and sends the multiple streaming media segments to at least two encoding servers 106. At least two encoding servers 106 whose overall encoding rate is not less than the frame rate of the live streaming media source data perform parallel encoding on the streaming media segments obtained by slicing, and send the obtained encoded video data of each encoding server to the merging server 108. The merging server 108 merges the encoded video data of each encoding server to obtain the live streaming media encoded data for sending to the live client. The merging server 108 can send the merged live streaming media encoded data to each second terminal 1022, so that the second terminal 1022 can decode and play the live streaming media encoded data, and the audience user can watch the network live broadcast through the second terminal 1022.

[0043] Among them, the terminals including the first terminal 1021 and the second terminal 1022 can be, but are not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Each server such as the slicing server 104, the encoding server 106 and the merge server 108 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0044] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0045] Among them, cloud computing refers to the delivery and use mode of IT infrastructure, which means obtaining required resources through the network in an on-demand and easily scalable manner; in a broad sense, cloud computing refers to the delivery and use mode of services, which means obtaining required services through the network in an on-demand and easily scalable manner. This service can be related to IT and software, the Internet, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the promotion of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from the previous parallel and distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model in terms of concept.

[0046] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions. At present, the storage method of the storage system is to create a logical volume. When creating a logical volume, a physical storage space is allocated for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, the data is stored on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object. Therefore, when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object. The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0047] In one embodiment, Figure 2 As shown, a method for processing live streaming media data is provided, and the method is executed by a computer device. In this embodiment, the method is applied to Figure 1 The following steps are used to illustrate the various servers in the example:

[0048] Step 202, cache the live streaming source data, in response to the streaming duration of the cached live streaming source data reaching the target cache duration, slice the cached live streaming source data according to a preset slice granularity to obtain multiple streaming segments that can be independently encoded; the target cache duration is positively correlated with the preset slice granularity.

[0049] Among them, streaming media technology refers to a technology that compresses a series of media data and transmits them in segments in the network in a stream manner to achieve real-time transmission of audio and video on the network for viewing. Live streaming media refers to the streaming media generated in network live broadcasts. The live streaming media source data is the original streaming media data directly generated by the anchor during the live broadcast. Streaming media duration refers to the cumulative duration of the cached live streaming media source data when caching the live streaming media source data. The target cache duration is a pre-set cache duration that needs to be achieved. The target cache duration can be set according to actual needs to ensure that parallel encoding processing can be performed through multiple encoding servers. Slice granularity refers to the duration of a unit slice when slicing live streaming media source data. The target cache duration is positively correlated with the preset slice granularity, that is, the higher the preset slice granularity, the larger the corresponding target cache duration value, that is, more live streaming media source data needs to be cached before triggering slicing processing. Streaming media segments refer to independently coded slice segments obtained after slicing the live streaming media source data. Each streaming media segment is an independently encodable segment, which can avoid cross-influence between different streaming media segments, thereby ensuring that the merged encoded video data can be played smoothly and clearly.

[0050] Specifically, when the anchor user triggers a live broadcast on the Internet, the server can cache the live streaming source data generated by the anchor end, and the server counts the streaming duration of the cached live streaming source data. When the streaming duration reaches the preset target cache duration, it indicates that the server has cached a sufficient amount of live streaming source data, and can support multiple encoding servers for parallel encoding. The server slices the cached live streaming source data according to the preset slice granularity, and specifically can be divided into equal time lengths according to the preset slice granularity to obtain multiple streaming media segments with the same duration that can be independently encoded. The target cache duration is positively correlated with the preset slice granularity. The slice granularity can be set according to actual needs, such as 0.5 seconds, 1 second or 2 seconds, etc., and based on the set slice granularity, the corresponding target cache duration can be set so that the target cache duration is positively correlated with the preset slice granularity. For example, when the preset slice granularity is 1 second, the target cache duration can be set to 12 seconds, and when the preset slice granularity is 0.5 seconds, the target cache duration can be set to 6 seconds.

[0051] Step 204: at least two encoding servers are used to encode the streaming media segments obtained by slicing in parallel to obtain encoded video data of each encoding server; the overall encoding rate of the at least two encoding servers is not less than the frame rate of the live streaming media source data.

[0052] The encoding server is a server used for video encoding processing, and the number of encoding servers is at least two, so that at least two encoding servers can perform parallel encoding processing. In the process of parallel encoding, each encoding server can encode the assigned streaming media segment. For example, the obtained streaming media segment includes 12 segments, and the slicing granularity of each streaming media segment is 1 second, that is, the duration of each streaming media segment is 1 second, then 12 encoding servers can encode a streaming media segment respectively, so that the 12 encoding servers simultaneously perform synchronous encoding on the 12 streaming media segments obtained by slicing, and obtain the encoded video data of each encoding server. The encoded video data is the video encoding result obtained by the encoding server for the streaming media segment. The streaming media segment can be compressed by encoding to reduce the data volume of the streaming media segment and obtain the encoded video data that is easy to transmit and store. The overall encoding rate refers to the encoding rate of each encoding server as a whole when each encoding server performs encoding in parallel, and the frame rate of the live streaming media source data refers to the number of pictures generated per unit time of the live streaming media source data.

[0053] Specifically, at least two encoding servers are used to encode the streaming media segments obtained by slicing in parallel, and each encoding server obtains its own encoded video data, and the number of encoding servers matches the number of streaming media segments obtained by slicing. In specific applications, the number of encoding servers that trigger parallel encoding in a unit time is the same as the number of streaming media segments obtained by slicing in a unit time, so that each streaming media segment obtained by slicing in a unit time can be encoded in parallel by different encoding servers, which can improve the encoding processing efficiency. The overall encoding rate of at least two encoding servers performing parallel encoding is not less than the frame rate of the live streaming media source data, so that the overall encoding rate of parallel encoding can match the frame rate of the live streaming media source data, avoiding jamming due to encoding processing efficiency problems during live broadcasting, ensuring smooth playback of live streaming media, and thus ensuring the real-time nature of live broadcasting.

[0054] Step 206: merge the encoded video data of each encoding server to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

[0055] The live streaming media encoding data is the encoding merging result obtained by merging the encoded video data obtained by each encoding server. The live streaming media encoding data is used to send to the live client to be played in the live client. Specifically, the server can merge the encoded video data of each encoding server to obtain the live streaming media encoding data for sending to the live client, thereby realizing the encoding processing of the live streaming media data. In specific applications, the processing of merging the encoded video data of each encoding server and the processing of slicing the cached live streaming media source data according to the preset slicing granularity can be performed by the same server or by different servers. For example, the processing of slicing the cached live streaming media source data according to the preset slicing granularity can be performed by a slicing server, the parallel encoding of the streaming media segments obtained by slicing can be performed by multiple encoding servers, and the merging of the encoded video data of each encoding server can be performed by a merging server. The slicing server and the merging server can be set to the same server or different servers set separately.

[0056] In the above-mentioned live streaming media data processing method, when the streaming duration of the cached live streaming media source data reaches the target cache duration, it is triggered to slice the cached live streaming media source data according to the preset slice granularity to obtain multiple streaming media segments that can be independently encoded, and the streaming media segments obtained by slicing are encoded in parallel through at least two encoding servers whose overall encoding rate is not less than the frame rate of the live streaming media source data, and the encoded video data obtained by each encoding server are merged to obtain live streaming media encoded data for sending to the live client. In the processing of live streaming media data, the streaming duration of the cached live streaming media source data reaches the target cache duration, and the overall encoding rate of at least two encoding servers is not less than the frame rate of the live streaming media source data, which can ensure that the processing efficiency of parallel encoding of the streaming media segments obtained by slicing by at least two encoding servers matches the frame rate of the live streaming media source data, avoiding the jamming caused by the encoding processing efficiency problem during the live broadcast process, thereby ensuring the real-time nature of the live broadcast.

[0057] In one embodiment, the preset slicing granularity includes an integer multiple granularity of a preset picture group; in response to the streaming duration of the cached live streaming source data reaching the target cache duration, slicing the cached live streaming source data according to the preset slicing granularity to obtain multiple streaming segments that can be independently encoded, including: counting the streaming duration of the cached live streaming source data; when the streaming duration reaches the preset target cache duration, slicing the cached live streaming source data according to the integer multiple granularity of the preset picture group to obtain multiple streaming segments that can be independently encoded.

[0058] The preset slice granularity includes an integer multiple granularity of a preset picture group, where a picture group refers to an interval between two intra-coded image frames during video encoding, and an intra-coded image frame refers to an image frame that can be encoded using only information of the frame without reference to other image frames. The preset slice granularity is an integer multiple granularity of a preset picture group, thereby ensuring that the streaming media segments obtained by slicing according to the preset slice granularity are all independently encodable slice segments.

[0059] Specifically, when the server caches live streaming source data, it can count the streaming duration of the cached live streaming source data, and when the streaming duration reaches the preset target cache duration, it indicates that the currently cached live streaming source data can already support multiple encoding servers for parallel encoding processing, and then the server slices the cached live streaming source data according to the integer multiple granularity of the preset picture group to obtain multiple streaming segments that can be independently encoded. During the encoding process, each picture group can be independently encoded, and slicing the live streaming source data according to the integer multiple granularity of the picture group can ensure that the multiple streaming segments obtained by slicing can be independently encoded, which is conducive to ensuring the playback effect of the live streaming encoded data.

[0060] In this embodiment, the server slices the cached live streaming source data according to the granularity of an integer multiple of the preset picture group, which can ensure that the multiple streaming segments obtained by slicing are independently encodable segments, and effectively isolate the encoding and decoding between each streaming segment, thereby facilitating the playback effect of the live streaming encoded data.

[0061] In one embodiment, the cached live streaming source data is sliced ​​according to the granularity of an integer multiple of a preset picture group to obtain a plurality of independently encodable streaming segments, including: slicing the cached live streaming source data at equal granularity according to the granularity of an integer multiple of a preset picture group to obtain a plurality of live streaming slice data of the same duration and independently encodable; and encapsulating the plurality of live streaming slice data respectively according to a target encapsulation format to obtain a plurality of streaming segments belonging to the target encapsulation format.

[0062] Among them, equal-granularity slicing means that the granularity of each segment is the same when slicing, such as 1 second or 0.5 second. Live streaming media slice data is obtained by slicing the cached live streaming media source data according to equal granularity. Each live streaming media slice data is obtained by slicing according to equal granularity, that is, the duration of each live streaming media slice data is the same; and each live streaming media slice data is obtained by slicing according to the granularity of an integer multiple of the preset picture group, that is, each live streaming media slice data is an independently encodable slice segment. The encapsulation format, also called a container, is to put the encoded and compressed video track and audio track into a file according to a certain format. As a container of the encoded video, the encapsulation format can include head / tail identifiers and video description information to identify the video included in the encapsulation format. The encapsulation format can include various formats such as avi, flv, MP4, rmvb, mkv or ts. The target encapsulation format is a pre-set encapsulation format.

[0063] Specifically, the server slices the cached live streaming source data at equal granularity according to the granularity of an integer multiple of a preset picture group, and obtains a plurality of live streaming slice data that can be independently encoded, and each live streaming slice data has the same duration. The server queries a preset target encapsulation format, and respectively encapsulates the plurality of live streaming slice data according to the target encapsulation format, and obtains a plurality of streaming segments belonging to the target encapsulation format. The server transcapsulates the sliced ​​live streaming slice data into the target encapsulation format, so as to obtain a plurality of streaming segments whose encapsulation format is the target encapsulation format, such as obtaining a plurality of streaming segments whose encapsulation format is the ts format.

[0064] In this embodiment, the server slices the cached live streaming source data into equal granularity according to the granularity of an integer multiple of the preset picture group, which can ensure that the multiple streaming media segments obtained by slicing are segments of the same duration and can be independently encoded, and the live streaming media slice data obtained by slicing is packaged into multiple streaming media segments belonging to the target encapsulation format, so that the encoding and decoding of each streaming media segment are effectively isolated, which is conducive to ensuring the playback effect of the live streaming media encoded data.

[0065] In one embodiment, the streaming media segments are a target number, the target number is positively correlated with the frame rate of the live streaming media source data, and the target number is negatively correlated with the independent encoding rate of the encoding server; the streaming media segments obtained by slicing are encoded in parallel by at least two encoding servers to obtain the encoded video data of each encoding server, including: the streaming media segments obtained by slicing are encoded in parallel by at least the target number of encoding servers to obtain the encoded video data of each encoding server; wherein the overall encoding rate of at least the target number of encoding servers is not less than the frame rate of the live streaming media source data.

[0066] Among them, the streaming media segment is the target number, that is, the target number of streaming media segments is obtained by slicing. The target number is positively correlated with the frame rate of the live streaming source data, and the target number is negatively correlated with the independent encoding rate of the encoding server, that is, the larger the frame rate of the live streaming source data, the larger the target number; and the larger the independent encoding rate of the encoding server, the smaller the target number. The independent encoding rate refers to the encoding rate when a single encoding server performs encoding independently.

[0067] Specifically, when parallel encoding is performed through multiple encoding servers, the streaming media segments obtained by slicing are encoded in parallel through at least a target number of encoding servers to obtain the encoded video data of each encoding server. That is, the number of encoding servers is not less than the number of streaming media segments, and each streaming media segment can correspond to an encoding server, so that the streaming media segments are encoded in parallel through at least a target number of encoding servers. Among them, the overall encoding rate of at least the target number of encoding servers is not less than the frame rate of the live streaming media source data, that is, the overall encoding rate of multiple encoding servers is greater than or equal to the frame rate of the live streaming media source data, so that the encoding processing efficiency for the live streaming media source data can be matched with the frame rate, and the jamming caused by the encoding processing efficiency problem can be avoided, thereby ensuring the real-time nature of the live broadcast.

[0068] In this embodiment, by using at least a target number of encoding servers whose overall encoding rate is not less than the frame rate of the live streaming source data, the streaming segments obtained by slicing are encoded in parallel to obtain encoded video data of each encoding server. This can achieve matching of the encoding processing efficiency and frame rate of the live streaming source data, avoid freezes caused by encoding processing efficiency issues, and thus ensure the real-time nature of the live broadcast.

[0069] In one embodiment, the plurality of streaming media segments belong to a target encapsulation format; Figure 3 As shown, at least two encoding servers are used to encode the streaming media segments obtained by slicing in parallel to obtain the encoded video data of each encoding server, including:

[0070] Step 302: Decapsulate and decode the streaming media segments obtained by slicing and belonging to the target encapsulation format in parallel through at least two encoding servers to obtain streaming media data.

[0071] Among them, multiple streaming media segments belong to the target encapsulation format, that is, multiple streaming media segments are respectively encapsulated into the target encapsulation format. Decapsulation refers to removing the encapsulation format to obtain video encoding data; decoding refers to decoding the video encoding data to restore it into video data. Specifically, at least two encoding servers are used to perform parallel decapsulation and decoding on the streaming media segments obtained by slicing and belonging to the target encapsulation format, and each encoding server decapsulates and decodes the streaming media segments assigned to it to obtain streaming media data. The streaming media data is the slice data that has not been encoded and encapsulated.

[0072] Step 304: each encoding server encodes its own streaming media data in parallel to obtain its own encoding segment.

[0073] Specifically, each encoding server encodes its own streaming media data in parallel, specifically based on a preset encoding algorithm, such as H.265 encoding algorithm, to obtain encoding segments of each encoding server. The encoding segments are encoding result segments obtained after video encoding of streaming media data.

[0074] Step 306: Each encoding server encapsulates its own encoding segments in parallel according to the target encapsulation format to obtain the encoded video data of each encoding server.

[0075] Specifically, each encoding server encapsulates the encoded segments obtained in parallel according to the target encapsulation format, that is, encapsulates the encoded segments again according to the target encapsulation format to obtain the encoded video data of each encoding server, and the encoded video data and the streaming media segments all belong to the target encapsulation format.

[0076] In this embodiment, after the streaming media segments belonging to the target encapsulation format are decapsulated and decoded, they are encoded by the encoding server, and the encoded segments obtained by encoding are re-encapsulated into the target encapsulation format to obtain encoded video data belonging to the target encapsulation format. By encapsulating the encoded video data into the target encapsulation format of the streaming media segments, it is convenient to manage each encoded video data, which is beneficial to improving the processing efficiency of live streaming media data.

[0077] In one embodiment, the live streaming media data processing method also includes: obtaining a preset slicing granularity configured for processing live streaming media source data; determining a live processing delay according to the preset slicing granularity, the frame rate of the live streaming media source data, and the independent encoding rate of the encoding server; the live processing delay is positively correlated with the preset slicing granularity and the frame rate of the live streaming media source data, respectively, and the live processing delay is negatively correlated with the independent encoding rate of the encoding server; determining a target cache duration based on the live processing delay; the target cache duration is not less than the live processing delay.

[0078] Among them, the preset slice granularity can be pre-configured before processing the live streaming source data, such as 1 second or 0.5 second or other slice granularity. Live processing delay refers to the live delay caused by processing the live streaming data. For example, the frame rate of the live streaming source data is 60FPS, and the independent encoding rate of the encoding server is 5FPS (Frame Per Second, refresh rate). If the preset slice granularity is 1 second, the live processing delay can be 60*1 / 5=12 seconds. FPS is a definition in the field of images, which refers to the number of frames transmitted per second. Generally speaking, it refers to the number of frames of animation or video. FPS is a measure of the amount of information used to save and display dynamic videos. The more frames per second, the smoother the displayed action will be. The live processing delay is positively correlated with the preset slice granularity and the frame rate of the live streaming source data, and the live processing delay is negatively correlated with the independent encoding rate of the encoding server. That is, the larger the preset slice granularity or the frame rate of the live streaming source data, the larger the live processing delay; the larger the independent encoding rate of the encoding server, the smaller the live processing delay.

[0079] Specifically, the server can obtain the preset slice granularity configured for processing the live streaming source data, which can be flexibly configured according to actual needs. The larger the preset slice granularity, the longer the duration of the streaming media segment obtained by slicing. The server determines the frame rate of the live streaming source data and the independent encoding rate of the encoding server, and calculates the live processing delay according to the preset slice granularity, frame rate and independent encoding rate. In a specific implementation, the live processing delay can be calculated by the formula live processing delay = preset slice granularity * frame rate / independent encoding rate. The live processing delay is positively correlated with the preset slice granularity and the frame rate of the live streaming source data, respectively, and the live processing delay is negatively correlated with the independent encoding rate of the encoding server. That is, in order to reduce the live processing delay, when the frame rate of the live streaming source data is constant, it can be achieved by increasing the independent encoding rate of the encoding server or reducing the preset slice granularity. The server determines the target cache duration based on the live processing delay, so that the target cache duration is not less than the live processing delay, thereby ensuring that the cached live streaming source data can cover the live processing delay and prevent jamming during live video playback. For example, when the live broadcast processing delay is 12 seconds, the target cache duration set by the server can be greater than or equal to 12 seconds.

[0080] In this embodiment, the server determines the live processing delay according to the preset slice granularity, the frame rate of the live streaming source data and the independent encoding rate of the encoding server, and determines the target cache time based on the live processing delay, so that the target cache time is not less than the live processing delay, thereby ensuring that the cached live streaming source data can cover the live processing delay, preventing freezes in the live video playback, and ensuring the real-time nature of the live broadcast.

[0081] In one embodiment, the encoded video data of each encoding server are merged to obtain live streaming media encoded data, including: merging the encoded video data of each encoding server in timestamp order to obtain merged encoded data; converting the encapsulation format of the merged encoded data to obtain live streaming media encoded data that conforms to the live transmission encapsulation format.

[0082] Among them, the timestamp order refers to the order of the generation time of the live streaming media encoded data corresponding to the encoded video data. The merged encoded data is the merged data obtained by merging the encoded video data. The live transmission encapsulation format refers to the encapsulation format used for live streaming media, such as HLS (HTTP Live Streaming, a streaming media transmission protocol based on HTTP), DASH (Dynamic Adaptive Streaming over HTTP, dynamic adaptive streaming based on HTTP) and other encapsulation formats. HLS is a dynamic bitrate adaptive technology, including an m3u8 index file, TS media segment file and key encryption string file. DASH is a dynamic adaptive stream based on HTTP, similar to the HLS protocol. It uses an adaptive bitrate streaming technology to enable high-quality streaming media to be transmitted over the HTTP protocol.

[0083] Specifically, the server can determine the timestamp sequence of the encoded video data of each encoding server, and merge the encoded video data in a chronological order according to the timestamp sequence to obtain merged encoded data. The server converts the merged encoded data into an encapsulation format to obtain live streaming media encoded data that conforms to the live transmission encapsulation format. The live streaming media encoded data conforms to the live transmission encapsulation format, which is conducive to efficiently sending the live streaming media encoded data to the live client for playback.

[0084] Furthermore, the live streaming media data processing method further includes: sending the live streaming media encoded data to the live client via the content distribution network, so as to play it on the live client.

[0085] Among them, the Content Delivery Network (CDN) is a new type of network content service system, which is built on the IP (Internet Protocol) network and provides content distribution and services based on the efficiency requirements, quality requirements and content order of content access and application. The content delivery network represents a network application service model with high quality, high efficiency and distinct network order built on the network. Specifically, the server sends the live streaming media encoded data to the live client through the content delivery network, so that the live streaming media encoded data can be played in real time on the live client.

[0086] In this embodiment, the server will merge the encoded data and convert the encapsulation format to obtain live streaming media encoded data that conforms to the live transmission encapsulation format, and send the live streaming media encoded data to the live client for playback through the content distribution network, thereby ensuring the real-time performance of the live streaming media playback.

[0087] In one embodiment, the live streaming media data processing method further includes: determining the maximum overall encoding rate of at least two encoding servers; when it is determined that the maximum overall encoding rate does not match the frame rate of the live streaming media source data, adjusting the server quantity of the at least two encoding servers so that the maximum overall encoding rate of the adjusted encoding servers matches the frame rate of the live streaming media source data.

[0088] The maximum overall encoding rate refers to the maximum encoding rate that can be achieved by at least two encoding servers when performing parallel encoding processing. The maximum overall encoding rate is related to the computing performance of each encoding server itself and the resource occupancy state of the encoding server. That is, the maximum overall encoding rate can be the real-time overall encoding rate of at least two encoding servers, and the encoding servers can correspond to different maximum overall encoding rates when they are in different resource occupancy states.

[0089] Specifically, the server can determine the maximum overall encoding rate of at least two encoding servers, which can be determined based on the independent encoding rates of each encoding server, and can also be determined based on the resource occupancy status of each encoding server. The server compares the maximum overall encoding rate with the frame rate of the live streaming media source data. If it is determined that the maximum overall encoding rate does not match the frame rate of the live streaming media source data, such as when it is determined that the maximum overall encoding rate is less than the frame rate of the live streaming media source data, the server can adjust the number of servers of the encoding servers, such as increasing the number of servers of the encoding servers, so that the maximum overall encoding rate of the adjusted encoding servers matches the frame rate of the live streaming media source data, thereby enabling real-time processing of the live streaming media source data. In a specific application, if the server determines that the maximum overall encoding rate is less than the frame rate of the live streaming media source data, the server can increase the number of encoding servers to increase the maximum overall encoding rate and ensure the real-time performance of the live broadcast; if the maximum overall encoding rate is greater than the frame rate of the live streaming media source data, the server can reduce the number of encoding servers to reduce the maximum overall encoding rate to save operating resources.

[0090] In this embodiment, when the maximum overall encoding rate of at least two encoding servers does not match the frame rate of the live streaming source data, the server can adjust the number of encoding servers to make the maximum overall encoding rate match the frame rate of the live streaming source data, thereby ensuring the real-time nature of the live broadcast and saving operating resources.

[0091] In one embodiment, at least two encoding servers respectively encode the streaming media segments obtained by slicing in parallel through encoding steps. The encoding steps include: dividing each video frame in the streaming media segment to obtain a respective encoding unit of each video frame; predicting each encoding unit of each video frame and obtaining residual data of each encoding unit based on the obtained prediction value; encoding each residual data of each video frame to obtain encoded video data of the streaming media segment.

[0092] Wherein, when the streaming media segments obtained by slicing are encoded in parallel by at least two encoding servers, each encoding server encodes the respective streaming media segments through the encoding steps. The encoding unit is a basic unit for predictive encoding. The residual data can be obtained according to the difference between the encoding unit and its corresponding prediction value.

[0093] Specifically, the encoding server can divide each video frame in the streaming media segment respectively to obtain the respective encoding units of each video frame. Each video frame in the streaming media segment can be divided into multiple encoding units. The encoding server predicts the respective encoding units of each video frame, and specifically can predict based on the prediction unit in the encoding unit to obtain the prediction value corresponding to the encoding unit. The encoding server obtains the residual data of each encoding unit based on the obtained prediction value, and specifically can determine the difference between the encoding unit and its corresponding prediction value by the encoding server, and obtain the residual data based on the difference. The encoding server can encode the respective residual data of each video frame, so as to obtain the encoded video data of the streaming media segment.

[0094] In this embodiment, the encoding server divides each video frame in the streaming media segment into encoding units, and determines the residual data of the encoding unit based on the predicted value obtained by predicting the encoding unit, encodes the residual data, and obtains the encoded video data of the streaming media segment, so that encoding can be performed based on the difference between the video frames to achieve compression processing of the video frames, thereby reducing the data volume of the encoded video data.

[0095] In one embodiment, the coding unit includes a prediction unit; predicting the coding units of each video frame, and obtaining residual data of each coding unit based on the obtained prediction value, including: predicting the prediction units of each video frame according to a target prediction mode to obtain prediction data; determining the target prediction mode from intra-frame prediction and inter-frame prediction; determining the original residual of each coding unit based on the prediction data obtained by the prediction unit and the prediction unit; performing data transformation on the original residual of each coding unit to obtain residual data of each coding unit.

[0096] Among them, the coding unit includes a prediction unit, which specifies all prediction modes of the coding unit. The information related to the prediction is defined in the prediction unit, such as the direction of intra-frame prediction, the segmentation method of inter-frame prediction, motion vector prediction, and the reference image index number of inter-frame prediction, all of which belong to the category of the prediction unit. The target prediction mode refers to the prediction method for predicting the prediction unit, and the target prediction mode is determined from the intra-frame prediction and inter-frame prediction. Intra-frame prediction refers to a prediction mode that uses the spatial correlation of the image to predict the current coding block using the surrounding reconstructed pixel values; inter-frame prediction refers to a prediction mode that uses the correlation between consecutive images to eliminate the temporal redundancy of video information through motion estimation and motion compensation coding methods, so as to use the previously encoded reconstructed frame as a reference frame for prediction. The prediction data is the prediction result obtained by predicting the prediction unit. The original residual is the original difference between the prediction data and the prediction unit. The residual data is obtained by performing data transformation on the original residual, such as DCT (Discrete Cosine Transform) transformation and quantization processing.

[0097] Specifically, the encoding server determines the prediction unit in the encoding unit, and predicts the prediction unit of each video frame according to the target prediction mode to obtain prediction data. Specifically, the prediction unit can be predicted according to intra-frame prediction or inter-frame prediction to obtain prediction data. The encoding server can determine the original residual of each encoding unit based on the prediction data and the prediction unit. For example, the encoding server can determine the difference between the prediction data and the prediction unit, and obtain the original residual of the encoding unit based on the difference. The encoding server performs data transformation on the original residual of each encoding unit, such as DCT transformation and quantization processing, to obtain the residual data of each encoding unit, and can perform encoding based on the residual data to obtain the encoded video data of the streaming media segment, thereby realizing the encoding processing of the streaming media segment.

[0098] In this embodiment, the encoding server performs prediction according to intra-frame prediction or inter-frame prediction through the prediction unit in the encoding unit, and determines the original residual of each encoding unit based on the obtained prediction data and the prediction unit. The encoding server performs data transformation on the original residual of each encoding unit to obtain the residual data of each encoding unit, so that encoding can be performed based on the residual data of each encoding unit to achieve compression processing of the video frame, thereby reducing the data amount of the encoded video data.

[0099] The present application also provides an application scenario, and the application scenario applies the live streaming media data processing method. Specifically, the application of the live streaming media data processing method in the application scenario is as follows:

[0100] The anchor user broadcasts live on the Internet through a terminal, and the terminal sends the collected live streaming media source data to the live streaming server. The live streaming server caches the live streaming media source data, and in response to the streaming media duration of the cached live streaming media source data reaching the target cache duration, the live streaming server slices the cached live streaming media source data according to the preset slice granularity to obtain multiple streaming media segments that can be independently encoded, wherein the target cache duration is positively correlated with the preset slice granularity. The live streaming server sends multiple streaming media segments to the encoding server cluster, so that at least two encoding servers in the encoding server cluster perform parallel encoding on the streaming media segments obtained by slicing, and obtain the respective encoded video data of each encoding server. Among them, the overall encoding rate of at least two encoding servers is not less than the frame rate of the live streaming media source data. Each encoding server in the encoding server cluster sends the encoded video data obtained by each encoding to the live streaming server. The live streaming server merges the respective encoded video data of each encoding server to obtain the live streaming media encoded data, and sends the live streaming media encoded data to the live streaming client to play on the live streaming client.

[0101] The present application also provides an application scenario, and the application scenario applies the live streaming media data processing method. Specifically, the application of the live streaming media data processing method in the application scenario is as follows:

[0102] In this embodiment, the live streaming media data includes audio and video data during the live broadcast process, among which video is the main carrier for information presentation and dissemination. It is predicted that in the future, more than 80% of personal consumer network traffic and more than 70% of industry application traffic will be video data. At present, video technology is undergoing an evolution from high definition (1080P) to ultra high definition (UHD, Ultra High Definition). The picture resolutions of 4K ultra high definition and 8K ultra high definition are 4 times and 16 times that of high definition, respectively. The ultra high definition video industry is an economic activity related to ultra high definition video acquisition, production, transmission and presentation. The development of the ultra high definition video industry will effectively drive the replacement and upgrading of terminal products and the updating of production and broadcasting equipment, drive the development of upstream chip, panel and other industries, and promote broadband network investment and construction, 5G (5th-Generation Mobile Communication Technology, fifth generation mobile communication technology) network commercialization and business development. According to the standards of ITU (International Telecommunication Union), ultra-high-definition video is divided into two specifications: 4K and 8K, supporting a total of 9 frame rates: 23.976fps, 24fps, 25fps, 29.97fps, 30fps, 50fps, 59.94fps, 60fps and 120fps. Real-time encoding of ultra-high-definition video (4K / 8K / 16K) has always been a pain point in the industry. The resolution pixels of 4K and 8K ultra-high-definition videos are 4 times and 16 times that of 1080P, respectively. In addition, 4K / 8K FPS is basically 60FPS+, which brings about an increase in encoding computing power of dozens of times that of 1080P. At present, most of the video industry is based on 1080P, and a small number of long videos support 4K video sources, but they are all offline long video encoding. In the field of real-time live broadcasting, there are very few 4K / 8K live broadcasts, and occasionally there are few that are achieved through encoding with special encoding equipment. The price of professional 8K encoding equipment for 8K channels is high. The computing power requirements for real-time encoding of ultra-high-definition video and the high cost of professional equipment have hindered the development of the ultra-high-definition live broadcast industry.

[0103] Based on this, the live streaming media data processing method provided in this embodiment can use a general cloud server (Cloud Virtual Machine, CVM) distributed architecture to perform real-time high FPS (>=60) encoding of ultra-high-definition 4K / 8K / 16K videos, meet low-cost real-time ultra-high-definition video live broadcast and high-quality encoding, which is conducive to promoting industry development and has good application prospects in CCTV (Closed Circuit Television) and OTT (Over The Top) industries.

[0104] Specifically, regarding the ITU's definition of ultra-high-definition video, the definition of ultra-high-definition video comes from the video resolution of 4K and above, but simply improving the resolution cannot well reflect the improvement in visual effects. It is necessary to simultaneously break through the six dimensions of technology: high resolution, high frame rate, high color depth, wide color gamut, high dynamic range and three-dimensional sound to bring a subversive, more appealing and immersive on-site experience. Figure 4 As shown in the figure, in the video resolution comparison, the pixels of high definition (HD) are 1280*720; the pixels of full high definition (Full HD) are 1920*1080; the pixels of quad high definition (Quad HD) are 256*1440; the pixels of 4K ultra high definition (UHD) are 3840*2160; the pixels of 5K ultra high definition+ are 5120*2880; the pixels of 8K ultra high definition are 7680*4320. According to ITU standards, ultra-high-definition video is divided into two specifications: 4K and 8K. Among them, 4K refers to 4K ultra-high definition (UHD4K), with a horizontal resolution of 3840, a vertical resolution of 2160, an aspect ratio of 16:9, and a total number of pixels of about 8.3 million; 8K refers to 8K ultra-high definition (UHD8K), with a horizontal resolution of 7680, a vertical resolution of 4320, an aspect ratio of 16:9, and a total number of pixels of about 33.2 million. In addition, there are also some application explorations for 16K (15360x8640, 130 million pixels) ultra-high definition in some metaverse VR (Virtual Reality) scenes. The latest video coding standards AV2 and VVC can also support 16K coding. In the ITU specification, ultra-high-definition video supports 9 frame rates: 23.976fps, 24fps, 25fps, 29.97fps, 30fps, 50fps, 59.94fps, 60fps and 120fps. With the huge increase in network bandwidth and encoding computing power, the ITU standard removed the interlaced field encoding method and only retained the progressive frame encoding method. In terms of color depth, the ITU increased the original 8bit to 10bit and 12bit, of which 10bit is used for 4K ultra-high definition and 12bit is used for 8K ultra-high definition. The above improvements further improve the smoothness, delicacy and color level of ultra-high-definition video.

[0105] Furthermore, if Figure 5As shown in the figure, in the audio and video processing flow, after the audio and video to be processed are input, the audio and video are decapsulated (libavformat demuxing), and the decapsulated encoded data is decoded (libavcodec decoding). The decoded audio and video data can be encoded (libavcodecencoding), and the encoding and decoding process supports various formats, such as H.264 / H.265 / AV1 / H.266, etc. After the encoding is completed, it is encapsulated (libavcodec muxing) and output, which can be encapsulated into various encapsulation formats such as Mp4 / MOV / T5 format. In addition, the decoded audio and video data can also be encoded with enhancement (libavfilter apply effects), and the audio and video data after the encoding enhancement process are encoded and encapsulated in turn to obtain the output audio and video encoded data. In the coding enhancement process, it can specifically include super-resolution / interpolation, SDR (Standard Dynamic Range, standard dynamic range image) / HDR (High-Dynamic Range, high dynamic range image), text watermark, denoising / deburring, de-jittering, color enhancement, subtitles, splicing, sprites and other processing. The computing power of audio and video processing is mainly in the encoding stage. The resolution pixels of 4K and 8K ultra-high-definition videos are 4 times and 16 times that of 1080P respectively, and the 4K / 8K FPS is basically a high refresh rate of 60FPS+, so the encoding computing power brought is dozens of times that of 1080P. H.265 / H.266 / AV1 supports 4K / 8K encoding, and the latest generation of video encoding standards H.266 / AV2 already supports 16K encoding. Ultra-high-definition 4K / 8k video real-time encoding has always been a pain point in the industry.

[0106] Furthermore, for video encoding processing, taking H.265 as an example, a frame of image is sent to the encoder, and first divided into coding tree units (Coding Tree Uint, CTU) according to the 64x64 block size, and then obtained by depth division. Coding units (Coding Uint, CU), each CU contains prediction units (Predict Unit, PU) and transform units (TransformUnit, TU). Each PU is predicted to obtain a prediction value, and the prediction value is subtracted from the input image data to obtain a residual, and then DCT transformation and quantization are performed to obtain residual coefficients, and the residual coefficients are sent to the entropy coding module to output the bitstream. At the same time, after the residual coefficient is inversely quantized and inversely transformed, the residual value of the reconstructed image is obtained, and then added to the prediction value to obtain a reconstructed image. After the reconstructed image is filtered in the loop, it enters the reference frame queue as the reference image of the next frame, and then is encoded backward in sequence. When predicting, starting from the largest coding unit (LCU, Largest Code Unit), each layer is divided downward layer by layer according to the quadtree, and recursive calculation is performed. First, divide from top to bottom. From depth = 0, the 64x64 block is first divided into 4 32x32 sub-CUs. Then one of the 32x32 sub-CUs is further divided into 4 16x16 sub-CUs, and so on, until depth = 3, the CU size is 8x8. Then, pruning is performed from bottom to top. The RDcost of the four 8x8 CUs is summed (denoted as cost1), and compared with the RDcost of the corresponding 16x16 CU of the previous level (denoted as cost2). If cost1 is less than cost2, the 8x8 CU division is retained, otherwise continue to prune upwards and compare layer by layer. Finally, find the optimal CU depth division. PU prediction is divided into intra-frame prediction and frame-level prediction. First, within the same prediction type, different PUs are compared to find the optimal segmentation mode, and then compared with the intra-frame and inter-frame modes to find the optimal prediction mode under the current CU; at the same time, the CU is adaptively transformed based on the quadtree structure (Residual Quad-tree Transform, RQT) to find the optimal TU mode. Finally, a frame of image is divided into CUs, and the PUs and TUs corresponding to the CUs, so as to perform prediction processing through the PUs. In addition, a frame of video image can also be encoded into one or more slices, each slice containing an integer number of macroblocks, that is, each slice has at least one macroblock, and at most contains the macroblocks of the entire image. The purpose of slices is to limit the spread and transmission of bit errors and keep the coded slices independent of each other. Macroblocks are the basic unit of coding. A coded image must first be divided into multiple blocks before it can be processed. Macroblocks are composed of an integer number of blocks.

[0107] like Figure 6As shown, when the current frame Fn is input for encoding, a prediction is performed based on the current frame Fn to obtain a prediction value, and the prediction value is subtracted from the current frame Fn to obtain a residual, and the residual is transformed and quantized in turn to obtain a residual coefficient, and the residual system is entropy coded to obtain the encoding result of the current frame Fn. In addition, the residual coefficient is inversely quantized and inversely transformed to obtain a residual value of the reconstructed image, and the residual value of the reconstructed image is added to the prediction value of the current frame Fn to obtain a reconstructed image. The reconstructed image is processed by deblocking (DB) and sample adaptive offset (SAO) to obtain a reference frame F'n, which is used as a reference image for the next frame. Further, in the specific prediction, when inter-frame prediction is performed for the current frame Fn, motion estimation (ME) and motion compensation (MC) are performed based on the reference frame F'n-1 to obtain an inter-frame prediction result. When performing intra-frame prediction processing, intra-frame prediction selection is performed based on the current frame Fn to implement intra-frame prediction and obtain an intra-frame prediction result. The prediction value of the current frame Fn is obtained by comparing the inter-frame prediction result with the inter-frame prediction result.

[0108] Furthermore, PU has a total of 8 segmentation modes, while TU has only 2 segmentation modes: segmentation or non-division. H.265 still uses the widely accepted hybrid coding framework (intra-frame coding, inter-frame coding, transform, quantization, and coding) that has been adopted since H.263, such as intra-frame prediction and inter-frame prediction based on motion compensation, as well as transform coding and high-performance entropy coding. Compared with previous coding standards, H.265 adopts many innovative technologies, such as flexible segmentation structure and residual coding structure based on large-size quadtree, adaptive motion parameter coding (Adaptive Motion Parameter), adaptive in-loop filtering (Adaptive In-loop Filtering), etc. These new technologies improve the coding efficiency of the H.265 standard. CU is the most basic square coding unit, and the size of CU can range from 8×8 to the largest coding unit 64×64; PU is the basic unit for intra-frame prediction and inter-frame prediction, and the size can range from 4×4 to 64×64. In addition to the symmetric mode (Symmetric Motion Partition, SMP) 2N×2N, N×N, 2N×N and N×2N similar to H.264's partitioning method, H.265 also provides an asymmetric mode (Asymmetricmotion Partition, AMP), including 2N×nU, 2N×nD, nL×2N and nR×2N, and the capital letters represent the position of the partition block with shorter side length. Transform unit TU is the basic unit for transformation and quantization process. The shape of the transform unit depends on the partitioning mode of the prediction unit. When the prediction unit is square, the transform unit is also square, and the size is 4×4 to 32×32; when the prediction unit is not square, the transform unit will not be square, and the size is 32×8, 8×32, 16×4, 4×16. Each coding unit contains one or more transform units, but the partitioning of the transform unit must be within the quadtree structure of the coding unit. Figure 7 As shown in the figure, the coding tree unit CTU is divided into various coding units CU through depth, and the coding unit CU includes a prediction unit PU and a transform unit TU. Among them, the partition modes of the prediction unit PU include 2N×2N, N×N, 2N×N, N×2N, 2N×nU, 2N×Nd, nL×2N and nR×2N, etc.; the partition modes of the transform unit TU include partition and non-partition.

[0109] Taking HM16.8 as an example, H.265 encodes 1080P sequences and encodes 0.03 frames per second under 2M bandwidth. If a 10-minute video with a frame rate of 30fps is encoded at this speed, it will take 10*30 / 0.03=10000 minutes, about 166.67 hours. This encoding speed cannot be applied to engineering products. The encoder needs to accelerate engineering performance on the one hand, such as multi-threading, assembly instruction acceleration, etc. On the other hand, it can also trim the video standard algorithm. For example, in the H.265 encoding process, some trimming and skipping are performed for different video scenes CU, PU, ​​TU division and intra-frame and inter-frame prediction processing. Finally, the video encoding algorithm performance parameter bd-rate will have a certain loss, but it can make some compromises between engineering performance and compression rate to implement ultra-high-definition video encoding processing on general servers.

[0110] Specifically, the live streaming media data processing method provided in this embodiment is as follows: Figure 8 As shown, the overall processing of live streaming data processing is realized by slicing, encoding, and merging distributed cluster scheduling. According to the cluster load scheduling, the corresponding module server is scheduled for processing. Specifically, the video source is a 4K / 8K ultra-high-definition video, which can specifically include videos in MP4 / MOV / TS / MKV formats. The slicing server slices the ultra-high-definition video source according to the GOP (Group Of Pictures) multiple granularity. The GOP length is the video frame between two IDRs. The SPS (Sequence Paramater Set) and PPS (Picture Paramater Set) information of these video frames are independent. Slicing according to the GOP multiple granularity can obtain a video that can be played normally. Otherwise, if it is not sliced ​​according to the GOP granularity multiple, the random video decoding will cause abnormal screen distortion and other problems. Among them, a set of global parameters of a coded video sequence (Codedvideo sequence) is stored in the SPS. The so-called coded video sequence is a sequence composed of the structure of the pixel data of each frame of the original video after encoding. The parameters on which the encoded data of each frame depends are stored in the image parameter set. Generally, the NAL Unit of SPS and PPS is usually located at the beginning of the entire bitstream. However, in some special cases, these two structures may appear in the middle of the bitstream. The main reason may be that the decoder needs to start decoding in the middle of the bitstream and the encoder changes the parameters of the bitstream during the encoding process. PPS saves some encoding-related parameters in the current video segment of NAL Uint (unit package of the layer), such as entropy coding type, number of slices, whether weighted prediction, quantization and filtering parameters, etc.

[0111] Among them, in video encoding algorithms such as H.264 / H.265 / H.266 / AV1, images are organized in sequences. The first image of a sequence is called an IDR image (immediate refresh image), and IDR images are all I-frame images. IDR will cause the DPB (Decoded Picture Buffer) to be cleared, but I-frames will not. DPB is the decoded picture buffer. When decoding a video, it is necessary to cache the latest reference frames. This buffer is called DPB. The decoded images are cached in the DPB for subsequent frames to use as reference images. Therefore, the decoder needs to manage the images in the DPB according to certain rules. IDR images must be I images, but I images are not necessarily IDR images. There can be many I images in a sequence, and the images after the I-frame images can reference the images between the I-frame images as motion references. There can be many I-frame images in a sequence, and the images after the I-frame images can reference the images between the I-frame images as motion references. P frame refers to forward prediction coding frame, which indicates the difference between this frame and the previous key frame (or P frame). When decoding, the difference defined by this frame needs to be superimposed with the previously cached picture to generate the final picture. B frame refers to bidirectional prediction interpolation coding frame, which is a bidirectional difference frame, that is, B frame records the difference between this frame and the previous and next frames. B frame can be used as a reference frame for other B frames, or it can not be used as a reference frame for other B frames. GOP is the interval between two I frames.

[0112] The slice server transcapsulates the slice into a video clip in TS format. The generated slice video clips are encoded in real time and in parallel through multiple slice encoding servers. Since the source is cut into TS video clips, multiple slice encoding servers need to encode in parallel. Assuming that the FPS of the source video is F, the length of the slice is 1 second, and the minimum encoding speed of the slice encoding server is X FPS, then the total live delay D>=F / X. That is, before starting slicing, a video cache of >=D length must be accumulated before encoding begins to ensure that the total FPS of the subsequent encoding cluster is >= the original FPS of the video, and no jamming occurs.

[0113] Furthermore, the merging server transcodes the TS video segments encoded by the parallel sliced ​​videos and outputs them in the HLS / DASH encapsulation format. The HLS / DASH live video content container transmission format encapsulation is TS fragmentation, which is convenient for each slice encoding server to encode and output in parallel. If it is a format encapsulated in a video stream container format such as RTMP / FLV / WebRTC, there will be some container format encoding compatibility issues when merging the merge single stream output. In addition, if the compatibility requirements are met, in addition to HLS / DASH, it can also be transcoded into various encapsulation formats such as RTMP / FLV / WebRTC. Finally, the content distribution network CDN (Content Delivery Network) will live stream the transcoded HLS / DASH ultra-high-definition video format in real time for users to watch. Among them, the basic idea of ​​the content distribution network is to avoid bottlenecks and links on the Internet that may affect the speed and stability of data transmission as much as possible, so as to make content transmission faster and more stable. By placing node servers at various locations on the network to form a layer of intelligent virtual network on the existing Internet, the CDN system can redirect user requests to the service node closest to the user in real time based on comprehensive information such as network traffic and the connection and load status of each node, as well as the distance to the user and response time. Its purpose is to enable users to obtain the required content nearby, solve the problem of network congestion, and improve the response speed of users visiting websites. In a specific application, such as Fig. 9 As shown in the figure, when 8K video is encoded and broadcast live, it can stably support 60FPS 8K video real-time encoding processing.

[0114] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a live streaming media data processing system for implementing the live streaming media data processing method involved above. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more live streaming media data processing system embodiments provided below can refer to the limitations of the live streaming media data processing method above, and will not be repeated here.

[0116] In one embodiment, Fig.10 As shown, a live streaming media processing system 1000 is provided, which includes: a slicing server 1002, a merging server 1006, and an encoding server cluster 1004 including at least two encoding servers; wherein:

[0117] The slice server 1002 is used to cache the live streaming media data, and in response to the streaming media duration of the cached live streaming media data reaching the target cache duration, slice the cached live streaming media data according to the preset slice granularity to obtain a plurality of streaming media segments that can be independently encoded; the target cache duration is positively correlated with the preset slice granularity and positively correlated with the number of encoding servers in the encoding server cluster;

[0118] The encoding server cluster 1004 is used to perform parallel encoding of the streaming media segments obtained by slicing through at least two encoding servers to obtain encoded video data of each encoding server;

[0119] The merging server 1006 is used to merge the encoded video data of each encoding server to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

[0120] In one embodiment, the preset slicing granularity includes an integer multiple granularity of a preset picture group; the slicing server 1002 is also used to count the streaming duration of the cached live streaming source data; when the streaming duration reaches the preset target cache duration, the cached live streaming source data is sliced ​​according to the integer multiple granularity of the preset picture group to obtain multiple streaming segments that can be independently encoded.

[0121] In one embodiment, the streaming media segments are a target number, the target number is positively correlated with the frame rate of the live streaming media source data, and the target number is negatively correlated with the independent encoding rate of the encoding server; the encoding server cluster 1004 is also used to parallelly encode the streaming media segments obtained by slicing through at least the target number of encoding servers to obtain the encoded video data of each encoding server; wherein, the overall encoding rate of at least the target number of encoding servers is not less than the frame rate of the live streaming media source data.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a live streaming media data processing device for implementing the live streaming media data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more live streaming media data processing device embodiments provided below can refer to the limitations of the live streaming media data processing method above, and will not be repeated here.

[0123] In one embodiment, Fig.11 As shown, a live streaming media data processing device 1100 is provided, comprising: a live streaming slicing module 1102, a slice parallel encoding module 1104 and an encoded data merging module 1106, wherein:

[0124] The live streaming slicing module 1102 is used to cache live streaming media source data, and in response to the streaming media duration of the cached live streaming media source data reaching the target caching duration, slice the cached live streaming media source data according to a preset slicing granularity to obtain a plurality of streaming media segments that can be independently encoded; the target caching duration is positively correlated with the preset slicing granularity;

[0125] The slice parallel encoding module 1104 is used to perform parallel encoding on the streaming media segments obtained by slicing through at least two encoding servers to obtain respective encoded video data of each encoding server; the overall encoding rate of the at least two encoding servers is not less than the frame rate of the live streaming media source data;

[0126] The coded data merging module 1106 is used to merge the coded video data of each coding server to obtain live streaming media coded data; the live streaming media coded data is used to send to the live client.

[0127] In one embodiment, the preset slicing granularity includes an integer multiple granularity of a preset picture group; the live stream slicing module 1102 includes a cache statistics module and a cache slicing module; wherein: the cache statistics module is used to count the streaming duration of the cached live streaming media source data; the cache slicing module is used to slice the cached live streaming media source data according to the integer multiple granularity of the preset picture group when the streaming duration reaches the preset target cache duration, so as to obtain multiple streaming media segments that can be independently encoded.

[0128] In one embodiment, the cache slicing module includes an equal-granularity slicing module and a slice encapsulation module; wherein: the equal-granularity slicing module is used to slice the cached live streaming source data at an equal-granularity according to an integer multiple of a preset picture group, to obtain a plurality of live streaming slice data of the same duration and independently encodable; the slice encapsulation module is used to encapsulate a plurality of live streaming slice data respectively according to a target encapsulation format, to obtain a plurality of streaming media segments belonging to the target encapsulation format.

[0129] In one embodiment, the streaming media segments are a target number, the target number is positively correlated with the frame rate of the live streaming media source data, and the target number is negatively correlated with the independent encoding rate of the encoding server; the slice parallel encoding module 1104 is also used to parallelly encode the streaming media segments obtained by slicing through at least the target number of encoding servers to obtain the encoded video data of each encoding server; wherein, the overall encoding rate of at least the target number of encoding servers is not less than the frame rate of the live streaming media source data.

[0130] In one embodiment, multiple streaming media segments belong to a target encapsulation format; the slice parallel encoding module 1104 includes a data restoration module, an encoding processing module and an encoding encapsulation module; wherein: the data restoration module is used to perform parallel decapsulation and decoding of the streaming media segments obtained by slicing and belonging to the target encapsulation format through at least two encoding servers to obtain streaming media data; the encoding processing module is used to perform parallel encoding of the respective streaming media data through each encoding server to obtain the respective encoded segments of each encoding server; the encoding encapsulation module is used to perform parallel encapsulation of the respective encoded segments according to the target encapsulation format through each encoding server to obtain the respective encoded video data of each encoding server.

[0131] In one embodiment, it also includes a slice granularity acquisition module, a live broadcast delay determination module and a cache duration determination module; wherein: the slice granularity acquisition module is used to obtain the preset slice granularity configured for processing the live streaming media source data; the live broadcast delay determination module is used to determine the live broadcast processing delay according to the preset slice granularity, the frame rate of the live streaming media source data and the independent encoding rate of the encoding server; the live broadcast processing delay is positively correlated with the preset slice granularity and the frame rate of the live streaming media source data, respectively, and the live broadcast processing delay is negatively correlated with the independent encoding rate of the encoding server; the cache duration determination module is used to determine the target cache duration based on the live broadcast processing delay; the target cache duration is not less than the live broadcast processing delay.

[0132] In one embodiment, the encoding data merging module 1106 is also used to merge the encoded video data of each encoding server in the order of timestamps to obtain merged encoded data; convert the encapsulation format of the merged encoded data to obtain live streaming media encoded data that conforms to the live transmission encapsulation format; and also includes a data sending module, which is used to send the live streaming media encoded data to the live client through the content distribution network for playback on the live client.

[0133] In one embodiment, it also includes a server quantity adjustment module, which is used to determine the maximum overall encoding rate of at least two encoding servers; when it is determined that the maximum overall encoding rate does not match the frame rate of the live streaming media source data, the server quantity of the at least two encoding servers is adjusted so that the maximum overall encoding rate of the adjusted encoding servers matches the frame rate of the live streaming media source data.

[0134] In one embodiment, it also includes a coding unit acquisition module, a prediction module and a residual coding module; wherein: the coding unit acquisition module is used to divide each video frame in the streaming media segment respectively to obtain the respective coding units of each video frame; the prediction module is used to predict the respective coding units of each video frame, and obtain the residual data of each coding unit based on the obtained prediction value; the residual coding module is used to encode the respective residual data of each video frame to obtain the encoded video data of the streaming media segment.

[0135] In one embodiment, the coding unit includes a prediction unit; the prediction module is also used to predict the prediction unit of each video frame according to a target prediction mode to obtain prediction data; the target prediction mode is determined from intra-frame prediction and inter-frame prediction; based on the prediction data obtained by the prediction unit and the prediction unit, the original residual of each coding unit is determined; the original residual of each coding unit is data transformed to obtain the residual data of each coding unit.

[0136] Each module in the above-mentioned live streaming media data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0137] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.12As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store live streaming media data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a live streaming media data processing method is implemented.

[0138] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0139] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0140] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0141] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing live streaming media data, characterized in that: The method comprises: Cache live streaming media source data, and obtain a preset slice granularity configured for processing the live streaming media source data; Determine a live processing delay according to the preset slice granularity, the frame rate of the live streaming media source data, and the independent encoding rate of the encoding server; the live processing delay is positively correlated with the preset slice granularity and the frame rate of the live streaming media source data, respectively, and the live processing delay is negatively correlated with the independent encoding rate of the encoding server; Determine a target cache duration based on the live broadcast processing delay; the target cache duration is not less than the live broadcast processing delay; In response to the streaming duration of the cached live streaming source data reaching the target cache duration, slicing the cached live streaming source data according to the preset slicing granularity to obtain a plurality of independently encodable streaming segments; the target cache duration is positively correlated with the preset slicing granularity; The streaming media segments obtained by slicing are encoded in parallel by at least two encoding servers to obtain encoded video data of each encoding server; the overall encoding rate of at least two encoding servers is not less than the frame rate of the live streaming media source data; The encoded video data of each encoding server are merged to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

2. The method according to claim 1, characterized in that The preset slicing granularity includes an integer multiple granularity of a preset picture group; in response to the streaming media duration of the cached live streaming media source data reaching the target cache duration, slicing the cached live streaming media source data according to the preset slicing granularity to obtain a plurality of independently encodable streaming media segments, including: Count the streaming duration of the cached live streaming source data; When the duration of the streaming media reaches the target cache duration, the cached live streaming media source data is sliced ​​according to the granularity of integer multiples of the preset picture group to obtain a plurality of streaming media segments that can be independently encoded.

3. The method according to claim 2, characterized in that The step of slicing the cached live streaming media source data according to the integer multiple granularity of the preset picture group to obtain a plurality of streaming media segments that can be independently encoded includes: Slicing the cached live streaming media source data with equal granularity according to the integer multiple granularity of the preset picture group to obtain a plurality of live streaming media slice data with the same duration and capable of being encoded independently; According to the target encapsulation format, the plurality of live streaming media slice data are respectively encapsulated to obtain a plurality of streaming media segments belonging to the target encapsulation format.

4. The method according to claim 1, characterized in that: The streaming media segments are a target number, the target number is positively correlated with the frame rate of the live streaming media source data, and the target number is negatively correlated with the independent encoding rate of the encoding server; The step of encoding the streaming media segments obtained by slicing in parallel by at least two encoding servers to obtain the encoded video data of each encoding server includes: Parallel encoding of the streaming media segments obtained by slicing by at least the target number of encoding servers is performed to obtain respective encoded video data of each encoding server; Among them, the overall encoding rate of at least the target number of encoding servers is not less than the frame rate of the live streaming media source data.

5. The method according to claim 1, characterized in that: The plurality of streaming media segments are of a target encapsulation format; The step of encoding the streaming media segments obtained by slicing in parallel by at least two encoding servers to obtain the encoded video data of each encoding server includes: Decapsulating and decoding the streaming media segments obtained by slicing and belonging to the target encapsulation format in parallel by at least two encoding servers to obtain streaming media data; By means of the encoding servers, the respective streaming media data are encoded in parallel to obtain respective encoding segments of the encoding servers; Through each of the encoding servers, respective encoding segments are encapsulated in parallel according to the target encapsulation format to obtain respective encoded video data of each of the encoding servers.

6. The method according to claim 1, characterized in that The step of merging the encoded video data of each encoding server to obtain the live streaming media encoded data includes: Merging the encoded video data of each encoding server in the order of timestamps to obtain merged encoded data; Converting the combined coded data into a package format to obtain live streaming media coded data conforming to a live transmission package format; The method further comprises: The live streaming media encoded data is sent to the live client via a content distribution network so as to be played on the live client.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: determining a maximum overall encoding rate of at least two of the encoding servers; When it is determined that the maximum overall encoding rate does not match the frame rate of the live streaming source data, the server quantities of at least two of the encoding servers are adjusted so that the maximum overall encoding rate of the adjusted encoding servers matches the frame rate of the live streaming source data.

8. The method according to any one of claims 1 to 6, characterized in that: At least two of the encoding servers respectively perform parallel encoding on the streaming media segments obtained by slicing through encoding steps; the encoding steps include: Dividing each video frame in the streaming media segment respectively to obtain a coding unit for each video frame; Predicting the coding units of the respective video frames, and obtaining residual data of the coding units based on the obtained prediction values; The residual data of each of the video frames is encoded to obtain encoded video data of the streaming media segment.

9. The method according to claim 8, characterized in that The coding unit includes a prediction unit; the predicting of the coding units of the respective video frames and obtaining residual data of the coding units based on the obtained prediction values ​​includes: For each prediction unit of each video frame, prediction is performed according to a target prediction mode to obtain prediction data; the target prediction mode is determined from intra-frame prediction and inter-frame prediction; Determining an original residual of each of the coding units based on the prediction data obtained by the prediction unit and the prediction unit; Data transformation is performed on the original residual of each coding unit to obtain residual data of each coding unit.

10. A live streaming media processing system, characterized in that: The system comprises: a slicing server, a merging server and an encoding server cluster comprising at least two encoding servers; The slicing server is used to cache live streaming source data and obtain a preset slicing granularity configured for processing the live streaming source data; determine a live processing delay according to the preset slicing granularity, the frame rate of the live streaming source data and the independent encoding rate of the encoding server; the live processing delay is positively correlated with the preset slicing granularity and the frame rate of the live streaming source data, respectively, and the live processing delay is negatively correlated with the independent encoding rate of the encoding server; determine a target cache duration based on the live processing delay; the target cache duration is not less than the live processing delay; in response to the streaming duration of the cached live streaming data reaching the target cache duration, slice the cached live streaming data according to the preset slicing granularity to obtain a plurality of streaming segments that can be independently encoded; the target cache duration is positively correlated with the preset slicing granularity and is positively correlated with the number of encoding servers in the encoding server cluster; The encoding server cluster is used to perform parallel encoding of streaming media segments obtained by slicing through at least two of the encoding servers to obtain respective encoded video data of each encoding server; The merging server is used to merge the encoded video data of each encoding server to obtain live streaming media encoded data; the live streaming media encoded data is used to send to the live client.

11. The system according to claim 10, characterized in that The preset slice granularity includes an integer multiple granularity of a preset picture group; The slicing server is also used to count the streaming duration of the cached live streaming source data; when the streaming duration reaches the target cache duration, the cached live streaming source data is sliced ​​according to the integer multiple granularity of the preset picture group to obtain multiple streaming segments that can be independently encoded.

12. The system according to claim 10 or 11, characterized in that The streaming media segments are a target number, the target number is positively correlated with the frame rate of the live streaming media source data, and the target number is negatively correlated with the independent encoding rate of the encoding server; The encoding server cluster is also used to parallelly encode the streaming media segments obtained by slicing through at least the target number of encoding servers to obtain the encoded video data of each encoding server; wherein the overall encoding rate of at least the target number of encoding servers is not less than the frame rate of the live streaming media source data.

13. A live streaming media data processing device, characterized in that: The device comprises: Live stream slicing module, used to cache live streaming source data; A slice granularity acquisition module, used to acquire a preset slice granularity configured for processing the live streaming media source data; A live broadcast delay determination module, used to determine the live broadcast processing delay according to the preset slice granularity, the frame rate of the live broadcast streaming media source data and the independent encoding rate of the encoding server; the live broadcast processing delay is positively correlated with the preset slice granularity and the frame rate of the live broadcast streaming media source data, respectively, and the live broadcast processing delay is negatively correlated with the independent encoding rate of the encoding server; A cache duration determination module, used to determine a target cache duration based on the live broadcast processing delay; the target cache duration is not less than the live broadcast processing delay; The live streaming slicing module is further configured to slice the cached live streaming source data according to the preset slicing granularity to obtain a plurality of independently encodable streaming segments in response to the streaming duration of the cached live streaming source data reaching the target caching duration; the target caching duration is positively correlated with the preset slicing granularity; A slice parallel encoding module is used to encode the streaming media segments obtained by slicing in parallel through at least two of the encoding servers to obtain the encoded video data of each encoding server; the overall encoding rate of at least two of the encoding servers is not less than the frame rate of the live streaming media source data; The coding data merging module is used to merge the coded video data of each coding server to obtain live streaming media coded data; the live streaming media coded data is used to send to the live client.

14. The device according to claim 13, characterized in that The preset slice granularity includes an integer multiple granularity of a preset picture group; The live stream slicing module is also used to count the streaming duration of the cached live streaming media source data; when the streaming duration reaches the target cache duration, the cached live streaming media source data is sliced ​​according to the integer multiple granularity of the preset picture group to obtain multiple streaming media segments that can be independently encoded.

15. The device according to claim 14, characterized in that The live streaming slicing module is further used to slice the cached live streaming media source data into equal granularity according to the granularity of an integer multiple of the preset picture group, so as to obtain a plurality of live streaming media slice data of the same duration and independently encodable; According to the target encapsulation format, the plurality of live streaming media slice data are respectively encapsulated to obtain a plurality of streaming media segments belonging to the target encapsulation format.

16. The device according to claim 13, characterized in that The streaming media segments are a target number, the target number is positively correlated with the frame rate of the live streaming media source data, and the target number is negatively correlated with the independent encoding rate of the encoding server; The slice parallel encoding module is also used to parallel encode the streaming media segments obtained by slicing through at least the target number of encoding servers to obtain the encoded video data of each encoding server; wherein the overall encoding rate of at least the target number of encoding servers is not less than the frame rate of the live streaming media source data.

17. The device according to claim 13, characterized in that The plurality of streaming media segments are of a target encapsulation format; The slice parallel encoding module is also used to parallelly decapsulate and decode the streaming media segments obtained by slicing and belonging to the target encapsulation format through at least two of the encoding servers to obtain streaming media data; to parallelly encode the respective streaming media data through each of the encoding servers to obtain the respective encoded segments of each of the encoding servers; and to parallelly encapsulate the respective encoded segments according to the target encapsulation format through each of the encoding servers to obtain the respective encoded video data of each of the encoding servers.

18. The device according to claim 13, characterized in that The coded data merging module is further used to merge the coded video data of each coding server according to the order of timestamps to obtain merged coded data; Converting the combined coded data into a package format to obtain live streaming media coded data conforming to a live transmission package format; The device further includes: a data sending module, which is used to send the live streaming media encoded data to the live client via a content distribution network, so as to play it on the live client.

19. The device according to any one of claims 13 to 18, characterized in that The device also includes: The server quantity adjustment module is used to determine the maximum overall encoding rate of at least two of the encoding servers; when it is determined that the maximum overall encoding rate does not match the frame rate of the live streaming media source data, the server quantity of at least two of the encoding servers is adjusted so that the maximum overall encoding rate of the adjusted encoding servers matches the frame rate of the live streaming media source data.

20. The device according to any one of claims 13 to 18, characterized in that The device also includes: A coding unit obtaining module, used for dividing each video frame in the streaming media segment respectively to obtain a coding unit for each video frame; A prediction module, configured to predict the coding units of the respective video frames, and obtain residual data of the respective coding units based on the obtained prediction values; The residual encoding module is used to encode the residual data of each video frame to obtain the encoded video data of the streaming media segment.

21. The device according to claim 20, characterized in that The encoding unit includes a prediction unit; The prediction module is also used to predict the prediction units of each of the video frames according to the target prediction mode to obtain prediction data; the target prediction mode is determined from intra-frame prediction and inter-frame prediction; based on the prediction data obtained by the prediction unit and the prediction unit, the original residual of each of the coding units is determined; the original residual of each of the coding units is data transformed to obtain the residual data of each of the coding units.

22. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

24. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images