Live stream data processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a live broadcast stream data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: when live broadcast stream reproduction processing for a target live broadcast stream is triggered, acquiring pre-stored transmission layer data and application layer data corresponding to the target live broadcast stream; the transmission layer data and the application layer data are stored when the live broadcast end transmits live broadcast data of the target live broadcast stream to the edge node; combining the transmission layer data and the application layer data to obtain live broadcast data in the target live broadcast stream; transmitting the live broadcast data to an edge node; the live broadcast data is used for indicating the edge node to reproduce and transmit the live broadcast data according to the live broadcast transmission path of the target live broadcast stream. By adopting the method, the live broadcast stream can be accurately reproduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a live streaming data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of Internet technology, live streaming has become an indispensable part of people's daily life and work, and is widely used in entertainment, education, conferences, e-commerce and other fields. Live streaming technology achieves efficient and instant information dissemination through real-time transmission of audio and video data, greatly enriching the user's interactive experience. During the live broadcast process, due to various reasons such as network fluctuations, equipment failures, encoding and decoding errors, or excessive server load, the live stream may experience a series of problems such as freezes, delays, screen distortion, and interruptions. These problems not only seriously affect the audience's viewing experience, but also bring great challenges to the operation and maintenance personnel of the live broadcast platform. In the troubleshooting stage after the live broadcast problem occurs, once the problem appears and disappears, it is often difficult to accurately reproduce it, which is not conducive to locating and solving the live broadcast problem. Summary of the invention

[0003] Based on this, it is necessary to provide a live stream data processing method, device, computer equipment, computer-readable storage medium and computer program product that can accurately reproduce the live stream in response to the above technical problems.

[0004] In a first aspect, the present application provides a method for processing live streaming data, comprising:

[0005] When triggering the live stream reproduction process for the target live stream, the transport layer data and application layer data corresponding to the pre-stored target live stream are obtained; the transport layer data and application layer data are stored when the live broadcast end transmits the live data of the target live stream to the edge node;

[0006] Combine the transport layer data and the application layer data to obtain the live data in the target live stream;

[0007] The live broadcast data is transmitted to the edge node; the live broadcast data is used to instruct the edge node to reproduce and transmit the live broadcast data according to the live broadcast transmission path of the target live broadcast stream.

[0008] In a second aspect, the present application also provides a live streaming data processing device, including:

[0009] The pre-stored data acquisition module is used to acquire the transport layer data and application layer data corresponding to the pre-stored target live stream when the live stream reproduction processing for the target live stream is triggered; the transport layer data and application layer data are stored when the live broadcast end transmits the live data of the target live stream to the edge node;

[0010] The pre-stored data combination module is used to combine the transport layer data and the application layer data to obtain the live data in the target live stream;

[0011] The live data reproduction module is used to transmit the live data to the edge node; the live data is used to instruct the edge node to reproduce and transmit the live data according to the live transmission path of the target live stream.

[0012] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above live streaming data processing method are implemented.

[0013] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above live streaming data processing method are implemented.

[0014] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above live streaming data processing method are implemented.

[0015] The above-mentioned live stream data processing method, device, computer equipment, computer-readable storage medium and computer program product, when triggering the live stream reproduction processing for the target live stream, obtain the pre-stored transport layer data and application layer data corresponding to the target live stream, the transport layer data and application layer data are stored when the live broadcast end transmits the live data of the target live stream to the edge node, the transport layer data and the application layer data are combined to obtain the live data, and the live data is transmitted to the edge node to instruct the edge node to reproduce and transmit the live data according to the live transmission path of the target live stream. When triggering the live stream reproduction processing for the target live stream, the pre-stored transport layer data and application layer data are obtained and combined to obtain the live data in the target live stream, and the live data is transmitted to the edge node to reproduce and transmit according to the live transmission path of the target live stream, so that the live stream can be accurately reproduced through the pre-stored transport layer data and application layer data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is an application environment diagram of a live streaming data processing method in one embodiment;

[0018] Figure 2 A schematic diagram of a flow chart of a live streaming data processing method in one embodiment;

[0019] Figure 3 A schematic diagram of a process of pre-storing data in an embodiment;

[0020] Figure 4 A schematic diagram of the encapsulation format of transport layer data in one embodiment;

[0021] Figure 5 A schematic diagram of the encapsulation format of application layer data in one embodiment;

[0022] Figure 6 A schematic diagram of the structure of a live streaming data processing system in one embodiment;

[0023] Figure 7 A flowchart of a live streaming data processing method in another embodiment;

[0024] Figure 8 is a structural block diagram of a live streaming data processing device in one embodiment;

[0025] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] The live streaming data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the first terminal 102 and the second terminal 104 communicate with the edge node server 106 through the network respectively, and the data storage system of the edge node server 106 can store the data to be processed by the edge node server 106. The data storage system can be integrated on the edge node server 106, or it can be placed on the cloud or other network servers. The edge node server 106 can also communicate with the server 108 through the network, and the data storage system of the server 108 can store the data that the server 108 needs to process, and the data storage system can be integrated on the server 108, or it can be placed on the cloud or other network servers.

[0028] In the live broadcast system, the first terminal 102 can be a live broadcast terminal, and the second terminal 104 can be a live broadcast audience terminal, that is, the anchor can send live broadcast data to the edge node server 106 through the first terminal 102, and the edge node server 106 forwards the live broadcast data to the second terminal 104 for playback and display. During the live broadcast process, when the first terminal 102 transmits the live broadcast data of the target live broadcast stream to the edge node server 106, the server 108 can store the live broadcast data obtained by the edge node server 106, such as storing it in the data storage system corresponding to the server 108. When triggering the live broadcast stream reproduction processing for the target live broadcast stream, if the target live broadcast stream is abnormal and needs to be located and solved, the server 108 can obtain the pre-stored transport layer data and application layer data corresponding to the target live broadcast stream from the connected data storage system. The server 108 can combine the transport layer data and the application layer data to obtain the live broadcast data, and transmit the live broadcast data to the edge node server 106 to instruct the edge node server 106 to reproduce and transmit the live broadcast data according to the live broadcast transmission path of the target live broadcast stream. Specifically, the edge node server 106 can forward the live broadcast data to the second terminal 104, thereby realizing the reproduction transmission of the target live broadcast stream.

[0029] In some embodiments, the server 108 can also be integrated into the edge node server 106, that is, the edge node server 106 can directly store the obtained live data, and when the live stream reproduction processing for the target live stream is triggered, the edge node server 106 will reproduce and transmit the combined live data according to the live transmission path of the target live stream.

[0030] Among them, the first terminal 102 or the second terminal 104 can be, but not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The edge node server 106 or the server 108 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0031] In an exemplary embodiment, Figure 2 As shown, a live streaming data processing method is provided, and the method is applied to Figure 1 The server in the example is used to illustrate, including the following steps 202 to 206. Among them:

[0032] Step 202, when triggering the live stream reproduction processing for the target live stream, obtain the pre-stored transport layer data and application layer data corresponding to the target live stream; the transport layer data and application layer data are stored when the live broadcast end transmits the live data of the target live stream to the edge node.

[0033] Among them, live streaming, namely push streaming and pull streaming, is an application form of streaming media. Streaming media refers to audio and video that is played while being downloaded, not audio and video that is downloaded and cached in advance. Live streaming requires the current real-time audio and video data of the anchor to be transmitted, so that the anchor and the audience can interact in time, so the process of push streaming and pull streaming is required to form the real-time nature of live streaming. The target live streaming is the live streaming that needs to be reproduced. The live streaming reproduction process refers to the process of re-simulating and reproducing the specific scene and state of the problem through a series of technical means and methods after the live streaming has problems or exceptions, so as to locate, analyze and solve the problem. In the live streaming environment, the live streaming may be affected by many factors, such as network conditions, device performance, encoding and decoding problems, etc. These factors may cause abnormalities in the live streaming. When these abnormalities occur, in order to accurately find the root cause of the problem and take corresponding solutions, the live streaming needs to be reproduced.

[0034] Transport layer data refers to data belonging to the transport layer in the target live stream, and the transport layer data is transmitted based on the transport layer protocol; application layer data refers to data belonging to the application layer in the target live stream, and the application layer data is transmitted based on the application layer protocol. The live data in the target live stream can be divided into transport layer data and application layer data according to the protocol layer to which it belongs. For the transport layer data and application layer data, they can also be combined and restored to obtain the live data of the target live stream. The edge node is a node server closest to the end user, usually located in a geographical location close to the user. The edge node is responsible for processing tasks related to user interaction to reduce latency and improve user experience. For example, the edge node can be used as a node server in a content distribution network system to support the anchor's nearby streaming and the distribution and nearby access of live streams.

[0035] Optionally, during the live broadcast process, the live data of the live stream is sent from the live broadcast terminal to the edge node, and forwarded by the edge node to the audience terminal for playback. When the live data of the target live stream is transmitted to the edge node, the server can store the transport layer data and application layer data in the target live stream. When triggering the live stream reproduction processing for the target live stream, such as when the target live stream generates an abnormality and needs to be located, it can be considered that the live stream reproduction processing for the target live stream is triggered, and the server can obtain the pre-stored transport layer data and application layer data corresponding to the target live stream. In some embodiments, the server can obtain the corresponding pre-stored transport layer data and application layer data based on the live stream identification information of the target live stream, such as based on the live stream name of the target live stream.

[0036] Step 204: Combine the transport layer data and the application layer data to obtain the live data in the target live stream.

[0037] Exemplarily, the server may combine the obtained transport layer data and application layer data to obtain the live data in the target live stream. In some embodiments, the server may encapsulate the transport layer data and application layer data according to the live data encapsulation format of the target live stream to obtain the live data in the target live stream.

[0038] Step 206, the live broadcast data is transmitted to the edge node; the live broadcast data is used to instruct the edge node to reproduce and transmit the live broadcast data according to the live broadcast transmission path of the target live broadcast stream.

[0039] Exemplarily, the server can transmit the obtained live broadcast data to the edge node, so that the edge node can reproduce and transmit the live broadcast data according to the live broadcast transmission path of the target live broadcast stream. Specifically, the live broadcast data can be transmitted to the corresponding audience terminal according to the live broadcast transmission path of the target live broadcast stream, thereby realizing the live broadcast stream reproduction processing for the target live broadcast stream.

[0040] In the above live stream data processing method, when triggering the live stream reproduction processing for the target live stream, the pre-stored transport layer data and application layer data corresponding to the target live stream are obtained, and the transport layer data and application layer data are stored when the live broadcast end transmits the live data of the target live stream to the edge node. The transport layer data and the application layer data are combined to obtain the live data, and the live data is transmitted to the edge node to instruct the edge node to reproduce and transmit the live data according to the live transmission path of the target live stream. When triggering the live stream reproduction processing for the target live stream, the pre-stored transport layer data and the application layer data are obtained and combined to obtain the live data in the target live stream, and the live data is transmitted to the edge node to reproduce and transmit according to the live transmission path of the target live stream, so that the live stream can be accurately reproduced through the pre-stored transport layer data and application layer data.

[0041] In an exemplary embodiment, when the live stream reproduction processing for the target live stream is triggered, the pre-stored transport layer data and application layer data corresponding to the target live stream are obtained, including: when there is a transmission abnormality in the target live stream, the live stream identification information of the target live stream is determined; based on the live stream identification information, the pre-stored transport layer encapsulation data and application layer encapsulation data corresponding to the target live stream are obtained; the transport layer encapsulation data and the application layer encapsulation data are respectively parsed to obtain the transport layer data and application layer data corresponding to the target live stream.

[0042] The live stream identification information is used to represent the live stream, and different live streams may correspond to different live stream identification information. For example, the live stream identification information may include identification information such as the live stream ID (Identification) and the live stream name. The transport layer encapsulation data is the encapsulation data obtained after encapsulating the transport layer data, and may be obtained by encapsulating the transport layer data according to a preset encapsulation format, and the transport layer encapsulation data may carry the transport layer data; the application layer encapsulation data is the encapsulation data obtained after encapsulating the application layer data, and may be obtained by encapsulating the application layer data according to a preset encapsulation format, and the application layer encapsulation data may carry the application layer data.

[0043] Exemplarily, when there is a transmission anomaly in the target live stream, such as when the transmission of the target live stream is interrupted, the server can determine to trigger the live stream reproduction processing for the target live stream, and the server can determine the live stream identification information of the target live stream, such as determining the live stream name of the target live stream. The server can obtain the pre-stored transport layer encapsulation data and application layer encapsulation data corresponding to the target live stream based on the live stream identification information. In some embodiments, the transport layer encapsulation data and the application layer encapsulation data can be stored separately, and the server can query and obtain the transport layer encapsulation data and the application layer encapsulation data respectively based on the live stream identification information. The server can parse the obtained transport layer encapsulation data and application layer encapsulation data respectively, so as to obtain the transport layer data and application layer data corresponding to the target live stream.

[0044] In this embodiment, when there is a transmission anomaly in the target live stream, the corresponding pre-stored transport layer encapsulation data and application layer encapsulation data are obtained according to the live stream identification information of the target live stream, and the transport layer encapsulation data and application layer encapsulation data are respectively parsed to obtain the corresponding transport layer data and application layer data, so that the live stream can be accurately reproduced through the pre-stored transport layer data and application layer data.

[0045] In an exemplary embodiment, the transport layer encapsulation data and the application layer encapsulation data are parsed respectively to obtain the transport layer data and the application layer data corresponding to the target live stream, including: parsing the transport layer encapsulation data according to the transport layer encapsulation format to obtain the transport layer data corresponding to the target live stream; parsing the application layer encapsulation data according to the application layer encapsulation format to obtain the application layer data corresponding to the target live stream.

[0046] Among them, the transport layer encapsulation format refers to the encapsulation format used for encapsulating the transport layer data. According to the transport layer encapsulation format, the transport layer encapsulation data can be parsed to restore the corresponding transport layer data. The application layer encapsulation format refers to the encapsulation format used for encapsulating the application layer data. According to the application layer encapsulation format, the application layer encapsulation data can be parsed to restore the corresponding application layer data. The transport layer encapsulation format and the application layer encapsulation format can be configured according to actual needs. According to the transport layer encapsulation format and the application layer encapsulation format, the live data in the live stream can be encapsulated and parsed, thereby ensuring the security of the live data.

[0047] Optionally, the server may determine a transport layer encapsulation format for the transport layer data, and parse the transport layer encapsulation data according to the transport layer encapsulation format to obtain the transport layer data corresponding to the target live stream. The server may determine an application layer encapsulation format for the application layer data, and parse the application layer encapsulation data according to the application layer encapsulation format to obtain the application layer data corresponding to the target live stream.

[0048] In this embodiment, the server parses the transport layer encapsulation data according to the transport layer encapsulation format to obtain transport layer data, and parses the application layer encapsulation data according to the application layer encapsulation format to obtain application layer data, so that the live stream can be accurately reproduced through the pre-stored transport layer data and application layer data.

[0049] In an exemplary embodiment, the transport layer data and the application layer data include a data arrival timestamp; the transport layer data and the application layer data are combined to obtain the live data in the target live stream, including: determining the matching relationship between the transport layer data and the application layer data according to the data arrival timestamp; representing the matching relationship as matching transport layer data and application layer data are combined to obtain the live data in the target live stream.

[0050] Among them, the data arrival timestamp is used to identify the time when the live stream data arrives at the edge node. For the transport layer data and application layer data of the same live stream, the transport layer data and the application layer data both include the data arrival timestamp of the live stream, that is, the transport layer data and the application layer data of the same live stream have the same data arrival timestamp. The matching relationship is used to associate the transport layer data with the application layer data. If the matching relationship is expressed as matched, it can be determined that the transport layer data and the application layer data belong to the live data at the same time in the same live stream, and can be combined to restore the live data. If the matching relationship is expressed as unmatched, it means that the transport layer data and the application layer data do not belong to the same time, that is, the transport layer data and the application layer data belong to the live data at different times, and it is impossible to accurately restore the corresponding live data based on the combination of the transport layer data and the application layer data.

[0051] Exemplarily, the server may determine the data arrival timestamps respectively included in the transport layer data and the application layer data, and the server may determine the matching relationship between the transport layer data and the application layer data according to the data arrival timestamp, thereby determining the matching relationship between each transport layer data and each application layer data. Based on the matching relationship, the server may combine the transport layer data and the application layer data whose matching relationship is matched, thereby obtaining the live data in the target live stream. In some embodiments, the server may determine the matching relationship between the transport layer data and the application layer data having the same data arrival timestamp as matched, and for the transport layer data and the application layer data having different data arrival timestamps, the server may determine the matching relationship between the two as mismatched.

[0052] In this embodiment, the server determines the matching relationship between the transport layer data and the application layer data based on the data arrival timestamp, and combines the transport layer data and application layer data with matching matching relationships to obtain the live data in the target live stream, thereby accurately determining the matching transport layer data and application layer data to be combined based on the data arrival timestamp, ensuring the accuracy of the combined live data, thereby ensuring the accuracy of the live stream reproduction.

[0053] In an exemplary embodiment, Figure 3 As shown, the live streaming data processing method also includes data pre-storage processing, specifically including steps 302 to 306. Among them:

[0054] Step 302: When it is detected that the live data in the target live stream is transmitted to the edge node, the live data in the target live stream is obtained.

[0055] The live broadcast data includes audio and video data transmitted in the target live broadcast stream. Optionally, during the live broadcast process, the server can detect the transmission process of the live broadcast data in the target live broadcast stream. When it is detected that the live broadcast data is transmitted to the edge node, the edge node can forward the received live broadcast data to the viewer terminal for playback. The server can obtain the live broadcast data transmitted to the edge node. In some embodiments, the server can analyze the live broadcast historical abnormal data to determine the live broadcast stream that is prone to abnormalities. When the target live broadcast stream is a live broadcast stream that is prone to abnormalities, the server can obtain the live broadcast data in the target live broadcast stream for pre-storage.

[0056] Step 304: parse and split the live broadcast data to obtain transport layer data and application layer data corresponding to the target live broadcast stream.

[0057] Exemplarily, the server may parse and split the acquired live data, thereby obtaining transport layer data and application layer data from the live data, and the server may store the transport layer data and application layer data in different ways.

[0058] Step 306: Store the transport layer data and the application layer data separately.

[0059] Optionally, the server may store the transport layer data and the application layer data separately. For example, the server may store the transport layer data and the application layer data in different storage spaces. In some embodiments, the server may also encapsulate the transport layer data and the application layer data separately, and store them separately based on the encapsulated data.

[0060] In this embodiment, when the live data in the target live stream is transmitted to the edge node, the server obtains the live data, parses and splits the live data into transport layer data and application layer data, and stores them separately. Therefore, when the live stream reproduction processing of the target live stream is triggered, the live stream can be accurately reproduced through the pre-stored transport layer data and application layer data.

[0061] In an exemplary embodiment, transport layer data and application layer data are stored separately, including: encapsulating the transport layer data according to the transport layer encapsulation format to obtain transport layer encapsulation data, wherein the transport layer encapsulation data carries the transport layer data; encapsulating the application layer data according to the application layer encapsulation format to obtain application layer encapsulation data, wherein the application layer encapsulation data carries the application layer data; and storing the transport layer encapsulation data and the application layer encapsulation data separately.

[0062] The transport layer encapsulation format is a format for encapsulating transport layer data, and the application layer encapsulation format is a format for encapsulating application layer data. The transport layer encapsulation format and the application layer encapsulation format can be pre-configured according to actual needs.

[0063] Exemplarily, the server may determine the configured transport layer encapsulation format and application layer encapsulation format, and encapsulate the transport layer data according to the transport layer encapsulation format to obtain transport layer encapsulation data carrying the transport layer data. The server may encapsulate the application layer data according to the application layer encapsulation format to obtain application layer encapsulation data carrying the application layer data. The server may store the transport layer encapsulation data and the application layer encapsulation data separately, such as the server may store the transport layer encapsulation data and the application layer encapsulation data in different storage spaces respectively. When triggering the live stream reproduction processing for the target live stream, the server may obtain the transport layer encapsulation data and the application layer encapsulation data respectively, and extract the transport layer data and the application layer data from the transport layer encapsulation data and the application layer encapsulation data respectively, and obtain the corresponding live data based on the combination of the transport layer data and the application layer data, thereby realizing the live stream reproduction processing for the target live stream.

[0064] In this embodiment, the server can encapsulate the transport layer data according to the transport layer encapsulation format, encapsulate the application layer data according to the application layer encapsulation format, and store the encapsulated transport layer encapsulation data and application layer encapsulation data respectively, so that when the live stream reproduction processing of the target live stream is triggered, the live stream can be accurately reproduced through the pre-stored transport layer data and application layer data.

[0065] The present application also provides an application scenario, and the application scenario applies the above-mentioned live streaming data processing method. Specifically, the application of the live streaming data processing method in the application scenario is as follows:

[0066] This application scenario involves the fields of IT and software development, specifically the video live streaming abnormality recording and mirroring reproduction system. With the rapid development of information technology, video live streaming technology has also been developing continuously, bringing a lot of convenience to people. At the same time, due to the particularity of some source live streaming, there will be some abnormalities in the live streaming of this source live streaming scenario. At present, video live streaming is generally built based on Nginx and RTMP (Real Time Messaging Protocol) modules. Nginx can be used to implement web services, reverse proxy and mail proxy servers, etc., with the advantages of lightweight and high performance.

[0067] In live video broadcasting, the anchor usually pushes the stream through the RTMP protocol, reaches the edge node of the CDN (Content Delivery Network), then forwards it to the parent node, and then forwards it to the central node. The audience can play the stream through various protocols, generally FLV (Flash Video), RTMP and HLS (HTTP Live Streaming, HTTP-based adaptive bitrate streaming media transmission protocol). Among them, the real-time message transmission protocol is a real-time message sending protocol based on the TCP / IP (Transmission Control Protocol / Internet Protocol) protocol suite. It has the advantages of stability and strong compatibility and is widely used in live broadcast and on-demand scenarios. The content distribution network is a virtual network added between the user and the server, which caches the requested content on the edge server to facilitate timely response to user requests. The audience's streaming playback request is also first requested to the edge node. If it does not hit, it returns to the upper node to pull the stream back to the source. If the audience has transcoding requirements, the central node interacts with the transcoding service when pulling the stream, and the transcoding service gives the corresponding transcoding stream, which is returned to the audience layer by layer by the central node. If some abnormalities occur after a host pushes the stream, such as direct push failure, failure to pull the stream and play in certain scenes, transcoding failure, etc., the problem needs to be located at this time. It is best to have an active live stream to facilitate the reproduction of the problem and solve it.

[0068] The existing solution will directly record the live stream. When a problem occurs, the problem is reproduced and located by parsing the recorded file. This method of directly using the live stream recording file to reproduce the problem can only locate some problems, and cannot completely reproduce the source stream. If the encapsulation format of the source live stream or the network flash and jitter cause a logical error in the live server, the source stream cannot be reproduced by recording the file, and the problem cannot be reproduced and located by this method, resulting in problems caused by abnormal live streams online that are difficult to reproduce and solve. Therefore, a method that can completely reproduce the live stream is needed to solve problems caused by abnormal live streams. Generally, such problems are difficult to troubleshoot from logs, and the efficiency of reproduction and positioning after active live streams is greatly improved.

[0069] Based on this, the live streaming data processing method provided in this embodiment involves a live streaming abnormality record and mirror reproduction method based on CDN edge nodes, which greatly improves the efficiency of reproducing and locating problems caused by live streaming abnormalities in the online environment. When the anchor pushes the stream to the CDN edge server, the live streaming data processing method provided in this embodiment uses the CDN edge node live streaming abnormality record and mirror system service to capture packets in real time, separate and parse the TCP layer data and the application layer data, encapsulate them separately, and upload them to the S3 object storage (Simple Storage Service) in real time. When a problem needs to be reproduced, the S3 object storage is read through the CDN edge node live streaming abnormality record and mirror system service, and the data is split into a specified network data format according to the TCP layer and the application layer, and pushed to the CDN edge server in real time. This method can restore the live stream by mirroring, and can completely reproduce the source live stream, greatly improving the efficiency of solving problems caused by abnormal live streams in the online environment.

[0070] Specifically, in the live streaming data processing method provided by this embodiment, after the client pushes the stream to the live edge node, the server can realize the live streaming abnormality recording and mirroring system service, and the specific server can capture packets in real time through Tcpdump, and can distinguish different live streams through source address, destination address, source port number and destination port number, so as to obtain the packet capture data of different live streams. Among them, Tcpdump is a command line network analysis tool widely used on Linux systems, which is used to capture and display data packets transmitted on the network.

[0071] The server reads the packet data in real time. The first thing it encounters is the network layer header. By analyzing the header protocol field, it can determine the transport layer protocol used by the data packet, such as TCP, and the TCP protocol number is 3. After confirming that the data packet uses the TCP protocol, the TCP header is parsed next. The TCP header contains information such as the source port number, destination port number, sequence number, confirmation number, header length, flag bit, etc. After the TCP header, there is the application layer data. The application layer data under the RTMP protocol includes information such as messages, message blocks, packet headers, data types, timestamps, and handshakes.

[0072] The server obtains TCP layer data and application layer data respectively, and designs private formats to encapsulate the two types of data. Figure 4 As shown, the TCP layer data private format includes source IP, source port number, destination IP, destination port number, data length, data arrival timestamp and actual TCP layer data. Among them, source IP, source port number, destination IP and destination port number are convenient for tracking the source and destination of data, and data arrival timestamp is the timestamp of TCP layer data packet arrival, in milliseconds.

[0073] like Figure 5As shown in the figure, the private format of application layer data includes source IP, source port number, destination IP, destination port number, data length, data arrival timestamp, message type, live stream name, data integrity identifier, encoding format and actual RTMP application layer data. The data arrival timestamp of application layer data is kept consistent, which facilitates the corresponding relationship between application layer data and TCP layer data when downloading, analyzing and restoring live streams. The message type identifies the type of message, such as audio, video, control message, etc. In RTMP, each stream has a name, and including this information in a private format can help identify and manage different streams. The data integrity identifier can be used to verify the integrity of the data, which helps to detect when the data is damaged or tampered with. The encoding format can be correctly decoded for different encoding formats during playback. The TCP layer data and application layer data are encapsulated in the private format mentioned above and uploaded to S3 object storage.

[0074] When reproducing, the server downloads the TCP layer data and application layer data of the specified live stream from the S3 object storage. For the TCP layer data, the TCP header information needs to be removed and parsed to obtain the Chunk data of the RTMP message. According to the provisions of the RTMP protocol, the application layer data is combined according to the Chunk data to restore the complete RTMP message, and the RTMP message is decoded and processed to extract the audio and video data and control information, etc. Finally, the data is pushed to the CDN edge server to mirror and reproduce the live stream.

[0075] In the live streaming data processing method provided in this embodiment, complete data is obtained by real-time packet capture, corresponding TCP layer data and application layer data are obtained through real-time analysis, the TCP layer data and application layer data are analyzed and processed according to two designed private formats, and uploaded to S3 object storage. When reproducing, the live streaming is restored by downloading the corresponding TCP layer data and application layer data for analysis and reorganization of the image, which is convenient for analyzing and locating service abnormalities caused by abnormal flows in online environments.

[0076] In the system corresponding to the live streaming data processing method provided in this embodiment, Figure 6As shown, it can include a live stream management module, a data acquisition module, a data analysis module, a data encapsulation module, a data upload module, a data download module, and a data sending module. Among them, the live stream management module receives, sends, and distributes all live streams; the data acquisition module captures packets through Tcpdump to obtain complete data; the data analysis module analyzes the complete data and encapsulates it into application layer data and TCP layer data respectively, and can also analyze and reorganize the encapsulated application layer data and TCP layer data to restore the live stream; the data upload module uploads the encapsulated application layer data and TCP layer data to the S3 object storage; the data download module obtains the encapsulated application layer data and TCP layer data from the S3 object storage; the data sending module sends the live stream data to the CDN edge server.

[0077] In a specific application, such as Figure 7 As shown in the figure, the server can obtain the complete data packet during the live broadcast process, extract the TCP layer data and the application layer data respectively, and encapsulate them according to the preset private format and upload them to the S3 object storage. When reproducing the live stream, the server can download the encapsulated TCP layer data and application layer data from the S3 object storage, and parse them according to the private format respectively to completely restore the live stream and send the live stream to the CDN edge node.

[0078] Specifically, assume that the edge, parent layer, and center IPs are ip1, ip2, and ip3 respectively, and there is a simulated client streaming machine with an IP of ip4. The first step is to use ffmpeg (audio and video processing tool) or OBS (Open Broadcaster Software, open source broadcast software) software to simulate the client streaming to the live edge node, and set the streaming name to / live / test. The live streaming abnormality record and mirror reproduction system service first obtains the PCAP (PacketCapture, captured data packet) file containing complete data through Tcpdump packet capture, and filters out the live streaming data by filtering the peer IP, port, local IP, and port 1935. The filtered data contains the application layer data and TCP layer data of the live streaming. In the second step, the live stream abnormality record and mirror reproduction system service obtains the source IP, source port number, destination IP, destination port number, data length, data arrival timestamp, message type, live stream name, data integrity identifier, encoding format, actual RTMP application layer data and actual TCP layer data, and then encapsulates the application layer data and TCP layer data respectively according to the corresponding private format, obtains the encapsulated application layer data and TCP layer data, and uploads them to the S3 object storage. In the third step, when the source live stream needs to be troubleshooted, the live stream abnormality record and mirror reproduction system service can download the application layer data and TCP layer data of the corresponding live stream from the S3 object storage through the relevant information. The downloaded data is parsed and restored through two private formats, and the restored live stream is pushed to the CDN edge node for problem analysis.

[0079] In the live streaming data processing method provided in this embodiment, firstly, the complete data of the live streaming is obtained by real-time packet capture through Tcpdump, and the corresponding TCP layer data and application layer data are obtained through real-time analysis. The TCP layer data and application layer data are analyzed, processed and packaged according to the two designed private formats, and uploaded to S3 object storage. When reproducing, the corresponding TCP layer data and application layer data are downloaded for analysis and reorganization of the image to restore the live streaming. Compared with the existing solution of directly using the recording file of the live streaming to reproduce the problem, the live streaming data processing method provided in this embodiment obtains the complete TCP layer data and application layer data, and analyzes, processes and packages the TCP layer data and application layer data respectively through the two designed private formats, and parses and restores the live streaming when reproducing, which can reproduce the live streaming more completely. Due to the encapsulation format of the source live streaming or the logical error of the live streaming server caused by the network flash and jitter, the live streaming data processing method provided in this embodiment can also reproduce the source live streaming, thereby improving the efficiency of online problem troubleshooting.

[0080] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0081] Based on the same inventive concept, the embodiment of the present application also provides a live stream data processing device for implementing the live stream data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more live stream data processing device embodiments provided below can refer to the limitations of the live stream data processing method above, and will not be repeated here.

[0082] In an exemplary embodiment, Figure 8 As shown, a live streaming data processing device 800 is provided, comprising: a pre-stored data acquisition module 802, a pre-stored data combination module 804 and a live streaming data reproduction module 806, wherein:

[0083] The pre-stored data acquisition module 802 is used to acquire the pre-stored transport layer data and application layer data corresponding to the target live stream when the live stream reproduction process for the target live stream is triggered; the transport layer data and application layer data are stored when the live broadcast end transmits the live data of the target live stream to the edge node;

[0084] The pre-stored data combination module 804 is used to combine the transport layer data and the application layer data to obtain the live data in the target live stream;

[0085] The live data reproduction module 806 is used to transmit the live data to the edge node; the live data is used to instruct the edge node to reproduce and transmit the live data according to the live transmission path of the target live stream.

[0086] In an exemplary embodiment, the pre-stored data acquisition module 802 is also used to determine the live stream identification information of the target live stream when there is a transmission anomaly in the target live stream; based on the live stream identification information, obtain the pre-stored transport layer encapsulation data and application layer encapsulation data corresponding to the target live stream; and parse the transport layer encapsulation data and the application layer encapsulation data respectively to obtain the transport layer data and application layer data corresponding to the target live stream.

[0087] In an exemplary embodiment, the pre-stored data acquisition module 802 is also used to parse the transport layer encapsulation data according to the transport layer encapsulation format to obtain the transport layer data corresponding to the target live stream; and to parse the application layer encapsulation data according to the application layer encapsulation format to obtain the application layer data corresponding to the target live stream.

[0088] In an exemplary embodiment, the transport layer data and the application layer data include a data arrival timestamp; the pre-stored data combination module 804 is also used to determine the matching relationship between the transport layer data and the application layer data according to the data arrival timestamp; the matching relationship is represented as a combination of matching transport layer data and application layer data to obtain the live data in the target live stream.

[0089] In an exemplary embodiment, it also includes a data pre-storage module, which is used to obtain the live data in the target live stream when it is detected that the live data in the target live stream is transmitted to the edge node; parse and split the live data to obtain the transport layer data and application layer data corresponding to the target live stream; and store the transport layer data and application layer data separately.

[0090] In an exemplary embodiment, the data pre-storage module is also used to encapsulate the transport layer data according to the transport layer encapsulation format to obtain the transport layer encapsulation data, which carries the transport layer data; encapsulate the application layer data according to the application layer encapsulation format to obtain the application layer encapsulation data, which carries the application layer data; and store the transport layer encapsulation data and the application layer encapsulation data separately.

[0091] Each module in the live streaming data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0092] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the live streaming data processing method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a live streaming data processing method is implemented.

[0093] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0094] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0095] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0096] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0097] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0098] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0099] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0100] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing live streaming data, characterized in that: The method comprises: When triggering the live stream reproduction process for the target live stream, obtaining the pre-stored transport layer data and application layer data corresponding to the target live stream; the transport layer data and the application layer data are stored when the live broadcast end transmits the live data of the target live stream to the edge node; Combining the transport layer data and the application layer data to obtain live data in the target live stream; The live broadcast data is transmitted to the edge node; the live broadcast data is used to instruct the edge node to reproduce and transmit the live broadcast data according to the live broadcast transmission path of the target live broadcast stream.

2. The method according to claim 1, characterized in that When triggering the live stream reproduction process for the target live stream, obtaining the pre-stored transport layer data and application layer data corresponding to the target live stream includes: When there is a transmission anomaly in the target live stream, determining live stream identification information of the target live stream; Based on the live stream identification information, obtaining the pre-stored transport layer encapsulation data and application layer encapsulation data corresponding to the target live stream; The transport layer encapsulation data and the application layer encapsulation data are parsed respectively to obtain the transport layer data and application layer data corresponding to the target live stream.

3. The method according to claim 2, characterized in that The parsing of the transport layer encapsulation data and the application layer encapsulation data respectively to obtain the transport layer data and the application layer data corresponding to the target live stream includes: Parsing the transport layer encapsulation data according to the transport layer encapsulation format to obtain the transport layer data corresponding to the target live stream; The application layer encapsulation data is parsed according to the application layer encapsulation format to obtain application layer data corresponding to the target live stream.

4. The method according to claim 1, characterized in that: The transport layer data and the application layer data include a data arrival timestamp; the combining the transport layer data and the application layer data to obtain the live data in the target live stream includes: Determining a matching relationship between the transport layer data and the application layer data according to the data arrival timestamp; The matching relationship is represented as combining the matching transport layer data and application layer data to obtain the live data in the target live stream.

5. The method according to claim 1, characterized in that: The method further comprises: When it is detected that the live data in the target live stream is transmitted to the edge node, the live data in the target live stream is acquired; Parsing and splitting the live broadcast data to obtain transport layer data and application layer data corresponding to the target live broadcast stream; The transport layer data and the application layer data are stored separately.

6. The method according to claim 5, characterized in that The storing the transport layer data and the application layer data separately includes: Encapsulating the transport layer data according to a transport layer encapsulation format to obtain transport layer encapsulation data, wherein the transport layer encapsulation data carries the transport layer data; Encapsulating the application layer data according to an application layer encapsulation format to obtain application layer encapsulation data, wherein the application layer encapsulation data carries the application layer data; The transport layer encapsulated data and the application layer encapsulated data are stored separately.

7. A live streaming data processing device, characterized in that: The device comprises: A pre-stored data acquisition module is used to acquire the pre-stored transport layer data and application layer data corresponding to the target live stream when triggering the live stream reproduction process for the target live stream; the transport layer data and the application layer data are stored when the live broadcast end transmits the live data of the target live stream to the edge node; A pre-stored data combination module, used for combining the transport layer data and the application layer data to obtain the live data in the target live stream; A live broadcast data reproduction module is used to transmit the live broadcast data to the edge node; the live broadcast data is used to instruct the edge node to reproduce and transmit the live broadcast data according to the live broadcast transmission path of the target live broadcast stream.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images