A method for predicting freeze in web live broadcast under RTP

Abstract

The present invention relates to a method for predicting freezes in web live broadcast under a real-time transport protocol, the method comprising: a plug-in program, a traffic collector, and an online adaptive freeze prediction model. The development program is applied to a web page to obtain client parameters, and its purpose is to provide basic parameters for determining subsequent freeze events and predicting and calculating corresponding features of freeze parameters. The traffic collector is used in a content service provider server to obtain network traffic that is about to be sent at the current moment. The periodically updated adaptive freeze prediction model is applied to the server side, and its purpose is to online predict freeze events in live broadcasts under the concept drift. Through the technical solution of the disclosed example, the freeze prediction of the live broadcast stream currently using the real-time transport protocol can be performed online and displayed in real time in the visual web page interface of the server administrator. This example enables the service provider to take timely measures on the live broadcast traffic that is about to be sent before the abnormality occurs, thereby improving user participation.

Description

Technical Field

[0001] The present invention relates to a technical method for predicting freezes in live network video streams under multiple applications, multiple network conditions and multiple audiences, and belongs to the field of network traffic analysis. Background Art

[0002] Video streaming accounts for a high proportion of network traffic, and live streaming traffic accounts for 68.2% of video streaming. For video service providers, it is important to provide high user service quality to increase user engagement. Commodity sales industries such as retail have also been impacted to varying degrees, and selling goods in the form of live streaming has become the mainstream commodity circulation model in the sales industry. At the same time, watching live streaming has gradually become a way for people to entertain themselves at home.

[0003] In the live broadcast scenario, the client has a small buffer. In the low-bandwidth scenario, the mismatch between the client buffer size and the received data size causes the occurrence of jamming. In this scenario, the client is prone to black screen or skipping a section of live content and then continuing to play. Many studies have shown that the number and duration of playback interruptions have the most serious impact on QOE. In contrast, users are willing to accept higher initial delays and higher video distortion. In addition, compared with on-demand video, live streaming has higher requirements in terms of latency, throughput, and quality switching. Analogous to other parameters that affect QOE indicators, such as initial delay, jamming will reduce the QOE value more quickly. The quality of live stream generation is mainly related to the uplink traffic of the live broadcast end and the downlink traffic of the user end. This paper aims to predict the occurrence of live broadcast jamming from the provider's perspective by analyzing the downlink live broadcast traffic of the web page combined with the client buffer occupancy. At present, the research on live broadcast traffic is limited to DASH or HLS protocol streams. Under the DASH and HLS protocols, the downlink live stream is played by integrating the data in the buffer for a period of time to generate fragments. Compared with the uplink live stream, this process has a higher latency. At present, web live broadcast traffic mostly uses RTP for data transmission. RTP can control video frames, so it can carry applications with high real-time requirements. Videos transmitted under RTP have a smaller transmission delay than those under DASH and HLS. However, researchers in various fields lack research related to RTP live streaming, and objective detection of QOE of downstream traffic of live streaming is also relatively vague.

[0004] Most of the existing methods for mapping video stream data to QOE based on passive measurement methods focus on machine learning methods or session modeling methods. In the session modeling method, statistical analysis is performed through the network layer or transport layer data structure to infer the video QOE. The machine learning method is to establish an association between observable indicators such as packet delay, loss and throughput on the network and video QOE through a model. At present, existing research on live streaming lacks objectivity and scalability in parameter assessment. Considering that providers need to adopt strategies to predict the quality loss of new sessions. However, its application scenario considers the same device as the client and the average bit rate experienced as method input, but the client devices are diverse and complex, and this method also lacks scalability. The research scenario is HTTP adaptive streaming, which has a relatively complete caching mechanism on the server side. In addition, for the prediction of QOE parameters, existing research lacks objectivity in assessment or increases the system workload while ensuring objectivity. In the process of offline training of QOE parameters, the scoring system still uses manual scoring, which increases the subjectivity of system judgment. In the research to ensure objectivity, the complete video needs to be input to obtain it, which increases the workload. In addition, existing research, whether it is for live streaming traffic or for the prediction of QOE parameters,

[0005] Therefore, in view of the current existing problems, the jam prediction is carried out under the mainstream real-time transport protocol (RTP) at this stage.

[0006] Due to the timeliness, both the server and the client frame buffer have small buffers. Therefore, it is necessary to consider the real-time update prediction model and online prediction of live broadcast jamming events under the small buffer in the existing research, so as to obtain high jamming prediction accuracy. The system designed in this paper can be applied to major live broadcast content providers at home and abroad. By running the traffic collection program and the client parameter collection program on the server, and developing a plug-in to start client parameter collection on the basis of client installation permission, the live broadcast jamming prediction model is run on the server managed by the live broadcast content provider, and it is updated periodically. This system helps live broadcast content service providers to predict client jamming events in time, so as to take countermeasures, such as compressing video quality, etc., thereby improving user participation. Summary of the invention

[0007] The present invention provides a method for predicting the freeze of web live broadcast that can be applied to multiple platforms. The present invention adopts the following technical scheme: a method for predicting the freeze of web live broadcast under RTP, which includes: a method for collecting playback parameters of multi-platform clients, a method for pre-distributing live broadcast traffic by automating the collection server, and a method for predicting the freeze of periodically updating the adaptive freeze. The method includes the following steps:

[0008] Step (1) The plug-in is installed on the user's browser after the user agrees to obtain information, and a regular expression is set to determine the domain name corresponding to the web page, so as to obtain the parameter information of the specified platform in a deterministic manner on different web live broadcast platforms.

[0009] Step (2) collecting video playback parameters of the webpage live broadcast client, obtaining user target user buffer information and live broadcast status;

[0010] Step (3) transmits the client parameters obtained in steps (1) and (2) back to the server in a cross-domain environment, so that the server obtains the client playback parameters in real time, and transmits the obtained client information in the form of a text file to a specified folder on the server.

[0011] Step (4) The traffic collector collects the client traffic to be sent by the server within a certain period, and collects and stores it in the specified folder of the server within a certain period of time.

[0012] Step (5) The server monitors the folder status in steps (3) and (4). When files are added to the traffic collection folder and the client parameter folder at the same time, the server starts the data processing program to process the collected traffic information and client parameters to obtain two predicted jamming features: codeword information and client frame buffer difference.

[0013] In step (6), the server extracts the RTP data segments from the collected traffic data to construct a spatiotemporal relationship structure, and combines the processed data features to send them into a periodically updated adaptive live broadcast freeze detection model, wherein the live broadcast freeze model is used to determine whether there is freeze in the live broadcast stream that the server is about to send.

[0014] Step (7) Based on steps (5) and (6), the server management site enters the management page initialization and obtains the jam result data that the server has predicted for the client.

[0015] Step (8) The server management site monitors the server model prediction results in real time and displays the result data in a visual website format.

[0016] In the first aspect, the method for collecting playback parameters of a multi-platform client provides a plug-in device that can collect status information during a live broadcast of a client, including:

[0017] Methods for obtaining video tags for different live broadcast websites:

[0018] Step (1.1) designs a web page plug-in to obtain designated website information and sets different web page filtering rules for different service providers. When the client user opens a web page, different filtering rules are set for different web pages by retrieving the domain name whitelist configured in the designed plug-in.

[0019] Step (1.2) sets different video tag acquisition methods for different live website platforms. The methods of obtaining tags currently summarized include: obtaining through tag names, such as getElementByTagname(); obtaining through the ID of the live video frame tag, such as getElementById(); obtaining through the class name of the live video frame tag, such as getElementByclassName(); different web pages have different methods for obtaining the tags of the live frames, and the classification statement is designed to determine the method of obtaining tags for the web page currently playing the live broadcast by the decision-making client user.

[0020] Methods for determining client buffer status information and live broadcast status at the millisecond level:

[0021] Step (2.1) obtains detailed status information of the client live broadcast for the acquired live video tag. The public javascript API of the video player based on HTML5 MSE can be used to obtain the current playback time of the video through the properties and real-time parameters of the web video player. The open API function currTime() is called to obtain the current playback time of the video;

[0022] Step (2.2) obtains the resolution information of the current video player by calling the videoWidth() and videoHeight() functions.

[0023] Step (2.3) calls the buffered.end() function to obtain the total buffering time of the current video service buffer; calls the buffered.start() function to obtain the initial total buffering time of the current video service buffer;

[0024] Step (2.4) performs data processing based on the initial information obtained above to obtain the buffer change status of the client and the buffered video frame information during the client user's live broadcast. The current playback time of the video is used to determine whether a freeze has occurred. If the current playback time of the video remains at the same number, it can be determined that a freeze has occurred; the total buffering time of the current video service buffer, and the difference between the current playback time of the video represent the duration of the video that has not been played in the current video service buffer; the initial total buffering time of the current video service buffer, this value is subtracted from the total buffering time of the current video service buffer to obtain the relative total buffering time of the current video service, which can be used as metadata for the duration of the video played in the buffer within the RTP data packet range; after the above steps, the client user playback information is obtained as follows: video resolution, timestamp, duration of the video not played in the buffer, and total buffering time of the buffer during the live broadcast cycle.

[0025] The obtained client parameters are sent back to the server, so that the server can obtain the previous playback parameters in real time:

[0026] Step (3.1) obtains the current timestamp on the client before and after uploading the data. The acquisition of the timestamp after uploading the data is triggered by receiving the server response. The two-way transmission time of this data transmission is obtained by the difference between the timestamps before and after data transmission. After average division, the transmission time required for the client to upload to the server under different bandwidths is obtained.

[0027] Step (3.2) sets a timer to periodically obtain client parameters, and follows the same data encoding and decoding method as the server. At the same time, the client data is uploaded to the server via ajax via the http protocol at intervals. In addition, when uploading in this way, cross-domain interaction needs to be considered, and the cross-domain value is set to true in the Body of its transmission. According to program testing, the system uploads a certain size of parameter file package to the server across domains under 2M bandwidth, and the time required is 267ms.

[0028] Step (3.3) Under the actual requirement of cross-domain upload, the client sets the cross-domain request identifier, and the data receiving server also sets the cross-domain interaction identifier, and transmits the parameters obtained by the client to the server.

[0029] In the second aspect, the method for automatically collecting live traffic from a server pre-distributes live traffic. This article provides a flow collector that can be run on a server to collect live traffic, including:

[0030] The method for the server to collect the client traffic to be sent within a certain period:

[0031] Step (4.1) opens a designated port (such as port 6088 in the test environment) to continuously listen to client requests and receive client data information according to the data transmission specifications negotiated with the client.

[0032] Step (4.2) The server collects the client traffic to be sent within a certain period. Receive client data information according to the negotiated data transmission specification. The information is carried in json data format, including timestamp information, client video live player resolution width, client video live player resolution length, playback time at the current timestamp, total buffering time at the current timestamp, and unplayed video time in the client buffer at the current timestamp.

[0033] Step (4.2) collects live traffic within a certain time range after receiving the client's parameter collection request, and collects data through the sniff function built into Python. A certain time collection range is set.

[0034] Step (4.3) forms the collected network traffic into a pcap file and stores it in the file directory specified by the server.

[0035] In the third aspect, the periodically updated adaptive jam prediction method provides a model that can be used for live streaming jam prediction, including:

[0036] Perform data processing on the collected traffic information and client parameters to obtain the method for predicting jamming features:

[0037] Step (5.1) The server sets up a folder detection program, uses the built-in component injection method of springboot, writes component files by itself and injects them into the container, and uses the @ServletComponentScan annotation at the program startup entry. This annotation can automatically scan all file classes annotated with @Component in the program. By implementing the ApplicationRunner interface, the monitoring information is initialized when the project starts. WatchService is used to observe all changes and events of the registered objects, and the folder to be detected is instantiated with the built-in function of the file monitor to instantiate the monitoring object, and the object is registered with WatchService to obtain the monitoring information pool bound to the folder. At this time, all change information in the folder will be imported into the monitoring pool. When there is no change in the monitored folder, the built-in function of springboot will be blocked in the monitoring program. After the monitored traffic collection folder and client parameter folder have changed, the data processing process will be entered. Extract the names of the files added to the traffic collection folder and the client parameter folder, call the known class that has been injected into springboot, the known class calls the Python jam prediction model, and receives the prediction result data obtained by the python model, and returns it to the configuration class.

[0038] After receiving the prediction result data obtained by the injection class, it is returned to the client service site with which the connection has been established via the websocket protocol program to visualize the prediction result data.

[0039] Step (5.2) obtains parameters such as timestamp and frame buffer status from the collected original live receiving end video analysis parameters through data processing. Parse the txt file to form json data, wherein the obtained json format list data is integrated into the following: using the current time value as the key, splicing the QOE object dictionary, using the timestamp value as the key, splicing the QOE object dictionary, using the jam start time as the key, storing the stallObject object of the modified non-jamming segment, using the jam start time as the key, storing the stallObject object of the modified jamming segment, storing the non-jamming segment time list [[start non-jamming time, non-jamming end time], [start non-jamming time, non-jamming end time], ..], storing the jamming segment time list [[start jamming time, jamming end time], [start jamming time, jamming end time], ..], recording the specific timestamp of non-jamming at the same current time, recording the specific timestamp of jamming at the same current time, recording the buffer change of jamming at the same current time, and recording the black screen time range.

[0040] Step (5.3) reads the file information in the specified directory of the server. For the collected pcap files, first use the extract function in the flowcontainer to extract them. Modify the extract function source code, and add the RTP information field to the source code to obtain the network traffic details information through the tshark command. Filter the RTP video stream by extracting and analyzing the reconstructed binary data. According to the information contained in the RTP data packet header, if the PT field of the RTP data segment header is in the range of 23 to 128, the RTP data stream is identified as a video data stream, and the other ranges are audio data streams. The purpose of this process is to obtain a pure RTP video stream from the collected original data stream. The data stream data corresponding to the freeze event can be located by obtaining the above parameters and timestamps.

[0041] Step (5.4) calculates the feature 1 required for the prediction model, and the frame buffer ratio difference calculation is designed as follows. The plug-in designed in step 101 can collect the difference in the length of the video not played in the frame buffer of the live receiving end at the start time and end time of the RTP video stream. This item reflects the actual video length played by the client within the time range of the RTP video stream, that is, the buffer consumption. Through the program, it is determined that under a 2M bandwidth, it will take 267ms for the client plug-in to upload the information to the server. Considering the normal network transmission rate of the client, we set it in the prediction process to use the length of the client's unplayed video at time t-0.2s instead of the client status at the current time t.

[0042] Step (5.5) calculates the feature 2 required by the prediction model, and the code calculation is designed as follows. The code data is calculated by the ratio of throughput to bit rate, which represents an estimate of how many bits of data each code element of the received video data carries. Parse the reorganized binary data of the collected data stream to obtain the total byte number lw_rtp of the RTP data segment numbered w. This item indicates the actual number of video payload bytes carried by the RTP data segment. The throughput is obtained by comparing the payload byte size carried by the RTP data segment with the time range corresponding to the RTP data segment. The bit rate of the wth RTP data segment depends on the actual RTP payload byte number and the client buffer cache situation. The video duration Vw stored in the buffer within the time range of the RTP data segment is the actual video duration. Among them, the buffer cache video duration Vw_i corresponding to the start timestamp of the RTP data segment is Bwi_end-Bwi_start. Similarly, the buffer cache video duration at the end timestamp. The actual video duration vw = vw_j-vw_i. The ratio of the RTP data segment payload byte size to the acquired frame buffer ratio difference is used as the transmission video bit rate value.

[0043] Method for constructing a representative RTP spatiotemporal relationship structure:

[0044] Step (6.1) deterministically extracts RTP data segments under different TCP streams by adding tshark to the original program code to obtain the reconstructed binary parameter information, and combines the command to obtain the rtp packet to obtain the IP data packet number range corresponding to the RTP data segment

[0045] Step (6.2) constructs a representative RTP spatiotemporal relationship structure. Deterministically extract RTP data segments under different TCP streams. Connect RTP data packets from different TCP data streams in timestamp order, and set different numbers for different streams. Connect the downlink live data packets with the same stream number, and connect the data packets with different stream numbers but adjacent in timestamp order. In this way, the spatiotemporal relationship structure of the RTP video stream is constructed. At the same time, the graph structure constructed within a certain time range is segmented to ensure that the scale of its sub-graph structure is fixed.

[0046] Step (6.3) divides the graph structure constructed within a certain time range to ensure that the scale of its sub-graph structure is fixed.

[0047] The prediction method of jamming for different application environments:

[0048] Step (7.1) selects the prediction model base model GGNN, which enhances GNN's ability to process input sequences. The model considers neighborhood nodes and selectively trains the model by combining the structural information between nodes and the node information itself. The model loss function uses an unweighted loss function to perform back propagation of the model to obtain its related gradient values.

[0049] Step (7.2) improves the model to improve the accuracy of jamming prediction. The model is designed to predict jamming in live broadcasts in different scenarios. When updating the loss value, multiple loss value integration is used to select the minimum loss value for feedback, and the feature count value corresponding to the loss value is increased. When predicting jamming events, the feature corresponding to the maximum feature count value is used for prediction. In the model output layer, different output layer architectures are designed for different numbers of features. As shown in the figure, output1 carries the output parameters when two features are used as parameters, and output2 carries the output parameters when a single feature is used as a parameter.

[0050] Step (7.3) combines the above steps to predict the jamming event. Obtain the characteristic parameters of the preceding 10 seconds and the subsequent 1 second corresponding to the jamming event. Then, send the obtained data to the designed optional feature prediction model to train the model. In the process of predicting the jamming event, this paper uses the next segment of client live traffic after the specified time to simulate the server collecting the live stream to be forwarded to the client, and combines the client parameters 0.2s before the specified time to send it to the data processing unit to form the data to be predicted, and finally obtain the jamming event.

[0051] To ensure that the additional delay caused by the server-side prediction of lag meets the client's requirements in the live broadcast scenario:

[0052] Step (8.1) sets up a performance detection program, sets up a timestamp acquisition program between the client plug-in program and the server receiving program, compares the difference in timestamps between data transmission interactions with the standard delay, and determines whether the delay caused by the predicted jam is within the tolerance range of the client.

[0053] Step (8.2) sets a certain range of total data volume and periodically updates the jam prediction model. In addition, before the new prediction model is updated, the new live stream data is cyclically detected to see whether it causes jams.

[0054] In a fourth aspect, the client real-time monitoring method for predicting jamming results provides a dynamically updated monitoring web page, including:

[0055] Method for initializing historical prediction result data for the server management station:

[0056] Step (9.1) After entering the homepage, the server management site immediately sends a websocket request to the server. In addition to the basic request conditions, the client random ID is attached to the websocket request and sent to the server. The server program uses the springboot framework. Under the full-duplex long connection of the websocket, the known class injected into springboot is called. The known class method traverses the prediction result file in the specified folder of the server, reads its content line by line, and uses the built-in string stream of java to splice it together to form an array list. At the same time, in this process, each line of prediction data is encoded and numbered.

[0057] After completing the basic data processing based on step (9.1), step (9.2) returns the historical prediction data of the client service site through the requested websocket channel by locking the request of the management site. After the server management site receives the historical prediction data obtained through step 10.1), it assigns the acquired data to the data list through the vue component and uses the v-for component to render the html dynamic tag. At the same time, after receiving the historical prediction data, the data list is arranged in reverse order and spliced ​​into the page data list. Through page dynamic rendering, the reverse historical prediction results are displayed in a table on the server administrator page.

[0058] Method for obtaining the jam prediction result data in real time from the server management station:

[0059] Step (10.1) sends a websocket full-duplex connection to obtain real-time data at the same time as step (9.1), and carries the client's random ID. After the server receives the request, it uses the random ID as the key and the session of the process as the value to store it on the server.

[0060] Step (10.2), based on step (10.1), obtains the known class injected into springboot, monitors the server traffic collection folder and the client parameter folder for file changes, and when files are added to the above two folders, jumps out of the blocking program, calls the external Python prediction model program through the springboot configuration file, and completes the prediction of the newly collected traffic.

[0061] The prediction results are generated by the Python model and returned to the springboot program. At the same time, the prediction results are returned to the actual websocket server program through the injection class. The server program uses the built-in string stream of java to splice the received data together to form an array list, and encodes and numbers each line of prediction data.

[0062] Step (10.3) is based on step (10.2), by locking the request channel of the management site, and returning the real-time prediction data of the client service site through the websocket channel of the specified ID. After receiving the data, the client splits the data in a comma-separated manner according to the data format specification negotiated with the server, so as to convert the string data obtained from the server into array data.

[0063] The basic array of the original dynamically rendered html tag is extracted and initialized into a new variable, the new array is initialized, the latest real-time data is used as the first predicted data in the new array, and the original predicted data is traversed at the same time, and the data is spliced ​​into the new array in order. At this time, the tag in the html file places the latest real-time prediction data at the top of the visual web page table through the v-for component, and continues to receive the implementation prediction data from the server and displays it in the administrator's web page.

[0064] Compared with the prior art, the present invention has the following advantages:

[0065] The present invention focuses on the popular live streaming protocol at this stage, and pioneers the research on the prediction of RTP live streaming jams. Based on its fragmented sequential transmission and the non-encrypted characteristics of the data packet header, the real-time parameters of the live traffic are obtained to carry out the research. At the same time, considering the real-time nature of the real-time transport protocol and the small buffer capacity, the system considers the additional delay overhead caused in the jam prediction process, and judges whether the prediction of live jams on different web live broadcast platforms (at home and abroad) can achieve the lowest delay overhead.

[0066] The design of the present invention makes the judgment of jamming events more objective. The designed plug-in program (Google) is applied to the client web page to collect jamming events and buffer conditions and send them to the server. This method is different from the traditional subjective human evaluation or the use of third-party tools to collect client playback parameters. At the same time, this method is based on the client user's self-selection of whether to allow the installation of the plug-in, which is in line with ethics and does not involve infringement of user privacy. At the same time, considering that if the user does not agree to install the plug-in, the bit rate characteristics of the network traffic clone are used alone to predict the jamming of the live broadcast, a higher prediction accuracy can also be achieved. This method makes the judgment of jamming events more objective. At the same time, the judgment of jamming events is at the millisecond level, which is more accurate than the current method.

[0067] The feature selection principle of the present invention is more in line with the live broadcast working mechanism. Based on the performance of the receiving device and the network conditions, the symbol coding and the frame buffer ratio difference are used as optional features for predicting freeze events. Compared with the direct network traffic features currently used, such as data packet or load length, RTT, etc., can be directly obtained from the network traffic based on timestamp information. However, the above features fail to reflect the direct correlation between live broadcast freezes and them. The prediction features selected by the present invention take into account the client network conditions and the insufficient decoding capabilities of the device, and respectively select the coding and the buffer ratio difference, that is, the length of the undecoded video in the buffer, as the prediction features.

[0068] The construction of the prediction model designed by the present invention is more in line with the actual state required by the client to predict the jamming. The present invention innovatively proposes a new graph structure construction method in the field of network flow. Construct the RTP spatiotemporal structure in the timestamp order, and design an automatic feature selection model for jamming prediction. Different from the traditional uplink and downlink composition method, the present invention connects the RTP data segments under the same stream, and connects the RTP data segments under different TCP stream data but connected in the timestamp order to construct. At the same time, based on (3), when training the model, different prediction features should be selected in different application scenarios to make predictions. At the same time, considering the dynamic nature of client devices and network conditions over time, it is set to update the model under a certain amount of data, and re-predict the new jamming data after the model is updated. Under periodic model updates, the generated model can better meet the current client jamming prediction requirements.

[0069] The present invention can obtain a higher prediction accuracy rate under the condition of less workload. The data set constructed for the invention test is huge. The system is constructed to contain 30,249,080 RTP data streams, 45,361 TCP data streams, and the total size of the data stream files is 55G. Under this data set, the designed prediction model can achieve an offline prediction accuracy rate of 99.17%. In addition, in the online real-time prediction process, the prediction can achieve a prediction accuracy rate of 97.44% in the case of a smaller amount of data, and at the same time, the prediction can achieve a prediction accuracy rate of 100% in the case of dense jams. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Design a framework diagram for the RTP live stream freeze prediction method;

[0071] Figure 2 This is the architecture diagram of the RTP live stream freeze prediction system;

[0072] Figure 3 A diagram of the visualization system platform for RTP live stream freeze prediction. DETAILED DESCRIPTION

[0073] In combination with the above-described steps, this section implements some cases in combination with the description of the relevant technical solution steps in the specification.

[0074] Embodiment 1: A method for predicting freezes in web live broadcast under RTP, comprising: a method for collecting playback parameters of a multi-platform client, a method for pre-distributing live broadcast traffic by an automated collection server, and a method for predicting freezes in a periodically updated adaptive manner. The method comprises the following steps:

[0075] Methods for obtaining video tags for different live broadcast websites:

[0076] Step (1.1) Design a web plug-in to obtain the specified website information. The test cases focus on domestic and foreign live broadcast platforms, including Douyu, Kuaishou, Youtube and Twitch live broadcast platforms. Set different web filtering rules for different service providers. For example, under the conditions of multiple web pages opened by the user, set: matches": ["https: / / *.youtube.com / *", "https: / / *.twitch.tv / *", "https: / / *.douyu.com / *", "https: / / *.younow.com / *"]. When the client user opens the web page, the domain name whitelist configured in the designed plug-in is retrieved to determine whether the plug-in is applied to the web page. At the same time, different filtering rules are set for different web pages to prevent the collection of on-demand data.

[0077] Step (1.2) sets different methods for obtaining video tags for different live website platforms. For Youtube and Twitch live websites, obtain through tag name, such as getElementByTagname(); for Douyu live platform, obtain through ID of live video frame tag, such as getElementById(); for Kuaishou live platform, obtain through class name of live video frame tag, such as getElementByclassName(); different web pages have different ways of obtaining tags of live frames. The classification statement is designed to determine the method of obtaining tags for the web page currently playing the live broadcast by the client user. The purpose of this operation is to obtain tag information through different methods. When a program bug occurs, tag information can also be obtained through other methods to prevent errors from affecting the normal execution of the program.

[0078] Method for determining client buffer status information and live broadcast status at millisecond level:

[0079] Step (2.1) obtains detailed status information of the client live broadcast for the acquired live video tag. The public javascript API of the video player based on HTML5 MSE can be used to obtain the current playback time of the video through the properties and real-time parameters of the web video player. The open API function currTime() is called to obtain the current playback time of the video;

[0080] Step (2.2) obtains the resolution information of the current video player by calling the videoWidth() and videoHeight() functions. The purpose of this step is to provide preliminary support for the subsequent data processing of obtaining the front-end feature parameters.

[0081] Step (2.3) calls the buffered.end() function to obtain the total buffering time of the current video service buffer; calls the buffered.start() function to obtain the initial total buffering time of the current video service buffer;

[0082] Step (2.4) performs data processing based on the initial information obtained above to obtain the buffer change status of the client and the buffered video frame information during the client user's live broadcast. The current playback time of the video is used to determine whether a freeze has occurred. If the current playback time of the video remains at the same number, for example, the current playback time is always maintained at 0, it can be determined that a freeze has occurred, and when the playback time is always 0, it means that the live video has not yet been played and is in the initial buffering stage; the total buffering time of the current video service buffer and the difference between the current playback time of the video represent the duration of the video that has not been played in the current video service buffer; the initial total buffering time of the current video service buffer, this value is subtracted from the total buffering time of the current video service buffer to obtain the relative total buffering time of the current video service, which can be used as metadata for the duration of the video played in the buffer within the range of the RTP data packet; after the above steps, the client user playback information is obtained as follows: video resolution, timestamp, duration of the video not played in the buffer, and total buffering time of the buffer during the live broadcast cycle.

[0083] The obtained client parameters are sent back to the server, so that the server can obtain the previous playback parameters in real time:

[0084] Step (3.1) obtains the current timestamp on the client before and after uploading the data. The acquisition of the timestamp after uploading the data is triggered by receiving the server response. The two-way transmission time of this data transmission is obtained by the difference between the timestamps before and after data transmission. After average division, the transmission time required for the client to upload to the server under different bandwidths is obtained.

[0085] Step (3.2) sets a timer to periodically obtain client parameters, and follows the same data encoding and decoding method as the server. At the same time, the client data is uploaded to the server at intervals through ajax via the http protocol. In addition, when uploading in this way, cross-domain interaction needs to be considered, and the cross-domain is set to true in the Body of its transmission. According to program testing, the system uploads a parameter file package of a certain size to the server across domains under 2M bandwidth, and the time required is 267ms. The role of this step is to unify the data receiving and sending formats of the front and back ends, and at the same time determine the processing method of the cross-domain environment during the data transmission path to prevent data from being sent or received. In addition, by determining the average transmission time of data upload and distribution under a certain bandwidth, it is possible to evaluate whether the transmission delay caused by data transmission is within the acceptable range for users. If it exceeds the threshold, the model needs to be adjusted to reduce the delay.

[0086] Step (3.3) Under the actual requirement of cross-domain upload, the client sets the cross-domain request identifier, and the data receiving server also sets the cross-domain interaction identifier, and transmits the parameters obtained by the client to the server.

[0087] Embodiment 2:

[0088] A stream collector that can run on the server to collect live traffic.

[0089] Method for the server to collect the client traffic to be sent within a certain period:

[0090] Step (4.1) opens the designated port 6088 to continuously listen to client requests and receives client data information according to the data transmission specifications negotiated with the client.

[0091] Step (4.2) The server collects the client traffic to be sent within a certain period. According to the negotiated data transmission specification, that is, {"timestamp": timestamp, "width": resolution width, "height": resolution height, "currTime": play time at the current timestamp, "bufferAllTime": total buffering time, "buffingTime": unplayed time}, the client data information is received. The information is carried in the json data format, including timestamp information, client video live player resolution width, client video live player resolution length, play time at the current timestamp, total buffering time at the current timestamp, and unplayed video time in the client buffer at the current timestamp. This process is based on the data format specified by the front and back ends in step 105. After receiving the information transmitted by the above client, appropriate features are extracted as model trainable features. After receiving the client collection parameters, if the obtained header information data value is 1 after parsing, the live traffic within a certain time range is collected, and data is collected through the sniff function built into Python, and a certain time collection range is set. The purpose of this operation is to collect the live broadcast traffic information pre-sent by the server in real time, so as to form the basic traffic data for subsequent jamming prediction.

[0092] Step (4.3) uses the wpcap function to convert the collected network traffic into a pcap file and then stores it in the specified file directory of the server. This operation facilitates the extraction of data in folders during subsequent model training and model testing. At the same time, when updating data, old data can be deleted and new data can be added in folders.

[0093] Embodiment 3:

[0094] A method for predicting freezes in live broadcasts can achieve a high prediction accuracy.

[0095] Perform data processing on the collected traffic information and client parameters to obtain the method for predicting jamming features:

[0096] Step (5.1) The server sets the folder detection program, uses the springboot built-in component injection method, writes the component file MonitorConfig class and injects it into the container, and annotates the program startup entry SpringApplication.run() with the @ServletComponentScan annotation, which can automatically scan all file classes annotated with @Component in the program. By implementing the ApplicationRunner interface, the monitoring information is initialized when the project starts. Use WatchService to observe all changes and events of the registered objects, use the file monitor built-in function to instantiate the monitoring object for the traffic collection folder and client parameter folder to be detected, and use the sub-function register under the Paths built-in function to register the object to WatchService, and obtain the monitoring information pool bound to the folder. At this time, all change information in the folder will be imported into the monitoring pool. When there is no change in the monitored folder, the springboot built-in function will be blocked in the monitoring program. This process is completed by the Webservice built-in takes() function. When the monitored traffic collection folder and client parameter folder have changed, Watchkey releases the monitored file process and enters the cyclic data processing process. Extract the names of the files added to the traffic collection folder and the client parameter folder, call the known class ExecController that has been injected into springboot, and inject the known class into the actual class via @Controller. The submethod submit() in the class undertakes the task of calling the Python jam prediction model by the known class. Through the built-in exec function of springboot, the shell command under the unified server is executed in the server. After executing the exec function, the process function is used to wait for the command execution to complete. The process is set to wait for 40ms. During this period, the error stream during the execution of the shell command is obtained through the process.getErrorStream() function, so that the later debuggers can locate the error position when problems occur. After waiting for the shell command to complete, the BufferedReader() function is combined with the readLine() function to read its content line by line, and the java built-in string stream StringBuilder is used to splice it together to form a string stream. After converting the string stream into a string, it is distributed with "\n" line separators and filled into the string array result. After knowing all the prediction result files, the last prediction data is extracted and returned to the configuration class.

[0097] After receiving the prediction result data obtained by the injection class, in the WebsocketMap list, the session corresponding to the specified client ID is extracted by extracting the key-value pair through the known client ID, and the prediction result data is returned to the client service site that has established a connection through the websocket protocol program to visualize the prediction result data.

[0098] Step (5.2) obtains parameters such as timestamp and frame buffer status from the collected original live receiving end video analysis parameters through data processing. Parse the txt file to form json data, where the obtained json format list data is integrated into stallingDict = dict() - based on the current time value as the key, splice the QOE object dictionary, TimestampsDict = dict() - based on the timestamp value as the key, splice the QOE object dictionary, unstallingListDict = dict() - using the jam start time as the key, store the stallObject object of the modified non-jamming segment, stallingListDict = dict() - using the jam start time as the key, store the stallObject object of the modified jam, unstallingTimeList = [] - store the non-jamming segment time list [[start non-jamming time, non-stuttering end time], [start non-stuttering time, non-stuttering end time], ..], stallingTimeList = []-storage card segment time list [[start stuttering time, stuttering end time], [start stuttering time, stuttering end time], ..], unstallingTimeListDetail = []-a two-dimensional list recording the specific timestamps of non-stuttering at the same current time, stallingTimeListDetail = []-a two-dimensional list recording the specific timestamps of stuttering at the same current time, buffedTimeListAll = []-a two-dimensional list recording the buffer changes of stuttering at the same current time, blacktimelist = []-a two-dimensional list recording the black screen time range. The purpose of this step is to organize the obtained front-end parameters and put them into subsequent model training and stuttering tests in a more intuitive way.

[0099] Step (5.3) For the collected pcap file, first use the extract function in flowcontainer to extract it. Modify the extract function source code, and add the RTP information field to the source code to obtain network traffic details through the tshark command, such as adding [e",'tcp.reassembled_in',#Mark the fragmentation, that is, which fragment the data packet belongs to, "-e",'http.content_length_header',#The byte length of the obtained ts segment, "-e",'http.response_for.uri',#Determine the name of the response ts, "-e",'http.file_data',#Get the corresponding list of ts names and durations (hls list),"-e",'rtcp.ssrc.fraction',#Packet loss rate displayed under RTCP data packet,"-e",'rtcp.ssrc.dlsr',#Timestamp of the source in RR or SR from the previous SR or RR,"-e",'rtp.payload']. Filter RTP video stream by extracting and analyzing reconstructed binary data. According to the information contained in the RTP data packet header, if the PT field in the RTP data segment header is in the range of 23 to 128, the RTP data stream is identified as a video data stream, and the rest of the range is an audio data stream. The purpose of this process is to obtain a pure RTP video stream from the collected original data stream. The data stream data corresponding to the freeze event can be located by obtaining the above parameters and timestamps. At the same time, the timestamp, frame buffer status and other parameters are obtained from the collected original live receiving end video analysis parameters through data processing.

[0100] Step (5.4) calculates the feature 1 required by the prediction model, and the calculation design of the frame buffer ratio difference is as follows. The plug-in designed in step 101 can collect the difference in the length of the video not played in the frame buffer of the live receiving end at the start time 1658535014.969637s and the end time 1658535014.970011s of the RTP video stream. Extract the data under the corresponding txt file, and obtain the buffingTime under the corresponding start and end timestamps in the json file. This item reflects the actual video length played by the client within the time range of the RTP video stream, that is, the buffer consumption. The program determines that under 2M bandwidth, it will take 267ms for the client plug-in to upload information to the server. Considering the normal network transmission rate of the client, we set it in the prediction process to use the length of the client's unplayed video at time t-0.2s instead of the current time t client state. The purpose of this step is to replace the parameter change characteristics of the client affected by the pre-sent traffic with the client characteristic data in a previous period of time within a certain error range according to the transmission and calculation delay obtained in step 105, so as to prevent the missing of characteristics.

[0101] Step (5.5) calculates the feature 2 required by the prediction model, and the code calculation is designed as follows. The code data is calculated by the ratio of throughput to bit rate, which represents the estimated value of how many bits of data each code element of the received video data carries. The total byte number of the RTP data segment numbered 1 is obtained by parsing the reorganized binary data of the collected data stream, which is 7589Bytes. This item indicates the actual number of video payload bytes carried by this RTP data segment. The throughput is obtained by comparing the payload byte size carried by the RTP data segment with the time range corresponding to the RTP data segment. The bit rate of the first RTP data segment depends on the actual RTP payload byte number and the client buffer cache situation. The video duration stored in the buffer within the time range of the RTP data segment is the actual video duration. By obtaining the json data in txt, the bufferAllTime field corresponds to the buffer cache video duration corresponding to the start timestamp of the RTP data segment under the start timestamp and the end timestamp. Similarly, the buffer cache video duration under the end timestamp. The ratio of the RTP data segment payload byte size to the difference in the obtained frame buffer ratio is used as the transmission video bit rate value. You can also directly use the time ratio, i.e., codetime / time, where codetime is the buffer cache video duration obtained from the txt file, and time is the duration of the RTP data segment during the actual transmission process. This step is used to calculate the characteristic parameters and codeword characteristics required for the jamming test model.

[0102] Method for constructing a representative RTP spatiotemporal relationship structure:

[0103] Step (6.1) deterministically extracts RTP data segments under different TCP streams by adding tshark to the original program code to obtain the reconstructed binary parameter information, and combines the command to obtain the rtp packet to obtain the IP data packet number range corresponding to the RTP data segment

[0104] Step (6.2) constructs a representative RTP spatiotemporal relationship structure. Deterministically extract RTP data segments under different TCP streams. Connect RTP data packets from different TCP data streams in timestamp order, and set different numbers for different streams. Connect downlink live data packets with the same stream number, and connect data packets with different stream numbers but adjacent in timestamp order. In this way, the spatiotemporal relationship structure of the RTP video stream is constructed.

[0105] Step (6.3) divides the graph structure constructed within a certain time range to ensure that the scale of its sub-graph structure is fixed. The graph structure constructed within a certain time range is divided. This case tests the prediction accuracy corresponding to each of the 5, 15, 20, 25, 30, 35, 40, and 50 unit sub-graphs. The purpose of this operation is to ensure that the scale of its sub-graph structure is fixed.

[0106] The prediction method of jamming for different application environments:

[0107] Step (7.1) selects the prediction model base model GGNN, which enhances the processing ability of GNN for input sequences. The model considers the neighborhood nodes and selectively trains the model by combining the structural information between nodes and the information of the nodes themselves. The model loss function uses an unweighted loss function to perform back propagation of the model to obtain its relevant gradient values. The model considers the unequal spatiotemporal relationship structure between nodes, conforms to the irregular characteristics of network flow, and constructs the jam prediction model in the undirected graph mode by combining the relationship between data packets through the graph structure and the gating unit.

[0108] Step (7.2) improves the model on the basis of step 305 to improve the accuracy of jamming prediction. The model is designed to predict jamming in live broadcasts in different scenarios. When updating the loss value, multiple loss values ​​are integrated to select the minimum loss value for feedback, and the feature count value corresponding to the loss value is increased. When predicting jamming events, the feature corresponding to the maximum feature count value is used for prediction. In the model output layer, different output layer architectures are designed for different numbers of features. As shown in the figure, output1 carries the output parameters when two features are used as parameters, that is, when the two features of bit rate coding and frame buffer ratio difference are used as feedback loss values; output2 carries the output parameters when a single feature is used as a parameter. By setting two output models, it is possible to select the corresponding output model at the same time after selecting features in different situations to achieve the output of adaptive jamming prediction.

[0109] Step (7.3) combines the above steps to predict the jamming event. Obtain the feature parameters of the preceding 10 seconds and the subsequent 1 second corresponding to the jamming event. Subsequently, the obtained data is sent to the designed selectable feature model to train the model. In the process of predicting jamming events, this paper uses the next segment of client live traffic after the specified time to simulate the server collecting the live stream to be forwarded to the client, and combines the client parameters 0.2s before the specified time to send it to the data processing unit to form the data to be predicted, and finally the jamming event is obtained. According to incomplete statistics, under the conditions of 200 experiments with automatic feature selection, there are 197 times when two feature sets are selected for jamming prediction, 1 time only the frame buffer occupancy difference is selected for jamming prediction, and 2 times only the bit rate coding feature is selected for jamming prediction. In the above cases, the prediction accuracy of the live broadcast freeze on the Kuaishou webpage platform can reach 98.33% under the mixed conditions of freeze and non-freeze data, the prediction accuracy of the live broadcast freeze on the Youtube platform can reach 99.17% under the mixed conditions of freeze and non-freeze data, and the prediction accuracy of the live broadcast freeze on the Twitch platform can reach 99.13% under the mixed conditions of freeze and non-freeze data. The prediction delay of the three is less than the minimum standard of the international delay specification, that is, less than 185ms. At the same time, in the above online prediction process, if the freeze event of the test sample occurs frequently, the prediction accuracy can reach 100%.

[0110] To ensure that the additional delay caused by the server-side prediction of lag meets the client's requirements in the live broadcast scenario:

[0111] Step (8.1) sets up a performance detection program, sets up a timestamp acquisition program between the client plug-in program and the server receiving program, compares the difference in timestamps between data transmission interactions with the standard delay, and determines whether the delay caused by the predicted jam is within the tolerance range of the client.

[0112] Step (8.2) sets a certain range of total data volume and periodically updates the jam prediction model. In addition, before the new prediction model is updated, the new live stream data is cyclically detected to see whether it causes jams.

[0113] Embodiment 4:

[0114] A visual monitoring and management platform.

[0115] Method for initializing historical prediction result data for the server management station:

[0116] Step (9.1) After entering the homepage, the server management site immediately sends a websocket request ["ws: / / "+dataurl+":"+port+" / wstotal / 1"] to the server. In addition to the basic request conditions, the client random ID (1) is attached to the websocket request and sent to the server. Among them, "ws" is the websocket request prefix, "dataurl" is the server URL, "port" is the server open port, "wstotal" is the request suffix that uniquely identifies this websocket connection, and "1" is the client ID. The server program uses the springboot framework. Under the full-duplex long connection of websocket, the client ID is added to the server-side websocket collection webSocketMap, and the ID transmitted by the client is used as the KEY and the session as the value. The known class ExecController injected into springboot is called. The known class is injected into the actual class via @Controller. The method in this class traverses the prediction result file in the specified folder of the server, reads its content line by line through the BufferedReader() function combined with the readLine() function, and uses the built-in java string stream StringBuilder to splice it together to form a string stream. After converting the string stream into a string, it is distributed at intervals with "\n" line separators and filled into the string array tempresult. It traverses tempresult and adds a serial number before each line of data to complete the encoding numbering.

[0117] Step (9.2) After completing the basic data processing based on step (9.1), the request of the management site is locked by the client ID, the corresponding session is extracted according to the ID in the webSocketMap to lock the websocket channel, and the historical prediction data message of the client service site is returned through the getAsyncRemote().sendText(message) in the websocket dependency built in springboot. After sending the request, the server management site continues to receive the historical prediction data evt.data obtained through step (9.1) through onmessage, and through the vue component, the attrFrom array of the historical prediction data in the data is assigned in the initialized vue component, so as to facilitate the assignment of the acquired data to the data list, and use the v-for component to render the html dynamic tag. In addition, the visualization page table title array is initialized and assigned in the data as ["serial number", "name", "running time", "precision", "recall rate", "accuracy", "comprehensive score"], the purpose of which is to better enable monitoring managers to view the prediction data. At the same time, after receiving the historical prediction data in string format, the data stream is split in the form of "\n" line separator to form the prediction data array oldDataInfo. Through the for loop, the data list is sorted in reverse order and spliced ​​into the page data list. After dynamic page rendering, the reverse historical prediction results are displayed in a table on the server administrator page.

[0118] Method for obtaining the jam prediction result data in real time from the server management station:

[0119] Step (10.1) At the same time as step (9.1), a websocket full-duplex connection ["ws: / / "+dataurl+":"+port+" / ws2 / 1"] is sent through new WebSocket (url) to obtain real-time data, where "ws" is the websocket request prefix, "dataurl" is the server URL, "port" is the server open port, "ws2" is the request suffix that uniquely identifies this websocket connection, and "1" is the client ID and carries the client random ID. After the server receives the request, it uses the random ID as the key and the session of the process as the value to store it in the server's webSocketMap.

[0120] Step (10.2) is based on step (10.1), and obtains the known class injected into springboot. In the known class, the server traffic collection folder and the client parameter folder are monitored for file changes. When files are added to the above two folders, the blocking program is jumped out, and the external Python prediction model program is called through the springboot configuration file to complete the prediction of the newly collected traffic. The prediction result is generated by the Python model and stored in the server-specified file. After completing the shell command operation, the server program reads the last line of the file through the readline function in BufferedReader(), and uses the built-in string stream of java to receive the read file content. After converting the string stream into a string, it is distributed with "\n" line separators and filled into the string array result, and at the same time, the serial number of the last line of data in the overall historical prediction data is calculated, and it is spliced ​​into the string array in a comma-separated manner, and the prediction result is returned to the actual websocket server program through the injection class.

[0121] Step (10.3) is based on step (10.2). When the client ID is known, the session channel is uniquely obtained according to the client ID in WebsocketMap, and the latest real-time prediction data is returned to the client service site. After the client receives the data, it receives the data in the disappearance event, extracts its data field, and assigns it to the Javascript 6 variable. According to the data format specification agreed with the server, the data is split in a comma-separated manner to convert the string data obtained from the server into array data. The basic array of the original dynamically rendered html tag is extracted and initialized into a new variable, the new array is initialized, and the latest real-time data is returned to the client service site.

[0122] [34,1658544419465.pcap,0.6548786163330078,0.9537,0.9429,0.9447,0.9429] is used as the first prediction data of the new array. The original prediction data is traversed by the for loop, and data processing is performed at the same time. The prediction result data less than 100% returned by the backend to the frontend is converted into percentage data. At the same time, the data is spliced ​​into the new array in order to obtain the historical prediction result array with the real-time new jam prediction data. At this time, the tag in the html file places the latest real-time prediction data at the top of the visual web page table after the v-for component, and continuously receives the implementation prediction data from the server and displays it in the administrator web page, where the table header is: ["Serial number", "Name", "Run time", "Precision", "Recall", "Accuracy", "Comprehensive score"], and the corresponding data is the jam prediction data of the same client at different timestamps.

[0123] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. A method for predicting freeze in webpage live broadcast under RTP, the method comprising: The plug-in program, the traffic collector, and the periodically updated adaptive live broadcast freeze prediction model are characterized by the following: Step (1) The plug-in program is installed on the user's browser after the user agrees to obtain the information, and a regular expression is set to determine the domain name corresponding to the web page, so as to obtain the parameter information of the specified platform in a deterministic manner on different web live broadcast platforms; Step (2) collecting web live broadcast client playback parameters, obtaining target user buffer information and live broadcast status; Step (3) transmits the client playback parameters acquired in steps (1) and (2) back to the server in a cross-domain environment, so that the server acquires the client playback parameters in real time, and transmits the acquired client playback parameters in the form of a text file to a folder designated by the server; Step (4) The traffic collector collects the client traffic that the server is about to send within a certain period, and collects and stores it in a specified folder on the server within a certain period of time. Step (5) The server monitors the folder status in steps (3) and (4). When files are added to both the traffic collection folder and the client playback parameter folder, the server starts a data processing program to process the collected traffic information and client playback parameters, and obtains two predicted jamming features, namely, symbol code system information and client frame buffer difference. Step (6) the server extracts the RTP data segments from the collected traffic data and constructs a spatiotemporal relationship structure; Step (7) combines the processed data features in step (6) and sends them to a periodically updated adaptive live streaming jam prediction model, wherein the live streaming jam prediction model is used to determine whether there is jam in the live streaming stream to be sent by the server; Step (8) in a live broadcast scenario, ensures through testing that the additional delay overhead caused by the server's predicted jamming meets the client's requirements; Step (9): Based on steps (5) and (6), the server management site enters the management page for initialization and obtains the prediction result data of the server; Step (10) the server management site monitors the server prediction result data in real time and displays the prediction result data in a visual website format; Among them, in step (5), data processing is performed on the collected traffic information and client playback parameters to obtain predicted jamming features, which includes: Step (5.1) monitors the server traffic collection folder and the client playback parameter folder, and when files are added to the above two folders, starts the data processing program; Step (5.2) processes the parameters collected by the client to locate the duration of the unplayed video in the target user buffer at the specified timestamp; Step (5.3) processes the collected network traffic pcap packets, extracts the RTP header information, and filters the pure video stream data; Step (5.4) obtains the total load within a certain timestamp range based on the extracted video traffic information, and obtains the throughput by dividing the total load by the timestamp information; Step (5.5) determines the video bit rate information within a certain timestamp range by combining the network video stream length information and the acquired client playback parameters; In step (6), an RTP time-space relationship structure is constructed, which includes: Step (6.1) deterministically extracts RTP data segments under different TCP network flows by adding tshark to the original program code to obtain the reconstructed binary parameter information, and combines the RTP packet acquisition command to obtain the IP data packet number range corresponding to the RTP data segment; Step (6.2) connects the RTP data segments from the same TCP network stream, and further connects the RTP data segments from different TCP network streams to construct a graph structure; Step (6.3) splits the graph structure constructed within a certain timestamp range to ensure that the size of its sub-graph structure is fixed; In step (7), the prediction of jamming is performed for different application environments, which includes: Step (7.1) selects a live broadcast freeze prediction model, considers the graph structure relationship between RTP data packets, and selects different weight relationships between data packets; Step (7.2) considers the differences in loss values ​​brought by different features in multiple scenarios, and performs multi-loss integrated feedback to make features adapt to feature training; Step (7.3) selects different features for jamming prediction according to different application scenarios, so that jamming prediction can be adaptively performed according to different client network environments and different client hardware device processing capabilities.

2. The method for predicting freeze of live webpage broadcast under RTP according to claim 1, characterized in that: In step (1), video tags are obtained for different live broadcast websites, which includes: Step (1.1) The web page plug-in obtains the domain name information of the designated website, sets the website regular expression in the plug-in configuration file, sets different web page filtering rules for different service providers, and configures the function file application URL; Step (1.2) sets different video tag acquisition methods for live streaming website platforms from different sources, including based on tag class name, tag ID and tag itself.

3. The method for predicting freeze of live webpage broadcast under RTP according to claim 1, characterized in that: In step (2), the client buffer status information and live broadcast status are determined at the millisecond level, which includes: Step (2.1) obtains the client's current playback timestamp starting from the first frame of live broadcast for the acquired live video tag; Step (2.2) obtains the client live video resolution length and video resolution width; Step (2.3) obtains the total video buffering time of the client live video; Step (2.4) obtains the duration of the live video that has not been played by the client, and the duration of the video in the buffer that has not been decoded by the client.

4. The method for predicting freeze of live webpage broadcast under RTP according to claim 1, characterized in that: In step (3), the obtained client playback parameters are transmitted back to the server, so that the server obtains the client playback parameters in real time, which includes: Step (3.1) obtains the current timestamp on the client before and after uploading the data, wherein the acquisition of the timestamp after uploading the data is triggered by receiving the server response, and the two-way transmission time of this data transmission is obtained by the difference between the timestamps before and after the data transmission, and the transmission time required for the client to upload to the server under different bandwidths is obtained after the average division; Step (3.2) organizes the obtained information into a data format that can be parsed by the server, and sets a timer in the plug-in program to periodically transmit the obtained client playback parameters back to the server; In step (3.3), under the actual requirement of cross-domain uploading, the client sets the cross-domain request identifier and receives data at the same time. The server also sets the cross-domain interaction identifier and transmits the parameters obtained by the client to the server.

5. The method for predicting freeze of live webpage broadcast under RTP according to claim 1, characterized in that: In step (4), the server collects the client traffic to be sent within a certain period, which includes: Step (4.1) opens a designated port to listen to client requests and receives client data information according to the data transmission specification negotiated with the client; Step (4.2) collects network traffic within a certain timestamp range after receiving the client's request to collect parameters; Step (4.3) forms the collected network traffic into a pcap file and stores it in the file directory specified by the server.

6. The method for predicting freeze of live webpage broadcast under RTP according to claim 1, characterized in that: In step (8), ensuring that the additional delay overhead caused by the server's predicted jamming in the live broadcast scenario meets the client's requirements includes: Step (8.1) sets up a performance detection program, sets up a timestamp acquisition program between the client plug-in program and the server receiving program, compares the difference in timestamps between data transmission interactions with the standard delay, and determines whether the delay caused by the predicted jam is within the tolerance range of the client. Step (8.2) sets a certain data volume range and periodically updates the live broadcast jamming prediction model. In addition, before the new live broadcast jamming prediction model is updated, the new live broadcast stream data is cyclically detected to observe whether it causes jamming.

7. The method for predicting freeze of live webpage broadcast under RTP according to claim 1, characterized in that: In step (9), the server management station initializes the historical prediction result data, which includes: After the initialization webpage loading in step (9.1) is completed, the historical prediction result data is obtained through the full-duplex websocket protocol. After the server receives the websocket request carrying the request client ID, it adds the requested user to the websocketMap collection, traverses the file content of the server storing the prediction result data, and performs corresponding data processing on the content before returning it to the administrator front-end interface. Step (9.2) the server management station interface sorts the received data in reverse order and displays the latest traffic collection file prediction result data in a table at the top; In step (10), the server management site obtains prediction result data in real time, which includes: Step (10.1) uses websocket to establish a TCP long connection, independent of initializing websocket, and opens a full-duplex connection in the real-time information acquisition module. Step (10.2) uses the spring boot injection mechanism to obtain the method class that has been injected into the container in the websocket configuration file. After continuously detecting the server folder status, run the live broadcast jam prediction model, obtain the prediction result data, and transmit it to the front end through the TCP channel. Step (10.3) The front end receives the prediction result data transmitted by the back-end medium under the websocket connection, and rearranges the original prediction result data list, placing the latest real-time prediction result data at the top of the list, which contains the serial number, traffic collection file name, jam prediction duration, precision, recall rate, accuracy and F1 measurement score. The above information is arranged in reverse order from top to bottom in the form of a table.

Images