Collaborative live broadcast interactive data processing method, system and device, and medium

Abstract

The invention provides a collaborative live broadcast interaction data processing method, system and device, and a medium. The method comprises the following steps: collecting anchor side data and platform side data; and the artificial intelligence prediction model predicts a bandwidth state prediction sequence, a packet loss rate prediction sequence and an interaction delay prediction sequence of each anchor in a first preset time based on the multi-dimensional time sequence. And dynamically optimizing the live broadcast data stream of each anchor according to a live broadcast data stream adaptive strategy based on the bandwidth state prediction sequence, the packet loss rate prediction sequence and the interaction delay prediction sequence. And constructing an interaction experience function based on the interaction delay prediction sequence, carrying out global evaluation on the interaction delay of the multiple anchors, and dynamically adjusting the interaction between the anchors and the audiences according to a collaborative live broadcast interaction adjustment strategy. And determining synchronization reference time based on the video frame timestamps of all the anchors, and performing corresponding video synchronization processing according to the time difference between the video frame timestamp of each anchor and the synchronization reference time.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a collaborative live interactive data processing method, system, device and medium. Background Art

[0002] With the development of multi-anchor collaborative live broadcasting, more and more live broadcasting scenarios require multiple anchors to interact with each other at the same time and jointly output content. However, due to factors such as different geographical distribution, inconsistent network environment, and complex audience interaction behaviors, some anchors often experience problems such as freezes, high latency, or asynchronous playback, which seriously affects the overall collaborative live broadcast audience viewing experience.

[0003] In the process of collaborative live broadcast by multiple anchors, existing technologies are often unable to accurately coordinate the video stream quality of each anchor, and lack effective interactive optimization and video timing synchronization mechanisms, resulting in obvious playback differences between different anchors. This problem is particularly prominent when bandwidth fluctuates or audience interaction is frequent. Summary of the invention

[0004] The present application provides a collaborative live interactive data processing method, system, device and medium to solve the problems of the prior art.

[0005] In a first aspect, the present application provides a collaborative live interactive data processing method, comprising: Collaborative live broadcast data collection, collecting anchor-side data and platform-side data, the anchor-side data includes bandwidth, interaction delay, packet loss rate, jitter and video frame timestamp, the platform-side data includes the number of anchors, interaction frequency and number of viewers, and generating multi-dimensional time series based on the anchor-side data and platform-side data; Data prediction based on an artificial intelligence prediction model, the artificial intelligence prediction model predicts a bandwidth status prediction sequence, a packet loss rate prediction sequence, and an interaction delay prediction sequence for each anchor within a first preset time based on the multi-dimensional time series; Dynamic optimization of live data streams, based on the bandwidth status prediction sequence, packet loss rate prediction sequence and interaction delay prediction sequence, dynamically optimizing the live data stream of each anchor according to the live data stream adaptive strategy; Collaborative live broadcast interaction optimization, building an interactive experience function based on the interactive delay prediction sequence, globally evaluating the interaction delay of multiple anchors, and dynamically adjusting the interaction between anchors and viewers according to the collaborative live broadcast interaction adjustment strategy; Multi-host video timing synchronization determines the synchronization reference time based on the video frame timestamps of all the hosts, and performs corresponding video synchronization processing according to the time difference between the video frame timestamp of each host and the synchronization reference time.

[0006] In a possible design, the live data stream adaptive strategy includes a dynamic bit rate adjustment strategy and a resolution and frame rate adjustment strategy; The dynamic bitrate adjustment strategy is to calculate and adjust the dynamic bitrate of each anchor in real time according to the bandwidth fluctuation prediction sequence, the packet loss rate prediction sequence and the interaction delay prediction sequence; The resolution and frame rate adjustment strategy is to adjust the resolution and frame rate of each anchor in real time according to the dynamic bit rate, including: When the dynamic bit rate is greater than or equal to 6 Mbps, the resolution is 4K and the frame rate is 60 FPS; When the dynamic bit rate is greater than or equal to 4 Mbps and less than 6 Mbps, the resolution is 2K and the frame rate is 60 FPS; When the dynamic bit rate is greater than or equal to 2.5Mbps and less than 4Mbps, the resolution is 1080P and the frame rate is 60FPS; When the dynamic bit rate is greater than or equal to 1.5Mbps and less than 2.5Mbps, the resolution is 1080P and the frame rate is 30FPS; When the dynamic bit rate is greater than or equal to 1.0 Mbps and less than 1.5 Mbps, the resolution is 720P and the frame rate is 30 FPS; When the dynamic bit rate is greater than or equal to 0.6 Mbps and less than 1.0 Mbps, the resolution is 480P and the frame rate is 30 FPS; When the dynamic bit rate is greater than or equal to 0.4 Mbps and less than 0.6 Mbps, the resolution is 360P and the frame rate is 30 FPS.

[0007] In a possible design, the interactive experience function is a triggering basis for the collaborative live broadcast interactive adjustment strategy; When the interactive experience function is greater than the interactive optimization trigger threshold, executing the collaborative live broadcast interactive adjustment strategy; The interaction optimization trigger threshold is used to determine whether the interaction experience function is within an acceptable range.

[0008] In a possible design, the collaborative live broadcast interaction adjustment strategy includes: Calculate the target interaction delay threshold of each anchor; Comparing the interaction delay prediction sequence of each of the anchors with the corresponding target interaction delay threshold; When the predicted interaction delay sequence of the anchor is greater than the corresponding target interaction delay threshold, the anchor is determined to be an interaction optimization object; Interaction optimization is performed on the anchors determined to be the objects of interaction optimization, and the interaction optimization includes reducing the frequency of sending barrages, reducing the frequency of comment interactions, simplifying reward animation effects, and merging gift display effects.

[0009] In one possible design, the target interaction delay threshold is a dynamic threshold.

[0010] In a possible design, the multi-host video timing synchronization includes: Based on the video frame timestamps collected within the second preset time, generating a video frame timestamp sequence for each anchor; According to the video frame timestamp sequences of all the anchors, a maximum value strategy is adopted to determine the synchronization reference time; Calculate the time difference between the video frame timestamp of each of the anchors and the synchronization reference time; According to the time difference, inserting a frame or dropping a frame is performed on each of the anchor's video frames, including: When the time difference is greater than 0, performing a frame insertion operation on the video frame of the anchor; When the time difference is less than 0, a frame dropping operation is performed on the video frame of the anchor; When the time difference is 0, the video frame of the host is not processed.

[0011] In one possible design, the interpolation operation is an interpolation algorithm based on an artificial intelligence prediction model, which predicts and generates intermediate transition frames by inputting adjacent video frames into the artificial intelligence prediction model.

[0012] In a second aspect, the present application provides a collaborative live interactive data processing system, comprising: A collaborative live broadcast data collection module collects data from the anchor side and the platform side. The anchor side data includes bandwidth, interaction delay, packet loss rate, jitter and video frame timestamp. The platform side data includes the number of anchors, interaction frequency and number of viewers. A multi-dimensional time series is generated based on the anchor side data and the platform side data. A data prediction module based on an artificial intelligence prediction model, wherein the artificial intelligence prediction model predicts a bandwidth status prediction sequence, a packet loss rate prediction sequence, and an interaction delay prediction sequence for each anchor within a first preset time based on the multi-dimensional time series; A live data stream dynamic optimization module dynamically optimizes the live data stream of each anchor according to a live data stream adaptive strategy based on the bandwidth status prediction sequence, the packet loss rate prediction sequence and the interaction delay prediction sequence; A collaborative live broadcast interaction optimization module constructs an interactive experience function based on the interactive delay prediction sequence, performs a global evaluation of the multi-host interaction delay, and dynamically adjusts the interaction between the host and the audience according to the collaborative live broadcast interaction adjustment strategy; The multi-anchor video timing synchronization module determines the synchronization reference time based on the video frame timestamps of all the anchors, and performs corresponding video synchronization processing according to the time difference between the video frame timestamp of each anchor and the synchronization reference time.

[0013] In a third aspect, the present application provides an electronic device, including: processor; and, A memory, configured to store executable instructions of the processor; The processor is configured to perform any possible method described in the first aspect by executing the executable instructions.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any possible method described in the first aspect.

[0015] The collaborative live interactive data processing method, system, device and medium provided in this application, based on the collected anchor side data and platform side data, intelligently predicts the live data stream through an artificial intelligence prediction model, and realizes dynamic optimization of the live data stream and collaborative live interactive optimization based on the prediction results, effectively reducing the interactive delay and freeze problems in the collaborative live broadcast process of multiple anchors. At the same time, through the timing synchronization of multiple anchor videos, the precise alignment of the audio and video content of multiple anchors is achieved, further improving the audience's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] Figure 1 It is a flowchart of a collaborative live interactive data processing method according to an exemplary embodiment of the present application; Figure 2 It is a flowchart of a collaborative live broadcast interaction adjustment strategy according to an exemplary embodiment of the present application; Figure 3 This is a schematic diagram of a process for synchronizing the timing of multi-host videos according to an exemplary embodiment of the present application; Figure 4 is a structural diagram of a collaborative live interactive data processing system according to an exemplary embodiment of the present application; Figure 5 It is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application.

[0018] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0019] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0020] Figure 1 FIG. 1 is a flow chart of a collaborative live interactive data processing method according to an exemplary embodiment of the present application. Figure 1 As shown, the method provided in this embodiment includes: Step S101: Collaborative live broadcast data collection, collecting anchor side data and platform side data. The anchor side data includes bandwidth, interaction delay, packet loss rate, jitter and video frame timestamp. The platform side data includes the number of anchors, interaction frequency and number of viewers. A multi-dimensional time series is generated based on the anchor side data and the platform side data.

[0021] In this step, a corresponding multidimensional time series can be generated based on the collected anchor-side data and platform-side data. The following example illustrates the corresponding relationship between the anchor-side data, platform-side data and the multidimensional time series.

[0022] For example, the collected data on the anchor side and the platform side are shown in the following table: Table 1 Data from the anchor side and the platform side

[0023] The resulting multidimensional time series looks like this:

[0024] Where X is a multidimensional time series, is the feature vector of the ith anchor at time t, for:

[0025] Corresponding to the input feature vector in Table 1 for:

[0026] The multidimensional time series X corresponding to Table 1 is:

[0027] Step S102: Data prediction based on an artificial intelligence prediction model, where the artificial intelligence prediction model is based on a multi-dimensional time series and predicts a bandwidth status prediction sequence, a packet loss rate prediction sequence, and an interaction delay prediction sequence for each anchor within a first preset time.

[0028] In this step, the artificial intelligence prediction model is built based on the long short-term memory network (LSTM) and trained on the historical collaborative live broadcast data set using supervised learning. The collaborative live broadcast data set used is a multi-dimensional time series composed of anchor-side data and platform-side data. The anchor-side data includes bandwidth, interaction delay, packet loss rate, and jitter; the platform-side data includes the number of anchors, interaction frequency, and number of viewers.

[0029] The data types and structures input in the model training phase and the prediction phase are consistent, and are used to model the evolution of each anchor's network status over time. The artificial intelligence prediction model predicts the anchor's bandwidth status, interaction delay, and packet loss rate within the first preset time by learning the time-dependent characteristics of the multi-dimensional time series, and outputs the corresponding prediction sequence results. The role of each collected data in model training and prediction is as follows: Bandwidth: Collect continuous bandwidth values ​​within a fixed window time T to construct the historical bandwidth time series of each anchor. The historical bandwidth time series is one of the input feature dimensions of the artificial intelligence prediction model. It is used by the artificial intelligence prediction model to learn the dynamic evolution of bandwidth status over time, and then predict the bandwidth status of each anchor within the first preset time t, and output the bandwidth status prediction sequence of each anchor. The prediction results are used to adjust the dynamic bit rate, resolution and frame rate on the anchor side in real time.

[0030] Interaction delay: Collect continuous interaction delay values ​​within a fixed window time T. The interaction delay is the return response delay from the host side to the platform side, and construct a historical interaction delay sequence for each host. The historical interaction delay sequence is one of the input feature dimensions of the artificial intelligence prediction model, which is used for the artificial intelligence prediction model to learn the fluctuation characteristics of network transmission delay, and then predict the interaction delay state of each host within the first preset time t, and output the interaction delay prediction sequence of each host. The prediction results are used to adjust the dynamic bit rate, resolution and frame rate of the host side in real time, and as a collaborative live interaction adjustment strategy, to collaboratively adjust the interaction between each host and the audience.

[0031] Packet loss rate: collect continuous packet loss rates within a fixed window time T, and construct a historical packet loss rate time series for each anchor. The packet loss rate is used to characterize the proportion of data packets lost during network transmission on the anchor side, and is an important indicator for measuring network stability. The historical packet loss rate time series is one of the input feature dimensions of the artificial intelligence prediction model. It is used for the artificial intelligence prediction model to learn the evolution characteristics of network transmission anomalies and packet loss fluctuations, and then predict the packet loss status of each anchor within the first preset time t, and output the packet loss rate prediction sequence of each anchor. The prediction results are used to adjust the dynamic bit rate, resolution and frame rate on the anchor side in real time.

[0032] Jitter, collect continuous network jitter values ​​within a fixed time window T, and construct a historical jitter time series for each anchor. Jitter refers to the degree of fluctuation in the arrival time between consecutive data packets, and is an important indirect indicator of network stability. As one of the input features of the artificial intelligence prediction model, the historical jitter time series plays an auxiliary modeling role in the generation of bandwidth status prediction series, packet loss rate prediction series, and interactive delay prediction series during the training and prediction process of the artificial intelligence prediction model, which helps to improve the overall prediction accuracy and stability. For example, when the network is in the early stage of congestion, the bandwidth and packet loss rate have not changed significantly, and the jitter usually increases first. The model can use the historical jitter time series to capture potential network fluctuation trends, thereby adjusting the output prediction results in advance and improving the timeliness and accuracy of the prediction.

[0033] The number of anchors, collects the anchor number statistics within a fixed time window T, and constructs a time series of the historical number of anchors. The number of anchors is obtained by real-time statistics on the platform side, and is used to represent the total number of anchors participating in the interaction in the current collaborative live broadcast scenario. The time series of the historical number of anchors is not used as an input feature dimension in the training and prediction process of the artificial intelligence prediction model, but as a control parameter to guide the artificial intelligence prediction model to determine the number of anchors that need to perform the current prediction task, that is, to control the number of model instantiations.

[0034] Interaction frequency, collects the number of interactive messages per unit time of each anchor within a fixed time window T, and constructs a historical interaction frequency time series for each anchor. The interaction frequency reflects the interactive activity between the anchor and the audience, and is a key factor affecting the load of the interactive channel. As one of the input feature dimensions of the artificial intelligence prediction model, the historical interaction frequency time series plays an auxiliary modeling role in the process of generating the interaction delay prediction sequence, thereby improving the accuracy of the interaction delay prediction. When the interaction frequency increases, the transmission load on the anchor side increases accordingly, which may cause fluctuations in the return path response. The artificial intelligence prediction model can use this to identify potential interaction delay surge trends in advance.

[0035] The number of viewers collects the audience access statistics of each anchor's live broadcast room within a fixed time window T, and constructs a time series of the historical number of viewers for each anchor. As one of the measurement indicators of the system transmission load, the number of viewers will affect the stability of the network transmission link. As one of the input feature dimensions of the artificial intelligence prediction model, the time series of the historical number of viewers plays an auxiliary modeling role in the process of generating bandwidth status prediction sequences and packet loss rate prediction sequences, thereby improving the accuracy of bandwidth status prediction sequences and packet loss rate prediction sequences. Especially in high-concurrency live broadcast scenarios, the artificial intelligence prediction model can use the characteristics of the number of viewers to perceive the trend of network congestion, thereby more accurately modeling the risk of packet loss or bandwidth fluctuations.

[0036] The fixed window time T and the predicted first preset time t are configurable preset times. The fixed window time T ranges from 15 seconds to 60 seconds, and the predicted first preset time t ranges from 5 seconds to 10 seconds. i is an integer between 1 and N, and N is the number of anchors.

[0037] The artificial intelligence prediction model is trained through collaborative live broadcast data sets, with the optimization goal of minimizing the error between the predicted value and the true value. During the training process, the mean square error (MSE) of multivariate regression is used as the loss function to guide the model to continuously extract time-dependent features in the input sequence, thereby improving the accuracy of predicting bandwidth status, interaction delay and packet loss rate within the first preset time t.

[0038] The input of the artificial intelligence prediction model is a multidimensional time series of i anchors in n consecutive time steps, and the multidimensional time series is as follows:

[0039] Wherein, i is an integer between 1 and N, N is the number of anchors, and t is the predicted first preset time.

[0040] is the time series of the i-th anchor in the prediction of the first preset time t, The expression is:

[0041] in, is the historical bandwidth time series of the i-th anchor in the prediction of the first preset time t, is the historical interaction delay sequence of the i-th anchor within the predicted first preset time t, is the time series of the historical packet loss rate of the i-th anchor in the prediction of the first preset time t, is the historical jitter time series of the i-th anchor in the prediction of the first preset time t, is the time series of the number of historical anchors within the first preset time t, is the historical interaction frequency time series of the ith anchor within the predicted first preset time t, is the time series of the number of historical viewers of the i-th anchor within the predicted first preset time t, i is an integer between 1 and N, and N is the number of anchors.

[0042] The output of the artificial intelligence prediction model is the prediction sequence of i anchors, where the prediction sequence of the i-th anchor is:

[0043] in, is the bandwidth status prediction sequence of the i-th anchor; is the packet loss rate prediction sequence of the i-th anchor; is the interaction delay prediction sequence of the i-th anchor, i is an integer between 1 and N, and N is the number of anchors.

[0044] The bandwidth status prediction sequence, packet loss rate prediction sequence and interaction delay prediction sequence of the i-th anchor are:

[0045] in, is the bandwidth prediction value of the i-th anchor at time t+1; is the predicted value of packet loss rate of the i-th anchor at time t+1; is the predicted value of the interaction delay of the i-th anchor at time t+1; t+n is the first preset time.

[0046] Step S103: Dynamically optimize the live data stream, based on the bandwidth status prediction sequence, the packet loss rate prediction sequence and the interactive delay prediction sequence, and dynamically optimize the live data stream of each anchor according to the live data stream adaptive strategy.

[0047] In this step, the live data stream adaptive strategy includes a dynamic bit rate adjustment strategy and a resolution and frame rate adjustment strategy; The dynamic bitrate adjustment strategy is to calculate and adjust the dynamic bitrate of each anchor in real time according to the bandwidth fluctuation prediction sequence, packet loss rate prediction sequence and interaction delay prediction sequence. The calculation formula of the dynamic bitrate is:

[0048] in, is the bandwidth fluctuation prediction sequence of the i-th anchor, α is the weight coefficient of bandwidth, β is the weight coefficient of packet loss rate, is the packet loss rate prediction sequence of the i-th anchor, is the weight coefficient of interaction delay, is the interaction delay prediction sequence of the i-th anchor.

[0049] In this embodiment, the weight coefficient Can be preset, the three weight coefficients must meet the constraints , to ensure the numerical stability and controllability of dynamic bit rate calculation.

[0050] The resolution and frame rate adjustment strategy is to adjust the resolution and frame rate of each anchor in real time according to the dynamic bit rate, including: When the dynamic bit rate is greater than or equal to 6Mbps, the resolution is 4K and the frame rate is 60FPS; When the dynamic bit rate is greater than or equal to 4Mbps and less than 6Mbps, the resolution is 2K and the frame rate is 60FPS; When the dynamic bit rate is greater than or equal to 2.5Mbps and less than 4Mbps, the resolution is 1080P and the frame rate is 60FPS; When the dynamic bit rate is greater than or equal to 1.5Mbps and less than 2.5Mbps, the resolution is 1080P and the frame rate is 30FPS; When the dynamic bit rate is greater than or equal to 1.0Mbps and less than 1.5Mbps, the resolution is 720P and the frame rate is 30FPS; When the dynamic bit rate is greater than or equal to 0.6Mbps and less than 1.0Mbps, the resolution is 480P and the frame rate is 30FPS; When the dynamic bit rate is greater than or equal to 0.4Mbps and less than 0.6Mbps, the resolution is 360P and the frame rate is 30FPS.

[0051] Step S104: Collaborative live broadcast interaction optimization, constructing an interactive experience function based on the interactive delay prediction sequence, performing a global evaluation of the multi-host interaction delay, and dynamically adjusting the interaction between the host and the audience according to the collaborative live broadcast interaction adjustment strategy.

[0052] In this step, first, an interactive experience function is constructed based on the interactive delay prediction sequence. The interactive experience function is the triggering basis for the collaborative live broadcast interactive adjustment strategy, which is used to globally evaluate the interactive quality status and optimize the interaction of all anchors based on the evaluation results to improve the interactive stability and consistency during the collaborative live broadcast process.

[0053] When the interactive experience function is greater than the interactive optimization trigger threshold, the collaborative live broadcast interactive adjustment strategy is executed.

[0054] The calculation formula of the interactive experience function is:

[0055] in, is the penalty weight coefficient, which is used to adjust the system's sensitivity to delay fluctuations; The interaction delay prediction sequence for the i-th anchor; is the global interaction delay threshold.

[0056] In this embodiment, The larger the value, the more sensitive it is to the difference in interaction delay between anchors, and the more likely it is to trigger the collaborative live interaction adjustment strategy; The smaller the value, the more tolerant it is to the interaction delay differences between anchors. The collaborative live broadcast interaction adjustment strategy is triggered only when the interaction delay differences are large. The value can be preset and the range is between 0.5-2.0.

[0057] The global interaction delay threshold calculation formula is:

[0058] in, The mean of the interaction delay prediction series of all anchors; is the adjustment coefficient, which is used to control the system's tolerance to delay jitter; The standard deviation of the interaction delay prediction series for all streamers.

[0059] In this embodiment, The larger the value, the more tolerant the interaction delay differences between anchors are. This is suitable for scenarios with a high tolerance for interaction delay. The smaller the value, the more sensitive it is to the difference in interaction delay, and the easier it is to trigger the collaborative live interaction adjustment strategy. The value can be preset and the range is between 0.1-1.0.

[0060] The interactive experience function QoE is used to measure the overall deviation of the current multi-host interaction delay. The higher the interactive experience function QoE value, the greater the interactive delay between hosts and the worse the interactive experience. In order to globally determine whether each host needs to start the collaborative live interaction adjustment strategy, it is necessary to set the interactive optimization trigger threshold to determine whether the interactive experience function QoE is within an acceptable range. The calculation formula for the interactive optimization trigger threshold is:

[0061] in, The number of anchors; The maximum delay deviation tolerance that a single streamer can tolerate.

[0062] In this embodiment, It is used to determine whether the host's interaction delay exceeds the acceptable range, thereby deciding whether to perform interaction optimization for the host. The value can be preset and the value range is 100ms-500ms.

[0063] Step S105: Multi-host video timing synchronization: determine the synchronization reference time based on the video frame timestamps of all hosts, and perform corresponding video synchronization processing according to the time difference between each host's video frame timestamp and the synchronization reference time.

[0064] Figure 2 FIG. 1 is a flow chart of a collaborative live interactive adjustment strategy according to an exemplary embodiment of the present application. Figure 2 As shown, the collaborative live broadcast interaction adjustment strategy provided in this embodiment includes: Step S201: Calculate the target interaction delay threshold of each anchor.

[0065] In this step, the target interaction delay threshold is a dynamic threshold, and the calculation formula of the target interaction delay threshold is:

[0066] in, is the mean of the interaction delay prediction sequence of the i-th anchor; is the adjustment coefficient; is the standard deviation of the interaction delay prediction series of the i-th anchor.

[0067] In this embodiment, Used to calculate the target interaction delay threshold for each anchor to control the anchor's tolerance for interaction delay differences. The value can be preset and the range is between 0.1-1.0.

[0068] Step S202: Compare the interaction delay prediction sequence of each anchor with the corresponding target interaction delay threshold.

[0069] Step S203: When the predicted interaction delay sequence of the anchor is greater than the corresponding target interaction delay threshold, the anchor is determined to be an interaction optimization object.

[0070] Step S204: Interaction optimization is performed on the anchor determined to be an interaction optimization target, and the interaction optimization includes reducing the frequency of sending barrages, reducing the frequency of comment interactions, simplifying reward animation effects, and merging gift display effects.

[0071] Figure 3 FIG. 1 is a flow chart of multi-host video timing synchronization according to an exemplary embodiment of the present application. Figure 3 As shown, the method provided in this embodiment includes: Step S301: Generate a video frame timestamp sequence for each anchor based on the video frame timestamps collected within a second preset time.

[0072] In this step, it is necessary to count the video frame timestamp sequence of each anchor. For example, the frame rate of the i-th anchor is 30FPS, and the video frame timestamp sequence within 1 second is:

[0073] in, The timestamp of the first frame of the i-th anchor in that second; is the 30th frame timestamp of the i-th anchor in that second. If the second preset time is 2 seconds and the frame rate of the i-th anchor is 30FPS, then the video frame timestamp sequence of the anchor is Contains the values ​​of the 60 video frame timestamps.

[0074] Step S302: Determine the synchronization reference time using the maximum value strategy according to the video frame timestamp sequences of all hosts.

[0075] In this step, the maximum value strategy is to select the largest video frame timestamp from the video frame timestamp sequence of all anchors as the synchronization reference time, and perform frame insertion or frame drop operations on the video frames of other anchors based on the synchronization reference time. The maximum value strategy can ensure that the audio and video content of all anchors will not be played ahead of time, thereby maintaining synchronization consistency. The calculation formula of the maximum value strategy is:

[0076] Where n is the number of anchors.

[0077] Step S303: Calculate the time difference between the video frame timestamp of each anchor and the synchronization reference time.

[0078] In this step, the time difference is calculated as:

[0079] In step S304, a frame insertion operation or a frame drop operation is performed on each anchor's video frame according to the time difference.

[0080] include: When the time difference is greater than 0, it means that the anchor's video frame is delayed, and the anchor's video frame is interpolated; When the time difference is less than 0, it means that the anchor's video frame is ahead, and the anchor's video frame is dropped; When the time difference is 0, it means that the video frames of the anchor have been aligned, and the video frames of the anchor are not processed.

[0081] In this step, the interpolation operation is an interpolation algorithm based on an artificial intelligence prediction model, which predicts and generates intermediate transition frames by inputting adjacent video frames into the artificial intelligence prediction model. The interpolation algorithm uses the artificial intelligence prediction model to learn the motion characteristics, image content changes, and time relationships between video frames, and generates one or more interpolation frames to compensate for the differences in the playback time of the host video stream.

[0082] In this embodiment, considering that the time difference of video frames between anchors during the collaborative live broadcast of multiple anchors may exceed the single frame interval, in order to achieve more precise video timing alignment, the artificial intelligence prediction model used for frame insertion operation needs to support the generation of multiple intermediate transition frames. The input of the artificial intelligence prediction model is the adjacent frame images at any two time points, and according to the set time interpolation factor, it outputs any number of intermediate frames to make up for the frame time difference.

[0083] The artificial intelligence prediction model used for interpolation operations can adopt an interpolation neural network structure based on time control guidance, such as the RIFE (Real-Time Intermediate Flow Estimation) framework. This type of model does not rely on complex optical flow modeling, but completes inter-frame motion relationship estimation and multi-frame fusion prediction based on a lightweight convolution structure, which is highly efficient and deployable.

[0084] The AI ​​prediction model for interpolation can be trained using the Vimeo90K dataset. The training samples usually consist of three consecutive frames, with the previous and next frames as input and the middle frame as a supervision label. During the training process, the AI ​​prediction model aims to minimize the pixel error between the predicted middle frame and the real middle frame. The loss function uses L1 loss to optimize the model's interpolation accuracy and image restoration capabilities.

[0085] The AI ​​prediction model used for frame insertion controls the insertion of one frame per unit time interval. The input of the artificial intelligence prediction model is the current frame and the previous frame of the anchor. Calculate the number of intermediate frames to be inserted and set the corresponding time interpolation factor. Specifically, first preset a time step threshold Y, for example, the Y value is 33 milliseconds, and the time difference is Divide by the time step threshold Y, and the quotient is the number of intermediate frames to be inserted.

[0086] To insert multiple intermediate transition frames, divide the time position between the current frame and the previous frame at equal intervals, and set multiple time interpolation factors. For example, if three frames need to be inserted, 0.25, 0.5, and 0.75 can be set as the time interpolation factors, indicating that the inserted frames are at the 1 / 4, 1 / 2, and 3 / 4 positions between the current frame and the previous frame on the time axis.

[0087] The artificial intelligence prediction model used for interpolation operations predicts and generates each intermediate transition frame in sequence according to the selected input frame image and interpolation factor, which is used to make up for the time difference of the anchor and achieve timing alignment of multiple anchor video frames.

[0088] In this embodiment, when the time difference of a certain anchor is detected The frame dropping operation includes: first, searching for the frame with the closest timestamp in the received frame sequence of the anchor. The frame is used as the target alignment frame; then all advanced frames later than this frame are discarded to ensure that the playback frame is consistent with the synchronization reference time. Advanced frames refer to frames in a host video frame sequence with timestamps later than All frames. To achieve timing alignment of multiple anchor video frames, such frames will be discarded and only the frames with the closest timestamp will be retained. The frames are used as synchronized playback frames.

[0089] Figure 4 1 is a schematic diagram of a collaborative live interactive data processing system according to an exemplary embodiment of the present application. Figure 4 As shown, the collaborative live broadcast interactive data processing system 400 provided in this embodiment includes: a collaborative live broadcast data acquisition module 410, a data prediction module 420 based on an artificial intelligence prediction model, a live broadcast data stream dynamic optimization module 430, a collaborative live broadcast interactive optimization module 440 and a multi-anchor video timing synchronization module 450.

[0090] The collaborative live broadcast data collection module 410 collects anchor side data and platform side data. The anchor side data includes bandwidth, interaction delay, packet loss rate, jitter and video frame timestamp. The platform side data includes the number of anchors, interaction frequency and number of viewers. A multi-dimensional time series is generated based on the anchor side data and platform side data.

[0091] The data prediction module 420 is based on an artificial intelligence prediction model, and the artificial intelligence prediction model predicts the bandwidth status prediction sequence, packet loss rate prediction sequence and interaction delay prediction sequence of each anchor within a first preset time based on a multi-dimensional time series.

[0092] The live data stream dynamic optimization module 430 dynamically optimizes the live data stream of each anchor according to the live data stream adaptive strategy based on the bandwidth status prediction sequence, the packet loss rate prediction sequence and the interaction delay prediction sequence.

[0093] The collaborative live broadcast interaction optimization module 440 constructs an interactive experience function based on the interactive delay prediction sequence, performs a global evaluation of the multi-host interaction delay, and dynamically adjusts the interaction between the host and the audience according to the collaborative live broadcast interaction adjustment strategy.

[0094] The multi-anchor video timing synchronization module 450 determines the synchronization reference time based on the video frame timestamps of all the anchors, and performs corresponding video synchronization processing according to the time difference between the video frame timestamp of each anchor and the synchronization reference time.

[0095] Figure 5 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 5 As shown, an electronic device 500 provided in this embodiment includes: a processor 501 and a memory 502; wherein: The memory 502 is used to store computer programs, and the memory may also be a flash memory.

[0096] The processor 501 is used to execute the execution instructions stored in the memory to implement each step in the above method. For details, please refer to the relevant description in the above method embodiment.

[0097] Optionally, the memory 502 may be independent or integrated with the processor 501 .

[0098] When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include: The bus 503 is used to connect the memory 502 and the processor 501 .

[0099] This embodiment further provides a readable storage medium, in which a computer program is stored. When at least one processor of an electronic device executes the computer program, the electronic device executes the methods provided in the above-mentioned various implementation modes.

[0100] This embodiment also provides a program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements the methods provided in the above various embodiments.

[0101] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0102] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A collaborative live interactive data processing method, characterized in that: include: Collaborative live broadcast data collection, collecting anchor-side data and platform-side data, the anchor-side data includes bandwidth, interaction delay, packet loss rate, jitter and video frame timestamp, the platform-side data includes the number of anchors, interaction frequency and number of viewers, and generating multi-dimensional time series based on the anchor-side data and platform-side data; Data prediction based on an artificial intelligence prediction model, the artificial intelligence prediction model predicts a bandwidth status prediction sequence, a packet loss rate prediction sequence, and an interaction delay prediction sequence for each anchor within a first preset time based on the multi-dimensional time series; Dynamic optimization of live data streams, based on the bandwidth status prediction sequence, packet loss rate prediction sequence and interaction delay prediction sequence, dynamically optimizing the live data stream of each anchor according to the live data stream adaptive strategy; Collaborative live broadcast interaction optimization, building an interactive experience function based on the interactive delay prediction sequence, globally evaluating the interaction delay of multiple anchors, and dynamically adjusting the interaction between anchors and viewers according to the collaborative live broadcast interaction adjustment strategy; Multi-host video timing synchronization determines the synchronization reference time based on the video frame timestamps of all the hosts, and performs corresponding video synchronization processing according to the time difference between the video frame timestamp of each host and the synchronization reference time.

2. The collaborative live interactive data processing method according to claim 1, characterized in that: The live data stream adaptive strategy includes a dynamic bit rate adjustment strategy and a resolution and frame rate adjustment strategy; The dynamic bitrate adjustment strategy is to calculate and adjust the dynamic bitrate of each anchor in real time according to the bandwidth fluctuation prediction sequence, the packet loss rate prediction sequence and the interaction delay prediction sequence; The resolution and frame rate adjustment strategy is to adjust the resolution and frame rate of each anchor in real time according to the dynamic bit rate, including: When the dynamic bit rate is greater than or equal to 6 Mbps, the resolution is 4K and the frame rate is 60 FPS; When the dynamic bit rate is greater than or equal to 4 Mbps and less than 6 Mbps, the resolution is 2K and the frame rate is 60 FPS; When the dynamic bit rate is greater than or equal to 2.5Mbps and less than 4Mbps, the resolution is 1080P and the frame rate is 60FPS; When the dynamic bit rate is greater than or equal to 1.5Mbps and less than 2.5Mbps, the resolution is 1080P and the frame rate is 30FPS; When the dynamic bit rate is greater than or equal to 1.0 Mbps and less than 1.5 Mbps, the resolution is 720P and the frame rate is 30 FPS; When the dynamic bit rate is greater than or equal to 0.6 Mbps and less than 1.0 Mbps, the resolution is 480P and the frame rate is 30 FPS; When the dynamic bit rate is greater than or equal to 0.4 Mbps and less than 0.6 Mbps, the resolution is 360P and the frame rate is 30 FPS.

3. The collaborative live interactive data processing method according to claim 1, characterized in that: The interactive experience function is the triggering basis for the collaborative live broadcast interactive adjustment strategy; When the interactive experience function is greater than the interactive optimization trigger threshold, executing the collaborative live broadcast interactive adjustment strategy; The interaction optimization trigger threshold is used to determine whether the interaction experience function is within an acceptable range.

4. The collaborative live interactive data processing method according to claim 3 is characterized in that: The collaborative live broadcast interaction adjustment strategy includes: Calculate the target interaction delay threshold of each anchor; Comparing the interaction delay prediction sequence of each of the anchors with the corresponding target interaction delay threshold; When the predicted interaction delay sequence of the anchor is greater than the corresponding target interaction delay threshold, the anchor is determined to be an interaction optimization object; Interaction optimization is performed on the anchors determined to be the objects of interaction optimization, and the interaction optimization includes reducing the frequency of sending barrages, reducing the frequency of comment interactions, simplifying reward animation effects, and merging gift display effects.

5. The collaborative live interactive data processing method according to claim 4 is characterized in that: The target interaction delay threshold is a dynamic threshold.

6. The collaborative live interactive data processing method according to claim 1, characterized in that: The multi-host video timing synchronization includes: Based on the video frame timestamps collected within the second preset time, generating a video frame timestamp sequence for each anchor; According to the video frame timestamp sequences of all the anchors, a maximum value strategy is adopted to determine the synchronization reference time; Calculate the time difference between the video frame timestamp of each of the anchors and the synchronization reference time; According to the time difference, inserting a frame or dropping a frame is performed on each of the anchor's video frames, including: When the time difference is greater than 0, performing a frame insertion operation on the video frame of the anchor; When the time difference is less than 0, a frame dropping operation is performed on the video frame of the anchor; When the time difference is 0, the video frame of the host is not processed.

7. The collaborative live interactive data processing method according to claim 6, characterized in that: The interpolation operation is an interpolation algorithm based on an artificial intelligence prediction model, which predicts and generates intermediate transition frames by inputting adjacent video frames into the artificial intelligence prediction model.

8. A collaborative live interactive data processing system, characterized in that: include: A collaborative live broadcast data collection module collects data from the anchor side and the platform side. The anchor side data includes bandwidth, interaction delay, packet loss rate, jitter and video frame timestamp. The platform side data includes the number of anchors, interaction frequency and number of viewers. A multi-dimensional time series is generated based on the anchor side data and the platform side data. A data prediction module based on an artificial intelligence prediction model, wherein the artificial intelligence prediction model predicts a bandwidth status prediction sequence, a packet loss rate prediction sequence, and an interaction delay prediction sequence for each anchor within a first preset time based on the multi-dimensional time series; A live data stream dynamic optimization module dynamically optimizes the live data stream of each anchor according to a live data stream adaptive strategy based on the bandwidth status prediction sequence, the packet loss rate prediction sequence and the interaction delay prediction sequence; A collaborative live broadcast interaction optimization module constructs an interactive experience function based on the interactive delay prediction sequence, performs a global evaluation of the multi-host interaction delay, and dynamically adjusts the interaction between the host and the audience according to the collaborative live broadcast interaction adjustment strategy; The multi-anchor video timing synchronization module determines the synchronization reference time based on the video frame timestamps of all the anchors, and performs corresponding video synchronization processing according to the time difference between the video frame timestamp of each anchor and the synchronization reference time.

9. An electronic device, characterized in that: include: processor; as well as, A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images