A method for generating live broadcast replay data, a live broadcast system, and an electronic device

Abstract

This application relates to the technical field of video data processing, and provides a method for generating live replay data, a live broadcast system, and an electronic device. The method includes: responding to a live replay recording instruction, obtaining live record data, and creating a target container; starting a live client in the target container, and entering an auto-play page through the live client; playing and recording the live record data based on the auto-play page to generate live replay data; the method for obtaining video data includes: receiving the original data uploaded by the live client, where the original data is the local video data that the live client obtains and stores in real time through a browser during the live streaming stage, and the local video data is also used to generate a live stream based on the WebRTC protocol, so that the live client can perform streaming during the live streaming stage based on the live stream; generating video data based on the original data. Based on this, the audio and video quality of the replay video can be improved, and thus the on-demand viewing experience of users can be enhanced.

Description

Technical Field

[0001] This application relates to the technical field of video data processing, and in particular, to a method for generating live replay data, a live broadcast system, and an electronic device. Background Art

[0002] WebRTC (Web Real-Time Communication) is an open standard and technology that supports real-time audio and video communication and data transmission between browsers. It was initiated by Google and is standardized by the W3C (World Wide Web Consortium) and the IETF (Internet Engineering Task Force). The main goal of WebRTC is to enable developers to implement peer-to-peer (P2P) real-time communication functions in web pages through simple JavaScript APIs without installing additional plugins or software. Therefore, it is widely used in video live broadcasts.

[0003] However, during the live broadcast implemented based on WebRTC, WebRTC dynamically adjusts the encoding parameters of video and audio, compresses the original data, or drops frames, etc., to adapt to the current network conditions, thereby ensuring the real-time nature of data transmission. Correspondingly, the viewing experience of the live video will be affected, including reduced resolution, frame skipping, etc. If the replay video is directly recorded based on the live stream, there will also be problems with the deterioration of the audio and video quality of the replay video, thus affecting the viewing experience of the replay video. Summary of the Invention

[0004] In order to ensure the file quality of the replay video and improve the viewing experience of the replay video, an embodiment of this application provides a method for generating live replay data. The method is applied to a server and includes the steps of: responding to a live replay recording instruction, obtaining live record data, and creating a target container; starting a live client in the target container and entering an auto-play page through the live client; playing and recording the live record data based on the auto-play page to generate live replay data; where the live record data includes video data recorded in real time during the live broadcast, and the acquisition method of the video data includes: during or after the live broadcast, receiving the original data uploaded by the live client, where the original data is local video data that the live client obtains and stores in real time through the browser during the live streaming stage, and the local video data is also used to generate a live stream based on the WebRTC protocol, so that the live client performs streaming based on the live stream during the live streaming stage; generating the video data based on the original data.

[0005] Based on the above technical solution, in the application scenario of implementing live streaming based on the WebRTC technology, video data is generated by using the original data for the recording of the playback video, thereby ensuring the audio-visual quality of the playback video, enhancing the viewing experience of the playback video, and avoiding the problem of poor audio-visual quality caused by directly using the live stream data for playback recording; it is possible to create a corresponding container for the live broadcast room to generate the playback data, and in the demand scenario where the concurrency of the playback recording requirements is relatively large, the corresponding playback data can be independently generated, thereby improving the generation efficiency of the playback data.

[0006] In one implementation, the method for generating the video data based on the original data includes: determining the time interval corresponding to the original data; determining the type of the original data according to the time interval and the target playback period of the live playback data; if the type of the original data is segment data, obtaining the live stream data, and replacing the segment corresponding to the time interval in the live stream data with the original data to obtain the video data.

[0007] Based on the above technical solution, it is possible to identify the type of the original data and select the corresponding video data acquisition method according to different types. On the one hand, it is possible to allow the storage range of the original data to be selected according to the storage capacity of the user's live end, and on the other hand, it can ensure that the original data can be used as much as possible to generate the video data, thereby ensuring the playback quality of the video data.

[0008] In one implementation, the method for replacing the segment corresponding to the time interval in the live stream data with the original data to obtain the video data includes: determining the overlapping endpoints of the time interval and the target playback period, determining a matching interval based on the overlapping endpoints, and obtaining all the key frames of the original data and the live stream data within the matching interval; comparing each of the key frames based on image recognition technology to determine the key frames with the same picture content in the original data and the live stream data, and determining the original data replacement point and the live stream data replacement point based on the same key frames; generating a replacement segment based on the original data replacement point, and determining a replacement interval based on the live stream data replacement point; replacing the corresponding segment in the replacement interval of the live stream data with the replacement segment.

[0009] Based on the above technical solution, by first determining the overlapping endpoints and determining the matching interval based on the overlapping endpoints, and then positioning the same key frames to respectively determine the replacement points of the original data and the live stream data, thereby realizing segment replacement, it is possible to ensure the playback fluency of the video data obtained after replacement.

[0010] In one implementation, the method for obtaining the video data further includes: after the live broadcast ends, receiving an m3u8 file uploaded by a third-party platform, and obtaining live stream data by accessing the m3u8 file; performing replacement processing on the live stream data based on the original data to obtain the video data.

[0011] In one implementation, the live broadcast record data further includes whiteboard data that is recorded in real time during the live broadcast, and playing the live broadcast record data includes synchronously playing the video data and the whiteboard data based on timestamps.

[0012] Based on the above technical solution, synchronous recording of whiteboard data can be achieved, enabling on-demand users to also synchronously view the operation data of the whiteboard, including shared files uploaded, marking behaviors on the files, etc.

[0013] In one implementation, the live broadcast record data further includes behavior data that is recorded in real time during the live broadcast, and playing the live broadcast record data includes, while playing the video data, reproducing the behavior data by running a script file.

[0014] Based on the above technical solution, it is possible to reproduce the operation behaviors of each live broadcast user during the live broadcast, thereby maintaining the consistency of the pictures viewed by on-demand and live broadcast users and avoiding omission of live broadcast information.

[0015] Based on the same inventive concept, an embodiment of the present application further provides a live broadcast system, which includes a server, multiple live broadcast user terminals, and a live broadcast control terminal. Among them, the live broadcast user terminals and the live broadcast control terminal are implemented based on a browser and are communicatively connected to the server, and the server is used to generate live broadcast playback data based on the above method.

[0016] In one implementation, the live broadcast user terminal is used to obtain local video data during the live broadcast streaming stage, convert the local video into a live stream based on the WebRTC protocol, and perform streaming. At the same time, based on a preset storage policy, part or all of the local video data is saved locally, and during the live broadcast pulling stage or after the live broadcast ends, the saved local video data is sent to the server as the original data.

[0017] In one implementation, the live broadcast user terminal saving part of the local video data locally based on a preset storage policy includes: obtaining network transmission data, and saving the local video data when the network transmission data indicates that the network quality meets the preset requirements; where the preset requirements are obtained based on historical live broadcast data analysis, and when the network quality meets the preset requirements, there is a situation where the audio and video quality of the live stream converted from the local video data deteriorates based on the WebRTC protocol.

[0018] Based on the above technical solution, by monitoring the network quality in real time, it is possible to locate the live stream data with deteriorated audio and video quality, so as to achieve precise replacement. On the one hand, it can reduce the storage amount of the original data, and on the other hand, it can ensure that the segments with deteriorated audio and video quality caused by network quality in the live stream data are all replaced, ensuring the playback quality of the video data.

[0019] In addition, an embodiment of the present application further provides an electronic device, which includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the above method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application.

[0021] In order to more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0022] Figure 1 The structure diagram of the live broadcast system provided by the embodiment of this application is shown.

[0023] Figure 2 The flowchart of the method for generating live broadcast playback data provided by the embodiment of this application is shown.

[0024] Figure 3 The flowchart of the method for obtaining video data in the embodiment of this application is shown.

[0025] Figure 4 The flowchart of the method for generating video data based on original data in the embodiment of this application is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

[0027] In the description of the embodiments of this application, unless otherwise specified, the meaning of "a plurality" refers to two or more.

[0028] The features, structures, or characteristics in this application can be combined in one or more embodiments in any suitable manner. In various embodiments of this application, the magnitude of the serial numbers of the various processes does not imply the order of execution, and the order of execution of the various processes should be determined by their functions and internal logics, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0029] Some optional features in the embodiments of this application can, in some scenarios, be implemented independently without relying on other features, solve corresponding technical problems, and achieve corresponding effects. In some scenarios, they can also be combined with other features according to requirements.

[0030] In this application, unless otherwise specified, the same or similar parts between various embodiments can be referred to each other. In various embodiments of this application, if there is no special instruction and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be cross-referred. The technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships. The implementation manners of this application do not constitute a limitation to the protection scope of this application.

[0031] The embodiments of this application will be described in detail below with reference to the drawings.

[0032] Please refer to Figure 1 , the embodiments of this application provide a live broadcast system, including a server 110, a plurality of live broadcast user terminals 120, and a live broadcast control terminal 130. Among them, the live broadcast user terminals 120 and the live broadcast control terminal 130 are implemented based on a browser and are communicatively connected to the server 110.

[0033] The live broadcast user terminal 120 is implemented based on a browser. In implementation, each live broadcast user terminal 120 can be correspondingly configured with a corresponding live broadcast role. In one example, user roles such as a lecturer, a host, and an audience can be set. In the actual application process, corresponding operation permissions can be configured for different user roles, including but not limited to microphone usage permissions, screen sharing permissions, etc.

[0034] The user can access the management background provided by the server 110 through the browser to create a live broadcast room and configure the live broadcast roles participating in the live broadcast. The management background creates a corresponding access link according to the live broadcast room roles configured by the user. The user invites other users participating in the live broadcast to enter the live broadcast room in the corresponding roles by sharing the access link. Among them, the access link can be directly accessed through the browser, and the users participating in the live broadcast can enter the live broadcast room through the local browser without installing a client or other plugins.

[0035] In the implementation of this application, the live user terminal is also used to obtain local video data during the live video streaming stage, convert the local video into a live stream based on the WebRTC protocol, and perform video streaming. At the same time, based on a preset storage policy, part or all of the local video data is saved locally, and during the live video pulling stage or after the live broadcast ends, the saved local video data is sent to the server as the original data for the server to generate a playback video.

[0036] In the implementation, the preset storage policy includes selecting to save all or part of the local video data based on storage control parameters.

[0037] Specifically, while creating the access links corresponding to each live role, the server 110 can estimate the video streaming duration corresponding to each live user terminal, determine the storage control parameters according to the estimated duration corresponding to each live user terminal, and send the storage control parameters to the corresponding live user terminal after the user enters the live broadcast room through the access link.

[0038] In one example, the server 110 can estimate the video streaming duration of each live user terminal according to the live role and quantity in the live broadcast room. For the live user terminals with an estimated duration greater than or equal to 50% of the total duration of the target playback period, the storage control parameters for saving part of the local video data are sent; for the live user terminals with an estimated duration less than 50% of the total duration of the target playback period, the storage control parameters for saving all of the local video data are sent.

[0039] The live user terminal determines the saving method of the local video data according to the received storage control parameters.

[0040] In one implementation, when the live user terminal determines to save only part of the local video data, during the video streaming stage, it obtains the network transmission data in real time, judges whether the network quality meets the preset requirements based on the network transmission data, and saves the locally generated video data in real time when the network quality meets the preset requirements, and stops saving the local video data when the network quality does not meet the preset requirements. In this way, the data storage pressure on the live user terminal can be reduced.

[0041] In a specific example, the network transmission data can be obtained by calling RTCPeerConnection.getStats(), and then based on the network transmission data, it is determined whether the network quality meets the preset requirements to determine whether to save the local video data. The network transmission data includes bandwidth and round-trip time. The preset requirements include that the bandwidth has reached the first threshold and the round-trip time continues to increase, then it is determined that the network quality meets the preset requirements, where the first threshold can be the maximum available bandwidth of the live user terminal, and the maximum available bandwidth can be obtained based on the network configuration data of the live user terminal, or determined according to the real-time transmission situation or historical data of the live user terminal.

[0042] Optionally, the live client may trigger the operation of saving local video data only when the network quality meets the preset requirements, and send all the saved local video clip data to the server as the original data; or it may first save all the local video data during the streaming stage, and after the streaming ends, analyze the network transmission data to determine the time interval when the network quality meets the preset requirements, and intercept the local video data based on the time interval to obtain the clip data corresponding to the time interval and send it to the server as the original data.

[0043] In another implementation, the live client may save the local video data based on the time interval sent by the live control end during the streaming stage.

[0044] Specifically, during the live broadcast, the live control end is in the pulling stream stage. Based on this, the live control end can determine the time interval by obtaining the video quality during the pulling stream process.

[0045] In one example, the live control end can obtain the packet loss rate during the live broadcast, determine the time period when the packet loss rate exceeds the second threshold as the time interval, and send it to the corresponding pushing stream end, so that the pushing stream end can save the local video data within the time interval. Among them, the second threshold can be obtained through historical data analysis. When the packet loss rate exceeds the second threshold, the audio-visual quality of the live stream data deteriorates significantly.

[0046] In another example, the live control end can determine the time interval by receiving the original data saving instruction sent by the user and send it to the corresponding pushing stream end. Specifically, the administrator user can monitor the live video quality in real time during the live broadcast. When it is found that the video picture display quality deteriorates or there is abnormal audio-visual acceleration, the data saving instruction is input. The live control end determines the time interval according to the time information specified in the data saving instruction or the sending time of the instruction.

[0047] For example, the start time in the time information specified in the instruction can be advanced by a preset duration, and the end time can be delayed by a preset duration to expand the time period corresponding to the time information, so as to obtain the time interval. In this way, the problem that the required original data cannot be obtained due to untimely user operations can be avoided to a large extent. In the case of determining the time interval based on the sending time of the instruction, the sending time can also be advanced by a preset duration as the start time of the time interval, and when the stop saving instruction sent by the user is received, the receiving time of the instruction is determined as the end time of the time interval.

[0048] In yet another example, the live broadcast control terminal can generate time intervals based on the above two examples simultaneously and send them to the live broadcast user terminal. The live broadcast user terminal can determine the storage range of local video data based on all the received time intervals.

[0049] The live broadcast user terminal 120 is also used to record behavior data during the live broadcast. The behavior data includes the action behaviors of turning on or off the camera or microphone and the corresponding time points, and sends them to the server 110.

[0050] The live broadcast control terminal 130 is used to manage the live broadcast room. When the management background creates access links corresponding to different live broadcast roles, it also creates an access link for accessing the live broadcast control terminal. The live broadcast room administrator can open the access link through a browser to enter the live broadcast control terminal, and thus manage the live broadcast room based on the live broadcast control terminal, including starting or ending the live broadcast, selecting a screen layout template for the live broadcast room, setting the warm-up screen before the live broadcast starts, and saving the received live stream data during the live broadcast and sending it to the server.

[0051] The server 110 is used to provide a management background to implement the creation of the live broadcast room and generate high-quality live broadcast replay data for the live broadcast content of the live broadcast room for subsequent on-demand use.

[0052] Please refer to Figure 2 , the method for generating live broadcast replay data provided by the embodiments of the present application specifically includes the following steps.

[0053] S210, in response to the live broadcast replay recording instruction, obtain the live broadcast record data and create a target container.

[0054] In implementation, the live broadcast replay recording instruction can be input by the user through the live broadcast control terminal before or during the live broadcast, or can be automatically generated based on the set trigger conditions.

[0055] In one example, the trigger conditions include that when the popularity of the live broadcast room exceeds the first popularity threshold, a live broadcast replay recording instruction is generated to instruct the server to determine the live broadcast period with the live broadcast popularity value exceeding the second popularity threshold during the live broadcast as the target replay period, and generate replay data corresponding to the target replay period, where the first popularity threshold is greater than or equal to the second popularity threshold, and the specific value can be set according to the application requirements of the user or the platform; the popularity of the live broadcast room can be determined according to the number of viewers of the live broadcast room.

[0056] The live broadcast replay recording instruction is used to trigger the server to generate live broadcast replay data and specify the target replay period corresponding to the replay data. That is to say, the server can generate replay data for the entire live broadcast according to the live broadcast replay recording instruction, or generate replay data for a certain live broadcast period or multiple live broadcast periods.

[0057] The live record data includes video data recorded in real time during the live broadcast. Please refer to Figure 3 , and the method for obtaining the video data specifically includes the following steps.

[0058] S310, during the live broadcast or after the live broadcast ends, receive the original data uploaded by the live user terminal.

[0059] Among them, the original data is the local video data obtained and stored in real time by the live user terminal through the browser during the live streaming stage. The local video data is also used to generate a live stream based on the WebRTC protocol, so that the live user terminal can perform streaming based on the live stream during the streaming stage.

[0060] It should be noted that the local video data is high-fidelity data directly obtained based on the local device and not generated into a live stream by WebRTC. During the live broadcast, since the WebRTC protocol will dynamically adjust the resolution and bitrate according to the network status to ensure real-time data transmission, the quality of the live audio and video presented based on the live stream data will also change accordingly, resulting in situations that affect the live viewing experience, such as blurred images, dropped frames, and stuttering. If the playback data is directly recorded based on the live stream data, the playback video will also have the same problems, thus affecting the viewing experience of on-demand users. Based on this, by obtaining the original data to generate the playback data, the playback quality of the playback video can be guaranteed.

[0061] S320, generate video data based on the original data.

[0062] In implementation, since the requirements for generating playback data in different live rooms are different and the corresponding types of original data are also different, the server needs to first determine the type of the original data and perform corresponding processing according to the type of the original data to obtain the video data. Please refer to Figure 4 , in an implementation, the method for generating video data based on the original data specifically includes the following steps.

[0063] S410, determine the time interval corresponding to the original data.

[0064] Among them, the time interval corresponding to the original data can be determined based on the start time and end time of the original data. In an example, the live user terminal can add a time mark to the original data when sending the original data to record the start time and end time corresponding to the original data. Based on this, the server can determine the corresponding time interval based on the time mark carried in the original data.

[0065] S420, determine the type of the original data according to the time interval and the target playback period of the live playback data.

[0066] In implementation, the target playback period is determined based on the live playback recording instruction. If the time interval of all the original data sent by each live user terminal in the live room can completely cover the target playback period, the type of the original data is determined as complete data; if it cannot completely cover the target playback period, the type of the original data is determined as segment data.

[0067] S430, in the case where the type of the original data is segment data, obtain the live stream data, and replace the segment corresponding to the time interval in the live stream data with the original data to obtain the video data.

[0068] Among them, the live stream data is the push stream data during the live broadcast. In an example, the live stream data is received and saved by the live control end during the live broadcast. For example, the live control end can save all the live stream data received during the live broadcast, intercept the corresponding live stream data according to the target playback period, or only save it for the target playback period to directly obtain the live stream data.

[0069] In another example, the live stream data can also be obtained based on a third-party platform. Among them, the third-party platform is implemented based on the Agora technology. Specifically, in some application scenarios, different live broadcast scenario requirements can be adapted by combining the use of the WebRTC technology and the Agora technology. For example, when there are a large number of viewers in the live room, the low-latency interaction between the anchor and the viewers can be realized based on the WebRTC technology, and at the same time, the live stream is distributed to a large number of viewers through the Agora technology. Based on this, after the live broadcast ends, the server can receive the m3u8 file uploaded by the Agora server and obtain the live stream data by accessing the m3u8 file.

[0070] In implementation, there may be a situation where the live stream data is inconsistent with the original data. Therefore, during the process of replacing the segment corresponding to the time interval in the live stream data with the original data, it is necessary to first determine the segment corresponding to the time interval.

[0071] In an example, since the type of the original data is data segment, it indicates that the time interval corresponding to the original data overlaps with the target playback period at least partially. Therefore, the overlapping endpoints of the time interval and the target playback period can be first determined, the matching interval can be determined based on the overlapping endpoints, and all the key frames of the original data and the live stream data within the matching interval can be obtained, and the key frames with the same picture content in the original data and the live stream data can be determined based on the image recognition technology. The original data replacement point and the live stream data replacement point can be determined based on the same key frames, the replacement segment can be generated based on the original data replacement point, and the replacement interval can be determined based on the live stream data replacement point, and the replacement segment is used to replace the corresponding segment in the replacement interval of the live stream data.

[0072] For example, the time interval of an original data is from 12:00:00 to 12:10:01, and the target playback period includes from 12:05:02 to 12:30:00. That is to say, the server determines according to the instruction that the live data in the period from 12:05:02 to 12:30:00 needs to be recorded. And the live user side saves and uploads the local video data from 12:00:00 to 12:10:01 according to the change of network quality. At the same time, the live control side records the live video during the period from 12:05:02 to 12:30:00. During the live process, WebRTC may drop frames or accelerate the live video, resulting in a difference between the playback progress of the live video and the original data. Therefore, even if the time of the live user side and the live control side is synchronized, there will still be a situation where the original data and the live stream data are out of sync within the same time period. For example, the original data at 12:10:01 is recorded at 12:10:03 after transmission processing.

[0073] To avoid the situation of unsmooth playback progress caused by directly replacing the original data, the overlapping endpoints 12:05:02 and 12:10:01 can be first determined, and with a time difference range of 2 seconds. Taking the overlapping endpoint 12:10:01 as an example, the matching interval from 12:09:59 to 12:10:03 is determined. The key frames of the original data and the live stream data within the range from 12:09:59 to 12:10:03 are respectively obtained and compared to determine the time points with the same key frames. For example, the time point corresponding to the same key frame in the original data is 12:10:00, and the time point in the live stream data is 12:10:01. Based on the same method, the replacement point 12:05:00 corresponding to the overlapping endpoint 12:05:02 in the original data can be determined.

[0074] In this way, during the replacement, the time point corresponding to the same key frame in the original data can be determined as the replacement point of the original data and intercepted to obtain the original data for replacement. At the same time, the time point corresponding to the same key frame in the live stream data is determined as the replacement point to determine the replacement segment. That is, the segment from 12:05:00 to 12:10:00 in the original data is used to replace the segment from 12:05:02 to 12:10:01 in the live stream data, so as to obtain the video data for generating the playback data.

[0075] Based on this, by using the high-fidelity original data to replace the live stream data, the segments with deteriorated audio and video quality in the live stream data can be repaired, thus ensuring the viewing experience of the playback data. And by positioning the replacement point based on the key frames, the audio and video smoothness of the playback data can be guaranteed.

[0076] S440. When the type of the original data is complete data, determine the original data as video data.

[0077] In implementation, when it is determined that the type of the original data is complete data, the original data can be directly determined as video data, or the original data can be intercepted according to the target playback period to obtain video data that meets the requirements of the target playback period.

[0078] When the server obtains the live record data, a target container is created based on the task of generating the current live playback data. In one example, the creation of the target container can be implemented based on the docker technology.

[0079] S220. Start the live client in the target container and enter the auto-play page through the live client.

[0080] In implementation, after the target container is created, a browser is started in the container by loading a predefined image file, and the auto-play page is entered through the browser. Among them, the auto-play page is a pre-created web page dedicated to recording live playback data. The auto-play page can realize the auto-play of video data and perform synchronous recording.

[0081] S230. Play the live record data based on the auto-play page and record it to generate live playback data.

[0082] In implementation, the auto-play page can auto-play the video data and record it to generate live playback data in MP4 format.

[0083] Based on the above technical solution, not only can the live data be converted into on-demand data, but also the on-demand data can be optimized by using high-fidelity original data, which can ensure the audio-visual quality of the on-demand data. In addition, in the process of video data processing, by matching key frames to determine the replacement segments, the influence of segment replacement on the audio-visual smoothness can be effectively reduced.

[0084] During some live broadcasts, a whiteboard tool is used to assist in presenting PPTs and for writing and marking. However, the real-time information flow generated by some whiteboard tools is independent of the live stream. Therefore, the live stream data saved by the live broadcast control terminal does not include the display data related to the whiteboard. To enable the synchronous display of the whiteboard screen during on-demand playback, in some embodiments of the present application, the live broadcast record data further includes whiteboard data recorded in real time during the live broadcast. The whiteboard data can be obtained by the live broadcast control terminal saving the received whiteboard data, for example, by decrypting the received SCTP (Stream Control Transmission Protocol) protocol packets. Correspondingly, in the step of playing the live broadcast record data, it further includes synchronously playing the video data and the whiteboard data based on timestamps. It should be noted that since the playback time of the video data obtained by replacing the original data may have changed compared to the playback time of the live stream data. For example, when a 5-second segment of original data is used to replace a 4-second segment of live stream data, the corresponding playback time needs to be reorganized to maintain the accuracy of the time information. The whiteboard data has been synchronized with the live stream data during the live broadcast. Thus, before synchronously playing the video data and the whiteboard data, it is also necessary to align the time of the whiteboard data with the video data. Among them, the alignment can be performed according to the replacement points of the live stream data and the timestamps of the whiteboard data, and the corresponding whiteboard data within the replacement segment is adapted, including extending or reducing the time of the whiteboard static screen, so as to achieve the alignment of the playback time of the whiteboard data and the original data.

[0085] To completely restore the operation behaviors of users during the live broadcast and for synchronous display, in some embodiments of the present application, each live user terminal also records the behavior data of users during the live broadcast and sends it to the server for saving in the live broadcast record data. Therefore, the live broadcast record data further includes behavior data recorded in real time during the live broadcast. Correspondingly, playing the live broadcast record data also includes, while playing the video data, reproducing the behavior data by running a script file. In one example, the behavior data is saved as a json file, and the JavaScript script in the automatic playback page will make corresponding state changes during the live broadcast according to the json file, achieving the purpose of simulating the live broadcast screen.

[0086] In addition, an embodiment of the present application further provides an electronic device, which includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements the method in any implementation manner in the embodiment of the present application. Among them, the processor may adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, and is used to execute relevant programs to implement the method in any implementation manner in the embodiment of the present application.

[0087] The processor may also be an integrated circuit electronic device with signal processing capabilities. In the implementation process, each step of the method in any implementation manner in the embodiment of the present application may be completed by the integrated logic circuit in the hardware of the processor or the instruction in the form of software.

[0088] The above-mentioned processor may also be a general-purpose processor, a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiment of the present application may be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor.

[0089] The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the functions required to be executed by the units included in the data processing device of the embodiment of the present application, or executes the method in any implementation manner in the embodiment of the present application.

[0090] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-mentioned method embodiment.

[0091] Those skilled in the art can understand that all or part of the steps in the above implementation methods can be completed by instructing relevant hardware through a program. This program is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0092] The above are all preferred embodiments of this application. It does not limit the protection scope of this application accordingly. Therefore, all equivalent changes made according to the structure, shape, and principle of this application should be covered within the protection scope of this application.

Claims

1. A method for generating live replay data, characterized in that, The method is applied to a server and includes the steps of: In response to a live replay recording instruction, obtain live record data and create a target container; Start a live client in the target container and enter an auto-play page through the live client; Play the live record data on the auto-play page and record it to generate live replay data; wherein, the live record data includes video data recorded in real time during the live process, and the acquisition method of the video data includes: During the live process or after the live ends, receive the original data uploaded by the live client, where the original data is local video data that the live client obtains and stores in real time through the browser during the live streaming stage and sends to the server during the live pulling stage or after the live ends. The local video data is also used to generate a live stream based on the WebRTC protocol so that the live client performs streaming during the live streaming stage based on the live stream; Determine the original data as the video data, or generate the video data by performing segment replacement on the live stream data based on the original data; wherein, the method for the live client to obtain local video data in real time through the browser during the live streaming stage includes: During the live streaming stage, obtain network transmission data in real time, determine whether the network quality meets the preset requirements based on the network transmission data. When the network quality meets the preset requirements, save the locally generated video data in real time. When the network quality does not meet the preset requirements, stop saving the local video data; the network transmission data includes bandwidth and round-trip time, and the preset requirements include that the bandwidth has reached a first threshold and the round-trip time continues to increase; or, During the live streaming stage, save the local video data within the time interval sent by the live control end. The live control end determines the time interval in which the packet loss rate exceeds a second threshold during the live process and sends it to the corresponding live streaming end.

2. The method according to claim 1, wherein The method for generating the video data based on the original data includes: Determine the time interval corresponding to the original data; Determine the type of the original data according to the time interval and the target playback period of the live replay data; If the type of the original data is segment data, obtain the live stream data, and replace the segment corresponding to the time interval in the live stream data with the original data to obtain the video data.

3. The method according to claim 2, wherein The method for replacing the segment corresponding to the time interval in the live stream data with the original data to obtain the video data includes: Determine the overlapping endpoints of the time interval and the target playback period, determine a matching interval based on the overlapping endpoints, and obtain all the key frames of the original data and the live stream data within the matching interval; compare each of the key frames based on image recognition technology to determine the key frames with the same picture content in the original data and the live stream data, and determine the original data replacement point and the live stream data replacement point based on the same key frames; generate a replacement segment based on the original data replacement point, and determine a replacement interval based on the live stream data replacement point; replace the corresponding segment of the replacement interval in the live stream data with the replacement segment.

4. The method according to claim 1, wherein The method for obtaining the video data further includes: After the live broadcast ends, receive the m3u8 file uploaded by a third-party platform, and obtain the live stream data by accessing the m3u8 file; Perform a replacement process on the live stream data based on the original data to obtain the video data.

5. The method according to claim 1, characterized in that The live broadcast record data further includes whiteboard data recorded in real time during the live broadcast, and playing the live broadcast record data includes synchronously playing the video data and the whiteboard data based on timestamps.

6. The method according to claim 1 or 5, characterized in that, The live broadcast record data further includes behavior data recorded in real time during the live broadcast, and playing the live broadcast record data includes, while playing the video data, reproducing the behavior data by running a script file.

7. A live broadcast system, characterized in that, The live broadcast system includes a server, multiple live broadcast client terminals, and a live broadcast control terminal. Among them, the live broadcast client terminals and the live broadcast control terminal are implemented based on a browser and are communicatively connected to the server, and the server is used to generate live broadcast playback data based on the method described in any one of claims 1 to 6.

8. The system according to claim 7, characterized in that, The live broadcast client terminal is used to obtain local video data during the live stream pushing stage, convert the local video into a live stream based on the WebRTC protocol, and perform live stream pushing. At the same time, based on a preset storage strategy, save part or all of the local video data locally, and send the saved local video data to the server as the original data after the live stream pulling stage or after the live broadcast ends.

9. The system according to claim 8, wherein The live broadcast client terminal saving part of the local video data locally based on a preset storage strategy includes: obtaining network transmission data, and saving the local video data when the network transmission data indicates that the network quality meets the preset requirements; wherein, the preset requirements are obtained based on historical live broadcast data analysis, and when the network quality meets the preset requirements, there is a situation where the audio and video quality of the live stream converted from the local video data deteriorates based on the WebRTC protocol.

10. An electronic device, characterized in that, It includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements the method described in any one of claims 1 to 6.

Images