Live broadcast playback data generation method, live broadcast system and electronic equipment

Abstract

The invention relates to the technical field of video data processing, and provides a live broadcast playback data generation method, a live broadcast system and electronic equipment, and the method comprises the steps: obtaining live broadcast record data in response to a live broadcast playback recording instruction, and creating a target container; starting a live broadcast user side in the target container, and entering an automatic playing page through the live broadcast user side; playing and recording the live broadcast record data based on the automatic playing page to generate live broadcast playback data; the video data acquisition mode comprises: receiving original data uploaded by a live broadcast user side, the original data being local video data acquired and stored by the live broadcast user side in real time through a browser in a live broadcast stream pushing stage, and the local video data being further used for generating a live broadcast stream based on a WebRTC protocol; the live broadcast user side carries out stream pushing in a live broadcast stream pushing stage based on the live broadcast stream; the video data is generated based on the original data, so that the sound and picture quality of the playback video can be improved, and the on-demand watching experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the technical field of video data processing, and in particular to a live broadcast playback data generation method, a live broadcast system and an electronic device. Background Art

[0002] WebRTC (Web Real-Time Communication) is an open standard and technology that supports real-time audio and video communication and data transmission between browsers. It was initiated by Google and has been supported by W3C (World Wide Web Consortium) and IETF (Internet Engineering Task Force). The main goal of WebRTC is to enable developers to implement peer-to-peer (P2P) real-time communication functions in web pages through simple JavaScript APIs without installing additional plug-ins or software. Therefore, it is widely used in live video broadcasting.

[0003] However, during the live broadcast process based on WebRTC, WebRTC will dynamically adjust the encoding parameters of the video and audio according to the network transmission quality, compress the original data, or drop frames to adapt to the current network conditions, thereby ensuring the real-time data transmission. Accordingly, the viewing experience of the live video will be affected, including resolution reduction, frame skipping, etc. If the video is directly recorded and played back based on the live stream, the playback video will also have the problem of deterioration in audio and video quality, thus affecting the viewing experience of the playback video. Summary of the invention

[0004] In order to ensure the file quality of the playback video and improve the viewing experience of the playback video, an embodiment of the present application provides a method for generating live playback data, which is applied to a server and includes the steps of: responding to a live playback recording instruction, obtaining live recording data, and creating a target container; starting a live broadcast user terminal in the target container, and entering an automatic playback page through the live broadcast user terminal; playing and recording the live broadcast recording data based on the automatic playback page to generate live broadcast playback data; wherein the live broadcast recording data includes video data recorded in real time during the live broadcast process, and the method for obtaining the video data includes: receiving the original data uploaded by the live broadcast user terminal during the live broadcast process or after the live broadcast ends, wherein the original data is the local video data obtained and stored in real time by the live broadcast user terminal through the browser during the live broadcast push stage, and the local video data is also used to generate a live stream based on the WebRTC protocol, so that the live broadcast user terminal pushes the stream based on the live stream during the live broadcast push stage; the video data is generated based on the original data.

[0005] Based on the above technical solution, in the application scenario of live broadcast based on WebRTC technology, video data is generated by using original data for recording playback video, thereby ensuring the sound and picture quality of the playback video, improving the viewing experience of the playback video, and avoiding the problem of poor sound and picture quality caused by directly using live stream data for playback recording; it is possible to create a corresponding container for the live broadcast room to generate playback data, and in the demand scenario with a large concurrency of playback recording requirements, the corresponding playback data can be generated independently, thereby improving the generation efficiency of playback data.

[0006] In one implementation, the method for generating the video data based on the original data includes: determining a time interval corresponding to the original data; determining the type of the original data based on the time interval and a target playback period of the live playback data; if the type of the original data is fragment data, obtaining live stream data, replacing the fragment in the live stream data corresponding to the time interval with the original data, and obtaining the video data.

[0007] Based on the above technical solution, the type of original data can be identified and the corresponding video data acquisition method can be selected according to different types. On the one hand, it can allow the storage range of the original data to be selected according to the storage capacity of the user's live broadcast end. On the other hand, it can ensure that the original data can be used to generate video data as much as possible, thereby ensuring the playback quality of the video data.

[0008] In one implementation, the method of replacing the segment corresponding to the time interval in the live streaming data with the original data to obtain the video data includes: determining the overlapping endpoints of the time interval and the target playback time period, determining a matching interval based on the overlapping endpoints, and obtaining all key frames of the original data and the live streaming data located in the matching interval; comparing each of the key frames based on image recognition technology to determine the key frames with the same picture content in the original data and the live streaming data, and determining the original data replacement points and the live streaming data replacement points based on the same key frames; generating a replacement segment based on the original data replacement point, and determining a replacement interval based on the live streaming data replacement point; and replacing the corresponding segment of the replacement interval in the live streaming data with the replacement segment.

[0009] Based on the above technical solution, by first determining the overlapping endpoints, and then determining the matching interval based on the overlapping endpoints, and then locating the same key frames, the replacement points of the original data and the live stream data are determined respectively, thereby realizing fragment replacement, which can ensure the smoothness of the playback of the video data obtained after replacement.

[0010] In one implementation, the method for obtaining the video data also includes: after the live broadcast ends, receiving an m3u8 file uploaded by a third-party platform, and obtaining live stream data by accessing the m3u8 file; and replacing the live stream data based on the original data to obtain the video data.

[0011] In one implementation, the live broadcast record data further includes whiteboard data recorded in real time during the live broadcast process, and the playing of the live broadcast record data includes synchronously playing the video data and the whiteboard data based on a timestamp.

[0012] Based on the above technical solution, synchronous recording of whiteboard data can be achieved, so that on-demand users can also synchronously view the whiteboard operation data, including uploaded shared files, file marking behavior, etc.

[0013] In one implementation, the live broadcast record data also includes behavior data recorded in real time during the live broadcast process, and playing the live broadcast record data includes reproducing the behavior data by running a script file while playing the video data.

[0014] Based on the above technical solution, it is possible to reproduce the operation behaviors of each live broadcast user during the live broadcast process, thereby maintaining the consistency of the images viewed by the on-demand and live broadcast users and avoiding omission of live broadcast information.

[0015] Based on the same inventive concept, an embodiment of the present application also provides a live broadcast system, which includes a server, multiple live broadcast user terminals and a live broadcast control terminal, wherein the live broadcast user terminals and the live broadcast control terminal are implemented based on a browser and are communicatively connected to the server, and the server is used to generate live broadcast playback data based on the above method.

[0016] In one implementation, the live broadcast user terminal is used to obtain local video data in the live broadcast streaming stage, convert the local video into a live broadcast stream based on the WebRTC protocol, and push the stream. At the same time, based on a preset storage strategy, part or all of the local video data is saved locally, and after the live broadcast streaming stage or the end of the live broadcast, the saved local video data is sent to the server as the original data.

[0017] In one implementation, the live broadcast user terminal locally saves part of the local video data based on a preset storage strategy, including: obtaining network transmission data, and saving the local video data when the network transmission data indicates that the network quality meets the preset requirements; wherein the preset requirements are obtained based on an analysis of historical live broadcast data, and when the network quality meets the preset requirements, the live broadcast stream converted from the local video data based on the WebRTC protocol has deteriorated audio and video quality.

[0018] Based on the above technical solution, by real-time monitoring of network quality, it is possible to locate live streaming data with deteriorated audio and video quality, thereby achieving accurate replacement. On the one hand, it can reduce the storage amount of original data, and on the other hand, it can ensure that all segments in the live streaming data with deteriorated audio and video quality caused by network quality are replaced, thereby ensuring the playback quality of the video data.

[0019] In addition, an embodiment of the present application also provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and the program or instruction implements the above method when executed by the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings constituting a part of the present application are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A structural schematic diagram of a live broadcast system provided in an embodiment of the present application is shown.

[0023] Figure 2 A flow chart of a method for generating live playback data provided in an embodiment of the present application is shown.

[0024] Figure 3 A flow chart of a method for acquiring video data in an embodiment of the present application is shown.

[0025] Figure 4 A flow chart of a method for generating video data based on original data in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more.

[0028] The features, structures or characteristics in this application may be combined in one or more embodiments in any suitable manner. In various embodiments of this application, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0029] Some optional features in the embodiments of the present application may be implemented independently in some scenarios without relying on other features to solve corresponding technical problems and achieve corresponding effects. They may also be combined with other features in some scenarios as needed.

[0030] In this application, unless otherwise specified, the same or similar parts between the various embodiments can refer to each other. In the various embodiments of this application, if there is no special description and logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships. The implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0031] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0032] Please refer to Figure 1 An embodiment of the present application provides a live broadcast system, including a server 110, multiple live broadcast user terminals 120 and a live broadcast control terminal 130, wherein the live broadcast user terminals 120 and the live broadcast control terminal 130 are implemented based on a browser and are communicated with the server 110.

[0033] The live broadcast client 120 is implemented based on a browser. In implementation, each live broadcast client 120 may be configured with a corresponding live broadcast role. In one example, user roles such as lecturer, host, and audience may be set. In actual application, corresponding operation permissions may be configured for different user roles, including but not limited to microphone use permissions, screen sharing permissions, etc.

[0034] The user can access the management backend provided by the server 110 through a browser to create a live broadcast room and configure the live broadcast role participating in the live broadcast. The management backend creates a corresponding access link according to the live broadcast room role configured by the user. The user invites other users participating in the live broadcast to enter the live broadcast room with the corresponding role by sharing the access link. The access link can be directly accessed through the browser, and the users participating in the live broadcast can enter the live broadcast room through the local browser without installing a client or other plug-ins.

[0035] In the implementation of this application, the live broadcast user terminal is also used to obtain local video data during the live broadcast streaming stage, convert the local video into a live stream based on the WebRTC protocol, and push the stream. At the same time, based on a preset storage strategy, part or all of the local video data is saved locally, and after the live broadcast streaming stage or the end of the live broadcast, the saved local video data is sent to the server as original data for the server to generate a playback video.

[0036] In implementation, the preset storage strategy includes selecting to save all or part of the local video data based on the storage control parameters.

[0037] Specifically, while creating access links corresponding to each live broadcast role, server 110 can estimate the streaming duration corresponding to each live broadcast user terminal, determine storage control parameters based on the estimated duration corresponding to each live broadcast user terminal, and send storage control parameters to the corresponding live broadcast user terminal after the user enters the live broadcast room through the access link.

[0038] In one example, server 110 can estimate the streaming duration of each live broadcast user terminal based on the live broadcast role and number of the live broadcast room, and send storage control parameters for saving part of the local video data to the live broadcast user terminal whose estimated duration is greater than or equal to 50% of the total duration of the target playback period; and send storage control parameters for saving all local video data to the live broadcast user terminal whose estimated duration is less than 50% of the total duration of the target playback period.

[0039] The live broadcast client determines the storage method of the local video data according to the received storage control parameters.

[0040] In one implementation, when the live broadcast user terminal determines to save only part of the local video data, it obtains the network transmission data in real time during the streaming stage, and determines whether the network quality meets the preset requirements based on the network transmission data, and saves the local video data generated in real time if the network quality meets the preset requirements, and stops saving the local video data if the network quality does not meet the preset requirements. In this way, the data storage pressure of the live broadcast user terminal can be reduced.

[0041] In a specific example, network transmission data can be obtained by calling RTCPeerConnection.getStats(), and then whether the network quality meets the preset requirements based on the network transmission data is determined to determine whether to save the local video data. The network transmission data includes bandwidth and round-trip time. The preset requirements include that the bandwidth has reached a first threshold and the round-trip time continues to increase, then it is determined that the network quality meets the preset requirements, where the first threshold can be the maximum bandwidth available to the live broadcast user end, and the maximum available bandwidth can be obtained based on the network configuration data of the live broadcast user end, or it can be determined based on the real-time transmission situation or historical data of the live broadcast user end.

[0042] Optionally, the live broadcast user end can trigger the operation of saving local video data only when the network quality meets the preset requirements, and send all the saved local video fragment data as original data to the server; or all local video data in the streaming stage can be saved first, and after the streaming is completed, the network transmission data can be analyzed to determine the time interval when the network quality meets the preset requirements, and the local video data can be intercepted based on the time interval to obtain the fragment data corresponding to the time interval, and sent to the server as original data.

[0043] In another implementation, the live broadcast user terminal can save local video data based on the time interval sent by the live broadcast control terminal during the streaming stage.

[0044] Specifically, the live broadcast control end is in the streaming stage during the live broadcast process. Based on this, the live broadcast control end can determine the time interval by obtaining the video quality during the streaming process.

[0045] In one example, the live broadcast control end can obtain the packet loss rate during the live broadcast process, and determine the period when the packet loss rate exceeds the second threshold as the time interval, and send it to the corresponding streaming end, so that the streaming end can save the local video data in the time interval. The second threshold can be obtained based on historical data analysis. When the packet loss rate exceeds the second threshold, the sound and picture quality of the live streaming data will obviously deteriorate.

[0046] In another example, the live broadcast control end can determine the time interval by receiving the original data saving instruction sent by the user, and send it to the corresponding streaming end. Specifically, the administrator user can monitor the live video quality in real time during the live broadcast. When the video display quality deteriorates or the audio and video are abnormally accelerated, the data saving instruction is input, and the live broadcast control end determines the time interval according to the time information specified in the data saving instruction or the sending time of the instruction.

[0047] For example, the start time in the time information specified in the instruction can be advanced by a preset time, and the end time can be delayed by a preset time, so as to expand the time period corresponding to the time information, thereby obtaining a time interval. In this way, the problem of not being able to obtain the required original data due to untimely user operation can be avoided to a large extent. In the case of determining the time interval based on the sending time of the instruction, the sending time can also be advanced by a preset time as the starting time of the time interval, and when the stop saving instruction sent by the user is received, the receiving time of the instruction is determined as the end time of the time interval.

[0048] In another example, the live broadcast control end may generate time intervals based on the above two examples at the same time and send them to the live broadcast user end. The live broadcast user end may determine the storage range of the local video data based on all the received time intervals.

[0049] The live broadcast client 120 is also used to record behavior data during the live broadcast process, wherein the behavior data includes actions of turning on or off the camera or microphone and corresponding time points, and sends it to the server 110 .

[0050] The live broadcast control terminal 130 is used to manage the live broadcast room. While creating access links corresponding to different live broadcast roles, the management background also creates an access link for accessing the live broadcast control terminal. The live broadcast room administrator can open the access link through the browser to enter the live broadcast control terminal, and then manage the live broadcast room based on the live broadcast control terminal, including starting or ending the live broadcast, selecting a screen layout template for the live broadcast room, setting the warm-up screen to be broadcast before the live broadcast starts, and saving the received live stream data during the live broadcast and sending it to the server.

[0051] The server 110 is used to provide a management background to realize the creation of a live broadcast room, and to generate high-quality live broadcast playback data for the live broadcast content of the live broadcast room for subsequent on-demand use.

[0052] Please refer to Figure 2 The live broadcast playback data generation method provided in the embodiment of the present application specifically includes the following steps.

[0053] S210, in response to the live playback recording instruction, obtaining live recording data and creating a target container.

[0054] In implementation, the live playback recording instruction may be input by the user through the live control terminal before or during the live broadcast, or may be automatically generated based on a set trigger condition.

[0055] In one example, the trigger condition includes generating a live playback recording instruction when the popularity of the live broadcast room exceeds a first popularity threshold, so as to instruct the server to determine the live broadcast period during the live broadcast in which the live broadcast popularity value exceeds a second popularity threshold as the target playback period, and generate playback data corresponding to the target playback period, wherein the first popularity threshold is greater than or equal to the second popularity threshold, and the specific value can be set according to the application requirements of the user or platform; the popularity of the live broadcast room can be determined according to the number of viewers in the live broadcast room.

[0056] The live playback recording instruction is used to trigger the server to generate live playback data and specify the target playback time period corresponding to the playback data. That is to say, the server can generate playback data for the entire live broadcast according to the live playback recording instruction, or it can generate playback data for a certain live broadcast time period or multiple live broadcast time periods.

[0057] Live broadcast data includes video data recorded in real time during the live broadcast. Figure 3 , the method for obtaining video data specifically includes the following steps.

[0058] S310, during or after the live broadcast, receiving original data uploaded by the live broadcast user terminal.

[0059] Among them, the original data is the local video data that the live broadcast user end obtains and stores in real time through the browser during the live broadcast streaming stage. The local video data is also used to generate a live stream based on the WebRTC protocol, so that the live broadcast user end can push the stream based on the live stream during the streaming stage.

[0060] It is worth noting that local video data is high-fidelity data directly obtained from local devices and generated through WebRTC live streaming. During the live broadcast process, since the WebRTC protocol will dynamically adjust the resolution and bit rate according to the network status to ensure real-time data transmission, the quality of the live audio and video presented based on the live streaming data will also change accordingly, resulting in blurred image quality, frame loss, freezes and other situations that affect the live viewing experience. If the playback data is recorded directly based on the live streaming data, the playback video will also have the same problems, thus affecting the viewing experience of on-demand users. Based on this, by obtaining the original data to generate the playback data, the playback quality of the playback video can be guaranteed.

[0061] S320, generating video data based on the original data.

[0062] In practice, since the playback data generation requirements of different live broadcast rooms are different, the corresponding raw data types are also different. Therefore, the server needs to first determine the type of raw data and perform corresponding processing according to the type of raw data to obtain video data. Please refer to Figure 4 In one implementation, the method for generating video data based on original data specifically includes the following steps.

[0063] S410, determining a time interval corresponding to the original data.

[0064] Among them, the time interval corresponding to the original data can be determined based on the start time and end time of the original data. In one example, the live broadcast user terminal can add a time mark to the original data when sending the original data to record the start time and end time corresponding to the original data. Based on this, the server can determine the corresponding time interval based on the time mark carried in the original data.

[0065] S420: Determine the type of the original data according to the time interval and the target playback period of the live playback data.

[0066] During implementation, the target playback period is determined based on the live playback recording instruction. If the time interval of all original data sent by each live user terminal in the live broadcast room can completely cover the target playback period, the type of the original data is determined to be complete data; if it cannot completely cover the target playback period, the type of the original data is determined to be fragment data.

[0067] S430, when the type of the original data is segment data, obtain live streaming data, and replace the segment corresponding to the time interval in the live streaming data with the original data to obtain video data.

[0068] The live streaming data is the streaming data pushed during the live broadcast process. In one example, the live streaming data is received and saved by the live broadcast control terminal during the live broadcast process. For example, the live broadcast control terminal can save all the live streaming data received during the live broadcast process, and extract the corresponding live streaming data according to the target playback period, or only save the live streaming data for the target playback period to directly obtain the live streaming data.

[0069] In another example, live streaming data can also be obtained based on a third-party platform, where the third-party platform is implemented based on the sound network technology. Specifically, in some application scenarios, WebRTC technology and sound network technology can be combined to adapt to different live scene requirements. For example, when there are a large number of viewers in the live broadcast room, low-latency interaction between the anchor and the audience can be achieved based on WebRTC technology, and the live stream can be distributed to a large number of viewers through the sound network technology. Based on this, after the live broadcast ends, the server can receive the m3u8 file uploaded by the sound network server and obtain the live streaming data by accessing the m3u8 file.

[0070] In implementation, there may be a situation where the live streaming data is inconsistent with the original data. Therefore, in the process of replacing the segment corresponding to the time interval in the live streaming data with the original data, the segment corresponding to the time interval needs to be determined first.

[0071] In one example, since the type of the original data is a data segment, it indicates that the time interval corresponding to the original data at least partially overlaps with the target playback period. Therefore, the overlapping endpoints of the time interval and the target playback period can be determined first, and the matching interval can be determined based on the overlapping endpoints. All key frames of the original data and the live stream data within the matching interval are obtained, and each key frame is compared based on image recognition technology to determine the key frames with the same picture content in the original data and the live stream data. Based on the same key frames, the original data replacement points and the live stream data replacement points are determined, and based on the original data replacement points, a replacement segment is generated, and the replacement interval is determined based on the live stream data replacement points, and the replacement segment is used to replace the corresponding segment of the replacement interval in the live stream data.

[0072] For example, the time interval of the original data is 12:00:00-12:10:01, and the target playback period includes 12:05:02-12:30:00. That is to say, the server determines according to the instruction that the live broadcast data of the period of 12:05:02-12:30:00 needs to be recorded, and the live broadcast user end saves and uploads the local video data of 12:00:00-12:10:01 according to the changes in network quality. At the same time, the live broadcast control end records the live video during 12:05:02-12:30:00. During the live broadcast process, WebRTC may drop frames or accelerate the live video, resulting in differences between the playback progress of the live video and the original data. Therefore, even if the time of the live broadcast user end and the live broadcast control end is synchronized, there may still be a situation where the original data and the live stream data in the same time period are not synchronized. For example, the original data at 12:10:01 is recorded at 12:10:03 seconds after transmission processing.

[0073] In order to avoid the situation where the playback progress is not smooth after the original data is directly replaced, 12:05:02 and 12:10:01 can be determined as overlapping endpoints, and the time difference range is 2 seconds. Taking the overlapping endpoint 12:10:01 as an example, 12:09:59-12:10:03 is determined as the matching interval, and the key frames of the original data and the live streaming data within 12:09:59-12:10:03 are obtained respectively, and compared to determine the time point with the same key frame, for example, the time point corresponding to the same key frame in the original data is 12:10:00, and the time point in the live streaming data is 12:10:01. Based on the same method, the replacement point 12:05:00 corresponding to the overlapping endpoint 12:05:02 in the original data can be determined.

[0074] Thus, when replacing, the time point corresponding to the same key frame in the original data can be determined as the replacement point of the original data, and intercepted to obtain the original data for replacement, and the time point corresponding to the same key frame in the live streaming data can be determined as the replacement point to determine the replacement segment. That is, the 12:05:00-12:10:00 segment in the original data is replaced with the 12:05:02-12:10:01 segment in the live streaming data to obtain the video data for generating playback data.

[0075] Based on this, by replacing the live streaming data with high-fidelity original data, the segments with poor audio and video quality in the live streaming data can be repaired, thereby ensuring the viewing experience of the playback data. In addition, by locating the replacement point based on the key frame, the audio and video smoothness of the playback data can be guaranteed.

[0076] S440: When the type of the original data is complete data, determine the original data as video data.

[0077] In implementation, when it is determined that the type of original data is complete data, the original data can be directly determined as video data, or the original data can be intercepted according to the target playback period to obtain video data that meets the requirements of the target playback period.

[0078] When the server obtains the live broadcast recording data, it creates a target container based on the task generated for the current live broadcast playback data. In one example, the creation of the target container can be implemented based on the Docker technology.

[0079] S220, starting a live broadcast client in the target container, and entering an automatic play page through the live broadcast client.

[0080] In implementation, after the target container is created, a browser is opened in the container by loading a predefined image file, and the automatic play page is entered through the browser. The automatic play page is a pre-created web page dedicated to recording live playback data. The automatic play page can realize automatic playback of video data and synchronous recording.

[0081] S230, playing and recording the live broadcast record data based on the automatic play page to generate live broadcast playback data.

[0082] In implementation, the auto-play page can automatically play the video data and record it to generate live playback data in MP4 format.

[0083] Based on the above technical solution, not only can the live data be converted into on-demand data, but also the audio and video quality of the on-demand data can be guaranteed by optimizing the live stream data using high-fidelity original data. In addition, in the process of video data processing, the replacement segment can be determined by matching key frames, which can effectively reduce the impact of segment replacement on the smoothness of audio and video.

[0084] In some live broadcasts, whiteboard tools are used to assist in displaying PPT, and to write and mark. However, the real-time information stream generated by some whiteboard tools is independent of the live broadcast stream. Therefore, the live broadcast stream data saved by the live broadcast control terminal does not include whiteboard-related display data. In order to achieve the synchronous display of the whiteboard screen during the on-demand process, in some embodiments of the present application, the live broadcast recording data also includes whiteboard data recorded in real time during the live broadcast process, wherein the whiteboard data can be obtained by saving the received whiteboard data by the live broadcast control terminal, for example, based on the decryption of the received SCTP (Stream Control Transmission Protocol) protocol packet. Correspondingly, the step of playing the live broadcast recording data also includes synchronously playing the video data and the whiteboard data based on the timestamp. It is worth noting that the playback time of the video data obtained by replacing the original data may have changed compared to the playback time of the live stream data. For example, when a 5-second original data is used to replace a 4-second live stream data, the corresponding playback time needs to be reorganized to keep the time information accurate. The whiteboard data has been synchronized with the live stream data during the live broadcast. Therefore, before synchronously playing the video data and the whiteboard data, the time of the whiteboard data and the video data needs to be aligned. The whiteboard data can be aligned with the timestamp of the whiteboard data according to each replacement point of the live stream data, and the corresponding whiteboard data in the replacement segment can be adapted, including extending or reducing the time of the static whiteboard image, so as to achieve the alignment of the playback time of the whiteboard data with the original data.

[0085] In order to completely restore the user's operation behavior during the live broadcast and display it synchronously, in some embodiments of the present application, each live broadcast user terminal also records the user's behavior data during the live broadcast and sends it to the server to be saved in the live broadcast record data. Therefore, the live broadcast record data also includes real-time recording of the behavior data during the live broadcast. Correspondingly, playing the live broadcast record data also includes playing the video data while reproducing the behavior data by running the script file. In one example, the behavior data is saved as a json file, and the JavaScript script in the automatic playback page will make corresponding state changes during the live broadcast according to the json file, so as to achieve the purpose of simulating the live broadcast screen.

[0086] In addition, an embodiment of the present application also provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a method as in any one of the implementations in the embodiments of the present application; wherein the processor may adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, for executing relevant programs to implement the method in any one of the implementations in the embodiments of the present application.

[0087] The processor may also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any implementation of the embodiments of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in the form of software.

[0088] The above-mentioned processor can also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined and executed.

[0089] The software module may be located in a random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines its hardware to complete the functions required to be performed by the units included in the data processing device of the embodiment of the present application, or executes the method in any one of the implementation modes in the embodiment of the present application.

[0090] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method when executed by a processor.

[0091] Those skilled in the art can understand that all or part of the steps in the above-mentioned implementation method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0092] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating live playback data, characterized in that: The method is applied to a server and comprises the steps of: In response to the live playback recording instruction, obtain live recording data and create a target container; Start a live broadcast client in the target container, and enter an automatic play page through the live broadcast client; The live broadcast record data is played and recorded based on the automatic play page to generate live broadcast playback data; wherein the live broadcast record data includes video data recorded in real time during the live broadcast process, and the video data is obtained by: During or after the live broadcast, receiving the original data uploaded by the live broadcast user end, wherein the original data is the local video data acquired and stored in real time by the live broadcast user end through the browser during the live broadcast streaming stage, and the local video data is also used to generate a live broadcast stream based on the WebRTC protocol, so that the live broadcast user end performs streaming based on the live broadcast stream during the live broadcast streaming stage; The video data is generated based on the raw data.

2. The method according to claim 1, characterized in that The method for generating the video data based on the original data comprises: Determine a time interval corresponding to the original data; Determining the type of the original data according to the time interval and the target playback period of the live playback data; If the type of the original data is segment data, live streaming data is acquired, and the segment corresponding to the time interval in the live streaming data is replaced with the original data to obtain the video data.

3. The method according to claim 2, characterized in that The method of replacing the segment corresponding to the time interval in the live streaming data with the original data to obtain the video data comprises: Determine the overlapping endpoints of the time interval and the target playback time period, determine the matching interval based on the overlapping endpoints, and obtain all key frames of the original data and the live streaming data within the matching interval; compare the key frames based on image recognition technology, determine the key frames with the same picture content in the original data and the live streaming data, and determine the original data replacement points and the live streaming data replacement points based on the same key frames; generate a replacement segment based on the original data replacement point, and determine the replacement interval based on the live streaming data replacement point; replace the corresponding segment of the replacement interval in the live streaming data with the replacement segment.

4. The method according to claim 1, characterized in that: The video data acquisition method further includes: After the live broadcast ends, receive the m3u8 file uploaded by the third-party platform, and obtain the live stream data by accessing the m3u8 file; The live streaming data is replaced based on the original data to obtain the video data.

5. The method according to claim 1, characterized in that The live broadcast record data also includes whiteboard data recorded in real time during the live broadcast process, and playing the live broadcast record data includes synchronously playing the video data and the whiteboard data based on a timestamp.

6. The method according to claim 1 or 5, characterized in that: The live broadcast record data also includes behavior data recorded in real time during the live broadcast process, and playing the live broadcast record data includes reproducing the behavior data by running a script file while playing the video data.

7. A live broadcast system, characterized in that: The live broadcast system includes a server, multiple live broadcast user terminals and a live broadcast control terminal, wherein the live broadcast user terminals and the live broadcast control terminal are implemented based on a browser and are communicatively connected to the server, and the server is used to generate live broadcast playback data based on the method described in any one of claims 1 to 6.

8. The system according to claim 7, characterized in that The live broadcast user terminal is used to obtain local video data in the live broadcast streaming stage, convert the local video into a live broadcast stream based on the WebRTC protocol, and push the stream. At the same time, based on a preset storage strategy, part or all of the local video data is saved locally, and in the live broadcast streaming stage or after the live broadcast ends, the saved local video data is sent to the server as the original data.

9. The system according to claim 8, characterized in that The live broadcast user terminal locally saves part of the local video data based on a preset storage strategy, including: obtaining network transmission data, and saving the local video data when the network transmission data indicates that the network quality meets the preset requirements; wherein the preset requirements are obtained based on an analysis of historical live broadcast data, and when the network quality meets the preset requirements, the live broadcast stream obtained by converting the local video data based on the WebRTC protocol has deteriorated audio and video quality.

10. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction implements the method according to any one of claims 1 to 6 when executed by the processor.

Images