A live streaming and microphone connection time axis playback method, device, equipment and medium

Abstract

The application relates to a live broadcast, a time axis playback method, device, equipment and medium. The method comprises the following steps: a server end starts an ffmpeg process to pull the client stream data in real time, synchronously generates recording and screenshot files of each client stream, and records and stores the related stream information of the live broadcast in real time; a host recording stream is dynamically loaded for live broadcast playback, the time axis of the live broadcast playback is the host recording time length, the screenshot serial number of the on-and-off microphone of the live broadcast is analyzed, a progress bar of the live broadcast playback is simulated, the microphone information is marked on the time axis based on the position of the progress bar, and a preview picture is generated; meanwhile, the microphone recording stream is jumped and synchronously played based on the position of the progress bar, and the playback of the whole live broadcast is realized through the adaptive adjustment of the playing position and the picture size. The method can meet the time axis synchronous playing of the multi-stream live broadcast, and improves the efficiency and accuracy of the live broadcast playback review.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network live broadcast, in particular to a live broadcast co-mic time axis playback method, device, equipment and medium. BACKGROUND

[0002] With the rapid development of Internet technology, especially the popularity of mobile Internet, network live broadcast has become a widely popular interactive form. In the live broadcast process, various interactive play methods have emerged, such as live broadcast co-mic and live broadcast PK. These play methods need to dynamically adjust the picture layout of the live broadcast room to adapt to the joining and leaving of different participants.

[0003] Since the picture layout of the live broadcast room will dynamically change with the user's up-mic and down-mic, when playing back the live broadcast content, the review platform needs to accurately restore the picture layout at different time points and synchronously load the live broadcast / recording broadcast streams of multiple users to ensure that the review content is consistent with the actual display at the time of live broadcast. For example, time axis synchronization: if user A up-mic at 10 minutes of anchor C's live broadcast, user B joins after 8 minutes of co-mic, and user B disconnects after 3 minutes of co-mic, the review platform needs to dynamically load the streams of user A, user B and anchor C according to the time axis and keep the playback synchronized. The picture consistency needs to be ensured to be completely consistent with the live broadcast, such as picture size, proportion, conversation order, etc., to avoid missing or misjudging due to display inadaptation.

[0004] Currently, a common implementation method is to mix streams on the server side, that is, to dynamically synthesize multiple video streams according to the up-mic and down-mic events of the client. However, mixing streams involves real-time encoding and synthesis of multiple audio and video streams, which consumes a lot of computing resources and requires high server performance. In addition, the mixing stream service needs to deal with complex audio and video synchronization, coding optimization and other problems, which is difficult to develop. Moreover, because there are frequent up-mic and down-mic and layout adjustment during co-mic, the mixing stream service needs to adjust the picture layout in real time, which is easy to cause event loss, stream processing exception, picture disorder and other problems. SUMMARY

[0005] Therefore, it is necessary to provide a live broadcast co-mic time axis playback method, device, equipment and medium to realize time axis synchronous playback of multiple streams and ensure accurate restoration of the live broadcast scene during review.

[0006] A live broadcast co-mic time axis playback method, the method comprising:

[0007] The server side starts the ffmpeg process to pull the client stream data in real time by receiving the start broadcasting information sent by the anchor client in the live broadcast room or the up-mic and down-mic information sent by the co-mic client, synchronously generates recording and screenshot files of each client stream, and records and stores the co-mic related stream information in the live broadcast co-mic process in real time;

[0008] The live broadcast playback is performed by dynamically loading the anchor recording broadcast stream, and the anchor recording time length is used as a time axis of the live broadcast playback. By analyzing the anchor and the co-host screenshot sequence numbers at the co-host on and off times in the co-host related stream information, the time offset of the co-host on and off times relative to the anchor recording stream and the time offset of the co-host on time relative to the co-host recording stream are calculated respectively. The progress bar of the live broadcast playback is simulated and generated by the time offset alignment, and the co-host information is marked on the time axis and the preview picture is generated based on the position of the progress bar.

[0009] The live broadcast playback time is obtained by detecting the position change of the progress bar in real time. When the live broadcast playback time reaches the time offset of the co-host on time relative to the anchor recording stream, the co-host recording stream is automatically loaded, and the co-host recording stream time is jumped to the time offset of the co-host on time relative to the co-host recording stream for synchronous playback. Meanwhile, the playback of the entire live broadcast is realized by adaptive adjustment of the playing position and the picture size.

[0010] In one of the embodiments, the ffmpeg process is started to pull the client stream data in real time, and the recording and screenshot files of each client stream are generated synchronously and regularly, including:

[0011] When the client stream data includes video stream and audio stream, the real-time client stream data is parsed into media data, and the recording and screenshot files with sequentially increasing sequence numbers are generated synchronously and regularly according to the preset recording time interval and screenshot time interval.

[0012] When the client stream data only contains audio stream, the ffmpeg process is restarted to generate recording files with sequentially increasing sequence numbers regularly according to the preset recording time interval, and the timestamp is recorded. Virtual id is generated synchronously and regularly according to the preset screenshot time interval, and it is detected in real time whether the video stream is restored. After the video stream is restored, the ffmpeg process starts to generate recording and screenshot files with sequentially increasing sequence numbers synchronously and regularly.

[0013] In one of the embodiments, the co-host related stream information in the live co-hosting process is recorded in real time, including:

[0014] When the server receives the on and off requests sent by the co-host client, the on and off events, the on time point, the anchor screenshot sequence number at the co-host on time , the co-host screenshot sequence number at the co-host on time , the anchor screenshot sequence number at the co-host off time , the co-host screenshot sequence number at the co-host off time , the anchor stream address, the co-host stream address, the co-hosting time length and the co-hosting sequence number are recorded as the co-host related stream information in the live co-hosting process, and are stored in the database.

[0015] In one of the embodiments, the server stores the screenshot sequence number of the anchor and the screenshot sequence number of the co-host when the co-host goes on air in the database directly if the screenshot sequence number has been generated at that time; otherwise, it stores NULL or a placeholder value first and starts a screenshot sequence number scanning at a fixed time, and updates the NULL or placeholder value in the database to start_imgid and start_link_imgid fields when the screenshot sequence number is scanned.

[0016] In one of the embodiments, the time offset of the co-host going on air and off air relative to the anchor live broadcast stream and the time offset of the co-host going on air relative to the co-host live broadcast stream are calculated by parsing the screenshot sequence numbers of the anchor and the co-host when the co-host goes on air and off air in the co-host related stream information, and a progress bar of the live broadcast playback is simulated by time offset alignment, and the co-host information is marked on the time axis based on the position of the progress bar and a preview image is generated, including:

[0017] According to the co-host duration and the anchor screenshot sequence number difference during the co-hosting, the real interval duration of the anchor screenshot during the co-hosting is calculated , which is represented as ;

[0018] According to the anchor screenshot sequence number and when the co-host goes on air, the position offset start time point of the co-host going on air relative to the anchor live broadcast stream is calculated , which is represented as ; according to the anchor screenshot sequence number and when the co-host goes off air, the position offset end time point of the co-host going off air relative to the anchor live broadcast stream is calculated , which is represented as ;

[0019] According to the co-host duration and the co-host screenshot sequence number difference during the co-hosting, the real interval duration of the co-host screenshot during the co-hosting is calculated , which is represented as ;

[0020] According to the co-host screenshot sequence number and when the co-host goes on air, the position offset start time point of the co-host going on air relative to the co-host live broadcast stream is calculated , which is represented as ;

[0021] In the manner of time offset alignment, that is As a time offset less than the anchor recording time, find the position of the anchor recording time as the time axis of the live broadcast playback Value alignment, simulate the progress bar of the live broadcast playback, and calculate the position of the progress bar as ; wherein, is the anchor recording time;

[0022] Mark the information of the mic-in at the time axis , and start the time point according to the position offset of the mic-in relative to the mic-in recording stream , generate a playback object, load the mic-in recording stream and jump to for playback when the marker point is clicked, and display the preview picture during the mic-in when the mouse hovers over the marker point;

[0023] Iterate through all the mic-in information in the live room, and repeat the above steps until all the mic-in information is marked on the time axis and the corresponding preview picture is generated.

[0024] In one embodiment, when the live broadcast playback time reaches the time offset of the mic-in relative to the anchor recording stream, the mic-in recording stream is automatically loaded, and the mic-in recording stream time is jumped to the time offset of the mic-in relative to the mic-in recording stream for synchronous playback. At the same time, through adaptive adjustment of the playback position and picture size, the playback of the entire live broadcast is realized, including:

[0025] When the live broadcast playback time is greater than or equal to , the mic-in recording stream is automatically loaded and the mic-in recording stream time is jumped to , a playback object is generated and synchronous playback is realized, and the playback object is placed in the foreground for playback;

[0026] When the live broadcast playback time is greater than or equal to , the playback object of the current mic-in is removed and destroyed, so that it is not visible in the foreground;

[0027] Get the playback objects and mic-in sequences of all mic-ins in the current live room, and adaptively adjust the playback object position and picture size to realize the playback of the entire live broadcast.

[0028] A time axis playback device for live mic-in, the device comprising:

[0029] A recording screenshot synchronization generation module for the server to start the ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the anchor client in the live room or the mic-in and mic-out information sent by the mic-in client, synchronously generate recording and screenshot files of each client stream, and record and store the mic-in related stream information in real time during the live mic-in process;

[0030] The mic sharing mark module is used for dynamically loading the anchor recording broadcast stream for live broadcast playback, and using the anchor recording time length as a time axis of the live broadcast playback. The mic sharing related stream information is analyzed to obtain the screenshot sequence numbers of the anchor and the mic sharer when the mic sharer goes on and off the mic. The time offset of the mic sharer relative to the anchor recording stream when the mic sharer goes on and off the mic and the time offset of the mic sharer relative to the mic sharer recording stream when the mic sharer goes on the mic are calculated respectively. The progress bar of the live broadcast playback is simulated and generated by the time offset alignment. The mic sharer information is marked on the time axis based on the position of the progress bar, and a preview image is generated.

[0031] The synchronous playing module is used for obtaining the live broadcast playback time by detecting the position change of the progress bar in real time. When the live broadcast playback time reaches the time offset of the mic sharer relative to the anchor recording stream when the mic sharer goes on the mic, the mic sharer recording stream is automatically loaded, and the time of the mic sharer recording stream is jumped to the time offset of the mic sharer relative to the mic sharer recording stream when the mic sharer goes on the mic for synchronous playing. Meanwhile, the playback of the whole live broadcast is realized by the adaptive adjustment of the playing position and the picture size.

[0032] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0033] The server side starts the ffmpeg process to pull the stream data of the client in real time by receiving the broadcasting information sent by the anchor client in the live broadcast room or the on and off mic information sent by the mic sharer client, synchronously generates the recording and screenshot files of each client stream in real time, and records and stores the mic sharing related stream information in the live broadcast process.

[0034] The mic sharing mark module is used for dynamically loading the anchor recording broadcast stream for live broadcast playback, and using the anchor recording time length as a time axis of the live broadcast playback. The mic sharing related stream information is analyzed to obtain the screenshot sequence numbers of the anchor and the mic sharer when the mic sharer goes on and off the mic. The time offset of the mic sharer relative to the anchor recording stream when the mic sharer goes on and off the mic and the time offset of the mic sharer relative to the mic sharer recording stream when the mic sharer goes on the mic are calculated respectively. The progress bar of the live broadcast playback is simulated and generated by the time offset alignment. The mic sharer information is marked on the time axis based on the position of the progress bar, and a preview image is generated.

[0035] The synchronous playing module is used for obtaining the live broadcast playback time by detecting the position change of the progress bar in real time. When the live broadcast playback time reaches the time offset of the mic sharer relative to the anchor recording stream when the mic sharer goes on the mic, the mic sharer recording stream is automatically loaded, and the time of the mic sharer recording stream is jumped to the time offset of the mic sharer relative to the mic sharer recording stream when the mic sharer goes on the mic for synchronous playing. Meanwhile, the playback of the whole live broadcast is realized by the adaptive adjustment of the playing position and the picture size.

[0036] A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the following steps:

[0037] The server end starts the ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the anchor client in the live room or the on / off microphone information sent by the microphone client, synchronously generates the recording and screenshot files of each client stream, and records and stores the microphone-related stream information in real time during the live microphone process.

[0038] The anchor recording stream is dynamically loaded for live playback, and the anchor recording time length is used as the time axis of the live playback. By analyzing the screenshot serial numbers of the anchor and the microphone user when the microphone user goes on and off the microphone, the time offset of the microphone user relative to the anchor recording stream when the microphone user goes on and off the microphone, and the time offset of the microphone user relative to the microphone user recording stream when the microphone user goes on the microphone are calculated respectively. The progress bar of the live playback is simulated by time offset alignment, and the microphone user information is marked on the time axis based on the position of the progress bar and a preview image is generated.

[0039] By detecting the position change of the progress bar in real time, the live playback time is obtained. When the live playback time reaches the time offset of the microphone user relative to the anchor recording stream when the microphone user goes on the microphone, the microphone user recording stream is automatically loaded, and the microphone user recording stream is synchronously played by jumping to the time offset of the microphone user relative to the microphone user recording stream when the microphone user goes on the microphone. At the same time, by adaptively adjusting the playing position and the picture size, the whole live playback is realized.

[0040] The above-mentioned time axis playback method, device, equipment and medium for live microphone, by synchronously generating the recording and screenshot files of each client stream, the screenshot serial numbers can correspond to the recording time, and the time axis synchronization between different recording streams can be realized, solving the problem that the audit personnel cannot continuously view the whole live microphone playback in the background. Compared with the existing server-side mixing, which requires a large server cost, the present application only stores the microphone-related stream information on the server end, has low requirements for server performance and development difficulty, is conducive to lightweight design, can simulate the generation of the progress bar of the live playback by analyzing the screenshot serial number relationship when the microphone user goes on and off the microphone, and can mark the microphone user information on the time axis based on the position of the progress bar and generate a preview image, so that the playing positioning is more accurate, and the audit personnel can directly observe the on / off microphone / pk situation of the whole live from the progress bar. Based on the progress bar, the microphone user recording stream can be quickly and accurately jumped and synchronously played, greatly improving the audit efficiency when auditing the live playback, avoiding the omission when viewing a single live playback, ensuring accurate reproduction of the live scene during auditing, and ensuring the consistency of the live playback picture through adaptive adjustment of the playing position and the picture. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 An application scenario diagram of a live microphone connection time axis playback method in an embodiment;

[0042] Figure 2 A flowchart of a live microphone connection time axis playback method in an embodiment;

[0043] Figure 3 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0044] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0045] A live microphone connection time axis playback method provided by the present application can be applied in an application environment as shown in Figure 1 . In the application environment, a client 102 communicates with a server 104 through a network. The client 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0046] In an embodiment, as shown in Figure 2 , a live microphone connection time axis playback method is provided. Taking the server in Figure 1 as an example, the method comprises the following steps:

[0047] In step 202, the server starts an ffmpeg process to pull the stream data of the client in real time by receiving the broadcasting start information sent by the host client in the live room or the on / off microphone information sent by the microphone user client, synchronously generates the recording and screenshot files of each client stream in real time, and records and stores the microphone related stream information in the live microphone connection process.

[0048] The ffmpeg process is an open source cross-platform audio and video processing framework composed of a series of libraries and command line tools, and is mainly used for multimedia processing tasks such as recording, transcoding, editing, streaming media transmission and the like. By synchronously generating the recording and screenshot files of each client in the same process, the screenshot serial number can correspond to the recording time, avoiding the audio and video out of synchronization due to processing delay, and realizing the time axis synchronization between different recording streams.

[0049] At step 204, the anchor recording and broadcasting stream is dynamically loaded for live playback, and the anchor recording time length is used as the time axis of the live playback. By analyzing the screenshot sequence numbers of the anchor and the co-host at the co-host's on and off times in the co-host related stream information, the time offset of the co-host relative to the anchor recording stream at the co-host's on and off times is calculated, and the time offset of the co-host relative to the co-host recording stream at the co-host's on time is calculated. The progress bar of the live playback is simulated by time offset alignment, and the co-host information is marked on the time axis based on the position of the progress bar and a preview image is generated.

[0050] It should be understood that, compared with the existing server-side mixed flow which requires a large server cost, the present application only stores the co-host related stream information on the server side, and the requirements for server performance and development difficulty are low, which is conducive to lightweight design. By analyzing the screenshot sequence number relationship at the on and off times in the co-host related stream information, the progress bar of the live playback can be simulated, and the co-host information can be marked on the time axis based on the position of the progress bar and a preview image can be generated, so that the auditor can see the on and off situation of the entire live room at a glance, greatly improving the accuracy of rule violation capture and playback positioning.

[0051] At step 206, the live playback time is obtained by detecting the position change of the progress bar in real time. When the live playback time reaches the time offset of the co-host relative to the anchor recording stream at the co-host's on time, the co-host recording stream is automatically loaded, and the co-host recording stream time is jumped to the time offset of the co-host relative to the co-host recording stream at the co-host's on time for synchronous playback. At the same time, through adaptive adjustment of the playback position and the picture size, the playback of the entire live broadcast is realized.

[0052] It should be understood that, based on the progress bar, the co-host recording stream can be quickly and accurately jumped and played synchronously, greatly speeding up the auditing efficiency during live playback, avoiding omissions during live playback, ensuring accurate reproduction of the live scene during auditing, and ensuring the consistency of the live playback picture through adaptive adjustment of the playback position and the picture size.

[0053] In summary, the above-mentioned live co-host time axis playback method has the advantages of server lightweight and low design cost, and can not only meet the basic multi-stream live co-host time axis synchronous playback, but also mark the on and off information of each co-host on the time axis, realize fast jump and preview, speed up the auditing efficiency and ensure accurate reproduction of the live scene during auditing.

[0054] In one of the embodiments, the server starts the ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the anchor client in the live room or the information sent by the client in the live room, and when the client stream data includes video stream and audio stream, the real-time client stream data is parsed into media data, and the recording and screenshot files with sequentially increasing serial numbers are generated synchronously and regularly according to the preset recording time interval and screenshot time interval. The recording format is ts file, m3u8 file, etc., and the screenshot format is jpg file. The ts file can be set to one per 5 seconds, and the screenshot file can be set to one per 2 seconds.

[0055] Considering that the real-time stream exists the case that the user closes the camera, when the client stream data only contains the audio stream, the ffmpeg process is restarted, and the recording file with sequentially increasing serial numbers is generated regularly according to the preset recording time interval, and the timestamp is recorded, and the virtual id is generated synchronously and regularly according to the preset screenshot time interval, and whether the video stream is restored is detected in real time, and after the video stream is restored, the ffmpeg process continues to generate the recording and screenshot files with sequentially increasing serial numbers synchronously and regularly. The virtual id can mark the period without video stream, so as to avoid the time disorder of the screenshot and recording after the video stream is restored.

[0056] In one of the embodiments, when the server receives the request for going on and off the mic sent by the client, the on and off mic event, the on-mic time point, the anchor screenshot serial number when the client goes on the mic , the client screenshot serial number when the client goes on the mic , the anchor screenshot serial number when the client goes off the mic , the client screenshot serial number when the client goes off the mic , the anchor stream address, the client stream address, the mic time and the mic serial number are recorded as the mic-related stream information in the live mic process, and are stored in the database redis.

[0057] In one of the embodiments, when the server records the anchor screenshot serial number and the client screenshot serial number when the client goes on the mic, if the screenshot serial number has been generated at this time, it is directly stored in the database; otherwise, it is first stored as NULL or a placeholder value, and the screenshot serial number is scanned regularly, and when the screenshot serial number is scanned, the NULL or placeholder value in the database is updated to the start_imgid and start_link_imgid fields.

[0058] In one of the embodiments, the logic for marking the client information on the timeline and generating the preview image is as follows:

[0059] 1. According to the mic time and the difference between the anchor screenshot serial numbers during the mic, the real interval time of the anchor screenshot during the mic is calculated , which is represented as .

[0060] 2、According to the anchor screenshot sequence number when the mic is connected and , the position offset start time point of the mic when the mic is connected relative to the anchor recording broadcast stream is calculated , expressed as ; according to the anchor screenshot sequence number when the mic is disconnected and , the position offset end time point of the mic when the mic is disconnected relative to the anchor recording broadcast stream is calculated , expressed as .

[0061] 3、According to the mic duration and the difference between the mic screenshot sequence numbers during the mic, the real interval duration of the mic screenshot during the mic is calculated , expressed as .

[0062] 4、According to the mic screenshot sequence number when the mic is connected and , the position offset start time point of the mic when the mic is connected relative to the mic recording broadcast stream is calculated , expressed as .

[0063] 5、Through time offset alignment, that is as a time offset less than the anchor recording duration, at the position of the anchor recording duration as the time axis of live broadcast playback, find value alignment, simulate the generation of live broadcast playback progress bar, and calculate the position of the progress bar as ; wherein is the anchor recording duration.

[0064] 6、Mark the mic information at the time axis , and generate a playback object according to the position offset start time point of the mic when the mic is connected relative to the mic recording broadcast stream , load the mic recording broadcast stream and jump to for playback when the mark point is clicked, and display the preview picture during the mic when the mouse hovers over the mark point.

[0065] 7、Iterate through all the mic information in the live room, and repeat the above 1-6 until all the mic information is marked on the time axis and the corresponding preview picture is generated.

[0066] It should be understood that although the screenshot file is generated according to the preset screenshot time interval in live broadcast, due to the influence of server performance on audio and video push-pull stream, the real screenshot interval length deviates from the preset value, so the real screenshot interval length is calculated according to the ratio between the live streaming time and the difference value between the screenshot serial numbers during live streaming, which is beneficial to accurately align the time axis and ensure that the screenshot and audio and video stream are strictly synchronized during live broadcast playback, so that the audit platform can accurately locate the occurrence time of the illegal content.

[0067] In one of the embodiments, when the live broadcast playback time is greater than or equal to , the live streaming of the live streamer is automatically loaded and the live streaming time of the live streamer is jumped to , the playback object is generated and synchronized playback is realized, and at the same time, the playback object is placed in the foreground for playback; when the live broadcast playback time is greater than or equal to , the playback object of the current live streamer is eliminated and destroyed, so that it is invisible in the foreground; the playback objects and the live streaming order of all live streamers in the current live broadcast room are obtained, and adaptive adjustment of the position and size of the playback object (such as: when a single person plays, the video occupies the whole canvas, and when two people play, the two streams are divided into the canvas, etc.) is performed, so as to realize the playback of the whole live broadcast.

[0068] In one of the embodiments, a time axis playback device for live streaming and live streaming is provided, which comprises:

[0069] The recording and broadcasting screenshot synchronous generation module is used for starting the ffmpeg process to pull the client stream data in real time through receiving the broadcasting information sent by the host client in the live broadcast room or the on and off microphone information sent by the live streamer client, synchronously generating the recording and broadcasting and screenshot files of each client stream, and recording and storing the live streaming related stream information in real time during the live streaming;

[0070] The live streaming marking module is used for dynamically loading the host recording stream for live broadcast playback, taking the host recording time as the time axis of live broadcast playback, calculating the time offset of the live streamer relative to the host recording stream when the live streamer turns on and off the microphone and the time offset of the live streamer relative to the live streamer recording stream when the live streamer turns on the microphone through analyzing the screenshot serial numbers of the host and the live streamer when the live streamer turns on and off the microphone in the live streaming related stream information, simulating the generation of the progress bar of live broadcast playback through time offset alignment, and marking the live streamer information on the time axis based on the position of the progress bar and generating a preview picture;

[0071] The synchronous playing module is configured to acquire a live broadcast playback time by detecting a position change of the progress bar in real time, and automatically load the live broadcast stream of the micer when the live broadcast playback time reaches a time offset of the micer's going online relative to the anchor's recording broadcast stream, and jump to the time offset of the micer's going online relative to the micer's recording broadcast stream to synchronously play the recording broadcast stream of the micer. Meanwhile, the synchronous playing module is configured to realize playback of the whole live broadcast by adaptively adjusting a playing position and a screen size.

[0072] The specific limitations of the live broadcast micer time axis playback device can be seen from the limitations of the live broadcast micer time axis playback method, which will not be repeated here. Each module in the live broadcast micer time axis playback device can be realized by software, hardware, and a combination thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0073] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 3 The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store data. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a live broadcast micer time axis playback method.

[0074] Those skilled in the art can understand that Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0075] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the following steps:

[0076] The server end starts an ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the host client or the on-off microphone information sent by the microphone client in the live room, synchronously generates the recording and screenshot files of each client stream, and records and stores the microphone related stream information in real time during the live microphone process.

[0077] The host recording stream is dynamically loaded for live playback, and the host recording time length is used as the time axis of the live playback. The screenshot sequence numbers of the host and the microphone during the on-off microphone of the microphone are analyzed, the time offset of the microphone relative to the host recording stream during the on-off microphone of the microphone is calculated, and the time offset of the microphone relative to the microphone recording stream during the on-microphone of the microphone is calculated. The progress bar of the live playback is simulated by time offset alignment, and the microphone information is marked on the time axis based on the position of the progress bar and a preview image is generated.

[0078] The position change of the progress bar is detected in real time to obtain the live playback time. When the live playback time reaches the time offset of the microphone relative to the host recording stream during the on-microphone of the microphone, the microphone recording stream is automatically loaded, and the microphone recording stream time is jumped to the time offset of the microphone relative to the microphone recording stream during the on-microphone of the microphone for synchronous playback. Meanwhile, adaptive adjustment of the playback position and the picture size realizes the playback of the entire live broadcast.

[0079] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the following steps:

[0080] The server end starts an ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the host client or the on-off microphone information sent by the microphone client in the live room, synchronously generates the recording and screenshot files of each client stream, and records and stores the microphone related stream information in real time during the live microphone process.

[0081] The host recording stream is dynamically loaded for live playback, and the host recording time length is used as the time axis of the live playback. The screenshot sequence numbers of the host and the microphone during the on-off microphone of the microphone are analyzed, the time offset of the microphone relative to the host recording stream during the on-off microphone of the microphone is calculated, and the time offset of the microphone relative to the microphone recording stream during the on-microphone of the microphone is calculated. The progress bar of the live playback is simulated by time offset alignment, and the microphone information is marked on the time axis based on the position of the progress bar and a preview image is generated.

[0082] By detecting the position change of the progress bar in real time, the live playback time is obtained, when the live playback time reaches the time offset of the micer's on-mic time relative to the anchor's recording stream, the micer's recording stream is automatically loaded, and the micer's recording stream time is jumped to the time offset of the micer's on-mic time relative to the micer's recording stream for synchronous playback, at the same time, through the adaptive adjustment of the playing position and the screen size, the playback of the whole live broadcast is realized.

[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM) and the like.

[0084] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0085] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application.

Claims

1. A live stream live-mic timeline playback method, characterized in that, The method comprises: The server starts the ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the host client in the live room or the information sent by the client who wants to join the live broadcast, synchronously generates the recording and screenshot files of each client stream, and records and stores the live broadcast and the related stream information in real time. The host recording stream is dynamically loaded for live broadcast playback, and the time axis of the live broadcast playback is the host recording time length. By analyzing the screenshot sequence numbers of the host and the client who wants to join the live broadcast when the client who wants to join the live broadcast is on or off, the time offset of the client who wants to join the live broadcast relative to the host recording stream when the client who wants to join the live broadcast is on or off, and the time offset of the client who wants to join the live broadcast relative to the client recording stream when the client who wants to join the live broadcast is on are calculated, the progress bar of the live broadcast playback is simulated by time offset alignment, and the client who wants to join the live broadcast information is marked on the time axis based on the position of the progress bar and a preview image is generated. By detecting the position change of the progress bar in real time, the live broadcast playback time is obtained. When the live broadcast playback time reaches the time offset of the client who wants to join the live broadcast relative to the host recording stream when the client who wants to join the live broadcast is on, the client recording stream is automatically loaded, and the client recording stream is synchronously played by jumping to the time offset of the client who wants to join the live broadcast relative to the client recording stream when the client who wants to join the live broadcast is on. Meanwhile, by adaptively adjusting the playing position and the picture size, the whole live broadcast is played back.

2. The live co-watch timeline playback method of claim 1, wherein, The ffmpeg process is started to pull the client stream data in real time, and the recording and screenshot files of each client stream are synchronously generated at regular intervals, including: When the client stream data includes video stream and audio stream, the real-time client stream data is parsed into media data, and the recording and screenshot files with sequentially increasing sequence numbers are synchronously generated at regular intervals according to the preset recording time interval and the screenshot time interval; When the client stream data only contains audio stream, the ffmpeg process is restarted, and the recording files with sequentially increasing sequence numbers are generated at regular intervals according to the preset recording time interval. At the same time, the timestamp is recorded, and the virtual id is synchronously generated at regular intervals according to the preset screenshot time interval. The video stream is detected in real time whether it is restored. After the video stream is restored, the ffmpeg process is started to synchronously generate the recording and screenshot files with sequentially increasing sequence numbers.

3. The live co-watch timeline playback method of claim 2, wherein, The live broadcast and the related stream information in the live broadcast and the related stream information in the live broadcast are recorded and stored in real time, including: The server records the on-mic event, the on-mic time point, the anchor screenshot sequence number when the on-mic user goes on-mic, the on-mic user screenshot sequence number when the on-mic user goes on-mic, the anchor screenshot sequence number when the on-mic user goes off-mic, the on-mic user screenshot sequence number when the on-mic user goes off-mic, the anchor stream address, the on-mic user stream address, the on-mic duration, and the on-mic sequence number as the on-mic related stream information in the live on-mic process, and stores them in the database. The server records the on-mic event, the on-mic time point, the anchor screenshot sequence number when the on-mic user goes on-mic, the on-mic user screenshot sequence number when the on-mic user goes on-mic, the anchor screenshot sequence number when the on-mic user goes off-mic, the on-mic user screenshot sequence number when the on-mic user goes off-mic, the anchor stream address, the on-mic user stream address, the on-mic duration, and the on-mic sequence number as the on-mic related stream information in the live on-mic process, and stores them in the database.​​​ 4. The live co-watch timeline playback method of claim 3, wherein, When the server records the host screenshot sequence number and the client who wants to join the live broadcast screenshot sequence number when the client who wants to join the live broadcast is on, if the screenshot sequence number has been generated at this time, it is directly stored in the database; otherwise, it is first stored as NULL or a placeholder value, and the screenshot sequence number is scanned at regular intervals. When the screenshot sequence number is scanned, the NULL or placeholder value in the database is updated to the start_imgid and start_link_imgid fields.

5. The live co-watch timeline playback method of claim 3, wherein, The time offset of the microphone user when going on and off the microphone relative to the anchor recording stream and the time offset of the microphone user when going on the microphone relative to the microphone user recording stream are calculated by analyzing the screenshot sequence numbers of the anchor and the microphone user when the microphone user goes on and off the microphone in the microphone related stream information, and a progress bar of the live broadcast playback is simulated by time offset alignment, and the microphone user information is marked on the time axis based on the position of the progress bar and a preview image is generated, including: According to the length of the microphone And the difference between the anchor screenshot sequence numbers during the microphone, the real interval length of the anchor screenshot during the microphone is calculated , expressed as ; According to the anchor screenshot sequence number when the mic is connected And , the position offset start time point of the mic when the mic is connected relative to the anchor recording broadcast stream is calculated , expressed as ; according to the anchor screenshot sequence number when the mic is disconnected And , the position offset end time point of the mic when the mic is disconnected relative to the anchor recording broadcast stream is calculated , expressed as ; According to the length of the microphone And the difference between the screenshot serial numbers of the microphone during the microphone, the real interval length of the screenshots of the microphone during the microphone is calculated , expressed as ; According to the sequence number of the screenshot of the speaker when the speaker goes online And , the position offset start time point of the speaker when the speaker goes online relative to the recording and broadcasting stream of the speaker is calculated , expressed as ; By means of time offset alignment, that is As a time offset less than the anchor recording length, at the position of the anchor recording length as the time axis of the live broadcast playback, find Value alignment, simulate the progress bar of live broadcast playback, and calculate the position of the progress bar as ; wherein The anchor recording length In the time axis The information of the co-host is marked, and the starting time point is offset according to the position of the co-host in the co-host recording stream when the co-host gets on the microphone The playing object is generated, the co-host recording stream is loaded and jumps to For playing at the marked point, and the preview picture during the co-host is displayed when the mouse hovers over the marked point; All microphone user information in the live broadcast room is traversed, and the above steps are repeated until all microphone user information is marked on the time axis and the corresponding preview image is generated.

6. The live co-watch timeline playback method of claim 5, wherein, When the live broadcast playback time reaches the time offset of the microphone user when going on the microphone relative to the anchor recording stream, the microphone user recording stream is automatically loaded and synchronized played by jumping to the time offset of the microphone user when going on the microphone relative to the microphone user recording stream, and the whole live broadcast playback is realized by adaptive adjustment of the playing position and the picture size. When the live playback time is greater than or equal to , the live playback stream of the host is automatically loaded, and the live playback stream time of the host is jumped to , a playback object is generated to realize synchronous playback, and the playback object is placed in the foreground for playing. When the live play back time is greater than or equal to , the playing object of the current mic is eliminated and destroyed, and is invisible in the foreground. The playing objects and the microphone sequence of all microphone users in the current live broadcast room are obtained, and adaptive adjustment of the playing object position and the picture size is performed to realize the whole live broadcast playback.

7. A live stream timeline playback apparatus, comprising: The device comprises: A recording and screenshot synchronization generation module for the server to start an ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the anchor client in the live broadcast room or the on and off microphone information sent by the microphone user client, synchronously generate recording and screenshot files of each client stream, and record and store the microphone related stream information in the live broadcast microphone process in real time; A microphone marking module for dynamically loading the anchor recording stream for live broadcast playback, taking the anchor recording time as the time axis of the live broadcast playback, analyzing the screenshot sequence numbers of the anchor and the microphone user when the microphone user goes on and off the microphone in the microphone related stream information, calculating the time offset of the microphone user when going on and off the microphone relative to the anchor recording stream and the time offset of the microphone user when going on the microphone relative to the microphone user recording stream, simulating the generation of the progress bar of the live broadcast playback by time offset alignment, and marking the microphone user information on the time axis based on the position of the progress bar and generating a preview image; A synchronous playing module for acquiring the live broadcast playback time by detecting the position change of the progress bar in real time, automatically loading the microphone user recording stream and synchronously playing it by jumping to the time offset of the microphone user when going on the microphone relative to the microphone user recording stream when the live broadcast playback time reaches the time offset of the microphone user when going on the microphone relative to the anchor recording stream, and realizing the whole live broadcast playback by adaptive adjustment of the playing position and the picture size.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 6.

Images