Timeline playback method and device for live broadcast microphone connection, equipment and medium

Abstract

The invention relates to a live broadcast microphone connection time axis playback method and device, equipment and a medium. The method comprises the steps that a server side starts an ffmpeg process to pull client stream data in real time, recording and broadcasting and screenshot files of client streams are synchronously and regularly generated, and microphone connection related stream information is recorded and stored in real time; dynamically loading the anchor recorded broadcast stream for live playback, taking the anchor recorded broadcast duration as a time axis of live playback, simulating and generating a progress bar of live playback by analyzing a screenshot sequence number of a microphone connector in microphone connection related stream information when the microphone connector gets on and off the microphone, marking the microphone connector information on the time axis based on the position of the progress bar, and generating a preview; and meanwhile, based on the position of the progress bar, the recorded broadcast stream of the microphone connector is skipped and synchronously played, and the playback of the whole live broadcast is realized through the self-adaptive adjustment of the playing position and the picture size. According to the method, time axis synchronous playing of multi-path stream live broadcast microphone connection can be met, and the live broadcast playback auditing efficiency and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network live broadcast, in particular to a live broadcast co-mic time axis playback method, device, equipment and medium. BACKGROUND

[0002] With the rapid development of Internet technology, especially the popularity of mobile Internet, network live broadcast has become a widely popular interactive form. In the live broadcast process, various interactive play methods have been derived, such as live broadcast co-mic and live broadcast PK. These play methods need to dynamically adjust the picture layout of the live broadcast room to adapt to the joining and leaving of different participants.

[0003] Since the picture layout of the live broadcast room will dynamically change with the user's up-mic and down-mic, when playing back the live broadcast content, the review platform needs to accurately restore the picture layout at different time points and synchronously load the live broadcast / recording broadcast streams of multiple users to ensure that the review content is consistent with the actual display at the time of live broadcast. For example, time axis synchronization: if user A up-mic at 10 minutes of anchor C's live broadcast, user B joins after 8 minutes of co-mic, and user B disconnects after 3 minutes of co-mic, the review platform needs to dynamically load the streams of user A, user B and anchor C according to the time axis and keep the playback synchronized. Picture consistency: when reviewing, the picture size, proportion, conversation order, etc. need to be completely consistent with those at the time of live broadcast to avoid missing or misjudging due to display inadaptation.

[0004] Currently, a common implementation method is to mix streams on the server side, that is, to dynamically synthesize multiple video streams according to the up-mic and down-mic events of the client. However, mixing streams involves real-time encoding and synthesis of multiple audio and video streams, which consumes a lot of computing resources and requires high server performance. Moreover, the mixing stream service needs to deal with complex audio and video synchronization, encoding optimization and other issues, which is difficult to develop. In addition, because there are frequent up-mic and down-mic and layout adjustment during co-mic, the mixing stream service needs to adjust the picture layout in real time, which may easily lead to event loss, stream processing exception, picture disorder and other problems. SUMMARY

[0005] Therefore, it is necessary to provide a live broadcast co-mic time axis playback method, device, equipment and medium to realize time axis synchronous playback of multiple streams and ensure accurate restoration of the live broadcast scene during review.

[0006] A live broadcast co-mic time axis playback method, the method comprising: The server side starts the ffmpeg process to pull the client stream data in real time by receiving the start broadcasting information sent by the anchor client in the live broadcast room or the up-mic and down-mic information sent by the co-mic client, synchronously generates recording and screenshot files of each client stream, and records and stores the co-mic related stream information in the live broadcast co-mic process in real time; The live broadcast playback is performed by dynamically loading the anchor recording broadcast stream, and the anchor recording time length is used as a time axis of the live broadcast playback. By analyzing the anchor and the co-host screenshot sequence numbers at the co-host on and off times in the co-host related stream information, the time offset of the co-host on and off times relative to the anchor recording stream and the time offset of the co-host on time relative to the co-host recording stream are calculated respectively. The progress bar of the live broadcast playback is simulated and generated by the time offset alignment, and the co-host information is marked on the time axis and the preview picture is generated based on the position of the progress bar. The live broadcast playback time is obtained by detecting the position change of the progress bar in real time. When the live broadcast playback time reaches the time offset of the co-host on time relative to the anchor recording stream, the co-host recording stream is automatically loaded, and the co-host recording stream time is jumped to the time offset of the co-host on time relative to the co-host recording stream for synchronous playback. Meanwhile, the playback of the entire live broadcast is realized by adaptive adjustment of the playing position and the picture size.

[0007] In one of the embodiments, the ffmpeg process is started to pull the client stream data in real time, and the recording and screenshot files of each client stream are generated synchronously and regularly, including: The ffmpeg process is started to pull the client stream data in real time. When the client stream data includes video stream and audio stream, the real-time client stream data is parsed into media data, and the recording and screenshot files with sequentially increasing sequence numbers are generated synchronously and regularly according to the preset recording time interval and screenshot time interval. When the client stream data only contains audio stream, the ffmpeg process is restarted to generate recording files with sequentially increasing sequence numbers regularly according to the preset recording time interval, and the timestamp is recorded. The virtual id is generated synchronously and regularly according to the preset screenshot time interval, and it is detected in real time whether the video stream is restored. After the video stream is restored, the ffmpeg process starts to generate recording and screenshot files with sequentially increasing sequence numbers synchronously and regularly.

[0008] In one of the embodiments, the co-host related stream information in the live co-hosting process is recorded in real time, including: When the co-host client sends the on and off requests, the server records the on and off events, the on time point, the anchor screenshot sequence number at the co-host on time , the co-host screenshot sequence number at the co-host on time , the anchor screenshot sequence number at the co-host off time , the co-host screenshot sequence number at the co-host off time , the anchor stream address, the co-host stream address, the co-hosting time length, and the co-hosting sequence as the co-host related stream information in the live co-hosting process, and stores them in the database.

[0009] In one of the embodiments, the server stores the screenshot sequence number of the anchor and the screenshot sequence number of the co-host when the co-host goes on air in the database directly if the screenshot sequence number has been generated at that time; otherwise, it stores as NULL or a placeholder value first and starts a screenshot sequence number timing scan. When the screenshot sequence number is scanned, the NULL or placeholder value in the database is updated to the start_imgid and start_link_imgid fields.

[0010] In one of the embodiments, by parsing the screenshot sequence numbers of the anchor and the co-host when the co-host goes on and off air in the co-host related stream information, the time offset of the co-host relative to the anchor live broadcast stream when the co-host goes on and off air is calculated respectively, and the time offset of the co-host relative to the co-host live broadcast stream when the co-host goes on air is calculated. In the manner of time offset alignment, the progress bar of the live broadcast playback is simulated to generate, and the co-host information is marked on the time axis based on the position of the progress bar and a preview image is generated, including: According to the co-host duration and the difference between the anchor screenshot sequence numbers during the co-hosting, the real interval duration of the anchor screenshot during the co-hosting is calculated , which is represented as ; According to the anchor screenshot sequence number when the co-host goes on air and , the position offset start time point of the co-host relative to the anchor live broadcast stream when the co-host goes on air is calculated , which is represented as ; according to the anchor screenshot sequence number when the co-host goes off air and , the position offset end time point of the co-host relative to the anchor live broadcast stream when the co-host goes off air is calculated , which is represented as ; According to the co-host duration and the difference between the co-host screenshot sequence numbers during the co-hosting, the real interval duration of the co-host screenshot during the co-hosting is calculated , which is represented as ; According to the co-host screenshot sequence number when the co-host goes on air and , the position offset start time point of the co-host relative to the co-host live broadcast stream when the co-host goes on air is calculated , which is represented as ; In the manner of time offset alignment, that is as a time offset less than the anchor live broadcast duration, the value alignment is found on the position of the anchor live broadcast duration as the time axis of the live broadcast playback , the progress bar of the live broadcast playback is simulated to generate, and the position of the progress bar is calculated as ; wherein, recording time of the host; In the timeline mark the information of the mic-in user, and start the time point according to the position offset of the mic-in time of the mic-in user relative to the recording stream of the mic-in user generate a play object, load the recording stream of the mic-in user and jump to for playing when the marker is clicked, and display a preview picture during the mic-in when the mouse hovers over the marker; repeat the above steps until all the mic-in user information is marked on the timeline and the corresponding preview picture is generated.

[0011] In one of the embodiments, when the live broadcast playback time reaches the time offset of the mic-in time of the mic-in user relative to the recording stream of the host, the recording stream of the mic-in user is automatically loaded, and the recording stream of the mic-in user is time-jumped to the time offset of the mic-in time of the mic-in user relative to the recording stream of the mic-in user for synchronous playing. Meanwhile, through adaptive adjustment of the playing position and the picture size, the playback of the whole live broadcast is realized, including: When the live broadcast playback time is greater than or equal to , the recording stream of the mic-in user is automatically loaded and time-jumped to , a play object is generated and synchronous playing is realized, and the play object is placed in the foreground for playing; When the live broadcast playback time is greater than or equal to , the play object of the current mic-in user is eliminated and destroyed, so that it is invisible in the foreground; Get the play objects and mic-in sequences of all the mic-in users in the current live broadcast room, and adaptively adjust the position and size of the play objects to realize the playback of the whole live broadcast.

[0012] A timeline playback device for live broadcast mic-in, the device comprising: A recording screenshot synchronous generation module for starting an ffmpeg process to pull the stream data of the client in real time by receiving the broadcasting information sent by the host client in the live broadcast room or the mic-in and mic-out information sent by the mic-in client, synchronously generating the recording and screenshot files of each client stream, and recording and storing the mic-in related stream information in real time during the live broadcast mic-in process; A mic-in marking module for dynamically loading the recording stream of the host for live broadcast playback, taking the recording time of the host as the timeline of the live broadcast playback, calculating the time offset of the mic-in time of the mic-in user relative to the recording stream of the host and the time offset of the mic-in time of the mic-in user relative to the recording stream of the mic-in user by analyzing the screenshot serial numbers of the host and the mic-in user at the mic-in and mic-out times of the mic-in user in the mic-in related stream information, simulating the generation of the progress bar of the live broadcast playback by time offset alignment, and marking the information of the mic-in user on the timeline and generating a preview picture based on the position of the progress bar. The synchronous playing module is configured to acquire a live broadcast playback time by detecting a position change of the progress bar in real time, automatically load a live broadcast stream of the co-host when the live broadcast playback time reaches a time offset of the co-host going online relative to the anchor recording broadcast stream, and synchronize playing of the live broadcast stream of the co-host by jumping to the time offset of the co-host going online relative to the live broadcast stream of the co-host, while realizing playback of the whole live broadcast through adaptive adjustment of a playing position and a picture size.

[0013] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: The server end starts an ffmpeg process to pull client stream data in real time by receiving broadcasting information sent by an anchor client or going online / offline information sent by a co-host client in a live broadcast room, synchronously generates recording and screenshot files of each client stream, and records and stores co-host related stream information in a live broadcast co-host process in real time; The live broadcast stream of the anchor is dynamically loaded for live broadcast playback, and a recording time length of the anchor is used as a time axis of the live broadcast playback, the screenshot serial numbers of the anchor and the co-host when the co-host goes online / offline in the co-host related stream information are analyzed, the time offset of the co-host going online / offline relative to the live broadcast stream of the anchor and the time offset of the co-host going online relative to the live broadcast stream of the co-host are calculated respectively, the progress bar of the live broadcast playback is simulated by time offset alignment, and the co-host information is marked on the time axis based on the position of the progress bar and a preview picture is generated; The live broadcast playback time is acquired by detecting a position change of the progress bar in real time, the live broadcast stream of the co-host is automatically loaded when the live broadcast playback time reaches the time offset of the co-host going online relative to the live broadcast stream of the anchor, and the live broadcast stream of the co-host is synchronized and played by jumping to the time offset of the co-host going online relative to the live broadcast stream of the co-host, while realizing playback of the whole live broadcast through adaptive adjustment of a playing position and a picture size.

[0014] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps: The server end starts an ffmpeg process to pull client stream data in real time by receiving broadcasting information sent by an anchor client or going online / offline information sent by a co-host client in a live broadcast room, synchronously generates recording and screenshot files of each client stream, and records and stores co-host related stream information in a live broadcast co-host process in real time; The dynamic loading of the anchor recording and broadcasting stream is used for live broadcast playback, and the anchor recording time is used as the time axis of the live broadcast playback. By analyzing the screenshot sequence numbers of the anchor and the co-host at the time of the co-host going on and off the microphone in the co-host related stream information, the time offset of the co-host relative to the anchor recording stream at the time of the co-host going on and off the microphone is calculated, and the time offset of the co-host relative to the co-host recording stream at the time of the co-host going on the microphone is calculated. The progress bar of the live broadcast playback is simulated and generated by the time offset alignment, and the co-host information is marked on the time axis based on the position of the progress bar and a preview image is generated. By detecting the position change of the progress bar in real time, the live broadcast playback time is obtained. When the live broadcast playback time reaches the time offset of the co-host relative to the anchor recording stream at the time of the co-host going on the microphone, the co-host recording stream is automatically loaded, and the co-host recording stream is synchronized and played by jumping to the time offset of the co-host relative to the co-host recording stream at the time of the co-host going on the microphone. At the same time, the adaptive adjustment of the playing position and the picture size is realized to realize the playback of the entire live broadcast.

[0015] The above-mentioned live broadcast co-host time axis playback method, device, equipment and medium synchronously generate recording and broadcasting of each client stream and screenshot files at regular intervals, so that the screenshot sequence numbers can correspond to the recording time, and the time axis synchronization between different recording streams can be realized, solving the pain point problem that the audit personnel cannot continuously view the entire live broadcast co-host playback in the background. Compared with the existing server mixing, a great server cost is required. The present application only stores the co-host related stream information on the server side, has low requirements for server performance and development difficulty, is conducive to lightweight design, can simulate the generation of the progress bar of the live broadcast playback by analyzing the screenshot sequence number relationship at the time of going on and off the microphone in the co-host related stream information, and can mark the co-host information on the time axis based on the position of the progress bar and generate a preview image, so that the playing positioning is more accurate, and the audit personnel can directly observe the going on and off the microphone / pk situation of the entire live broadcast from the progress bar. Based on the progress bar, the co-host recording stream can be quickly and accurately jumped and synchronized and played, greatly improving the audit efficiency when auditing the live broadcast playback, avoiding the omission when viewing a single live broadcast playback, ensuring accurate restoration of the live broadcast scene during auditing, and ensuring the consistency of the live broadcast playback picture through adaptive adjustment of the playing position and the picture. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The application scenario diagram of the live broadcast co-host time axis playback method in one embodiment; Figure 2 The flowchart of the live broadcast co-host time axis playback method in one embodiment; Figure 3 The internal structure diagram of the computer equipment in one embodiment. DETAILED DESCRIPTION

[0017] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0018] The live microphone connection time axis playback method provided by the present application can be applied to the application environment as shown in Figure 1 . The client 102 communicates with the server 104 through the network. The client 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0019] In one embodiment, as shown in Figure 2 , a live microphone connection time axis playback method is provided. The method is described below by taking the server in Figure 1 as an example, which includes the following steps: Step 202, the server starts the ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the host client in the live room or the on-off microphone information sent by the microphone client, synchronously generates the recording and screenshot files of each client stream, and records and stores the microphone related stream information in real time during the live microphone connection process.

[0020] The ffmpeg process is an open source cross-platform audio and video processing framework composed of a series of libraries and command line tools, which is mainly used for multimedia processing tasks such as audio and video recording, transcoding, editing, streaming media transmission, etc. By synchronously generating the recording and screenshot files of each client through the same process, the screenshot sequence number can correspond to the recording time, avoiding the audio and video out of sync caused by processing delay, and realizing the time axis synchronization between different recording streams.

[0021] Step 204, dynamically load the host recording stream for live playback, and use the host recording time length as the time axis of live playback. By analyzing the screenshot sequence numbers of the host and the microphone during the microphone on-off of the microphone related stream information, the time offset of the microphone relative to the host recording stream during the microphone on-off of the microphone, and the time offset of the microphone relative to the microphone recording stream during the microphone on of the microphone are calculated respectively. The progress bar of live playback is simulated by time offset alignment, and the microphone information is marked on the time axis and the preview picture is generated based on the position of the progress bar.

[0022] It should be understood that, compared with the existing server-side mixed flow, the application only stores the relevant information of the server-side mixed flow, which has low requirements for server performance and development difficulty, is conducive to lightweight design, and can simulate the generation of the progress bar of the live broadcast playback by analyzing the screenshot number relationship at the time of going online and offline, and then mark the information of the online and offline person on the time axis based on the position of the progress bar and generate a preview image, so that the auditor can see the online and offline situation of the entire live broadcast room at a glance, which greatly improves the accuracy of the violation capture and playback positioning.

[0023] In step 206, the live broadcast playback time is obtained by detecting the position change of the progress bar in real time. When the live broadcast playback time reaches the time offset of the online person going online relative to the anchor recording and broadcasting stream, the online person recording and broadcasting stream is automatically loaded, and the online person recording and broadcasting stream time is jumped to the time offset of the online person going online relative to the online person recording and broadcasting stream for synchronous playback. At the same time, through adaptive adjustment of the playback position and the picture size, the playback of the entire live broadcast is realized.

[0024] It should be understood that, based on the progress bar, the online person recording and broadcasting stream can be quickly and accurately jumped and played synchronously, which greatly speeds up the auditing efficiency during live broadcast playback, avoids missing during live broadcast playback, ensures accurate restoration of the live broadcast scene during auditing, and ensures the consistency of the live broadcast playback picture through adaptive adjustment of the playback position and the picture size.

[0025] In summary, the above-mentioned live broadcast online time axis playback method has the advantages of server lightweight and low design cost, and can not only meet the basic multi-stream live broadcast online time axis synchronous playback, but also mark the online and offline information of each online person on the time axis, realize fast jump and preview, speed up the auditing efficiency, and ensure accurate restoration of the live broadcast scene during auditing.

[0026] In one of the embodiments, the server side starts the ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the anchor client in the live broadcast room or the online and offline information sent by the online person client. When the client stream data includes video stream and audio stream, the real-time client stream data is parsed into media data, and the recording and broadcasting and screenshot files with sequentially increasing serial numbers are generated synchronously and regularly according to the preset recording and broadcasting time interval and screenshot time interval. The recording and broadcasting format is ts file, m3u8 file, etc., and the screenshot format is jpg file. The ts file can be set to 5 seconds per file, and the screenshot file can be set to 2 seconds per file.

[0027] Considering that there is a case of user closing the camera in real-time stream, and the client stream data only contains audio stream, the ffmpeg process is restarted, and the recording file with continuously increasing serial number is generated at preset recording time interval, the timestamp is recorded, the virtual id is generated at preset screenshot time interval, and it is detected in real time whether the video stream is restored. When the video stream is restored, the ffmpeg process continues to generate the recording and screenshot file with continuously increasing serial number at the same time. The virtual id can mark the period without video stream, so as to avoid time disorder of screenshot and recording after the video stream is restored.

[0028] In one of the embodiments, the server records the following information as the live streaming related information during the live streaming process when receiving the request for going on and off the microphone sent by the client of the co-streamer: the on and off microphone event, the on-microphone time point, the anchor screenshot serial number when the co-streamer goes on the microphone , the co-streamer screenshot serial number when the co-streamer goes on the microphone , the anchor screenshot serial number when the co-streamer goes off the microphone , the co-streamer screenshot serial number when the co-streamer goes off the microphone , the anchor stream address, the co-streamer stream address, the co-streaming time length, and the co-streaming serial number, and stores them in the database redis.

[0029] In one of the embodiments, when recording the anchor screenshot serial number and the co-streamer screenshot serial number when the co-streamer goes on the microphone, if the screenshot serial number has been generated at this time, it is directly stored in the database; otherwise, it is first stored as NULL or a placeholder value, and the screenshot serial number is scanned at a timing, and when the screenshot serial number is scanned, the NULL or placeholder value in the database is updated to the start_imgid and start_link_imgid fields.

[0030] In one of the embodiments, the logic for marking the co-streamer information on the timeline and generating the preview image is as follows: 1. According to the co-streaming time length and the anchor screenshot serial number difference during the co-streaming, the real interval time length of the anchor screenshot during the co-streaming is calculated , which is represented as .

[0031] 2. According to the anchor screenshot serial number and when the co-streamer goes on the microphone, the position offset start time point of the co-streamer going on the microphone relative to the anchor recording stream is calculated , which is represented as ; according to the anchor screenshot serial number and when the co-streamer goes off the microphone, the position offset end time point of the co-streamer going off the microphone relative to the anchor recording stream is calculated , which is represented as .

[0032] 3、According to the length of the mic and the difference between the screenshot sequence numbers of the mic during the mic, the real interval length of the screenshots of the mic during the mic is calculated , which is represented as .

[0033] 4、According to the screenshot sequence number of the mic when the mic is on and , the position offset start time point of the mic when the mic is on relative to the recording stream of the mic is calculated , which is represented as .

[0034] 5、Through time offset alignment, that is as a time offset less than the length of the anchor recording, at the position of the anchor recording length as the time axis of live broadcast playback, find value alignment, simulate the generation of live broadcast playback progress bar, and calculate the position of the progress bar as ; wherein is the length of the anchor recording.

[0035] 6、Mark the mic information at the time axis , and generate a playback object according to the position offset start time point of the mic when the mic is on relative to the recording stream of the mic , load the recording stream of the mic and jump to for playback when the marker is clicked, and display the preview picture during the mic when the mouse hovers over the marker.

[0036] 7、Iterate through all the mic information in the live room and repeat the above 1-6 until all the mic information is marked on the time axis and the corresponding preview picture is generated.

[0037] It should be understood that although the screenshot file is generated according to the preset screenshot time interval during live broadcast, due to the influence of server performance on audio and video push-pull stream, there is a deviation between the real screenshot interval length and the preset value, so according to the ratio between the length of the mic and the difference between the screenshot sequence numbers during the mic, the real screenshot interval length is calculated, which is beneficial to accurately align the time axis and ensure that the screenshot and audio and video stream are strictly synchronized during live broadcast playback, so that the audit platform can accurately locate the time when the violation occurs.

[0038] In one of the embodiments, when the live broadcast playback time is greater than or equal to , the recording stream of the mic is automatically loaded and the time of the recording stream of the mic is jumped to , a playback object is generated and synchronous playback is realized, and the playback object is placed in the foreground for playback; when the live broadcast playback time is greater than or equal to When the current microphone is removed, the playing object of the current microphone is removed and destroyed, and the playing object is invisible in the foreground. The playing object and the microphone sequence of all microphones in the current live room are obtained, and the position and size of the playing object are adaptively adjusted (for example, when a single person plays, the video occupies the entire canvas, and when two people play, the two streams are divided into the canvas). The playback of the entire live broadcast is realized.

[0039] In one embodiment, a live microphone time axis playback device is provided, comprising: The recording and broadcasting screenshot synchronization generation module is configured to start an ffmpeg process to pull client stream data in real time on the server side by receiving the broadcasting information sent by the anchor client in the live room or the on and off microphone information sent by the microphone client, synchronously generate recording and broadcasting and screenshot files of each client stream, and record and store the microphone related stream information in real time during the live microphone process. The microphone marking module is configured to dynamically load the anchor recording and broadcasting stream for live playback, use the anchor recording time as the time axis of the live playback, analyze the screenshot numbers of the anchor and the microphone when the microphone is on and off in the microphone related stream information, calculate the time offset of the microphone relative to the anchor recording stream when the microphone is on and off, and the time offset of the microphone relative to the microphone recording stream when the microphone is on, simulate the generation of the progress bar of the live playback through the time offset alignment, and mark the microphone information on the time axis based on the position of the progress bar and generate a preview picture. The synchronous playback module is configured to detect the position change of the progress bar in real time, obtain the live playback time, automatically load the microphone recording and broadcasting stream when the live playback time reaches the time offset of the microphone relative to the anchor recording stream when the microphone is on, and jump to the time offset of the microphone recording and broadcasting stream when the microphone is on to synchronize the playback of the microphone recording and broadcasting stream. At the same time, through adaptive adjustment of the playing position and the picture size, the playback of the entire live broadcast is realized.

[0040] For specific limitations of a live microphone time axis playback device, refer to the limitations of a live microphone time axis playback method in the foregoing, which will not be repeated here. Each module in the live microphone time axis playback device described above can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0041] In one embodiment, a computer device, which can be a server, has an internal structure diagram as shown in Figure 3As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to realize a live streaming and microphone connection time axis playback method.

[0042] Those skilled in the art can understand that, Figure 3 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0043] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps: The server side starts the ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the host client in the live streaming room or the on-off microphone information sent by the microphone client, synchronously generates the recording and screenshot files of each client stream, and records and stores the microphone related stream information in the live streaming microphone process in real time; The host recording stream is dynamically loaded for live streaming playback, and the host recording time length is used as the time axis of live streaming playback. By analyzing the screenshot sequence numbers of the host and the microphone during the microphone on-off of the microphone related stream information, the time offset of the microphone relative to the host recording stream during the microphone on-off of the microphone, and the time offset of the microphone relative to the microphone recording stream during the microphone on of the microphone are calculated respectively. The progress bar of live streaming playback is simulated by time offset alignment, and the microphone information is marked on the time axis based on the position of the progress bar and a preview picture is generated; By detecting the position change of the progress bar in real time, the live streaming playback time is obtained. When the live streaming playback time reaches the time offset of the microphone relative to the host recording stream during the microphone on of the microphone, the microphone recording stream is automatically loaded, and the microphone recording stream time is jumped to the time offset of the microphone relative to the microphone recording stream during the microphone on of the microphone for synchronous playback. At the same time, through adaptive adjustment of the playing position and the picture size, the playback of the whole live streaming is realized.

[0044] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps: The server end starts the ffmpeg process to pull the client stream data in real time by receiving the starting broadcasting information sent by the anchor client in the live room or the on-off microphone information sent by the microphone client, synchronously generates the recording and screenshot files of each client stream in real time, and records and stores the microphone related stream information in the live microphone process; The anchor recording stream is dynamically loaded for live playback, and the anchor recording time length is used as the time axis of the live playback. By analyzing the anchor and microphone screenshot serial numbers when the microphone is on and off in the microphone related stream information, the time offset of the microphone relative to the anchor recording stream when the microphone is on and off, and the time offset of the microphone relative to the microphone recording stream when the microphone is on are calculated respectively. The progress bar of the live playback is simulated by time offset alignment, and the microphone information is marked on the time axis based on the position of the progress bar and a preview image is generated. By detecting the position change of the progress bar in real time, the live playback time is obtained. When the live playback time reaches the time offset of the microphone relative to the anchor recording stream when the microphone is on, the microphone recording stream is automatically loaded, and the microphone recording stream time is jumped to the time offset of the microphone relative to the microphone recording stream when the microphone is on for synchronous playback. At the same time, through adaptive adjustment of the playing position and the picture size, the whole live playback is realized.

[0045] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0046] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of the technical features is considered to be within the scope of the present disclosure.

[0047] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the protection scope of the present application.

Claims

1. A live stream live-mic timeline playback method, characterized in that, The method comprises: The server starts the ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the host client in the live room or the information sent by the client who wants to join the live broadcast, synchronously generates the recording and screenshot files of each client stream, and records and stores the live broadcast and the related stream information in real time. The host recording stream is dynamically loaded for live broadcast playback, and the time axis of the live broadcast playback is the host recording time length. By analyzing the screenshot sequence numbers of the host and the client who wants to join the live broadcast when the client who wants to join the live broadcast is on or off, the time offset of the client who wants to join the live broadcast relative to the host recording stream when the client who wants to join the live broadcast is on or off, and the time offset of the client who wants to join the live broadcast relative to the recording stream of the client who wants to join the live broadcast when the client who wants to join the live broadcast is on are calculated, the progress bar of the live broadcast playback is simulated by time offset alignment, and the client who wants to join the live broadcast information is marked on the time axis based on the position of the progress bar and a preview image is generated. By detecting the position change of the progress bar in real time, the live broadcast playback time is obtained. When the live broadcast playback time reaches the time offset of the client who wants to join the live broadcast relative to the host recording stream when the client who wants to join the live broadcast is on, the recording stream of the client who wants to join the live broadcast is automatically loaded, and the recording stream of the client who wants to join the live broadcast is synchronized and played by jumping to the time offset of the client who wants to join the live broadcast relative to the recording stream of the client who wants to join the live broadcast when the client who wants to join the live broadcast is on. Meanwhile, by adaptively adjusting the playing position and the picture size, the whole live broadcast is played back.

2. The live co-watch timeline playback method of claim 1, wherein, The ffmpeg process is started to pull the client stream data in real time, and the recording and screenshot files of each client stream are synchronously generated at regular intervals, including: When the client stream data includes video stream and audio stream, the real-time client stream data is parsed into media data, and the recording and screenshot files with sequentially increasing sequence numbers are synchronously generated at regular intervals according to the preset recording time interval and the screenshot time interval; When the client stream data only contains audio stream, the ffmpeg process is restarted, and the recording files with sequentially increasing sequence numbers are generated at regular intervals according to the preset recording time interval. At the same time, the timestamp is recorded, and the virtual id is synchronously generated at regular intervals according to the preset screenshot time interval. The video stream is detected in real time whether it is restored. After the video stream is restored, the ffmpeg process is started to synchronously generate the recording and screenshot files with sequentially increasing sequence numbers at regular intervals.

3. The live co-watch timeline playback method of claim 2, wherein, The live broadcast and the related stream information in the live broadcast and the related stream information in the live broadcast are recorded and stored in real time, including: When the server receives a request from a participant's client to go on or off the microphone, it records the event, the time of going on, and the sequence number of the screenshot taken by the host when the participant went on. Screenshot number of the person who joined the live stream. Screenshot number of the host when the other person in the live chat leaves the microphone. Screenshot number of the person who left the live chat. The streamer's address, the co-host's address, the co-host's address, the co-host's duration, and the co-host's order are used as co-host-related stream information during the live streaming process and are stored in the database.

4. The live co-watch timeline playback method of claim 3, wherein, When the server records the host screenshot sequence number and the client who wants to join the live broadcast screenshot sequence number when the client who wants to join the live broadcast is on, if the screenshot sequence number has been generated at this time, it is directly stored in the database; otherwise, it is first stored as NULL or a placeholder value, and the screenshot sequence number is scanned at regular intervals. When the screenshot sequence number is scanned, the NULL or placeholder value in the database is updated to the start_imgid and start_link_imgid fields.

5. The live co-watch timeline playback method of claim 3, wherein, The time offset of the microphone user when going on and off the microphone relative to the anchor recording stream and the time offset of the microphone user when going on the microphone relative to the microphone user recording stream are calculated by analyzing the screenshot sequence numbers of the anchor and the microphone user when the microphone user goes on and off the microphone in the microphone related stream information, and a progress bar of the live broadcast playback is simulated by time offset alignment, and the microphone user information is marked on the time axis based on the position of the progress bar and a preview image is generated, including: According to the length of the microphone And the difference between the anchor screenshot sequence numbers during the microphone, the real interval length of the anchor screenshot during the microphone is calculated , expressed as ; According to the anchor screenshot sequence number when the mic is connected And , the position offset start time point of the mic when the mic is connected relative to the anchor recording broadcast stream is calculated , expressed as ; according to the anchor screenshot sequence number when the mic is disconnected And , the position offset end time point of the mic when the mic is disconnected relative to the anchor recording broadcast stream is calculated , expressed as ; According to the length of the microphone And the difference between the screenshot serial numbers of the microphone during the microphone, the real interval length of the screenshots of the microphone during the microphone is calculated , expressed as ; According to the sequence number of the screenshot of the speaker when the speaker goes online And , the position offset start time point of the speaker when the speaker goes online relative to the recording and broadcasting stream of the speaker is calculated , expressed as ; By means of time offset alignment, that is As a time offset less than the anchor recording length, at the position of the anchor recording length as the time axis of the live broadcast playback, find Value alignment, simulate the progress bar of live broadcast playback, and calculate the position of the progress bar as ; wherein The anchor recording length In the time axis The information of the co-host is marked, and the starting time point is offset according to the position of the co-host in the co-host recording stream when the co-host gets on the microphone The playing object is generated, the co-host recording stream is loaded and jumps to For playing at the marked point, and the preview picture during the co-host is displayed when the mouse hovers over the marked point; All microphone user information in the live broadcast room is traversed, and the above steps are repeated until all microphone user information is marked on the time axis and the corresponding preview image is generated.

6. The live co-watch timeline playback method of claim 5, wherein, When the live broadcast playback time reaches the time offset of the microphone user when going on the microphone relative to the anchor recording stream, the microphone user recording stream is automatically loaded and synchronized played by jumping to the time offset of the microphone user when going on the microphone relative to the microphone user recording stream, and the whole live broadcast playback is realized by adaptive adjustment of the playing position and the picture size. When the live playback time is greater than or equal to , the live playback stream of the host is automatically loaded, and the live playback stream time of the host is jumped to , a playback object is generated to realize synchronous playback, and the playback object is placed in the foreground for playing. When the live play back time is greater than or equal to , the playing object of the current mic is eliminated and destroyed, and is invisible in the foreground. The playing objects and the microphone sequence of all microphone users in the current live broadcast room are obtained, and adaptive adjustment of the playing object position and the picture size is performed to realize the whole live broadcast playback.

7. A live stream timeline playback apparatus, comprising: The device comprises: A recording and screenshot synchronization generation module for the server to start an ffmpeg process to pull the client stream data in real time by receiving the broadcasting information sent by the anchor client in the live broadcast room or the on and off microphone information sent by the microphone user client, synchronously generate recording and screenshot files of each client stream, and record and store the microphone related stream information in the live broadcast microphone process in real time; A microphone marking module for dynamically loading the anchor recording stream for live broadcast playback, taking the anchor recording time as the time axis of the live broadcast playback, analyzing the screenshot sequence numbers of the anchor and the microphone user when the microphone user goes on and off the microphone in the microphone related stream information, calculating the time offset of the microphone user when going on and off the microphone relative to the anchor recording stream and the time offset of the microphone user when going on the microphone relative to the microphone user recording stream, simulating the generation of the progress bar of the live broadcast playback by time offset alignment, and marking the microphone user information on the time axis based on the position of the progress bar and generating a preview image; A synchronous playing module for acquiring the live broadcast playback time by detecting the position change of the progress bar in real time, automatically loading the microphone user recording stream and synchronously playing it by jumping to the time offset of the microphone user when going on the microphone relative to the microphone user recording stream when the live broadcast playback time reaches the time offset of the microphone user when going on the microphone relative to the anchor recording stream, and realizing the whole live broadcast playback by adaptive adjustment of the playing position and the picture size.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 6.

Images