Live broadcast pull stream gasket method, device and medium

Abstract

The embodiment of the invention provides a live broadcast pull stream gasket method, equipment and a medium, and belongs to the technical field of communication, and the method specifically comprises the steps: 1, starting a first pull stream device and a second pull stream device to pull a live broadcast stream, taking a stream pulled by the first pull stream device as a main stream, taking a stream pulled by the second pull stream device as a standby stream, and taking the standby stream as a standby stream; the main stream is subjected to coding stream pushing live broadcast through a stream pushing device; step 2, when the main stream is detected to be disconnected, performing audio resampling on the standby stream, then synchronizing the audio and video of the standby stream after the timestamp of the standby stream is updated, and performing coding stream pushing live broadcast through a stream pushing device; and step 3, detecting whether the mainstream is reconnected or not in real time, and when the mainstream is reconnected, updating the timestamp of the mainstream and carrying out coding stream-pushing live broadcast on the mainstream through a stream pushing device. According to the scheme of the invention, when the pull stream is interrupted or the quality is reduced, the playing fluency is guaranteed through the dynamic strategy.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of communication technologies, and in particular, to a live streaming padding method, device, and medium. Background Art

[0002] Currently, the technical background of live streaming and fault handling padding covers protocols (RTMP / HLS / WebRTC), encoding layers (H.264 / AV1), transport layers (QUIC / FEC), and decision-making layers (ABR / CDN scheduling), continuously addressing core challenges such as high concurrency, low latency, and weak network resistance.

[0003] However, the padding source will repeatedly send padding files. The durations of the padding files are unequal, but at the end of the file, it is necessary to re-pull the padding stream, and the timestamps of the re-pulled padding stream often have various changes, which easily leads to problems with audio and video synchronization.

[0004] It can be seen that there is an urgent need for a live streaming padding method that can ensure the smoothness of live streaming. Summary of the Invention

[0005] In view of this, embodiments of the present disclosure provide a live streaming padding method, device, and medium, which at least partially solve the problem of poor live streaming smoothness caused by padding in the prior art.

[0006] In a first aspect, embodiments of the present disclosure provide a live streaming padding method, including:

[0007] Step 1, start a first puller and a second puller to pull the live stream respectively. Take the live stream pulled by the first puller as the main stream, take the live stream pulled by the second puller as the backup stream, and encode and push the main stream for live streaming through a pusher;

[0008] Step 2, when it is detected that the main stream is disconnected, perform audio resampling on the backup stream, then update the timestamp of the backup stream and synchronize the audio and video of the backup stream, and encode and push the backup stream for live streaming through a pusher;

[0009] Step 3, continuously detect whether the main stream reconnects. When the main stream reconnects, update the timestamp of the main stream and encode and push the main stream for live streaming through a pusher.

[0010] According to a specific implementation manner of embodiments of the present disclosure, the live stream includes video data and audio data.

[0011] According to a specific implementation manner of embodiments of the present disclosure, after the step of taking the live stream pulled by the second puller as the backup stream, the method further includes:

[0012] Synchronously save the backup stream to the memory buffer in the form of a file.

[0013] According to a specific implementation manner of an embodiment of the present disclosure, step 2 specifically includes:

[0014] Step 2.1, when it is detected that the main stream is disconnected, decode the standby stream in the memory buffer to obtain video data and audio data;

[0015] Step 2.2, perform audio resampling on the audio data to generate new audio data;

[0016] Step 2.3, update the current timestamps of the video data and the new audio data;

[0017] Step 2.4, calculate the difference between the timestamps of each frame of the video data and the new audio data in real time;

[0018] Step 2.5, adjust the duration of the current frame of the video data according to the difference and the synchronization formula, and after achieving audio-visual synchronization, perform encoding and push-streaming live broadcast through a pusher.

[0019] According to a specific implementation manner of an embodiment of the present disclosure, step 2.2 specifically includes:

[0020] Recombine the audio data, put the resampled audio data into a fifo queue, send it according to the size of the encoded data, and at the same time calculate the timestamp using the time base of the main stream to obtain new audio data.

[0021] According to a specific implementation manner of an embodiment of the present disclosure, the step of timestamp update includes:

[0022] Take the product of the duration of the current video frame and the time base of this video frame as the new increment value of the current video playback duration;

[0023] Take the product of the duration of the current audio frame and the time base of this audio frame as the new increment value of the current audio playback duration.

[0024] According to a specific implementation manner of an embodiment of the present disclosure, the expression of the timestamp update is

[0025] pts_sec += frame.duration * av_q2d(frame.tbn)

[0026] Wherein, frame.duration represents the duration of the current frame of the decoded data, and frame.tbn is the time base of the decoded data container. When switching between the main stream and the standby stream, frame.tbn will be converted to the corresponding time base.

[0027] According to a specific implementation manner of an embodiment of the present disclosure, the expression of the synchronization formula is

[0028] frame.duration-

[0029] = 0.01 * (current_video_pts_sec_

[0030] - current_audio_pts_sec_) / av_q2d(frame.tbn)

[0031] wherein, current_video_pts_sec_ represents the video display timestamp of the pushed stream, in seconds, current_audio_pts_sec_ represents the audio playback timestamp of the pushed stream, in seconds, av_q2d represents converting the time base in the form of numerator and denominator into a double-precision floating point number, and the calculation process of av_q2d(frame.tbn) is as follows:

[0032]

[0033] wherein, num is the numerator of the time base and den is the denominator of the time base.

[0034] In a second aspect, an embodiment of the present disclosure further provides an electronic device, which includes:

[0035] at least one processor; and,

[0036] a memory communicatively connected to the at least one processor; wherein,

[0037] the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the live streaming pull shim method in the foregoing first aspect or any implementation manner of the first aspect.

[0038] In a third aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions for causing a computer to execute the live streaming pull shim method in the foregoing first aspect or any implementation manner of the first aspect.

[0039] In a fourth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computing program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is enabled to execute the live streaming pull shim method in the foregoing first aspect or any implementation manner of the first aspect.

[0040] The live streaming pulling buffer solution in the embodiments of the present disclosure includes: Step 1, start the first stream puller and the second stream puller to pull the live stream respectively. Take the stream pulled by the first stream puller as the main stream, and take the stream pulled by the second stream puller as the backup stream. Then encode and push the main stream for live streaming through a pusher; Step 2, when it is detected that the main stream is disconnected, perform audio resampling on the backup stream, then update the timestamp of the backup stream and synchronize the audio and video of the backup stream, and encode and push the backup stream for live streaming through a pusher; Step 3, detect in real time whether the main stream is reconnected. When the main stream is reconnected, update the timestamp of the main stream and encode and push the main stream for live streaming through a pusher.

[0041] The beneficial effects of the embodiments of the present disclosure are as follows: Through the solution of the present disclosure, when the stream pulling is interrupted or the quality deteriorates, the smoothness of playback is guaranteed through dynamic strategies. Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0043] Figure 1 It is a schematic flowchart of a live streaming pulling buffer method provided by the embodiments of the present disclosure;

[0044] Figure 2 It is a schematic structural diagram of a live streaming pulling buffer device provided by the embodiments of the present disclosure;

[0045] Figure 3 It is a schematic diagram of an electronic device provided by the embodiments of the present disclosure. Detailed Embodiments

[0046] The following will describe the embodiments of the present disclosure in detail with reference to the drawings.

[0047] The following specific examples illustrate the implementation manners of the present disclosure. Those skilled in the art can easily understand other advantages and effects of the present disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of them. The present disclosure can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present disclosure belong to the scope of protection of the present disclosure.

[0048] Note that the following description relates to various aspects of embodiments within the scope of the appended claims. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is for illustrative purposes only. Based on this disclosure, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of the aspects set forth herein can be used to implement a device and / or practice a method. Additionally, this device can be implemented and this method can be practiced using other structures and / or functionality in addition to one or more of the aspects set forth herein.

[0049] It should also be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of this disclosure. The diagrams only show the components related to this disclosure and are not drawn according to the number, shape, and size of the components in actual implementation. The form, quantity, and ratio of each component in its actual implementation can be arbitrarily changed, and the layout form of its components may also be more complex.

[0050] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the aspects described can be practiced without these specific details.

[0051] An embodiment of this disclosure provides a live stream pulling shim method, which can be applied to the live broadcast process in an Internet scenario.

[0052] See Figure 1 , which is a schematic flowchart of a live stream pulling shim method provided by an embodiment of this disclosure.

[0053] As Figure 1 shown, the method mainly includes the following steps:

[0054] Step 1, start the first stream puller and the second stream puller to pull the live stream respectively, use the live stream pulled by the first stream puller as the main stream, use the live stream pulled by the second stream puller as the backup stream, and encode and push the main stream through a pusher for live broadcast;

[0055] Optionally, the live stream includes video data and audio data.

[0056] Optionally, after the step of using the live stream pulled by the second stream puller as the backup stream, the method further includes:

[0057] Synchronously save the backup stream to the memory buffer in the form of a file.

[0058] In specific implementation, two pullers and one pusher can be started. One puller pulls the main stream and the other pulls the backup stream, and one pusher is used for pushing the stream. When pulling the backup stream, since the backup stream is a file and the end of the file will be disconnected, a method of saving the backup stream to memory is adopted. After the backup stream is saved to memory, the timestamp becomes stable, and at the same time, the bandwidth pressure of pulling the stream can be reduced.

[0059] Step 2: When it is detected that the main stream is disconnected, perform audio resampling on the backup stream, then update the timestamp of the backup stream and synchronize the audio and video of the backup stream, and perform encoding and pushing live through the pusher;

[0060] Based on the above embodiments, the specific steps of step 2 include:

[0061] Step 2.1: When it is detected that the main stream is disconnected, decode the backup stream in the memory buffer to obtain video data and audio data;

[0062] Step 2.2: Perform audio resampling on the audio data to generate new audio data;

[0063] Step 2.3: Update the current timestamps of the video data and the new audio data;

[0064] Step 2.4: Calculate the difference between the timestamps of each frame of the video data and the new audio data in real time;

[0065] Step 2.5: Adjust the duration of the current frame of the video data according to the difference and the synchronization formula to achieve audio-visual synchronization, and then perform encoding and pushing live through the pusher.

[0066] Further, the specific steps of step 2.2 include:

[0067] Recombine the audio data, put the resampled audio data into a fifo queue, send it according to the size of the encoded data, and at the same time calculate the timestamp using the time base of the main stream to obtain new audio data.

[0068] Further, the steps of updating the timestamp include:

[0069] Take the product of the duration of the current video frame and the time base of this video frame as the new increment value of the current video playing duration;

[0070] Take the product of the duration of the current audio frame and the time base of this audio frame as the new increment value of the current audio playing duration.

[0071] Further, the expression for updating the timestamp is

[0072] pts_sec += frame.duration * av_q2d(frame.tbn)

[0073] Among them, frame.duration represents the duration of the current decoded data frame, and frame.tbn is the time base of the decoded data container. When the main and backup streams are switched, frame.tbn will be converted to the corresponding time base.

[0074] Furthermore, the expression of the synchronization formula is

[0075] frame.duration -

[0076] = 0.01 * (current_video_pts_sec_

[0077] - current_audio_pts_sec_) / av_q2d(frame.tbn)

[0078] Among them, current_video_pts_sec_ represents the video display timestamp of the pushed stream in seconds, current_audio_pts_sec_ represents the audio playback timestamp of the pushed stream in seconds, av_q2d represents converting the time base in the form of numerator and denominator into a double-precision floating-point number, and the calculation process of av_q2d(frame.tbn) is:

[0079]

[0080] Among them, num is the numerator of the time base, and den is the denominator of the time base.

[0081] In specific implementation, when it is detected that the main stream is disconnected, the main stream puller can be restarted to pull the stream again. The backup stream puller adds the decoded data frames to the encoding queue. The encoder of the pusher sends the encoded data packets to the client, and the timestamps of the data packets will be resynchronized according to the duration of the frames before encoding. When the audio sampling rates are different, audio resampling is required. After audio resampling, the timestamps of the backup stream need to be synchronized in seconds using the time base of the main stream. In the case of AAC, since the pusher has already set the frame_size of the audio according to the parameters of the main stream and cannot be changed, it is necessary to reorganize the resampled audio data, and the timestamps of the reorganized audio data need to be calculated according to the new sampling rate.

[0082] The current playing duration of the video, video_pts_sec, and the current playing duration of the audio, audio_pts_sec, are in seconds.

[0083] Update of the current video playing duration video_pts_sec: The product of the duration of the current video frame and the time base of this video frame is used as the new increment value of the current video playing duration. The video frame to be encoded currently can be the main video frame or the video frame of the alternate stream.

[0084] Update of the current audio playing duration video_pts_sec: The product of the duration of the current audio frame and the time base of this audio frame is used as the new increment value of the current audio playing duration. The audio frame to be encoded currently can be the main audio frame or the audio frame of the alternate stream.

[0085] Update formula: pts_sec += frame.duration * av_q2d(frame.tbn) where frame is the decoded data, duration is the duration of this frame, and tbn is the time base of the container. When switching between the main and alternate streams, tbn will be converted to the corresponding time base.

[0086] The difference between video_pts_sec and audio_pts_sec will be detected for each frame. If the difference exceeds ±0.3s, adjustment is required. If it is positive, it means the video is faster than the audio, and frame.duration will be decreased in proportion to the difference. If it is negative, it means the video is slower than the audio, and frame.duration will be increased in proportion to the difference. The adjustment is as follows: frame.duration -= 0.01 * (current_video_pts_sec_ - current_audio_pts_sec_) / av_q2d(frame.tbn); The difference in the playing durations of the audio and video is converted into a time representation based on the time base of the current frame and then multiplied by an adjustable coefficient to adjust the duration of the current frame. The difference in audio - video out - of - sync can be spread over the durations of multiple frames for correction to reduce the situation of picture jumping.

[0087] When audio resampling is performed, the audio data needs to be reorganized. Assume that the mainstream audio AAC encoding has a sampling rate of 48 kHz, and the sampling rate of the alternate stream is 44.1 kHz. The push stream is AAC encoded with a frame size of 1024. The audio of the alternate stream is resampled at 48 kHz. The original audio length of 1024 becomes 1024 * 48 / 44.1. If the resample interface of ffmpeg is directly used, ffmpeg will directly truncate the audio data, resulting in audio-video desynchronization. We reorganized the audio data, put the resampled audio data into a FIFO queue, and sent it in sizes of 1024. At the same time, the timestamp calculation for 1024 uses the mainstream time base. This is equivalent to dividing the original audio data of approximately 10 frames into 11 frames for transmission, and the duration of each frame is reduced by about 10%. Using the mainstream time base to calculate exactly obtains the correct value.

[0088] Step 3, real-time detect whether the mainstream is reconnected. When the mainstream is reconnected, update the timestamp of the mainstream and encode and push the mainstream for live streaming through a pusher.

[0089] Specifically in implementation, when pulling the alternate stream for live streaming, it is also possible to real-time detect whether the mainstream is reconnected. When the mainstream is reconnected, update the timestamp of the mainstream and encode and push the mainstream for live streaming through a pusher.

[0090] The live streaming pull stream shim method provided in this embodiment improves the stability and smoothness of live streaming by introducing an alternate stream and an intelligent switching mechanism. When the mainstream is disconnected, the alternate stream can quickly take over to avoid live streaming interruption. At the same time, the system can real-time detect the recovery of the mainstream and smoothly switch back to the mainstream to ensure that the user's viewing experience is not affected. Through timestamp update and audio resampling, the switching process between the alternate stream and the mainstream is almost seamless, avoiding problems such as audio-video desynchronization or stuttering. In addition, the solution enhances the fault tolerance of the system, so that even if there are network problems, the alternate stream can supplement the mainstream to avoid live streaming interruption caused by a single stream source failure. Generally speaking, this solution improves the redundancy backup, automatic recovery, and real-time synchronization mechanisms, not only improving the stability of live streaming and the fault tolerance of the system, but also ensuring that users obtain a smooth and fluent viewing experience.

[0091] Corresponding to the above method embodiment, refer to Figure 2 , this disclosure embodiment also provides a live streaming pull stream shim device 10, including:

[0092] Mainstream puller 101, used to real-time pull the mainstream (RTMP / HTTP protocol).

[0093] Alternate stream puller 102, used to pull local / remote alternate stream files (such as MP4 / TS format).

[0094] A memory buffer 103 for circularly storing audio - video frames after decoding the backup stream.

[0095] A timestamp synchronization module 104 for calculating the difference between the timestamps of the front and back frames and generating a continuous time base.

[0096] A pusher 105, including an encoder (H.264 / AAC) and a sending module, for encoding audio - video data and pushing the stream to the client.

[0097] A fault detector 106 for monitoring the mainstream status (heartbeat packet / data timeout).

[0098] An audio resampling module 107 for converting the audio sampling rate of the backup stream to adapt to the mainstream parameters.

[0099] Figure 2 The device shown can correspondingly execute the content in the above - mentioned method embodiments. For parts not described in detail in this embodiment, refer to the content recorded in the above - mentioned method embodiments and will not be elaborated here.

[0100] See Figure 3 , this embodiment of the present disclosure also provides an electronic device 30, which includes: at least one processor and a memory communicatively connected to the at least one processor. Wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor can execute the live stream pulling shim method in the foregoing method embodiments.

[0101] This embodiment of the present disclosure also provides a non - transitory computer - readable storage medium, which stores computer instructions for causing the computer to execute the live stream pulling shim method in the foregoing method embodiments.

[0102] This embodiment of the present disclosure also provides a computer program product, which includes a computing program stored on a non - transitory computer - readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the live stream pulling shim method in the foregoing method embodiments.

[0103] Next, refer to Figure 3 , which shows a schematic structural diagram of an electronic device 30 suitable for implementing the embodiments of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in - vehicle terminals (such as in - vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3The electronic device shown is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0104] As Figure 3 shown, the electronic device 30 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 301, which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage device 308 into the random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 30 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0105] Generally, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device 30 to communicate with other devices wirelessly or wiredly to exchange data. Although the electronic device 30 with various devices is shown in the figure, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices may be implemented or had alternatively.

[0106] In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above functions defined in the methods of the embodiments of the present disclosure are executed.

[0107] It should be noted that the computer-readable medium described above can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0108] The above computer-readable medium can be included in the above electronic device; or it can exist separately without being assembled into the electronic device.

[0109] The above computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can perform the relevant steps of the above method embodiments.

[0110] Alternatively, the above computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can perform the relevant steps of the above method embodiments.

[0111] Computer program code for performing the operations of this disclosure can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by connecting through the Internet using an Internet service provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and this module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order from that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0113] The units described in the embodiments of this disclosure can be implemented in software or in hardware.

[0114] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0115] As described above, this is only the specific implementation manner of this disclosure, but the protection scope of this disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by this disclosure should be covered by the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be subject to the protection scope of the claims.

Claims

1. A live streaming gasket method, characterized in that: include: Step 1: Start the first stream puller and the second stream puller to pull the live stream respectively, use the stream pulled by the first stream puller as the main stream, use the stream pulled by the second stream puller as the backup stream, and encode the main stream through the stream pusher for live streaming; Step 2: When the main stream is detected to be disconnected, the audio of the backup stream is resampled, and then the timestamp of the backup stream is updated to synchronize the audio and video of the backup stream, and the stream is encoded and live streamed through the stream pusher; Step 3: Check in real time whether the main stream is reconnected. When the main stream is reconnected, update the timestamp of the main stream and encode and stream the main stream through the stream pusher for live broadcast.

2. The method according to claim 1, characterized in that The live stream includes video data and audio data.

3. The method according to claim 1, characterized in that After the step of pulling the second flow puller as a backup flow, the method further comprises: Synchronously save the alternate stream to a file in a memory buffer.

4. The method according to claim 3, characterized in that: The step 2 specifically includes: Step 2.1, when the main stream is detected to be disconnected, the backup stream in the memory buffer is decoded to obtain video data and audio data; Step 2.2, resampling the audio data to generate new audio data; Step 2.3, updating the current timestamps of the video data and the new audio data; Step 2.4, calculating the difference between the timestamps of each frame of the video data and the new audio data in real time; Step 2.5, adjust the duration of the current frame of the video data according to the difference and the synchronization formula, and after achieving audio and video synchronization, encode and push the stream for live broadcast through the pusher.

5. The method according to claim 4, characterized in that The step 2.2 specifically includes: The audio data is reorganized, the resampled audio data is put into a fifo queue, and sent according to the size of the encoded data. At the same time, the timestamp is calculated using the mainstream time base to obtain new audio data.

6. The method according to claim 4, characterized in that The step of updating the timestamp includes: The product of the duration of the current video frame and the time base of the video frame is used as the new added value of the current video playback duration; The product of the duration of the current audio frame and the time base of the audio frame is used as the new added value of the current audio playback duration.

7. The method according to claim 4, characterized in that The expression for the timestamp update is: pts_sec+=frame.duration*av_q2d(frame.tbn) Among them, frame.duration indicates the duration of the current frame of the decoded data, and frame.tbn is the time base of the decoded data container. When the main stream and the backup stream are switched, frame.tbn will be converted to the corresponding time base.

8. The method according to claim 4, characterized in that The expression of the synchronization formula is: frame.duration - =0.01*(current_video_pts_sec_ -current_audio_pts_sec_) / av_q2d(frame.tbn) Among them, current_video_pts_sec_ represents the video display timestamp of the pushed stream, in seconds, current_audio_pts_sec_ represents the audio playback timestamp of the pushed stream, in seconds, and av_q2d represents the conversion of the time base in the form of numerator and denominator into double-precision floating point numbers. The calculation process of av_q2d(frame.tbn) is as follows: Among them, num is the numerator of the time base and den is the denominator of the time base.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the live streaming gasket method described in any one of claims 1-8.

10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the live streaming gasket method described in any one of claims 1 to 8.

Images