Video conference live broadcast platform recommendation method based on RTMP stream pushing

Abstract

The invention provides a video conference live broadcast platform recommendation method based on RTMP stream pushing, and the method comprises the steps: carrying out the RTMP stream pushing of a speaker end, carrying out the data forwarding of the speaker end and an audience end through a server, and generating audio and video streams with different code rates; the server performs audio separation and code rate conversion: after receiving the data stream pushed by the speaker end, the server separates the video stream from the audio stream, and then performs code rate conversion on the data stream; the audience side is connected to the conference address, and a user obtains the address of the CDN pull stream server for watching; and adaptively regulating and controlling the code rate. According to the video conference live broadcast platform recommendation method based on RTMP stream pushing, by combining the advantages of different video conferences and the characteristic that the number of video conference watching users is generally larger than the number of live broadcast interactive users, the cost can be remarkably reduced by using CDN stream pushing, meanwhile, the network condition can be automatically adapted by using a code rate self-adaption technology, the jamming probability is reduced, and the user experience is improved. The method has strong versatility, and can realize the effect of finishing large conference live broadcast through a small number of live broadcast channels.

Description

Technical Field

[0001] The present invention relates to the technical field of online conferencing, and particularly to a method for recommending a video conferencing live platform based on RTMP streaming. Background Art

[0002] Online conferences are usually billed according to the number of channels or minutes. Different video conferencing providers have their own advantages and disadvantages in terms of the number of participating users, the number of channels, the number of users who can connect to the microphone, and the payment for value-added functions. The conference platform can allocate video conference room resources of different providers based on the meeting duration and the number of people.

[0003] In the video conferencing live scenario, there are usually two roles: speakers and audiences. Speakers can demonstrate courseware and screen recordings, while audiences can only watch and have limited connection to the microphone. Since both the number of video conferences and the minutes are billed, in the prior art, direct viewing by audiences in large conferences through the live platform will result in more cost consumption, and the large traffic generated in the live channel will also affect the transmission stability of network data, thus causing live freezes and affecting the live effect. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the purpose of the present invention is to provide a method for recommending a video conferencing live platform based on RTMP streaming to solve the problems raised in the above background art. The present invention uses CDN streaming to significantly reduce costs, and at the same time uses bitrate adaptation technology to automatically adapt to network conditions and reduce the probability of freezing, and has strong versatility.

[0005] To achieve the above purpose, the present invention is realized through the following technical solutions: A method for recommending a video conferencing live platform based on RTMP streaming, including RTMP streaming at the speaker end, data forwarding between the speaker end and the audience end through a server, and generating audio and video streams with different bitrates; audio separation and bitrate conversion at the server end, after the server end receives the data stream pushed by the speaker end, separating the video and audio streams, and then performing bitrate conversion on the data stream; the audience end connecting to the conference address, users entering the conference waiting room by scanning a code or clicking on a web address, and when the conference reaches the specified time, after the waiting room completes the security verification of the user information, obtaining the CDN pull stream server address for viewing; bitrate adaptive regulation.

[0006] Further, during the RTMP streaming process, the courseware and screen recording are mixed and streamed to the streaming media server end. The speaker end uses the client of a third-party online conference platform to generate video mixing, and the video mixing includes: the speaker's camera, ppt courseware, and the audio stream is collected by the system microphone.

[0007] Further, the speaker client is used to complete functions such as ppt courseware demonstration and screen recording; the video stream uses the H.264 encoding format and the audio stream uses the AAC encoding format.

[0008] Furthermore, during the bitrate conversion process, the ffmpeg multimedia framework is adopted and includes the following: the streaming media server separates the audio through ffmpeg, and the separated audio stream is pushed to the viewer side through websocket; the streaming media server completes the resolution conversion through ffmpeg, and nginx-rtmp is used for deployment; the streaming media server provides an HLS service based on the nginx-rtmp module to generate all available video segment information in real time; the streaming media server provides a TCP service to maintain a long connection with the viewer side, and when the viewer side needs live interaction, it accesses the data stream and performs multi-channel audio and video streaming.

[0009] Furthermore, the video segment information includes the address and duration, and the viewer side obtains the video stream through the CDN network for viewing.

[0010] Furthermore, after obtaining the permission of the speaker or the system, the viewer side sends messages or files to the chat server through the websocket long connection channel; the viewer side views the discussion materials through the CDN pull stream server address, where the CDN pull stream server is used to send the audio and video data stream to the viewer side to provide a reliable live viewing service; the viewer side slides the timeline through the multimedia control to view the replay; the speaker side uses the dynamic form tool on the mobile side to complete the configuration of the questionnaire and sign-in form, and the viewer side completes the corresponding actions on the web page. The viewer side realizes audio access through the WebRTC technology and sends the audio data stream to the streaming media access layer server through the websocket long connection channel, and the server performs streaming and then completes the co-hosting function.

[0011] Furthermore, the bitrate adaptive regulation process includes the following: the client and the streaming media server perform real-time communication through two-way websocket; the bitrate of the current video stream is obtained through the websocket signaling data, and the received bitrate is calculated.

[0012] Furthermore, one way of the websocket is used for streaming media signaling distribution, including information such as video progress, bitrate, and timeline, and the other way is used for data distribution, including chat messages and WebRTC audio data transmission.

[0013] Furthermore, the bitrate calculation formula is: bitrate = received packet size / received packet time.

[0014] Furthermore, when the calculated bitrate is lower than the bitrate obtained from the signaling data, a message is sent to the server through the websocket to automatically switch to the low-bitrate channel.

[0015] Advantages of the present invention:

[0016] 1. The video conferencing live streaming platform recommendation method based on RTMP streaming combines the advantages of different video conferences and the characteristic that the number of users watching video conferences is usually more than that of live interactive users. Using CDN streaming can significantly reduce costs. At the same time, the bitrate adaptation technology is used to automatically adapt to the network situation and reduce the chance of buffering, with strong versatility.

[0017] 2. The video conferencing live streaming platform recommendation method based on RTMP streaming performs bitrate conversion by RTMP streaming to the streaming media server, and then completes the live viewing of the audience through CDN push-pull streaming, which can realize the function of completing large-scale conference live streaming with a small number of live channels. Since the number and minutes of video conferences are both billed, the audience only calculates the bandwidth and traffic fees for pulling the stream through CDN, with obvious price advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is the schematic diagram of the application method in the embodiment of the present invention;

[0019] Figure 2 is the flowchart of a video conferencing live streaming platform recommendation method based on RTMP streaming of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] To make the technical means, creative features, achieved purposes and functions of the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.

[0021] Please refer to Figures 1 to 2 , the present invention provides a technical solution: a video conferencing live streaming platform recommendation method based on RTMP streaming. In this method, the speaker uses a video conferencing system to hold a meeting, performs bitrate conversion by RTMP streaming to the streaming media server, and then completes the live viewing of the audience through CDN push-pull streaming, which can realize the function of completing large-scale conference live streaming with a small number of live channels. The recommendation method combines the advantages of different video conferences and the characteristic that the number of users watching video conferences is usually more than that of live interactive users. Using CDN streaming can significantly reduce costs. At the same time, the bitrate adaptation technology is used to automatically adapt to the network situation and reduce the chance of buffering, with strong versatility. The specific contents are as follows:

[0022] 1. RTMP Streaming at the Speaker End

[0023] The speaker end and the audience end cannot directly transmit video streams and need the server for data forwarding. And to ensure that the audience end can watch smoothly under different network conditions, audio and video streams with different bitrates need to be generated. RTMP streaming can be set on the speaker live client, and the courseware and screen recording are mixed and streamed to the streaming media server. The following technologies are mainly used:

[0024] 1.1 The client of the third-party online meeting platform is used at the speaker side to generate video mixing, and the video mixing includes: the speaker's camera and the PPT courseware. The audio stream is collected by the system microphone;

[0025] 1.2 The video stream adopts the H.264 encoding format. Since H.264 adopts intra-frame compression and spatial prediction coding technology, it can significantly reduce the size of the data stream without losing too much quality;

[0026] 1.3 The audio stream adopts the AAC encoding format, supports multiple sampling rates and bit depths, and at the same time has low encoding and decoding delays, and can provide high-quality audio at a low bit rate;

[0027] 1.4 RTMP is a protocol for real-time data transmission, mainly used for the transmission of audio and video streams and streaming media services, and has the characteristics of high reliability and real-time performance, multiplexing, flow control, and bandwidth adaptation.

[0028] 2. Audio separation and bitrate conversion on the server side

[0029] In this embodiment, after the server side receives the data stream pushed by the speaker side, the video and audio streams are separated, and then the bitrate of the data stream is converted to ensure smooth viewing in different network conditions. ffmpeg is a feature-rich open-source multimedia framework that supports functions such as video encoding, decoding, and format conversion, and can be completed in the following ways:

[0030] 2.1 The streaming media server completes audio separation through ffmpeg, and the separated audio stream is pushed to the viewer side through websocket. The reference code is as follows:

[0031]

[0032] 2.2 The streaming media server completes resolution conversion through ffmpeg, and nginx-rtmp can be used for deployment. Some configuration files are as follows:

[0033]

[0034]

[0035] 2.3 HLS (HTTP Live Streaming) is an adaptive bitrate streaming media transmission protocol proposed by Apple based on HTTP. The streaming media server provides an HLS service based on the nginx-rtmp module, and real-time generates all available video segment information (such as addresses and durations) at present. The viewer side obtains the video stream through the CDN network for viewing. Some nginx configuration files are as follows:

[0036]

[0037]

[0038] 3.4 The streaming media server provides TCP service and maintains a long connection with the viewer side. When the viewer side needs live interaction, it accesses the data stream and performs multi-channel audio and video streaming. Reference command:

[0039] ffmpeg - i input1.mp4 - i input2.mp4 - filter_complex "[0:v]scale=w=640:h=480[video1];[1:v]scale=w=640:h=480[video2];[video1][video2]concat=n=2:v=1[outv]" - map "[outv]" output.mp4

[0040] 3. Conference functions on the viewer side

[0041] In this embodiment, on the participating end, the web page side is usually used for more convenient viewing. Users enter the meeting waiting room by scanning the code or clicking on the web page address. When the meeting reaches the specified time, after the waiting room completes the security verification of the user information, it obtains the CDN pull stream server address for viewing. The viewer side also has the following functions:

[0042] 3.1 When the viewer side obtains the permission from the speaker or the system, it can use the websocket long connection channel to send messages or files to the chat server;

[0043] 3.2 Viewing meeting materials, such as meeting courseware, meeting picture materials, etc.;

[0044] 3.3 Sliding the timeline through the multimedia control to view the playback, including viewing the video clips edited by the meeting organizer;

[0045] 3.4 The speaker side uses the dynamic form tool on the mobile side to complete the configuration of the questionnaire and sign - in form, and the viewer side completes the corresponding actions on the web page;

[0046] 3.5 The viewer side realizes audio access through WebRTC technology, and uses the websocket long connection channel to send the audio data stream to the streaming media access layer server. After the server performs streaming, the co - hosting function is completed. The reference code is as follows:

[0047]

[0048]

[0049] 4. Bitrate adaptation

[0050] In this embodiment, in addition to completing reliable audio and video streaming, the online meeting system also needs to reduce the impact of network fluctuations on the live broadcast quality. In different network scenarios, such as 3G, 4G, Wi-Fi, etc., there will be obvious differences in the viewing experience of the client. An effective method is needed to identify network changes, which can be achieved through the following methods:

[0051] 4.1 The client and the streaming media server use two-way websocket for real-time communication. One way is for streaming media signaling to be sent down, including information such as video progress, bitrate, time axis, etc. The other way is for data to be sent down, including chat messages and WebRTC audio data transmission;

[0052] 4.2 Obtain the bitrate of the current video stream through the websocket signaling data, and calculate the received bitrate. The bitrate calculation formula is: the size of the received data packet / the time of the received data packet. For example, use the following code:

[0053]

[0054]

[0055] When the calculated bitrate is lower than the bitrate obtained from the signaling data, a message can be sent to the server through websocket to automatically switch to a low-bitrate channel.

[0056] Please refer to Figure 1 , the structure functions in this embodiment are as follows: Figure 1 in

[0057] (1) Speaker client: Complete functions such as ppt courseware demonstration and screen recording;

[0058] (2) Streaming media access server: Responsible for receiving the push stream from the speaker client; responsible for receiving the WebRTC audio data stream of the viewing end;

[0059] (3) Streaming media server: Responsible for streaming media bitrate conversion, streaming media signaling data distribution, and streaming media bitrate coordination;

[0060] (4) CDN push server: Responsible for sending audio and video data streams to the viewer side and providing a reliable live viewing service;

[0061] (5) Viewer side: Responsible for authentication and network adaptation, providing rich meeting room customization functions, such as co-hosting, whiteboard, questionnaire, etc.; Network status detection and coordinating the bitrate with the streaming media server through the message server;

[0062] (6) Back-end business server: Responsible for business services other than the live stream, including authentication, business logic processing, data persistence, etc.;

[0063] (7) Message Server: Responsible for maintaining a long connection with the viewer side and providing the two-way data transmission capability of BS, including signaling data and service data transmission.

[0064] The foregoing has shown and described the basic principles, main features and advantages of the present invention. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. Therefore, in any aspect, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

[0065] In addition, it should be understood that although this specification is described according to embodiments, not every embodiment only contains an independent technical solution. This narrative manner of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for recommending a video conferencing live streaming platform based on RTMP streaming, characterized in that, It includes RTMP pushing on the speaker side. Data is forwarded between the speaker side and the viewer side through a server, and audio and video streams with different bitrates are generated. Audio separation and bitrate conversion on the server side. After receiving the data stream pushed by the speaker side, the server separates the video and audio streams and then performs bitrate conversion on the data stream. The viewer side connects to the meeting address. Users enter the meeting waiting room by scanning the code or clicking on the web address. When the meeting reaches the specified time, after the waiting room completes the security verification of the user information, it obtains the CDN pull stream server address for viewing. Bitrate adaptive regulation.

2. A method for recommending a video conferencing live platform based on RTMP streaming according to claim 1, characterized in that: During the RTMP pushing process, the courseware and screen recording are mixed and pushed to the streaming media server. The speaker side uses the client of a third-party online meeting platform to generate video mixing, which includes the speaker's camera and ppt courseware. The audio stream is collected by the system microphone.

3. A method for recommending a video conferencing live broadcast platform based on RTMP streaming according to claim 2, characterized in that: The speaker client is used to complete functions such as ppt courseware demonstration and screen recording. The video stream uses the H.264 encoding format and the audio stream uses the AAC encoding format.

4. A method for recommending a video conferencing live streaming platform based on RTMP streaming, as claimed in claim 1, wherein: During the bitrate conversion process, the ffmpeg multimedia framework is adopted and includes the following: The streaming media server completes audio separation through ffmpeg, and the separated audio stream is pushed to the viewer side through websocket. The streaming media server completes resolution conversion through ffmpeg, and nginx-rtmp is used for deployment. The streaming media server provides an HLS service based on the nginx-rtmp module and generates information on all currently available video segments in real time. The streaming media server provides a TCP service to maintain a long connection with the viewer side. When the viewer side needs live interaction, it accesses the data stream and performs multi-channel audio and video streaming.

5. A method for recommending a video conferencing live streaming platform based on RTMP streaming, as claimed in claim 4, wherein: The video segment information includes the address and duration, and the viewer side obtains the video stream through the CDN network for viewing.

6. A method for recommending a video conferencing live streaming platform based on RTMP streaming, as claimed in claim 1, wherein: After obtaining the permission of the speaker or the system, the viewer side sends messages or files to the chat server through the websocket long connection channel. The viewer side views the meeting materials through the CDN pull stream server address. The CDN pull stream server is used to send the audio and video data stream to the viewer side to provide a reliable live viewing service. The viewer side slides the timeline through the multimedia control to view the playback. The speaker side uses the dynamic form tool on the mobile device to complete the configuration of the questionnaire and sign-in form. The viewer side completes the corresponding actions on the web page. The viewer side realizes audio access through the WebRTC technology and sends the audio data stream to the streaming media access layer server through the websocket long connection channel. After the server performs streaming, the co-hosting function is completed.

7. A method for recommending a video conferencing live broadcast platform based on RTMP streaming, as claimed in claim 1, wherein The bitrate adaptive regulation process includes the following: The client and the streaming media server perform real-time communication through two-way websocket. Obtain the bitrate of the current video stream through the websocket signaling data and calculate the received bitrate.

8. A method for recommending a video conferencing live platform based on RTMP streaming, as claimed in claim 7, wherein: One path of the websocket is used for streaming media signaling distribution, including information such as video progress, bitrate, and timeline. The other path is used for data distribution, including chat messages and WebRTC audio data transmission.

9. A method for recommending a video conferencing live platform based on RTMP streaming as claimed in claim 8, wherein, The formula for calculating the bit rate is: bit rate = size of received data packet / time of received data packet.

10. A method for recommending a video conferencing live streaming platform based on RTMP streaming, as claimed in claim 1, wherein: When the calculated bit rate is lower than the bit rate obtained from signaling data, a message is sent to the server via websocket to automatically switch to a low-bitrate channel.

Images