Multi-terminal live broadcast stream pushing management system based on cloud service

Abstract

The invention discloses a multi-terminal live broadcast stream pushing management system based on cloud service, and the system comprises the steps: intercepting to-be-played audio and video streams of a video playing application when the video playing application is detected to be started; determining a flow pushing server according to the obtained flow pushing address; pushing the audio and video streams to the stream pushing server based on a preset streaming media protocol; obtaining a signaling state of the target plug-flow live broadcast; the signaling state comprises a signaling operation state or a signaling end state, the signaling operation state is determined by a live broadcast signaling generated by an anchor terminal in response to a live broadcast operation control, and the signaling end state is determined by a live broadcast signaling generated by the anchor terminal in response to a live broadcast end control.

Description

Technical Field

[0001] The present invention relates to the field of live streaming and streaming technology, and more specifically, to a multi-terminal live streaming and streaming management system based on cloud services. Background Art

[0002] With the development of the internet, more and more people are exchanging information online. Live streaming, as a simple and convenient way to interact, is gaining popularity, and its formats are becoming more diverse. Streaming live streaming, for example, is widely used to promote product promotions in e-commerce live streams, as it allows for more intuitive dissemination of live broadcast information. The live data stream in push streaming is generated through a third-party channel and is separate from the control of the live signaling generated during the live streaming process. The host side of a push streaming live stream primarily serves as the signaling control end. When an abnormality such as standby occurs on the host side, it may not be detected in time, resulting in the interruption of the live stream, making it impossible for viewers to view the live stream. To resume the live stream, the host must create a new live streaming room and inform the audience of the change, resulting in poor live streaming quality and a reduced viewing experience. Mainstream streaming devices utilize dedicated encoding chips combined with network communication devices to create a dedicated streaming product, meeting the needs of push streaming. However, these products can only push video data from locally connected cameras, limiting the video source's richness. Summary of the Invention

[0003] (1) Technical problems solved

[0004] In response to the problems existing in the prior art, the present invention provides a multi-terminal live streaming and streaming management system based on cloud services to solve the technical problem mentioned in the background technology that the installation position and size of the mounting bracket for different models of cameras are fixed and inconvenient to adjust.

[0005] (2) Technical solution

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multi-terminal live streaming and streaming management system based on cloud services, comprising the following steps:

[0007] Step 1: When detecting that a video playback application is started, intercepting the audio and video stream to be played by the video playback application; determining a streaming server according to the acquired streaming address; and pushing the audio and video stream to the streaming server based on a preset streaming protocol;

[0008] Step 2: Obtain the signaling status of the target live streaming; the signaling status includes a signaling running status or a signaling ending status. The signaling running status is determined by the live signaling generated by the anchor terminal in response to the live signaling running control, and the signaling ending status is determined by the live signaling generated by the anchor terminal in response to the live signaling ending control.

[0009] Step 3: setting the parameters of the camera; the camera establishes a connection with the texture of the open graphics rendering library OpenGL through the surface texture SurfaceTexture, performs beautification by processing the texture, and the beautified data is rendered into the drawing buffer of the open graphics rendering library OpenGL through the texture and drawn, and then a preview is obtained, including entering a loop for processing the video stream, the loop includes processing frame by frame, the video frame is output to the texture receiving interface frame by frame, the texture object needs to obtain the latest frame data by updating the current frame buffer, the open embedded surface OES texture directly receives data, converts the open embedded surface OES texture into a 2D texture, performs a filtering and beauty algorithm on the 2D texture, and draws through the mechanism of the open graphics rendering library OpenGL;

[0010] Step 4: If the signaling state is the signaling running state, monitoring the live data stream within the data stream acquisition time period; if the live data stream is acquired within the data stream acquisition time period, determining that the target push stream live broadcast is in the live broadcast progress state, retaining the target push stream live broadcast, so that the anchor terminal enters the target push stream live broadcast when it recovers from the abnormal exit state to the normal live broadcast state;

[0011] Step 5: The video is encoded into a specified format through the multimedia digital signal codec MediaCodeC, and the video stream data of the camera is passed to the frame buffer of the open graphics rendering library OpenGL through the texture. The frame buffer data is further exchanged to the input surface Surface buffer created by the multimedia digital signal codec MediaCodeC, and the data is encoded and processed. Then, the encoded data is obtained through the output buffer OutPutBuffer; the real-time message transmission Rtmp protocol is used to push the obtained encoded data video stream.

[0012] The present invention is further configured so that, before the step of intercepting the audio and video stream to be played by the video playback application when the video playback application is detected to be started, it also includes: periodically sending a first UDP data packet carrying a streaming address request to the streaming management end based on the UDP protocol, and the streaming management end obtains the streaming address request by parsing the first UDP data packet; wherein, the streaming management end is a terminal that manages, configures and / or monitors the streaming process based on the UDP data reported by the OTT set-top box; and receives the streaming address fed back by the streaming management end based on the streaming address request.

[0013] The present invention is further configured such that, after the step of pushing the audio and video stream to the streaming server based on a preset streaming media protocol, it also includes: triggering fault prompt information when a fault is detected in the current streaming process; sending a second UDP data packet carrying the fault prompt information to the streaming management end based on the UDP protocol; the streaming management end feeds back a restart instruction when detecting the presence of the fault prompt information in the second UDP data; and restarting based on the received restart instruction.

[0014] The present invention is further configured such that the obtaining of the signaling status of the target live streaming includes: if a live streaming end signaling generated by the anchor terminal in response to the live streaming end control is received, then determining that the signaling status of the target live streaming is the signaling end status; if a live streaming operation signaling generated by the anchor terminal in response to the live streaming operation control is received, or no live streaming signaling sent by the anchor terminal is received, then determining that the signaling status of the target live streaming is the signaling operation status; the live streaming operation control is a control in the anchor terminal other than the live streaming end control.

[0015] The present invention is further configured such that, before the step of pushing the audio and video stream to the streaming server based on the preset streaming protocol, it also includes: judging whether the transmission protocol corresponding to the audio and video stream is the preset streaming protocol; if not, parsing the audio and video stream to obtain the audio and video stream to be encapsulated; accordingly, the step of pushing the audio and video stream to the streaming server based on the preset streaming protocol includes: pushing the audio and video stream to be encapsulated to the streaming server based on the preset streaming protocol.

[0016] The present invention is further configured such that the setting of the camera parameters includes: setting the camera parameters, preview callback method, exporting the camera's video frame stream through the surface texture SurfaceTexture, and constructing the surface texture SurfaceTexture object through the texture object in the open graphics rendering library OpenGL, so that when the camera's video frame stream is exported to the surface texture SurfaceTexture, it can be connected with the texture, and the texture includes an open embedded surface OES texture.

[0017] The present invention is further configured as follows: if not, parsing the audio and video stream to obtain the audio and video stream to be encapsulated, including: if not, performing streaming media protocol parsing on the audio and video stream to obtain the audio and video stream to be processed; parsing the encapsulation format of the audio and video stream to be processed to obtain a video stream in a video encoding format and an audio stream in an audio encoding format; and using the video stream in the video encoding format and the audio stream in the audio encoding format as the audio and video stream to be encapsulated.

[0018] (3) Beneficial effects

[0019] Compared with the existing technology, the present invention provides a multi-terminal live streaming and streaming management system based on cloud services, which has the following beneficial effects:

[0020] The present invention provides a method for real-time video beautification and streaming in terminal-based live broadcast on the Internet, including a detailed description of the beautification, rendering, encoding, streaming and life cycle management involved in the entire live broadcast process from camera to streaming. Through the scheme of the present invention, real-time video beautification and streaming can be realized in terminals such as mobile phones in live broadcast on the Internet. In addition, when the video playback application is detected to be started, the audio and video stream to be played by the video playback application is intercepted, and then the streaming server is determined according to the acquired streaming address, and finally the audio and video stream is pushed to the streaming server based on the preset streaming media protocol. Since the present invention intercepts the audio and video stream to be played by the video playback application through the set-top box, and then pushes the intercepted audio and video stream to the streaming server, compared with the prior art that pushes the stream through a dedicated encoding chip plus a network communication device, which can only push the locally accessed camera video data, the above-mentioned live streaming streaming method of the present invention avoids the source end of the video source being limited to the camera device input, and further enriches the source end of the video source. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a schematic diagram of the overall structure of a multi-terminal live streaming management system based on cloud services. DETAILED DESCRIPTION

[0022] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meaning as commonly understood by ordinary technicians in the technical field to which this application belongs.

[0024] Example 1:

[0025] See also Figure 1 A multi-terminal live streaming and streaming management system based on cloud services includes the following steps:

[0026] Step 1: When detecting that a video playback application is started, intercepting the audio and video stream to be played by the video playback application; determining a streaming server according to the acquired streaming address; and pushing the audio and video stream to the streaming server based on a preset streaming protocol;

[0027] Step 2: Obtain the signaling status of the target live streaming; the signaling status includes a signaling running status or a signaling ending status. The signaling running status is determined by the live signaling generated by the anchor terminal in response to the live signaling running control, and the signaling ending status is determined by the live signaling generated by the anchor terminal in response to the live signaling ending control.

[0028] Step 3: setting the parameters of the camera; the camera establishes a connection with the texture of the open graphics rendering library OpenGL through the surface texture SurfaceTexture, performs beautification by processing the texture, and the beautified data is rendered into the drawing buffer of the open graphics rendering library OpenGL through the texture and drawn, and then a preview is obtained, including entering a loop for processing the video stream, the loop includes processing frame by frame, the video frame is output to the texture receiving interface frame by frame, the texture object needs to obtain the latest frame data by updating the current frame buffer, the open embedded surface OES texture directly receives data, converts the open embedded surface OES texture into a 2D texture, performs a filtering and beauty algorithm on the 2D texture, and draws through the mechanism of the open graphics rendering library OpenGL;

[0029] Step 4: If the signaling state is the signaling running state, monitoring the live data stream within the data stream acquisition time period; if the live data stream is acquired within the data stream acquisition time period, determining that the target push stream live broadcast is in the live broadcast progress state, retaining the target push stream live broadcast, so that the anchor terminal enters the target push stream live broadcast when it recovers from the abnormal exit state to the normal live broadcast state;

[0030] Step 5: The video is encoded into a specified format through the multimedia digital signal codec MediaCodeC, and the video stream data of the camera is passed to the frame buffer of the open graphics rendering library OpenGL through the texture. The frame buffer data is further exchanged to the input surface Surface buffer created by the multimedia digital signal codec MediaCodeC, and the data is encoded and processed. Then, the encoded data is obtained through the output buffer OutPutBuffer; the real-time message transmission Rtmp protocol is used to push the obtained encoded data video stream.

[0031] In this embodiment, before the step of intercepting the audio and video stream to be played by the video playback application when detecting that the video playback application is started, it also includes: periodically sending a first UDP data packet carrying a streaming address request to the streaming management end based on the UDP protocol, and the streaming management end obtains the streaming address request by parsing the first UDP data packet; wherein, the streaming management end is a terminal that manages, configures and / or monitors the streaming process based on the UDP data reported by the OTT set-top box; and receives the streaming address fed back by the streaming management end based on the streaming address request.

[0032] In this embodiment, after the step of pushing the audio and video stream to the streaming server based on the preset streaming protocol, it also includes: triggering fault prompt information when a fault is detected in the current streaming process; sending a second UDP data packet carrying the fault prompt information to the streaming management end based on the UDP protocol; the streaming management end feeds back a restart instruction when detecting the presence of the fault prompt information in the second UDP data; and restarting based on the received restart instruction.

[0033] Example 2:

[0034] A multi-terminal live streaming and streaming management system based on cloud services includes the following steps:

[0035] Step 1: When detecting that a video playback application is started, intercepting the audio and video stream to be played by the video playback application; determining a streaming server according to the acquired streaming address; and pushing the audio and video stream to the streaming server based on a preset streaming protocol;

[0036] Step 2: Obtain the signaling status of the target live streaming; the signaling status includes a signaling running status or a signaling ending status. The signaling running status is determined by the live signaling generated by the anchor terminal in response to the live signaling running control, and the signaling ending status is determined by the live signaling generated by the anchor terminal in response to the live signaling ending control.

[0037] Step 3: setting the parameters of the camera; the camera establishes a connection with the texture of the open graphics rendering library OpenGL through the surface texture SurfaceTexture, performs beautification by processing the texture, and the beautified data is rendered into the drawing buffer of the open graphics rendering library OpenGL through the texture and drawn, and then a preview is obtained, including entering a loop for processing the video stream, the loop includes processing frame by frame, the video frame is output to the texture receiving interface frame by frame, the texture object needs to obtain the latest frame data by updating the current frame buffer, the open embedded surface OES texture directly receives data, converts the open embedded surface OES texture into a 2D texture, performs a filtering and beauty algorithm on the 2D texture, and draws through the mechanism of the open graphics rendering library OpenGL;

[0038] Step 4: If the signaling state is the signaling running state, monitoring the live data stream within the data stream acquisition time period; if the live data stream is acquired within the data stream acquisition time period, determining that the target push stream live broadcast is in the live broadcast progress state, retaining the target push stream live broadcast, so that the anchor terminal enters the target push stream live broadcast when it recovers from the abnormal exit state to the normal live broadcast state;

[0039] Step 5: The video is encoded into a specified format through the multimedia digital signal codec MediaCodeC, and the video stream data of the camera is passed to the frame buffer of the open graphics rendering library OpenGL through the texture. The frame buffer data is further exchanged to the input surface Surface buffer created by the multimedia digital signal codec MediaCodeC, and the data is encoded and processed. Then, the encoded data is obtained through the output buffer OutPutBuffer; the real-time message transmission Rtmp protocol is used to push the obtained encoded data video stream.

[0040] In this embodiment, the signaling state of the target push-stream live broadcast is obtained, including: if the live broadcast end signaling generated by the anchor terminal in response to the live broadcast end control is received, then the signaling state of the target push-stream live broadcast is determined to be the signaling end state; if the live broadcast operation signaling generated by the anchor terminal in response to the live broadcast operation control is received, or the live broadcast signaling sent by the anchor terminal is not received, then the signaling state of the target push-stream live broadcast is determined to be the signaling operation state; the live broadcast operation control is a control in the anchor terminal other than the live broadcast end control; before the step of pushing the audio and video stream to the push-stream server based on the preset streaming media protocol, it also includes: judging whether the transmission protocol corresponding to the audio and video stream is the preset streaming media protocol; if not, The audio and video stream is parsed to obtain the audio and video stream to be packaged; accordingly, the step of pushing the audio and video stream to the streaming server based on the preset streaming protocol includes: pushing the audio and video stream to be packaged to the streaming server based on the preset streaming protocol; the setting of the camera parameters includes: setting the camera parameters, preview callback method, exporting the camera's video frame stream through the surface texture SurfaceTexture, and constructing the surface texture SurfaceTexture object through the texture object in the open graphics drawing library OpenGL, so that when the camera's video frame stream is exported to the surface texture SurfaceTexture, it can be connected with the texture, and the texture includes an open embedded surface OES texture.

[0041] The present invention is further configured as follows: if not, parsing the audio and video stream to obtain the audio and video stream to be encapsulated, including: if not, performing streaming media protocol parsing on the audio and video stream to obtain the audio and video stream to be processed; parsing the encapsulation format of the audio and video stream to be processed to obtain a video stream in a video encoding format and an audio stream in an audio encoding format; and using the video stream in the video encoding format and the audio stream in the audio encoding format as the audio and video stream to be encapsulated.

[0042] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A multi-terminal live streaming and streaming management system based on cloud services, comprising the following steps: Step 1: When detecting that a video playback application is started, intercepting the audio and video stream to be played by the video playback application; Determine the streaming server based on the acquired streaming address; Pushing the audio and video stream to the streaming server based on a preset streaming protocol; Step 2: Get the signaling status of the target live stream; The signaling status includes a signaling running status or a signaling ending status. The signaling running status is determined by the live signaling generated by the anchor terminal in response to the live signaling running control, and the signaling ending status is determined by the live signaling generated by the anchor terminal in response to the live signaling ending control. Step 3: setting the parameters of the camera; the camera establishes a connection with the texture of the open graphics rendering library OpenGL through the surface texture SurfaceTexture, performs beautification by processing the texture, and the beautified data is rendered into the drawing buffer of the open graphics rendering library OpenGL through the texture and drawn, and then a preview is obtained, including entering a loop for processing the video stream, the loop includes processing frame by frame, the video frame is output to the texture receiving interface frame by frame, the texture object needs to obtain the latest frame data by updating the current frame buffer, the open embedded surface OES texture directly receives data, converts the open embedded surface OES texture into a 2D texture, performs a filtering and beauty algorithm on the 2D texture, and draws through the mechanism of the open graphics rendering library OpenGL; Step 4: If the signaling state is the signaling running state, monitoring the live data stream within the data stream acquisition time period; if the live data stream is acquired within the data stream acquisition time period, determining that the target push stream live broadcast is in the live broadcast progress state, retaining the target push stream live broadcast, so that the anchor terminal enters the target push stream live broadcast when it recovers from the abnormal exit state to the normal live broadcast state; Step 5: Encode the video into a specified format using MediaCodec. The camera's video stream data is passed to the OpenGL frame buffer via textures. The frame buffer data is then transferred to the input surface buffer created by MediaCodec. The data is then encoded and processed, and the encoded data is obtained through the output buffer OutPutBuffer. The obtained encoded data video stream is pushed using the real-time message transmission Rtmp protocol.

2. The multi-terminal live streaming and streaming management system based on cloud services according to claim 1 is characterized by: Before the step of intercepting the audio and video stream to be played by the video playback application when detecting that the video playback application is started, the method also includes: periodically sending a first UDP data packet carrying a streaming address request to the streaming management end based on the UDP protocol, and the streaming management end obtains the streaming address request by parsing the first UDP data packet; wherein the streaming management end is a terminal that manages, configures and / or monitors the streaming process based on the UDP data reported by the OTT set-top box; and receiving the streaming address fed back by the streaming management end based on the streaming address request.

3. The multi-terminal live streaming and streaming management system based on cloud services according to claim 1 is characterized by: After the step of pushing the audio and video stream to the streaming server based on the preset streaming protocol, it also includes: triggering fault prompt information when a fault is detected in the current streaming process; sending a second UDP data packet carrying the fault prompt information to the streaming management end based on the UDP protocol; the streaming management end feeds back a restart instruction when detecting the presence of the fault prompt information in the second UDP data; and restarting based on the received restart instruction.

4. The multi-terminal live streaming and streaming management system based on cloud services according to claim 1 is characterized by: The obtaining of the signaling status of the target live streaming includes: if a live streaming end signaling generated by the anchor terminal in response to the live streaming end control is received, determining that the signaling status of the target live streaming is the signaling end state; if a live streaming start signaling generated by the anchor terminal in response to the live streaming start control is received, or no live streaming signaling sent by the anchor terminal is received, determining that the signaling status of the target live streaming is the signaling start state; the live streaming start control is a control in the anchor terminal other than the live streaming end control.

5. The multi-terminal live streaming and streaming management system based on cloud services according to claim 4 is characterized by: Before the step of pushing the audio and video stream to the streaming server based on the preset streaming protocol, it also includes: judging whether the transmission protocol corresponding to the audio and video stream is the preset streaming protocol; if not, parsing the audio and video stream to obtain the audio and video stream to be encapsulated; accordingly, the step of pushing the audio and video stream to the streaming server based on the preset streaming protocol includes: pushing the audio and video stream to be encapsulated to the streaming server based on the preset streaming protocol.

6. The multi-terminal live streaming and streaming management system based on cloud services according to claim 5, characterized in that: The camera parameters are set, including: setting the camera parameters, preview callback method, exporting the camera's video frame stream through the surface texture SurfaceTexture, and constructing the surface texture SurfaceTexture object through the texture object in the open graphics rendering library OpenGL, so that when the camera's video frame stream is exported to the surface texture SurfaceTexture, it can be connected with the texture, and the texture includes an open embedded surface OES texture.

7. The multi-terminal live streaming and streaming management system based on cloud services according to claim 6 is characterized by: If not, the step of parsing the audio and video stream to obtain the audio and video stream to be encapsulated includes: if not, performing streaming media protocol parsing on the audio and video stream to obtain the audio and video stream to be processed; parsing the encapsulation format of the audio and video stream to be processed to obtain a video stream in a video encoding format and an audio stream in an audio encoding format; and using the video stream in the video encoding format and the audio stream in the audio encoding format as the audio and video stream to be encapsulated.

Images