Live broadcast conference stream pushing method and conference live broadcast system

Abstract

The embodiment of the invention provides a live broadcast conference stream pushing method and a conference live broadcast system. The method comprises the following steps: under the condition that auxiliary stream audios and videos exist in a live conference, coding the auxiliary stream audios and videos and main stream audios and videos of the live conference into a picture-in-picture layout based on a set picture-in-picture mode; and carrying out bypass stream pushing on the live broadcast conference to a live broadcast client in the picture-in-picture layout. Through the embodiment of the invention, the problem that the main stream audio and video and the auxiliary stream audio and video are not displayed in the same picture in the conference live broadcast in the related technology is solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of live video processing, and in particular to a live conference streaming method and a conference live broadcast system. Background Art

[0002] In the related art, during a conference live broadcast, the mainstream audio and video and the auxiliary audio and video are pushed separately through bypass channels, so that the contents of the mainstream audio and video and the auxiliary audio and video cannot be arranged and displayed on the same screen at the same time, which will increase the difficulty of integrating the live broadcast client. Summary of the invention

[0003] The embodiments of the present invention provide a live conference streaming method and a conference live broadcast system, so as to at least solve the problem in the related art that the live conference broadcast does not support the layout of the mainstream audio and video and the auxiliary audio and video for display on the same screen.

[0004] According to one embodiment of the present invention, a method for streaming a live conference is provided, comprising: in the case where auxiliary stream audio and video exist in the live conference, encoding the auxiliary stream audio and video and the mainstream audio and video of the live conference into a picture-in-picture layout based on a set picture-in-picture mode; and bypass streaming the live conference to a live client in the picture-in-picture layout.

[0005] According to another embodiment of the present invention, a conference live broadcast system is provided, including: an encoding module, which is used to encode the auxiliary stream audio and video and the mainstream audio and video of the live conference into a picture-in-picture layout based on the set picture-in-picture mode when there is auxiliary stream audio and video in the live conference; and a streaming module, which is used to bypass the live conference to the live broadcast client in the picture-in-picture layout.

[0006] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0007] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0008] Through the above-mentioned embodiments of the present invention, since a picture-in-picture mode is provided, when auxiliary stream audio and video exist in a live conference, the auxiliary stream audio and video can be arranged in the same screen with the mainstream audio and video based on the picture-in-picture mode, and pushed to the user together. Therefore, the problem in the related art that the mainstream audio and video and the auxiliary stream audio and video are not supported to be arranged in the same screen for display in the conference live broadcast can be solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a hardware structure block diagram of a computer terminal for running a live conference streaming method according to an embodiment of the present invention;

[0010] Figure 2 This is a network architecture diagram of a network device for running a live conference streaming method according to an embodiment of the present invention;

[0011] Figure 3 is a flowchart of a live conference streaming method according to an embodiment of the present invention;

[0012] Figure 4 is a structural block diagram of a conference live broadcast system according to an embodiment of the present invention;

[0013] Figure 5 is a flowchart of a conference live broadcast method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0015] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0016] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal for running a live conference streaming method according to an embodiment of the present invention. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0017] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the live conference streaming method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0018] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other computer terminals through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0019] The present application embodiment can be run on Figure 2 In the network architecture shown in Figure 2 It is shown that the network architecture includes: a conference live broadcast system, a live broadcast client and multiple conference terminals.

[0020] Conference terminal is the terminal for participating in live conferences. It is the part actually used by users and also the provider of live broadcast sources.

[0021] The conference terminals include: App0, App1, App2...AppN (N is an integer greater than zero), App0 is the conference terminal that requests to start a live conference and is also the conference terminal that initiates the mainstream audio and video; App1, App2...AppN are conference terminals that initiate / stop sharing / stop auxiliary stream audio and video.

[0022] The conference live broadcast system includes:

[0023] The CCS submodule is a submodule that interacts with App0 and is used to receive App0's request to start a live conference and set a picture-in-picture mode setting message, and send the request to start a live conference and set a picture-in-picture mode setting message to the CCM submodule.

[0024] The CCM submodule is used to receive and process requests to start a live conference, set the picture-in-picture mode, and share / stop auxiliary stream audio and video messages, and set the media routing link according to whether there is an auxiliary stream in the conference;

[0025] The FCS submodule is used for terminal auxiliary stream control, that is, decoding the received mainstream audio and video and auxiliary stream audio and video, and then encoding the decoded mainstream audio and video and auxiliary stream audio and video into a picture-in-picture layout, and transmitting the mainstream audio and video and auxiliary stream audio and video to the VPS submodule in the picture-in-picture layout, and pushing the mainstream audio and video and auxiliary stream audio and video to the live broadcast client through the VPS submodule;

[0026] MPS submodule: In the absence of auxiliary stream audio and video sharing, MPS receives and decodes the mainstream audio and video, and transmits the decoded mainstream audio and video to the VPS submodule, which pushes the mainstream audio and video to the live broadcast client through the VPS submodule;

[0027] The VPS submodule receives the live stream (main stream, auxiliary stream audio and video) according to the media routing link, and pushes the media stream of the picture-in-picture layout or the mainstream single-screen layout to the live client after encoding and decoding;

[0028] The live client module is used to collect the media stream pushed by the conference live system module, locally decode and encode to complete video synthesis, and is also the receiver of the live picture-in-picture media stream.

[0029] In this embodiment, a live conference streaming method running on the above-mentioned computer terminal or network architecture is provided. Figure 3 is a flow chart of a live conference streaming method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0030] Step S302: if there is auxiliary stream audio and video in the live conference, encode the auxiliary stream audio and video and the main stream audio and video of the live conference into a picture-in-picture layout based on the set picture-in-picture mode;

[0031] Before step S302 of this embodiment, the method further includes: when a live broadcast conference is initiated or in progress, according to a setting message for setting the picture-in-picture mode, setting the conference live broadcast system to support the picture-in-picture mode.

[0032] Setting the conference live broadcast system to support picture-in-picture mode enriches the live broadcast screen layout, which can improve the user experience during video conference live broadcasts, and reduces the difficulty of live broadcast client integration, which helps to improve the efficiency of live broadcast client development.

[0033] In step S302 of this embodiment, the auxiliary stream audio and video and the mainstream audio and video of the live conference are encoded into a picture-in-picture layout based on the set picture-in-picture mode, including: setting a first media routing link, and receiving the auxiliary stream audio and video and the mainstream audio and video through the first media routing link; respectively decoding the received auxiliary stream audio and video and the mainstream audio and video, and encoding the decoded auxiliary stream audio and video and the mainstream audio and video into a picture-in-picture layout.

[0034] In one embodiment, when the auxiliary stream audio and video does not exist in the live conference, the method further includes: setting a second media routing link, and receiving the mainstream audio and video through the second media routing link; and bypassing the mainstream audio and video to the live client in a single-screen layout.

[0035] Step S304: bypass streaming the live conference to the live client in the picture-in-picture layout.

[0036] Live streaming refers to transmitting real-time audio and video data to a server via the network (functionally equivalent to Figure 2 The server then pushes the data to the viewers in real time (i.e. Figure 2 Live broadcast client in ) to achieve real-time live broadcast effect.

[0037] Streaming can be achieved in various ways, such as using devices such as cameras and microphones to capture audio and video data, and then using streaming software to push the data to the server. Viewers can connect to the server through the network, receive the audio and video data pushed by the server, and watch the live content in real time.

[0038] In one embodiment, when a live conference is initiated or in progress, the method further includes one of the following: setting the definition of the mainstream audio and video and the auxiliary audio and video in the picture-in-picture layout; or setting the definition of the mainstream audio and video data in the single-screen layout. The definition includes at least high definition and standard definition.

[0039] In one embodiment, the definition of the mainstream audio and video and the auxiliary audio and video in the picture-in-picture layout are both set to high definition; or the definition of the mainstream audio and video and the auxiliary audio and video in the picture-in-picture layout are both set to standard definition; or, the definition of the mainstream audio and video in the picture-in-picture layout is set to high definition, and the definition of the auxiliary audio and video is set to standard definition; or, the definition of the mainstream audio and video in the picture-in-picture layout is set to standard definition, and the definition of the auxiliary audio and video is set to high definition.

[0040] For a single-screen layout, you can set the mainstream audio and video in the single-screen layout to standard definition or high definition.

[0041] After step S304 of this embodiment, the method further includes: when the auxiliary stream audio and video has stopped sharing, updating the screen layout of the live conference to a single-screen layout; and bypassing the mainstream audio and video with the single-screen layout.

[0042] In one embodiment, the method further includes: when a conference participant initiates a new auxiliary stream audio and video, updating the auxiliary stream audio and video in the picture-in-picture layout to the new auxiliary stream audio and video.

[0043] Through the above steps, since the picture-in-picture mode is set, when there is auxiliary stream audio and video in the live conference, the auxiliary stream audio and video can be arranged in the same screen with the mainstream audio and video based on the picture-in-picture mode, and pushed to the user together. Therefore, the problem in the related art that the mainstream audio and video and the auxiliary stream audio and video are not supported to be arranged in the same screen for display in the conference live broadcast can be solved.

[0044] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a computer terminal, etc.) to execute the methods described in each embodiment of the present invention.

[0045] In this embodiment, a conference live broadcast system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0046] Figure 4 is a structural block diagram of a conference live broadcast system according to an embodiment of the present invention. Figure 4 As shown, the system includes: an encoding module 10 and a streaming module 20.

[0047] The encoding module 10 is used for encoding the auxiliary stream audio and video and the main stream audio and video of the live conference into a picture-in-picture layout based on the set picture-in-picture mode when there is auxiliary stream audio and video in the live conference;

[0048] The streaming module 20 is used to perform bypass streaming of the live conference to the live client in the picture-in-picture layout.

[0049] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0050] To facilitate the understanding of the technical solution provided by the present invention, a detailed description will be given below in conjunction with embodiments of specific scenarios.

[0051] The embodiment of the present invention provides a conference live broadcast method that supports a picture-in-picture mode in which the live broadcast source is the main stream and the auxiliary stream, which can improve the viewing experience of video conferences in current remote office, online education and other scenarios. Figure 2 In the network architecture, it involves conference terminals, conference live broadcast service systems and live broadcast clients. In the conference live broadcast method, the conference live broadcast system supports the application of picture-in-picture mode, including the setting of picture-in-picture mode and the bypass push of picture-in-picture mode, specifically:

[0052] There are participants in the live conference (i.e. Figure 2 If you share audio and video with App1~AppN in the live broadcast, the live broadcast screen will automatically switch to the picture-in-picture layout;

[0053] When other participants take over and send new auxiliary audio and video, the live broadcast system can update the sub-screen in the picture-in-picture layout in real time;

[0054] When no one is sharing auxiliary audio or video in the meeting, the live broadcast screen can automatically switch to the main single screen layout.

[0055] Figure 5 is a flow chart of a conference live broadcast method according to an embodiment of the present invention. Figure 5 As shown, the method comprises the following steps:

[0056] Step S502, setting the picture-in-picture mode.

[0057] In this embodiment, the conference live broadcast system is enabled in advance to support the setting of the picture-in-picture mode, so that when there is auxiliary stream audio and video in the live conference, the main stream and auxiliary stream audio and video can be arranged in the same screen by picture-in-picture layout.

[0058] In one embodiment, when the live broadcast system initiates a conference or during a conference, the live broadcast source is set to a picture-in-picture mode.

[0059] Step S504: determine whether there is auxiliary stream audio or video in the live conference.

[0060] When the conference live broadcast system determines that there is no auxiliary stream audio or video in the live broadcast conference, step S506 is executed;

[0061] When the conference live broadcast system determines that there is auxiliary stream audio and video in the live broadcast conference, step S508 is executed.

[0062] Step S506: Process the media stream in a non-picture-in-picture mode.

[0063] Specifically, when there is no auxiliary stream audio or video in the live conference, the mainstream audio and video are processed into a mainstream single-screen layout, and the mainstream audio and video are bypassed and pushed to the live client in the mainstream single-screen layout.

[0064] Step S508: Process the media stream in a picture-in-picture mode.

[0065] Specifically, when auxiliary stream audio and video exist in a live conference, the mainstream audio and video and the auxiliary stream audio and video are processed into a picture-in-picture layout, and the auxiliary stream audio and video and the mainstream audio and video are bypassed and pushed to the live client in the picture-in-picture layout.

[0066] For example: When a participant in the meeting initiates a desktop stream, the live screen mode is automatically switched to the picture-in-picture mode. When the auxiliary stream in the live conference changes (that is, other participants initiate a new auxiliary stream audio and video), the picture-in-picture sub-screen content is changed synchronously; when the auxiliary stream audio and video in the conference stops sharing, the live screen mode is automatically switched to the non-picture-in-picture mode.

[0067] In one embodiment, there are multiple sub-pictures in the picture-in-picture layout, which are used to layout the media streams of more than two conference terminals in the same picture.

[0068] In one embodiment, the clarity of the main picture and sub-picture in the picture-in-picture layout can be customized, supporting settings such as the main picture and sub-picture are both HD, the main picture and sub-picture are both SD, and the main picture and sub-picture are a combination of HD+SD.

[0069] In one embodiment, the live broadcast client collects the bypass push stream of the conference live broadcast system and processes the synthesized video screen locally.

[0070] Specifically, when the conference live broadcast system bypasses the mainstream audio and video and pushes it to the live broadcast client, the live broadcast client encodes and decodes the mainstream audio and video locally to obtain a mainstream single screen containing only the mainstream audio and video, and displays the live broadcast content of App0 in the live broadcast client.

[0071] When the conference live broadcast system bypasses and pushes the mainstream audio and video and the auxiliary audio and video to the live broadcast client, the live broadcast client encodes and decodes the mainstream audio and video and the auxiliary audio and video locally to obtain a picture-in-picture layout screen containing the mainstream audio and video and the auxiliary audio and video, and displays the live broadcast content of any conference terminal App1 to N and the live broadcast content of App0 in the live broadcast client.

[0072] Through the above steps, the conference live broadcast service system sets the conference live broadcast mode to the picture-in-picture mode. The conference terminal joins the live broadcast conference and initiates auxiliary stream sharing. The conference live broadcast service system encodes and decodes the auxiliary stream and the main video into a picture-in-picture layout and pushes the stream to the live broadcast client for other user terminals to watch in the picture-in-picture mode on demand. In the live broadcast conference, auxiliary stream sharing is canceled, and the conference live broadcast service system updates the screen layout to only the main video single screen and pushes the stream to the live broadcast client, so that other user terminals can watch the main stream of the conference. In this way, the live broadcast screen layout can be optimized, the user experience of video conference live broadcast can be improved, and the integration difficulty of the live broadcast client can be reduced, and the development efficiency of the live broadcast client can be improved.

[0073] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0074] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0075] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0076] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0077] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0078] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for streaming a live conference, characterized in that: include: In the case where auxiliary stream audio and video exist in the live conference, encoding the auxiliary stream audio and video and the mainstream audio and video of the live conference into a picture-in-picture layout based on the set picture-in-picture mode; The live conference is streamed to the live client in a bypass manner using the picture-in-picture layout.

2. The method according to claim 1, characterized in that Before encoding the auxiliary stream audio and video and the mainstream audio and video in the live broadcast into a picture-in-picture layout based on the set picture-in-picture mode, the method further includes: When a live broadcast conference is initiated or in progress, the conference live broadcast system is set to support the picture-in-picture mode according to a setting message for setting the picture-in-picture mode.

3. The method according to claim 1, characterized in that Encoding the auxiliary stream audio and video and the mainstream audio and video in the live broadcast into a picture-in-picture layout based on the set picture-in-picture mode, including: Setting a first media routing link, and receiving the auxiliary stream audio and video and the main stream audio and video through the first media routing link; The received auxiliary stream audio and video and the mainstream audio and video are decoded respectively, and the decoded auxiliary stream audio and video and the mainstream audio and video are encoded into a picture-in-picture layout.

4. The method according to claim 1, characterized in that In the case that the auxiliary stream audio and video does not exist in the live conference, the method further includes: Setting a second media routing link, and receiving the mainstream audio and video through the second media routing link; The mainstream audio and video are bypassed and pushed to the live broadcast client in a single-screen layout.

5. The method according to claim 3 or 4, characterized in that: When a live broadcast conference is initiated or in progress, the method further includes one of the following: Setting the definition of the mainstream audio and video and the auxiliary audio and video in the picture-in-picture layout; The definition of the mainstream audio and video data in the single-screen layout is set.

6. The method according to claim 1, characterized in that After the live conference is streamed to the live client in the picture-in-picture layout through a bypass channel, the method further includes: When the auxiliary stream audio and video has stopped sharing, updating the screen layout of the live conference to a single screen layout; The mainstream audio and video are streamed in a bypass manner using the single-screen layout.

7. The method according to claim 1, characterized in that: The method further comprises: When a conference participant initiates a new auxiliary stream audio and video, the auxiliary stream audio and video in the picture-in-picture layout is updated to the new auxiliary stream audio and video.

8. A conference live broadcast system, characterized in that: include: An encoding module, used for encoding the auxiliary stream audio and video and the mainstream audio and video of the live conference into a picture-in-picture layout based on a set picture-in-picture mode when there is auxiliary stream audio and video in the live conference; The streaming module is used to bypass the live conference and stream it to the live client in the picture-in-picture layout.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images