Live broadcast conference stream pushing method and conference live broadcasting system

Abstract

Embodiments of the present disclosure provide a live broadcast conference stream pushing method and a conference live broadcasting system. The method comprises: when an auxiliary stream of audio and video exists in a live broadcast conference, on the basis of a set picture-in-picture mode, encoding the auxiliary stream of audio and video and a main stream of audio and video of the live broadcast conference into a picture-in-picture layout; and pushing the live broadcast conference stream in the picture-in-picture layout to a live broadcasting client via bypassing.

Description

Live conference streaming method and conference live broadcast system

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure is based on Chinese patent application CN202311273108.3 filed on September 26, 2023, entitled “Live Conference Streaming Method and Conference Live Broadcast System”, and claims the priority of the patent application, and all the contents disclosed therein are incorporated into this disclosure by reference. Technical Field

[0003] The embodiments of the present disclosure relate to the technical field of live video processing, and more particularly, to a live conference streaming method and a conference live broadcast system. Background Art

[0004] In related technologies, during conference live broadcasts, the main audio and video and auxiliary audio and video are pushed separately through bypass channels, making it impossible to layout and display the content of the main audio and video and auxiliary audio and video on the same screen at the same time, which will increase the difficulty of integrating the live broadcast client.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a live conference streaming method and a conference live broadcast system to at least solve the problem in the related art that conference live broadcasts do not support displaying the main stream audio and video and the auxiliary stream audio and video in the same screen.

[0007] According to one embodiment of the present disclosure, a method for streaming a live conference is provided, comprising: in the case where auxiliary stream audio and video exist in the live conference, encoding the auxiliary stream audio and video and the mainstream audio and video of the live conference into a picture-in-picture layout based on a set picture-in-picture mode; and bypass streaming the live conference to a live client in the picture-in-picture layout.

[0008] According to another embodiment of the present disclosure, a conference live broadcast system is provided, including: an encoding module, configured to, when there is auxiliary stream audio and video in the live conference, encode the auxiliary stream audio and video and the mainstream audio and video of the live conference into a picture-in-picture layout based on a set picture-in-picture mode; and a streaming module, configured to bypass the live conference to the live broadcast client in the picture-in-picture layout.

[0009] According to another embodiment of the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when running.

[0010] According to another embodiment of the present disclosure, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG1 is a hardware structure block diagram of a computer terminal for running a live conference streaming method according to an embodiment of the present disclosure;

[0012] FIG2 is a network architecture diagram of a network device for executing a live conference streaming method according to an embodiment of the present disclosure;

[0013] FIG3 is a flow chart of a method for streaming a live conference according to an embodiment of the present disclosure;

[0014] FIG4 is a structural block diagram of a conference live broadcast system according to an embodiment of the present disclosure;

[0015] FIG5 is a flowchart of a conference live broadcast method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings and in conjunction with embodiments.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0018] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a hardware structure block diagram of a computer terminal that runs a live conference streaming method in an embodiment of the present disclosure. As shown in Figure 1, the computer terminal may include one or more (only one is shown in Figure 1) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that the structure shown in Figure 1 is only for illustration, and it does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include more or fewer components than those shown in Figure 1, or have a configuration different from that shown in Figure 1.

[0019] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the live conference streaming method in the embodiment of the present disclosure. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0020] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other computer terminals via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0021] The embodiment of the present application can run on the network architecture shown in Figure 2. As shown in Figure 2, the network architecture includes: a conference live broadcast system, a live broadcast client and multiple conference terminals.

[0022] The conference terminal is the terminal for participating in live conferences. It is the part actually used by users and also the provider of the live broadcast source.

[0023] Conference terminals include: App0, App1, App2...AppN (N is an integer greater than zero). App0 is the conference terminal that requests to start a live conference and is also the conference terminal that initiates the mainstream audio and video; App1, App2...AppN are conference terminals that initiate / stop sharing / stop auxiliary stream audio and video.

[0024] The conference live broadcast system includes:

[0025] The CCS submodule is a submodule that interacts with App0 and is used to receive App0's request to start a live conference and set the picture-in-picture mode setting message, and send the request to start a live conference and the message to set the picture-in-picture mode setting to the CCM submodule.

[0026] The CCM submodule is used to receive and process requests to start a live conference, set the picture-in-picture mode, and share / stop content stream audio and video messages, and set the media routing link based on whether content streams exist in the conference;

[0027] The FCS submodule is responsible for terminal auxiliary stream control. It decodes the received main audio and video and auxiliary stream audio and video, then encodes the decoded main audio and video and auxiliary stream audio and video into a picture-in-picture layout. The picture-in-picture layout transmits the main audio and video and auxiliary stream audio and video to the VPS submodule, which pushes the main audio and video and auxiliary stream audio and video to the live broadcast client.

[0028] The MPS submodule receives and decodes the mainstream audio and video without auxiliary stream audio and video sharing, and transmits the decoded mainstream audio and video to the VPS submodule, which pushes the mainstream audio and video to the live broadcast client through the VPS submodule;

[0029] The VPS submodule receives live streams (main stream, auxiliary stream audio and video) according to the media routing link, encodes and decodes them, and pushes the media streams in picture-in-picture layout or mainstream single-screen layout to the live client;

[0030] The live broadcast client module is used to collect the media stream pushed by the conference live broadcast system module, locally decode and encode to complete video synthesis, and is also the receiver of the live picture-in-picture media stream.

[0031] In this embodiment, a method for pushing live conference streaming running on the above-mentioned computer terminal or network architecture is provided. FIG3 is a flow chart of the method for pushing live conference streaming according to an embodiment of the present disclosure. As shown in FIG3 , the flow chart includes the following steps:

[0032] Step S302: If auxiliary audio and video streams exist in the live conference, encoding the auxiliary audio and video streams and the main audio and video streams of the live conference into a picture-in-picture layout based on the set picture-in-picture mode;

[0033] Before step S302 of this embodiment, the method further includes: when a live broadcast conference is initiated or during the live broadcast conference, according to a setting message for setting the picture-in-picture mode, setting the conference live broadcast system to support the picture-in-picture mode.

[0034] Setting the conference live broadcast system to support picture-in-picture mode enriches the live broadcast screen layout, which can improve the user experience during video conference live broadcasts, and reduces the difficulty of live broadcast client integration, which helps improve the efficiency of live broadcast client development.

[0035] In step S302 of this embodiment, the auxiliary stream audio and video and the mainstream audio and video of the live conference are encoded into a picture-in-picture layout based on the set picture-in-picture mode, including: setting a first media routing link, and receiving the auxiliary stream audio and video and the mainstream audio and video through the first media routing link; respectively decoding the received auxiliary stream audio and video and the mainstream audio and video, and encoding the decoded auxiliary stream audio and video and the mainstream audio and video into a picture-in-picture layout.

[0036] In one embodiment, when the auxiliary stream audio and video does not exist in the live conference, the method further includes: setting a second media routing link and receiving the mainstream audio and video through the second media routing link; and bypassing the mainstream audio and video to the live client in a single-screen layout.

[0037] Step S304: Push the live conference to the live client in a bypass channel using the picture-in-picture layout.

[0038] Live streaming refers to the transmission of real-time audio and video data via the network to a server (functionally equivalent to the conference live broadcast system in Figure 2), and then the server pushes this data to viewers in real time (i.e., the live broadcast client in Figure 2), achieving a real-time live broadcast effect.

[0039] Push streaming can be achieved in various ways, such as using devices such as cameras and microphones to capture audio and video data, and then using streaming software to push the data to the server. Viewers can connect to the server through the Internet, receive the audio and video data pushed by the server, and watch the live content in real time.

[0040] In one embodiment, when a live conference is initiated or in progress, the method further includes one of the following: setting the definition of the main audio and video and the auxiliary audio and video in the picture-in-picture layout; or setting the definition of the main audio and video data in the single-screen layout. Definitions include at least high definition and standard definition.

[0041] In one embodiment, the definition of the mainstream audio and video and the auxiliary audio and video in the picture-in-picture layout are both set to high definition; or the definition of the mainstream audio and video and the auxiliary audio and video in the picture-in-picture layout are both set to standard definition; or the definition of the mainstream audio and video in the picture-in-picture layout is set to high definition, and the definition of the auxiliary audio and video are both set to standard definition; or the definition of the mainstream audio and video in the picture-in-picture layout is set to standard definition, and the definition of the auxiliary audio and video are both set to high definition.

[0042] For a single-screen layout, you can set the mainstream audio and video in the single-screen layout to standard definition or high definition.

[0043] After step S304 of this embodiment, the method further includes: updating the screen layout of the live conference to a single-screen layout when the auxiliary stream audio and video has stopped being shared; and bypassing the mainstream audio and video with the single-screen layout.

[0044] In one embodiment, the method further includes: when a conference participant initiates a new auxiliary stream audio and video, updating the auxiliary stream audio and video in the picture-in-picture layout to the new auxiliary stream audio and video.

[0045] Through the above steps, thanks to the Picture-in-Picture mode, when a live conference includes a secondary audio or video stream, the secondary audio or video can be placed on the same screen as the main audio or video, and streamed to the user together. This solves the problem in related art where live conference broadcasts do not support placing the main audio or video and the secondary audio or video on the same screen.

[0046] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or computer terminal, etc.) to execute the methods described in each embodiment of the present disclosure.

[0047] In this embodiment, a conference live broadcast system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation methods. The details that have been described will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0048] FIG4 is a structural block diagram of a conference live broadcast system according to an embodiment of the present disclosure. As shown in FIG4 , the system includes: an encoding module 10 and a streaming module 20 .

[0049] The encoding module 10 is configured to encode the auxiliary audio and video stream and the main audio and video of the live conference into a picture-in-picture layout based on the set picture-in-picture mode when auxiliary audio and video streams are present in the live conference;

[0050] The streaming module 20 is configured to perform a bypass streaming of the live conference to the live client in the picture-in-picture layout.

[0051] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0052] To facilitate understanding of the technical solutions provided by the present disclosure, embodiments of specific scenarios will be described in detail below.

[0053] The disclosed embodiments provide a conference live broadcast method that supports a picture-in-picture mode with a live stream source as the main stream and auxiliary stream, which can improve the viewing experience of video conferences in current scenarios such as remote office and online education. The method is applied to the network architecture of Figure 2, that is, it involves a conference terminal, a conference live broadcast service system, and a live broadcast client. In this conference live broadcast method, the conference live broadcast system supports the application of picture-in-picture mode, including the setting of picture-in-picture mode and the bypass push of picture-in-picture mode.

[0054] In one embodiment, if a participant (i.e., App1 to AppN in FIG2 ) in a live conference shares a consonant video, the live screen will automatically switch to a picture-in-picture layout;

[0055] When other participants preemptively send new content streams, the live broadcast system can update the sub-screens in the picture-in-picture layout in real time.

[0056] When no one is sharing auxiliary audio or video in the meeting, the live broadcast screen can automatically switch to the main single screen layout.

[0057] FIG5 is a flow chart of a conference live broadcast method according to an embodiment of the present disclosure. As shown in FIG5 , the method includes the following steps:

[0058] Step S502: Set the picture-in-picture mode.

[0059] In this embodiment, the conference live broadcast system is pre-enabled to support the setting of picture-in-picture mode, so that when there is auxiliary stream audio and video in the live conference, the main stream and auxiliary stream audio and video can be arranged in the same screen using picture-in-picture layout.

[0060] In one embodiment, when the live broadcast system initiates a conference or during the conference, the live broadcast source is set to picture-in-picture mode.

[0061] Step S504: Determine whether there is auxiliary audio or video stream in the live conference.

[0062] If the conference live broadcast system determines that there is no auxiliary stream audio or video in the live conference, step S506 is executed;

[0063] When the conference live broadcast system determines that there is auxiliary stream audio and video in the live broadcast conference, step S508 is executed.

[0064] Step S506: Process the media stream in a non-picture-in-picture mode.

[0065] In one embodiment, when there is no auxiliary audio or video stream in a live conference, the mainstream audio and video are processed into a mainstream single-screen layout, and the mainstream audio and video are bypassed and pushed to the live client in the mainstream single-screen layout.

[0066] Step S508: Process the media stream in picture-in-picture mode.

[0067] In one embodiment, when auxiliary audio and video streams exist in a live conference, the main audio and video streams and the auxiliary audio and video streams are processed into a picture-in-picture layout, and the auxiliary audio and video streams and the main audio and video streams are bypassed and pushed to the live client in the picture-in-picture layout.

[0068] For example, when a participant initiates a desktop stream in a conference, the live screen mode automatically switches to picture-in-picture mode. When the content stream changes (i.e., another participant initiates a new content stream audio or video), the content of the picture-in-picture sub-screen changes synchronously. When the content stream audio or video stops sharing in a conference, the live screen mode automatically switches to non-picture-in-picture mode.

[0069] In one embodiment, there are multiple sub-pictures in the picture-in-picture layout, which are used to arrange the media streams of more than two conference terminals in the same picture.

[0070] In one embodiment, the clarity of the main picture and sub-picture in the picture-in-picture layout can be customized, supporting settings such as the main picture and sub-picture are both high-definition, the main picture and sub-picture are both standard-definition, and the main picture and sub-picture are a combination of high-definition and standard-definition.

[0071] In one embodiment, the live broadcast client collects the bypass stream of the conference live broadcast system and processes the synthesized video images locally.

[0072] In one embodiment, when the conference live broadcast system bypasses the mainstream audio and video and pushes it to the live broadcast client, the live broadcast client encodes and decodes the mainstream audio and video locally to obtain a mainstream single screen containing only the mainstream audio and video, and displays the live broadcast content of App0 in the live broadcast client.

[0073] When the conference live broadcast system bypasses and pushes the mainstream audio and video and auxiliary audio and video to the live broadcast client, the live broadcast client locally encodes and decodes the mainstream audio and video and auxiliary audio and video to obtain a picture-in-picture layout screen containing the mainstream audio and video and auxiliary audio and video. The live broadcast client displays the live broadcast content of any conference terminal App1 to N and the live broadcast content of App0.

[0074] Through the above steps, the live conference service system sets the conference live broadcast mode to picture-in-picture mode. When a conference terminal joins the live conference and initiates auxiliary stream sharing, the live conference service system encodes and decodes the auxiliary stream and the main video into a picture-in-picture layout and pushes it to the live broadcast client for other user terminals to watch in picture-in-picture mode on demand. During the live conference, auxiliary stream sharing is canceled, and the live conference service system updates the screen layout to a single screen with only the main video, which is pushed to the live broadcast client for other user terminals to watch the main conference content. This optimizes the live broadcast screen layout, improves the user experience of video conference live broadcasts, reduces the difficulty of live broadcast client integration, and improves the efficiency of live broadcast client development.

[0075] An embodiment of the present disclosure further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above method embodiments when run.

[0076] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0077] An embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0078] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0079] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0080] Obviously, those skilled in the art should understand that the modules or steps of the present disclosure described above can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed across a network composed of multiple computing devices, they can be implemented using program code executable by the computing device, and thus, they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in a different order than herein, or they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any particular combination of hardware and software.

[0081] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations of the present disclosure are possible. Any modifications, equivalent substitutions, or improvements made within the principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A method for streaming a live conference, comprising: In the case where auxiliary stream audio and video exist in the live conference, encoding the auxiliary stream audio and video and the mainstream audio and video of the live conference into a picture-in-picture layout based on the set picture-in-picture mode; The live conference is streamed to the live client in a bypass manner using the picture-in-picture layout.

2. The method according to claim 1, wherein: Before encoding the auxiliary stream audio and video and the mainstream audio and video in the live broadcast into a picture-in-picture layout based on the set picture-in-picture mode, the method further includes: When a live broadcast conference is initiated or in progress, the conference live broadcast system is set to support the picture-in-picture mode according to a setting message for setting the picture-in-picture mode.

3. The method according to claim 1, wherein: The auxiliary stream audio and video and the main stream audio and video in the live broadcast are encoded into a picture-in-picture layout based on the set picture-in-picture mode, including: Setting a first media routing link, and receiving the auxiliary stream audio and video and the main stream audio and video through the first media routing link; The received auxiliary stream audio and video and the mainstream audio and video are decoded respectively, and the decoded auxiliary stream audio and video and the mainstream audio and video are encoded into a picture-in-picture layout.

4. The method according to claim 1, wherein: In the case that the auxiliary stream audio and video does not exist in the live conference, the method further includes: Setting a second media routing link, and receiving the mainstream audio and video through the second media routing link; The mainstream audio and video are bypassed and pushed to the live broadcast client in a single-screen layout.

5. The method according to claim 3 or 4, wherein: When a live broadcast conference is initiated or in progress, the method further includes one of the following: Setting the definition of the mainstream audio and video and the auxiliary audio and video in the picture-in-picture layout; The definition of the mainstream audio and video data in the single-screen layout is set.

6. The method according to claim 1, wherein: After the live conference is streamed to the live client in the picture-in-picture layout through a bypass channel, the method further includes: When the auxiliary stream audio and video has stopped sharing, updating the screen layout of the live conference to a single screen layout; The mainstream audio and video are streamed in a bypass manner using the single-screen layout.

7. The method according to claim 1, further comprising: When a conference participant initiates a new auxiliary stream audio and video, the auxiliary stream audio and video in the picture-in-picture layout is updated to the new auxiliary stream audio and video.

8. A conference live broadcast system, comprising: an encoding module, configured to encode the auxiliary stream audio and video and the mainstream audio and video of the live conference into a picture-in-picture layout based on a set picture-in-picture mode when there is auxiliary stream audio and video in the live conference; The streaming module is configured to perform bypass streaming of the live conference to the live client in the picture-in-picture layout.

9. A computer-readable storage medium having a computer program stored therein, wherein: When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of claims 1 to 7 when executing the computer program.

Images