Livestreaming co-streaming method, device, and storage medium

Abstract

Embodiments of the present disclosure provide a livestreaming co-streaming method, a device, and a storage medium. The method comprises: acquiring, by any livestreaming participant client in a first multi-person co-streaming livestreaming room, a first real-time multimedia stream of at least one livestreaming participant client in the first multi-person co-streaming livestreaming room, and playing back the first real-time multimedia stream; on the basis of a cross-room co-streaming instruction, acquiring public stream data of a second multi-person co-streaming livestreaming room, wherein the public stream data is formed by combining second real-time multimedia streams of multiple livestreaming participant clients in the second multi-person co-streaming livestreaming room; and playing back the public stream data. During cross-room co-streaming, only one path of public stream data needs to be acquired for an opposite multi-person co-streaming livestreaming room, thereby relieving the pressure on the performance of livestreaming clients, reducing the bandwidth and traffic costs, and improving livestreaming co-streaming experience.

Description

Live broadcast and microphone connection method, device and storage medium

[0001] This application claims priority to the Chinese invention patent application entitled “Live Broadcasting Method, Device and Storage Medium” filed on June 16, 2023, with application number 202310721520.0. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present disclosure relate to the field of computer and network communication technology, and more particularly to a live broadcast and microphone connection method, device, and storage medium. Background Art

[0003] During live streaming, live broadcasters often use microphones to interact with other broadcasters. Based on this method, the current broadcaster can invite other broadcasters to join the live broadcast.

[0004] In existing technologies, when live streaming with microphones connected, each participant's client needs to obtain the audio and video streams of all other participants from the server. If there are many participants, this puts pressure on the client's performance, increases bandwidth and data costs, and is prone to problems, seriously affecting the user experience.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a live broadcast and microphone connection method, device, and storage medium to reduce the pressure on the live broadcast client performance when connecting to microphones across rooms, reduce bandwidth and traffic costs, and improve the live broadcast and microphone connection experience.

[0007] In a first aspect, an embodiment of the present disclosure provides a live broadcast connection method, which is applied to any live broadcast participant client in a first multi-person live broadcast room. The method includes: obtaining a first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room and playing it; based on a cross-room connection instruction, obtaining public stream data of a second multi-person live broadcast room, wherein the public stream data is a confluence of second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room; and playing the public stream data.

[0008] In a second aspect, an embodiment of the present disclosure provides a live broadcast and microphone connection method, which is applied to a server, and the method includes: pushing a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast and microphone connection room and a second real-time multimedia stream of at least one live broadcast participant client in a second multi-person live broadcast and microphone connection room to the live broadcast participant clients in the respective multi-person live broadcast and microphone connection rooms; based on a cross-room live broadcast and microphone connection instruction, pushing first public stream data corresponding to the first multi-person live broadcast and microphone connection room to the live broadcast participant clients in the second multi-person live broadcast and microphone connection room, and pushing second public stream data corresponding to the second multi-person live broadcast and microphone connection room to the live broadcast participant clients in the first multi-person live broadcast and microphone connection room.

[0009] In a third aspect, an embodiment of the present disclosure provides a live broadcast participant client, comprising: a local room stream pulling unit, for obtaining a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room with microphones connected, and playing it; a public stream pulling unit, for obtaining public stream data of a second multi-person live broadcast room with microphones connected based on a cross-room live broadcast instruction, wherein the public stream data is formed by merging the second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room with microphones connected; a control unit, for controlling the playback of the public stream data

[0010] In a fourth aspect, an embodiment of the present disclosure provides a server, including: a same-room streaming push unit, for pushing a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room and a second real-time multimedia stream of at least one live broadcast participant client in a second multi-person live broadcast room to the live broadcast participant clients in the respective multi-person live broadcast rooms; a public stream streaming push unit, for pushing first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant clients in the second multi-person live broadcast room based on a cross-room live broadcast instruction, and pushing second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant clients in the first multi-person live broadcast room.

[0011] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live broadcast and microphone connection method described in the first aspect and various possible designs of the first aspect, or the live broadcast and microphone connection method described in the second aspect and various possible designs of the second aspect.

[0012] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the live broadcast and microphone connection method described in the first aspect and various possible designs of the first aspect, or the live broadcast and microphone connection method described in the second aspect and various possible designs of the second aspect are implemented.

[0013] In the seventh aspect, an embodiment of the present disclosure provides a computer program product, including computer-executable instructions. When a processor executes the computer-executable instructions, it implements the live broadcast and microphone connection method described in the first aspect and various possible designs of the first aspect, or the live broadcast and microphone connection method described in the second aspect and various possible designs of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] FIG1a is an example diagram of an interface of a method for live broadcasting and connecting microphones across rooms in the prior art;

[0016] FIG1b is a system diagram of a method for cross-room live broadcast and microphone connection in the prior art;

[0017] FIG2 is a system diagram of a live broadcast and microphone connection method provided by an embodiment of the present disclosure;

[0018] FIG3 is a flowchart of a live broadcast and microphone connection method according to an embodiment of the present disclosure;

[0019] FIG4 is a flowchart of a live broadcast and microphone connection method provided by another embodiment of the present disclosure;

[0020] FIG5 is a schematic diagram of a server architecture provided by an embodiment of the present disclosure;

[0021] FIG6 is a structural block diagram of a live broadcast participant client provided by an embodiment of the present disclosure;

[0022] FIG7 is a structural block diagram of a server provided by an embodiment of the present disclosure;

[0023] FIG8 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0025] During live streaming, live streamers often use a live broadcasting method to interact with other streamers. This method allows the current streamer to invite other streamers to join the live stream. In existing technologies, when live streaming is performed via live broadcasting, each participating client needs to pull the audio and video streams of all other participating clients in the live stream room from the server.

[0026] In a cross-room live broadcast scenario, multiple hosts are already connected to the mic in one live broadcast room, and multiple hosts are also connected to the mic in another live broadcast room. The two live broadcast rooms can then connect to each other (for example, to host a battle between the two rooms), allowing all hosts in the cross-room connected rooms to communicate. The host client in each live broadcast room can simultaneously view the images of all hosts in both live broadcast rooms, as shown in Figure 1a.

[0027] In existing cross-room live broadcasting methods, each participant client in Live Room 1 not only pulls the audio and video streams from all other participants in its own room, but also needs to push its own audio and video streams to the opposite room. Furthermore, it also needs to pull the audio and video streams from all other participants in the opposite room. This process is illustrated in Figure 1b. Based on the server, each participant client in Live Room 1 not only pushes audio and video streams to Live Room 1, but also to Live Room 2. Similarly, each participant client in Live Room 2 not only pushes audio and video streams to Live Room 2, but also to Live Room 2. Thus, each participant client in Live Room 1 not only pulls the audio and video streams from other participants in its own room, but also pulls the audio and video streams pushed to Live Room 1 by each participant client in Live Room 2. Each participant client in Live Room 2 not only pulls the audio and video streams from other participants in its own room, but also pulls the audio and video streams pushed to Live Room 2 by each participant client in Live Room 1.

[0028] In cross-room live broadcast, if there are many participants, more audio and video streams need to be pulled, and each audio and video stream needs to run a decoder to decode, which puts performance pressure on the client. For example, if 11 audio and video streams need to be pulled, 11 decoders need to be run. Since there are many audio and video streams to be pulled, and each audio and video stream is a low-latency RTC (Real-Time Communications) audio and video stream, the bandwidth and traffic costs are high; and problems are prone to occur. For example, at the end of the cross-room live broadcast, it is necessary to stop pulling the audio and video streams of all live broadcast participant clients in the opposite live broadcast room. However, if the audio and video stream of any live broadcast participant client in the opposite live broadcast room does not stop pushing the stream to this live broadcast room, the audio and video streams of each live broadcast participant client in this live broadcast room can still be pulled, resulting in crosstalk problems, which seriously affects the experience.

[0029] In order to solve the above technical problems, the present disclosure provides a live broadcast connection method. In order to reduce the number of real-time multimedia streams, it is possible to consider merging multiple real-time multimedia streams, that is, to adopt the MCU (Multipoint Conferencing Unit) architecture in the WebRTC technology. However, due to the delay in the MCU architecture, if all real-time multimedia streams are merged, the real-time performance of the live broadcast connection will be affected. Considering that when connecting to the microphone across rooms, in certain scenarios such as battles between rooms, the live broadcast participants in the same multi-person live broadcast room need to talk in real time, while the live broadcast participants in different multi-person live broadcast rooms do not need to talk in real time (perhaps one multi-person live broadcast room mutes another multi-person live broadcast room), and the real-time performance requirements are not high, the MCU architecture and SFU (Selective Forwarding Unit) architecture (existing technology) in the WebRTC technology can be combined. That is, only the real-time multimedia streams of each live broadcast participant client in the other multi-person live broadcast room are merged into a common stream data, while the real-time multimedia streams of each live broadcast participant client in the current room are still pulled separately in real time. This allows users to pull only the shared stream data from the other multi-room live broadcast room, eliminating the need to pull the real-time multimedia streams from each participant's client. This reduces the number of decoders running, alleviating the performance strain on the live broadcast client and lowering bandwidth and data costs. Furthermore, terminating a cross-room live broadcast is simple, reducing crosstalk and improving the live broadcast experience.

[0030] Specifically, the live broadcast method process may include obtaining the first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room through any live broadcast participant client in the first multi-person live broadcast room, and playing it; obtaining the public stream data of the second multi-person live broadcast room based on the cross-room live broadcast instruction, wherein the public stream data is a multimedia stream formed by the confluence of the second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room; and playing the public stream data. At this time, as shown in Figure 2, the data pulled by any live broadcast participant client in the first multi-person live broadcast room includes the first real-time multimedia stream of at least one other live broadcast participant client in the current first multi-person live broadcast room, and the public stream data of the second multi-person live broadcast room, thereby realizing cross-room live broadcast. The other live broadcast participant clients in the first multi-person live broadcast room and the live broadcast participant clients in the second multi-person live broadcast room are similar to Figure 2 and are not shown one by one here.

[0031] It should be noted that the user information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] The live broadcast and microphone connection method disclosed in the present invention will be described in detail below with reference to specific embodiments.

[0033] Refer to Figure 3, which is a flow chart of a live broadcast and microphone connection method provided by an embodiment of the present disclosure. The method of this embodiment can be applied in a live broadcast participant client, and the live broadcast and microphone connection method includes:

[0034] S201: Obtain and play a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room.

[0035] In this embodiment, the live broadcast participant client in the first multi-person live broadcast room is the client participating in the live broadcast room. Multiple live broadcast participant clients (including hosts and guests) in the first multi-person live broadcast room are connected to each other. This means that each live broadcast participant client in the first multi-person live broadcast room can access the real-time multimedia stream (referred to as the first real-time multimedia stream) of all other live broadcast participant clients in the first multi-person live broadcast room, excluding its own client, and play it on the client interface. This allows each live broadcast participant in the first multi-person live broadcast room to not only see its own video on its client interface, but also simultaneously play the video and audio of other live broadcast participants in the first multi-person live broadcast room. Of course, any live broadcast participant client in the first multi-person live broadcast room can also disable video and audio capture. This means that the live broadcast participant client does not capture its own first real-time multimedia stream and does not send its own first real-time multimedia stream to the server, but can obtain the first real-time multimedia streams of other live broadcast participant clients from the server. This means that the live broadcast participant client cannot see its own video on the client interface, but can play the video and audio of other live broadcast participants in the first multi-person live broadcast room. Therefore, any live broadcast participant client in the first multi-person live broadcast room can obtain the first real-time multimedia stream of at least one other live broadcast participant client from the server. If all other live broadcast participant clients turn off video and audio capture, they can only play their own first real-time multimedia stream.

[0036] Optionally, the first multi-person live broadcast room is an RTC room. RTC, also known as WebRTC (Web Real-Time Communications), is a real-time communication technology that allows network applications or sites to establish peer-to-peer connections between browsers without the help of an intermediary, thereby enabling the transmission of video streams and / or audio streams or any other data. Accordingly, the first real-time multimedia stream is an RTC stream. Each live broadcast participant client in the RTC room can push its own RTC stream to the RTC room in the server, and can also pull the RTC streams of other live broadcast participant clients from the RTC room in the server.

[0037] Similarly, the second multi-person live broadcast room is another multi-person live broadcast room, and the live broadcast participant client of the first multi-person live broadcast room is the client participating in the live broadcast in the second multi-person live broadcast room. The multiple live broadcast participants (including anchors and guests) in the second multi-person live broadcast room are also connected by microphone. That is, each live broadcast participant client in the second multi-person live broadcast room can obtain the real-time multimedia stream (recorded as the second real-time multimedia stream) of at least one other live broadcast participant client in the second multi-person live broadcast room and play it in the client interface. This allows each live broadcast participant in the second multi-person live broadcast room to play the acquired picture and sound of at least one other live broadcast participant in the second multi-person live broadcast room on its client, and can also see its own picture (if the video and audio collection is turned off, the user cannot see his own picture).

[0038] S202: Acquire public stream data of the second multi-person live broadcast room, wherein the public stream data is formed by merging second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room.

[0039] In this embodiment, when a cross-room microphone connection is required between a first multi-person live broadcast room and a second multi-person live broadcast room, any live broadcast participant client in any multi-person live broadcast room can initiate a cross-room microphone connection request to the other multi-person live broadcast room. After any live broadcast participant client in the other multi-person live broadcast room agrees to the cross-room microphone connection request, the cross-room microphone connection between the first multi-person live broadcast room and the second multi-person live broadcast room begins.

[0040] Since, when a cross-room connection is made between a first multi-person live broadcast room and a second multi-person live broadcast room, in scenarios such as inter-room battles, real-time communication is required between the participants in the same multi-person live broadcast room, while real-time communication is not required between the participants in different multi-person live broadcast rooms. It is also possible that one multi-person live broadcast room will mute another multi-person live broadcast room. Therefore, the real-time requirements are not high. In addition, in order to avoid the need to pull the real-time multimedia stream of each live broadcast participant client in the other multi-person live broadcast room for cross-room connection, and to run a decoder for each real-time multimedia stream, which puts pressure on client performance and increases bandwidth and traffic costs, in this embodiment, the real-time multimedia stream of each live broadcast participant client in the other multi-person live broadcast room is merged into a common stream data. In this way, only the common stream data of the other multi-person live broadcast room needs to be pulled, without having to pull the real-time multimedia streams of multiple live broadcast participant clients in the other multi-person live broadcast room separately.

[0041] In RTC technology, RTC rooms allow users to directly access multimedia streams within an RTC room without entering the room. This is referred to as a public stream. This public stream is a confluence of the multimedia streams within the RTC room. Therefore, in this embodiment, the same method for obtaining public streams in RTC rooms can be applied to obtain public stream data for each multi-person live broadcast room. Furthermore, public stream data can still be published via RTC streaming to minimize latency.

[0042] Optionally, the public stream data of the first multi-person live broadcast room and the public stream data of the second multi-person live broadcast room can be configured with identification information (ID) of the public stream data respectively. The identification information of the public stream data can be configured by the server, or can also be configured by the live broadcast participant client, and each live broadcast participant client of the first multi-person live broadcast room knows the identification information of the public stream data of the second multi-person live broadcast room, and each live broadcast participant client of the second multi-person live broadcast room knows the identification information of the public stream data of the first multi-person live broadcast room. Further, any live broadcast participant client of the first multi-person live broadcast room can obtain the public stream data of the second multi-person live broadcast room based on the identification information of the public stream data of the second multi-person live broadcast room. Any live broadcast participant client of the second multi-person live broadcast room can obtain the public stream data of the first multi-person live broadcast room based on the identification information of the public stream data of the first multi-person live broadcast room. Optionally, the public stream data can be obtained according to the identification information of the public stream data through the publish / subscribe mode.

[0043] S203: Play the public stream data.

[0044] In this embodiment, after any live broadcast participant client of the first multi-person live broadcast room obtains the public stream data of the second multi-person live broadcast room, it can play the public stream data. At this time, it is still necessary to continue to obtain the first real-time multimedia stream of other live broadcast participant clients in the first multi-person live broadcast room for playback to ensure the real-time performance of the first multi-person live broadcast room. Similarly, for any live broadcast participant client of the second multi-person live broadcast room, after obtaining the public stream data of the first multi-person live broadcast room, it can play the public stream data. It is also still necessary to continue to obtain the second real-time multimedia stream of other live broadcast participant clients in the second multi-person live broadcast room for playback to ensure the real-time performance of the second multi-person live broadcast room.

[0045] For any live broadcast participant client in the second multi-person live broadcast room, the steps are similar to those for any live broadcast participant client in the first multi-person live broadcast room, and will not be repeated here.

[0046] In this embodiment, when it is necessary to end the cross-room live broadcast, any live broadcast participant client of the first multi-person live broadcast room can directly stop obtaining and playing the public stream data of the second multi-person live broadcast room, and no crosstalk problem will occur.

[0047] The live broadcast method provided in this embodiment obtains the first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room through any live broadcast participant client in the first multi-person live broadcast room, and plays it; based on the cross-room broadcast instruction, obtains the public stream data of the second multi-person live broadcast room, wherein the public stream data is formed by the confluence of the second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room; and plays the public stream data. When connecting to the live broadcast room across rooms, only one channel of public stream data needs to be obtained for the other multi-person live broadcast room, avoiding pulling the real-time multimedia stream of each live broadcast participant client in the other multi-person live broadcast room, reducing bandwidth and traffic costs, and alleviating pressure on the performance of the live broadcast client. Since the number of streams pulled is reduced, the number of running decoders is reduced, the CPU overhead is reduced, and at the same time, the number of images that need to be rendered is reduced during GPU rendering, the drawing instructions are reduced, and the GPU overhead is reduced. In addition, the cross-room broadcast ending process is simple, crosstalk is not likely to occur, and the live broadcast experience is improved.

[0048] Based on any of the above embodiments, another embodiment of the present disclosure provides a live broadcast and microphone connection method, the live broadcast and microphone connection method comprising:

[0049] S301: Obtain and play a first real-time multimedia stream from at least one live broadcast participant client in a first multi-person live broadcast room. This step is the same as S201.

[0050] S302: Based on the cross-room live broadcast instruction, obtain and play a second real-time multimedia stream of multiple live broadcast participant clients in a second multi-person live broadcast room.

[0051] In this embodiment, when a cross-room connection is required between a first multi-person live broadcast room and a second multi-person live broadcast room, any live broadcast participant client in either multi-person live broadcast room can initiate a cross-room connection request to the other multi-person live broadcast room. After any live broadcast participant client in the other multi-person live broadcast room agrees to the cross-room connection request, the cross-room connection between the first multi-person live broadcast room and the second multi-person live broadcast room begins, and a cross-room connection instruction is issued to each live broadcast participant client in the two multi-person live broadcast rooms. Considering that the server-side stream converging operation takes a certain amount of time, each live broadcast participant client can obtain the real-time multimedia streams of multiple live broadcast participant clients in the other multi-person live broadcast room according to the cross-room connection instruction, thereby ensuring a rapid response to the cross-room connection instruction without waiting for public stream data.

[0052] For example, any live broadcast participant client in the first multi-person live broadcast room obtains the second real-time multimedia stream of multiple live broadcast participant clients in the second multi-person live broadcast room, and plays it in the client interface, so that in addition to playing its own picture and the picture and sound of at least one other live broadcast participant in the first multi-person live broadcast room (if the video and audio capture is turned off, the user cannot see its own picture), the live broadcast participant client can also simultaneously play the pictures and sounds of multiple live broadcast participants in the second multi-person live broadcast room. Any live broadcast participant client in the second multi-person live broadcast room obtains the first real-time multimedia stream of multiple live broadcast participant clients in the first multi-person live broadcast room, and plays it in the client interface, so that in addition to playing its own picture (if the video and audio capture is turned off, the user cannot see its own picture) and the picture and sound of at least one other live broadcast participant in the second multi-person live broadcast room, the live broadcast participant client can also simultaneously play the pictures and sounds of multiple live broadcast participants in the first multi-person live broadcast room.

[0053] Optionally, in response to the cross-room live broadcast instruction, multiple live broadcast participant clients in the second multi-person live broadcast room push their respective second real-time multimedia streams to the first multi-person live broadcast room on the server. Multiple live broadcast participant clients in the first multi-person live broadcast room push their respective first real-time multimedia streams to the second multi-person live broadcast room on the server.

[0054] Optionally, the server can also perform cross-room streaming in response to the cross-room live broadcast instruction. Specifically, the first real-time multimedia stream of multiple live broadcast participant clients in the first multi-person live broadcast room on the server is pushed to the second multi-person live broadcast room, and the second real-time multimedia stream of multiple live broadcast participant clients in the second multi-person live broadcast room on the server is pushed to the first multi-person live broadcast room. Each live broadcast participant client can simultaneously obtain the first real-time multimedia stream and the second real-time multimedia stream from its respective multi-person live broadcast room on the server for playback.

[0055] S303: Based on the cross-room live broadcast instruction, obtain the public stream data of the second multi-person live broadcast room.

[0056] In this embodiment, after the server merges the second real-time multimedia streams to obtain common stream data, any live broadcast participant client in the first multi-person live broadcast room can obtain the common stream data of the second multi-person live broadcast room. Similarly, after the server merges the first real-time multimedia streams to obtain common stream data, any live broadcast participant client in the second multi-person live broadcast room can obtain the common stream data of the first multi-person live broadcast room. For details, see S202.

[0057] S304: Play the common stream data, and stop acquiring and playing the second real-time multimedia stream.

[0058] In this embodiment, after any live broadcast participant client in the first multi-person live broadcast room obtains the public stream data of the second multi-person live broadcast room, it can play the public stream data to replace the second real-time multimedia streams played by multiple live broadcast participant clients in the second multi-person live broadcast room. In other words, the acquisition (mute operation) and playback of the multiple second real-time multimedia streams can be stopped, but the acquisition and playback of the first real-time multimedia stream of at least one other live broadcast participant client in the first multi-person live broadcast room must still be continued to ensure the real-time performance of the first multi-person live broadcast room.

[0059] Similarly, any live broadcast participant client in the second multi-person live broadcast room, after acquiring the public stream data of the first multi-person live broadcast room, can play the public stream data to replace the second real-time multimedia stream played by multiple live broadcast participant clients in the first multi-person live broadcast room. In other words, the acquisition (mute operation) and playback of multiple first real-time multimedia streams can be stopped, but the acquisition and playback of the second real-time multimedia stream of at least one other live broadcast participant client in the second multi-person live broadcast room must still be continued to ensure the real-time performance of the second multi-person live broadcast room.

[0060] Optionally, in order to ensure smooth switching between each second real-time multimedia stream and the public stream data without lag, any live participant client in the first multi-person live broadcast room can, after obtaining the public stream data, keep playing each second real-time multimedia stream while decoding the public stream data. After the first frame of the public stream data is successfully decoded, the second real-time multimedia stream and the public stream data are switched. That is, the public stream data is played, and the acquisition and playback of multiple second real-time multimedia streams is stopped. Of course, if the decoding of the first frame of the public stream data fails, the acquisition and playback of multiple second real-time multimedia streams can continue.

[0061] Optionally, if any live participant client in the first multi-person live broadcast room experiences a playback failure while playing the public stream data, such as due to a transmission interruption or decoding error, multiple second real-time multimedia streams can be obtained and played. In other words, the public stream data can be switched back to the multiple second real-time multimedia streams to ensure the continuation of the live broadcast. Alternatively, if the live participant clients in the first multi-person live broadcast room and the second multi-person live broadcast room need to conduct a real-time cross-room call, the public stream data can also be switched back to the multiple second real-time multimedia streams to ensure the real-time nature of the cross-room call.

[0062] Based on any of the above embodiments, the obtaining of the public stream data of the second multi-person live broadcast room described in S202 may specifically include: in response to a mute instruction for the second multi-person live broadcast room, obtaining the first type of public stream data of the second multi-person live broadcast room from the server, wherein the first type of public stream data is formed by the server end merging the real-time video streams of multiple live broadcast participant clients in the second multi-person live broadcast room; or in response to an unmute instruction for the second multi-person live broadcast room, obtaining the second type of public stream data of the second multi-person live broadcast room from the server, wherein the second type of public stream data is formed by the server end merging the real-time audio and video streams of multiple live broadcast participant clients in the second multi-person live broadcast room.

[0063] In this embodiment, any live broadcast participant client (e.g., anchor client) of the first multi-person live broadcast room can control the mute of the second multi-person live broadcast room. That is, each live broadcast participant client of the first multi-person live broadcast room cannot hear the voice of each live broadcast participant client of the second multi-person live broadcast room, but can see the picture. By triggering the mute command of the second multi-person live broadcast room, the server can only merge the real-time video stream of the second multi-person live broadcast room when obtaining the public stream data of the second multi-person live broadcast room, and the first type of public stream data of the second multi-person live broadcast room is obtained by merging. When any live broadcast participant client of the first multi-person live broadcast room obtains and plays the first type of public stream data of the second multi-person live broadcast room, the second multi-person live broadcast room is muted.

[0064] Similarly, any live broadcast participant client (such as the host client) of the first multi-person live broadcast room can control the second multi-person live broadcast room to unmute. That is, each live broadcast participant client in the first multi-person live broadcast room can hear the voice of each live broadcast participant client in the second multi-person live broadcast room and see the picture. By triggering the unmute instruction for the second multi-person live broadcast room, the server can merge the real-time audio and video streams of the second multi-person live broadcast room when obtaining the public stream data of the second multi-person live broadcast room, and obtain the second type of public stream data of the second multi-person live broadcast room. When any live broadcast participant client of the first multi-person live broadcast room obtains and plays the second type of public stream data of the second multi-person live broadcast room, the second multi-person live broadcast room is unmuted.

[0065] The process of any live broadcast participant client (such as the host client) of the second multi-person live broadcast room controlling the mute and unmute of the second multi-person live broadcast room is similar to the above process and will not be repeated here.

[0066] Based on any of the above embodiments, when the server merges the real-time video streams of each live broadcast participant client in the second multi-person live broadcast room, it can obtain the audio and video status information and / or volume information of the real-time multimedia streams of each live broadcast participant client in the second multi-person live broadcast room. In some embodiments, the audio and video status information may include whether there is sound, whether the sound is background music, or whether it is a human voice, and the volume information may be human voice volume information and / or background music volume information. Furthermore, the server can carry the audio and video status information and / or volume information in the public stream data, so that when any live broadcast participant client in the first multi-person live broadcast room decodes the public stream data of the second multi-person live broadcast room, it can obtain the audio and video status information and / or volume information of the real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room. When playing the public stream data, the corresponding audio and video status information and / or volume information can be displayed in the display area corresponding to each live broadcast participant in the public stream data, so that it can be determined which live broadcast participant client in the second multi-person live broadcast room is speaking.

[0067] When any live broadcast participant client in the first multi-person live broadcast room obtains a first real-time multimedia stream from at least one other live broadcast participant client in the current first multi-person live broadcast room, each first real-time multimedia stream carries its own audio and video status information and / or volume information. When any live broadcast participant client in the first multi-person live broadcast room plays any first real-time multimedia stream, the corresponding audio and video status information and / or volume information can be displayed in the display area corresponding to the first real-time multimedia stream. This allows the user to determine which live broadcast participant client in the first multi-person live broadcast room is currently emitting sound.

[0068] Based on any of the above embodiments, during the cross-room live broadcast process, the audience client can simultaneously watch the live broadcast process of each live broadcast participant in each multi-person live broadcast room.

[0069] Optionally, any live broadcast participant client in the first multi-person live broadcast room can combine the first real-time multimedia streams of each live broadcast participant client in the first multi-person live broadcast room with the common stream data. The combined multimedia stream is uploaded to the server, which pushes it to the viewer client for playback, allowing the viewer client to watch the cross-room live broadcast process.

[0070] Optionally, considering that merging by the live broadcast participant clients increases the performance pressure on the live broadcast client, the server can also perform the merging. The server can merge the public stream data of the first multi-person live broadcast room with the public stream data of the second multi-person live broadcast room, and push the merged multimedia stream to the viewer client for playback, allowing the viewer client to watch the cross-room mic-connection process.

[0071] The live broadcast connection method, device and storage medium provided by the embodiments of the present disclosure obtain the first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room through any live broadcast participant client in the first multi-person live broadcast room, and play it; based on the cross-room connection instruction, obtain the public stream data of the second multi-person live broadcast room, wherein the public stream data is the second real-time multimedia stream of multiple live broadcast participant clients in the second multi-person live broadcast room; play the public stream data. When connecting to the live broadcast room across rooms, only one channel of public stream data needs to be obtained for the other multi-person live broadcast room, avoiding pulling the real-time multimedia stream of each live broadcast participant client in the other multi-person live broadcast room. It also reduces the number of running decoders, reduces the pressure on the performance of the live broadcast client, and reduces the bandwidth and traffic costs. In addition, the cross-room connection termination process is simple, crosstalk is not likely to occur, and the live broadcast connection experience is improved.

[0072] Refer to Figure 4, which is a flowchart of a live broadcast and microphone connection method provided by an embodiment of the present disclosure. The method of this embodiment can be applied in a server, and the live broadcast and microphone connection method includes:

[0073] S401: Push a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room and a second real-time multimedia stream of at least one live broadcast participant client in a second multi-person live broadcast room to the live broadcast participant clients in their respective multi-person live broadcast rooms;

[0074] S402. Based on the cross-room live broadcast instruction, the first public stream data corresponding to the first multi-person live broadcast room is pushed to the live broadcast participant client in the second multi-person live broadcast room, and the second public stream data corresponding to the second multi-person live broadcast room is pushed to the live broadcast participant client in the first multi-person live broadcast room.

[0075] Specifically, after receiving the cross-room live broadcast instruction, the real-time multimedia streams in the same multi-person live broadcast room are merged respectively to obtain the public stream data corresponding to the first multi-person live broadcast room and the public stream data corresponding to the second multi-person live broadcast room, and then the first public stream data and the second public stream data are pushed to the live broadcast participant clients of the other multi-person live broadcast room for playback.

[0076] The live broadcast and microphone connection method in this embodiment is the live broadcast and microphone connection method on the server side in the above S201-S203 corresponding embodiments. Its implementation principle and technical effects can be found in the above embodiments and will not be repeated here.

[0077] It should be noted that this embodiment can be executed on the same server or in a server cluster. For example, the transmission task of the real-time multimedia stream can be executed on one or more servers, while the task of merging the real-time multimedia stream to obtain public stream data can be executed on another one or more servers. The servers that perform the two tasks are controlled in one server cluster as much as possible (for example, in a computer room in the same area) to reduce the time consumed by the real-time multimedia stream to detour. An optional server-side architecture can be shown in Figure 5. The live broadcast participant client in the second multi-person live broadcast room pushes the second real-time multimedia stream to server A, and the live broadcast participant client in the second multi-person live broadcast room can be connected through server A. Similarly, the live broadcast participant client in the first multi-person live broadcast room can be connected through server C. When connecting across rooms, server A sends the second real-time multimedia stream of each live broadcast participant client in the second multi-person live broadcast room to server B, and server B performs the merging operation to obtain the public stream data of the second multi-person live broadcast room, and then pushes the public stream data of the second multi-person live broadcast room to server C. Server C pushes the public stream data of the second multi-person live broadcast room to the clients of each live broadcast participant in the first multi-person live broadcast room.

[0078] Based on the above embodiment, another embodiment of the present disclosure provides a live broadcast and microphone connection method, which includes:

[0079] S501: Push the first real-time multimedia stream of each live broadcast participant client in the first multi-person live broadcast room and the second real-time multimedia stream of each live broadcast participant client in the second multi-person live broadcast room to the live broadcast participant clients in their respective multi-person live broadcast rooms.

[0080] S502: In response to the cross-room live broadcast instruction, the first real-time multimedia stream of the multiple live broadcast participant clients in the first multi-person live broadcast room and the second real-time multimedia stream of the multiple live broadcast participant clients in the second multi-person live broadcast room are pushed to the live broadcast participant clients in the other multi-person live broadcast room respectively.

[0081] S503: Combine multiple real-time multimedia streams in the same multi-person live broadcast room to obtain public stream data corresponding to the first multi-person live broadcast room and public stream data corresponding to the second multi-person live broadcast room.

[0082] S504: Push the first public stream data and the second public stream data to the clients of each live broadcast participant in the other party's multi-person live broadcast room for playback.

[0083] The live broadcast and microphone connection method in this embodiment is the live broadcast and microphone connection method on the server side in the above S301-S304 corresponding embodiments. Its implementation principle and technical effects can be found in the above embodiments and will not be repeated here.

[0084] On the basis of the above embodiment, the merging of the first real-time multimedia streams to obtain the first public stream data and the merging of the second real-time multimedia streams to obtain the second public stream data include: responding to a mute instruction of any multi-person live broadcast room, merging the real-time video streams of multiple live broadcast participant clients in the multi-person live broadcast room to obtain the first type of public stream data of the multi-person live broadcast room; or responding to an unmute instruction of any multi-person live broadcast room, merging the real-time audio and video streams of multiple live broadcast participant clients in the multi-person live broadcast room to obtain the second type of public stream data of the multi-person live broadcast room.

[0085] Based on any of the above embodiments, the live broadcast and microphone connection method also includes: when merging the public stream data of any multi-person live broadcast room, obtaining the audio and video status information and / or volume information of the real-time multimedia stream of multiple live broadcast participant clients in the multi-person live broadcast room, and carrying it in the public stream data.

[0086] Based on any of the above embodiments, the live broadcast and microphone connection method also includes: merging the first public stream data and the second public stream data to obtain a merged multimedia stream, and pushing the merged multimedia stream to the audience client for playback; or receiving the merged multimedia stream uploaded by any live broadcast participant client, and pushing the merged multimedia stream to the audience client for playback, wherein the uploaded merged multimedia stream is a multimedia stream obtained by merging the real-time multimedia streams of each live broadcast participant client in the multi-person live broadcast room where the live broadcast participant client is located and the public stream data of the other party's multi-person live broadcast room.

[0087] Corresponding to the live broadcast participant client-side live broadcast connection method described in the previous embodiment, Figure 6 is a block diagram of the live broadcast participant client provided in the present embodiment. For ease of illustration, only the portions relevant to the present embodiment are shown. Referring to Figure 6 , the live broadcast participant client 600 includes: a local room stream pulling unit 601, a public stream pulling unit 602, and a control unit 603.

[0088] Among them, the room stream pulling unit 601 is used to obtain the first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room and play it; the public stream pulling unit 602 is used to obtain the public stream data of the second multi-person live broadcast room based on the cross-room live broadcast instruction, wherein the public stream data is the second real-time multimedia stream merged by multiple live broadcast participant clients in the second multi-person live broadcast room; the control unit 603 is used to control the playback of the public stream data.

[0089] In one or more embodiments of the present disclosure, when obtaining the first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room, the local room stream pulling unit 601 is used to: obtain the first real-time multimedia stream of at least one other live broadcast participant client in the first multi-person live broadcast room except any one of the live broadcast participant clients from the server.

[0090] In one or more embodiments of the present disclosure, the live broadcast participant client 600 also includes a cross-room streaming unit 604, which is used to: before obtaining the public stream data of the second multi-person live broadcast room, obtain each of the second real-time multimedia streams based on the cross-room live broadcast instruction and play them; the control unit 603 is also used to stop obtaining and playing the second real-time multimedia stream after playing the public stream data.

[0091] In one or more embodiments of the present disclosure, when obtaining the public stream data of the second multi-person live broadcast room, the public stream pulling unit 602 is used to: obtain identification information of the public stream data; after merging multiple second real-time multimedia streams on the server side to obtain the public stream data, obtain the public stream data from the server side according to the identification information.

[0092] In one or more embodiments of the present disclosure, when obtaining the public stream data of the second multi-person live broadcast room, the public stream pulling unit 602 is used to: in response to a mute instruction for the second multi-person live broadcast room, obtain the first type of public stream data of the second multi-person live broadcast room from the server, wherein the first type of public stream data is formed by the server end merging the real-time video streams of multiple live broadcast participant clients in the second multi-person live broadcast room; or in response to an unmute instruction for the second multi-person live broadcast room, obtain the second type of public stream data of the second multi-person live broadcast room from the server, wherein the second type of public stream data is formed by the server end merging the real-time audio and video streams of multiple live broadcast participant clients in the second multi-person live broadcast room.

[0093] In one or more embodiments of the present disclosure, when stopping acquisition and playback of each second real-time multimedia stream, the control unit 603 is configured to: stop acquisition and playback of the second real-time multimedia stream after the first frame of the common stream data is successfully decoded.

[0094] In one or more embodiments of the present disclosure, the common stream data carries audio and video status information and / or volume information of each second real-time multimedia stream; when the control unit 603 controls the playback of the common stream data, it also includes: displaying the corresponding audio and video status information and / or volume information in the display area corresponding to each live broadcast participant in the common stream data.

[0095] In one or more embodiments of the present disclosure, the cross-room stream pulling unit 604 is used to: pull each second real-time multimedia stream from the first multi-person live broadcast room on the server side when obtaining each second real-time multimedia stream respectively, wherein each second real-time multimedia stream is a plurality of live broadcast participant clients in the second multi-person live broadcast room in response to the cross-room live broadcast instruction and push their respective second real-time multimedia streams to the first multi-person live broadcast room on the server side.

[0096] In one or more embodiments of the present disclosure, the live broadcast participant client also includes a streaming unit for: in response to the cross-room microphone connection instruction, pushing the first real-time multimedia stream collected in real time by any live broadcast participant client to the second multi-person microphone live broadcast room on the server side.

[0097] In one or more embodiments of the present disclosure, the control unit 603 is further configured to: if the playback of the common stream data fails, control the cross-room stream pulling unit 604 to obtain and play each of the second real-time multimedia streams.

[0098] In one or more embodiments of the present disclosure, the streaming unit is also used to: merge the first real-time multimedia stream of each live broadcast participant client in the first multi-person live broadcast room and the public stream data, and upload the merged multimedia stream to the server for pushing by the server to the audience client for playback.

[0099] The live broadcast participant client provided in this embodiment can be used to execute the technical solution of the method embodiment on the live broadcast participant client side. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0100] Corresponding to the live broadcast and microphone connection method on the server side of the above embodiment, Figure 7 is a structural block diagram of the server provided by the embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Referring to Figure 7, the server 700 includes: a same-room streaming unit 701, a merging unit 702, and a public stream streaming unit 703.

[0101] Among them, the same-room streaming unit 701 is used to push the first real-time multimedia stream of each live broadcast participant client in the first multi-person live broadcast room and the second real-time multimedia stream of each live broadcast participant client in the second multi-person live broadcast room to the live broadcast participant clients in their respective multi-person live broadcast rooms.

[0102] The public stream pushing unit 703 is used to push the first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant client in the second multi-person live broadcast room, and push the second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant client in the first multi-person live broadcast room.

[0103] In addition, the server 700 also includes a merging unit 702, which is used to merge multiple real-time multimedia streams in the same multi-person live broadcast room to obtain public stream data corresponding to the first multi-person live broadcast room and public stream data corresponding to the second multi-person live broadcast room.

[0104] In one or more embodiments of the present disclosure, the server 700 also includes a cross-room streaming unit 704, which is used to: before pushing the first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant client in the second multi-person live broadcast room, and pushing the second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant client in the first multi-person live broadcast room, based on the cross-room live broadcast instruction, push the first real-time multimedia stream of multiple live broadcast participant clients in the first multi-person live broadcast room and the second real-time multimedia stream of multiple live broadcast participant clients in the second multi-person live broadcast room to the live broadcast participant client in the other multi-person live broadcast room respectively.

[0105] In one or more embodiments of the present disclosure, the merging unit 702 is further used to: in response to a mute instruction for any multi-person live broadcast room, merge the real-time video streams of multiple live broadcast participant clients in the multi-person live broadcast room to obtain the first type of public stream data of the multi-person live broadcast room; or in response to an unmute instruction for any multi-person live broadcast room, merge the real-time audio and video streams of multiple live broadcast participant clients in the multi-person live broadcast room to obtain the second type of public stream data of the multi-person live broadcast room.

[0106] In one or more embodiments of the present disclosure, the merging unit 702 is also used to: when obtaining the public stream data of any multi-person live broadcast room, obtain the audio and video status information and / or volume information of the real-time multimedia stream of multiple live broadcast participant clients in the multi-person live broadcast room, and carry it in the public stream data.

[0107] In one or more embodiments of the present disclosure, the server further includes a viewer streaming unit configured to: merge the first public stream data and the second public stream data to obtain a merged multimedia stream, and push the merged multimedia stream to a viewer client for playback; or

[0108] Receive the merged multimedia stream uploaded by any live broadcast participant client, and push the merged multimedia stream to the audience client for playback, where the uploaded merged multimedia stream is the multimedia stream merged by the real-time multimedia streams of each live broadcast participant client in the multi-person live broadcast room where the live broadcast participant client is located and the public stream data of the other party's multi-person live broadcast room.

[0109] The server provided in this embodiment can be used to execute the technical solution of the above-mentioned server-side method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0110] Referring to FIG8 , a schematic diagram of the structure of an electronic device 800 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 800 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG8 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0111] As shown in Figure 8, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage device 808 to the random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0112] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although FIG8 shows an electronic device 800 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0113] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0114] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0115] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0116] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0117] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0120] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0121] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] In a first aspect, according to one or more embodiments of the present disclosure, a live broadcast connection method is provided, which is applied to any live broadcast participant client in a first multi-person live broadcast room. The method includes: obtaining a first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room and playing it; based on a cross-room live broadcast instruction, obtaining public stream data of a second multi-person live broadcast room, wherein the public stream data is formed by merging the second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room; and playing the public stream data.

[0123] According to one or more embodiments of the present disclosure, obtaining the first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room includes: obtaining the first real-time multimedia stream of at least one other live broadcast participant client in the first multi-person live broadcast room except any one of the live broadcast participant clients from the server.

[0124] According to one or more embodiments of the present disclosure, the method further includes: before obtaining the public stream data of the second multi-person live broadcast room, based on the cross-room live broadcast instruction, respectively obtaining the second real-time multimedia stream and playing it; after playing the public stream data, stopping the acquisition and playback of the second real-time multimedia stream.

[0125] According to one or more embodiments of the present disclosure, obtaining the public stream data of the second multi-person live broadcast room includes: obtaining identification information of the public stream data; after merging multiple second real-time multimedia streams on the server side to obtain the public stream data, obtaining the public stream data from the server side according to the identification information.

[0126] According to one or more embodiments of the present disclosure, obtaining the public stream data of the second multi-person live broadcast room includes: in response to a mute instruction for the second multi-person live broadcast room, obtaining the first type of public stream data of the second multi-person live broadcast room from the server, wherein the first type of public stream data is formed by the server end combining the real-time video streams of multiple live broadcast participant clients in the second multi-person live broadcast room; or in response to an unmute instruction for the second multi-person live broadcast room, obtaining the second type of public stream data of the second multi-person live broadcast room from the server, wherein the second type of public stream data is formed by the server end combining the real-time audio and video streams of multiple live broadcast participant clients in the second multi-person live broadcast room.

[0127] According to one or more embodiments of the present disclosure, stopping acquiring and playing each of the second real-time multimedia streams includes: stopping acquiring and playing each of the second real-time multimedia streams after the first frame of the common stream data is successfully decoded.

[0128] According to one or more embodiments of the present disclosure, the public stream data carries audio and video status information and / or volume information of multiple second real-time multimedia streams; when playing the public stream data, it also includes: displaying the corresponding audio and video status information and / or volume information in the display area corresponding to each live broadcast participant in the public stream data.

[0129] According to one or more embodiments of the present disclosure, respectively obtaining each second real-time multimedia stream includes: pulling each second real-time multimedia stream from the first multi-person live broadcast room on the server side, wherein each second real-time multimedia stream is pushed by each live broadcast participant client in the second multi-person live broadcast room to the first multi-person live broadcast room on the server side in response to the cross-room live broadcast instruction.

[0130] According to one or more embodiments of the present disclosure, the method further includes: in response to the cross-room live broadcast instruction, pushing the first real-time multimedia stream collected in real time by any live broadcast participant client to the second multi-person live broadcast room on the server.

[0131] According to one or more embodiments of the present disclosure, the method further includes: if the common stream data fails to be played, acquiring the second real-time multimedia stream and playing it.

[0132] According to one or more embodiments of the present disclosure, the method also includes: merging the first real-time multimedia stream of each live broadcast participant client in the first multi-person live broadcast room and the public stream data, and uploading the merged multimedia stream to the server for pushing by the server to the audience client for playback.

[0133] In a second aspect, according to one or more embodiments of the present disclosure, a live broadcast connection device is provided, which is applied to a server. The method includes: pushing a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room and a second real-time multimedia stream of at least one live broadcast participant client in a second multi-person live broadcast room to the live broadcast participant clients in the respective multi-person live broadcast rooms; based on a cross-room live broadcast instruction, pushing the first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant client in the second multi-person live broadcast room, and pushing the second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant client in the first multi-person live broadcast room.

[0134] According to one or more embodiments of the present disclosure, before pushing the first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant client in the second multi-person live broadcast room, and pushing the second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant client in the first multi-person live broadcast room, it also includes: based on the cross-room live broadcast instruction, pushing the first real-time multimedia stream of multiple live broadcast participant clients in the first multi-person live broadcast room and the second real-time multimedia stream of multiple live broadcast participant clients in the second multi-person live broadcast room to the live broadcast participant client in the other multi-person live broadcast room respectively.

[0135] According to one or more embodiments of the present disclosure, the method also includes: in response to a mute instruction for any multi-person live broadcast room, merging the real-time video streams of multiple live broadcast participant clients in the multi-person live broadcast room to obtain the first type of public stream data of the multi-person live broadcast room; or in response to an unmute instruction for any multi-person live broadcast room, merging the real-time audio and video streams of multiple live broadcast participant clients in the multi-person live broadcast room to obtain the second type of public stream data of the multi-person live broadcast room.

[0136] According to one or more embodiments of the present disclosure, the method further includes: when merging the public stream data of any multi-person live broadcast room, obtaining the audio and video status information and / or volume information of the real-time multimedia stream of multiple live broadcast participant clients in the multi-person live broadcast room, and carrying it in the public stream data.

[0137] According to one or more embodiments of the present disclosure, the method also includes: merging the first public stream data and the second public stream data to obtain a merged multimedia stream, and pushing the merged multimedia stream to the audience client for playback; or receiving the merged multimedia stream uploaded by any live broadcast participant client, and pushing the merged multimedia stream to the audience client for playback, wherein the uploaded merged multimedia stream is a multimedia stream obtained by merging the real-time multimedia streams of each live broadcast participant client in the multi-person live broadcast room where the live broadcast participant client is located and the public stream data of the other party's multi-person live broadcast room.

[0138] In a third aspect, according to one or more embodiments of the present disclosure, a live broadcast participant client is provided, including: a local room stream pulling unit, used to obtain a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room and play it; a public stream pulling unit, used to obtain public stream data of a second multi-person live broadcast room based on a cross-room live broadcast instruction, wherein the public stream data is formed by merging the second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room; and a control unit, used to control the playback of the public stream data.

[0139] According to one or more embodiments of the present disclosure, when the stream pulling unit in this room obtains the first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room, it is used to: obtain the first real-time multimedia stream of at least one other live broadcast participant client in the first multi-person live broadcast room except any one of the live broadcast participant clients from the server.

[0140] According to one or more embodiments of the present disclosure, the live broadcast participant client also includes a cross-room streaming unit, which is used to: before obtaining the public stream data of the second multi-person live broadcast room, obtain each of the second real-time multimedia streams based on the cross-room live broadcast instruction and play them; the control unit is also used to stop obtaining and playing each of the second real-time multimedia streams after playing the public stream data.

[0141] According to one or more embodiments of the present disclosure, when the public stream pulling unit obtains the public stream data of the second multi-person live broadcast room, it is used to: obtain the identification information of the public stream data; after merging multiple second real-time multimedia streams on the server side to obtain the public stream data, obtain the public stream data from the server side according to the identification information.

[0142] According to one or more embodiments of the present disclosure, when obtaining the public stream data of the second multi-person live broadcast room, the public stream pulling unit is used to: in response to a mute instruction for the second multi-person live broadcast room, obtain the first type of public stream data of the second multi-person live broadcast room from the server end, wherein the first type of public stream data is formed by the server end merging the real-time video streams of multiple live broadcast participant clients in the second multi-person live broadcast room; or in response to an unmute instruction for the second multi-person live broadcast room, obtain the second type of public stream data of the second multi-person live broadcast room from the server end, wherein the second type of public stream data is formed by the server end merging the real-time audio and video streams of multiple live broadcast participant clients in the second multi-person live broadcast room.

[0143] According to one or more embodiments of the present disclosure, when stopping acquisition and playback of each second real-time multimedia stream, the control unit is configured to: stop acquisition and playback of the second real-time multimedia stream after the first frame of the common stream data is successfully decoded.

[0144] According to one or more embodiments of the present disclosure, the common stream data carries audio and video status information and / or volume information of multiple second real-time multimedia streams; when the control unit controls the playback of the common stream data, it also includes: displaying the corresponding audio and video status information and / or volume information in the display area corresponding to each live broadcast participant in the common stream data.

[0145] According to one or more embodiments of the present disclosure, when the cross-room stream pulling unit obtains each of the second real-time multimedia streams respectively, it is used to: pull each of the second real-time multimedia streams from the first multi-person live broadcast room on the server side, wherein each of the second real-time multimedia streams is pushed by each live broadcast participant client in the second multi-person live broadcast room to the first multi-person live broadcast room on the server side in response to the cross-room live broadcast instruction.

[0146] According to one or more embodiments of the present disclosure, the live broadcast participant client also includes a streaming unit for: in response to the cross-room microphone connection instruction, pushing the first real-time multimedia stream collected in real time by any live broadcast participant client to the second multi-person microphone live broadcast room on the server side.

[0147] According to one or more embodiments of the present disclosure, the control unit is further configured to: if the playback of the common stream data fails, re-control the cross-room stream pulling unit to obtain and play each of the second real-time multimedia streams.

[0148] According to one or more embodiments of the present disclosure, the streaming unit is also used to: merge the first real-time multimedia stream of each live broadcast participant client in the first multi-person live broadcast room and the public stream data, and upload the merged multimedia stream to the server for being pushed by the server to the audience client for playback.

[0149] In a fourth aspect, according to one or more embodiments of the present disclosure, a server is provided, including: a same-room streaming pushing unit, for pushing a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room and a second real-time multimedia stream of at least one live broadcast participant client in a second multi-person live broadcast room to the live broadcast participant clients in the respective multi-person live broadcast rooms; a public stream streaming pushing unit, for pushing the first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant clients in the second multi-person live broadcast room based on a cross-room live broadcast instruction, and pushing the second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant clients in the first multi-person live broadcast room.

[0150] According to one or more embodiments of the present disclosure, the server also includes a cross-room streaming unit, which is used to: before pushing the first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant client in the second multi-person live broadcast room, and before pushing the second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant client in the first multi-person live broadcast room, based on the cross-room live broadcast instruction, push the first real-time multimedia stream of multiple live broadcast participant clients in the first multi-person live broadcast room and the second real-time multimedia stream of multiple live broadcast participant clients in the second multi-person live broadcast room to the live broadcast participant client in the other multi-person live broadcast room.

[0151] According to one or more embodiments of the present disclosure, the merging unit is also used to: in response to a mute instruction for any multi-person live broadcast room, merge the real-time video streams of the German live broadcast participant clients in the multi-person live broadcast room to obtain the first type of public stream data of the multi-person live broadcast room; or in response to an unmute instruction for any multi-person live broadcast room, merge the real-time audio and video streams of multiple live broadcast participant clients in the multi-person live broadcast room to obtain the second type of public stream data of the multi-person live broadcast room.

[0152] According to one or more embodiments of the present disclosure, the merging unit is also used to: when obtaining the public stream data of any multi-person live broadcast room, obtain the audio and video status information and / or volume information of the real-time multimedia stream of multiple live broadcast participant clients in the multi-person live broadcast room, and carry it in the public stream data.

[0153] According to one or more embodiments of the present disclosure, the server also includes an audience streaming unit, which is used to: merge the first public stream data and the second public stream data to obtain a merged multimedia stream, and push the merged multimedia stream to the audience client for playback; or receive the merged multimedia stream uploaded by any live broadcast participant client, and push the merged multimedia stream to the audience client for playback, wherein the uploaded merged multimedia stream is a multimedia stream obtained by merging the real-time multimedia streams of each live broadcast participant client in the multi-person live broadcast room where the live broadcast participant client is located and the public stream data of the other party's multi-person live broadcast room.

[0154] In a fifth aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live broadcast and microphone connection method described in the first aspect and various possible designs of the first aspect, or the live broadcast and microphone connection method described in the second aspect and various possible designs of the second aspect.

[0155] In the sixth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the processor executes the computer-executable instructions, the live broadcast and microphone connection method described in the first aspect and various possible designs of the first aspect, or the live broadcast and microphone connection method described in the second aspect and various possible designs of the second aspect are implemented.

[0156] In the seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including computer execution instructions. When a processor executes the computer execution instructions, the live broadcast and microphone connection method described in the first aspect and various possible designs of the first aspect, or the live broadcast and microphone connection method described in the second aspect and various possible designs of the second aspect is implemented.

[0157] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0158] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0159] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A live broadcast and microphone connection method, applied to any live broadcast participant client in a first multi-person microphone connection live broadcast room, the method comprising: Acquire a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room, and play the stream; Based on the cross-room live broadcast instruction, obtain public stream data of the second multi-person live broadcast room, wherein the public stream data is formed by merging second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room; The common stream data is played.

2. The method according to claim 1, wherein: The step of obtaining a first real-time multimedia stream of at least one live broadcast participant client in a first multi-person live broadcast room includes: A first real-time multimedia stream of at least one other live broadcast participant client except any one of the live broadcast participant clients in the first multi-person live broadcast room is obtained from the server.

3. The method according to claim 1, wherein: The method further comprises: Before obtaining the public stream data of the second multi-person live broadcast room, based on the cross-room live broadcast instruction, respectively obtain the second real-time multimedia stream and play it; After playing the common stream data, stop acquiring and playing the second real-time multimedia stream.

4. The method according to any one of claims 1 to 3, wherein: The obtaining of the public stream data of the second multi-person live broadcast room includes: Acquire identification information of the public stream data; After the server merges the plurality of second real-time multimedia streams to obtain the common stream data, the server acquires the common stream data from the server according to the identification information.

5. The method according to claim 1, wherein: The obtaining of the public stream data of the second multi-person live broadcast room includes: In response to a mute instruction for a second multi-person live broadcast room, obtaining first-category public stream data for the second multi-person live broadcast room from a server, wherein the first-category public stream data is formed by combining real-time video streams of multiple live broadcast participant clients in the second multi-person live broadcast room with the server; or In response to the unmute instruction for the second multi-person live broadcast room, the first The second type of public stream data of a multi-person live broadcast room connected by microphones is formed by the server converging the real-time audio and video streams of multiple live broadcast participant clients in the second multi-person live broadcast room connected by microphones.

6. The method according to claim 3, wherein: The stopping of acquiring and playing the second real-time multimedia stream includes: After the first frame of the common stream data is decoded successfully, the acquisition and playback of the second real-time multimedia stream is stopped.

7. The method according to claim 1, wherein: The common stream data carries the audio and video status information and / or volume information of each of the second real-time multimedia streams; When playing the common stream data, the method further includes: The corresponding audio and video status information and / or volume information is displayed in the display area corresponding to each live broadcast participant in the public stream data.

8. The method according to claim 3, wherein: The obtaining the second real-time multimedia streams respectively includes: The second real-time multimedia stream is pulled from the first multi-person live broadcast room on the server side, wherein the second real-time multimedia stream is pushed by multiple live broadcast participant clients in the second multi-person live broadcast room to the first multi-person live broadcast room on the server side in response to the cross-room live broadcast instruction.

9. The method according to claim 8, wherein: Also includes: In response to the cross-room microphone connection instruction, the first real-time multimedia stream collected in real time by the client of any live broadcast participant is pushed to the second multi-person microphone connection live broadcast room of the server.

10. The method according to any one of claims 1 to 3, wherein: Also includes: If the common stream data fails to be played, the second real-time multimedia stream is obtained and played.

11. The method according to claim 1, wherein: Also includes: The first real-time multimedia stream of each live broadcast participant client in the first multi-person live broadcast room and the public stream data are merged, and the merged multimedia stream is uploaded to the server end for being pushed by the server end to the audience client for playback.

12. A live broadcast and microphone connection method, applied to a server, the method comprising: The first real-time multimedia client of at least one live broadcast participant in the first multi-person live broadcast room is The second real-time multimedia stream and the second real-time multimedia stream of at least one live broadcast participant client in the second multi-person live broadcast room are pushed to the live broadcast participant clients in the respective multi-person live broadcast rooms; Based on the cross-room live broadcast instruction, the first public stream data corresponding to the first multi-person live broadcast room is pushed to the live broadcast participant client in the second multi-person live broadcast room, and the second public stream data corresponding to the second multi-person live broadcast room is pushed to the live broadcast participant client in the first multi-person live broadcast room.

13. The method according to claim 12, wherein: Before pushing the first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant client in the second multi-person live broadcast room, and pushing the second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant client in the first multi-person live broadcast room, the method further includes: Based on the cross-room live broadcast instruction, the first real-time multimedia stream of multiple live broadcast participant clients in the first multi-person live broadcast room and the second real-time multimedia stream of multiple live broadcast participant clients in the second multi-person live broadcast room are pushed to the live broadcast participant clients in the other multi-person live broadcast room respectively.

14. The method according to claim 12, wherein: The method further comprises: In response to a mute instruction for any multi-person live broadcast room, real-time video streams of multiple live broadcast participant clients in the multi-person live broadcast room are merged to obtain first-category public stream data of the multi-person live broadcast room; or In response to an unmute instruction for any multi-person live broadcast room, real-time audio and video streams of multiple live broadcast participant clients in the multi-person live broadcast room are merged to obtain second-category public stream data of the multi-person live broadcast room.

15. The method according to any one of claims 12 to 14, wherein: Also includes: Merging the first public stream data and the second public stream data to obtain a merged multimedia stream, and pushing the merged multimedia stream to a viewer client for playback; or Receive the merged multimedia stream uploaded by any live broadcast participant client, and push the merged multimedia stream to the audience client for playback, wherein the uploaded merged multimedia stream is a multimedia stream merged by the real-time multimedia streams of each live broadcast participant client in the multi-person live broadcast room where the live broadcast participant client is located and the public stream data of the other party's multi-person live broadcast room.

16. A live broadcast participant client, comprising: The stream pulling unit in this room is used to obtain a first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room and play it; A public stream pulling unit, configured to obtain public stream data of a second multi-person live broadcast room based on a cross-room live broadcast instruction, wherein the public stream data is a confluence of second real-time multimedia streams of multiple live broadcast participant clients in the second multi-person live broadcast room; A control unit is used to control the playing of the common stream data.

17. A server comprising: The same-room streaming unit is used to push a first real-time multimedia stream of at least one live broadcast participant client in the first multi-person live broadcast room and a second real-time multimedia stream of at least one live broadcast participant client in the second multi-person live broadcast room to the live broadcast participant clients in the respective multi-person live broadcast rooms; The public stream pushing unit is used to push the first public stream data corresponding to the first multi-person live broadcast room to the live broadcast participant clients in the second multi-person live broadcast room based on the cross-room live broadcast instruction, and push the second public stream data corresponding to the second multi-person live broadcast room to the live broadcast participant clients in the first multi-person live broadcast room.

18. An electronic device comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 15.

19. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 15 is implemented.

20. A computer program product, comprising computer executable instructions, and when a processor executes the computer executable instructions, the method according to any one of claims 1 to 15 is implemented.

Images