Multi-person microphone connection method and device

Abstract

The present application provides a method and device for connecting multiple people to a microphone, wherein the method includes: obtaining audio streams corresponding to at least two connected microphone users; creating a corresponding sub-track for each audio stream, transmitting a low-bitrate audio stream corresponding to the audio stream through the sub-track, and storing the correspondence between the sub-track and the connected microphone users; generating a mixed stream based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-track; pushing the mixed stream to the user end, wherein the low-bitrate audio stream of the sub-track in the mixed stream is used to instruct the user end to determine and identify the audio attributes of each connected microphone user at the current time based on the correspondence. In this way, by creating a corresponding sub-track for each connected microphone user and transmitting the low-bitrate audio stream corresponding to the audio stream of each connected microphone user, it is possible to transmit the corresponding audio attributes to the user end while transmitting the audio stream of each connected microphone user to the user end, thereby avoiding the delay of the audio attributes and ensuring the user experience of the connected microphone users.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more particularly to a method for connecting multiple people to a microphone. The present application also relates to a device for connecting multiple people to a microphone, a computing device, and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer and Internet technologies, online business has developed rapidly. More and more users can interact online to meet their daily life, study and work needs. For example, in order to meet people's interactive needs when watching live broadcasts, anchors can interact with other anchors or viewers via microphones to improve the live broadcast effect. For example, multiple users can participate in online meetings via microphones, making it easier to hold meetings anytime and anywhere and reducing meeting costs.

[0003] In the prior art, in a scenario where multiple people are connected to the microphone, the user end plays a mixed stream of multiple audio and video. At this time, the audio properties of the audio can be saved on the server end. When the user end receives the mixed stream of multiple connected users, it establishes a signaling channel with the server end to obtain the corresponding audio properties, thereby rendering different effects.

[0004] However, in the above method, the audio attributes are obtained by establishing a signaling channel with the server, polling request or server push, and the distribution of the audio stream is usually through the content distribution network CDN. The audio attributes obtained by the user end cannot be completely consistent with the audio stream in time, and there is a time delay in the audio attributes, which makes it impossible to fully synchronize the audio attributes and the audio stream, affecting the user experience of the connected microphone user. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a method for connecting multiple people to a microphone. The present application also relates to a device for connecting multiple people to a microphone, a computing device, and a computer-readable storage medium to solve the technical problem in the prior art that there is a delay in audio attributes, resulting in the inability to fully synchronize audio attributes and audio streams.

[0006] According to a first aspect of an embodiment of the present application, a multi-person microphone connection method is provided, which is applied to a content distribution network, and includes:

[0007] Get the audio streams corresponding to at least two connected users;

[0008] Create a corresponding sub-track for each audio stream, transmit the low-bitrate audio stream corresponding to the audio stream through the sub-track, and store the correspondence between the sub-track and the connected user;

[0009] Generate a mixed stream based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-track;

[0010] The mixed stream is pushed to the user end, wherein the low-bitrate audio stream of the sub-track in the mixed stream is used to instruct the user end to determine and identify the audio attributes of each connected microphone user at the current time based on the corresponding relationship.

[0011] According to a second aspect of an embodiment of the present application, a multi-person microphone connection method is provided, which is applied to a user terminal and includes:

[0012] Receive the mixed stream pushed by the content distribution network, and obtain the low-bitrate audio stream of each sub-track from the mixed stream;

[0013] Based on the low bit rate audio stream of each sub-track, determine the audio attribute of each sub-track at the current time;

[0014] Obtain the correspondence between the sub-tracks and the connected users, determine the audio properties of each connected user at the current time based on the audio properties of each sub-track at the current time, and identify the audio properties corresponding to each connected user in the connected interface at the current time.

[0015] According to a third aspect of an embodiment of the present application, a content distribution network is provided, including:

[0016] A first acquisition module is configured to acquire audio streams corresponding to at least two connected microphone users;

[0017] A creation module is configured to create a corresponding sub-track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-track, and store a corresponding relationship between the sub-track and the connected microphone user;

[0018] A generation module is configured to generate a mixed stream based on the audio streams corresponding to the connected microphone users and the low bit rate audio streams of the sub-tracks;

[0019] The push module is configured to push the mixed stream to the user end, wherein the low-bitrate audio stream of the sub-track in the mixed stream is used to instruct the user end to determine and identify the audio attributes of each connected microphone user at the current time based on the corresponding relationship.

[0020] According to a fourth aspect of an embodiment of the present application, a user terminal is provided, including:

[0021] A second acquisition module is configured to receive a mixed stream pushed by a content distribution network, and obtain a low-bitrate audio stream of each sub-track from the mixed stream;

[0022] A determination module, configured to determine the audio properties of each sub-track at a current time based on the low bit rate audio stream of each sub-track;

[0023] The identification module is configured to obtain the correspondence between the sub-audio tracks and the connected users, determine the audio properties of each connected user at the current time based on the audio properties of each sub-audio track at the current time, and identify the audio properties corresponding to each connected user in the connected interface at the current time.

[0024] According to a fifth aspect of an embodiment of the present application, a multi-person microphone connection system is provided, the multi-person microphone connection system comprising a content distribution network and a user terminal;

[0025] The content distribution network is configured to obtain audio streams corresponding to at least two connected microphone users; create a corresponding sub-audio track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-audio track, and store a corresponding relationship between the sub-audio track and the connected microphone users; generate a mixed stream based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-audio track; and push the mixed stream to the user end;

[0026] The user end is configured to receive the mixed stream pushed by the content distribution network, obtain the low-bitrate audio stream of each sub-track from the mixed stream; determine the audio properties of each sub-track at the current time based on the low-bitrate audio stream of each sub-track; obtain the correspondence between the sub-track and the connected microphone user, determine the audio properties of each connected microphone user at the current time based on the audio properties of each sub-track at the current time, and identify the audio properties corresponding to each connected microphone user in the connected microphone interface at the current time.

[0027] According to a sixth aspect of an embodiment of the present application, a computing device is provided, including:

[0028] Memory and processor;

[0029] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement any of the operating steps of the above-mentioned multi-person microphone connection method.

[0030] According to a seventh aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, any step of the above-mentioned multi-person microphone connection method is implemented.

[0031] In the multi-microphone connection method provided in the embodiment of the present application, a content distribution network can obtain audio streams corresponding to at least two connected microphone users; create a corresponding sub-audio track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-audio track, and store the correspondence between the sub-audio track and the connected microphone users; generate a mixed stream based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-audio track; push the mixed stream to the user end, wherein the low-bitrate audio stream of the sub-audio track in the mixed stream is used to instruct the user end to determine and identify the audio attributes of each connected microphone user at the current time based on the correspondence.

[0032] In this case, the content distribution network can generate a corresponding low-bitrate audio stream for each audio stream, and create a corresponding sub-audio track, transmit the low-bitrate audio stream through the sub-audio track, and then generate a mixed stream based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-audio track. In addition to the audio streams corresponding to each connected microphone user that need to be played, that is, the actual audio content, the mixed stream also includes the low-bitrate audio streams of each sub-audio track. The low-bitrate audio stream is used to determine the audio properties, so that after the user end receives the mixed stream, it can directly parse the mixed stream to obtain the audio stream to be played and the low-bitrate audio streams of each sub-audio track at the same time, so as to calculate and determine the audio properties of each connected microphone user at the current time based on the low-bitrate audio streams of each sub-audio track, and mark them. In this way, by creating a corresponding sub-audio track for each connected microphone user and transmitting the low-bitrate audio stream corresponding to the audio stream of each connected microphone user, it is possible to transmit the audio stream of each connected microphone user to the user end while transmitting the corresponding audio attributes to the user end, thereby avoiding the delay of audio attributes. The audio attributes obtained by the user end are completely synchronized with the audio stream, ensuring that the currently speaking connected microphone user identified in the current connected microphone interface is synchronized with the currently actually speaking connected microphone user, thereby ensuring the user experience of the connected microphone users. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of a two-person microphone connection processing process provided by an embodiment of the present application;

[0034] Figure 2 This is a flow chart of a multi-person microphone connection method provided by an embodiment of the present application;

[0035] Figure 3a is a flow chart of another multi-person microphone connection method provided by an embodiment of the present application;

[0036] Figure 3b is a schematic diagram of a microphone connection interface provided by an embodiment of the present application;

[0037] Figure 4 It is a schematic diagram of a processing process of a multi-person microphone connection method applied to a two-person microphone connection scenario in a live broadcast provided by an embodiment of the present application;

[0038] Figure 5 is a structural diagram of a content distribution network provided by an embodiment of the present application;

[0039] Figure 6 This is a schematic diagram of the structure of a user terminal provided by an embodiment of the present application;

[0040] Figure 7 This is a structural diagram of a multi-person microphone connection system provided by an embodiment of the present application;

[0041] Figure 8 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0042] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0043] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0044] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0045] First, the terms involved in one or more embodiments of the present application are explained.

[0046] Anchor: The user who initiates a live broadcast. The audience will see the live images and sounds in the anchor's live broadcast room.

[0047] Microphone users: users who interact with the initiator through a microphone. In a conference scenario, all participants can see the microphone screen and hear the participants' speeches. In a live broadcast scenario, microphone users refer to users who interact with the anchor through a microphone. When the microphone is connected, other viewers in the live broadcast room can also see / hear the screen and sound of the live broadcast room. Microphone users may be other anchors or ordinary viewers of the live broadcast room. They may be one person or multiple people.

[0048] Stream: An audio or video stream usually contains both video images and audio sounds, but may also contain only video images or only audio sounds.

[0049] Live stream: The stream distributed to the live broadcast room so that the audience can actually see it. In the non-connected microphone state, it is the same as the host's stream, that is, the image / sound of the host's live broadcast. In the connected microphone state, it is a mixed stream that contains the image / sound of both the host and the connected microphone user.

[0050] Mixed stream: Mix multiple audio and video streams of the host and the connected users into one audio and video stream, which is then distributed to the live broadcast room as a live stream for viewers to watch.

[0051] DBFS: Decibels relative to full scale, a volume representation method commonly used in digital audio systems. The value range is usually -96 to 0dBFS, with 0 representing the maximum volume.

[0052] Audio track: These are the parallel "tracks" you see in the sequencer software. Each track defines the properties of the track, such as the timbre, timbre library, number of channels, input / output ports, volume, etc.

[0053] It should be noted that when multiple people are connected to the microphone, the user end can only know that someone is talking when listening to the mixed stream of multiple people talking, but cannot know who is talking, let alone the volume of the talking. Therefore, when mixing the streams, the audio properties corresponding to the audio (volume, speaker identification, etc.) can be saved on the server end; when presenting on the end user end, the user end obtains the audio properties from the server end, and renders different effects when playing the audio stream, such as speaker icons, sound columns, avatar special effects, etc.

[0054] For example, take the live broadcast scene as an example. Figure 1 is a schematic diagram of a two-person microphone connection processing process provided by an embodiment of the present application, such as Figure 1 As shown, the host pushes the host stream to the mixing module of the content distribution network, and the connected microphone user pushes the connected microphone user stream to the mixing module of the content distribution network. The mixing module mixes the host stream and the connected microphone user stream to generate a live stream, and at the same time determines the audio attributes corresponding to the host stream and the connected microphone user stream, and stores the audio attributes to the server. The content distribution network pushes the live stream to the live broadcast room for users to watch or listen. The live broadcast room queries the corresponding audio attributes from the server, and renders the corresponding speaker and volume information in the connected microphone interface based on the audio attributes.

[0055] As can be seen from the above, the method of obtaining audio attributes is to establish a signaling channel with the server, obtain it through polling requests or server push; and the distribution of audio streams is usually through the content delivery network (Content Delivery Network, CDN). Generally speaking, the typical delay of audio streams distributed through the content delivery network is about 3 to 10 seconds, and the typical delay of obtaining signals through the signaling channel is usually hundreds of milliseconds to one or two seconds; therefore, the audio attributes obtained by the user end cannot be completely consistent with the audio stream in time, so there is a problem of incomplete synchronization, such as a person is already speaking, but the icon displayed is just the previous person's speaking icon.

[0056] Therefore, a multi-person microphone connection method is provided in an embodiment of the present application, and multiple audio tracks are generated during the mixing process. In addition to the main audio track (first audio track) which is the actual audio content after the mixing process, and maintains high quality and high bit rate, the others are sub-audio tracks assigned to each user participating in the microphone connection, and the corresponding low-quality and low-bit rate audio streams are transmitted, which are only used for the player side to determine the audio properties. The audio streams corresponding to each microphone-connected user and the low-bit rate audio streams of each sub-audio track are packaged to generate a mixed stream, which is distributed to the user end, so that when the user end receives the mixed stream, the audio stream to be played and the low-bit rate audio streams of each sub-audio track can be obtained by directly parsing the mixed stream. Based on the low-bit rate audio stream, the audio properties of each microphone-connected user at the current time can be determined, and the audio properties are obtained in full synchronization with the audio stream, thereby ensuring that the sound heard by the audience is completely synchronized with the sound icon / sound column, etc., which improves the user experience.

[0057] In the present application, a multi-person microphone connection method is provided. The present application also relates to a multi-person microphone connection device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0058] Figure 2 A flowchart of a multi-person live broadcast method according to an embodiment of the present application is shown, which is applied to a content distribution network and specifically includes the following steps 202-208:

[0059] Step 202: Obtain audio streams corresponding to at least two connected microphone users.

[0060] Specifically, the users who need to interact with each other include the users who initiate the connection and the users who participate in the connection. In the online conference scenario, the users who connect to the microphone are the participants of the online conference; in the live broadcast scenario, the users who connect to the microphone are the anchor who initiates the connection, as well as the audience or other anchors who participate in the connection. In addition, the audio streams corresponding to at least two users who connect to the microphone are the audio data that the subsequent user terminal needs to play.

[0061] In practical applications, the multi-person microphone connection method provided in the embodiment of the present application can be applied to pure audio microphone connection scenarios, and can also be applied to audio and video microphone connection scenarios. In the pure audio microphone connection scenario, the audio stream corresponding to the microphone user is the multimedia stream pushed by the microphone user; in the audio and video microphone connection scenario, the audio stream corresponding to the microphone user is the audio stream in the multimedia stream pushed by the microphone user.

[0062] It should be noted that the audio stream of each user participating in the live broadcast can be pushed to the content distribution network, which will mix the audio stream and push it to each user end for playback after obtaining the mixed stream. The user end can be the client of the user participating in the live broadcast or the client of the user watching the live broadcast.

[0063] For example, taking the case of two users connected to the microphone in a live broadcast scenario, the host end that initiates the connection collects the voice data of the connected hosts, generates an audio stream, and pushes it to the mixing module of the content distribution network; the user end that connects the microphone collects the voice data of the connected users, generates an audio stream, and pushes it to the mixing module of the content distribution network. At this point, the content distribution network can obtain the audio streams corresponding to the two connected users, so as to subsequently process the audio streams of each connected user.

[0064] Step 204: Create a corresponding sub-track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-track, and store the corresponding relationship between the sub-track and the connected user.

[0065] In practical applications, a sub-track generation sub-module can be added to the mixing module of the content distribution network. The sub-track generation sub-module is responsible for creating a corresponding sub-track for each input audio stream, and generating a low-bitrate audio stream corresponding to the audio stream, and transmitting the low-bitrate audio stream through the sub-track. Among them, each input audio stream needs to be mixed and pushed to the user end for playback, and the low-bitrate audio stream corresponding to each audio stream is only used to transmit the audio properties of the corresponding audio stream, and the audio properties are used to indicate the current speaking user. Among them, there are multiple audio properties corresponding to the audio, such as the sender of the audio, that is, the user ID of the user corresponding to the audio, the volume information of the audio, etc. The volume information can use any common representation method, such as dBFS commonly used in digital audio systems, that is, 0dB as the maximum value and -96dB as the minimum value, or a more intuitive and simple digital expression of 0 to 100.

[0066] It should be noted that each time the mixing module receives an audio stream, it can create a corresponding sub-track for the audio stream, and save the sub-track and the user ID of the connected user in a one-to-one correspondence for subsequent user-side inquiries. The low-bitrate audio stream corresponding to the audio stream is transmitted through the sub-track, and the low-bitrate audio stream can be used by the user side to determine the audio properties of the corresponding sub-track, thereby determining the audio properties of the corresponding connected user. In other words, through the low-bitrate audio streams of each sub-track, the audio properties of the connected user at the current time can be transmitted to the user side, such as which connected user is speaking at the current time and how loud the volume is, thereby ensuring the synchronization of the audio stream and the audio properties.

[0067] In an optional implementation of this embodiment, a corresponding sub-track is created for each audio stream, and a low-bitrate audio stream corresponding to the audio stream is transmitted through the sub-track. The specific implementation process may be as follows:

[0068] For the first audio stream, re-encode the first audio stream into a second audio stream, wherein the bit rate of the second audio stream is lower than that of the first audio stream, and the first audio stream is the audio stream corresponding to any connected microphone user;

[0069] A target sub-audio track is created, and a second audio stream is transmitted through the target sub-audio track, wherein the target sub-audio track corresponds to a microphone-connected user of the first audio stream.

[0070] It should be noted that, since the data transmitted in the sub-track is only used by the user end to determine the audio attributes and does not need to be played to the user, the first audio stream can be re-encoded into a second audio stream with a lower bit rate, that is, the bit rate of the second audio stream can be much lower than the first audio stream, and the second audio stream only needs to be able to determine the corresponding audio attributes, such as volume information. By analogy, the audio stream corresponding to each connected microphone user can be used as the first audio stream to generate a corresponding low-bit rate audio stream, and create a corresponding sub-track to transmit the low-bit rate audio stream.

[0071] In practical applications, when re-encoding the first audio stream into the second audio stream, the target bit rate of the second audio stream can be determined first. The target bit rate can be any value less than the original bit rate of the first audio stream. The smaller the target bit rate is, the more bandwidth can be saved.

[0072] For example, taking the anchor scenario as an example, assuming that the stream mixing module receives a 192kbps audio stream pushed by the anchor who initiated the joint microphone, the sub-track generation submodule in the stream mixing module can re-encode the 192kbps audio stream into an 8kbps audio stream to save 184kbps of bandwidth, create a corresponding sub-track, and transmit the 8kbps audio stream through the sub-track. In addition, the user ID of the anchor who initiated the joint microphone can be stored in correspondence with the track ID of the sub-track to indicate the correspondence between the joint microphone user and the sub-track.

[0073] In an embodiment of the present application, the first audio stream can be re-encoded into a second audio stream with low quality and low bit rate. The second audio stream is a low bit rate audio stream. The low bit rate audio stream is transmitted through a sub-audio track. While synchronously transmitting the audio attributes to the user end, the bandwidth is saved as much as possible.

[0074] In an optional implementation of this embodiment, the corresponding relationship between the sub-track and the connected microphone user is stored, and the specific implementation process may be as follows:

[0075] Arrange the sub-tracks in the order in which they were created;

[0076] Based on the arrangement order of each sub-track, the correspondence between the sub-track and the connected microphone user is stored.

[0077] It should be noted that in order not to add additional identification information to the low-bitrate audio stream transmitted by the sub-tracks, the sub-tracks can be arranged in the order of creation, and the correspondence between the sub-tracks and the connected users can be stored based on the arrangement order of the sub-tracks. In this way, the sub-tracks can be directly identified by the arrangement order of the sub-tracks, which is simple and convenient, without the need to introduce other identification information, and thus bandwidth is saved.

[0078] In an optional implementation of this embodiment, the correspondence between the sub-tracks and the users connected to the microphone is stored based on the arrangement order of the sub-tracks. The specific implementation process may be as follows:

[0079] For the first sub-track, the arrangement order of the first sub-track in each sub-track is used as the track identifier of the first sub-track, wherein the first sub-track is any one of the sub-tracks;

[0080] Determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream;

[0081] The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

[0082] In actual applications, the arrangement order of the first sub-track in each sub-track is used as the track identifier of the first sub-track, the microphone user identifier of the push user of the target audio stream corresponding to the first sub-track is determined, and the track identifier of the first sub-track and the microphone user identifier are stored in correspondence. By analogy, each sub-track can be used as the first sub-track, the corresponding microphone user identifier is determined, and the track identifier of each sub-track and the corresponding microphone user identifier are stored one by one.

[0083] In addition, during the microphone connection process, for each new user joining the microphone connection, a corresponding sub-track is created for it, and the corresponding relationship between the sub-track and the microphone connection user is updated.

[0084] For example, for user 1 who initiates the live broadcast, a corresponding sub-track is created, and the sub-track is arranged in an order of 1; for user 2 who is the first to join the live broadcast, a corresponding sub-track is created, and the sub-track is arranged in an order of 2; for user 3 who is the second to join the live broadcast, a corresponding sub-track is created, and the sub-track is arranged in an order of 3; ...; and so on, a correspondence table between sub-tracks and live broadcast users is obtained as shown in Table 1 below.

[0085] Table 1. Correspondence between sub-tracks and connected microphone users

[0086] subtrack Connected microphone users 1st sub track Connected user 1 Second sub-track Connected user 2 The third sub-track Connected microphone user 3 …… ……

[0087] In addition, in actual applications, the original audio stream in the main audio track and the low-bitrate audio streams of each sub-audio track need to be mixed into a mixed stream and pushed to the user end. Therefore, the mixed stream received by the user end includes the main audio track and each sub-audio track. The track identifier of the main audio track can be always set to 0, and each sub-audio track can be arranged from 1 based on the creation order, so that each audio track is identified based on the arrangement order of each audio track for the user end to identify and distinguish.

[0088] In the embodiment of the present application, each sub-track is directly identified by the order in which it is created, so that each sub-track and the connected microphone user are stored in correspondence, which is simple and convenient, does not require the introduction of other identification information, and saves bandwidth.

[0089] In an optional implementation of this embodiment, the corresponding relationship between the sub-track and the connected microphone user is stored, and the specific implementation process may also be:

[0090] Assign track identifiers to each sub-track;

[0091] For a first sub-audio track, determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream, wherein the first sub-audio track is any one of the sub-audio tracks;

[0092] The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

[0093] In practical applications, the sub-track generation sub-module in the mixing module can assign corresponding track identifiers to each generated sub-track, then determine the microphone user identifier of the push user of the target audio stream corresponding to the first sub-track, and store the track identifier of the first sub-track and the microphone user identifier in correspondence. By analogy, each sub-track can be used as the first sub-track, determine the corresponding microphone user identifier, and store the track identifier and the corresponding microphone user identifier of each sub-track one by one.

[0094] It should be noted that since the audio track identifier is randomly assigned by the sub-track generation submodule, when the corresponding low-bitrate audio stream is transmitted through the sub-track, the audio track identifier needs to be carried in the low-bitrate audio stream, so that when the subsequent user end receives the mixed stream, it can determine which sub-track transmits the low-bitrate audio stream, thereby determining the corresponding connected microphone user.

[0095] For example, for user 1 who initiates the live broadcast, a corresponding sub-track is created, and a track identifier X1 is assigned to the sub-track; for user 2 who is the first to join the live broadcast, a corresponding sub-track is created, and a track identifier X2 is assigned to the sub-track; for user 3 who is the second to join the live broadcast, a corresponding sub-track is created, and a track identifier X3 is assigned to the sub-track; and so on, to obtain the correspondence table between sub-tracks and live broadcast users as shown in Table 2 below.

[0096] Table 2. Correspondence between sub-tracks and connected microphone users

[0097] subtrack Connected microphone users X1 Connected user 1 X2 Connected user 2 X3 Connected microphone user 3 …… ……

[0098] In an embodiment of the present application, the sub-track generation sub-module in the mixing module can assign corresponding track identifiers to each generated sub-track to store each sub-track and the connected user in correspondence. The created sub-tracks do not need to be arranged in sequence and can be arranged randomly, which has high flexibility and can adapt to different application scenarios.

[0099] In an optional implementation of this embodiment, the corresponding relationship between the sub-track and the connected microphone user is stored, and the specific implementation process may be as follows:

[0100] Report the correspondence between the sub-track and the connected microphone user to the configuration center service.

[0101] It should be noted that, since the correspondence between the sub-audio track and the connected microphone user will only be updated when a new connected microphone user is added and changes less frequently, the sub-audio track generation submodule in the mixing module stores the sub-audio track and the connected microphone user in a one-to-one correspondence. After obtaining the correspondence between the sub-audio track and the connected microphone user, the sub-audio track generation submodule can report the correspondence between the sub-audio track and the connected microphone user to the configuration center service. The subsequent user terminal can obtain it from the configuration center service when needed. The correspondence between the sub-audio track and the connected microphone user does not need to be carried in the mixed stream and transmitted to the user terminal in real time, saving bandwidth.

[0102] Step 206: Generate a mixed stream based on the audio streams corresponding to the connected users and the low-bitrate audio streams of the sub-tracks.

[0103] It should be noted that the audio stream corresponding to each connected user is the actual audio content that needs to be played by the subsequent user end, and the low-bitrate audio stream of each sub-audio track is used to transmit the audio properties of the corresponding audio stream. Therefore, a mixed stream can be generated based on the audio stream corresponding to each connected user and the low-bitrate audio stream of each sub-audio track, so that the mixed stream can include both the actual audio content that needs to be played and the audio properties of the audio stream, thereby ensuring that the audio stream and audio properties are fully synchronized and avoiding audio property delays.

[0104] In an optional implementation of this embodiment, a mixed stream is generated based on the audio streams corresponding to each connected microphone user and the low bit rate audio streams of each sub-track. The specific implementation process may be as follows:

[0105] Mix the individual audio streams into an original audio stream and transmit the original audio stream through the main audio track;

[0106] The original audio stream of the main audio track and the low-bitrate audio streams of each sub-track are packaged into a mixed stream.

[0107] In actual applications, the mixing sub-module of the stream mixing module has been updated to support the input of multiple additional sub-audio tracks. The function of the original audio mixing remains unchanged, that is, the audio streams of each connected user are mixed and encoded into an original audio stream, which is transmitted through the main audio track. The original audio stream is the audio content that the subsequent user end needs to actually play. At the same time, it supports mixing the low-bitrate audio streams of each sub-audio track into the mixed stream.

[0108] It should be noted that the low-bitrate audio stream of each sub-track can be directly packaged together with the original audio stream of the main track to obtain a mixed stream. There is no need to re-encode the low-bitrate audio stream of each sub-track. You only need to package the low-bitrate audio stream of each sub-track into the mixed stream, with basically no additional performance consumption.

[0109] In an embodiment of the present application, the original audio stream of the main track and the low-bitrate audio streams of each sub-track can be packaged into a mixed stream, which includes multiple audio streams, the original audio stream of the main track is the audio content actually played, and the low-bitrate audio streams of each sub-track can be used to determine the audio properties of the audio stream. In this way, while transmitting the audio content actually played, the audio properties of the audio stream are transmitted through the low-bitrate audio streams transmitted by each sub-track, so that the audio properties are completely synchronized with the audio stream, thereby ensuring that the sound heard by the audience is completely synchronized with the sound icon / sound column, etc. seen, thereby improving the user experience.

[0110] Step 208: Push the mixed stream to the user end, wherein the low-bitrate audio stream of the sub-track in the mixed stream is used to instruct the user end to determine and identify the audio attributes of each connected microphone user at the current time based on the corresponding relationship.

[0111] In actual applications, after the mixing module of the content distribution network obtains the mixed stream, it can push the mixed stream to the user end. In addition to the original audio stream that needs to be played, the mixed stream can also include the low-bitrate audio stream of each sub-track. Therefore, after the user end receives the mixed stream pushed by the content distribution network, it can parse the mixed stream to obtain the low-bitrate audio stream of each sub-track, and determine and identify the audio attributes of each connected user at the current time by combining the correspondence between the obtained sub-track and the connected user.

[0112] Specifically, the audio attributes of the connected user at the current time may include the connected user ID, volume information, etc. A sound icon and / or a volume icon corresponding to the volume information may be added to the connected interface to identify who is currently speaking and the volume of the speech.

[0113] It should be noted that the low-bitrate audio stream of each sub-track has been brought into the mixed stream. Based on the low-bitrate audio stream, the corresponding sub-track identification, volume information and other audio properties can be determined. While playing the original audio stream of the main track in the mixed stream to the user, it is possible to determine in real time which connected microphone user is currently speaking and the volume of the speech. Therefore, some effects can be drawn in real time, such as avatar special effects, speakers, sound columns, etc. of the corresponding host / connected microphone user, to achieve a better experience.

[0114] In the multi-person microphone connection method provided in the embodiment of the present application, the content distribution network can generate a corresponding low-bitrate audio stream for each audio stream, and create a corresponding sub-audio track, and transmit the low-bitrate audio stream through the sub-audio track, and then generate a mixed stream based on the audio streams corresponding to each microphone-connected user and the low-bitrate audio streams of each sub-audio track. In addition to the audio streams corresponding to each microphone-connected user that need to be played, that is, the actual audio content, the mixed stream also includes the low-bitrate audio streams of each sub-audio track. The low-bitrate audio stream is used to determine the audio properties, so that after the user end receives the mixed stream, it can directly parse the mixed stream to simultaneously obtain the audio stream to be played and the low-bitrate audio streams of each sub-audio track, so as to calculate and determine the audio properties of each microphone-connected user at the current time based on the low-bitrate audio streams of each sub-audio track, and mark them. In this way, by creating a corresponding sub-audio track for each connected microphone user and transmitting the low-bitrate audio stream corresponding to the audio stream of each connected microphone user, it is possible to transmit the audio stream of each connected microphone user to the user end while transmitting the corresponding audio attributes to the user end, thereby avoiding the delay of audio attributes. The audio attributes obtained by the user end are completely synchronized with the audio stream, ensuring that the currently speaking connected microphone user identified in the current connected microphone interface is synchronized with the currently actually speaking connected microphone user, thereby ensuring the user experience of the connected microphone users.

[0115] Figure 3aA flowchart of another multi-person microphone connection method provided according to an embodiment of the present application is shown, which is applied to a user end and specifically includes the following steps 302-306:

[0116] Step 302: Receive the mixed stream pushed by the content distribution network, and obtain the low-bitrate audio stream of each sub-track from the mixed stream.

[0117] It should be noted that after the mixing module of the content distribution network obtains the mixed stream, it can push the mixed stream to the user end. In addition to the original audio stream to be played, the mixed stream also includes the low-bitrate audio stream of each sub-track. Therefore, after the user end receives the mixed stream pushed by the content distribution network, it can parse the mixed stream and obtain the low-bitrate audio stream of each sub-track from the mixed stream, so as to facilitate the subsequent determination of the audio properties of each sub-track based on the low-bitrate audio stream of each sub-track.

[0118] Step 304: Based on the low bit rate audio stream of each sub-track, determine the audio attribute of each sub-track at the current time.

[0119] Specifically, the audio attributes of the sub-track at the current time may include a sub-track identifier, volume information at the current time, and the like.

[0120] In an optional implementation of this embodiment, the volume information of the currently transmitted audio may be calculated based on the phonemes in the low bit rate audio stream to determine the corresponding audio attributes, that is, based on the low bit rate audio stream of each sub-track, the audio attributes of each sub-track at the current time are determined. The specific implementation process may be as follows:

[0121] For the first low-bit-rate audio stream, obtain the phoneme information of the transmitted audio from the first low-bit-rate audio stream, and obtain the track identifier of the sub-track corresponding to the first low-bit-rate audio stream;

[0122] Determine the volume information of the transmitted audio according to the phoneme information;

[0123] The volume information and the track identifier are used as audio attributes of the second sub-track, wherein the second sub-track is the track corresponding to the first low-bitrate audio stream.

[0124] Among them, phoneme is the smallest unit or smallest speech segment that constitutes a syllable. It is the smallest linear speech unit divided from the perspective of sound quality. Phoneme is a concrete physical phenomenon.

[0125] It should be noted that in the live broadcast scenario, the user-side player can receive a mixed stream of multiple audio tracks. Except for the first main audio track whose content is the real live broadcast room content and is played directly, each other sub-audio track only extracts a low-bitrate audio stream for calculating and determining the audio properties of the sub-audio track.

[0126] In actual applications, the user end can obtain the phoneme information of the transmitted audio and the track identifier of the sub-track corresponding to the first low-bitrate audio stream from the first low-bitrate audio stream; then, the volume information of the transmitted audio can be determined based on the phoneme information; after that, the volume information and track identifier are used as the audio attributes of the second sub-track, and so on. Each low-bitrate audio stream can be used as the first low-bitrate audio stream, thereby determining the audio attributes of each sub-track. In this way, the audio attribute can indicate the source sub-track, thereby indicating the corresponding connected microphone user, and can indicate the volume information of the current time audio.

[0127] Among them, volume represents the intensity of sound, also known as loudness, intensity or energy. It refers to the subjective feeling of the human ear to the size and strength of the sound heard. The objective evaluation scale is the amplitude of the sound. This feeling comes from the pressure generated when the object vibrates, that is, the sound pressure. The vibration of the object transmits its vibration energy through different media.

[0128] In specific implementation, the volume information of the transmitted audio can be calculated by analogy with the signal amplitude within a phoneme. Specifically, it can be calculated in the following two ways: taking the sum of the absolute values ​​(signal absolute values) of each phoneme in the transmitted audio as the volume information of the transmitted audio; taking the sum of the square values ​​of the signal values ​​of each phoneme in the transmitted audio, taking the logarithm with base 10, and multiplying it by 10 to obtain the volume information of the transmitted audio. Of course, the volume information of the transmitted audio can also be calculated in other ways, and the embodiments of the present application are not limited to this.

[0129] In an embodiment of the present application, the audio properties of each sub-track can be calculated based on the low-bitrate audio stream of each sub-track, thereby achieving simultaneous acquisition of the audio content actually to be played and the corresponding audio properties, avoiding time delays in the audio properties, and ensuring that the audio properties obtained by the user end are completely synchronized with the audio stream, thereby ensuring the user experience of the connected microphone user.

[0130] Step 306: Obtain the correspondence between the sub-tracks and the connected users, determine the audio properties of each connected user at the current time based on the audio properties of each sub-track at the current time, and identify the audio properties corresponding to each connected user in the connected interface at the current time.

[0131] It should be noted that after the user end determines the audio properties of each sub-track at the current time, it needs to determine the audio properties of each connected user based on the correspondence between the sub-track and the connected user. Therefore, the user end needs to obtain the correspondence between the sub-track and the connected user.

[0132] In an optional implementation of this embodiment, the corresponding relationship between the sub-track and the connected microphone user is obtained, and the specific implementation process may be as follows:

[0133] When the corresponding relationship update conditions are met, the corresponding relationship between the sub-track and the connected microphone user is obtained from the configuration center service.

[0134] In actual applications, the mixing module of the content distribution network will upload the correspondence between the sub-audio track and the connected microphone user to the configuration center service. Therefore, the user end can obtain the correspondence between the sub-audio track and the connected microphone user from the configuration center service when the correspondence update conditions are met. The correspondence between the sub-audio track and the connected microphone user does not need to be carried in the mixed stream for real-time reception, saving bandwidth.

[0135] Among them, the corresponding relationship update condition refers to a pre-set condition, and the user determines whether the user terminal currently needs to obtain the latest correspondence between the sub-audio track and the connected microphone user. For example, the corresponding relationship update condition can be when the connected microphone starts, when a new connected microphone user is added, when the update timing is reached, etc.

[0136] In an optional implementation of this embodiment, the audio attributes of each connected microphone user include volume information and a user identifier of the connected microphone user. The audio attributes corresponding to each connected microphone user are identified in the connected microphone interface at the current time. The specific implementation process may be as follows:

[0137] For a first microphone-connected user, determine volume information corresponding to the first microphone-connected user, wherein the first microphone-connected user is any one of the microphone-connected users;

[0138] When the volume information satisfies the set volume condition, determining, according to the first user identifier, a first user avatar of the first microphone-connected user in the microphone-connected interface;

[0139] A sound icon and / or a volume icon corresponding to the volume information is added to the first user's avatar.

[0140] The volume condition is a pre-set rule for determining whether the connected user is speaking, so as to avoid misidentifying noise as the connected user speaking. For example, the volume condition is set to be greater than 40 decibels.

[0141] In actual applications, for each connected user, it can be determined whether the volume is greater than the volume threshold. If it is greater than the volume threshold, it means that the connected user is speaking. At this time, the user avatar of the connected user can be found, and a voice icon and / or a volume icon corresponding to the volume information can be added to the user avatar.

[0142] Among them, the sound icon indicates that the connected user is currently speaking, and the sound icon can be pre-set, such as a small speaker icon, a bold border, an avatar special effect or other graphic icons; the volume icon can indicate the volume at the current time, such as the volume icon can be a volume bar, a specific value, or other graphic icons that can indicate the volume.

[0143] For example, in a live broadcast scenario, four users are connected to the microphone. Figure 3b is a schematic diagram of a microphone connection interface provided by an embodiment of the present application, such as Figure 3b As shown in the figure, the users connected to the microphone are user 1, user 2, user 3 and user 4. Figure 3b The avatars of the four users are displayed in the video. The 0th audio track in the mixed stream received by the user end is the main audio track. The original audio stream in the main audio track is the content that needs to be played in the live broadcast room (the mixed audio stream after the audio streams of the four connected users are mixed). The 1st to 4th audio tracks are sub-audio tracks. The low-bitrate audio streams of each sub-audio track can determine the audio properties corresponding to each sub-audio track. Assume that the audio properties corresponding to the 1st sub-audio track are: sub-audio track 1, volume 0, the audio properties corresponding to the 2nd sub-audio track are: sub-audio track 2, volume 50, the audio properties corresponding to the 3rd sub-audio track are: sub-audio track 3, volume 10, and the audio properties corresponding to the 4th sub-audio track are: sub-audio track 4, volume 0.

[0144] Assuming that the correspondence between the sub-tracks and the connected users obtained by the user end is that sub-track 1 corresponds to user 1, sub-track 2 corresponds to user 2, sub-track 3 corresponds to user 3, and sub-track 4 corresponds to user 4, the audio attributes of each connected user can be determined as follows: user 1, volume 0, user 2, volume 50, user 3, volume 10, user 4, volume 0. At this time, only the volume of user 2 exceeds the remaining threshold of 40, indicating that user 2 is speaking at this time, and the speaking volume is 50. Find the user 2 avatar corresponding to user 2 in the current connected interface, add a small speaker icon to the avatar, and add a volume bar. The current size of the volume bar indicates that the volume is 50.

[0145] In the method for connecting multiple people to a microphone provided by an embodiment of the present application, a user terminal can receive a mixed stream pushed by a content distribution network, and the mixed stream includes the original audio stream after mixing the audio streams corresponding to each connected microphone user, and the low-bitrate audio stream of each sub-track. Therefore, after the user terminal receives the mixed stream pushed by the content distribution network, it can directly parse the mixed stream to simultaneously obtain the audio stream to be played and the low-bitrate audio stream of each sub-track, thereby determining the audio properties of each connected microphone user at the current time based on the low-bitrate audio stream of each sub-track, and marking the audio properties in the connected microphone interface to render different effects. In this way, by creating a corresponding sub-track for each connected microphone user and transmitting the low-bitrate audio stream corresponding to the audio stream of each connected microphone user, it is possible to transmit the corresponding audio properties to the user terminal while transmitting the audio stream of each connected microphone user to the user terminal, avoiding the existence of delay in the audio properties, and the audio properties obtained by the user terminal are completely synchronized with the audio stream, ensuring that the connected microphone user currently speaking identified in the current connected microphone interface is synchronized with the connected microphone user currently speaking, thereby ensuring the user experience of the connected microphone user.

[0146] The following combination Figure 4 , taking the application of the multi-person microphone connection method provided by this application in a two-person microphone connection scenario in a live broadcast as an example, the multi-person microphone connection method is further explained. Figure 4 A schematic diagram of a processing process of a multi-person microphone connection method applied to a two-person microphone connection scenario in a live broadcast provided by an embodiment of the present application is shown, which specifically includes the following steps:

[0147] Taking the live broadcast scenario as an example, a typical two-person microphone scene consists of a microphone host and a microphone participant. The host stream pushed by the microphone host and the microphone participant will enter the stream mixing module respectively. The host stream can include the host audio stream, and the microphone stream can include the microphone audio stream.

[0148] The sub-audio track generation sub-module in the mixing module can generate a low-bit-rate audio stream for the host corresponding to the host audio stream, and create a corresponding sub-audio track 1, through which the host's low-bit-rate audio stream is transmitted; and, it can also generate a low-bit-rate audio stream for the connected microphone corresponding to the connected microphone audio stream, and create a corresponding sub-audio track 2, through which the low-bit-rate audio stream for the connected microphone is transmitted.

[0149] The mixing submodule in the mixing module can mix the host audio stream and the connected microphone audio stream into the original audio stream, and transmit the original audio stream through the main audio track; and the original audio stream of the main audio track and the host's low-bitrate audio stream of sub-audio track 1 and the connected microphone low-bitrate audio stream of sub-audio track 2 can be packaged into a live stream, and the live stream can be pushed to the live broadcast room.

[0150] In addition, when creating each sub-track, the sub-track generation submodule can also store the correspondence between the sub-track and the connected microphone user, and report it to the configuration center service. Therefore, the live broadcast room can also obtain the correspondence between the sub-track and the connected microphone user from the configuration center service, parse the live broadcast stream, obtain the host's low-bitrate audio stream of sub-track 1, and determine the audio properties of sub-track 1 based on the host's low-bitrate audio stream, that is, the audio properties of the host's audio stream pushed by the host; and, it is also possible to obtain the connected microphone low-bitrate audio stream of sub-track 2, and determine the audio properties of sub-track 2 based on the connected microphone low-bitrate audio stream, that is, the audio properties of the connected microphone audio stream pushed by the connected microphone participant. While the original audio stream of the main audio track is played in the live broadcast room for users to listen to, the corresponding audio properties can also be identified in the connected microphone interface.

[0151] In the multi-person microphone connection method provided in the embodiment of the present application, the content distribution network can generate a corresponding low-bitrate audio stream for each audio stream, and create a corresponding sub-audio track, and transmit the low-bitrate audio stream through the sub-audio track, and then generate a mixed stream based on the audio streams corresponding to each microphone-connected user and the low-bitrate audio streams of each sub-audio track. In addition to the audio streams corresponding to each microphone-connected user that need to be played, that is, the actual audio content, the mixed stream also includes the low-bitrate audio streams of each sub-audio track. The low-bitrate audio stream is used to determine the audio properties, so that after the user end receives the mixed stream, it can directly parse the mixed stream to simultaneously obtain the audio stream to be played and the low-bitrate audio streams of each sub-audio track, so as to calculate and determine the audio properties of each microphone-connected user at the current time based on the low-bitrate audio streams of each sub-audio track, and mark them. In this way, by creating a corresponding sub-audio track for each connected microphone user and transmitting the low-bitrate audio stream corresponding to the audio stream of each connected microphone user, it is possible to transmit the audio stream of each connected microphone user to the user end while transmitting the corresponding audio attributes to the user end, thereby avoiding the delay of audio attributes. The audio attributes obtained by the user end are completely synchronized with the audio stream, ensuring that the currently speaking connected microphone user identified in the current connected microphone interface is synchronized with the currently actually speaking connected microphone user, thereby ensuring the user experience of the connected microphone users.

[0152] Corresponding to the above method embodiment, the present application also provides a content distribution network embodiment, Figure 5 FIG. 1 shows a schematic diagram of a content distribution network provided by an embodiment of the present application. Figure 5 As shown, the device comprises:

[0153] A first acquisition module 502 is configured to acquire audio streams corresponding to at least two microphone-connected users;

[0154] The creation module 504 is configured to create a corresponding sub-track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-track, and store a correspondence between the sub-track and the connected microphone user;

[0155] A generating module 506 is configured to generate a mixed stream based on the audio streams corresponding to the connected microphone users and the low bit rate audio streams of the sub-tracks;

[0156] The push module 508 is configured to push the mixed stream to the user end, wherein the low-bitrate audio stream of the sub-track in the mixed stream is used to instruct the user end to determine and identify the audio attributes of each connected microphone user at the current time based on the corresponding relationship.

[0157] Optionally, the generating module 506 is further configured to:

[0158] Mix the individual audio streams into an original audio stream and transmit the original audio stream through the main audio track;

[0159] The original audio stream of the main audio track and the low-bitrate audio streams of each sub-track are packaged into a mixed stream.

[0160] Optionally, the creation module 504 is further configured to:

[0161] For the first audio stream, re-encode the first audio stream into a second audio stream, wherein the bit rate of the second audio stream is lower than that of the first audio stream, and the first audio stream is the audio stream corresponding to any connected microphone user;

[0162] A target sub-audio track is created, and a second audio stream is transmitted through the target sub-audio track, wherein the target sub-audio track corresponds to a microphone-connected user of the first audio stream.

[0163] Optionally, the creation module 504 is further configured to:

[0164] Arrange the sub-tracks in the order in which they were created;

[0165] Based on the arrangement order of each sub-track, the correspondence between the sub-track and the connected microphone user is stored.

[0166] Optionally, the creation module 504 is further configured to:

[0167] For the first sub-track, the arrangement order of the first sub-track in each sub-track is used as the track identifier of the first sub-track, wherein the first sub-track is any one of the sub-tracks;

[0168] Determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream;

[0169] The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

[0170] Optionally, the creation module 504 is further configured to:

[0171] Assign track identifiers to each sub-track;

[0172] For a first sub-audio track, determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream, wherein the first sub-audio track is any one of the sub-audio tracks;

[0173] The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

[0174] Optionally, the creation module 504 is further configured to:

[0175] Report the correspondence between sub-tracks and connected microphone users to the configuration center service.

[0176] The content distribution network provided in the embodiment of the present application can generate a corresponding low-bitrate audio stream for each audio stream, and create a corresponding sub-audio track, and transmit the low-bitrate audio stream through the sub-audio track. Then, based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-audio track, a mixed stream is generated. In addition to the audio streams corresponding to each connected microphone user that need to be played, that is, the actual audio content, the mixed stream also includes the low-bitrate audio streams of each sub-audio track. The low-bitrate audio stream is used to determine the audio properties, so that after the user end receives the mixed stream, it can directly parse the mixed stream to obtain the audio stream to be played and the low-bitrate audio streams of each sub-audio track at the same time, so as to calculate and determine the audio properties of each connected microphone user at the current time based on the low-bitrate audio streams of each sub-audio track, and mark them. In this way, by creating a corresponding sub-audio track for each connected microphone user and transmitting the low-bitrate audio stream corresponding to the audio stream of each connected microphone user, it is possible to transmit the audio stream of each connected microphone user to the user end while transmitting the corresponding audio attributes to the user end, thereby avoiding the delay of audio attributes. The audio attributes obtained by the user end are completely synchronized with the audio stream, ensuring that the currently speaking connected microphone user identified in the current connected microphone interface is synchronized with the currently actually speaking connected microphone user, thereby ensuring the user experience of the connected microphone users.

[0177] The above is a schematic solution of a content distribution network of this embodiment. It should be noted that the technical solution of the content distribution network and the technical solution of the multi-person microphone connection method described above are of the same concept. For details not described in detail in the technical solution of the content distribution network, please refer to the description of the technical solution of the multi-person microphone connection method described above.

[0178] Corresponding to the above method embodiment, the present application also provides a user end embodiment, Figure 6 FIG. 1 shows a schematic diagram of the structure of a user terminal provided by an embodiment of the present application. Figure 6 As shown, the device comprises:

[0179] The second acquisition module 602 is configured to receive the mixed stream pushed by the content distribution network, and obtain the low-bitrate audio stream of each sub-track from the mixed stream;

[0180] The determination module 604 is configured to determine the audio attribute of each sub-track at the current time based on the low bit rate audio stream of each sub-track;

[0181] The identification module 606 is configured to obtain the correspondence between the sub-audio tracks and the connected users, determine the audio properties of each connected user at the current time based on the audio properties of each sub-audio track at the current time, and identify the audio properties corresponding to each connected user in the connected interface at the current time.

[0182] Optionally, the audio attributes of each connected microphone user include volume information and a user identifier of the connected microphone user; the identification module 606 is further configured to:

[0183] For a first microphone-connected user, determine volume information corresponding to the first microphone-connected user, wherein the first microphone-connected user is any one of the microphone-connected users;

[0184] When the volume information satisfies the set volume condition, determining, according to the first user identifier, a first user avatar of the first microphone-connected user in the microphone-connected interface;

[0185] A sound icon and / or a volume icon corresponding to the volume information is added to the first user's avatar.

[0186] Optionally, the identification module 606 is further configured to:

[0187] When the corresponding relationship update conditions are met, the corresponding relationship between the sub-track and the connected microphone user is obtained from the configuration center service.

[0188] Optionally, the determination module 604 is further configured to:

[0189] For the first low-bit-rate audio stream, obtain the phoneme information of the transmitted audio from the first low-bit-rate audio stream, and obtain the track identifier of the sub-track corresponding to the first low-bit-rate audio stream;

[0190] Determine the volume information of the transmitted audio according to the phoneme information;

[0191] The volume information and the track identifier are used as audio attributes of the second sub-track, wherein the second sub-track is the track corresponding to the first low-bitrate audio stream.

[0192] The user terminal provided by the embodiment of the present application can receive a mixed stream pushed by a content distribution network, and the mixed stream includes the original audio stream after mixing the audio streams corresponding to each connected microphone user, and the low-bitrate audio stream of each sub-track. Therefore, after the user terminal receives the mixed stream pushed by the content distribution network, it can directly parse the mixed stream to obtain the audio stream to be played and the low-bitrate audio stream of each sub-track at the same time, thereby determining the audio properties of each connected microphone user at the current time based on the low-bitrate audio stream of each sub-track, and marking the audio properties in the connected microphone interface to render different effects. In this way, by creating corresponding sub-tracks for each connected microphone user and transmitting the low-bitrate audio stream corresponding to the audio stream of each connected microphone user, it is achieved that while transmitting the audio stream of each connected microphone user to the user terminal, the corresponding audio properties are transmitted to the user terminal, avoiding the delay of the audio properties, and the audio properties obtained by the user terminal are completely synchronized with the audio stream, ensuring that the connected microphone user currently speaking identified in the current connected microphone interface is synchronized with the connected microphone user actually speaking, thereby ensuring the user experience of the connected microphone user.

[0193] The above is a schematic scheme of a user terminal of this embodiment. It should be noted that the technical scheme of the user terminal and the technical scheme of the multi-person microphone connection method described above are of the same concept, and the details not described in detail in the technical scheme of the user terminal can be found in the description of the technical scheme of the multi-person microphone connection method described above.

[0194] Corresponding to the above method embodiment, the present application also provides a multi-person microphone connection system embodiment, Figure 7 FIG. 1 shows a schematic diagram of a multi-person microphone connection system provided by an embodiment of the present application. Figure 7 As shown, the multi-person live broadcast system includes a content distribution network 702 and a user terminal 704;

[0195] The content distribution network 702 is configured to obtain audio streams corresponding to at least two connected microphone users; create a corresponding sub-track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-track, and store the corresponding relationship between the sub-track and the connected microphone users; generate a mixed stream based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-track; and push the mixed stream to the user end;

[0196] The user end 704 is configured to receive the mixed stream pushed by the content distribution network, obtain the low-bitrate audio stream of each sub-track from the mixed stream; determine the audio properties of each sub-track at the current time based on the low-bitrate audio stream of each sub-track; obtain the correspondence between the sub-track and the connected microphone user, determine the audio properties of each connected microphone user at the current time based on the audio properties of each sub-track at the current time, and identify the audio properties corresponding to each connected microphone user in the connected microphone interface at the current time.

[0197] In the multi-person live broadcast system provided by the embodiment of the present application, the content distribution network can generate a corresponding low-bitrate audio stream for each audio stream, and create a corresponding sub-audio track, and transmit the low-bitrate audio stream through the sub-audio track. Then, based on the audio streams corresponding to each live broadcast user and the low-bitrate audio streams of each sub-audio track, a mixed stream is generated. In addition to the audio streams corresponding to each live broadcast user that need to be played, that is, the actual audio content, the mixed stream also includes the low-bitrate audio streams of each sub-audio track. The low-bitrate audio stream is used to determine the audio properties, so that after the user end receives the mixed stream, it can directly parse the mixed stream to simultaneously obtain the audio stream to be played and the low-bitrate audio streams of each sub-audio track, so as to calculate and determine the audio properties of each live broadcast user at the current time based on the low-bitrate audio streams of each sub-audio track, and mark them. In this way, by creating a corresponding sub-audio track for each connected microphone user and transmitting the low-bitrate audio stream corresponding to the audio stream of each connected microphone user, it is possible to transmit the audio stream of each connected microphone user to the user end while transmitting the corresponding audio attributes to the user end, thereby avoiding the delay of audio attributes. The audio attributes obtained by the user end are completely synchronized with the audio stream, ensuring that the currently speaking connected microphone user identified in the current connected microphone interface is synchronized with the currently actually speaking connected microphone user, thereby ensuring the user experience of the connected microphone users.

[0198] The above is a schematic scheme of a multi-person microphone connection system of this embodiment. It should be noted that the technical scheme of the multi-person microphone connection system and the technical scheme of the multi-person microphone connection method described above are of the same concept, and the details not described in detail in the technical scheme of the multi-person microphone connection system can be referred to the description of the technical scheme of the multi-person microphone connection method described above.

[0199] Figure 8 The block diagram of a computing device according to an embodiment of the present application is shown. The components of the computing device 800 include but are not limited to a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and the database 850 is used to store data.

[0200] The computing device 800 also includes an access device 840 that enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0201] In one embodiment of the present application, the above components of the computing device 800 and Figure 8 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 8 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0202] The computing device 800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 800 may also be a mobile or stationary server.

[0203] The processor 820 is used to execute the following computer executable instructions to implement any of the operating steps of the above-mentioned multi-person microphone connection method.

[0204] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the multi-person microphone connection method described above are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the multi-person microphone connection method described above.

[0205] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the operating steps of the above-mentioned multi-person microphone connection method.

[0206] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multi-person microphone connection method described above are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the multi-person microphone connection method described above.

[0207] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0208] Computer instructions include computer program codes, which may be in source code form, object code form, executable files, or some intermediate form, etc. Computer readable media may include: any entity or device capable of carrying computer program codes, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0209] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0210] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0211] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A method for connecting multiple people to microphones, characterized in that: Applied to content delivery networks, including: Get the audio streams corresponding to at least two connected users; Create a corresponding sub-track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-track, and store the corresponding relationship between the sub-track and the connected microphone user; Generate a mixed stream based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-track; Pushing the mixed stream to the user terminal, wherein the low-bitrate audio stream of the sub-track in the mixed stream is used to instruct the user terminal to determine and identify the audio attributes of each connected microphone user at the current time based on the corresponding relationship; The corresponding relationship between the stored sub-tracks and the connected microphone users includes: Arrange the sub-tracks in the order of creation; Based on the arrangement order of the sub-tracks, storing the corresponding relationship between the sub-tracks and the users connected to the microphone; The storing the correspondence between the sub-tracks and the connected microphone users based on the arrangement order of the sub-tracks includes: For a first sub-track, using the arrangement order of the first sub-track in the sub-tracks as a track identifier of the first sub-track, wherein the first sub-track is any one of the sub-tracks; Determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream; The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

2. The method for connecting multiple people to microphones according to claim 1, characterized in that: The generating of a mixed stream based on the audio streams corresponding to the connected microphone users and the low bit rate audio streams of the sub-tracks includes: Mixing the respective audio streams into an original audio stream, and transmitting the original audio stream through a main audio track; The original audio stream of the main audio track and the low-bitrate audio streams of each sub-audio track are packaged into the mixed stream.

3. The method for connecting multiple people to microphones according to claim 1, characterized in that: The step of creating a corresponding sub-track for each audio stream and transmitting a low-bitrate audio stream corresponding to the audio stream through the sub-track includes: For a first audio stream, re-encode the first audio stream into a second audio stream, wherein the bit rate of the second audio stream is lower than that of the first audio stream, and the first audio stream is an audio stream corresponding to any microphone-connected user; A target sub-audio track is created, and the second audio stream is transmitted through the target sub-audio track, wherein the target sub-audio track corresponds to a microphone-connected user of the first audio stream.

4. The method for connecting multiple people to microphones according to any one of claims 1 to 3, characterized in that: The correspondence between the stored sub-tracks and the connected microphone users includes: Assign track identifiers to each sub-track; For a first sub-audio track, determining a target audio stream corresponding to the first sub-audio track, and determining a microphone-connected user identifier corresponding to the target audio stream, wherein the first sub-audio track is any one of the sub-audio tracks; The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

5. The method for connecting multiple people via microphones according to any one of claims 1 to 3, characterized in that: The correspondence between the stored sub-tracks and the connected microphone users includes: The correspondence between the sub-track and the connected microphone user is reported to the configuration center service.

6. A method for connecting multiple people to microphones, characterized in that: Applied to the user side, including: Receive a mixed stream pushed by a content distribution network, and obtain a low-bitrate audio stream of each sub-track from the mixed stream; Determine the audio properties of each sub-track at a current time based on the low bit rate audio stream of each sub-track; Obtaining the correspondence between the sub-tracks and the connected-microphone users, determining the audio attributes of the connected-microphone users at the current time based on the audio attributes of the sub-tracks at the current time, and identifying the audio attributes corresponding to the connected-microphone users in the connected-microphone interface at the current time; The corresponding relationship between the sub-track and the connected microphone user includes: Arrange the sub-tracks in the order of creation; Based on the arrangement order of the sub-tracks, storing the corresponding relationship between the sub-tracks and the users connected to the microphone; The storing the correspondence between the sub-tracks and the connected microphone users based on the arrangement order of the sub-tracks includes: For a first sub-track, using the arrangement order of the first sub-track in the sub-tracks as a track identifier of the first sub-track, wherein the first sub-track is any one of the sub-tracks; Determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream; The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

7. The method for connecting multiple people to microphones according to claim 6, characterized in that: The audio attributes of each connected microphone user include volume information and a user ID of the connected microphone user; The identifying the audio attributes corresponding to each microphone-connected user in the microphone-connected interface at the current time includes: For a first connected-microphone user, determining volume information corresponding to the first connected-microphone user, wherein the first connected-microphone user is any one of the connected-microphone users; When the volume information satisfies the set volume condition, determining, according to the first user identifier, a first user avatar of the first microphone-connected user in the microphone-connected interface; A sound icon and / or a volume icon corresponding to the volume information is added to the first user avatar.

8. The method for connecting multiple people to microphones according to claim 6, characterized in that: The obtaining of the correspondence between the sub-track and the connected microphone user includes: When the corresponding relationship update condition is met, the corresponding relationship between the sub-track and the connected microphone user is obtained from the configuration center service.

9. The method for connecting multiple people via microphones according to any one of claims 6 to 8, characterized in that: The step of determining the audio properties of each sub-track at a current time based on the low bit rate audio stream of each sub-track comprises: For the first low-bit-rate audio stream, obtain the phoneme information of the transmitted audio from the first low-bit-rate audio stream, and obtain the track identifier of the sub-track corresponding to the first low-bit-rate audio stream; Determining volume information of the transmitted audio according to the phoneme information; The volume information and the audio track identifier are used as audio attributes of a second sub-audio track, wherein the second sub-audio track is the audio track corresponding to the first low-bitrate audio stream.

10. A content distribution network, characterized in that: include: A first acquisition module is configured to acquire audio streams corresponding to at least two connected microphone users; A creation module, configured to create a corresponding sub-track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-track, and store a correspondence between the sub-track and the connected microphone user; A generation module is configured to generate a mixed stream based on the audio streams corresponding to the connected microphone users and the low bit rate audio streams of the sub-tracks; A push module is configured to push the mixed stream to the user terminal, wherein the low-bitrate audio stream of the sub-track in the mixed stream is used to instruct the user terminal to determine and identify the audio attributes of each connected microphone user at the current time based on the corresponding relationship; The corresponding relationship between the stored sub-tracks and the connected microphone users includes: Arrange the sub-tracks in the order of creation; Based on the arrangement order of the sub-tracks, storing the corresponding relationship between the sub-tracks and the users connected to the microphone; The storing the correspondence between the sub-tracks and the connected microphone users based on the arrangement order of the sub-tracks includes: For a first sub-track, using the arrangement order of the first sub-track in the sub-tracks as a track identifier of the first sub-track, wherein the first sub-track is any one of the sub-tracks; Determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream; The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

11. A user terminal, characterized in that: include: A second acquisition module is configured to receive a mixed stream pushed by a content distribution network, and acquire a low-bitrate audio stream of each sub-track from the mixed stream; A determination module, configured to determine the audio properties of each sub-track at a current time based on the low bit rate audio stream of each sub-track; an identification module configured to obtain a correspondence between a sub-track and a connected microphone user, determine the audio attributes of each connected microphone user at the current time based on the audio attributes of each sub-track at the current time, and identify the audio attributes corresponding to each connected microphone user in the connected microphone interface at the current time; The corresponding relationship between the sub-track and the connected microphone user includes: Arrange the sub-tracks in the order of creation; Based on the arrangement order of the sub-tracks, storing the corresponding relationship between the sub-tracks and the users connected to the microphone; The storing the correspondence between the sub-tracks and the connected microphone users based on the arrangement order of the sub-tracks includes: For a first sub-track, using the arrangement order of the first sub-track in the sub-tracks as a track identifier of the first sub-track, wherein the first sub-track is any one of the sub-tracks; Determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream; The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

12. A multi-person microphone connection system, characterized in that: The multi-person live broadcast system includes a content distribution network and a user terminal; The content distribution network is configured to obtain audio streams corresponding to at least two connected microphone users; create a corresponding sub-track for each audio stream, transmit a low-bitrate audio stream corresponding to the audio stream through the sub-track, and store a corresponding relationship between the sub-track and the connected microphone users; generate a mixed stream based on the audio streams corresponding to each connected microphone user and the low-bitrate audio streams of each sub-track; and push the mixed stream to the user end; The user terminal is configured to receive a mixed stream pushed by a content distribution network, obtain a low-bitrate audio stream of each sub-track from the mixed stream; determine the audio properties of each sub-track at a current time based on the low-bitrate audio stream of each sub-track; obtain a correspondence between a sub-track and a connected microphone user, determine the audio properties of each connected microphone user at a current time based on the audio properties of each sub-track at a current time, and identify the audio properties corresponding to each connected microphone user in a connected microphone interface at a current time; The corresponding relationship between the stored sub-tracks and the connected microphone users includes: Arrange the sub-tracks in the order of creation; Based on the arrangement order of the sub-tracks, storing the corresponding relationship between the sub-tracks and the users connected to the microphone; The storing the correspondence between the sub-tracks and the connected microphone users based on the arrangement order of the sub-tracks includes: For a first sub-track, using the arrangement order of the first sub-track in the sub-tracks as a track identifier of the first sub-track, wherein the first sub-track is any one of the sub-tracks; Determine a target audio stream corresponding to the first sub-audio track, and determine a microphone-connected user identifier corresponding to the target audio stream; The track identifier of the first sub-track and the microphone-connected user identifier are stored correspondingly.

13. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the operating steps of the multi-person microphone connection method described in any one of claims 1-5 or claims 6-9.

14. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the operating steps of the multi-person microphone connection method described in any one of claims 1-5 or claims 6-9.

15. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 or claims 6 to 9 are implemented.

Images