Chorus system and chorus method

Abstract

The present application relates to the technical field of audio processing, and the present application provides a chorus system and a chorus method. The chorus system provided by the present application includes: at least two chorus host devices and a chorus server; the chorus host device includes an output mixing module, and the chorus server includes a mixing module; the mixing module is used to receive the input audio of each of the chorus host devices, mix the input audio to obtain mixed audio, and transmit the mixed audio to each of the output mixing modules; the output mixing module is used to receive the mixed audio output by the mixing module, and output the mixed audio to a playback device for playback. The solution provided by the present application solves the problems of severe delay and poor synchronization of Internet chorus.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular, to a chorus system and a chorus method. Background Art

[0002] Currently, there are two main chorus modes on the market, one is a stand-alone chorus mode, and the other is a chorus mode based on Internet streaming media transmission.

[0003] The stand-alone chorus mode refers to the chorus singers using one or more audio input devices connected to the same audio processing device to sing. Due to the limitations of equipment and other environmental factors, the chorus singers can only gather in a space with a short distance from each other to sing. In order to gather together, the chorus singers need to spend a lot of time, which leads to this mode greatly limiting the participation of the chorus singers.

[0004] The chorus mode based on Internet streaming can effectively eliminate this limitation, but its disadvantages are also obvious. Since the audio is transmitted through the Internet, the network equipment and paths that the data passes through are difficult to control, so the delay from the source to the destination is generally serious. Chorus requires the chorus members to sing together, which is highly sensitive to time. The delay will greatly affect the performance of the chorus. Summary of the invention

[0005] In view of the problems of severe delay and poor synchronization of the current chorus mode based on Internet streaming media transmission, the present application provides a chorus system.

[0006] In a first aspect, the chorus system provided by the present application comprises:

[0007] at least two chorus host devices and a chorus server;

[0008] The chorus host device includes an output mixing module, and the chorus server includes a mixing module;

[0009] The mixing module is used to receive input audio from each of the chorus host devices, mix the input audio to obtain mixed audio, and transmit the mixed audio to each of the output mixing modules;

[0010] The output mixing module is used to receive the mixed audio output by the mixing module, and output the mixed audio to the playing device for playing.

[0011] In an optional embodiment of the first aspect, the chorus system further includes:

[0012] An input distributor and a mixer in the mixing module that is arranged corresponding to the input distributor;

[0013] The input distributor is arranged on the chorus server;

[0014] The input distributor is arranged corresponding to each of the chorus host devices, and is used to receive the input audio of the corresponding chorus host device, and distribute the input audio to the mixers arranged corresponding to other input distributors;

[0015] The mixing module includes at least two mixers, and the mixers are arranged corresponding to the input distributors;

[0016] The mixer is used to receive and mix input audio distributed by other input distributors other than the corresponding one to obtain mixed audio.

[0017] In an optional embodiment of the first aspect, the output mixing module is further used to receive input audio from the chorus host device, and mix the input audio with the mixed audio output by the mixer to form playback audio.

[0018] In an optional embodiment of the first aspect, the chorus host device includes a master chorus host device and at least one slave chorus host device, and the input distributor includes a master input distributor receiving input audio of the master chorus host device and a slave input distributor receiving input audio of the slave chorus host device;

[0019] The mixing module includes a master mixer and a slave mixer;

[0020] The main mixer is used to receive the input audio distributed by the slave input distributor, perform mixing processing to obtain the main mixed audio, and transmit the main mixed audio to the output mixing module of the main chorus host device;

[0021] The slave mixer is used to receive the input audio distributed by the master input distributor and other slave input distributors, perform mixing processing to obtain slave mixed audio, and transmit the slave mixed audio to the output mixing module of the slave chorus host device.

[0022] In an optional embodiment of the first aspect, the chorus system further includes a recording module;

[0023] The recording module is connected to one of the output mixing modules and is used to record the playback audio.

[0024] In an optional embodiment of the first aspect, the chorus host device further includes an input mixing module for sending a plurality of input audios generated by the chorus host device to input distributors with the same number.

[0025] In an optional embodiment of the first aspect, the recording module is disposed in the chorus server.

[0026] In an optional embodiment of the first aspect, the chorus server is built into the main chorus host device.

[0027] In an optional embodiment of the first aspect, the chorus system further includes:

[0028] A signaling server, a first RTC client, and a second RTC client;

[0029] The chorus server integrates the signaling server and the first RTC client, wherein the signaling server and the first RTC client transmit SDP information and Candidate information to each other;

[0030] The chorus host device integrates the second RTC client, wherein the second RTC client and the first RTC client mutually transmit RTP protocol information, and the second RTC client and the signaling server mutually transmit SDP information and Candidate information obtained from the first RTC client;

[0031] The candidate information includes RTP / RTCP protocol information, and the input audio and mixed audio include RTP / RTCP protocol information.

[0032] In a second aspect, the present application further provides a chorus method, which is based on the chorus system provided in the first aspect, and the steps include:

[0033] When it is detected that the amount of the played audio data downloaded and cached to the local resource server meets the singing threshold, a singing start request is sent to the chorus server;

[0034] receiving singing start information fed back in response to the singing start request after the chorus server has accumulated singing start requests from at least two chorus host devices, and starting synchronous singing according to the singing start information; wherein the singing start information includes a singing start time frame;

[0035] The mixed audio is received by the output mixing module and output to a playing device for playing.

[0036] In an optional embodiment of the second aspect, after using the output mixing module to receive the mixed audio, the method further includes:

[0037] The mixed audio is mixed with the locally received input audio to form a playback audio.

[0038] In an optional embodiment of the second aspect, the starting of synchronized singing according to the singing start information includes:

[0039] Parsing the singing start information, extracting the singing start time frame, calibrating its own clock, and obtaining a calibrated singing start time frame;

[0040] According to the calibrated singing start time frame, the synchronous singing start is started.

[0041] In an optional embodiment of the second aspect, before detecting that the amount of mixed audio data downloaded and cached to the local resource server meets the singing-ready threshold, the process further includes:

[0042] It is detected that the mixed audio does not exist locally, and the mixed audio is downloaded from the resource server.

[0043] In a third aspect, the present application also provides another chorus method, which is based on the chorus system provided in the first aspect, and the steps include:

[0044] Receive and accumulate singing start requests from at least two chorus host devices;

[0045] Acquire the current time frame and generate singing start information according to the current time frame, and send the singing start information to the chorus host device;

[0046] Transmitting the mixed audio to the output mixing module, so that the mixed output audio formed after the mixed audio and the input audio received by the chorus host device are mixed and processed is output and played in the output mixing module;

[0047] The singing start request is generated by each chorus host device when it is detected that the data volume of the mixed audio downloaded and cached to the local resource server meets the singing start threshold.

[0048] In an optional embodiment of the third aspect, the start time frame is determined by a clock of the chorus server.

[0049] In an optional embodiment of the third aspect, the singing start request includes audio parameters supported by the chorus host device;

[0050] The sending of the singing start information to the chorus host device comprises:

[0051] Extracting audio parameters commonly supported by all the chorus host devices that issue a start-singing request as audio parameters of the current chorus method, adding notification information of the audio duration and the designated chorus time to each data packet received by the start-singing request to form start-singing information;

[0052] The singing start information is distributed to each of the chorus host devices.

[0053] In an optional embodiment of the third aspect, the audio parameters are stored in a data portion of the RTP / RTCP protocol information;

[0054] The chorus method further comprises:

[0055] The data portion of the RTP / RTCP protocol information is clipped.

[0056] In an optional embodiment of the third aspect, the clipping of the data portion of the RTP / RTCP protocol information includes:

[0057] Acquire a plurality of consecutive data packets and the duration corresponding to each of the data packets from the input audio;

[0058] Numbering the data packets and calculating corresponding timestamps;

[0059] The original timestamp parameters of the input audio are cropped.

[0060] In an optional embodiment of the third aspect, the clipping of the data portion of the RTP / RTCP protocol information includes:

[0061] The multiplexing parameters of the input audio corresponding to the mixed audio are trimmed; wherein the multiplexing parameters include SSRC and CSRC.

[0062] In an optional embodiment of the third aspect, the chorus method further includes:

[0063] When it is detected that the transmission delay time of the input audio and the mixed audio is lower than a preset transmission delay threshold, a transmission path optimization request is sent to the SDN controller to optimize the transmission path between the chorus host device and the chorus server.

[0064] The technical solution provided by the embodiments of the present disclosure has the following beneficial effects:

[0065] In the chorus scheme provided in the embodiment of the present application, based on the chorus system formed by two or more chorus host devices and a chorus server, a mixing module is added to the chorus server, and the mixing module is used to mix the input audio received from the chorus host device to obtain the mixed audio, and output it to the output mixing module of the chorus host device for playback. In the present application, the mixing module is used to align and mix the received input audio, and the aligned and mixed mixed audio is simultaneously sent to each chorus host device, so that the mixed audio received by each chorus host device after mixing multiple input audios is synchronized, overcoming the output delay problem caused by the network chorus of multiple chorus host devices.

[0066] Additional aspects and advantages of the present application will be partially given in the following description, which will become obvious from the following description, or will be learned through practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The above and / or additional aspects and advantages will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0068] Figure 1 A schematic diagram of a module of a chorus system provided in one embodiment of the present application;

[0069] Figure 2 A schematic diagram of a module of a chorus system provided by another embodiment of the present application;

[0070] Figure 3 A schematic diagram of a module of a chorus system provided in yet another embodiment of the present application;

[0071] Figure 4 A signaling structure diagram of a chorus system provided in an embodiment of the present application;

[0072] Figure 5 A schematic flow chart of a chorus method provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of a judgment flow for sending a start-singing request provided in an embodiment of the present application;

[0074] Figure 7 A schematic flow chart of another chorus method provided in an embodiment of the present application;

[0075] Figure 8 This is a timing diagram of the synchronous singing of the chorus method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The present application will be further described below in conjunction with the accompanying drawings and exemplary embodiments, wherein the same reference numerals in the accompanying drawings all refer to the same components. In addition, if the detailed description of the known technology is not necessary for illustrating the features of the present application, it will be omitted.

[0077] Reference Figure 1 , Figure 1 A schematic diagram of a module of a chorus system provided in one embodiment of the present application.

[0078] A chorus system provided by an embodiment of the present application includes at least two chorus host devices and a chorus server.

[0079] The chorus server includes a mixing module, and each chorus host device includes an output mixing module. The mixing module mixes the input audio received from the chorus host device and transmits the mixed audio to each chorus host device.

[0080] The output mixing module receives the mixed audio and plays it using a playing device.

[0081] In the chorus system provided in the present application, on the basis of the chorus system formed by two or more chorus host devices and a chorus server, a mixing module is added to the chorus server, and the mixing module is used to mix the input audio received from the chorus host device to obtain the mixed audio, and output it to the output mixing module of the chorus host device for playback. In the present application, the mixing module is used to align and mix the received input audio, and the aligned and mixed mixed audio is simultaneously sent to each chorus host device, so that the mixed audio received by each chorus host device after mixing multiple input audios is synchronized, thereby overcoming the output delay problem caused by the network chorus of multiple chorus host devices.

[0082] On the basis of the above embodiment, the chorus system further includes: an input distributor arranged in the chorus server. The input distributor is arranged corresponding to each chorus host device, and distributes the received input audio to the mixer arranged corresponding to other input distributors. The input audio received by the mixer is at least two input audios distributed by other input distributors other than the input distributor arranged corresponding to the mixer. The mixer mixes the multi-channel input audio to obtain mixed audio.

[0083] In the chorus system, each mixer has a corresponding input distributor, each input distributor has a corresponding chorus host device, and a chorus system includes at least two mixers and two input distributors.

[0084] Furthermore, the output mixing module in the chorus host device can also directly receive the input audio of the chorus host device, and further mix the input audio collected by itself with the mixed audio obtained by the mixer to obtain the playback audio, and output it to the playback device for playback.

[0085] like Figure 1 As shown, this embodiment includes chorus host devices A, chorus host devices B0 to chorus host devices Bn numbered A, B0 to Bn, and correspondingly, the chorus server includes input distributors A, input distributors B0 to input distributors Bn numbered A, B0 to Bn. Among them, the input distributor A is connected to the chorus host device A, the input distributor B0 is connected to the chorus host device B0, and the input distributor Bn is connected to the chorus host device Bn, and each input distributor receives the input audio collected by the chorus host device with the same number and transmits it to the mixer with a different number.

[0086] Based on the above embodiment, the multiple mixers in the mixing module are also numbered A, B0 to Bn. Each mixer is set corresponding to the input distributor with the same number, and the input distributor has the same number as the corresponding chorus host device. In this embodiment, the input audio received by the mixer is the input audio forwarded by other input distributors other than the input distributor set corresponding to it, that is, the number of each mixer is different from the number of the input distributor forwarding the input audio received by it. After the mixer mixes the input audio, the mixed audio is output to the output mixing module of the chorus host device with the same number as it.

[0087] In this embodiment, reference Figure 1 , connected to the input arrow of mixer A are input distributors B0 to Bn. After the input audios collected from input distributors B0 to Bn are mixed by mixer A, mixed audio is formed, and then output to the output mixing module in the chorus host device A connected thereto.

[0088] For the above-mentioned embodiment, after receiving the input audios distributed from the input distributor B0 to the input distributor Bn, the mixer A aligns the input audios with reference to its own clock and mixes the aligned input audios.

[0089] In this embodiment, when the chorus host device receives its own input audio collected by the output mixing module in the chorus host device A, the input audio collected by the output mixing module is aligned with the mixed audio based on the clock error between the chorus server and the chorus host device A and with reference to the timestamp of the received mixed audio, to form a playback audio that is a mixture of the original sound received by the input audio device of the chorus host device A and the mixed audio received by the chorus host devices B0 to Bn, and the audio is output to the playback device and played.

[0090] In addition, in addition to the above-mentioned embodiment, the mixer A can also be connected to all the input distributors respectively, that is, the mixer A not only receives the input audio distributed by the input distributor B0 to the input distributor Bn, but also receives the input audio received by the input distributor A, and performs audio mixing in the mixer A, and directly outputs it to the output mixing module in the chorus host device A.

[0091] The transmission and mixing principles from other mixers B0 to Bn are the same as those in the above-mentioned embodiments.

[0092] Furthermore, the plurality of chorus host devices involved in the above embodiment include a master chorus host device and at least one slave chorus host device. Figure 1To explain, the chorus host device A forms the main initiating system A, that is, the chorus is initiated by the user through the chorus host device A, and the chorus host devices B0 to Bn form the slave initiating system B0 to the slave initiating system Bn. The above-mentioned slave initiating system establishes a connection with the chorus server according to the singing request initiated by the chorus host device A, and participates in the chorus.

[0093] Correspondingly, according to the master-initiated and slave-initiated allocation of the chorus host device A and the chorus host devices B0 to Bn, the input distributors connected to the chorus host devices are respectively the master input distributor and the slave input distributor. The mixers connected to the output mixing modules of the corresponding chorus host devices are respectively the master mixer and the slave mixer. In this embodiment, the input distributor A is the master input distributor, and the other input distributors are slave input distributors. The transmission and mixing principles of the audio are consistent with those of the above-mentioned embodiments.

[0094] In this embodiment, the main mixer A receives the input audio distributed by the input distributors B0 to Bn, performs mixing processing to obtain the main mixed audio, and transmits the main mixed audio to the output mixing module of the main chorus host device;

[0095] The slave mixer is used to receive input audio distributed by the main input distributor A and other slave input distributors with different numbers, and perform mixing processing to obtain slave mixed audio, and transmit the slave mixed audio to the output mixing module of the slave chorus host device with the same number.

[0096] Reference Figure 2 , Figure 2 A schematic diagram of a module of a chorus system provided in another embodiment of the present application.

[0097] The chorus host device also includes an input mixing module. The input mixing module can collect multiple input audios, such as multiple input audios of the same live chorus, and send the multiple input audios to input distributors with the same number.

[0098] In addition, the input mixing module can also mix multiple input audios generated by the chorus host device to obtain input mixed audio and output it to the corresponding input distributor.

[0099] In this embodiment, the input mixing module can be connected to multiple audio input devices, such as microphones. The output mixing module is connected to its chorus host device, outputs the processed audio to the output audio device, outputs to the playback device and plays it, and the playback device can be a speaker.

[0100] The chorus server includes a mixing module. The input distributor is connected to the input mixing module in the chorus host device, and is used to receive input audio collected by the connected input mixing module and distribute the input audio to the mixing module.

[0101] The chorus system provided by the embodiment of the present application can obtain the input audio received by each chorus host device through the chorus server, use its input distributor to redistribute the obtained input audio, and use the mixer to mix multiple input audios, and feed the obtained mixed audio back to the corresponding chorus host device for output and playback, so that each input audio is aligned and mixed through the chorus server, and a mixed audio is formed to the corresponding chorus host device, avoiding the delay problem caused by each input audio in the transmission and processing. At the same time, it also reduces the amount of audio transmission data, reduces the load on the network, the chorus host device and the chorus server, and can better ensure the delay problem of audio data transmission, which is ultimately beneficial to improving the user experience of remote chorus.

[0102] The chorus system provided by the above embodiment can further include a recording module. In this embodiment, the recording module is connected to the output mixing module of one of the chorus host devices in the chorus system, and is used to record the mixed output audio provided by any of the above embodiments and save it. Figure 1-2 As shown, the recording module is set in the chorus server, and the recording module establishes a transmission link with the output mixing module of each chorus host device. According to the detection result of the first start-singing request, the chorus host device that initiates the first start-singing request is confirmed to be the main chorus host device, and the transmission link with its output mixing module is enabled to obtain the corresponding playback audio.

[0103] Reference Figure 3 , Figure 3 A schematic diagram of a module of a chorus system provided in another embodiment of the present application. Figure 1 or Figure 2 The difference between the embodiment of the present invention is that the chorus server is built into the chorus host device. According to the method of determining the main chorus host device as described above, the corresponding chorus server can be enabled as the server module of the main chorus host device to form a chorus and host system. The transmission and mixing principle of the audio data by the input distributor and the mixer is the same as that of the embodiment provided above.

[0104] In addition, in this embodiment, the recording module is built into the chorus host device. According to the method of determining the main chorus host device, the corresponding recording module can be enabled to record the mixed output audio, avoiding the simultaneous activation of multiple recording modules to increase the data processing burden of each chorus host device.

[0105] refer to Figure 4 , Figure 4 This is a signaling structure diagram of the chorus system provided in an embodiment of the present application.

[0106] The chorus device also includes: a signaling server, a first RTC client and a second RTC client.

[0107] The chorus server integrates the signaling server and the first RTC client, wherein the signaling server and the first RTC client transmit SDP information and Candidate information to each other;

[0108] The chorus host device integrates the second RTC client, wherein the second RTC client and the first RTC client mutually transmit RTP protocol information (Real-time Transport Protocol) as a real-time transport protocol. The second RTC client and the signaling server mutually transmit SDP information and Candidate information obtained from the first RTC client;

[0109] The candidate information includes RTP / RTCP protocol information, and the input audio and mixed audio include RTP / RTCP protocol information. RTCP (Real-time Control Protocol) is a real-time transport control protocol.

[0110] Among them, RTP protocol information (RTP, Real-time Transport Protocol). The connection of the RTP protocol information can be divided into two connections, namely, the RTP protocol information (RTP, Real-time Transport Protocol, Real-time Transport Protocol) used to transmit audio data and the RTCP protocol (Real-time Control Protocol, Real-time Transport Control Protocol) used for transmission control. The RTP protocol and the RTCP protocol are used in conjunction with each other, and there are two network connections with different ports for each audio transmission in the system. Among them, the RTP protocol is mainly used for audio data transmission. The RTCP protocol is not used to transmit audio data, but is only used to transmit the control data of the corresponding RTP protocol. Its main function is to provide quality of service feedback for the RTP protocol.

[0111] Based on a chorus system provided by any one of the above embodiments, the present application also provides a chorus method.

[0112] The execution end of the chorus method is a chorus host device.

[0113] Reference Figure 5 , Figure 5 A schematic flow chart of a chorus method provided in an embodiment of the present application.

[0114] With the chorus host device as the execution end, the present application provides a chorus method, the steps of which include:

[0115] S110, when it is detected that the amount of the played audio data downloaded and cached to the local source server meets the singing threshold, a singing start request is sent to the chorus server;

[0116] S120, receiving singing start information fed back by the chorus server in response to the singing start request after accumulating singing start requests from at least two chorus host devices, and starting synchronous singing according to the singing start information; wherein the singing start information includes a singing start time frame;

[0117] S130: Utilize the output mixing module to receive the mixed audio, and output it to a playback device for playback.

[0118] refer to Figure 5 In the above steps S110-S30, the chorus host device detects whether there is a local mixed output audio resource. If not, the chorus resource is downloaded from the resource server connected to the chorus server until the local mixed output audio cache reaches a preset singing threshold. The chorus host device can respond to the user's singing instruction and send a singing request to the chorus server. In this embodiment, the singing threshold is measured by time.

[0119] Reference Figure 6 , Figure 6 A schematic diagram of a determination process for sending a start-singing request provided in an embodiment of the present application.

[0120] exist Figure 6 In the example, the cache duration is the duration of the storage of the chorus resource. When the cache duration is less than the duration of the cached data, the cache duration is shorter than the duration of the cached data playback, and the cache is faster than the playback, the chorus host device sends a start request to the chorus server. Otherwise, the start request is not sent to the chorus server until the duration of the cached data is greater than the duration of the data to be cached, that is, the cached duration is longer than the download duration of the remaining uncached data. Otherwise, the cache is continued until the cache duration reaches 1 second.

[0121] The duration of data to be cached refers to the duration of data that has not yet been cached. Figure 6 The length of data to be cached conforms to the following formula: length of data to be cached = total data length - length of cached data. For example, if a song is 300 seconds long and 5 seconds of data has been cached, the length of data to be cached is 295 seconds.

[0122] The time to cache data is calculated based on the length of cached data and the time taken to cache data. Figure 6 The time to cache data meets the following formula: Time to cache data = total time ÷ cached data duration × (total data duration - cached data duration). For example, if a song is 300 seconds long and it takes 6 seconds to cache 5 seconds of song data, then the time to cache data = 6 ÷ 5 × (300-5) = 354 seconds.

[0123] The chorus host device sends a singing start request to the chorus server according to the singing start instruction of the user. The chorus server accumulates the singing start requests of at least two chorus host devices, responds to the singing start request, and feeds back singing start information to the chorus host device that sent the singing start request. According to the singing start information, all chorus host devices that sent the singing start request are synchronously started to start singing, that is, the chorus mode is started. When receiving the mixed audio transmitted by the mixer, each chorus host device that enters the chorus mode outputs the mixed audio.

[0124] On this basis, after receiving the mixed audio by the output mixing module, the method further includes:

[0125] The mixed audio is mixed with the locally received input audio to form a playback audio.

[0126] In this embodiment, the start-of-singing information includes a start-of-singing time frame.

[0127] In the above embodiment, starting the synchronous singing according to the singing start information in step S120 includes:

[0128] After the start singing information is obtained, the start singing information is parsed to extract the start singing time frame. In this embodiment, the start singing time frame is determined according to the clock of the chorus server. The clocks of each chorus host device are calibrated according to the start singing time frame to achieve clock synchronization of all chorus host devices and chorus servers for synchronous start singing. Synchronous start singing is started according to the calibrated start singing time frame.

[0129] In the chorus method provided above in the present application, the chorus server can send start-of-singing information to multiple chorus host devices, and calibrate and unify the start-of-singing time of each chorus host device according to the start-of-singing time frame in the start-of-singing information, thereby ensuring the synchronization of the network chorus and overcoming the problem of audio transmission asynchrony caused by transmission delay.

[0130] Reference Figure 7 , Figure 7 A schematic flow chart of another chorus method provided in an embodiment of the present application.

[0131] With the chorus server as the execution end, the present application also provides another chorus method. The steps of the chorus method include:

[0132] S210, receiving and accumulating singing start requests from at least two chorus host devices;

[0133] S220, acquiring a current time frame and generating singing start information according to the current time frame, and sending the singing start information to the chorus host device;

[0134] S230, transmitting the mixed audio to the output mixing module, so that the mixed output audio formed after the mixed audio and the input audio received by the chorus host device are mixed and processed is output and played in the output mixing module.

[0135] The singing start request is generated by each chorus host device when it is detected that the data volume of the mixed audio downloaded and cached to the local resource server meets the singing start threshold.

[0136] Reference Figure 8 , Figure 8 This is a timing diagram of the synchronous singing of the chorus method provided in an embodiment of the present application.

[0137] In the above steps S210-S230, refer to Figure 8 The synchronous singing can be divided into two stages: downloading resources and synchronous singing. In the downloading resource stage, the chorus host device downloads the chorus resource from the local resource server. The chorus resource is a mixed audio obtained by the chorus server. When the chorus resource downloaded and cached locally by the chorus host device reaches the singing threshold, the chorus host device enters the synchronous singing stage and sends a singing request to the chorus server.

[0138] When the chorus server receives at least two singing start requests from the chorus host device, it obtains the current time frame according to its own clock, forms the current singing start information according to the current time frame, and sends it to the chorus host device that issued the singing start request. According to the transmission and mixing processing of the input audio by the chorus system, the input audio from each chorus host device is mixed to obtain the mixed audio, and the mixed audio is transmitted to the output mixing module of the chorus host device.

[0139] The singing start time frame is determined by the clock of the chorus server.

[0140] The output mixing module mixes the input audio collected from the chorus host device and the mixed audio to obtain the playback audio, and outputs the playback audio to the playback device for playback.

[0141] In this embodiment, the function of the chorus server is streaming media transmission, which can be specifically divided into forwarding data without modification and mixing data and then outputting it.

[0142] The chorus method can overcome the technical problems of the existing chorus method, including: if the chorus server only forwards audio data, assuming that there are n chorus hosts performing chorus, the server needs to perform N downstream and N*(N-1) upstream, which will increase the network transmission pressure and easily lead to packet loss, delay, etc., but will not cause any loss to the audio.

[0143] Alternatively, if the chorus server mixes the audio and then outputs it, the server transmission is reduced to N downstream and N upstream, but the mixing process will not only cause delays, but also cause audio loss. In order to reduce the pressure on network transmission and reduce the delay and audio loss caused by the mixing process.

[0144] The singing start request provided above includes the audio parameters supported by the chorus host device.

[0145] The step S220 of sending the start-of-singing information to the chorus host device includes:

[0146] Extracting audio parameters commonly supported by all the chorus host devices that issue a start-singing request as audio parameters of the current chorus method, adding notification information of the audio duration and the designated chorus time to each data packet received by the start-singing request to form start-singing information;

[0147] The singing start information is distributed to each of the chorus host devices.

[0148] When the chorus host device initiates a singing start request to the chorus server, the singing start request includes audio parameters such as the audio encoding format and audio sampling rate supported by each chorus host device. After receiving the singing start requests from all chorus host devices, the chorus server will select the audio parameters used for this chorus from the audio parameters they all support, add the audio duration corresponding to each data packet and the notification information of the designated chorus time, form singing start information, and distribute it to each chorus host device that issued the singing start request.

[0149] Therefore, during the audio data transmission process, there is no need to carry audio parameters, only audio data needs to be transmitted, thereby reducing the data transmission load.

[0150] In the above embodiment, the audio parameters are stored in the data part of the RTP / RTCP protocol information.

[0151] On this basis, the chorus method further includes: cutting the data part of the RTP / RTCP protocol information.

[0152] Specifically, it may include trimming the timestamp parameter in the RTP protocol, which may specifically include the following steps:

[0153] A plurality of continuous data packets and the duration corresponding to each of the data packets are obtained from the input audio; the data packets are numbered and the corresponding timestamps are calculated; and the original timestamp parameters of the input audio are trimmed.

[0154] This embodiment is based on the fact that the audio data is continuous and the duration of each audio packet is known. Through the data packet number (Sequence Number), the timestamp corresponding to the data packet can be calculated, so the timestamp parameter in the RTP protocol can be removed. For example, each data packet corresponds to a duration of 5ms, and the numbering starts from 0, then the timestamp of the second data packet = (2-0)*5, that is, 10ms.

[0155] In addition, the multiplexing parameters of the input audio corresponding to the mixed audio are trimmed; wherein the multiplexing parameters include SSRC and CSRC.

[0156] In the chorus system, the chorus host device can generate multiple input audios, and mix them through a mixing module to form one mixed audio. Regardless of the input audio or the mixed audio, the RTP protocol information book remains unchanged, including SSRC (Synchronization source, used to identify the source of the RTP message, each RTP message can only have one SSRC), CSRC (Contributing source, used to identify the special source to which the RTP message belongs, each RTP message can contain 0 to 15 CSRCs). If there are multiple multiplexing parameters in the multiple input audios, that is, the same audio parameters, the repeated audio parameters can be trimmed. The chorus host device can distinguish the sources of the mixed input audios from the CSRC in the mixed audio.

[0157] The chorus method provided in the present application further includes: when it is detected that the transmission delay time of the input audio and the mixed audio is lower than a preset transmission delay threshold, a transmission path optimization request is sent to an SDN (Software-Defined Networking) controller so as to optimize the transmission path between the chorus host device and the chorus server. The transmission path may be a switching routing path between the chorus host device and the chorus server.

[0158] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A chorus system, characterized in that: include: at least two chorus host devices and a chorus server; The chorus host device includes an output mixing module, and the chorus server includes a mixing module; The mixing module is used to receive input audio from each of the chorus host devices, mix the input audio to obtain mixed audio, and transmit the mixed audio to each of the output mixing modules; The output mixing module is used to receive the mixed audio output by the mixing module, and output the mixed audio to the playback device for playback; The chorus system also includes: An input distributor and a mixer in the mixing module that is arranged corresponding to the input distributor; The input distributor is arranged on the chorus server; The input distributor is arranged corresponding to each of the chorus host devices, and is used to receive the input audio of the corresponding chorus host device, and distribute the input audio to the mixers arranged corresponding to other input distributors; The mixing module includes at least two mixers, and the mixers are arranged corresponding to the input distributors; The mixer is used to receive and mix input audio distributed by other input distributors other than the corresponding one to obtain mixed audio.

2. The chorus system according to claim 1, characterized in that: The output mixing module is also used to receive the input audio of the chorus host device, and mix the input audio with the mixed audio output by the mixer to form the playback audio.

3. The chorus system according to claim 1, characterized in that: The chorus host device includes a master chorus host device and at least one slave chorus host device, and the input distributor includes a master input distributor receiving input audio of the master chorus host device and a slave input distributor receiving input audio of the slave chorus host device; The mixing module includes a master mixer and a slave mixer; The main mixer is used to receive the input audio distributed by the slave input distributor, perform mixing processing to obtain the main mixed audio, and transmit the main mixed audio to the output mixing module of the main chorus host device; The slave mixer is used to receive the input audio distributed by the master input distributor and other slave input distributors, perform mixing processing to obtain slave mixed audio, and transmit the slave mixed audio to the output mixing module of the slave chorus host device.

4. The chorus system according to claim 2, characterized in that: Also includes a recording module; The recording module is connected to one of the output mixing modules and is used to record the playback audio.

5. The chorus system according to claim 1, characterized in that: The chorus host device further comprises an input mixing module for sending a plurality of input audios generated by the chorus host device to input distributors with the same number.

6. The chorus system according to claim 4, characterized in that: The recording module is arranged in the chorus server.

7. The chorus system according to claim 3, characterized in that: The chorus server is built in the main chorus host device.

8. The chorus system according to claim 1, characterized in that: Also includes: A signaling server, a first RTC client, and a second RTC client; The chorus server integrates the signaling server and the first RTC client, wherein the signaling server and the first RTC client transmit SDP information and Candidate information to each other; The chorus host device integrates the second RTC client, wherein the second RTC client and the first RTC client mutually transmit RTP protocol information, and the second RTC client and the signaling server mutually transmit SDP information and Candidate information obtained from the first RTC client; The candidate information includes RTP / RTCP protocol information, and the input audio and mixed audio include RTP / RTCP protocol information.

9. A chorus method, characterized in that: The chorus system according to any one of claims 1 to 8 is implemented, and the steps include: When it is detected that the amount of the played audio data downloaded and cached to the local resource server meets the singing threshold, a singing start request is sent to the chorus server; receiving singing start information fed back in response to the singing start request after the chorus server has accumulated singing start requests from at least two chorus host devices, and starting synchronous singing according to the singing start information; wherein the singing start information includes a singing start time frame; The output mixing module is used to receive the mixed audio and output it to the playing device for playing.

10. The chorus method according to claim 9, characterized in that: After receiving the mixed audio by the output mixing module, the method further includes: The mixed audio is mixed with the locally received input audio to form a playback audio.

11. The chorus method according to claim 9, characterized in that: The step of starting the synchronous singing according to the singing start information includes: Parsing the singing start information, extracting the singing start time frame, calibrating its own clock, and obtaining a calibrated singing start time frame; According to the calibrated singing start time frame, the synchronous singing start is started.

12. The chorus method according to claim 9, characterized in that: Before the detection that the amount of mixed audio data downloaded and cached to the local resource server meets the threshold for singing, the method further includes: It is detected that the mixed audio does not exist locally, and the mixed audio is downloaded from the resource server.

13. A chorus method, characterized in that: The chorus system according to any one of claims 1 to 8 is implemented, and the steps include: Receive and accumulate singing start requests from at least two chorus host devices; Acquire the current time frame and generate singing start information according to the current time frame, and send the singing start information to the chorus host device; Transmitting the mixed audio to the output mixing module, so that the mixed output audio formed after the mixed audio and the input audio received by the chorus host device are mixed and processed is output and played in the output mixing module; The singing start request is generated by each chorus host device when it is detected that the data volume of the mixed audio downloaded and cached to the local resource server meets the singing start threshold.

14. The chorus method according to claim 13, characterized in that: The singing start information includes a singing start time frame; The singing start time frame is determined by the clock of the chorus server.

15. The chorus method according to claim 13, characterized in that: The singing start request includes audio parameters supported by the chorus host device; The sending of the singing start information to the chorus host device comprises: Extracting audio parameters commonly supported by all the chorus host devices that issue a start-singing request as audio parameters of the current chorus method, adding notification information of the audio duration and the designated chorus time to each data packet received by the start-singing request to form start-singing information; The singing start information is distributed to each of the chorus host devices.

16. The chorus method according to claim 15, characterized in that: The audio parameters are stored in the data part of the RTP / RTCP protocol information; The chorus method further comprises: The data portion of the RTP / RTCP protocol information is clipped.

17. The chorus method according to claim 16, characterized in that: The step of cutting the data portion of the RTP / RTCP protocol information comprises: Acquire a plurality of consecutive data packets and the duration corresponding to each of the data packets from the input audio; Numbering the data packets and calculating corresponding timestamps; The original timestamp parameters of the input audio are cropped.

18. The chorus method according to claim 16, characterized in that: The step of cutting the data portion of the RTP / RTCP protocol information comprises: The multiplexing parameters of the input audio corresponding to the mixed audio are trimmed; wherein the multiplexing parameters include SSRC and CSRC.

19. The chorus method according to claim 13, characterized in that: Also includes: When it is detected that the transmission delay time of the input audio and the mixed audio is lower than a preset transmission delay threshold, a transmission path optimization request is sent to the SDN controller to optimize the transmission path between the chorus host device and the chorus server.

Images