A system and method for supporting quick expansion of multi-channel audio modules

Abstract

The application discloses a system and method supporting quick expansion of multi-channel audio modules, and belongs to the technical field of multi-channel audio processing. A render thread module is connected with an audio distribution module B, and is used for asking the audio distribution module B for audio data. At this time, a mixing module A mixes audio data acquired from an audio buffer of each single-channel audio stream unit into one-way audio data, and sends the one-way audio data to the audio distribution module B. The audio distribution module B sends the mixed audio data to a mixing module B. Meanwhile, microphone audio data in the audio buffer B is sent to the mixing module B. The mixing module B mixes the mixed audio data and the microphone audio data, and then performs sampling rate adjustment and volume adjustment. Finally, the mixed audio data is distributed through an audio distribution module C. The system can quickly add audio sources, switch any one or several audio sources in multi-channels, and quickly analyze a specified one or several audio sources.

Description

Technical Field

[0001] This invention relates to the field of multi-channel audio processing technology, and more specifically to a system and method that supports rapid expansion of multi-channel audio modules. Background Technology

[0002] Traditional audio processing solutions include hardware and software approaches. Hardware solutions rely on physical wiring and dedicated hardware. After wiring is completed, adding or removing audio sources or switching channels requires readjusting the hardware connections, making configuration cumbersome and inflexible. Software solutions, on the other hand, are cumbersome to implement in handling complex audio streams and lack flexibility. They cannot quickly add audio sources, switch any one or more audio sources from multiple channels, or quickly analyze specific audio sources. Therefore, they fail to meet the demands for flexibility and scalability in today's complex audio processing scenarios.

[0003] Based on this, the present invention designs a system and method that supports rapid expansion of multi-channel audio modules to solve the above problems. Summary of the Invention

[0004] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a system and method that supports rapid expansion of multi-channel audio modules.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A system that supports rapid expansion of multiple audio modules includes:

[0007] Several sets of single-channel audio stream units are used to independently process the audio data of each input channel;

[0008] Mixing module A is used to mix the audio data obtained from the audio buffers of each single audio stream unit into one audio stream; mixing module A is connected to audio distribution module B, and audio distribution module B is connected to mixing module B.

[0009] Audio distribution module B is used to send the mixed audio data to playback channel B, audio debugging module and mixing module B;

[0010] The microphone, as the audio data acquisition hardware, collects microphone audio data; the microphone is connected to the volume control module C.

[0011] Volume control module C is used to adjust the volume of microphone audio data; volume control module C is connected to audio buffer B.

[0012] Audio buffer B is used to temporarily store microphone audio data; audio buffer B is connected to mixing module B.

[0013] Mixing module B is used to mix the mixed audio data obtained from audio distribution module B and the microphone audio data obtained from audio buffer B into one audio data stream.

[0014] Sampling rate conversion module B is used to adjust the sampling rate of the mixed audio data and convert audio data of various sampling rates into audio data that can be played normally by the device.

[0015] Volume control module B is used to adjust the volume of the mixed audio data;

[0016] The audio distribution module C is used to send the mixed audio data, either encoded by the audio encoding module or unencoded, to the required channels.

[0017] The Render thread module is connected to the audio distribution module B and is used to request audio data from the audio distribution module B. At this time, the mixing module A mixes the audio data obtained from the audio buffer of each single audio stream unit into one audio data and sends it to the audio distribution module B. The audio distribution module B sends the mixed audio data to the mixing module B.

[0018] The Capture thread module is used to collect audio data in real time through the microphone, and after adjustment by the volume control module C, transmit the audio data to the audio buffer B; and when the Render thread module starts, it pulls the microphone audio data in the audio buffer B and sends it to the mixing module B.

[0019] Standard audio interface.

[0020] Furthermore, the single-channel audio stream unit includes an audio decoding module, a sampling rate conversion module A, a volume control module A, an audio distribution module A, and an audio buffer A connected in sequence; or the single-channel audio stream unit includes a sampling rate conversion module C, a volume control module D, and an audio distribution module D connected in sequence, as well as an audio buffer C and an audio buffer C1 connected to the audio distribution module D.

[0021] Furthermore, the audio decoding module is used to decode audio data transmitted over the network in AAC or OPUS format into raw audio data.

[0022] Sampling rate conversion module A is used to adjust the sampling rate of audio data;

[0023] Volume control module A is used to adjust the volume of audio data;

[0024] Audio distribution module A is used to send audio data to audio buffer A;

[0025] Audio buffer A is used to temporarily store audio data; audio buffer A is connected to the mixing module.

[0026] Furthermore, the audio decoding module uses an AAC decoding module or an OPUS decoding module.

[0027] Furthermore, the sampling rate conversion module C is used to adjust the sampling rate of the audio data;

[0028] Volume control module D is used to adjust the volume of audio data;

[0029] The audio distribution module D is used to send audio data to audio buffer C and audio buffer C1, so that downstream modules (such as mixing module A) can "retrieve" audio data from the buffers "on demand".

[0030] Playback channel A, acting as a channel proxy for the audio data output hardware, is used to send the audio data in audio buffer C1 to the selected output hardware, such as a speaker.

[0031] Audio buffer C is used to temporarily store audio data; audio buffer C is connected to the mixing module.

[0032] Furthermore, the audio distribution module B is connected to the playback channel B and the audio debugging module;

[0033] Playback channel B acts as a channel proxy for the audio data output hardware, used to send audio data to selected output hardware, such as speakers;

[0034] The audio debugging module is used to debug audio.

[0035] Furthermore, the audio encoding module is used to encode the mixed audio data into AAC or OPUS format and send it over the network.

[0036] Furthermore, the audio encoding module uses either an AAC encoding module or an OPUS encoding module.

[0037] Furthermore, the standard audio interface includes a sink interface for pushing audio data out and a source interface for retrieving audio data from the upper level.

[0038] To better achieve the objectives of this invention, this invention also provides a method for supporting rapid expansion of multi-channel audio modules, comprising the following steps:

[0039] Step 1: The Render thread module starts and requests audio data from the audio distribution module B. At this time, the mixing module A mixes the audio data obtained from the audio buffers of each single audio stream unit into one audio stream and sends it to the audio distribution module B. The audio distribution module B sends the mixed audio data to the mixing module B. Simultaneously, it pulls microphone audio data from the audio buffer B and sends it to the mixing module B. The mixing module B mixes the mixed audio data and the microphone audio data and sends it to the sampling rate conversion module B. The sampling rate conversion module B adjusts the sampling rate of the mixed audio data, converting audio data of various sampling rates into audio data that can be played normally by the device. Then, the volume is adjusted by the volume control module B. Finally, the audio distribution module C encodes the mixed audio data or sends it to each required channel, either encoded or unencoded.

[0040] Step 2: When adding one or more audio streams, connect the newly added single-channel audio stream processing unit to mixing module A;

[0041] Step 3: When playing the audio stream of the specified channel, add a pipe to the audio distribution module D and point it to the playback channel A, while unbinding the playback channel A from other audio distribution modules.

[0042] Step 4: When excluding a certain audio stream, disconnect the single audio stream processing unit from the mixing module A;

[0043] Step 5: When debugging a certain audio stream, add an audio debugging module and connect it to the audio distribution module for that stream. When debugging a certain stream or mixing module, add an audio debugging module and connect it to audio distribution module B.

[0044] Compared with the prior art, the advantages of this invention are: 1. Any audio stream can be quickly integrated into the audio processing system;

[0045] 2. It can obtain audio stream data from any node in the pipeline from the audio processing system, including single audio streams, mixed audio streams, etc.

[0046] 3. It allows switching between any one or several audio sources in multiple channels, and quickly debugging and analyzing a specified one or several audio sources. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0048] Figure 1 This is a flowchart of a method for supporting rapid expansion of multi-channel audio modules according to the present invention.

[0049] Figure 2 This diagram illustrates the normal flow of audio stream within a pipe and the method for obtaining the audio stream from any node within the pipe.

[0050] Figure 3 This diagram illustrates a method for quickly analyzing one or more audio channels when switching between multiple channels of any audio source. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0052] Example 1: Please refer to the appendix of the instruction manual. Figures 1-3 A system that supports rapid expansion of multiple audio modules includes:

[0053] Several sets of single-channel audio stream units are used to independently process the audio data of each input channel;

[0054] The single-channel audio stream unit includes an audio decoding module, a sampling rate conversion module A, a volume control module A, an audio distribution module A, and an audio buffer A connected in sequence.

[0055] The audio decoding module is used to decode audio data transmitted over the network in formats such as AAC or OPUS into raw audio data (PCM format).

[0056] The audio decoding module can be an AAC decoding module or an OPUS decoding module, etc.

[0057] Sampling rate conversion module A is used to adjust the sampling rate of audio data and convert audio data of various sampling rates into audio data that can be played normally by the device.

[0058] Volume control module A is used to adjust the volume of audio data;

[0059] Audio distribution module A is used to send audio data to audio buffer A, so that downstream modules (such as mixing module A) can "retrieve" audio data from the buffer "on demand";

[0060] Audio buffer A is used to temporarily store audio data, waiting for the Render thread module to start; audio buffer A is connected to the mixing module.

[0061] Alternatively, the single-channel audio stream unit may include a sampling rate conversion module C, a volume control module D, and an audio distribution module D connected in sequence, as well as an audio buffer C and an audio buffer C1 connected to the audio distribution module D.

[0062] The sampling rate conversion module C is used to adjust the sampling rate of audio data and convert audio data of various sampling rates into audio data that can be played normally by the device.

[0063] Volume control module D is used to adjust the volume of audio data;

[0064] The audio distribution module D is used to send audio data to audio buffer C and audio buffer C1, so that downstream modules (such as mixing module A) can "retrieve" audio data from the buffers "on demand".

[0065] Playback channel A, acting as a channel proxy for the audio data output hardware, is used to send the audio data in audio buffer C1 to the selected output hardware, such as a speaker.

[0066] Audio buffer C is used to temporarily store audio data, waiting for the Render thread module to start; audio buffer C is connected to the mixing module.

[0067] Mixing module A is used to mix the audio data obtained from the audio buffers (audio buffer A or audio buffer C) of each single audio stream unit into one audio stream.

[0068] Mixing module A is connected to audio distribution module B, and audio distribution module B is connected to playback channel B, audio debugging module, and mixing module B.

[0069] Audio distribution module B is used to send the mixed audio data to playback channel B, audio debugging module and mixing module B;

[0070] Playback channel B acts as a channel proxy for the audio data output hardware, used to send audio data to selected output hardware, such as speakers;

[0071] The audio debugging module is used to debug audio, such as quality verification and parameter adjustment, to ensure audio balance and noise-free audio.

[0072] The microphone, as the audio data acquisition hardware, collects microphone audio data; the microphone is connected to the volume control module C.

[0073] Volume control module C is used to adjust the volume of microphone audio data; volume control module C is connected to audio buffer B.

[0074] Audio buffer B is used to temporarily store microphone audio data, waiting for the Capture thread module to start; audio buffer B is connected to mixing module B.

[0075] Mixing module B is used to mix the mixed audio data obtained from audio distribution module B and the microphone audio data obtained from audio buffer B into one audio data stream.

[0076] Sampling rate conversion module B is used to adjust the sampling rate of the mixed audio data and convert audio data of various sampling rates into audio data that can be played normally by the device.

[0077] Volume control module B is used to adjust the volume of the mixed audio data;

[0078] The audio distribution module C is used to send the mixed audio data, either encoded by the audio encoding module or unencoded, to the required channels.

[0079] The audio encoding module is used to encode the mixed audio data (PCM format) into audio data in formats such as AAC or OPUS and send it to the network;

[0080] The audio encoding module can be an AAC encoding module or an OPUS encoding module, etc.

[0081] The Capture thread module is used to collect audio data in real time through the microphone, and after adjustment by the volume control module C, transmit the audio data to the audio buffer B; and when the Render thread module starts, it pulls the microphone audio data in the audio buffer B and sends it to the mixing module B.

[0082] The Render thread module, connected to the audio distribution module B, requests audio data from the audio distribution module B. At this time, the mixing module A mixes the audio data obtained from the audio buffers (audio buffer A or audio buffer C) of each single audio stream unit into one audio stream and sends it to the audio distribution module B. The audio distribution module B sends the mixed audio data back to the mixing module B. Simultaneously, it pulls microphone audio data from audio buffer B and sends it to the mixing module B. The mixing module B mixes the mixed audio data and microphone audio data, adjusts the sampling rate and volume, and finally distributes it to the required channels via the audio distribution module C.

[0083] The standard audio interface includes a sink interface for pushing audio data out and a source interface for obtaining audio data from the upper level.

[0084] Define unified audio input / output interfaces: sink and source. The sink interface provides the putFrame method for pushing audio data out, and the source interface provides the getFrame method for retrieving audio data from its parent source. Figure 1 As shown, some modules only support sink or source, while some templates support conversion between sink and source.

[0085] When one or more audio streams need to be added to the current system, simply connect the newly added single audio stream processing unit to the mixing module A to receive AAC, OPUS, or raw audio data (PCM format).

[0086] When it is necessary to play the audio stream of a specific channel, as shown in the single-channel audio processing unit 3, a pipe is added to the audio distribution module D and pointed to the playback channel A, while the binding of the playback channel A with other audio distribution modules is removed.

[0087] When it is necessary to exclude a certain audio stream, simply disconnect the single audio stream processing unit from the mixing module A, and the audio stream will be removed from the mixing module A.

[0088] When debugging a certain audio stream is required, add an audio debugging module and connect it to the audio distribution module (audio distribution module A or audio distribution module D). When debugging a certain channel or mixing module is required, add an audio debugging module and connect it to audio distribution module B.

[0089] The present invention can achieve the following technical effects:

[0090] (1) Any audio stream can be quickly integrated into the audio processing system;

[0091] (2) Audio stream data from any node in the pipeline can be obtained from the audio processing system, including single audio streams, mixed audio streams, etc.;

[0092] refer to Figure 2 Scenario 1: The audio stream within the pipe flows normally (without acquiring audio stream data within the pipe), and the stream from volume control module D is directly sent to audio buffer C. Scenario 2: When it is necessary to acquire the audio stream from any node within the pipe, an audio distribution module D is added to intercept the stream from the audio control module and distribute it to audio buffer C1, where it is then processed.

[0093] (3) It can switch any one or several audio sources in the multi-channel and quickly debug and analyze a specified one or several audio sources.

[0094] refer to Figure 3When you need to switch between any audio source from multiple channels, simply add or remove the corresponding audio buffer in mixing module A by calling AddMixerItem or RemoveMixerItem as shown in scenario one of the diagram. If you need to quickly analyze one or more specific audio sources, you can insert audio distribution module B after mixing module A as shown in scenario two, which will intercept and distribute one audio source to the audio debugging module.

[0095] Example 2: Please refer to the accompanying drawings in the instruction manual. Figure 1 A method for rapidly expanding multi-channel audio modules includes the following steps:

[0096] Step 1: The Render thread module starts and requests audio data from the audio distribution module B. At this time, the mixing module A mixes the audio data obtained from the audio buffers (audio buffer A or audio buffer C) of each single audio stream unit into one audio stream and sends it to the audio distribution module B. The audio distribution module B sends the mixed audio data to the mixing module B. Simultaneously, it pulls microphone audio data from audio buffer B and sends it to the mixing module B. The mixing module B mixes the mixed audio data and microphone audio data and sends it to the sampling rate conversion module B. The sampling rate conversion module B adjusts the sampling rate of the mixed audio data, converting audio data of various sampling rates into audio data that can be played normally by the device. Then, the volume is adjusted by the volume control module B. Finally, the audio distribution module C encodes the mixed audio data or sends it to each required channel, either directly or without encoding.

[0097] Step 2: When it is necessary to add one or more audio streams to the current system, connect the newly added single audio stream processing unit to the mixing module A (this can be done automatically or by manually triggering a business scenario), and AAC, OPUS or raw audio data (PCM format) can be fed in.

[0098] Step 3: When it is necessary to play the audio stream of a specific channel, as shown in the single-channel audio processing unit 3, add a pipe to the audio distribution module D and point it to the playback channel A. At the same time, unbind the playback channel A from other audio distribution modules (business-driven, such as receiving the audio and video stream of the group leader as a group, at this time only the group leader's audio stream is played, the source is pointed to the playback channel A, and the audio stream of the playback channel B is unbound, so that only the playback channel A has sound output).

[0099] Step 4: When it is necessary to exclude a certain audio stream, disconnect the single audio stream processing unit from the mixing module A. The audio stream can then be removed from the mixing module A (the connection will be automatically disconnected when an audio source goes offline, or you can manually trigger binding and unbinding by operating the source).

[0100] Step 5: When debugging a certain audio stream, add an audio debugging module to connect with the audio distribution module (audio distribution module A or audio distribution module D). When debugging a certain channel or mixing module, add an audio debugging module to connect with audio distribution module B. (When the developer is debugging the audio stream on the debugging page, clicking the debugging function will automatically trigger the connection between the audio distribution module and the audio source, and will distribute an audio source to the debugging file.)

[0101] The present invention can achieve the following technical effects:

[0102] (1) Any audio stream can be quickly integrated into the audio processing system;

[0103] (2) Audio stream data from any node in the pipeline can be obtained from the audio processing system, including single audio streams, mixed audio streams, etc.;

[0104] (3) It can switch any one or several audio sources in the multi-channel and quickly debug and analyze a specified one or several audio sources.

[0105] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A system that supports rapid expansion of multiple audio modules, characterized in that, include: Several sets of single-channel audio stream units are used to independently process the audio data of each input channel; Mixing module A is used to mix the audio data obtained from the audio buffers of each single audio stream unit into a single audio stream. Mixing module A is connected to audio distribution module B, and audio distribution module B is connected to mixing module B; Audio distribution module B is used to send the mixed audio data to playback channel B, audio debugging module and mixing module B; A microphone, as a hardware device for acquiring audio data, collects microphone audio data. The microphone is connected to the volume control module C; Volume control module C is used to adjust the volume of microphone audio data; volume control module C is connected to audio buffer B. Audio buffer B is used to temporarily store microphone audio data; Audio buffer B is connected to mixing module B; Mixing module B is used to mix the mixed audio data obtained from audio distribution module B and the microphone audio data obtained from audio buffer B into one audio data stream. Sampling rate conversion module B is used to adjust the sampling rate of the mixed audio data and convert audio data of various sampling rates into audio data that can be played normally by the device. Volume control module B is used to adjust the volume of the mixed audio data; The audio distribution module C is used to send the mixed audio data, either encoded by the audio encoding module or unencoded, to the required channels. The Render thread module is connected to the audio distribution module B and is used to request audio data from the audio distribution module B. At this time, the mixing module A mixes the audio data obtained from the audio buffer of each single audio stream unit into one audio data and sends it to the audio distribution module B. The audio distribution module B sends the mixed audio data to the mixing module B. The Capture thread module is used to collect audio data in real time through the microphone, and after adjustment by the volume control module C, transmit the audio data to the audio buffer B; and when the Render thread module starts, it pulls the microphone audio data in the audio buffer B and sends it to the mixing module B. Standard audio interface.

2. The system for rapidly expanding multi-channel audio modules according to claim 1, characterized in that, The single-channel audio stream unit includes an audio decoding module, a sampling rate conversion module A, a volume control module A, an audio distribution module A, and an audio buffer A connected in sequence; or the single-channel audio stream unit includes a sampling rate conversion module C, a volume control module D, and an audio distribution module D connected in sequence, as well as an audio buffer C and an audio buffer C1 connected to the audio distribution module D.

3. The system for rapidly expanding multi-channel audio modules according to claim 2, characterized in that, The audio decoding module is used to decode audio data encoded in AAC or OPUS format transmitted over the network into raw audio data. Sampling rate conversion module A is used to adjust the sampling rate of audio data; Volume control module A is used to adjust the volume of audio data; Audio distribution module A is used to send audio data to audio buffer A; Audio buffer A is used to temporarily store audio data; audio buffer A is connected to the mixing module.

4. The system for rapidly expanding multi-channel audio modules according to claim 2 or 3, characterized in that, The audio decoding module uses either an AAC decoding module or an OPUS decoding module.

5. The system for rapidly expanding multi-channel audio modules according to claim 3, characterized in that, Sampling rate conversion module C is used to adjust the sampling rate of audio data; Volume control module D is used to adjust the volume of audio data; Audio distribution module D is used to send audio data to audio buffer C and audio buffer C1; Playback channel A, acting as a channel proxy for the audio data output hardware, is used to send the audio data in audio buffer C1 to the selected output hardware. Audio buffer C is used to temporarily store audio data; audio buffer C is connected to the mixing module.

6. The system for rapidly expanding multi-channel audio modules according to claim 5, characterized in that, Audio distribution module B is connected to playback channel B and audio debugging module; Playback channel B acts as a channel proxy for the audio data output hardware, used to send audio data to the selected output hardware; The audio debugging module is used to debug audio.

7. The system for rapidly expanding multi-channel audio modules according to claim 1, characterized in that, The audio encoding module is used to encode the mixed audio data into AAC or OPUS format and send it over the network.

8. The system for rapidly expanding multi-channel audio modules according to claim 7, characterized in that, The audio encoding module uses either AAC or OPUS encoding.

9. The system for rapidly expanding multi-channel audio modules according to claim 6, characterized in that, The standard audio interface includes a sink interface for pushing audio data and a source interface for retrieving audio data from the parent device.

10. A method for rapidly expanding multi-channel audio modules, characterized in that, Includes the following steps: Step 1: The Render thread module starts and requests audio data from the audio distribution module B. At this time, the mixing module A mixes the audio data obtained from the audio buffers of each single audio stream unit into one audio stream and sends it to the audio distribution module B. The audio distribution module B sends the mixed audio data to the mixing module B. Simultaneously, it pulls microphone audio data from the audio buffer B and sends it to the mixing module B. The mixing module B mixes the mixed audio data and the microphone audio data and sends it to the sampling rate conversion module B. The sampling rate conversion module B adjusts the sampling rate of the mixed audio data, converting audio data of various sampling rates into audio data that the device can play normally. Then, the volume control module B adjusts the volume. Finally, the audio distribution module C encodes the mixed audio data or sends it to each required channel, either directly or without encoding. Step 2: When adding one or more audio streams, connect the newly added single-channel audio stream processing unit to mixing module A; Step 3: When playing the audio stream of the specified channel, add a pipe to the audio distribution module D and point it to the playback channel A, while unbinding the playback channel A from other audio distribution modules. Step 4: When excluding a certain audio stream, disconnect the single audio stream processing unit from the mixing module A; Step 5: When debugging a certain audio stream, add an audio debugging module and connect it to the audio distribution module for that stream. When debugging a certain stream or mixing module, add an audio debugging module and connect it to audio distribution module B.

Images