Audio mixing processing method and device, chip, bluetooth device and mixing system

Abstract

The application provides an audio mixing processing method and device, a chip, a Bluetooth device and a mixing system. The method comprises the following steps: S100, when it is detected that the mixing audio data of the second buffer is underloaded; then, it is detected whether the first PCM frame data and the second PCM frame data are underloaded; S200a, when neither of them is underloaded, the first PCM frame data and the second PCM frame data are added; S200b, when at least one of them is underloaded, the underloaded first PCM frame data and / or the second PCM frame data are added with mute frame data, and the underloaded first PCM frame data or the second PCM frame data are added with the first PCM frame data or the second PCM frame data output from the first buffer. The application can avoid the situation that the audio listening effect is poor due to the interference or transmission blockage of one audio stream.

Description

Technical Field

[0001] This invention relates to the field of audio data processing, and more particularly to an audio mixing processing method and apparatus, chip, Bluetooth device and mixing system. Background Technology

[0002] In today's era, with the continuous development of internet technology, the internet has entered the "live streaming era" and the "audio and video content era." Live streaming and audio and video content have rapidly developed into a new internet cultural industry. Simultaneously, more and more people are participating in live streaming or content creation to generate marketing or platform revenue, in both single-person and multi-person formats. Most connection solutions in multi-person formats use BLE (Bluetooth Low Energy) transmission for wireless microphones. This involves one host device acting as a receiver, connecting multiple wireless microphone transmitters. For example, two people might use wireless microphones to collect sound, creating two audio streams that are transmitted via Bluetooth to the receivers. The receivers then receive both audio streams. When the sampling rates are consistent, the audio mixing device can linearly superimpose the sampled data of each corresponding signal to form a mixed audio stream, which is then output to a DA or USB interface. However, one drawback of this traditional asynchronous transmission scheme is that the two audio streams are asynchronously added during mixing. If one audio stream is interfered with or blocked, the other stream cannot be detected and is easily blocked as well, resulting in poor audio reception at the receiving terminal. Summary of the Invention

[0003] Based on the above situation, the main objective of this invention is to provide an audio mixing processing method, apparatus, chip, Bluetooth device, and mixing system that can prevent poor audio reception at the receiving terminal due to interference or transmission blockage of one audio stream.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] This invention provides an audio mixing processing method, wherein a first buffer caches first channel PCM frame data and second channel PCM frame data, and a second buffer caches mixed audio data for digital-to-analog conversion to output analog audio signals;

[0006] The first PCM frame data and the second PCM frame data are obtained by decompressing the continuously received first audio data and second audio data; the mixed audio data is the data obtained after completing audio mixing.

[0007] The method includes:

[0008] Step S100: When it is detected that the mixed audio data in the second buffer is underloaded, then it is detected whether the current first PCM frame data and the second PCM frame data are underloaded.

[0009] Step S200a: When it is detected that neither the first channel PCM frame data nor the second channel PCM frame data is underloaded, the first channel PCM frame data and the second channel PCM frame data output from the first buffer are summed to obtain the mixed audio data, and the mixed audio data is cached in the second buffer;

[0010] Step S200b: When it is detected that at least one of the current first PCM frame data and the second PCM frame data is underloaded, the underloaded first PCM frame data and / or the underloaded second PCM frame data are summed with silent frame data, and the ununderloaded first PCM frame data or the second PCM frame data are summed with the first PCM frame data or the second PCM frame data output from the first buffer, and the mixed audio data obtained by the summing operation is cached in the second buffer.

[0011] Preferably, the method further includes: step Sb2: when it is detected that the buffer value of the mixed audio data in the second buffer is lower than a preset underload trigger value, it is determined that the mixed audio data in the second buffer has underloaded.

[0012] Preferably, the method further includes step Sb1, which is performed before step Sb2: when an interrupt signal is detected, it is detected whether the buffer value of the mixed audio data in the second buffer is lower than a preset underload trigger value.

[0013] Preferably, the interrupt signal is generated periodically or custom-generated.

[0014] Preferably, the interrupt signal is triggered when the amount of mixed audio data in the second buffer is lower than the first threshold.

[0015] Preferably, the method further includes: step Sa2: when it is detected that the amount of data of the first PCM frame data and / or the second PCM frame data in the current first buffer does not exceed the second threshold; then it is determined that the first PCM frame data and / or the second PCM frame data in the current first buffer are underloaded.

[0016] Preferably, the method further includes step Sa1, which is performed before step Sa2: when it is detected that at least one of the current first PCM frame data and the second PCM frame data has not been mixed, it is detected whether the amount of data of the first PCM frame data and / or the second PCM frame data in the current first buffer that has not been mixed has not exceeded a second threshold.

[0017] Preferably, the first PCM frame data and the second PCM frame data are equipped with an underload flag;

[0018] The underload flag includes a relatively independent step of setting the underload flag and a step of clearing the underload flag; in the step of setting the underload flag, the underload flag is set to underload, and in the step of clearing the underload flag, the underload flag is set to not underload.

[0019] The underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded.

[0020] Preferably, the first PCM frame data and the second PCM frame data are equipped with an underload flag;

[0021] In step Sa2 of the method, the underload flag corresponding to the first PCM frame data and / or the second PCM frame data whose data volume does not exceed the second threshold is set to underload;

[0022] The underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded.

[0023] Preferably, the method further includes: step S300: when the amount of data of the first channel PCM frame data and / or the second channel PCM frame data in the first buffer detected in real time exceeds a third threshold; then the corresponding underload flag is set to non-underload;

[0024] The step of setting the underload flag to underload is performed independently of the step of setting the underload flag to non-underload.

[0025] Preferably, the DAC module used for digital-to-analog conversion interrupts to determine whether the mixed audio data is underloaded; the method is applicable to the low-power Bluetooth multi-connection asynchronous transmission mode.

[0026] The present invention also provides an audio mixing processing apparatus, the audio mixing processing apparatus comprising:

[0027] The receiving module is used to continuously receive the first audio data and the second audio data, buffer them into the first buffer, and decompress them to obtain the first PCM frame data and the second PCM frame data.

[0028] The mixing module is used to perform audio mixing to obtain mixed audio data and cache the mixed audio data in a second buffer;

[0029] The DAC module is used to perform digital-to-analog conversion on the mixed audio data output from the second buffer and output an analog audio signal;

[0030] The first underload detection module is used to detect whether the mixed audio data is underloaded;

[0031] The second underload detection module is used to detect whether the current first channel PCM frame data and the second channel PCM frame data are underloaded.

[0032] When the first underload detection module detects an underload in the mixed audio data, the second underload detection module initiates a detection process.

[0033] When the second detection module detects that neither the first channel PCM frame data nor the second channel PCM frame data is underloaded, the mixing module performs a summation operation on the first channel PCM frame data and the second channel PCM frame data output from the first buffer to obtain the mixed audio data.

[0034] When the second detection module detects that at least one of the current first PCM frame data and the second PCM frame data is underloaded, in the mixing module, the underloaded first PCM frame data and / or the underloaded second PCM frame data are summed with silent frame data, and the non-underloaded first PCM frame data or the second PCM frame data are summed with the first PCM frame data or the second PCM frame data output from the first buffer.

[0035] Preferably, the first underload detection module includes: a buffer value detection module, used to detect the buffer value of the mixed audio data in the second buffer, and when the buffer value is lower than a preset underload trigger value, it is determined that the mixed audio data in the second buffer has underloaded.

[0036] Preferably, the first underload detection module further includes: an interrupt module, used to trigger the generation of an interrupt signal, and when the interrupt signal is generated, the buffer value detection module starts detection; the interrupt module is the interrupt of the DAC module.

[0037] Preferably, the first underload detection device includes a counter, which triggers the interrupt module to generate the interrupt signal when it detects that the amount of mixed audio data in the second buffer is lower than a preset first threshold.

[0038] Preferably, the second underload detection module includes: a data volume detection module, used to detect the data volume of the first channel PCM frame data and / or the second channel PCM frame data in the current first buffer, and when the data volume does not exceed a second threshold, it is determined that the first channel PCM frame data and / or the second channel PCM frame data in the current first buffer are underloaded.

[0039] Preferably, the second underload detection module further includes: a mixing state detection module, used to detect whether the current first channel PCM frame data and the second channel PCM frame data have been mixed. When at least one of them has not been mixed, the data volume detection module starts detection.

[0040] Preferably, the first PCM frame data and the second PCM frame data are equipped with an underload flag;

[0041] When the data volume detection module detects that the data volume of the first channel PCM frame data and / or the second channel PCM frame data in the current first buffer does not exceed the second threshold, the underload flag is set to underload.

[0042] The underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded.

[0043] Preferably, the audio mixing processing device further includes: a clear underload flag module, used to detect in real time whether the amount of data of the first channel PCM frame data and / or the second channel PCM frame data in the first buffer exceeds a third threshold; when it exceeds the threshold, the corresponding underload flag is set to non-underload.

[0044] The present invention also provides a chip including the audio mixing processing apparatus as described above; or including a storage medium storing a computer program that, when executed by a processor, can implement the audio mixing processing method as described above.

[0045] The present invention also provides a Bluetooth device, including the audio mixing processing apparatus as described above; or including a storage medium storing a computer program that, when executed by a processor, can implement the audio mixing processing method as described above.

[0046] The present invention also provides a mixing system, including the audio mixing processing apparatus as described above; or a storage medium storing a computer program, which, when executed by a processor, can implement the audio mixing processing method as described above; a first input module and a second input module;

[0047] The first channel of analog audio data and the second channel of analog audio data are input through the first input module and the second input module, respectively, and after processing, they are transmitted to the audio mixing processing device or the storage medium as the first channel of audio data and the second channel of audio data.

[0048] The audio mixing processing method, apparatus, chip, Bluetooth device, and mixing system provided by this invention can perform audio mixing processing using a silent frame when at least one of the first PCM frame data and the second PCM frame data is underloaded, thereby avoiding the situation where one audio stream is interfered with or the transmission is blocked, resulting in poor audio listening effect.

[0049] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0050] The preferred embodiments of the audio mixing processing method and apparatus, chip, Bluetooth device, and mixing system according to the present invention will be described below with reference to the accompanying drawings. (Figures:)

[0051] Figure 1 This is a schematic diagram of the main flow of the audio mixing processing method and apparatus according to an embodiment of the present invention.

[0052] Figure 2 A schematic diagram illustrating the specific process for detecting underload in the mixed audio signal.

[0053] Figure 3 A schematic diagram illustrating the specific process for detecting underload of the first PCM frame data and / or the second PCM frame data.

[0054] Figure 4 This is a schematic diagram of audio mixing processing when underload occurs. Detailed Implementation

[0055] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the following detailed description of the present invention, some specific details are described in detail, but well-known methods, processes, procedures, and elements are not described in detail in order to avoid obscuring the essence of the present invention.

[0056] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0057] Unless the context explicitly requires it, the words "comprising," "including," and similar terms throughout the specification and claims should be interpreted as encompassing rather than being exclusive or exhaustive; that is, meaning "including but not limited to."

[0058] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0059] This invention provides an audio mixing processing method. In this method, a first buffer caches first and second PCM frame data, and a second buffer caches mixed audio data for digital-to-analog conversion to output an analog audio signal. The first and second PCM frame data are obtained by decompressing continuously received first and second audio data. The mixed audio data is the data obtained after completing the audio mixing process.

[0060] Please see Figure 1 The method includes:

[0061] Step S100: When it is detected that the mixed audio data in the second buffer is underloaded, then it is detected whether the current first PCM frame data and the second PCM frame data are underloaded.

[0062] Step S200a: When it is detected that neither the first channel PCM frame data nor the second channel PCM frame data is underloaded, the first channel PCM frame data and the second channel PCM frame data output from the first buffer are summed to obtain the mixed audio data, and the mixed audio data is cached in the second buffer;

[0063] Step S200b: When it is detected that at least one of the current first PCM frame data and the second PCM frame data is underloaded, the underloaded first PCM frame data and / or the underloaded second PCM frame data are summed with silent frame data, and the ununderloaded first PCM frame data or the second PCM frame data are summed with the first PCM frame data or the second PCM frame data output from the first buffer, and the mixed audio data obtained by the summing operation is cached in the second buffer.

[0064] This invention provides an audio mixing processing method. When at least one of the first PCM frame data and the second PCM frame data is underloaded, a silent frame is used for audio mixing processing to solve the problem of audio mixing processing stalling under underload conditions. This can avoid the situation where one audio stream is interfered with or the transmission is blocked, resulting in poor audio listening effect of the receiving terminal.

[0065] This invention provides an audio mixing processing method for mixing multiple PCM frame data. The multiple PCM frame data can be two or more channels. It is understood that this embodiment uses two channels as an example for explanation, but it is not limited to two channels.

[0066] As one embodiment, PCM frame data is data obtained after analog audio data has undergone a series of processes such as noise reduction, analog-to-digital conversion, compression encoding, and decompression. The PCM frame data is buffered in a first buffer. For example, two wireless microphones acquire a first channel of analog audio data and a second channel of analog audio data. The acquired analog audio data is sampled to obtain digital signals, and the digital signals are compressed and encoded to obtain the first and second channel audio data. The first and second channel audio data are transmitted through the transmitting unit of the wireless microphone, and then decompressed to obtain the first and second channel PCM frame data, which are then buffered in the first buffer. It can be understood that the first and second channel PCM frame data are either decompressed and buffered in the first buffer, or the first and second channel audio data are buffered in the first buffer and then decompressed.

[0067] The first and second PCM frame data can be cached in the same buffer partition or buffered in different buffers. These different buffers are then collectively referred to as the first buffer. It is understood that there is no limitation that the first and second PCM frame data are stored in the same storage medium.

[0068] As one embodiment, the mixed audio data is the data obtained by mixing multiple audio signals, and the mixed audio data is buffered in a second buffer. For example, the first PCM frame data and the second PCM frame data output from the first buffer are processed by audio mixing in the mixing module to obtain the mixed audio data. After the mixed audio data undergoes digital-to-analog conversion by the DAC module, the receiving terminal outputs an analog audio signal.

[0069] In step S100, when it is detected that the mixed audio data in the second buffer is underloaded, it is then detected whether the current first PCM frame data and the second PCM frame data are underloaded.

[0070] It's understandable that when the mixed audio data in the second buffer is underloaded, meaning the final audio output is experiencing or tending towards stuttering or playback instability, it's necessary to check if the first and second PCM frame data are underloaded. If underload occurs, without any intervention, stuttering and playback instability will continue. This is because when the first and second PCM frame data are underloaded, no first and / or second PCM frame data is sent to the mixing module for audio mixing within a given time period, or the amount of data sent to the mixing module is insufficient. This will cause the mixing module to enter a waiting state. For example, if the mixing module receives the second PCM frame data while the first PCM frame data is underloaded, but has not received the first PCM frame data, the mixing module will remain in a waiting state and will not perform audio mixing, resulting in no mixed audio data generation and consequently, continuous stuttering and playback instability at the output.

[0071] As one embodiment, the method for detecting whether the mixed audio data in the second buffer is underloaded can be: detecting whether an interrupt signal occurs; if an interrupt signal occurs, detecting whether the buffer value of the mixed audio data in the second buffer is lower than a preset underload trigger value; when the buffer value of the mixed audio data in the second buffer is lower than the preset underload trigger value, it is considered that an underload has occurred.

[0072] Please see Figure 2 As one embodiment, the audio mixing processing method of the present invention further includes steps Sb1 and Sb2, which are executed sequentially.

[0073] In step Sb1, when an interrupt signal is detected, it is checked whether the buffer value of the mixed audio data in the second buffer is lower than a preset underload trigger value.

[0074] It is understandable that interrupt signals can be generated periodically or custom-defined.

[0075] In an embodiment where interrupt signals are generated periodically, the interrupt signal is generated once at a fixed interval (e.g., 1ms). In an embodiment where interrupt signals are generated customizablely, the user can define the interrupt triggering conditions. When the conditions are met, an interrupt signal is generated. In this embodiment, the interval between interrupt signals can be variable (e.g., the first interrupt is generated when the conditions are met 1ms after the system has been running. The second interrupt is generated when the conditions are met again after an interval of 10ms).

[0076] As one embodiment, the interrupt signal is triggered when the amount of mixed audio data in the second buffer falls below a first threshold. That is, in a custom embodiment...

[0077] The amount of mixed audio data can be detected using a counter. The counter counts the mixed audio data entering and exiting the second buffer, allowing for real-time measurement of the current amount of mixed audio data in the second buffer. For example, using a forward counter (increase for in, decrease for out), the amount of mixed audio data in the second buffer can be directly counted. Using a reverse counter (decrease for in, increase for out), the counter calculates the amount of mixed audio data that can be written to the second buffer. Combining this with the size of the second buffer, the total amount of mixed audio data in the second buffer can be calculated.

[0078] As one embodiment, the DAC module used for digital-to-analog conversion triggers an interrupt to determine whether the mixed audio data is underloaded.

[0079] It can be understood that the DAC module is a hardware processing module integrated into the integrated circuit chip. The DAC accesses the data (mixed audio data) in the shared buffer (second buffer) between the CPU and the DAC module, set by the software, through DMA (Direct Memory Access), and moves the data to the digital-to-analog converter circuit to process the digital signal and output the analog signal. Each direct access of the shared buffer by the DAC module results in one sampling point. Taking a positive counter as an example, the counter increments by 1 for each sampling point input. Each access of the mixed audio data in the shared buffer by the DAC module decrements the counter by 1. When the counter decrements to the set interrupt trigger value (first threshold), an interrupt is triggered, which in turn interrupts the CPU from entering the process of checking whether the buffer value of the mixed audio data in the second buffer is lower than the preset underload trigger value.

[0080] When the amount of mixed audio data in the second buffer falls below a first threshold, an interrupt signal is triggered. This reduces the drawbacks of frequent CPU interruptions caused by periodic interrupts, which can lead to system performance degradation. It also eliminates the delay caused by periodic interrupts requiring the filling of the DAC module's next sampling buffer. Furthermore, it allows users to set corresponding interrupt trigger points according to their actual business needs.

[0081] In step Sb2, when it is detected that the buffer value of the mixed audio data in the second buffer is lower than the preset underload trigger value, it is determined that the mixed audio data in the second buffer has been underloaded.

[0082] It is understandable that in actual systems or devices, not only the services of the above-mentioned layers use the interrupt of the DAC module. Therefore, when the interrupt signal is generated, it is impossible to predict whether the interrupt is triggered by the services of the above-mentioned layers or by other services or device configurations. For the robustness of the logic, step Sb2 is executed after the interrupt signal is generated.

[0083] It is understood that the buffer value can be a specific amount of data (in points) or a specific time. Taking data volume as an example, the preset underload trigger value is 10 sampling points (assuming it corresponds to 2ms). When the counter counts and the mixed audio data in the second buffer is 9 sampling points, the condition is met: the buffer value of the mixed audio data in the second buffer is lower than the preset underload trigger value, thus determining that the mixed audio data in the second buffer has experienced underload. Taking time as an example, the preset underload trigger value is 2ms. When the counter counts and the mixed audio data in the second buffer is 9 sampling points, the converted buffer value is 1.8ms. The buffer value of the mixed audio data is lower than the preset underload trigger value, and underload occurs.

[0084] As a modified embodiment, instead of detecting whether an interrupt signal has occurred, it directly confirms whether an underload has occurred by detecting whether the buffer value of the mixed audio data in the second buffer is lower than a preset underload trigger value. That is, step Sb1 can be omitted.

[0085] As one embodiment, when the mixed audio data in the second buffer is under-mixed, the method for detecting whether the current first PCM frame data and the second PCM frame data are under-mixed can be as follows: Check whether the current first PCM frame data and the second PCM frame data have been completely mixed. If at least one of the first PCM frame data and the second PCM frame data has not been completely mixed, then check whether the data volume of the corresponding unmixed first PCM frame data and / or second PCM frame data in the current first buffer does not exceed a second threshold. If it does not exceed the threshold, then an under-mixing is considered to have occurred. It can be understood that it is not necessary to check whether the data volume of the fully mixed PCM frame data exceeds the second threshold; only the PCM frame data that has not been completely mixed needs to be checked to see if its data volume exceeds the second threshold.

[0086] Please see Figure 3 As one embodiment, the audio mixing processing method of the present invention further includes steps Sa1 and Sa2, which are executed sequentially.

[0087] Step Sa1: When it is detected that at least one of the current first channel PCM frame data and the second channel PCM frame data has not been mixed, it is then detected whether the amount of the first channel PCM frame data and / or the second channel PCM frame data in the current first buffer that has not been mixed has not exceeded the second threshold.

[0088] Step Sa2: When it is detected that the amount of data of the first PCM frame data and / or the second PCM frame data in the first buffer does not exceed the second threshold, it is determined that the first PCM frame data and / or the second PCM frame data in the first buffer are underloaded.

[0089] In step S200a, when it is detected that neither the first PCM frame data nor the second PCM frame data is underloaded, the first PCM frame data and the second PCM frame data output from the first buffer are summed to obtain the mixed audio data, and the mixed audio data is cached in the second buffer.

[0090] In step S200b, when it is detected that at least one of the current first PCM frame data and the second PCM frame data is underloaded, the underloaded first PCM frame data and / or the underloaded second PCM frame data are summed with silent frame data, and the unloaded first PCM frame data or the second PCM frame data are summed with the first PCM frame data or the second PCM frame data output from the first buffer, and the mixed audio data obtained by the summing operation is cached in the second buffer.

[0091] Please see Figure 4 Taking the case where the first PCM frame data does not experience underload but the second PCM frame data does, during the time interval t1-tn, the first PCM frame data continuously outputs P1, P2, P3...Pn to the mixing module. After the second PCM frame data continuously outputs Q1 and Q2, underload occurs due to signal interference or weak signal (t3-t6). After t6, the underload is eliminated, and Q3, Q4...Qn are continuously output. The mixing module starts audio mixing processing at time t2, and adds the first and second PCM frame data output from the first buffer to obtain the mixed audio data P1+Q1 and P2+Q2. Since underload occurred during t3-t6, that is, the amount of second PCM frame data in the first buffer is detected to be lower than the second threshold, the second PCM frame data at this time is added with silent frame data, thereby enabling the mixing module to work normally and not be in a waiting state, ensuring the continuous transmission of the other data. After the underload state disappears (after t6), the mixing module continues to sum the first PCM frame data and the second PCM frame data output from the first buffer to obtain the mixed audio data.

[0092] Step S200b avoids misjudgment in critical signal states. For example, if there is signal instability (in a critical signal state), it is very likely that in S200a, the PCM frame data is not fully mixed, but there is actually data in the first buffer.

[0093] As a modified embodiment, step S200a can be omitted, and the determination of whether the first and second PCM frame data are underloaded can be made directly by judging step S200b. As another embodiment, the signal critical state can also be disregarded, step S200b can be omitted, and the determination of whether the first and second PCM frame data are underloaded can be made directly by judging step S20a.

[0094] As one embodiment, the first PCM frame data and the second PCM frame data are equipped with an underload flag. The underload flag includes a relatively independent step of setting the underload flag and a step of clearing the underload flag. In the step of setting the underload flag, the underload flag is set to underload (flag=1), and in the step of clearing the underload flag, the underload flag is set to non-underload (flag=0). The underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded. The relatively independent steps of setting and clearing the underload flag mean that they are not based on different judgment results from the same judgment process. The judgment processes of the two steps are independent of each other, which helps to handle the occurrence and disappearance of underload in a timely manner.

[0095] As one embodiment, the first PCM frame data and the second PCM frame data are equipped with an underload flag. In step Sa2 of the method, the underload flag corresponding to the first PCM frame data and / or the second PCM frame data whose data volume does not exceed the second threshold is set to underload. The underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded, so as to determine whether to proceed to the next step. In this step, if the data volume exceeds the second threshold, the underload flag is still set to underload, and no processing is performed on the underload flag in this step.

[0096] As one embodiment, the method further includes step S300: when the amount of data of the first channel PCM frame data and / or the second channel PCM frame data in the first buffer detected in real time exceeds a third threshold; then the corresponding underload flag is set to non-underload.

[0097] In this step, if the amount of data does not exceed the third threshold, the underload flag is still set to the non-underload state. In this step, the underload flag is not processed. That is, the step of setting the underload flag to underload state is relatively independent of the step of setting the underload flag to non-underload state.

[0098] As one embodiment, the audio mixing processing method includes the steps of: detecting whether the mixed audio data includes a first channel PCM frame data or a silence frame, a second channel PCM frame data or a silence frame; if it includes, then digital-to-analog conversion is performed; if it does not include, then the audio mixing is considered unsuccessful, the data is discarded, and PCM frame data is continuously received for mixing processing.

[0099] It is understood that the solution described in this invention is applicable to low-power Bluetooth multi-connection asynchronous transmission modes, such as Bluetooth 5.0 / 5.1-based BLE (Bluetooth Low Energy) links, which are asynchronous connectionless types. 。

[0100] The method in this invention includes determining whether the mixed audio data is in an underload state by interrupting the DAC module. An underload event is sent to the mixing module to detect the first PCM frame data and whether an asynchronous reception anomaly has occurred. The result is set as an underload flag (flag=1) in the audio stream's running state. The mixing module fills in empty data (silent frame data) based on whether the data reception is normal and whether the PCM frame data is in an underload flag (flag=1). This method effectively ensures that even if one channel experiences signal interference or anomaly, the normally received audio stream continues to generate mixed audio data through the mixing module and is buffered in the DAC module before entering the playback mode. This avoids interference to the other smooth audio data stream, effectively achieving non-interfering mixed output of the two audio data streams.

[0101] The present invention also provides an audio mixing processing device, which includes a receiving module, a mixing module, a DAC module, a first underload detection module, and a second underload detection module. The receiving module continuously receives first and second audio data, buffers them in a first buffer, and decompresses them to obtain first and second PCM frame data. The mixing module performs audio mixing to obtain mixed audio data and buffers the mixed audio data in a second buffer. The DAC module performs digital-to-analog conversion on the mixed audio data output from the second buffer to output an analog audio signal. The first underload detection module detects whether the mixed audio data is underloaded. The second underload detection module detects whether the current first and second PCM frame data are underloaded.

[0102] Specifically, when the first underload detection module detects an underload in the mixed audio data, the second underload detection module initiates a detection process; when the second detection module detects that neither the first nor the second PCM frame data is underloaded, the mixing module performs a summation operation on the first and second PCM frame data output from the first buffer to obtain the mixed audio data; when the second detection module detects that at least one of the first and second PCM frame data is underloaded, in the mixing module, the underloaded first and / or second PCM frame data is summed with silent frame data, and the non-underloaded first or second PCM frame data is summed with the first or second PCM frame data output from the first buffer.

[0103] As one embodiment, the first underload detection module is located in the DAC module.

[0104] As one embodiment, the audio mixing processing device is integrated on a single chip.

[0105] As one embodiment, the first underload detection module includes: a buffer value detection module, used to detect the buffer value of the mixed audio data in the second buffer, and when the buffer value is lower than a preset underload trigger value, it is determined that the mixed audio data in the second buffer has experienced an underload.

[0106] As one embodiment, the first underload detection module further includes: an interrupt module, used to trigger the generation of an interrupt signal, and when the interrupt signal is generated, the buffer value detection module starts detection; the interrupt module is the interrupt of the DAC module.

[0107] As one embodiment, the first underload detection device includes a counter, which triggers the interrupt module to generate the interrupt signal when it detects that the amount of mixed audio data in the second buffer is lower than a preset first threshold.

[0108] As one embodiment, the second underload detection module includes: a data volume detection module, used to detect the data volume of the first channel PCM frame data and / or the second channel PCM frame data in the current first buffer, and when the data volume does not exceed a second threshold, it is determined that the first channel PCM frame data and / or the second channel PCM frame data in the current first buffer are underloaded.

[0109] As one embodiment, the second underload detection module further includes: a mixing state detection module, used to detect whether the current first channel PCM frame data and the second channel PCM frame data have been mixed. When at least one of them has not been mixed, the data volume detection module starts detection.

[0110] As one embodiment, the first PCM frame data and the second PCM frame data are set with an underload flag; when the data volume detection module detects that the data volume of the first PCM frame data and / or the second PCM frame data in the current first buffer does not exceed the second threshold, the underload flag is set to underload; the underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded.

[0111] As one embodiment, the audio mixing processing device further includes: a clear underload flag module, used to detect in real time whether the amount of data of the first channel PCM frame data and / or the second channel PCM frame data in the first buffer exceeds a third threshold; when it exceeds the threshold, the corresponding underload flag is set to non-underload.

[0112] It can be understood that the audio mixing processing device is a device corresponding to the audio mixing processing method, and all the contents described in the embodiments of the audio mixing processing method are applicable to the audio mixing processing device.

[0113] The present invention also provides a chip including the audio mixing processing apparatus as described above; or including a storage medium storing a computer program that, when executed by a processor, can implement the audio mixing processing method as described above.

[0114] The present invention also provides a Bluetooth device, including the audio mixing processing apparatus as described above; or including a storage medium storing a computer program, which, when executed by a processor, can implement the audio mixing processing method as described above. The Bluetooth device may be a Bluetooth mixing system, a Bluetooth speaker, etc.

[0115] The present invention also provides a mixing system, including the audio mixing processing device as described above; or a storage medium storing a computer program, which, when executed by a processor, can implement the audio mixing processing method as described above; a first input module and a second input module; a first channel of analog audio data and a second channel of analog audio data are respectively input through the first input module and the second input module, and after processing, the first channel of audio data and the second channel of audio data are transmitted to the audio mixing processing device or the storage medium.

[0116] In one embodiment, the first input module and the second input module are slave devices, specifically wireless microphones, which are wirelessly connected to the master device via Bluetooth. The master device includes a receiving module, a mixing module, and a DAC module. The master device is integrated onto a single chip.

[0117] It is understood that the steps in the method provided by this invention can be implemented by hardware and / or software. When implemented by software or a combination of software and hardware, the steps can be executed by different program logics.

[0118] Those skilled in the art will understand that, without conflict, the above-mentioned preferred solutions can be freely combined and superimposed.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods, apparatus, systems, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] It should be understood that the above embodiments are merely exemplary and not restrictive. Various obvious or equivalent modifications or substitutions that can be made by those skilled in the art regarding the above details without departing from the basic principles of the present invention will be included within the scope of the claims of the present invention.

Claims

1. An audio mixing processing method, characterized in that, The first buffer caches the first and second PCM frame data, and the second buffer caches the mixed audio data for digital-to-analog conversion to output analog audio signals. The first PCM frame data and the second PCM frame data are obtained by decompressing the continuously received first audio data and second audio data; the mixed audio data is the data obtained after completing audio mixing. The method includes: Step S100: When it is detected that the mixed audio data in the second buffer is underloaded, then it is detected whether the current first PCM frame data and the second PCM frame data are underloaded. Step S200a: When it is detected that neither the first channel PCM frame data nor the second channel PCM frame data is underloaded, the first channel PCM frame data and the second channel PCM frame data output from the first buffer are summed to obtain the mixed audio data, and the mixed audio data is cached in the second buffer; Step S200b: When it is detected that at least one of the current first PCM frame data and the second PCM frame data is underloaded, the underloaded first PCM frame data and / or the underloaded second PCM frame data are summed with silent frame data, and the ununderloaded first PCM frame data or the second PCM frame data are summed with the first PCM frame data or the second PCM frame data output from the first buffer, and the mixed audio data obtained by the summing operation is cached in the second buffer.

2. The audio mixing processing method as described in claim 1, characterized in that, The method further includes: Step Sb2: When the buffer value of the mixed audio data in the second buffer is detected to be lower than the preset underload trigger value, it is determined that the mixed audio data in the second buffer has been underloaded.

3. The audio mixing processing method as described in claim 2, characterized in that, The method further includes step Sb1, which is performed before step Sb2: when an interrupt signal is detected, it is detected whether the buffer value of the mixed audio data in the second buffer is lower than a preset underload trigger value.

4. The audio mixing processing method as described in claim 3, characterized in that, The interrupt signal is generated periodically or custom-generated.

5. The audio mixing processing method as described in claim 3, characterized in that, When the amount of mixed audio data in the second buffer is lower than the first threshold, the interrupt signal is triggered.

6. The audio mixing processing method as described in claim 1, characterized in that, The method further includes: Step Sa2: When it is detected that the amount of data of the first PCM frame data and / or the second PCM frame data in the first buffer does not exceed the second threshold, it is determined that the first PCM frame data and / or the second PCM frame data in the first buffer are underloaded.

7. The audio mixing processing method as described in claim 6, characterized in that, The method further includes step Sa1, which is performed before step Sa2: when it is detected that at least one of the current first PCM frame data and the second PCM frame data has not been mixed, it is detected whether the amount of the first PCM frame data and / or the second PCM frame data in the current first buffer that has not been mixed has not exceeded a second threshold.

8. The audio mixing processing method as described in claim 1, characterized in that, Both the first PCM frame data and the second PCM frame data are equipped with an underload flag; The underload flag includes a relatively independent step of setting the underload flag and a step of clearing the underload flag; in the step of setting the underload flag, the underload flag is set to underload, and in the step of clearing the underload flag, the underload flag is set to not underload. The underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded.

9. The audio mixing processing method as described in claim 6, characterized in that, Both the first PCM frame data and the second PCM frame data are equipped with an underload flag; In step Sa2 of the method, the underload flag corresponding to the first PCM frame data and / or the second PCM frame data whose data volume does not exceed the second threshold is set to underload; The underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded.

10. The audio mixing processing method as described in claim 9, characterized in that, The method further includes: Step S300: When the amount of data of the first channel PCM frame data and / or the second channel PCM frame data in the first buffer detected in real time exceeds the third threshold, the corresponding underload flag is set to non-underload. The step of setting the underload flag to underload is performed independently of the step of setting the underload flag to non-underload.

11. The audio mixing processing method as described in claim 1, characterized in that, The DAC module used for digital-to-analog conversion triggers an interrupt to determine whether the mixed audio data is underloaded; The method is applicable to Bluetooth Low Energy multi-connection asynchronous transmission mode.

12. An audio mixing processing apparatus, characterized in that, The audio mixing processing device includes: The receiving module is used to continuously receive the first audio data and the second audio data, buffer them into the first buffer, and decompress them to obtain the first PCM frame data and the second PCM frame data. The mixing module is used to perform audio mixing to obtain mixed audio data and cache the mixed audio data in a second buffer; The DAC module is used to perform digital-to-analog conversion on the mixed audio data output from the second buffer and output an analog audio signal; The first underload detection module is used to detect whether the mixed audio data is underloaded; The second underload detection module is used to detect whether the current first channel PCM frame data and the second channel PCM frame data are underloaded. When the first underload detection module detects that the mixed audio data is underloaded, the second underload detection module starts the detection. When the second underload detection module detects that neither the first channel PCM frame data nor the second channel PCM frame data has experienced underload, the mixing module will perform a summation operation on the first channel PCM frame data and the second channel PCM frame data output from the first buffer to obtain the mixed audio data. When the second detection module detects that at least one of the current first PCM frame data and the second PCM frame data is underloaded, in the mixing module, the underloaded first PCM frame data and / or the underloaded second PCM frame data are summed with silent frame data, and the non-underloaded first PCM frame data or the second PCM frame data are summed with the first PCM frame data or the second PCM frame data output from the first buffer.

13. The audio mixing processing apparatus as claimed in claim 12, characterized in that, The first underload detection module includes: The buffer value detection module is used to detect the buffer value of the mixed audio data in the second buffer, and when the buffer value is lower than a preset underload trigger value, it is determined that the mixed audio data in the second buffer has underloaded.

14. The audio mixing processing apparatus as claimed in claim 13, characterized in that, The first underload detection module further includes: An interrupt module is used to trigger the generation of an interrupt signal, and when the interrupt signal is generated, the buffer value detection module starts detection; The interrupt module is the interrupt for the DAC module.

15. The audio mixing processing apparatus as claimed in claim 14, characterized in that, The first underload detection module includes a counter. When the counter detects that the amount of mixed audio data in the second buffer is lower than a preset first threshold, it triggers the interrupt module to generate the interrupt signal.

16. The audio mixing processing apparatus as claimed in claim 12, characterized in that, The second underload detection module includes: The data volume detection module is used to detect the data volume of the first channel PCM frame data and / or the second channel PCM frame data in the first buffer, and when the data volume does not exceed the second threshold, it is determined that the first channel PCM frame data and / or the second channel PCM frame data in the first buffer are underloaded.

17. The audio mixing processing apparatus as claimed in claim 16, characterized in that, The second underload detection module further includes: The mixing status detection module is used to detect whether the current first channel PCM frame data and the second channel PCM frame data have been mixed. When at least one of them has not been mixed, the data volume detection module starts detection.

18. The audio mixing processing apparatus as claimed in claim 16, characterized in that, Both the first PCM frame data and the second PCM frame data are equipped with an underload flag; When the data volume detection module detects that the data volume of the first channel PCM frame data and / or the second channel PCM frame data in the current first buffer does not exceed the second threshold, the underload flag is set to underload. The underload flag is queried to determine whether the first PCM frame data and the second PCM frame data are underloaded.

19. The audio mixing processing apparatus as claimed in claim 12, characterized in that, The audio mixing processing apparatus further includes: The underload flag clearing module is used to detect in real time whether the amount of data of the first channel PCM frame data and / or the second channel PCM frame data in the first buffer exceeds a third threshold; when it exceeds the threshold, the corresponding underload flag is set to non-underload.

20. A chip, characterized in that: Includes the audio mixing processing apparatus as described in any one of claims 12-19; ​​or includes a storage medium storing a computer program that, when executed by a processor, enables the audio mixing processing method as described in any one of claims 1-11.

21. A Bluetooth device, characterized in that: Includes the audio mixing processing apparatus as described in any one of claims 12-19; ​​or includes a storage medium storing a computer program that, when executed by a processor, enables the audio mixing processing method as described in any one of claims 1-11.

22. A mixing system, characterized in that: Includes an audio mixing processing apparatus as described in any one of claims 12-19; ​​or a storage medium storing a computer program that, when executed by a processor, can implement the audio mixing processing method as described in any one of claims 1-11; a first input module and a second input module; The first channel of analog audio data and the second channel of analog audio data are input through the first input module and the second input module, respectively, and after processing, they are transmitted to the audio mixing processing device or the storage medium as the first channel of audio data and the second channel of audio data.

Images