An audio data processing method and related apparatus

Abstract

The application discloses an audio data processing method and related device, and relates to the field of audio processing, and comprises the following steps: acquiring PCM data of each sound channel of a current audio frame of sound stream information based on a spatial audio algorithm; performing bidirectional filtering on the PCM data of the current audio frame based on the PCM data of a previous audio frame to obtain high-frequency signals and low-frequency signals; deleting the PCM data of the previous audio frame saved in a first area and moving the PCM data of the current audio frame from a second area to the first area; performing downmix processing on the high-frequency signals and the low-frequency signals respectively to obtain high-frequency stereo signals and low-frequency stereo signals; and merging the high-frequency stereo signals and the low-frequency stereo signals to obtain an audio rendering signal of the current audio frame. The application realizes real-time output of an audio rendering signal with clear and stable levels under a low-delay condition based on frame-level bidirectional filtering operation.

Description

Technical Field

[0001] This application relates to the field of audio data processing technology, and in particular to an audio data processing method and related apparatus. Background Technology

[0002] In scenarios with high real-time audio processing requirements (such as music variety shows), it is necessary to process the live sound captured by microphones in real time and output the generated audio data to the internet. Currently, when processing live sound, spatial audio algorithms are used to downmix both high and low frequencies simultaneously, resulting in overlapping high-frequency and low-frequency sound images. For example, if the high-frequency vocal image is located in the center of the stage and the low-frequency piano sound is located around the perimeter, downmixing both simultaneously will cause the sound image locations of the high-frequency vocals and the low-frequency piano sound to overlap, failing to represent the actual placement of sound sources on the stage and resulting in poor stereo sound. Summary of the Invention

[0003] In view of the above problems, this application provides an audio data processing method and related apparatus to achieve real-time processing of audio data. The specific solution is as follows:

[0004] The first aspect of this application provides an audio data processing method, including:

[0005] PCM data for each channel of the current audio frame is obtained based on spatial audio algorithms to acquire sound flow information.

[0006] If the current audio frame is not the first frame, the PCM data of the current audio frame is stored in the second area of ​​the buffer. The buffer includes the first area and the second area. The second area is arranged after the first area. The first area stores the PCM data of the previous audio frame.

[0007] Based on the PCM data of the previous audio frame, the PCM data of the current audio frame is bidirectionally filtered to obtain high-frequency and low-frequency signals. The PCM data of the previous audio frame stored in the first region is deleted, and the PCM data of the current audio frame is moved from the second region to the first region.

[0008] The high-frequency signal and the low-frequency signal are downmixed separately to obtain the high-frequency stereo signal and the low-frequency stereo signal.

[0009] The high-frequency stereo signal and the low-frequency stereo signal are merged to obtain the audio rendering signal of the current audio frame.

[0010] Optional, bidirectional filtering process includes:

[0011] The PCM data in the buffer is filtered to obtain the filtered result, and the forward buffer data is saved. The filtered result is then reverse-processed to obtain the reverse-filtered result. The reverse-filtered result is then subjected to a second filtering process to obtain the second filtering result, and the reverse buffer data is saved. The second filtering result is then reverse-processed a second time to obtain the second reverse-filtered result. The second reverse-filtered result is then segmented to obtain the high-frequency and low-frequency signals for each channel.

[0012] Optionally, the PCM data in the buffer is filtered to obtain the filtered result, and the forward buffer data is saved, including:

[0013] If the current audio frame is the first frame, perform the first filtering process on the PCM data in the buffer to obtain the filtering result, and save the forward buffer data;

[0014] If the current audio frame is not the first frame, the PCM data in the buffer is subjected to a second filtering process to obtain the filtering result, and the first two digits of the filtering result are removed, and the forward buffer data is saved.

[0015] Optionally, a second filtering process is performed on the PCM data in the buffer to obtain the filtered result, including:

[0016] Extract the last M bits of the PCM data from the previous audio frame to obtain the input buffer data. The length of M is the length of the filter parameters.

[0017] The PCM data of the previous audio frame is filtered to obtain the processing result, and the last M bits of the processing result are extracted to obtain the initial input conditions.

[0018] Input the input buffer data, initial input conditions, and PCM data of the current audio frame into the filter to obtain the output data of the current audio frame;

[0019] The initial input conditions are assigned to the first M bits of the output data to obtain the filtering result.

[0020] Optionally, the PCM data of the current audio frame is bidirectionally filtered based on the PCM data of the previous audio frame to obtain high-frequency and low-frequency signals, including:

[0021] Obtain the filter parameters for each channel of the current audio frame. The filter parameters include low-frequency input filter parameters, low-frequency feedback filter parameters, high-frequency input filter parameters, and high-frequency feedback filter parameters.

[0022] Based on the PCM data of the previous audio frame, low-frequency input filtering parameters, and low-frequency feedback filtering parameters, the PCM data of the current audio frame is bidirectionally filtered to obtain the low-frequency signal of each channel.

[0023] Based on the PCM data of the previous audio frame, high-frequency input filtering parameters, and high-frequency feedback filtering parameters, the PCM data of the current audio frame is bidirectionally filtered to obtain the high-frequency signal of each channel.

[0024] Optionally, the result of the second inverse filtering is segmented to obtain the high-frequency and low-frequency signals for each channel, including:

[0025] If the current frame is the first frame, when the second inverse filtering result of the current frame is obtained, the first nine bits of the second inverse filtering result are cut off, and then the second inverse filtering result of the first 1024 sampling points is extracted to obtain the high frequency signal and low frequency signal of each channel.

[0026] If the current frame is not the first frame, then when obtaining the second inverse filtering result of the current frame, the first 1024 sampling points of the second inverse filtering result are extracted to obtain the high-frequency signal and low-frequency signal of each channel.

[0027] Optionally, the process of down-mixing the high-frequency signal and the low-frequency signal separately includes:

[0028] The low-frequency signal of each channel is downmixed to obtain a low-frequency stereo signal;

[0029] The high-frequency signal of each channel is processed in the middle channel to obtain the sound field extension signal;

[0030] The sound field extension signal is downmixed using HRTF to obtain a high-frequency stereo signal.

[0031] A second aspect of this application provides an audio data processing apparatus, comprising:

[0032] The data acquisition module is used to acquire the PCM data of each channel of the current audio frame based on the spatial audio algorithm;

[0033] The data update module is used to store the PCM data of the current audio frame into the second area of ​​the buffer if the current audio frame is not the first frame. The buffer includes a first area and a second area, with the second area following the first area. The first area stores the PCM data of the previous audio frame.

[0034] The filtering module is used to perform bidirectional filtering on the PCM data of the current audio frame based on the PCM data of the previous audio frame to obtain high-frequency signals and low-frequency signals, delete the PCM data of the previous audio frame stored in the first region, and move the PCM data of the current audio frame from the second region to the first region.

[0035] The downmixing module is used to downmix the high-frequency signal and the low-frequency signal respectively to obtain the high-frequency stereo signal and the low-frequency stereo signal;

[0036] The data merging module is used to merge high-frequency stereo signals and low-frequency stereo signals to obtain the audio rendering signal of the current audio frame.

[0037] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the audio data processing method of the first aspect or any implementation thereof.

[0038] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0039] Memory is used to store computer programs;

[0040] The processor is used to execute computer programs to enable electronic devices to implement the audio data processing method of the first aspect or any implementation thereof.

[0041] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the audio data processing method described in the first aspect or any implementation thereof.

[0042] By employing the aforementioned technical solution, this application avoids cross-frame boundary distortion and ensures real-time performance by introducing a buffer during frame-by-frame processing. Based on this, the audio signal of each channel is divided into high-frequency and low-frequency components, and the high-frequency and low-frequency signals are down-mixed separately to generate low-frequency stereo signals and high-frequency stereo signals, improving the accuracy of sound image localization and the resolution of spatial distribution. Finally, the low-frequency and high-frequency stereo signals of each channel are added together to obtain the binaural rendering signal. Compared to existing binaural rendering methods for files that use conventional bidirectional filtering and cannot be applied in real-time live streaming systems, this application reduces computational complexity while maintaining the balance of low-frequency energy. The binaural rendering signal obtained by synthesizing high and low frequency signals can output clear and stable sound field results in real time under low latency conditions. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A flowchart illustrating an audio data processing method provided in this application;

[0045] Figure 2 A flowchart illustrating an audio data processing algorithm provided in this application;

[0046] Figure 3 This application provides a schematic diagram of the structure of an audio data processing device;

[0047] Figure 4 A schematic block diagram of an electronic device provided in this application. Detailed Implementation

[0048] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0049] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0050] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0051] This application provides an audio data processing method, such as... Figure 1 The method may include the following steps:

[0052] S101: PCM data of each channel of the current audio frame is obtained based on the spatial audio algorithm to obtain the sound flow information.

[0053] Optionally, in one embodiment, the spatial audio algorithm can be the Jingcai Sound algorithm, Dolby algorithm, and DTS:X algorithm. The current audio frame refers to the frame data currently being processed in the audio stream's time sequence. It is an audio signal segment divided by sampling points, typically including several sampling points, capable of completely reflecting the sound data within that time segment. A channel refers to an independent sound channel in an audio system, such as the common left channel, right channel, and the front, rear, and bass channels in multi-channel surround sound. Different channels carry different audio information, used to reconstruct a stereo sound field in space. In this embodiment, the number of channels can be 10, including the left channel, right channel, center channel, bass channel, left surround channel, right surround channel, left front top channel, right front top channel, left rear top channel, and right rear top channel, with the last four channels referred to as sky channels. PCM data, or Pulse Code Modulation data, is a discrete digital signal obtained by sampling, quantizing, and encoding analog audio signals. It can fully reflect the waveform characteristics of the original sound and is the fundamental data in audio processing.

[0054] Optionally, in this embodiment, the system can directly decode the input audio stream information frame by frame to obtain PCM data. The PCM data is then divided into frames according to a fixed sampling point length (e.g., 1024 sampling points), and mapped to PCM data corresponding to multiple channels according to the number of channels, to ensure the accuracy of the data input. The formula for obtaining the PCM data of each channel of the current audio frame is as follows: ,in, This refers to all PCM data contained in the acoustic flow information, and Given a matrix of [1024, 10], This represents the audio frame counter, indicating how many audio frames are contained in the PCM data (this applies to all the following formulas). This represents an audio frame with a length of 1024 sample points.

[0055] Specifically, amplitude normalization can be performed on the PCM data during decoding to unify the amplitudes of different channels into a reasonable range. This avoids excessive volume differences between channels that could lead to poor subsequent processing results, ensuring that the timing and sampling accuracy of the PCM data meet the requirements of different playback devices. By accurately parsing the PCM data into frame-by-frame and channel-by-channel PCM data, not only is complete and reliable input data provided for subsequent buffering and filtering processing, but the distortion-free parsing of the PCM data is also ensured, thereby improving the accuracy and controllability of subsequent PCM data processing.

[0056] S102: If the current audio frame is not the first frame, the PCM data of the current audio frame is stored in the second region of the buffer. The buffer includes the first region and the second region. The second region is arranged after the first region. The first region stores the PCM data of the previous audio frame.

[0057] Optionally, in this embodiment, the buffer is a pre-allocated storage space in the system memory used to temporarily store the PCM data of each channel, ensuring the continuity and integrity of data during processing. The first region is used to store the PCM data of the previous audio frame, and the second region is used to store the PCM data of the current audio frame. The data in the second region is arranged after the first region, thus forming an ordered dual-region structure, providing continuous input data for subsequent bidirectional filtering. Specifically, when the input audio frame is the first frame, the PCM data of the first frame is placed in the first region without any other operations, while the PCM data of the second frame is placed in the second region.

[0058] Specifically, the buffer can be represented as: ,in, This serves as a buffer, used to buffer input information to ensure continuity of data processing.

[0059] when hour, and The same, when At that time, the previous frame With the current frame Together they form a buffer zone. in front A 2048-byte buffer is then formed. The buffer updates data sequentially; when writing new PCM data, the PCM data of the previous audio frame is stored in the first region, while the PCM data of the current audio frame is written to the second region, thus ensuring sequential storage of adjacent frames. In another implementation, the buffer employs a double-buffered structure, with the first and second regions alternating as input and output. This allows data to be written to one region while data in the other region is accessed by the IIR filter, improving the parallelism of data reading and writing. In yet another implementation, the buffer can be designed as a circular queue. When the buffer is full, newly written PCM data overwrites the oldest frame. This design is particularly suitable for scenarios with high real-time requirements, such as audio variety shows or live broadcasts, enabling the system to operate continuously and stably without data loss due to buffer overflow. By establishing a buffer to buffer input data, the PCM data of the previous and current audio frames are stored continuously and orderly in the buffer, providing the complete input data required for subsequent bidirectional filtering. This not only improves the coherence and stability of data processing but also optimizes storage space utilization, thus balancing real-time performance and processing accuracy.

[0060] S103: Based on the PCM data of the previous audio frame, perform bidirectional filtering on the PCM data of the current audio frame to obtain high-frequency and low-frequency signals, delete the PCM data of the previous audio frame stored in the first region, and move the PCM data of the current audio frame from the second region to the first region.

[0061] Optionally, in this embodiment, bidirectional filtering can refer to a signal processing method that combines forward filtering and inverse filtering. Forward filtering is used to initially separate high-frequency and low-frequency signals in the audio signal, while inverse filtering is used to eliminate phase distortion that may be introduced during forward filtering, thereby obtaining a phase-free filtering result. High-frequency signals refer to the high-frequency components of the audio signal in the PCM data extracted during the filtering process, including the clarity, brightness, and detail of the sound. Low-frequency signals refer to the low-frequency components of the audio signal in the PCM data retained during the filtering process, mainly reflecting the thickness, atmosphere, and spatial immersion of the sound. Specifically, bidirectional filtering is performed in the order of forward filtering, inverse filtering, secondary filtering, and secondary inverse filtering, and redundant data is removed by cutting to ensure that the final high-frequency and low-frequency signals are smoother and more accurate.

[0062] In another implementation, bidirectional filtering can employ a segmented processing strategy, dividing the current audio frame into multiple sub-segments. Each sub-segment undergoes bidirectional filtering with the previous audio frame, and the results of these sub-segments are then concatenated to form complete high-frequency and low-frequency signals. This approach is suitable for processing large amounts of data and can effectively reduce the computational burden per operation. This embodiment effectively separates high-frequency and low-frequency signals and avoids phase distortion, thereby achieving a clear representation of sound layers and providing high-quality input signals for subsequent downmixing and audio rendering.

[0063] S104: Perform downmixing on the high-frequency and low-frequency signals respectively to obtain high-frequency stereo signals and low-frequency stereo signals.

[0064] Optionally, in this embodiment, downmixing can refer to the process of converting high-frequency and low-frequency signals into stereo signals according to certain rules. Its purpose is to reduce the number of channels while preserving as much of the original sound field's spatial characteristics and sonic layering as possible. When downmixing high-frequency signals, a channel-side processing method is typically used, decomposing the high-frequency components of multiple channels into center and side signals, thereby enhancing the sound field's expansion. When downmixing low-frequency signals, a loudness downmixing operation is often used, weighted and fused from low-frequency signals of different channels. This ensures that the low-frequency signals still have sufficient power and immersion after stereo synthesis, preserving or even enhancing the spatial and layering of the sound. This ensures that the audio signal presents a good listening effect on different playback devices, making it particularly suitable for the immersive sound effects requirements of music variety shows.

[0065] S105: Merge the high-frequency stereo signal and the low-frequency stereo signal to obtain the audio rendering signal of the current audio frame.

[0066] Optionally, in this embodiment, the audio rendering signal can refer to the final playable audio data output after fusing and spatializing multiple signals. The merging process typically involves aligning the amplitude, phase, and timing of the high-frequency stereo signal and the low-frequency stereo signal of each channel, and then mixing them according to a set synthesis ratio to obtain a rendering result containing full-band information. Specifically, the high-frequency stereo signal and the low-frequency stereo signal can be directly weighted and superimposed to obtain a full-bandwidth audio rendering signal, generating an audio rendering signal that balances clarity and richness, so that the output sound has both detailed representation and maintains the overall spatial atmosphere, enhancing the audience's immersive experience in audio variety shows.

[0067] In one embodiment, the bidirectional filtering process includes:

[0068] The PCM data in the buffer is filtered to obtain the filtered result, and the forward buffer data is saved. The filtered result is then reverse-processed to obtain the reverse-filtered result. The reverse-filtered result is then subjected to a second filtering process to obtain the second filtering result, and the reverse buffer data is saved. The second filtering result is then reverse-processed a second time to obtain the second reverse-filtered result. The second reverse-filtered result is then segmented to obtain the high-frequency and low-frequency signals for each channel.

[0069] Specifically, firstly, an IIR filter can be used to filter the PCM data in the buffer to obtain the filtering result. The current audio frame refers to the PCM data segment currently being processed. The buffer is an area in the system memory used to temporarily store audio frame data. The PCM data stored in the buffer is the input digitized audio signal.

[0070] An IIR filter is an infinite impulse response filter that can achieve high filtering accuracy with limited computation, making it suitable for real-time audio processing. Subsequently, the formula is used... The filtering result undergoes inverse processing, which involves reversing the data sequence of the filtered result in time. This process eliminates the phase delay that might be introduced by the forward filtering, thereby improving signal fidelity. After inverse processing, the following formula is applied... The obtained inverse filtering result is subjected to a second filtering process to obtain the second filtering result. and use the formula Save reverse buffer data This refers to the inverse buffer data, which is the intermediate result generated during the inverse filtering process. Next, the formula is used... The result of the second filtering is then subjected to a second inverse processing to obtain the final second inverse filtering result. The purpose of the second inversion process is to further eliminate residual phase distortion, making the output signal smoother and more natural. Finally, the formula is used. The result of the second inverse filtering is then segmented. Segmentation refers to removing invalid data introduced by zero padding during the filtering process, retaining only the valid sampling points, thereby obtaining the high-frequency and low-frequency signals for each channel. The high-frequency signal represents the part of the audio that contains clarity and detail, while the low-frequency signal represents the part of the audio that embodies atmosphere and spatiality.

[0071] In another embodiment, an IIR filter can be used to filter the PCM data in the buffer, and then the formula is applied. The filtering result is then processed in reverse to obtain the inverse filtering result. Then, the formula is applied... The obtained inverse filtering result is subjected to a second filtering process to obtain the second filtering result, and the corresponding inverse buffer data is saved. Finally, the result of the second filtering is applied using the formula... The reverse processing is performed again to obtain the secondary inverse filtering result, using the formula... The result of the secondary inverse filtering is then segmented to extract the effective portion, ultimately yielding the high-frequency and low-frequency signals for each channel.

[0072] In one embodiment, filtering the PCM data in the buffer to obtain the filtering result and saving the forward buffer data includes:

[0073] If the current audio frame is the first frame, the PCM data in the buffer is subjected to a first filtering process to obtain the filtering result, and the forward buffer data is saved;

[0074] If the current audio frame is not the first frame, the PCM data in the buffer is subjected to a second filtering process to obtain a filtering result, and the first two digits of the filtering result are removed, and the forward buffer data is saved.

[0075] Specifically, when the current audio frame is the first frame, to ensure the smoothness of the filtering process, the input data in the buffer is padded with zeros, using the formula... The buffer is padded with zeros, where, Represents the zero-padding function Represents the parameters of the waveguide. , indicating taking The product of lengths.

[0076] Firstly, an IIR filter can be used to process the PCM data in the buffer using the formula... The first filtering process is performed to obtain the filtering result, where The IIR filter function defines the current audio frame as the currently processed PCM data segment, and the first frame as the starting frame when the audio stream enters the system for processing. This first frame does not contain any historical frame data and therefore requires special handling. The buffer is a region in system memory used to temporarily store audio frame data; the PCM data stored in the buffer is the input digitized audio signal. An IIR filter is an infinite impulse response filter that can achieve high filtering accuracy with limited computation, making it suitable for real-time audio processing. The filtering result is generated and then processed using the formula... Save the forward buffer data, where Forward buffer data refers to the intermediate results recorded during the filtering process, which are then used in subsequent reverse filtering to ensure consistency between the pre- and post-processing. Indicates the selection of output results The last two characters. When the current audio frame being processed by the system is not the first frame, that is, the audio frame is not the starting frame of the PCM data in the time series, but any frame after the first frame, the system will use an IIR filter to perform a second filtering process on the PCM data in the buffer. The buffer refers to the storage area used by the system to store audio frame data.

[0077] First, use the formula. The PCM data in the buffer undergoes a second filtering process to obtain the filtered result, where... This is the second filtering function. To ensure the stability of the filter output and avoid redundant calculations, the formula is used. The first two digits of the filtered result are discarded to remove invalid sampling points that may be caused by the initial conditions of the filter. Afterwards, the filtered result is generated and processed using the formula... The forward buffer data is saved to provide continuous input conditions in subsequent filtering calculations. As can be seen above, this application, through a combination of zero-padding, forward filtering, reverse filtering, secondary filtering, and secondary reverse filtering in the first frame, ensures that high-precision filtering results can still be obtained in the first frame even without historical reference. This not only avoids the boundary distortion problem that occurs in the first frame stage in traditional methods, but also improves the fidelity of high-frequency and low-frequency signals in each channel, thereby ensuring good continuity and stability of the entire audio stream in subsequent processing. For non-first frames, combining historical data from the previous frame for filtering avoids data distortion caused by the continuation of filter states.

[0078] In one embodiment, a second filtering process is performed on the PCM data in the buffer to obtain the filtered result, including:

[0079] Extract the last M bits of the PCM data from the previous audio frame to obtain the input buffer data, where M is the length of the filter parameters. Filter the PCM data from the previous audio frame to obtain the processing result, and extract the last M bits of the processing result to obtain the initial input conditions. Input the input buffer data, the initial input conditions, and the PCM data of the current audio frame into the filter to obtain the output data of the current audio frame. Assign the initial input conditions to the first M bits of the output data to obtain the filtering result.

[0080] The filter can be an IIR filter. Specifically, when using an IIR filter to perform a second filtering process on the PCM data in the buffer, the formula is first used. Extract the last M bits of the PCM data from the previous audio frame to obtain the input buffer data. ,in, The preceding audio frame refers to the frame immediately preceding the current audio frame. M represents the length of the filter parameters, typically corresponding to the order of the IIR filter or the state quantity to be maintained; in this embodiment, M can be 3. The input buffer data ensures the continuity of cross-frame processing. Subsequently, the formula is used... The PCM data of the previous audio frame is filtered to obtain the processed result, and the last M bits are extracted from this result as the initial input condition. The initial input conditions refer to the internal state variables of the IIR filter before entering the next frame. These conditions record the state of the IIR filter's processing of the previous frame's data, thus ensuring the consistency and stability of cross-frame filtering. Next, the formula is used... The input buffer data, initial input conditions, and PCM data of the current audio frame are input together into the IIR filter for a second filtering process, yielding the processing result. This is the PCM data of the current audio frame, which is the raw audio input to be processed. Combined with the input buffer data and initial input conditions, it forms a complete, continuous signal during the filtering process, avoiding discontinuities or distortion between frames. Finally, the system assigns the initial input conditions to the first M bits of the processing result, thereby obtaining the output data of the current audio frame.

[0081] As can be seen from the above, this application, by introducing input buffer data and initial input conditions, enables the second filtering process to maintain the continuity of the signal in the case of cross-frame processing, avoiding the problems of discontinuity and spectral distortion caused by cross-frame splicing in traditional methods. This improves the adaptability and robustness of the IIR filter in real-time audio processing scenarios and provides a reliable guarantee for the accurate separation of high-frequency and low-frequency signals in each channel.

[0082] In one embodiment, the PCM data of the current audio frame is bidirectionally filtered based on the PCM data of the previous audio frame to obtain high-frequency and low-frequency signals, and the method further includes:

[0083] Obtain the filter parameters for each channel of the current audio frame. The filter parameters include low-frequency input filter parameters, low-frequency feedback filter parameters, high-frequency input filter parameters, and high-frequency feedback filter parameters. Based on the PCM data of the previous audio frame, the low-frequency input filter parameters, and the low-frequency feedback filter parameters, perform bidirectional filtering on the PCM data of the current audio frame to obtain the low-frequency signal for each channel. Based on the PCM data of the previous audio frame, the high-frequency input filter parameters, and the high-frequency feedback filter parameters, perform bidirectional filtering on the PCM data of the current audio frame to obtain the high-frequency signal for each channel.

[0084] Specifically, firstly, a Bassworth crossover can be used to obtain the filter parameters for each channel of the current audio frame. A crossover is a module that splits a full-band signal into low-frequency and high-frequency components according to a set frequency threshold, providing corresponding parameter support for filtering different frequency bands. Filter parameters include low-frequency input filter parameters, low-frequency feedback filter parameters, high-frequency input filter parameters, and high-frequency feedback filter parameters. The input filter parameters control the path of external signals into the IIR filter, while the feedback filter parameters control the characteristics of the feedback loop within the IIR filter. Together, they determine the filter's response characteristics to signals in different frequency bands. For the left and right channels, 8000Hz can be used as the crossover point; for the left and right surround channels, 1000Hz; for the sky channels, 450Hz; and for the bass channels, no crossover is needed. Subsequently, based on the PCM data of the previous audio frame, the low-frequency input filter parameters, and the low-frequency feedback filter parameters, the PCM data of the current audio frame is bidirectionally filtered to obtain the low-frequency signal for each channel. Then, based on the PCM data of the previous audio frame, the high-frequency input filtering parameters, and the high-frequency feedback filtering parameters, the system performs bidirectional filtering on the PCM data of the current audio frame to obtain the high-frequency signal for each channel.

[0085] From the above, it can be concluded that this application, by introducing a crossover to obtain low-frequency input filtering parameters, low-frequency feedback filtering parameters, high-frequency input filtering parameters, and high-frequency feedback filtering parameters, not only achieves effective separation of high and low frequency components, but also eliminates phase distortion during bidirectional filtering, ensuring the fidelity and continuity of the output signal of each channel. Thus, in the playback of audio variety shows, it can simultaneously present a full low-frequency atmosphere and clear high-frequency details, enhancing the overall sound field's sense of layering and immersion.

[0086] In one embodiment, the result of the second inverse filtering is segmented to obtain the high-frequency and low-frequency signals for each channel, including:

[0087] If the current frame is the first frame, when obtaining the second inverse filtering result of the current frame, the first nine bits of the second inverse filtering result are removed, and then the second inverse filtering result of the first 1024 sampling points is extracted to obtain the high-frequency signal and low-frequency signal of each channel; if the current frame is not the first frame, when obtaining the second inverse filtering result of the current frame, the second inverse filtering result of the first 1024 sampling points is extracted to obtain the high-frequency signal and low-frequency signal of each channel.

[0088] Specifically, if the current frame is the first frame, when obtaining the second inverse filtering result for that frame, the first nine zero-padding bits of the second inverse filtering result are first removed. Since there is no historical data at the beginning, zero padding in the first frame is equivalent to giving the IIR filter a "buffer," allowing it time to transition from its initial state (usually zero state) to a steady state. This avoids unnatural jumps or oscillations in the output signal at the boundaries, preventing distortion at the beginning and end of the output signal. Therefore, after re-filtering, the first nine bits of data need to be removed to eliminate invalid data. After removing invalid data, the first 1024 sampling points are extracted from the second inverse filtering result. These 1024 sampling points correspond to the effective data length of the filtering process, ensuring that the high-frequency and low-frequency signals output by each channel have a uniform duration and structure. If the current frame is not the first frame, i.e., it is in the normal processing stage, then when obtaining the second inverse filtering result, it is not necessary to remove the first nine bits of data; instead, the first 1024 sampling points are directly extracted to obtain the high-frequency and low-frequency signals for each channel.

[0089] As can be seen from the above, this application eliminates invalid data supplemented by IIR due to insufficient initial conditions of the filter by introducing an additional cutting step when the current frame is the first frame. At the same time, it directly extracts valid sampling points in non-first frame stages, which improves processing efficiency and ensures the consistency of high-frequency and low-frequency signals in each channel in terms of time length and structure, thereby improving the smoothness and stability of the overall audio stream.

[0090] In one embodiment, the process of downmixing high-frequency signals and low-frequency signals respectively includes:

[0091] Loudness downmixing is performed on the low-frequency signal of each channel to obtain a low-frequency stereo signal; channel center-side processing is performed on the high-frequency signal of each channel to obtain a sound field extension signal; HRTF downmixing is performed on the sound field extension signal to obtain a high-frequency stereo signal.

[0092] Specifically, such as Figure 2 As shown, low-frequency signals refer to the audio information in the corresponding low-frequency band of the PCM data obtained through bidirectional filtering, mainly reflecting the energy and spatial atmosphere of the sound. Loudness downmixing refers to combining the low-frequency signals from multiple channels into a single low-frequency stereo signal while maintaining overall loudness balance. This low-frequency stereo signal preserves the energy distribution of the original audio and, through weighted averaging of the low-frequency signals from different channels, makes the sound fuller when played in both ears. For example... Figure 2 After obtaining the high-frequency signal as shown, the formula is used. and The high-frequency signal of each channel undergoes channel center-side processing. Channel center-side processing is a stereo processing method that produces a sound field extension signal. This extended signal has a wider spatial distribution and can enhance the localization of high-frequency components in the stereo sound field. is the sound field expansion factor, ranging from [1,2]. These represent the left channel, left surround channel, left front top channel, and left rear top channel. This represents the right channel, right surround channel, right front top channel, and right rear top channel, with each pair corresponding to the other. In other words, the left channel, left surround channel, left front top channel, and left rear top channel are represented by [missing information - likely a specific symbol or symbol]. Perform center-side processing, and apply the following to the right channel, right surround channel, right front overhead channel, and right rear overhead channel respectively. Mid-side processing yields the sound field extension signal. The high-frequency signal refers to the audio information in the corresponding high-frequency band of the PCM data obtained through bidirectional filtering, primarily reflecting the clarity and detail of the sound. Finally, the sound field extension signal undergoes HRTF downmixing. HRTF is a filtering method that simulates the characteristics of human hearing, incorporating the influence of the head, auricle, and shoulders on sound propagation during stereo synthesis, thereby enhancing spatial awareness and directionality. After HRTF downmixing, a high-frequency stereo signal is obtained. This high-frequency stereo signal not only maintains the clarity and detail of the high-frequency components but also possesses a stronger sense of spatial immersion. Finally, as shown... Figure 2 The above will merge the high-frequency stereo signal and the low-frequency stereo signal to obtain the audio rendering signal of the current audio frame.

[0093] As shown above, this application ensures the fullness and balance of the low-frequency range by performing loudness downmixing on the low-frequency signal. By performing channel mid-side processing on the high-frequency signal, the width and layering of the sound field are expanded. Combined with HRTF downmixing, the spatial positioning and immersive effect of the high-frequency range are effectively improved. This results in an audio rendering signal that possesses both a rich low-frequency atmosphere and a delicate high-frequency spatial sense during playback, significantly enhancing sound quality and the user's listening experience.

[0094] The above describes an audio data processing method provided by an embodiment of this application. The following describes an audio data processing apparatus that performs the above.

[0095] Please see Figure 3 , Figure 3 This is a schematic diagram of an audio data processing device provided in an embodiment of this application. Figure 3 As shown, the audio data processing device includes:

[0096] Data acquisition module 401 is used to acquire PCM data of each channel of the current audio frame based on spatial audio algorithm;

[0097] The data update module 402 is used to store the PCM data of the current audio frame into the second area of ​​the buffer if the current audio frame is not the first frame. The buffer includes a first area and a second area, the second area is arranged after the first area, and the first area stores the PCM data of the previous audio frame.

[0098] The filtering module 403 is used to perform bidirectional filtering on the PCM data of the current audio frame based on the PCM data of the previous audio frame to obtain high-frequency signals and low-frequency signals, delete the PCM data of the previous audio frame stored in the first region, and move the PCM data of the current audio frame from the second region to the first region.

[0099] The downmixing module 404 is used to downmix the high-frequency signal and the low-frequency signal respectively to obtain the high-frequency stereo signal and the low-frequency stereo signal.

[0100] The data merging module 405 is used to merge high-frequency stereo signals and low-frequency stereo signals to obtain the audio rendering signal of the current audio frame.

[0101] In one embodiment, the data update module 402 is specifically used for:

[0102] The PCM data in the buffer is filtered to obtain a filtering result, and the forward buffer data is saved. The filtering result is then reverse-processed to obtain a reverse filtering result. The reverse filtering result is then subjected to a second filtering process to obtain a second filtering result, and the reverse buffer data is saved. The second filtering result is then subjected to a second reverse processing to obtain a second reverse filtering result. The second reverse filtering result is then segmented to obtain the high-frequency signal and low-frequency signal for each channel.

[0103] In one embodiment, the data update module 402 is specifically used to: if the current audio frame is the first frame, perform a first filtering process on the PCM data in the buffer to obtain the filtering result, and save the forward buffer data;

[0104] If the current audio frame is not the first frame, the PCM data in the buffer is subjected to a second filtering process to obtain a filtering result, and the first two digits of the filtering result are removed, and the forward buffer data is saved.

[0105] In one embodiment, the filtering module 403 is specifically used to: extract the last M bits of the PCM data of the previous audio frame to obtain input buffer data, where the length of M is the length of the filter parameters;

[0106] The PCM data of the previous audio frame is filtered to obtain the processing result, and the last M bits of the processing result are extracted to obtain the initial input conditions.

[0107] Input the input buffer data, initial input conditions, and PCM data of the current audio frame into the filter to obtain the output data of the current audio frame;

[0108] The initial input conditions are assigned to the first M bits of the output data to obtain the filtering result.

[0109] In one embodiment, the filtering module 403 is specifically used to: obtain the filter parameters for each channel of the current audio frame, the filter parameters including low-frequency input filter parameters, low-frequency feedback filter parameters, high-frequency input filter parameters and high-frequency feedback filter parameters;

[0110] Based on the PCM data of the previous audio frame, low-frequency input filtering parameters, and low-frequency feedback filtering parameters, the PCM data of the current audio frame is bidirectionally filtered to obtain the low-frequency signal of each channel.

[0111] Based on the PCM data of the previous audio frame, high-frequency input filtering parameters, and high-frequency feedback filtering parameters, the PCM data of the current audio frame is bidirectionally filtered to obtain the high-frequency signal of each channel.

[0112] In one embodiment, the filtering module 403 is specifically used to: if the current frame is the first frame, when the second inverse filtering result of the current frame is obtained, the first nine bits of the second inverse filtering result are cut off, and then the second inverse filtering result of the first 1024 sampling points is extracted to obtain the high-frequency signal and low-frequency signal of each channel.

[0113] If the current frame is not the first frame, then when obtaining the second inverse filtering result of the current frame, the first 1024 sampling points of the second inverse filtering result are extracted to obtain the high-frequency signal and low-frequency signal of each channel.

[0114] In one embodiment, the downmixing module 404 and the data merging module 405 are specifically used to: perform loudness downmixing processing on the low-frequency signal of each channel to obtain a low-frequency stereo signal;

[0115] The high-frequency signal of each channel is processed in the middle channel to obtain the sound field extension signal;

[0116] The sound field extension signal is downmixed using HRTF to obtain a high-frequency stereo signal.

[0117] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic of an electronic device suitable for implementing the audio data processing method in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0118] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0119] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0120] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the audio data processing methods provided in this application.

[0121] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the audio data processing methods provided in this application.

[0122] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0124] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0125] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An audio data processing method, characterized in that, include: PCM data for each channel of the current audio frame is obtained based on spatial audio algorithms to acquire sound flow information. The PCM data of the current audio frame is stored in a buffer, wherein the buffer includes a first region and a second region, and the second region is arranged after the first region; if the current audio frame is the first frame, the PCM data of the current audio frame is stored in the first region, and the first region stores the PCM data of the previous audio frame; if the current audio frame is not the first frame, the PCM data of the current audio frame is stored in the second region. Based on the PCM data of the previous audio frame, the PCM data of the current audio frame is subjected to bidirectional filtering, including forward filtering and reverse filtering, to obtain the high-frequency signal and low-frequency signal of each channel of the current audio frame. If the current audio frame is not the first frame, then the PCM data of the previous audio frame stored in the first region is deleted and the PCM data of the current audio frame is moved from the second region to the first region. The high-frequency signal and the low-frequency signal are respectively downmixed to obtain a high-frequency stereo signal and a low-frequency stereo signal; The high-frequency stereo signal and the low-frequency stereo signal are combined to obtain the audio rendering signal of the current audio frame.

2. The audio data processing method according to claim 1, characterized in that, The bidirectional filtering process includes: The PCM data in the buffer is filtered to obtain a filtering result, and the forward buffer data is saved. The filtering result is then reverse-processed to obtain a reverse filtering result. The reverse filtering result is then subjected to a second filtering process to obtain a second filtering result, and the reverse buffer data is saved. The second filtering result is then subjected to a second reverse processing to obtain a second reverse filtering result. The second reverse filtering result is then segmented to obtain the high-frequency signal and low-frequency signal for each channel.

3. The audio data processing method according to claim 2, characterized in that, The process of filtering the PCM data in the buffer to obtain the filtering result and saving the forward buffer data includes: If the current audio frame is the first frame, the PCM data in the buffer is subjected to a first filtering process to obtain the filtering result, and the forward buffer data is saved; If the current audio frame is not the first frame, the PCM data in the buffer is subjected to a second filtering process to obtain a filtering result, and the first two digits of the filtering result are removed, and the forward buffer data is saved.

4. The audio data processing method according to claim 3, characterized in that, The PCM data in the buffer is subjected to a second filtering process to obtain the filtering result, including: Extract the last M bits of the PCM data from the previous audio frame to obtain the input buffer data, where the length of M is the length of the filter parameters; The PCM data of the previous audio frame is filtered to obtain the processing result, and the last M bits of the processing result are extracted to obtain the initial input conditions. The input buffer data, the initial input conditions, and the PCM data of the current audio frame are input into the filter to obtain the output data of the current audio frame; The initial input conditions are assigned to the first M bits of the output data to obtain the filtering result.

5. The audio data processing method according to claim 1, characterized in that, Based on the PCM data of the previous audio frame, the PCM data of the current audio frame is bidirectionally filtered to obtain high-frequency and low-frequency signals, including: Obtain the filter parameters for each channel of the current audio frame, including low-frequency input filter parameters, low-frequency feedback filter parameters, high-frequency input filter parameters, and high-frequency feedback filter parameters; Based on the PCM data of the previous audio frame, the low-frequency input filtering parameters, and the low-frequency feedback filtering parameters, the PCM data of the current audio frame is bidirectionally filtered to obtain the low-frequency signal of each channel. Based on the PCM data of the previous audio frame, the high-frequency input filtering parameters, and the high-frequency feedback filtering parameters, the PCM data of the current audio frame is bidirectionally filtered to obtain the high-frequency signal of each channel.

6. The audio data processing method according to claim 2, characterized in that, The step of segmenting the secondary inverse filtering result to obtain the high-frequency and low-frequency signals for each channel includes: If the current audio frame is the first frame, when the second inverse filtering result of the current audio frame is obtained, the first nine bits of the second inverse filtering result are removed, and then the second inverse filtering result of the first 1024 sampling points is extracted to obtain the high-frequency signal and low-frequency signal of each channel. If the current audio frame is not the first frame, then when obtaining the secondary inverse filtering result of the current audio frame, the secondary inverse filtering result of the first 1024 sampling points of the secondary inverse filtering result is extracted to obtain the high-frequency signal and low-frequency signal of each channel.

7. The audio data processing method according to claim 1, characterized in that, The process of down-mixing the high-frequency signal and the low-frequency signal respectively includes: The low-frequency signal of each channel is downmixed to obtain a low-frequency stereo signal; The high-frequency signal of each channel is processed in the middle channel to obtain the sound field extension signal; The sound field extension signal is subjected to HRTF downmixing to obtain a high-frequency stereo signal.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the audio data processing method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the audio data processing method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the audio data processing method as described in any one of claims 1 to 7.

Images