Audio processing method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure provides an audio processing method and device, electronic equipment, computer readable storage medium and computer program product, relating to the technical field of audio and video processing. The method comprises: obtaining a to-be-processed audio, classifying the to-be-processed audio to obtain an audio type of the to-be-processed audio; performing audio quality detection processing on the to-be-processed audio, and determining a first code rate corresponding to the to-be-processed audio in combination with the audio type; determining an audio feature of the to-be-processed audio, performing code rate prediction processing based on the audio feature to obtain a second code rate corresponding to the to-be-processed audio; and determining a target code rate corresponding to the to-be-processed audio based on the first code rate and the second code rate. Through audio classification and audio quality prediction processing on the to-be-processed audio, the corresponding target code rate is determined, which can effectively guarantee the audio quality while reducing the required audio compression code rate in the transmission process as much as possible, thereby reducing the transmission bandwidth.

Description

Technical Field

[0001] This disclosure relates to the field of audio and video processing technology, and in particular to an audio processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Audio encoding refers to the process of compressing audio signals using encoding techniques during audio transmission. Its purpose is to reduce the bandwidth requirements of audio streaming media and the storage size of audio files. Compression methods can be divided into lossy compression and lossless compression. The transmission bandwidth of audio streaming media and the compression quality of audio are closely related to the audio compression bitrate. Audio compression bitrate refers to the number of bits of compressed audio transmitted per unit of time.

[0003] Generally, when using the same compression method, a higher audio compression bitrate results in better audio quality, but also requires greater bandwidth for transmission. However, once the audio compression bitrate reaches a certain value, the change in audio quality becomes imperceptible to human hearing, while the bandwidth required for transmission continues to increase. Therefore, minimizing the bandwidth required for transmission while maintaining the perceived audio quality has become a crucial task in audio coding. However, current methods using raw signal processing algorithms may suffer from insufficient accuracy, affecting the determination of the audio bitrate. Summary of the Invention

[0004] This disclosure provides an audio processing method, apparatus, electronic device, computer-readable storage medium, and computer program product to at least solve the problem of insufficient accuracy in determining audio bitrate using deep learning technology in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, an audio processing method is provided, comprising: acquiring an audio to be processed; classifying the audio to be processed to obtain an audio type; performing sound quality detection processing on the audio to be processed and determining a first bitrate corresponding to the audio to be processed based on the audio type; determining audio features of the audio to be processed and performing bitrate prediction processing based on the audio features to obtain a second bitrate corresponding to the audio to be processed; and determining a target bitrate corresponding to the audio to be processed based on the first bitrate and the second bitrate.

[0006] In one exemplary embodiment of this disclosure, classifying the audio to be processed to obtain the audio type includes: acquiring a pre-constructed audio classification network; the audio classification network being trained based on the Mel spectrum feature values ​​and audio type labels corresponding to multiple sample audio segments; and classifying the audio to be processed through the audio classification network to obtain the audio type.

[0007] In an exemplary embodiment of this disclosure, the audio classification network is obtained based on the following steps: acquiring a training audio sample set and a pre-constructed initial classification network; the training audio sample set includes multiple sample audio segments and audio type labels for each sample audio segment; performing time-domain transformation processing on each sample audio segment to obtain time-domain audio segments corresponding to each sample audio segment; calculating Mel frequency cepstral coefficients for each time-domain audio segment to obtain initial Mel spectrum values ​​for each time-domain audio segment; determining Mel spectrum feature values ​​corresponding to each of the multiple time-domain audio segments based on each initial Mel spectrum value; and training the initial classification network based on the audio type labels corresponding to each of the multiple sample audio segments and the Mel spectrum feature values ​​to obtain the audio classification network.

[0008] In an exemplary embodiment of this disclosure, the audio quality detection processing includes effective bandwidth detection and audio complexity detection; the step of performing audio quality detection processing on the audio to be processed and determining the first bitrate corresponding to the audio to be processed in combination with the audio type includes: performing effective bandwidth detection on the audio to be processed to obtain the effective bandwidth of the audio to be processed; performing audio complexity detection on the audio to be processed to obtain the spectral richness of the audio to be processed; and performing fusion processing on the effective bandwidth, the spectral richness and the audio type to obtain the first bitrate.

[0009] In an exemplary embodiment of this disclosure, the step of detecting the effective bandwidth of the audio to be processed to obtain the effective bandwidth of the audio to be processed includes: obtaining the time domain interval in which the audio to be processed is located, calculating the frequency band energy of each frequency band of the audio to be processed within the time domain interval; determining the maximum frequency band energy among the multiple frequency band energies, and determining the audio bandwidth frequency band point of the audio to be processed based on the maximum frequency band energy; obtaining a pre-configured fast Fourier transform length and the audio sampling rate corresponding to the audio to be processed; and determining the effective bandwidth based on the audio bandwidth frequency band point, the fast Fourier transform length, and the audio sampling rate.

[0010] In an exemplary embodiment of this disclosure, the step of performing audio complexity detection on the audio to be processed to obtain the spectral richness of the audio to be processed includes: determining the total number of frequency bands corresponding to the audio to be processed; obtaining the frequency band energy of each frequency band of the audio to be processed in the time domain interval, and the frame sequence corresponding to the audio to be processed; the frame sequence includes multiple audio frames, each audio frame having a corresponding frame number; determining the number of frequency band truncations based on the multiple frequency band energies and the frame numbers of each audio frame; and determining the spectral richness based on the number of frequency band truncations and the total number of frequency bands.

[0011] In an exemplary embodiment of this disclosure, the step of fusing the effective bandwidth, the spectral richness, and the audio type to obtain the first bitrate includes: obtaining the audio sampling rate, upper bandwidth limit, and lower bandwidth limit corresponding to the audio to be processed; determining an effective bandwidth coefficient based on the audio sampling rate and the effective bandwidth; determining an upper bitrate bandwidth limit and a lower bitrate bandwidth limit for the audio to be processed based on the effective bandwidth coefficient, the upper bandwidth limit, and the lower bandwidth limit; and determining the first bitrate based on the spectral richness, the upper bitrate bandwidth limit, and the lower bitrate bandwidth limit.

[0012] In an exemplary embodiment of this disclosure, the step of performing bitrate prediction processing based on the audio features to obtain a second bitrate corresponding to the audio to be processed includes: acquiring a pre-constructed audio quality prediction network; the audio quality prediction network being trained based on training sample pairs consisting of sample audio segments and sample audio quality; the sample audio quality being obtained by performing multi-bitrate encoding processing on the sample audio segments; inputting the audio features into the audio quality prediction network for audio quality prediction processing to obtain the target audio quality of the audio to be processed; acquiring a pre-constructed bitrate-audio quality curve, and determining the second bitrate corresponding to the target audio quality based on the bitrate-audio quality curve.

[0013] In one exemplary embodiment of this disclosure, the audio quality prediction network is constructed through the following steps: obtaining a pre-constructed initial neural network and a training audio sample set; the training audio sample set includes multiple sample audio segments; performing multi-bitrate encoding processing on the sample audio segments to obtain multiple encoded audio segments corresponding to the sample audio segments; performing audio quality evaluation processing on the multiple encoded audio segments to obtain the corresponding sample audio quality; extracting features from the sample audio segments to obtain sample audio features corresponding to the sample audio segments; using the sample audio features of the sample audio segments and the corresponding sample audio quality as training sample pairs to train the initial neural network to obtain the audio quality prediction network.

[0014] In one exemplary embodiment of this disclosure, determining the target bitrate corresponding to the audio to be processed based on the first bitrate and the second bitrate includes: obtaining the current network status; and determining the target bitrate according to the current network status, the first bitrate, and the second bitrate.

[0015] In one exemplary embodiment of this disclosure, determining the target bitrate based on the current network state, the first bitrate, and the second bitrate includes: if the current network state is a first network state, then determining the target bitrate based on the audio type, the first bitrate, and the second bitrate; the first network state is when the network transmission rate is within a preset rate range.

[0016] In one exemplary embodiment of this disclosure, determining the target bitrate based on the current network state, the first bitrate, and the second bitrate further includes: if the current network state is a second network state, determining the larger bitrate value between the first bitrate and the second bitrate; the second network state is a network transmission rate greater than the maximum value corresponding to the preset rate interval; and using the larger bitrate value as the target bitrate.

[0017] In one exemplary embodiment of this disclosure, determining the target bitrate based on the current network state, the first bitrate, and the second bitrate further includes: if the current network state is a third network state, determining the smaller bitrate value between the first bitrate and the second bitrate; the third network state is a network transmission rate greater than the minimum value corresponding to the preset rate interval; and using the smaller bitrate value as the target bitrate.

[0018] According to a second aspect of the present disclosure, an audio processing apparatus is provided, comprising: an audio type determination module, configured to acquire audio to be processed, classify the audio to be processed, and obtain an audio type of the audio to be processed; a first bitrate determination module, configured to perform sound quality detection processing on the audio to be processed, and determine a first bitrate corresponding to the audio to be processed in combination with the audio type; a second bitrate determination module, configured to determine audio features of the audio to be processed, and perform bitrate prediction processing based on the audio features to obtain a second bitrate corresponding to the audio to be processed; and a target bitrate determination module, configured to determine a target bitrate corresponding to the audio to be processed based on the first bitrate and the second bitrate.

[0019] In one exemplary embodiment of this disclosure, the audio type determination module includes an audio type determination unit, configured to: acquire a pre-constructed audio classification network; the audio classification network is trained based on the Mel spectrum feature values ​​and audio type labels corresponding to multiple sample audio segments; and classify the audio to be processed through the audio classification network to obtain the audio type.

[0020] In one exemplary embodiment of this disclosure, the audio type determination module further includes a classification network training unit, configured to: acquire a training audio sample set and a pre-constructed initial classification network; the training audio sample set includes multiple sample audio segments and audio type labels for each sample audio segment; perform time-domain transformation processing on each sample audio segment to obtain a time-domain audio segment corresponding to each sample audio segment; calculate Mel frequency cepstral coefficients for each time-domain audio segment to obtain an initial Mel spectrum value for each time-domain audio segment; determine Mel spectrum feature values ​​corresponding to each of the multiple time-domain audio segments based on each initial Mel spectrum value; and train the initial classification network based on the audio type labels corresponding to each of the multiple sample audio segments and the Mel spectrum feature values ​​to obtain the audio classification network.

[0021] In an exemplary embodiment of this disclosure, the first bitrate determination module includes a first bitrate determination unit, configured to: perform effective bandwidth detection on the audio to be processed to obtain the effective bandwidth of the audio to be processed; perform audio complexity detection on the audio to be processed to obtain the spectral richness of the audio to be processed; and perform fusion processing on the effective bandwidth, the spectral richness and the audio type to obtain the first bitrate.

[0022] In an exemplary embodiment of this disclosure, the first bitrate determination unit includes an effective bandwidth determination subunit, configured to: acquire the time domain interval where the audio to be processed is located; calculate the frequency band energy of each frequency band of the audio to be processed within the time domain interval; determine the maximum frequency band energy among the multiple frequency band energies; determine the audio bandwidth frequency band point of the audio to be processed based on the maximum frequency band energy; acquire a pre-configured fast Fourier transform length and the audio sampling rate corresponding to the audio to be processed; and determine the effective bandwidth based on the audio bandwidth frequency band point, the fast Fourier transform length, and the audio sampling rate.

[0023] In an exemplary embodiment of this disclosure, the first bitrate determination unit includes a complexity determination subunit, configured to: determine the total number of frequency bands corresponding to the audio to be processed; obtain the frequency band energy of each frequency band of the audio to be processed in the time domain interval, and the frame sequence corresponding to the audio to be processed; the frame sequence includes multiple audio frames, each audio frame having a corresponding frame number; determine the number of frequency band truncations based on the multiple frequency band energies and the frame numbers of each audio frame; and determine the spectral richness based on the number of frequency band truncations and the total number of frequency bands.

[0024] In an exemplary embodiment of this disclosure, the first bitrate determination unit includes a first bitrate determination subunit, configured to: acquire the audio sampling rate, bandwidth upper limit, and bandwidth lower limit corresponding to the audio to be processed; determine an effective bandwidth coefficient based on the audio sampling rate and the effective bandwidth; determine the bitrate bandwidth upper limit and bitrate bandwidth lower limit of the audio to be processed based on the effective bandwidth coefficient, the bandwidth upper limit, and the bandwidth lower limit; and determine a first bitrate based on the spectral richness, the bitrate bandwidth upper limit, and the bitrate bandwidth lower limit.

[0025] In one exemplary embodiment of this disclosure, the second bitrate determination module includes a second bitrate determination unit, configured to: acquire a pre-constructed audio quality prediction network; the audio quality prediction network is trained based on training sample pairs consisting of sample audio segments and sample audio quality; the sample audio quality is obtained by performing multi-bitrate encoding processing on the sample audio segments; input the audio features into the audio quality prediction network for audio quality prediction processing to obtain the target audio quality of the audio to be processed; acquire a pre-constructed bitrate-quality curve, and determine the second bitrate corresponding to the target audio quality based on the bitrate-quality curve.

[0026] In one exemplary embodiment of this disclosure, the second bitrate determination module includes a prediction network determination unit, configured to: acquire a pre-constructed initial neural network and a training audio sample set; the training audio sample set includes multiple sample audio segments; perform multi-bitrate encoding processing on the sample audio segments to obtain multiple encoded audio segments corresponding to the sample audio segments; perform sound quality evaluation processing on the multiple encoded audio segments to obtain corresponding sample audio sound quality; perform feature extraction on the sample audio segments to obtain sample audio features corresponding to the sample audio segments; and use the sample audio features of the sample audio segments and the corresponding sample audio sound quality as training sample pairs to train the initial neural network to obtain the sound quality prediction network.

[0027] In one exemplary embodiment of this disclosure, the target bitrate determination module includes a target bitrate determination unit, configured to: obtain the current network state; and determine the target bitrate based on the current network state, the first bitrate, and the second bitrate.

[0028] In an exemplary embodiment of this disclosure, the target bitrate determination unit includes a first target bitrate determination subunit, configured to: if the current network state is a first network state, determine the target bitrate based on the audio type, the first bitrate, and the second bitrate; the first network state is that the network transmission rate is within a preset rate range.

[0029] In an exemplary embodiment of this disclosure, the target bitrate determination unit further includes a second target bitrate determination subunit, configured to: if the current network state is a second network state, determine the larger bitrate value between the first bitrate and the second bitrate; the second network state is a network transmission rate greater than the maximum value corresponding to the preset rate interval; and use the larger bitrate value as the target bitrate.

[0030] In an exemplary embodiment of this disclosure, the target bitrate determination unit further includes a third target bitrate determination subunit, configured to: if the current network state is a third network state, determine the smaller bitrate value between the first bitrate and the second bitrate; the third network state is a network transmission rate greater than the minimum value corresponding to the preset rate interval; and use the smaller bitrate value as the target bitrate.

[0031] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the audio processing method described in any of the preceding claims.

[0032] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the audio processing method described in any of the preceding claims.

[0033] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the audio processing method described in any one of the preceding claims.

[0034] The technical solutions provided by the embodiments of this disclosure offer at least the following advantages: On the one hand, calculating the target bitrate by fusing the first bitrate and the second bitrate can effectively improve the accuracy of the determined bitrate result. On the other hand, by performing audio classification and audio quality prediction processing on the audio to be processed separately to obtain the first bitrate and the second bitrate, the audio compression bitrate required during transmission can be reduced as much as possible while effectively ensuring audio quality, thereby reducing transmission bandwidth.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0037] Figure 1 This is a flowchart illustrating an audio processing method according to an exemplary embodiment.

[0038] Figure 2 This is a system block diagram illustrating the determination of the target bitrate of the audio to be processed, according to an exemplary embodiment.

[0039] Figure 3 This is a system block diagram illustrating a signal processing method for determining a first bitrate of the audio to be processed, according to an exemplary embodiment.

[0040] Figure 4 This is a schematic diagram illustrating the determination of the bitrate of an audio segment based on a bitrate-quality curve according to an exemplary embodiment.

[0041] Figure 5 This is a system block diagram illustrating, according to an exemplary embodiment, a method for predicting the second bitrate of an audio file using deep learning techniques.

[0042] Figure 6 This is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment.

[0043] Figure 7 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown schematically. Detailed Implementation

[0044] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0045] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0046] In the process of audio transmission, minimizing the bandwidth required for transmission while ensuring that the audio quality perceived by human hearing remains unchanged is an important task in the audio coding industry.

[0047] One implementation scheme utilizes signal processing algorithms to set the audio bitrate corresponding to an audio segment based on its classification results and audio feature parameters, using manually defined rules to reduce the overall bitrate while maintaining audio quality. However, signal processing algorithms have the following drawbacks: First, the manually defined rule method can only set the audio bitrate based on a few features and limited strategies, making it difficult to cover all audio content scenarios. Second, in real-world scenarios, different audio quality goals may exist to balance bandwidth costs and user experience in different scenarios. Setting different manually defined rules for different audio quality goals is relatively cumbersome and time-consuming.

[0048] Another approach utilizes deep learning to optimize audio bitrate based on encoded audio quality prediction. This involves using a neural network to learn the audio quality metrics of different audio content encoded at different bitrates. This allows for setting the optimal bitrate for different audio content with any target audio quality, thus achieving the goal of optimizing audio bitrate. However, this approach, while based on deep learning, suffers from insufficient accuracy due to limitations in complexity and network performance.

[0049] Figure 1 This is a flowchart illustrating an audio processing method according to an exemplary embodiment, such as... Figure 1 As shown, the audio processing method can be used in computer devices. The computer devices described in this disclosure may include mobile terminal devices such as mobile phones, tablets, laptops, PDAs, and personal digital assistants (PDAs), as well as fixed terminal devices such as desktop computers. This exemplary embodiment illustrates the application of the method to a computer device; it is understood that the method can also be applied to a server, and further to a system including both computer devices and a server, and implemented through the interaction between the computer device and the server. Specifically, it includes the following steps.

[0050] In step S110, the audio to be processed is obtained, and the audio to be processed is classified to obtain the audio type of the audio to be processed.

[0051] In one exemplary embodiment of this disclosure, the audio to be processed may be an audio segment to be encoded. Classification processing may be a process of determining the specific type of the audio to be processed. The audio type may be a specific type of the audio to be processed; for example, the audio type may include noise, speech, or music.

[0052] When the audio to be processed is acquired, it can be classified. For example, an audio classification network can be used to classify the audio and determine its corresponding audio type. For instance, the audio to be processed can be an audio signal x of length T, input to the audio classification network for classification to determine its corresponding audio type. After determining the corresponding audio type, the upper and lower bandwidth limits for the audio to be processed can be determined. Combining this with the previous latency results, the bandwidth limits for various audio types are determined as follows: noise can be 32-48 kbps, speech can be 48-64 kbps, and music can be 64-96 kbps.

[0053] In step S120, the audio to be processed is subjected to audio quality detection processing, and the first bit rate corresponding to the audio to be processed is determined in combination with the audio type.

[0054] In one exemplary embodiment of this disclosure, the audio quality detection processing may be a process of performing audio quality detection on the audio to be processed to determine the effective bandwidth and spectral richness corresponding to the audio to be processed. The first bitrate may be the bitrate value of the audio to be processed determined by a signal processing algorithm (such as an audio quality detection processing scheme).

[0055] refer to Figure 2 , Figure 2 This is a system block diagram illustrating the determination of the target bitrate of the audio to be processed according to an exemplary embodiment. In step S210, the audio can be processed for bitrate prediction using conventional bitrate prediction methods. For example, conventional bitrate prediction can be performed using signal processing algorithms. For the acquired audio to be processed, audio quality detection processing can be performed. The purpose of audio quality detection processing is mainly to determine the effective bandwidth and spectral richness of the audio to be processed. For example, signal processing algorithms can be used to perform audio quality detection processing on the audio to be processed to determine the corresponding effective bandwidth and spectral richness. When the effective bandwidth of the audio to be processed is higher, the audio compression bitrate is higher; when the spectral richness of the audio to be processed is richer, the audio compression bitrate is higher.

[0056] In step S130, the audio features of the audio to be processed are determined, and bitrate prediction processing is performed based on the audio features to obtain the second bitrate corresponding to the audio to be processed.

[0057] In one exemplary embodiment of this disclosure, the audio features may be features of the entire audio segment extracted from the audio to be processed. The bitrate prediction processing may be a process of predicting the encoded bitrate of the audio to be processed. The second bitrate may be the bitrate value determined by the bitrate prediction processing of the audio to be processed.

[0058] Continue to refer to Figure 2In step S220, feature extraction is performed on the audio to be processed, and the obtained multiple features are concatenated to obtain the corresponding multidimensional feature vector. For example, multiple frame features can be extracted for each audio frame of the audio to be processed, and then the statistical mean and variance of the frame features are used as the features of the entire audio segment. Typical frame features may include Mel-Frequency Cepstral Coefficients (MFCCs), subband energy ratio, etc.; where the subband energy ratio can be the ratio of the energy of the subbands obtained by dividing the spectrum to the total energy of the spectrum.

[0059] Alternatively, features can be extracted directly from the entire audio segment. For example, the effective bandwidth and audio richness of the audio to be processed can both be used as audio features. For instance, for an input audio segment x, multiple features are concatenated together to obtain a D-dimensional feature vector. If the audio to be processed is a 3-second audio segment, a 40-dimensional audio feature vector can be extracted after feature extraction.

[0060] After determining the audio features obtained through feature extraction, in step S230, bitrate prediction processing can be performed based on the audio features to obtain the second bitrate corresponding to the audio to be processed. For example, a deep learning bitrate prediction method can be used to determine the second bitrate corresponding to the audio to be processed.

[0061] In step S140, the target bitrate corresponding to the audio to be processed is determined based on the first bitrate and the second bitrate.

[0062] In one exemplary embodiment of this disclosure, the target bitrate may be the bitrate value used when encoding the audio to be processed, denoted as EB.

[0063] Continue to refer to Figure 2 In step S240, after obtaining the first bitrate and the second bitrate, the first bitrate and the second bitrate can be fused to obtain the target bitrate used for encoding the audio to be processed. The process of determining the target bitrate EB can be shown in Formula 1.

[0064] (Formula 1)

[0065] According to the audio processing method in this example embodiment, on the one hand, the target bitrate is calculated by fusing the first bitrate and the second bitrate, which can effectively improve the accuracy of the determined bitrate result. On the other hand, by performing audio classification and sound quality prediction processing on the audio to be processed to obtain the first bitrate and the second bitrate, the audio compression bitrate required during transmission can be reduced as much as possible while effectively ensuring audio quality, thereby reducing the transmission bandwidth.

[0066] The audio processing method in this example embodiment will be further explained below.

[0067] In one exemplary embodiment of this disclosure, classifying the audio to be processed to obtain the audio type includes: acquiring a pre-constructed audio classification network; training the audio classification network based on the Mel spectrum feature values ​​and audio type labels corresponding to multiple sample audio segments; and classifying the audio to be processed through the audio classification network to obtain the audio type.

[0068] The audio classification network can be a network structure used to classify the audio to be processed into different audio types. The sample audio segments can be audio segments used as training samples to train the audio classification network. The Mel spectrum feature values ​​can be specific numerical values ​​of the Mel spectrum-related features of the sample audio segments; for example, the Mel spectrum typically includes time-domain-frequency domain information, perceptually related amplitude information, and perceptually related frequency domain information. The audio type label can be used to identify the specific audio type of the sample audio segment.

[0069] refer to Figure 3 , Figure 3 This is a system block diagram illustrating a signal processing method for determining the first bit rate of the audio to be processed, according to an exemplary embodiment. In step S310, audio classification processing is performed on the audio to be processed. When classifying the audio, an audio classification network can be used, and the resulting audio type label 310 can include noise, speech, and music, etc. The audio classification network can be trained based on data pairs composed of Mel-spectrum feature values ​​corresponding to sample audio segments and their corresponding audio type labels. When training the audio classification network, multiple sample audio segments can be acquired, and a corresponding training audio sample set can be constructed based on the Mel-spectrum feature values ​​and audio type labels corresponding to each of the multiple sample audio segments. Furthermore, the Mel-spectrum feature values ​​corresponding to each of the acquired sample audio segments are used as input features, and the audio type labels corresponding to each of the multiple sample audio segments are used as the learning target of the network to train the model, thereby obtaining the audio classification network.

[0070] By inputting the audio to be processed into a trained audio classification network, the corresponding audio type can be obtained. Simultaneously, the upper and lower bandwidth limits for the audio to be processed can be determined, denoted as B respectively. max With B min In subsequent processing, the first bitrate corresponding to the audio to be processed can be determined by combining the audio type.

[0071] In one exemplary embodiment of this disclosure, the audio classification network is obtained based on the following steps: acquiring a training audio sample set and a pre-constructed initial classification network; the training audio sample set includes multiple sample audio segments and audio type labels for each sample audio segment; performing time-domain transformation processing on each sample audio segment to obtain time-domain audio segments corresponding to each sample audio segment; calculating Mel frequency cepstral coefficients for each time-domain audio segment to obtain initial Mel spectrum values ​​for each time-domain audio segment; determining Mel spectrum feature values ​​corresponding to each of the multiple time-domain audio segments based on each initial Mel spectrum value; and training the initial classification network based on the audio type labels and Mel spectrum feature values ​​corresponding to each of the multiple sample audio segments to obtain the audio classification network.

[0072] The training audio sample set can be the set used to train the initial classification network to obtain the audio classification network. The training audio sample set can consist of data pairs composed of multiple sample audio segments and their corresponding audio type labels. The initial classification network can be the initially constructed classification network structure. The sample audio segments can be the audio segments included in the training audio sample set. The audio type labels can be used to identify the audio type corresponding to the sample audio segments. The time-domain transformation processing can be performing a short-time Fourier transform on the sample audio segments to analyze them in the time domain. The time-domain audio segments can be obtained by applying a window function to the sample audio segments during the short-time Fourier transform process, decomposing the entire time-domain process into multiple small segments of equal length.

[0073] Mel-frequency cepstrum coefficients can be the coefficients that make up the Mel-frequency cepstrum; where the Mel-frequency cepstrum is a linear transformation of the logarithmic energy spectrum based on a nonlinear Mel scale of sound frequency. The initial values ​​of the Mel spectrum can be the initial feature values ​​obtained by extracting Mel-frequency cepstrum coefficient features from a time-domain audio segment. Mel spectrum feature values ​​can be the feature values ​​obtained by averaging the initial Mel spectrum values. The audio classification network can be the network structure used to classify the audio to be processed.

[0074] When training an audio classification network, a pre-constructed training audio sample set can be obtained. The training audio sample set can include multiple sample audio segments. For example, a sample audio segment can be an original audio signal x of length T, denoted as x(t), where t represents time and 0 < t ≤ T. Perform time-domain transformation processing on each sample audio segment in the training audio sample set to obtain the corresponding time-domain audio segments. For example, the Short-Time Fourier Transform (STFT) can be performed on each sample audio segment separately. Then, each sample audio segment x(t) can be represented in the time-frequency domain as shown in Equation 2:

[0075] (Equation 2)

[0076] where n is the frame sequence, 0 < n ≤ N, and N is the total number of frames; k is the central frequency sequence, 0 < k ≤ K, and K is the total number of frequency bands.

[0077] After obtaining the time-domain audio segments corresponding to the sample audio segments, the Mel Frequency Cepstral Coefficients (MFCC) can be calculated for the time-domain audio segments to obtain the initial Mel spectra values of the multiple time-domain audio segments. Specifically, it is shown in Equation 3.

[0078] (Equation 3)

[0079] where n is the frame sequence, 0 < n ≤ N, and N is the total number of frames; k is the central frequency sequence, 0 < k ≤ K, and K is the total number of frequency bands; mfcc() is a function for calculating the Mel Frequency Cepstral Coefficients for the time-domain audio segments.

[0080] After obtaining the multiple initial Mel spectra values, the Mel spectrum eigenvalue corresponding to the time-domain audio segment can be determined based on the multiple initial Mel spectra values. For example, the average value of the multiple initial Mel spectra values can be taken in the time dimension to obtain the Mel spectrum eigenvalue, as shown in Equation 4.

[0081] (Equation 4)

[0082] where can be the initial Mel spectra values corresponding to each time-domain audio segment; ave() can be an average value calculation function.

[0083] In other exemplary embodiments of the present disclosure, other calculation processes can also be performed on the initial Mel spectra values to obtain the corresponding Mel spectrum eigenvalues. For example, the median value of the multiple initial Mel spectra values can be taken in the time dimension as the corresponding Mel spectrum eigenvalue. The present disclosure does not make any special limitations on this.

[0084] After calculating the Mel spectrum feature value corresponding to the sample audio segment, the Mel spectrum feature value Mel(p) can be used as the input feature of the initial classification network. Each sample audio segment is labeled with its own corresponding audio type label. The initial classification network will determine the audio type label corresponding to each sample audio segment as the target of network learning. The network structure is designed, and the initial classification network is trained based on the training audio sample set to obtain the audio classification network used for audio classification.

[0085] In one exemplary embodiment of this disclosure, the audio quality detection processing includes effective bandwidth detection and audio complexity detection; performing audio quality detection processing on the audio to be processed and determining the first bitrate corresponding to the audio to be processed in combination with the audio type includes: performing effective bandwidth detection on the audio to be processed to obtain the effective bandwidth of the audio to be processed; performing audio complexity detection on the audio to be processed to obtain the spectral richness of the audio to be processed; and performing fusion processing on the effective bandwidth, spectral richness and audio type to obtain the first bitrate.

[0086] Effective bandwidth detection can be a process for determining the effective bandwidth of the audio signal to be processed. Effective bandwidth, also known as effective frequency band width, refers to the frequency range containing the main harmonic components. Audio complexity detection can be a process for detecting the complexity of the audio signal to be processed. Spectral richness can be the sparsity of the audio signal to be processed.

[0087] Continue to refer to Figure 3 In step S320, audio quality detection processing is performed on the audio to be processed. When calculating the bitrate of the audio to be processed using a signal processing algorithm, audio quality detection processing can be performed. This processing may include effective bandwidth detection and audio complexity detection to obtain the corresponding audio detection result 320. After effective bandwidth detection, the corresponding effective bandwidth is determined; after audio complexity detection, the corresponding spectral richness is determined. The obtained effective bandwidth, spectral richness, and audio type of the audio to be processed are then fused to obtain the first bitrate corresponding to the audio to be processed.

[0088] In one exemplary embodiment of this disclosure, effective bandwidth detection is performed on the audio to be processed to obtain the effective bandwidth of the audio to be processed, including: obtaining the time domain interval where the audio to be processed is located, calculating the frequency band energy of each frequency band of the audio to be processed in the time domain interval; determining the maximum frequency band energy among multiple frequency band energies, and determining the audio bandwidth frequency band point of the audio to be processed based on the maximum frequency band energy; obtaining a pre-configured fast Fourier transform length and the audio sampling rate corresponding to the audio to be processed; and determining the effective bandwidth based on the audio bandwidth frequency band point, the fast Fourier transform length, and the audio sampling rate.

[0089] Among them, the time domain interval can be the time domain range where the audio signal of the audio to be processed is located. The band energy refers to the energy corresponding to the specified frequency band extracted within the frequency spectrum. The maximum band energy can be the maximum value among multiple band energies. The audio bandwidth frequency band point can be the minimum position point in the audio to be processed where the difference between the maximum band energy and the preset energy is greater than the band energy. The fast Fourier transform (FFT) can be a general term for an efficient and fast calculation method for calculating the discrete Fourier transform (DFT) using a computer. The fast Fourier transform length can be the calculation factor used in the DFT calculation formula. The audio sampling rate can be how much the recording device samples the analog signal (such as the audio signal of the audio to be processed) per unit time.

[0090] When performing effective bandwidth detection on the audio to be processed, it can be based on the time domain audio segment corresponding to the audio to be processed. First, the time domain interval corresponding to the audio to be processed can be determined. For example, the time domain interval of the audio to be processed can be 0 < t ≤ T; where t is the duration of the audio to be processed. When the time domain interval where the audio to be processed is located is determined, the band energy of each frequency band of the audio to be processed within the above time domain interval during the T period can be calculated, as shown in Formula 5 specifically.

[0091] (Formula 5)

[0092] Among them, abs() is a function to find the absolute value of data; ave() is a function to calculate the average value; log() is a logarithmic function; X(n,k) can be the representation form of the audio to be processed within the time domain interval; n is the frame sequence, 0 < n ≤ N, and N is the total number of frames; k is the central frequency sequence, 0 < k ≤ K, and K is the total number of frequency bands.

[0093] After determining multiple band energies, the maximum band energy among the multiple band energies can be determined, as shown in Formula 6 specifically.

[0094] (Formula 6)

[0095] Among them, k is the central frequency sequence, 0 < k ≤ K, and K is the total number of frequency bands; max() is a function to find the maximum value.

[0096] After determining the maximum band energy, the audio bandwidth frequency band point of the audio to be processed can be determined according to the maximum band energy, as shown in Formula 7 specifically.

[0097] (Formula 7)

[0098] Then, the effective bandwidth of the audio to be processed is determined based on the audio bandwidth frequency point, the length of the fast Fourier transform, and the audio sampling rate, as shown in Formula 8.

[0099] (Formula 8)

[0100] Where EB can be the effective bandwidth of the audio to be processed, index_k is the audio bandwidth frequency point, fft_len is the FFT length, and Fs is the audio sampling rate. Through the above processing steps, the effective bandwidth corresponding to the audio to be processed can be determined.

[0101] In one exemplary embodiment of this disclosure, audio complexity detection is performed on the audio to be processed to obtain the spectral richness of the audio, including: determining the total number of frequency bands corresponding to the audio to be processed; obtaining the frequency band energy of each frequency band of the audio to be processed in the time domain interval, and the frame sequence corresponding to the audio to be processed; the frame sequence includes multiple audio frames, each audio frame having a corresponding frame number; determining the number of frequency band energy truncations based on the multiple frequency band energies and the frame numbers of each audio frame; and determining the spectral richness based on the number of frequency band energy truncations and the total number of frequency bands.

[0102] The total number of frequency bands can be the number of FFT frequency bands of the audio to be processed. The frame sequence can be the sequence of audio frames corresponding to the audio to be processed within the corresponding time domain interval. The audio frame sequence includes multiple audio frames. The frame number can be the number corresponding to each audio frame; for example, in the audio frame sequence, the audio frames can be numbered in ascending order. The frequency band energy can be the capability corresponding to each frequency band. The frequency band energy cut-off quantity can be the number of bins that represent the minimum ratio of the input audio energy to the preset value.

[0103] When calculating the audio complexity of the audio to be processed, the spectral richness / spectral sparsity index can be used to determine the audio complexity, thereby reducing the bitrate. The audio to be processed can correspond to a frame sequence composed of multiple audio frames in the corresponding time domain interval. Based on the frequency band energy of each audio frame in the frame sequence and the frame number of each audio frame, the corresponding frequency band energy truncation quantity can be determined. Specifically, audio richness (also known as spectral sparsity) can be defined as shown in Formula 9.

[0104] (Formula 9)

[0105] (Formula 10)

[0106] (Formula 11)

[0107] Where N is the number of FFT frequency bands, and n can be the corresponding number of each frequency band; C can be the number of frequency band energy cuts, and C can be the minimum value that makes Formula 10 true, that is, the minimum number of bins representing the 95% power value of the input audio, which can be calculated by Formula 11; S(n) can represent the frequency band energy. Through the above processing steps, the spectral richness corresponding to the audio to be processed can be determined.

[0108] In one exemplary embodiment of this disclosure, the effective bandwidth, spectral richness, and audio type are fused to obtain a first bitrate, including: obtaining the audio sampling rate, upper bandwidth limit, and lower bandwidth limit corresponding to the audio to be processed; determining the effective bandwidth coefficient based on the audio sampling rate and effective bandwidth; determining the upper and lower bitrate bandwidth limits of the audio to be processed based on the effective bandwidth coefficient, upper bandwidth limit, and lower bandwidth limit; and determining the first bitrate based on the spectral richness, upper bitrate bandwidth limit, and lower bitrate bandwidth limit.

[0109] The upper bandwidth limit can be the maximum value of the information interval formed by the intelligible information of a certain type of audio. The lower bandwidth limit can be the minimum value of the information interval formed by the intelligible information of a certain type of audio. The effective bandwidth coefficient can be a calculated coefficient used to determine the effective bandwidth based on the audio sampling rate and the effective bandwidth. The upper bitrate bandwidth limit can be the maximum bitrate bandwidth obtained by calculating the upper bandwidth limit and the effective bandwidth coefficient. The lower bitrate bandwidth limit can be the minimum bitrate bandwidth obtained by calculating the upper bandwidth limit and the effective bandwidth coefficient. The spectral sparsity coefficient can be a coefficient used to characterize the spectral sparsity of the audio to be processed.

[0110] Continue to refer to Figure 3 In step S330, bitrate allocation processing is performed based on the obtained audio type 310 and audio quality detection result 320 to obtain the first bitrate. After determining the effective bandwidth corresponding to the audio to be processed, the effective bandwidth coefficient can be determined according to the effective bandwidth and the audio sampling rate. The calculation process of the effective bandwidth coefficient is shown in Formula 12.

[0111] (Formula 12)

[0112] in, It can be the effective bandwidth coefficient of the audio to be processed; EB can be the effective bandwidth of the audio to be processed; Fs is the audio sampling rate.

[0113] After determining the audio type corresponding to the audio to be processed, the upper bandwidth limit B corresponding to the audio to be processed can also be determined. max With bandwidth lower limit B minBy combining the determined upper and lower bandwidth limits with the effective bandwidth coefficient, the upper and lower bitrate bandwidth limits corresponding to the audio to be processed can be determined. The specific calculation process is shown in Formulas 13 and 14.

[0114] (Formula 13)

[0115] (Formula 14)

[0116] in, It can be the effective bandwidth coefficient of the audio to be processed; B max It could be the bandwidth limit; B min It could be the lower limit of bandwidth; It can be the upper limit of the bitrate bandwidth of the audio to be processed; It can be the lower limit of the bitrate bandwidth of the audio to be processed.

[0117] After determining the upper and lower limits of the bitrate bandwidth, the first bitrate of the audio to be processed can be determined by combining the spectral richness, as shown in Formula 15.

[0118] (Formula 15)

[0119] in, It can be the upper limit of the bitrate bandwidth of the audio to be processed; It can be the lower limit of the bitrate bandwidth of the audio to be processed; It can be the spectral richness (spectral sparsity coefficient) of the audio to be processed. Another representation can be shown in Equation 16.

[0120] (Formula 16)

[0121] In one exemplary embodiment of this disclosure, bitrate prediction processing based on audio features is performed to obtain a second bitrate corresponding to the audio to be processed, including: acquiring a pre-constructed audio quality prediction network; training the audio quality prediction network based on training sample pairs composed of sample audio segments and sample audio quality; obtaining the sample audio quality by performing multi-bitrate encoding processing on the sample audio segments; inputting audio features into the audio quality prediction network for audio quality prediction processing to obtain the target audio quality of the audio to be processed; acquiring a pre-constructed bitrate-audio quality curve, and determining the second bitrate corresponding to the target audio quality based on the bitrate-audio quality curve.

[0122] The audio quality prediction network can be a network model used to predict the bitrate of the audio to be processed. The sample audio quality can be the audio quality obtained after quality evaluation processing of sample audio segments. Training sample pairs can be data pairs consisting of sample audio segments and their corresponding sample audio qualities. Multi-bitrate encoding processing can be the specific process of encoding sample audio segments using different bitrates. Bitrate prediction processing can be the process of using the audio quality prediction network to determine the bitrate of the audio to be processed. The target audio quality can be the fidelity of the audio signal to be processed. The bitrate-to-audio quality curve can be a curve used to express the correspondence between the audio's bitrate and audio quality.

[0123] To address the limitation of traditional signal processing algorithms in determining bitrates for various scenarios, this embodiment employs deep learning techniques to predict the bitrate of the audio to be processed, thereby obtaining a corresponding second bitrate. For example, a pre-constructed audio quality prediction network can be used to predict the bitrate of the audio. This audio quality prediction network can be trained using training sample pairs consisting of sample audio segments and sample audio quality as training data. The sample audio segments are used as input data, and the target audio quality corresponding to the sample audio segments is used as the training objective to construct a deep learning network model.

[0124] The training samples are input into the constructed deep learning network model for training, resulting in a sound quality prediction network used to determine the target sound quality. After obtaining the sound quality prediction network, audio features can be input into the sound quality prediction network for bitrate prediction processing to determine the target sound quality corresponding to the audio to be processed.

[0125] To determine the second bitrate of the audio to be processed, a pre-constructed bitrate-to-quality curve can be used. This curve can represent the relationship between the encoded bitrate and the objective audio quality of an audio segment, as shown in the reference. Figure 4 , Figure 4 This is a schematic diagram illustrating the determination of the bitrate for a specific audio segment based on a bitrate-quality curve, according to an exemplary embodiment. From Figure 4 It can be seen from this that Figure 4 The document presents the audio quality corresponding to 10 different bitrates for audio encoding, specifically the curves for N=10 and R=[24,32,40,48,56,64,72,80,88,96]; where asterisks (…) The value () represents the audio quality prediction S' corresponding to the bitrate sampling point R. Linear interpolation can be used to fit the data between two sampling points. For example... Figure 4 As shown, based on the bitrate-quality curve of an audio segment, for a certain target quality, the bitrate required for encoding the segment can be directly obtained, thereby determining the second bitrate of the audio to be processed.

[0126] In one exemplary embodiment of this disclosure, the audio quality prediction network is constructed through the following steps: obtaining a pre-constructed initial neural network and a training audio sample set; the training audio sample set includes multiple sample audio segments; performing multi-bitrate encoding processing on the sample audio segments to obtain multiple encoded audio segments corresponding to the sample audio segments; performing audio quality evaluation processing on the multiple encoded audio segments to obtain the corresponding sample audio quality; extracting features from the sample audio segments to obtain sample audio features corresponding to the sample audio segments; using the sample audio features of the sample audio segments and the corresponding sample audio quality as training sample pairs to train the initial neural network to obtain the audio quality prediction network.

[0127] The initial neural network can be a pre-built initial network model. The training audio sample set can be a sample dataset used to train the initial neural network to obtain the sound quality prediction network; the training audio sample set can consist of multiple sample audio segments. The encoded audio segment can be an audio segment obtained by encoding the sample audio segments at different bitrates. The sound quality evaluation process can be the process of determining the sound quality corresponding to the sample audio segment. The sample audio sound quality can be the fidelity of the sound signal of the sample audio segment. The training sample pair can be a data pair consisting of a sample audio segment and its corresponding sample audio sound quality.

[0128] refer to Figure 5 , Figure 5 This is a system block diagram illustrating a deep learning technique for predicting the second bitrate of audio to be processed, according to an exemplary embodiment. When constructing the audio quality prediction network, sample audio segments can first undergo data synthesis processing. In the data synthesis stage, for a sample audio segment 510 (audio segment x) (which may be speech, music, and ambient sound, or a mixture of several types of content), an audio coding algorithm, such as the High Efficiency Advanced Audio Coding (HE-AAC) algorithm, is used to encode N bitrates. For example, the bitrate used in the encoding process is R=[r1,r2,…,rN], and after encoding the sample audio segment, the output audio segment is Y=[y1,y2,…,yN].

[0129] For example, a typical bitrate setting is N=10, R=[24,32,40,48,56,64,72,80,88,96]. In step S510, objective audio quality is calculated. Based on the original input sample audio segment x and the encoded audio segment Y=[y1,y2,…,yN], an objective audio quality evaluation algorithm is used to obtain the objective audio quality S=[s1,s2,…,sN] corresponding to various bitrates. The objective audio quality here can adopt the Perceptual Evaluation of Audio Quality (PEAQ) index recommended by the International Telecommunication Union, or a linear fusion of multiple objective audio quality indices. If there is a large amount of subjectively labeled data, it can be changed to predict subjective audio quality, making the result closer to human subjective perception.

[0130] Through the above data synthesis methods, we can obtain a large number of original sample audio segments x and audio quality S data pairs obtained after multi-bitrate N encoding and audio evaluation processing, which can be used for model training of the audio quality prediction network.

[0131] Continue to refer to Figure 5 After performing a short-time Fourier transform on the sample audio segment, spectral data corresponding to multiple audio frames can be obtained. In step S520, feature extraction processing is performed on the obtained spectral data. The process of feature extraction for audio segments has already been described in the process of feature extraction for the audio to be processed, and will not be repeated in this embodiment. For the input sample audio segment x, multiple features are concatenated together to finally obtain a D-dimensional feature vector.

[0132] A pre-constructed initial neural network is obtained. The input data of the initial neural network can be a D-dimensional feature vector and an N-dimensional sound quality estimation result. The network structure of the initial neural network can be a simple multi-layer fully connected network with a total of L layers and K nodes in each layer (e.g., L=2, K=100). During model training, the model parameters are updated by minimizing the mean square error (MSE) function between the estimated objective quality S' and the reference objective quality S to train the model and obtain the final sound quality prediction network 520. In step S530, the audio features of the audio to be processed are input into the sound quality prediction network 520 for sound quality prediction processing, and the corresponding second bitrate is determined based on the obtained target sound quality.

[0133] In one exemplary embodiment of this disclosure, determining the target bitrate corresponding to the audio to be processed based on a first bitrate and a second bitrate includes: obtaining the current network status; and determining the target bitrate based on the current network status, the first bitrate, and the second bitrate.

[0134] The current network status can be the current network status for data transmission.

[0135] After determining the first and second bitrates, the current network status of the data transmission network can be obtained, which represents the network's data transmission rate. Based on the current network status, the two bitrates are fused to obtain the target bitrate for the audio to be processed. By combining the current network status to determine the target bitrate, the required audio compression bitrate during transmission can be reduced as much as possible while effectively ensuring audio quality, thereby reducing transmission bandwidth.

[0136] In one exemplary embodiment of this disclosure, determining the target bitrate based on the current network state, the first bitrate, and the second bitrate includes: if the current network state is a first network state, then determining the target bitrate based on the audio type, the first bitrate, and the second bitrate; the first network state is that the network transmission rate is within a preset rate range.

[0137] The first network state can be a network state where the network transmission rate is within a preset rate range. Network transmission rate refers to the speed at which data is transmitted, typically expressed in bits per second (bps), which represents the number of bits of binary data transmitted per second. The preset rate range can be a pre-configured range of network transmission rates, including a minimum and a maximum value.

[0138] If the current network state is the first network state, then the current data transmission network is considered to be in normal condition. When classifying the incoming audio, high bitrate is required for music scenarios, necessitating high quality. Conversely, in noisy scenarios, users have lower requirements for noise levels, so the bitrate should be minimized. Therefore, if the audio is determined to be noise, the minimum of the first and second bitrates is used; if it is determined to be music, the maximum of the first and second bitrates is used; and if it is determined to be speech, the average of the first and second bitrates is used. This ensures that the audio quality is maintained while minimizing the bitrate, regardless of the audio type. See Formula 17 for details.

[0139] (Formula 17)

[0140] In one exemplary embodiment of this disclosure, determining the target bitrate based on the current network state, the first bitrate, and the second bitrate further includes: if the current network state is a second network state, then determining the larger bitrate value between the first bitrate and the second bitrate; the second network state is when the network transmission rate is greater than the maximum value corresponding to a preset rate interval; and using the larger bitrate value as the target bitrate.

[0141] The second network state can be a state where the current network transmission rate is greater than the maximum value corresponding to the preset rate range. The larger bitrate value can be the larger bitrate value between the first bitrate and the second bitrate.

[0142] If the current network state is in the second network state, since the second network state can refer to the network transmission rate being greater than the maximum value corresponding to the preset rate range, the network state can be considered to be good. In this case, in order to ensure the audio quality of the audio to be processed, the larger bitrate value between the first bitrate and the second bitrate can be selected as the target bitrate to ensure people's requirements for sound quality, as shown in Formula 18.

[0143] (Formula 18)

[0144] In one exemplary embodiment of this disclosure, determining the target bitrate based on the current network state, the first bitrate, and the second bitrate further includes: if the current network state is a third network state, then determining the smaller bitrate value between the first bitrate and the second bitrate; the third network state is a network transmission rate greater than the minimum value corresponding to a preset rate interval; and using the smaller bitrate value as the target bitrate.

[0145] The third network state can be a state where the current network transmission rate is less than the minimum value corresponding to the preset rate range. The smaller bitrate value can be the smaller bitrate value between the first bitrate and the second bitrate.

[0146] If the current network state is in the third network state, since the third network state can refer to the network transmission rate being less than the minimum value corresponding to the preset rate range, the network state can be considered to be poor. In this case, in order to ensure the audio quality of the audio to be processed, the smaller bit rate value between the first bit rate and the second bit rate can be selected as the target bit rate to reduce the audio transmission bit rate as much as possible, as shown in Formula 19.

[0147] (Formula 19)

[0148] In summary, the process involves acquiring the audio to be processed, classifying it to determine its audio type, performing audio quality detection, and determining the first bitrate based on the audio type. Then, the audio features of the audio are identified, and bitrate prediction is performed based on these features to obtain the second bitrate. Finally, the target bitrate is determined based on the first and second bitrates. On one hand, by performing audio classification and audio quality prediction on the audio to obtain the first and second bitrates, audio quality can be effectively guaranteed while minimizing the audio compression bitrate required during transmission, thereby reducing transmission bandwidth. On the other hand, since the target bitrate is determined based on the first and second bitrates, the accuracy of the results can be effectively improved. Furthermore, combining traditional signal processing algorithms with deep learning prediction schemes makes the determined target bitrate highly applicable to various scenarios.

[0149] Figure 6 This is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment. (Refer to...) Figure 6 The audio processing device 600 includes an audio type determination module 610, a first bit rate determination module 620, a second bit rate determination module 630, and a target bit rate determination module 640.

[0150] Specifically, the audio type determination module 610 is used to acquire the audio to be processed, classify the audio to be processed, and obtain the audio type of the audio to be processed; the first bitrate determination module 620 is used to perform sound quality detection processing on the audio to be processed, and determine the first bitrate corresponding to the audio to be processed based on the audio type; the second bitrate determination module 630 is used to determine the audio features of the audio to be processed, perform bitrate prediction processing based on the audio features, and obtain the second bitrate corresponding to the audio to be processed; the target bitrate determination module 640 is used to determine the target bitrate corresponding to the audio to be processed based on the first bitrate and the second bitrate.

[0151] In one exemplary embodiment of this disclosure, the audio type determination module 610 includes an audio type determination unit, configured to: acquire a pre-constructed audio classification network; train the audio classification network based on the Mel spectrum feature values ​​and audio type labels corresponding to multiple sample audio segments; and classify the audio to be processed through the audio classification network to obtain the audio type.

[0152] In one exemplary embodiment of this disclosure, the audio type determination module further includes a classification network training unit, configured to: acquire a training audio sample set and a pre-constructed initial classification network; the training audio sample set includes multiple sample audio segments and audio type labels for each sample audio segment; perform time-domain transformation processing on each sample audio segment to obtain time-domain audio segments corresponding to each sample audio segment; calculate Mel frequency cepstral coefficients for each time-domain audio segment to obtain initial Mel spectrum values ​​for each time-domain audio segment; determine Mel spectrum feature values ​​corresponding to each of the multiple time-domain audio segments based on each initial Mel spectrum value; and train the initial classification network based on the audio type labels and Mel spectrum feature values ​​corresponding to each of the multiple sample audio segments to obtain an audio classification network.

[0153] In one exemplary embodiment of this disclosure, the first bitrate determination module 620 includes a first bitrate determination unit, configured to: perform effective bandwidth detection on the audio to be processed to obtain the effective bandwidth of the audio to be processed; perform audio complexity detection on the audio to be processed to obtain the spectral richness of the audio to be processed; and perform fusion processing on the effective bandwidth, spectral richness and audio type to obtain the first bitrate.

[0154] In one exemplary embodiment of this disclosure, the first bit rate determination unit includes an effective bandwidth determination subunit, configured to: obtain the time domain interval where the audio to be processed is located, calculate the frequency band energy of each frequency band of the audio to be processed within the time domain interval; determine the maximum frequency band energy among multiple frequency band energies, and determine the audio bandwidth frequency band point of the audio to be processed based on the maximum frequency band energy; obtain a pre-configured fast Fourier transform length and the audio sampling rate corresponding to the audio to be processed; and determine the effective bandwidth based on the audio bandwidth frequency band point, the fast Fourier transform length, and the audio sampling rate.

[0155] In an exemplary embodiment of this disclosure, the first bitrate determination unit includes a complexity determination subunit, configured to: determine the total number of frequency bands corresponding to the audio to be processed; acquire the frequency band energy of each frequency band of the audio to be processed in the time domain interval, and the frame sequence corresponding to the audio to be processed; the frame sequence includes multiple audio frames, each audio frame having a corresponding frame number; determine the number of frequency band energy truncations based on the multiple frequency band energies and the frame numbers of each audio frame; and determine the spectral richness based on the number of frequency band energy truncations and the total number of frequency bands.

[0156] In an exemplary embodiment of this disclosure, the first bitrate determination unit includes a first bitrate determination subunit, configured to: acquire the audio sampling rate, upper bandwidth limit, and lower bandwidth limit corresponding to the audio to be processed; determine the effective bandwidth coefficient based on the audio sampling rate and effective bandwidth; determine the upper bitrate bandwidth limit and lower bitrate bandwidth limit of the audio to be processed based on the effective bandwidth coefficient, upper bandwidth limit, and lower bandwidth limit; and determine the first bitrate based on the spectral richness, upper bitrate bandwidth limit, and lower bitrate bandwidth limit.

[0157] In one exemplary embodiment of this disclosure, the second bitrate determination module 630 includes a second bitrate determination unit, configured to: acquire a pre-constructed audio quality prediction network; train the audio quality prediction network based on training sample pairs consisting of sample audio segments and sample audio quality; obtain the sample audio quality by performing multi-bitrate encoding processing on the sample audio segments; input audio features into the audio quality prediction network for audio quality prediction processing to obtain the target audio quality of the audio to be processed; acquire a pre-constructed bitrate-audio quality curve, and determine the second bitrate corresponding to the target audio quality based on the bitrate-audio quality curve.

[0158] In one exemplary embodiment of this disclosure, the second bitrate determination module 630 further includes a prediction network determination unit, configured to: acquire a pre-constructed initial neural network and a training audio sample set; the training audio sample set includes multiple sample audio segments; perform multi-bitrate encoding processing on the sample audio segments to obtain multiple encoded audio segments corresponding to the sample audio segments; perform sound quality evaluation processing on the multiple encoded audio segments to obtain the corresponding sample audio sound quality; perform feature extraction on the sample audio segments to obtain sample audio features corresponding to the sample audio segments; and use the sample audio features of the sample audio segments and the corresponding sample audio sound quality as training sample pairs to train the initial neural network to obtain a sound quality prediction network.

[0159] In one exemplary embodiment of this disclosure, the target bitrate determination module 640 includes a target bitrate determination unit, configured to: obtain the current network state; and determine the target bitrate based on the current network state, a first bitrate, and a second bitrate.

[0160] In one exemplary embodiment of this disclosure, the target bitrate determination unit includes a first target bitrate determination subunit, configured to: determine a target bitrate based on audio type, a first bitrate, and a second bitrate if the current network state is a first network state; the first network state is when the network transmission rate is within a preset rate range.

[0161] In one exemplary embodiment of this disclosure, the target bit rate determination unit further includes a second target bit rate determination subunit, configured to: if the current network state is a second network state, determine the larger bit rate value between the first bit rate and the second bit rate; the second network state is a network transmission rate greater than the maximum value corresponding to a preset rate interval; and use the larger bit rate value as the target bit rate.

[0162] In an exemplary embodiment of this disclosure, the target bit rate determination unit further includes a third target bit rate determination subunit, configured to: if the current network state is a third network state, determine the smaller bit rate value between the first bit rate and the second bit rate; the third network state is a network transmission rate greater than the minimum value corresponding to a preset rate interval; and use the smaller bit rate value as the target bit rate.

[0163] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0164] The following is for reference. Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0165] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including storage unit 720 and processing unit 710), and a display unit 740.

[0166] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section above, according to various exemplary embodiments of this disclosure.

[0167] Storage unit 720 may include readable media in the form of volatile storage units, such as random access memory (RAM) 721 and / or cache memory 722, and may further include read-only memory (ROM) 723.

[0168] Storage unit 720 may include a program / utility 724 having a set (at least one) program module 725, such program module 725 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0169] Bus 730 can represent one or more of several types of bus structures, including memory cell bus or memory cell controller, peripheral bus, graphics acceleration port, processing unit, or local bus using any of the multiple bus structures.

[0170] Electronic device 700 can also communicate with one or more external devices 770 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0171] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of the device to perform the audio processing method described above. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0172] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the audio processing method described in any one of the preceding embodiments.

[0173] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0174] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio processing method, characterized in that, include: The audio to be processed is obtained, and the audio to be processed is classified to obtain the audio type of the audio to be processed. The audio to be processed is subjected to audio quality detection processing, and the first bitrate corresponding to the audio to be processed is determined in combination with the audio type; The audio features of the audio to be processed are determined, and bitrate prediction processing is performed based on the audio features to obtain the second bitrate corresponding to the audio to be processed. The target bitrate corresponding to the audio to be processed is determined based on the first bitrate and the second bitrate; The step of performing bitrate prediction processing based on the audio features to obtain the second bitrate corresponding to the audio to be processed includes: A pre-constructed audio quality prediction network is obtained; the audio quality prediction network is trained based on training sample pairs consisting of sample audio segments and sample audio quality; the sample audio quality is obtained by performing multi-bitrate encoding on the sample audio segments; The audio features are input into the audio quality prediction network for audio quality prediction processing to obtain the target audio quality of the audio to be processed. Obtain a pre-constructed bitrate-to-sound-quality curve, and determine the second bitrate corresponding to the target sound quality based on the bitrate-to-sound-quality curve; Determining the target bitrate corresponding to the audio to be processed based on the first bitrate and the second bitrate includes: Get the current network status; The target bitrate is determined based on the current network status, the first bitrate, and the second bitrate.

2. The method according to claim 1, characterized in that, The process of classifying the audio to be processed to obtain the audio type includes: Obtain a pre-constructed audio classification network; the audio classification network is trained based on the Mel-spectrum feature values ​​and audio type labels corresponding to multiple sample audio segments; The audio to be processed is classified using the audio classification network to obtain the audio type.

3. The method according to claim 2, characterized in that, The audio classification network is obtained based on the following steps: Obtain a training audio sample set and a pre-constructed initial classification network; the training audio sample set includes multiple sample audio segments and audio type labels for each sample audio segment; Each of the sample audio segments is subjected to time-domain transformation to obtain the time-domain audio segment corresponding to each sample audio segment. Mel frequency cepstral coefficients are calculated for each of the time-domain audio segments to obtain the initial Mel spectrum values ​​for each of the time-domain audio segments; Each of the multiple time-domain audio segments is determined according to its respective initial Mel spectrum value; Based on the audio type labels corresponding to each of the multiple sample audio segments and the Mel spectrum feature values, the initial classification network is trained to obtain the audio classification network.

4. The method according to claim 1, characterized in that, The audio quality detection process includes effective bandwidth detection and audio complexity detection; the audio quality detection process performed on the audio to be processed, and the determination of the first bitrate corresponding to the audio to be processed in combination with the audio type, includes: The effective bandwidth of the audio to be processed is obtained by performing effective bandwidth detection on the audio to be processed. The audio to be processed is subjected to audio complexity detection to obtain the spectral richness of the audio to be processed; The effective bandwidth, the spectral richness, and the audio type are fused to obtain the first bitrate.

5. The method according to claim 4, characterized in that, The step of detecting the effective bandwidth of the audio to be processed to obtain the effective bandwidth of the audio to be processed includes: Obtain the time domain interval where the audio to be processed is located, and calculate the frequency band energy of each frequency band of the audio to be processed within the time domain interval; Determine the maximum frequency band energy among the multiple frequency band energies, and determine the audio bandwidth frequency band point of the audio to be processed based on the maximum frequency band energy; Obtain the pre-configured Fast Fourier Transform length and the audio sampling rate corresponding to the audio to be processed; The effective bandwidth is determined based on the audio bandwidth frequency point, the fast Fourier transform length, and the audio sampling rate.

6. The method according to claim 4, characterized in that, The step of performing audio complexity detection on the audio to be processed to obtain the spectral richness of the audio to be processed includes: Determine the total number of frequency bands corresponding to the audio to be processed; The frequency band energy of each frequency band of the audio to be processed in the time domain interval is obtained, as well as the frame sequence corresponding to the audio to be processed; the frame sequence includes multiple audio frames, each audio frame having a corresponding frame number; The number of frequency band energies to be extracted is determined based on the multiple frequency band energies and the frame number of each audio frame. The spectral richness is determined based on the number of frequency band energy cuts and the total number of frequency bands.

7. The method according to claim 4, characterized in that, The process of fusing the effective bandwidth, the spectral richness, and the audio type to obtain the first bitrate includes: Obtain the audio sampling rate, upper bandwidth limit, and lower bandwidth limit corresponding to the audio to be processed; The effective bandwidth coefficient is determined based on the audio sampling rate and the effective bandwidth. The upper and lower bitrate bandwidth limits of the audio to be processed are determined based on the effective bandwidth coefficient, the upper bandwidth limit, and the lower bandwidth limit. The first bitrate is determined based on the spectral richness, the upper limit of the bitrate bandwidth, and the lower limit of the bitrate bandwidth.

8. The method according to claim 1, characterized in that, The sound quality prediction network is constructed through the following steps: Obtain a pre-constructed initial neural network and a training audio sample set; the training audio sample set includes multiple sample audio segments; The sample audio segment is subjected to multi-bitrate encoding to obtain multiple encoded audio segments corresponding to the sample audio segment; The audio quality of multiple encoded audio segments is evaluated to obtain the corresponding sample audio quality. Feature extraction is performed on the sample audio segment to obtain the sample audio features corresponding to the sample audio segment; The sample audio features and corresponding sample audio quality of the sample audio segments are used as training sample pairs to train the initial neural network, thereby obtaining the sound quality prediction network.

9. The method according to claim 1, characterized in that, Determining the target bitrate based on the current network state, the first bitrate, and the second bitrate includes: If the current network state is the first network state, then the target bitrate is determined based on the audio type, the first bitrate, and the second bitrate; the first network state is when the network transmission rate is within a preset rate range.

10. The method according to claim 1, characterized in that, The step of determining the target bitrate based on the current network state, the first bitrate, and the second bitrate further includes: If the current network state is the second network state, then the larger bitrate value between the first bitrate and the second bitrate is determined; the second network state is when the network transmission rate is greater than the maximum value corresponding to a preset rate range. The larger bitrate value is used as the target bitrate.

11. The method according to claim 1, characterized in that, The step of determining the target bitrate based on the current network state, the first bitrate, and the second bitrate further includes: If the current network state is the third network state, then the smaller bitrate value between the first bitrate and the second bitrate is determined; the third network state is when the network transmission rate is greater than the minimum value corresponding to a preset rate interval. The smaller bitrate value is used as the target bitrate.

12. An audio processing device, characterized in that, include: An audio type determination module is used to acquire the audio to be processed, classify the audio to be processed, and obtain the audio type of the audio to be processed. The first bitrate determination module is used to perform audio quality detection processing on the audio to be processed, and determine the first bitrate corresponding to the audio to be processed in combination with the audio type; The second bitrate determination module is used to determine the audio features of the audio to be processed, and perform bitrate prediction processing based on the audio features to obtain the second bitrate corresponding to the audio to be processed. A target bitrate determination module is used to determine the target bitrate corresponding to the audio to be processed based on the first bitrate and the second bitrate; The second bitrate determination module is used to obtain a pre-constructed audio quality prediction network; the audio quality prediction network is trained based on training sample pairs consisting of sample audio segments and sample audio quality; the sample audio quality is obtained by performing multi-bitrate encoding processing on the sample audio segments; The audio features are input into the audio quality prediction network for audio quality prediction processing to obtain the target audio quality of the audio to be processed. Obtain a pre-constructed bitrate-to-sound-quality curve, and determine the second bitrate corresponding to the target sound quality based on the bitrate-to-sound-quality curve; The target bit rate determination module is used to obtain the current network status; The target bitrate is determined based on the current network status, the first bitrate, and the second bitrate.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the audio processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the audio processing method as described in any one of claims 1 to 11.

Images