An ai-based cloud terminal dynamic audio processing method

Abstract

The application relates to the technical field of audio processing, and discloses an AI-based cloud terminal dynamic audio processing method, which aims to solve the problem that an existing scheme cannot dynamically adapt to scene changes, and mainly comprises the following steps: collecting environmental noise data, extracting environmental perception features after preprocessing, framing and frequency domain transformation; performing framing processing on playing audio data, extracting MFCC features, logarithmic short-time energy features and zero-crossing rate features, and obtaining a scene recognition result through a CNN+LSTM scene classification model; splicing the environmental perception features and the scene recognition result into a joint feature vector, inputting the joint feature vector into an AI model to obtain adaptive gain curve parameters, scene adaptive noise suppression parameters and anti-distortion dynamic range control parameters; processing the playing audio to obtain optimized audio data; and updating AI model parameters based on a reward value of user feedback data and through a PPO algorithm. The application can realize intelligent audio processing which dynamically adapts to the environment and the scene, and effectively improves the audio experience of the cloud terminal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to an AI-based cloud terminal dynamic audio processing method. BACKGROUND

[0002] With the rapid development of the Internet and cloud computing technology, the computing mode is gradually migrating from the local terminal to the cloud. Under this background, intelligent cloud terminals (such as cloud computers, cloud pads, cloud laptops, etc.) emerge as one of the core businesses of operators. Intelligent cloud terminals deploy computing, storage and rendering capabilities in the cloud, and the local terminal is only responsible for display and interaction, so that users can enjoy cloud desktops, online videos, cloud games and other high-level services with low-cost, lightweight and small hardware devices. This mode greatly expands the user's use scenarios and realizes the possibility of accessing cloud computing power anytime and anywhere.

[0003] However, in order to control costs and size, cloud terminals are usually equipped with small power and general quality speaker units, whose inherent physical characteristics result in insufficient loudness and narrow frequency response range of audio output. At the same time, the user's use environment is complex and variable, for example, in a noisy coffee shop, airport or open office, the environmental noise will seriously mask the weak audio signal of the cloud terminal, resulting in a sharp decrease in user listening clarity. In addition, the audio content consumed by users also has diversified characteristics, such as voice calls, music playback, game sound effects, video background sound, etc. The recording levels and dynamic ranges of these contents themselves differ greatly, causing the volume to be large or small when playing, and users need to manually adjust the volume frequently, resulting in a poor experience.

[0004] The existing cloud terminal audio processing scheme is mostly static or semi-static, lacking intelligence and adaptability, and cannot perceive the specific acoustic environment in which the user is located in real time, nor can it understand the differentiated needs of the user's business scenarios for audio processing. The specific reasons are as follows: first, the existing technology usually adopts fixed gain compensation, that is, a fixed volume gain value is preset according to the terminal model. This method cannot adapt to dynamically changing environmental noise, and in a quiet environment, it is easy to produce distortion or disturb others due to excessive amplification, and in a noisy environment, the gain is insufficient; second, the existing technology usually adopts traditional dynamic range compression, that is, the dynamic range of the audio signal is compressed with a fixed threshold, which can balance the volume fluctuation to a certain extent, but lacks differentiation of business scenarios, which can easily destroy the dynamic beauty of music or reduce the intelligibility of speech; third, the existing technology generally uses simple noise suppression, but is usually based on a fixed noise spectrum assumption, which cannot accurately identify and track complex and variable noise in the real environment, such as burst noise and periodic noise, and the suppression effect is limited and may introduce speech damage. SUMMARY

[0005] This invention aims to address the problems of existing cloud terminal audio processing solutions being unable to adapt to dynamically changing usage environments and diverse business scenarios, resulting in low volume, significant environmental noise interference, and drastic volume fluctuations for different audio content. It proposes an AI-based dynamic audio processing method for cloud terminals.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] A cloud terminal dynamic audio processing method based on AI, the method comprising:

[0008] Real-time environmental noise data is collected via a microphone. The real-time environmental noise data is preprocessed and framed to obtain a framed time-domain signal. The framed time-domain signal is then transformed in the frequency domain to obtain a noise power spectral density distribution. Based on the noise power spectral density distribution, the signal-to-noise ratio (SNR) and the noise-dominant frequency band distribution are calculated. Environmental perception features are obtained based on the SNR and the noise-dominant frequency band distribution. The environmental perception features include the energy proportion of the three frequency bands, the normalized signal-to-noise ratio (SNR), and the noise-dominant frequency band label.

[0009] The audio data played on the cloud terminal is processed by frame segmentation. The MFCC features, log short-time energy features and zero-crossing rate features of each frame of audio are extracted, concatenated and normalized to obtain a normalized feature vector. The normalized feature vector is input into a pre-trained CNN+LSTM scene classification model to obtain scene recognition results. The scene recognition results include the one-hot encoded vector of the scene category and the classification confidence.

[0010] The environmental perception features and the scene recognition results are concatenated into a joint feature vector. The joint feature vector is then input into an AI model to obtain audio processing parameters. Based on the audio processing parameters, the played audio data is processed to obtain optimized audio data. The audio processing parameters include adaptive gain curve parameters, scene adaptive noise suppression parameters, and distortion prevention dynamic range control parameters.

[0011] Collect user feedback data, calculate reward values ​​based on the user feedback data, and update the parameters of the AI ​​model based on the reward values ​​and the PPO algorithm optimized by the near-end strategy.

[0012] Furthermore, the formula for calculating the signal-to-noise ratio is as follows:

[0013] ;

[0014] in, Indicates the signal-to-noise ratio. Indicates the target signal power. Indicates noise power;

[0015] The noise-dominant frequency band distribution was obtained by calculating the energy proportions of the three frequency bands: low frequency (0-500Hz), mid frequency (500-4kHz), and high frequency (>4kHz).

[0016] Furthermore, the method for extracting the MFCC features includes:

[0017] Pre-emphasis processing: High-frequency components are boosted using a first-order FIR filter;

[0018] Fourier transform: Perform an FFT transform on the windowed data to obtain the power spectrum;

[0019] Mel filtering: The power spectrum is converted into a Mel frequency spectrum through 24 triangular Mel filter banks, the frequency range of which is 20Hz~20kHz;

[0020] Logarithmic energy calculation: Calculate the logarithmic energy of each Mel filter output using the following formula:

[0021] ;

[0022] in, Indicates the first The logarithmic energy of a Mel filter Indicates the first The frequency response of a Mel filter Represents frequency Fourier transform coefficients at the location;

[0023] DCT Transform: Perform Discrete Cosine Transform on the logarithmic energy, take the first 13 coefficients as the static features of MFCC, and calculate the first-order difference and the second-order difference to form a 39-dimensional MFCC feature vector.

[0024] Furthermore, the formula for extracting the logarithmic short-time energy feature is as follows:

[0025] Calculate short-time energy:

[0026] ;

[0027] in, Represents the logarithmic short-time energy. Indicates the first in the frame One sampling point, Indicates frame length;

[0028] The formula for extracting the zero-crossing rate feature is as follows:

[0029] ;

[0030] in, Indicates the zero-crossing rate. Represents a symbolic function.

[0031] Furthermore, the structure of the CNN+LSTM scene classification model includes:

[0032] CNN Feature Extraction Layer: Input a 41-dimensional normalized feature vector, pass through Convolutional Layer 1 using 32 3×3 filters and ReLU activation, outputting a feature map of size 1×39×32, then pass through Pooling Layer 1 for 2×2 max pooling, outputting a feature map of size 1×19×32; pass through Convolutional Layer 2 using 64 3×3 filters and ReLU activation, outputting a feature map of size 1×17×64, then pass through Pooling Layer 2 for 2×2 max pooling, outputting a feature map of size 1×8×64, finally flattened into a 512-dimensional vector;

[0033] LSTM sequence modeling layer: The input 512-dimensional vector is mapped to 128-dimensional features through a fully connected layer, and then processed by a bidirectional LSTM layer with 128 hidden units and a Dropout rate of 0.3, outputting 256-dimensional features.

[0034] Classification output layer: The 256-dimensional features are input, mapped to 64-dimensional features through a fully connected layer, and then mapped to 6-dimensional features through an output layer. The 6-dimensional features correspond to 6 types of scenes, and the probability distribution of each type of scene is output using the softmax activation function.

[0035] Among them, when the maximum probability When the probability is greater than the probability threshold, output the first... If there are no specific scene categories, then it is marked as an unknown scene. This indicates that the audio corresponding to the input normalized feature vector belongs to the th . The probability of each scene category, i.e., the classification confidence.

[0036] Furthermore, the adaptive gain curve parameters include a reference gain, a low-frequency weighting coefficient, a mid-frequency weighting coefficient, and a high-frequency weighting coefficient;

[0037] The audio playback data is processed based on adaptive gain curve parameters, including:

[0038] Calculate the target frequency band gain curve based on the adaptive gain curve parameters:

[0039] ;

[0040] in, Represents frequency Gain value at that point, Indicates the reference gain. This represents the normalized signal-to-noise ratio. Represents frequency The frequency band weights are determined by the energy proportions of the three frequency bands: ;in, Indicates low-frequency weights. Indicates the intermediate frequency weight. Indicates high-frequency weights, Indicates the proportion of low-frequency energy. Indicates the proportion of high-frequency energy;

[0041] The playback audio data is framed and subjected to FFT transformation to obtain a frequency domain signal. The frequency domain signal is then multiplied with the gain curve in the frequency domain to obtain a gain-adjusted frequency domain signal. An inverse FFT transformation is then performed on the gain-adjusted frequency domain signal to obtain gain-optimized audio data.

[0042] Furthermore, the scene adaptive noise suppression parameter includes noise suppression intensity, and the calculation formula for the noise suppression intensity is as follows:

[0043] ;

[0044] in, Indicates the noise suppression strength. This represents the normalized signal-to-noise ratio. This indicates the classification confidence level of the scene category in the scene recognition results;

[0045] The playback audio data is processed based on scene-adaptive noise suppression parameters, including:

[0046] The noise suppression algorithm is selected according to the scene category, and the gain-optimized audio data is denoised according to the noise suppression intensity and the corresponding noise suppression algorithm to obtain the denoised audio data.

[0047] Furthermore, noise suppression algorithms are selected based on scene category, including:

[0048] For low-frequency dominant noise scenarios, an improved spectral subtraction method is adopted, with the noise estimation window set to 500ms and the oversubtraction factor of the spectral subtraction method set to 4.0.

[0049] For mid-frequency dominant noise scenarios, Wiener filtering is used, with the signal-to-noise ratio threshold set to 10dB and the filter order set to 32.

[0050] For high-frequency dominant noise scenarios, wavelet thresholding is used for denoising, with the decomposition level set to 4 and a soft thresholding function. ,in, This represents the threshold for wavelet denoising. This represents the standard deviation of noise.

[0051] Furthermore, the anti-distortion dynamic range control parameters include compression ratio, dynamic threshold, attack time, release time, and hard limiter threshold;

[0052] The playback audio data is processed based on the anti-distortion dynamic range control parameters, including:

[0053] The instantaneous level of the noise-reduced audio data is monitored in real time. When the instantaneous level exceeds the dynamic threshold, the required gain reduction is calculated based on the compression ratio and the degree to which the instantaneous level exceeds the threshold. The gain reduction is then applied to compress the noise-reduced audio data according to the attack time.

[0054] When the instantaneous level falls below the dynamic threshold, the gain is restored to the normal level according to the release time;

[0055] For audio data that has undergone dynamic range compression, a hard limiter is used to clip signal peaks that exceed the hard limiter threshold to obtain optimized audio data.

[0056] Furthermore, the user feedback data includes users' subjective rating data, text feedback data, and behavioral feedback data;

[0057] The formula for calculating the reward value is as follows:

[0058] ;

[0059] in, Indicates the reward value. This represents the subjective rating reward, obtained through mapping subjective rating data. The reward for text feedback is obtained through text feedback data mapping. This represents the behavioral feedback reward, obtained through mapping behavioral feedback data. Indicates the weight of subjective rating rewards. This indicates the weight of the text feedback reward.

[0060] The beneficial effects of this invention are as follows: The AI-based dynamic audio processing method for cloud terminals provided by this invention constructs an audio processing system capable of dynamically sensing the environment and intelligently recognizing scenes through artificial intelligence technology, thereby bringing an adaptive and high-quality audio experience to cloud terminal users. Specifically, by analyzing environmental noise in real time and intelligently applying noise suppression and gain compensation, background noise can be effectively suppressed, highlighting useful audio signals, enabling users to hear the content clearly even in noisy environments, especially improving the experience of voice communication and video dialogue; by accurately identifying the current business scenario, such as music, games, and meetings, and applying matching audio processing parameters, it can preserve the dynamic beauty of music and the immersive sound effects of games while ensuring high speech intelligibility, providing users with the most suitable auditory enjoyment for the current content; through anti-distortion dynamic range control and adaptive gain adjustment, it can automatically balance audio content from different sources and with different recording levels, effectively avoiding the problem of fluctuating volume during playback, eliminating the need for users to frequently adjust the volume manually; the introduction of a reinforcement learning mechanism based on user feedback enables the system to continuously optimize itself, becoming increasingly aligned with users' personal listening preferences, greatly enhancing user satisfaction and reliance on the cloud terminal business platform. Attached Figure Description

[0061] Figure 1 A schematic diagram of the system architecture provided for the embodiments;

[0062] Figure 2 This is a flowchart illustrating the AI-based cloud terminal dynamic audio processing method provided in this embodiment. Detailed Implementation

[0063] To overcome the problem that existing cloud terminal audio processing solutions, due to their static and singular nature, cannot adapt to complex environments and diverse scenarios, resulting in poor user experience, this invention proposes a technical solution. This invention achieves precise adaptive adjustment of audio parameters through dynamic environmental perception and intelligent scene recognition, significantly improving audio clarity, comfort, and user satisfaction.

[0064] Specifically, in this invention, firstly, environmental noise data is collected and processed to obtain environmental perception features, thereby quantitatively describing the acoustic environment in which the user is located. Then, a hybrid deep learning model (CNN+LSTM) is used to perform scene recognition on the audio data played on the cloud terminal, understanding the type of audio content the user is consuming. Next, utilizing a non-linear mapping relationship from the joint feature vector to audio processing parameters pre-learned by the AI ​​model, the joint feature vector concatenated from the environmental perception features and scene recognition results is mapped to audio processing parameters. Among these parameters, the adaptive gain curve parameter determines how much gain to increase at different frequencies based on the signal-to-noise ratio and the scene; the scene-adaptive noise suppression parameter determines the intensity and range of noise reduction based on the noise-dominant frequency band and the scene; and the anti-distortion dynamic range control parameter dynamically compresses the dynamic range of the audio based on the short-term energy of the content and the scene, preventing overload clipping distortion of strong signals while boosting weak signals. Finally, user feedback data is quantified into a reward value. The PPO algorithm updates the parameters of the AI ​​model based on this reward value, enabling the system to continuously improve and adapt to individual user preferences.

[0065] The technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0066] Please see Figure 1 and Figure 2 The AI-based cloud terminal dynamic audio processing method provided in this embodiment includes the following steps:

[0067] Step 1: Collect real-time environmental noise data through a microphone, preprocess and frame the real-time environmental noise data to obtain the framed time-domain signal; perform frequency domain transformation on the framed time-domain signal to obtain the noise power spectral density distribution, and calculate the signal-to-noise ratio and noise-dominant frequency band distribution based on the noise power spectral density distribution. Obtain environmental perception features based on the signal-to-noise ratio and noise-dominant frequency band distribution. The environmental perception features include the energy proportion of the three frequency bands, the normalized signal-to-noise ratio, and the noise-dominant frequency band label.

[0068] The core principle of Step 1 is to transform the complex acoustic environment in the physical world into a series of quantifiable, machine-readable digital features through signal processing techniques. Its scientific basis lies in the frequency domain characteristics and masking effect of noise: noise in different environments has unique fingerprints in terms of frequency distribution and energy. By capturing these fingerprints, the system can understand the type of noisy environment it is in, thus providing a basis for decision-making in subsequent intelligent audio processing.

[0069] In practical applications, the ambient sound is first recorded in real time using the built-in microphone (sampling rate 44.1kHz, bit depth 16bit) to obtain ambient noise data. To avoid data redundancy, the raw ambient noise data is pre-filtered to retain the audible frequency band signal of 20Hz~20kHz. At the same time, a sliding window (window length 512ms, step size 256ms) is used for frame division to ensure temporal continuity.

[0070] Then, an FFT (2048 points) is performed on the framed data to convert the time-domain signal into a frequency-domain spectrum, obtaining the power spectral density (PSD) distribution of the environmental noise data. This PSD reflects the distribution of signal power at different frequency components and is regarded as the noise power spectral density distribution of the current frame.

[0071] Finally, based on the above noise power spectral density distribution, the signal-to-noise ratio and noise-dominant frequency band distribution are calculated, thereby obtaining three core environmental perception features: the energy proportion of the three frequency bands, the normalized signal-to-noise ratio, and the noise-dominant frequency band label.

[0072] The formula for calculating the signal-to-noise ratio is as follows:

[0073] ;

[0074] in, Indicates the signal-to-noise ratio. The target signal power can be obtained through initial calibration or by learning after detecting a prolonged low level. This represents noise power, which can be directly estimated from the power spectrum of the acquired environmental noise frame.

[0075] In practical applications, the entire frequency band is first divided into three sub-bands: a low-frequency band (e.g., 0-500Hz), a mid-frequency band (e.g., 500-4kHz), and a high-frequency band (e.g., above 4kHz). The total energy of the power spectral density within each band is calculated, and the energy of each band is divided by the total energy to obtain the energy proportion of the three bands. Next, the calculated signal-to-noise ratio (SNR) is mapped to the range [0, 1] to obtain the normalized SNR. Finally, noise-dominant frequency band labels are generated, marking the frequency band with the highest energy proportion as the dominant frequency band.

[0076] In this embodiment, the energy proportion of the three frequency bands, the normalized signal-to-noise ratio, and the noise-dominant frequency band label together constitute an environmental perception feature vector that comprehensively describes the current acoustic environment. This provides a three-dimensional portrait of the environment from three dimensions, offering AI models richer contextual information than traditional methods.

[0077] Step 2: Perform frame-by-frame processing on the audio data played on the cloud terminal, extract the MFCC features, log short-time energy features, and zero-crossing rate features of each frame of audio, concatenate and normalize them to obtain a normalized feature vector; input the normalized feature vector into the pre-trained CNN+LSTM scene classification model to obtain the scene recognition result, which includes the one-hot encoded vector of the scene category and the classification confidence.

[0078] In practical applications, the audio data is first divided into short frames of fixed length, such as 128ms. A Hamming window function is then applied to each frame to reduce the spectral leakage effect caused by signal truncation and improve the accuracy of subsequent frequency domain analysis.

[0079] Then, the MFCC features (39 dimensions) of each audio frame are extracted, specifically including steps 21 to 25:

[0080] Step 21, Pre-emphasis processing: High-frequency components are boosted using a first-order FIR filter;

[0081] Step 22, Fourier Transform: Perform an FFT transform on the windowed data (2048 transformation points) to obtain the power spectrum;

[0082] Step 23, Mel filtering: The power spectrum is converted into a Mel frequency spectrum through 24 triangular Mel filter banks (frequency range 20Hz~20kHz).

[0083] Step 24, Logarithmic Energy Calculation: Calculate the logarithmic energy of each Mel filter output using the following formula:

[0084] ;

[0085] in, Indicates the first The logarithmic energy of a Mel filter Indicates the first The frequency response of a Mel filter Represents frequency Fourier transform coefficients at the location;

[0086] Step 25, DCT Transform: Perform Discrete Cosine Transform on the logarithmic energy, take the first 13 coefficients as the static features of MFCC, and calculate the first-order difference and the second-order difference to form a 39-dimensional MFCC feature vector.

[0087] Next, the logarithmic short-time energy (1-dimensional) is calculated using the following formula:

[0088] ;

[0089] in, Represents the logarithmic short-time energy. Indicates the first in the frame One sampling point, Indicates the frame length.

[0090] The above formula compresses the dynamic range by performing a logarithmic transformation on the total energy, thereby enhancing the robustness of the characteristics at different volume levels.

[0091] Next, the zero-crossing rate feature (1-dimensional) is calculated using the following formula:

[0092] ;

[0093] in, Indicates the zero-crossing rate. Represents a symbolic function.

[0094] The 39-dimensional MFCC features, the 1-dimensional logarithmic short-time energy features, and the 1-dimensional zero-crossing rate features are concatenated to form a 41-dimensional composite feature vector. And perform Z-score standardization on the composite feature vector, that is .in, This represents 39-dimensional MFCC features. This indicates the composite eigenvector. The normalized feature vector obtained after standardization processing and These are the mean and standard deviation of each feature dimension, calculated from a large-scale training dataset during the model training phase. This step eliminates differences between different feature scales, making model training more stable and converging faster.

[0095] Finally, the obtained normalized feature vectors are input into the pre-trained CNN+LSTM scene classification model to obtain the scene recognition result. In this embodiment, the structure of the CNN+LSTM scene classification model includes:

[0096] CNN Feature Extraction Layer: Input a 41-dimensional normalized feature vector, pass through Convolutional Layer 1 using 32 3×3 filters and ReLU activation, outputting a feature map of size 1×39×32, then pass through Pooling Layer 1 for 2×2 max pooling, outputting a feature map of size 1×19×32; pass through Convolutional Layer 2 using 64 3×3 filters and ReLU activation, outputting a feature map of size 1×17×64, then pass through Pooling Layer 2 for 2×2 max pooling, outputting a feature map of size 1×8×64, finally flattened into a 512-dimensional vector;

[0097] LSTM sequence modeling layer: The input 512-dimensional vector is mapped to 128-dimensional features through a fully connected layer, and then processed by a bidirectional LSTM layer with 128 hidden units and a Dropout rate of 0.3, outputting 256-dimensional features.

[0098] Classification output layer: The 256-dimensional features are input, mapped to 64-dimensional features through a fully connected layer, and then mapped to 6-dimensional features through an output layer. The 6-dimensional features correspond to 6 types of scenarios (such as office scenario, traffic scenario, home scenario, shopping mall scenario, park scenario, and quiet scenario). The softmax activation function is used to output the probability distribution of each type of scenario.

[0099] Among them, when the maximum probability When the probability exceeds a threshold (e.g., 0.85), output the first... If a scene is classified into three categories, it is identified as an unknown scene. A one-hot encoded vector corresponding to the scene category is then output, where... This indicates that the audio corresponding to the input normalized feature vector belongs to the th . The probability of each scene category, i.e., the classification confidence.

[0100] Through the above process, the final scene recognition result, including the one-hot encoded vector of the scene category and the classification confidence, is obtained. By fusing three complementary features—MFCC (spectral detail), log energy (loudness), and zero-crossing rate (rough frequency domain estimation)—and combining the advantages of CNN and LSTM, complex audio scenes can be distinguished with high accuracy, providing a reliable prerequisite for subsequent scene adaptive processing.

[0101] Step 3: Concatenate the environmental perception features and the scene recognition results into a joint feature vector, input the joint feature vector into the AI ​​model to obtain audio processing parameters, process the played audio data based on the audio processing parameters to obtain optimized audio data, the audio processing parameters include adaptive gain curve parameters, scene adaptive noise suppression parameters, and distortion prevention dynamic range control parameters.

[0102] In this embodiment, a trained AI model learns a complex nonlinear mapping relationship from joint feature vectors to optimal audio processing parameters. The goal is to solve for a set of audio processing parameters that maximize the user's auditory experience under given environmental and content constraints. Finally, by applying the solved audio processing parameters, precise and adaptive control of the audio signal is achieved, thereby attaining the optimal balance between increasing loudness, suppressing noise, and preventing distortion.

[0103] In practical applications, the environment-aware feature vector output from step 1 is first concatenated with the scene recognition result vector output from step 2. For example, assuming the environment-aware features are 7-dimensional and the scene recognition result is 7-dimensional (6-dimensional one-hot encoding + 1-dimensional classification confidence), the concatenation forms a 14-dimensional joint feature vector. This joint feature vector is then input into a pre-trained AI model. The AI ​​model can be a multilayer perceptron (MLP) or a similar regression network, acting as a complex policy function. The AI ​​model outputs a set of specific, executable audio processing parameters, and finally, the played audio data is processed based on these parameters to obtain optimized audio data.

[0104] In this embodiment, the adaptive gain curve parameters include a reference gain, a low-frequency weighting coefficient, a mid-frequency weighting coefficient, and a high-frequency weighting coefficient.

[0105] The audio playback data is processed based on adaptive gain curve parameters, including:

[0106] Calculate the target frequency band gain curve based on the adaptive gain curve parameters:

[0107] ;

[0108] in, Represents frequency Gain value at that point, Indicates the reference gain. This represents the normalized signal-to-noise ratio. Represents frequency The frequency band weights are determined by the energy proportions of the three frequency bands: ;in, Indicates low-frequency weights. Indicates the intermediate frequency weight. Indicates high-frequency weights, Indicates the proportion of low-frequency energy. Indicates the proportion of high-frequency energy;

[0109] The playback audio data is framed and subjected to FFT transformation to obtain a frequency domain signal. The frequency domain signal is then multiplied with the gain curve in the frequency domain to obtain a gain-adjusted frequency domain signal. An inverse FFT transformation is then performed on the gain-adjusted frequency domain signal to obtain gain-optimized audio data.

[0110] In practical applications, the weight of each frequency band is first calculated based on the energy proportion of the three frequency bands in the noise-dominant frequency band. During the calculation, frequency bands with concentrated noise energy are assigned lower weights to avoid amplifying noise, while frequency bands with less noise are assigned higher weights to effectively enhance the useful signal. Then, for multiple frequencies... Select the corresponding frequency band weight based on its frequency band. and the corresponding reference gain Normalized signal-to-noise ratio and frequency band weight Multiplying them yields the gain curve for the target frequency band, where Ensure a higher gain is applied in noisy environments (low signal-to-noise ratio). Finally, perform frame segmentation and FFT transformation on the playback audio data to be processed to obtain the frequency domain signal. ,Will With gain curve After multiplying point by point in the frequency domain, an inverse FFT transform is performed to obtain gain-optimized audio data.

[0111] In this embodiment, the scene adaptive noise suppression parameter includes noise suppression intensity, and the calculation formula for the noise suppression intensity is as follows:

[0112] ;

[0113] in, Indicates the noise suppression strength. This represents the normalized signal-to-noise ratio. This indicates the classification confidence level of the scene category in the scene recognition results;

[0114] The playback audio data is processed based on scene-adaptive noise suppression parameters, including:

[0115] The noise suppression algorithm is selected according to the scene category, and the gain-optimized audio data is denoised according to the noise suppression intensity and the corresponding noise suppression algorithm to obtain the denoised audio data.

[0116] Among them, the noise suppression algorithm selected according to the scene category includes:

[0117] For low-frequency dominant noise scenarios, an improved spectral subtraction method is adopted, with the noise estimation window set to 500ms and the oversubtraction factor of the spectral subtraction method set to 4.0.

[0118] For mid-frequency dominant noise scenarios, Wiener filtering is used, with the signal-to-noise ratio threshold set to 10dB and the filter order set to 32.

[0119] For high-frequency dominant noise scenarios, wavelet thresholding is used for denoising, with the decomposition level set to 4 and a soft thresholding function. ,in, This represents the threshold for wavelet denoising. This represents the standard deviation of noise.

[0120] In practical applications, the noise suppression strength is first calculated based on the normalized signal-to-noise ratio and classification confidence. The noisier the environment (the lower the normalized signal-to-noise ratio) and the more confident the scene recognition (the higher the classification confidence), the greater the applied noise suppression strength.

[0121] Then, a noise suppression algorithm is selected based on the noise-dominant frequency band label:

[0122] Low-frequency dominance: An improved spectral subtraction method is employed. The noise spectrum is estimated using the signal from the first 500 ms, and a large over-subtraction factor (4.0) is used to effectively eliminate stable low-frequency hum.

[0123] Mid-frequency dominance: Wiener filtering is employed. By setting an appropriate signal-to-noise ratio threshold (10dB) and filter order (32), useful mid-frequency signals such as speech can be well preserved while suppressing noise.

[0124] High-frequency dominant: Wavelet thresholding is used for noise reduction. A four-level decomposition is performed, and a soft thresholding function is used to smooth high-frequency noise.

[0125] Finally, the audio data is optimized for gain, and denoising is performed based on the selected noise suppression algorithm and noise suppression strength to obtain the denoised audio data.

[0126] In this embodiment, the anti-distortion dynamic range control parameters include compression ratio, dynamic threshold, attack time, release time, and hard limiter threshold.

[0127] The playback audio data is processed based on the anti-distortion dynamic range control parameters, including:

[0128] The instantaneous level of the noise-reduced audio data is monitored in real time. When the instantaneous level exceeds the dynamic threshold, the required gain reduction is calculated based on the compression ratio and the degree to which the instantaneous level exceeds the threshold. The gain reduction is then applied to compress the noise-reduced audio data according to the attack time.

[0129] When the instantaneous level falls below the dynamic threshold, the gain is restored to the normal level according to the release time;

[0130] For audio data that has undergone dynamic range compression, a hard limiter is used to clip signal peaks that exceed the hard limiter threshold to obtain optimized audio data.

[0131] In practical applications, dynamic range compression is first performed: the instantaneous level of the noise-reduced audio data is monitored in real time. When the instantaneous level exceeds the dynamic threshold, the required gain reduction is calculated based on the compression ratio (e.g., 2:1, 4:1), and compression is applied rapidly with a very short attack time (e.g., 1-10ms) to prevent signal peaks from spiked. When the level falls back below the threshold, the gain is smoothly restored with a slower release time (e.g., 50-200ms) to avoid a breathing effect.

[0132] Then, peak limiting is applied: the compressed signal is finally passed through a hard limiter. This limiter sets a very high threshold (close to 0 dBFS) and clips any signal peaks exceeding this threshold as a last line of defense against digital distortion.

[0133] Finally, the optimized audio data is output to the speakers for playback.

[0134] The above steps, through a three-tiered processing approach (gain, noise reduction, and distortion prevention), cleverly balance the often conflicting goals of loudness enhancement, noise suppression, and sound quality preservation. For example, adaptive gain enhances perceived loudness while avoiding noise amplification through frequency band weighting; while dynamic range control ensures that the audio signal remains clear and undistorted even after significant gain and noise reduction. Furthermore, the calculation of gain curves and noise suppression intensity incorporates environmental signal-to-noise ratio and scene confidence, making the decisions intelligent and reasonable. The noise reduction strategy based on the noise-dominant frequency band makes the noise reduction effect more targeted and minimizes sound quality loss. The distortion prevention dynamic range control module effectively prevents pops and distortions caused by over-processing or sudden changes in the input signal, ensuring the safety and comfort of the output audio. The end-user experience is clear, comfortable, and moderately loud sound in any environment and with any content, significantly improving user satisfaction and product stickiness.

[0135] Step 4: Collect user feedback data, calculate the reward value based on the user feedback data, and update the parameters of the AI ​​model based on the reward value and the PPO algorithm optimized by the near-end strategy.

[0136] In practical applications, after a user uses a cloud terminal, such as after answering a call or watching a video, a pop-up interface requests a rating, allowing the user to provide subjective rating data, such as a satisfaction rating of 1-5 stars. A text input box is also provided, allowing users to enter descriptive feedback data, such as "the sound is a bit harsh" or "the bass is insufficient." Furthermore, user behavior feedback data during the playback of optimized audio data can be recorded as indirect feedback. Examples include volume adjustment behavior, mute operation, and premature service interruption.

[0137] Then, the reward value is calculated based on user feedback data, using the following formula:

[0138] ;

[0139] in, Indicates the reward value. This represents the subjective rating reward, obtained through mapping subjective rating data. The reward for text feedback is obtained through text feedback data mapping. This represents the behavioral feedback reward, obtained through mapping behavioral feedback data. Indicates the weight of subjective rating rewards. This indicates the weight of the text feedback reward.

[0140] Subjective rating rewards can linearly or non-linearly map user-submitted 1-5 star ratings to a numerical range; for example, 1 star maps to -2, and 5 stars to +2. Text feedback rewards can use sentiment analysis models (such as pre-trained BERT models) to perform sentiment analysis on user-input text, mapping positive sentiments (such as "clear" and "pleasant to listen to") to positive rewards and negative sentiments (such as "noisy" and "distorted") to negative rewards. For behavioral feedback rewards, manually increasing the volume can be recorded as -0.5, and not using the volume button for a long time can be recorded as +0.3.

[0141] Finally, the parameters of the AI ​​model are updated based on the PPO (Proximal Policy Optimization) algorithm. The PPO algorithm maximizes the expected cumulative reward by optimizing a loss function that includes a pruning and substitution objective function. Its core idea is to encourage beneficial actions while ensuring that the update magnitude of each policy is not too large, thereby guaranteeing the stability of the training process.

[0142] In practical applications, an interaction process is organized into a state-action-reward tuple containing a joint feature vector, audio processing parameters, and a reward value. A training cycle is then periodically initiated in the cloud. Using a batch of collected state-action-reward tuples, the probability ratio of the output audio processing parameters when processing the same joint feature vector is calculated between the current policy (the latest AI model) and the old policy (the AI ​​model that generated this data). Based on this probability ratio and the reward value, the PPO loss function is calculated. This function considers both increasing the probability of high-reward actions and limiting the step size of a single update through a pruning mechanism. The AI ​​model parameters are updated via backpropagation using the gradient descent algorithm. The updated AI model is then redeployed to the cloud terminal or made available for use by the terminal, thus completing one iteration of system optimization.

[0143] Through the above process, the system no longer relies solely on initial offline training but can continuously learn online through real user feedback, constantly fine-tuning its processing strategies to improve audio processing over time. By collecting feedback from individual users and using it to update the model, the system can gradually learn and adapt to the specific listening habits and preferences of users, providing a customized audio experience and greatly enhancing user engagement.

[0144] In summary, the AI-based dynamic audio processing method for cloud terminals provided in this embodiment comprehensively enhances the audio experience of cloud terminals in complex real-world scenarios by constructing a complete intelligent closed loop of perception-recognition-decision-optimization. Specifically, through environmental perception and intelligent gain, it can automatically increase the loudness of speech and useful signals in noisy environments, while effectively suppressing background noise and significantly improving audio clarity using scene-adaptive noise suppression technology. Through precise scene recognition, the system can switch processing strategies according to content type (such as music, games, and meetings), ensuring speech clarity while preserving the dynamic beauty of music and the immersive sound effects of games, providing the most suitable auditory comfort and immersion for the current content. Through anti-distortion dynamic range control, it automatically balances the volume fluctuations of content from different sources, solving the pain point of users frequently adjusting the volume manually. Through AI algorithms, it achieves audio quality at the software level that previously required high-end hardware (such as high-powered speakers and multi-microphone arrays), effectively breaking through the inherent bottleneck of low-cost, lightweight cloud terminals in audio hardware, and providing solid technical support for the successful promotion and long-term development of operator cloud terminal services.

Claims

1. A cloud terminal dynamic audio processing method based on AI, characterized in that, The method includes: Real-time environmental noise data is collected via a microphone. The real-time environmental noise data is preprocessed and framed to obtain a framed time-domain signal. The framed time-domain signal is then transformed in the frequency domain to obtain a noise power spectral density distribution. Based on the noise power spectral density distribution, the signal-to-noise ratio (SNR) and the noise-dominant frequency band distribution are calculated. Environmental perception features are obtained based on the SNR and the noise-dominant frequency band distribution. The environmental perception features include the energy proportion of the three frequency bands, the normalized signal-to-noise ratio (SNR), and the noise-dominant frequency band label. The audio data played on the cloud terminal is processed by frame segmentation. The MFCC features, log short-time energy features and zero-crossing rate features of each frame of audio are extracted, concatenated and normalized to obtain a normalized feature vector. The normalized feature vector is input into a pre-trained CNN+LSTM scene classification model to obtain scene recognition results. The scene recognition results include the one-hot encoded vector of the scene category and the classification confidence. The environmental perception features and the scene recognition results are concatenated into a joint feature vector. The joint feature vector is then input into an AI model to obtain audio processing parameters. Based on the audio processing parameters, the played audio data is processed to obtain optimized audio data. The audio processing parameters include adaptive gain curve parameters, scene adaptive noise suppression parameters, and distortion prevention dynamic range control parameters. Collect user feedback data, calculate reward values ​​based on the user feedback data, and update the parameters of the AI ​​model based on the reward values ​​and the PPO algorithm optimized by the near-end strategy.

2. The AI-based cloud terminal dynamic audio processing method according to claim 1, characterized in that, The formula for calculating the signal-to-noise ratio is as follows: ； in, Indicates the signal-to-noise ratio. Indicates the target signal power. Indicates noise power; The noise-dominant frequency band distribution was obtained by calculating the energy proportions of the three frequency bands: low frequency (0-500Hz), mid frequency (500-4kHz), and high frequency (>4kHz).

3. The AI-based cloud terminal dynamic audio processing method according to claim 1, characterized in that, The method for extracting MFCC features includes: Pre-emphasis processing: High-frequency components are boosted using a first-order FIR filter; Fourier transform: Perform an FFT transform on the windowed data to obtain the power spectrum; Mel filtering: The power spectrum is converted into a Mel frequency spectrum through 24 triangular Mel filter banks, the frequency range of which is 20Hz~20kHz; Logarithmic energy calculation: Calculate the logarithmic energy of each Mel filter output using the following formula: ； in, Indicates the first The logarithmic energy of a Mel filter Indicates the first The frequency response of a Mel filter Represents frequency Fourier transform coefficients at the location; DCT Transform: Perform Discrete Cosine Transform on the logarithmic energy, take the first 13 coefficients as the static features of MFCC, and calculate the first-order difference and the second-order difference to form a 39-dimensional MFCC feature vector.

4. The AI-based cloud terminal dynamic audio processing method according to claim 3, characterized in that, The formula for extracting the logarithmic short-time energy feature is as follows: Calculate short-time energy: ； in, Represents the logarithmic short-time energy. Indicates the first in the frame One sampling point, Indicates frame length; The formula for extracting the zero-crossing rate feature is as follows: ； in, Indicates the zero-crossing rate. Represents a symbolic function.

5. The AI-based cloud terminal dynamic audio processing method according to claim 4, characterized in that, The structure of the CNN+LSTM scene classification model includes: CNN Feature Extraction Layer: Input a 41-dimensional normalized feature vector, pass through Convolutional Layer 1 using 32 3×3 filters and ReLU activation, outputting a feature map of size 1×39×32, then pass through Pooling Layer 1 for 2×2 max pooling, outputting a feature map of size 1×19×32; pass through Convolutional Layer 2 using 64 3×3 filters and ReLU activation, outputting a feature map of size 1×17×64, then pass through Pooling Layer 2 for 2×2 max pooling, outputting a feature map of size 1×8×64, finally flattened into a 512-dimensional vector; LSTM sequence modeling layer: The input 512-dimensional vector is mapped to 128-dimensional features through a fully connected layer, and then processed by a bidirectional LSTM layer with 128 hidden units and a Dropout rate of 0.3, outputting 256-dimensional features. Classification output layer: The 256-dimensional features are input, mapped to 64-dimensional features through a fully connected layer, and then mapped to 6-dimensional features through an output layer. The 6-dimensional features correspond to 6 types of scenes, and the probability distribution of each type of scene is output using the softmax activation function. Among them, when the maximum probability When the probability is greater than the probability threshold, output the first... If there are no specific scene categories, then it is marked as an unknown scene. This indicates that the audio corresponding to the input normalized feature vector belongs to the th . The probability of each scene category, i.e., the classification confidence.

6. The AI-based cloud terminal dynamic audio processing method according to claim 1, characterized in that, The adaptive gain curve parameters include the reference gain, low-frequency weighting coefficient, mid-frequency weighting coefficient, and high-frequency weighting coefficient; The audio playback data is processed based on adaptive gain curve parameters, including: Calculate the target frequency band gain curve based on the adaptive gain curve parameters: ； in, Represents frequency Gain value at that point, Indicates the reference gain. This represents the normalized signal-to-noise ratio. Represents frequency The frequency band weights are determined by the energy proportions of the three frequency bands: ;in, Indicates low-frequency weights. Indicates the intermediate frequency weight. Indicates high-frequency weights, Indicates the proportion of low-frequency energy. Indicates the proportion of high-frequency energy; The playback audio data is framed and subjected to FFT transformation to obtain a frequency domain signal. The frequency domain signal is then multiplied with the gain curve in the frequency domain to obtain a gain-adjusted frequency domain signal. An inverse FFT transformation is then performed on the gain-adjusted frequency domain signal to obtain gain-optimized audio data.

7. The AI-based cloud terminal dynamic audio processing method according to claim 6, characterized in that, The scene adaptive noise suppression parameters include noise suppression intensity, and the calculation formula for the noise suppression intensity is as follows: ； in, Indicates the noise suppression strength. This represents the normalized signal-to-noise ratio. This indicates the classification confidence level of the scene category in the scene recognition results; The playback audio data is processed based on scene-adaptive noise suppression parameters, including: The noise suppression algorithm is selected according to the scene category, and the gain-optimized audio data is denoised according to the noise suppression intensity and the corresponding noise suppression algorithm to obtain the denoised audio data.

8. The AI-based cloud terminal dynamic audio processing method according to claim 7, characterized in that, The noise suppression algorithm is selected based on the scene category, including: For low-frequency dominant noise scenarios, an improved spectral subtraction method is adopted, with the noise estimation window set to 500ms and the oversubtraction factor of the spectral subtraction method set to 4.

0. For mid-frequency dominant noise scenarios, Wiener filtering is used, with the signal-to-noise ratio threshold set to 10dB and the filter order set to 32. For high-frequency dominant noise scenarios, wavelet thresholding is used for denoising, with the decomposition level set to 4 and a soft thresholding function. ,in, This represents the threshold for wavelet denoising. This represents the standard deviation of noise.

9. The AI-based cloud terminal dynamic audio processing method according to claim 7, characterized in that, The anti-distortion dynamic range control parameters include compression ratio, dynamic threshold, attack time, release time, and hard limiter threshold. The playback audio data is processed based on the anti-distortion dynamic range control parameters, including: The instantaneous level of the noise-reduced audio data is monitored in real time. When the instantaneous level exceeds the dynamic threshold, the required gain reduction is calculated based on the compression ratio and the degree to which the instantaneous level exceeds the threshold. The gain reduction is then applied to compress the noise-reduced audio data according to the attack time. When the instantaneous level falls below the dynamic threshold, the gain is restored to the normal level according to the release time; For audio data that has undergone dynamic range compression, a hard limiter is used to clip signal peaks that exceed the hard limiter threshold to obtain optimized audio data.

10. The AI-based cloud terminal dynamic audio processing method according to claim 1, characterized in that, The user feedback data includes users' subjective rating data and text feedback data; The formula for calculating the reward value is as follows: ； in, Indicates the reward value. This represents the subjective rating reward, obtained through mapping subjective rating data. The reward for text feedback is obtained through text feedback data mapping. This represents the behavioral feedback reward, obtained through mapping behavioral feedback data. Indicates the weight of subjective rating rewards. This indicates the weight of the text feedback reward.

Images