An audio sound effect enhancement processing method, system and storage medium

Abstract

The application relates to the technical field of digital audio processing, and discloses a sound effect enhancement processing method and system and a storage medium, wherein the sound effect enhancement processing method comprises the following steps: collecting environmental acoustic characteristics in real time, and constructing an environmental characteristic vector; deeply analyzing audio content, and generating a content characteristic function; fusing environmental and content characteristics in a joint characteristic space; identifying key audio components based on a spectrum attention algorithm; dynamically generating time-frequency domain processing parameters by applying an adaptive mapping function; and performing personalized adjustment according to user preferences; the application realizes true environmental adaptability, can adopt differentiated processing strategies for different types of audio content, improves voice intelligibility, music expressiveness and environmental sound authenticity, and through the collaborative optimization of the environment and the content, the system overcomes the limitations of traditional sound effect processing, effectively protects the hearing of users, reduces auditory fatigue, and improves the audio experience in various complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of digital audio processing technology, and more specifically, to a method, system, and storage medium for enhancing audio effects. Background Technology

[0002] With the rapid development of digital audio technology, audio processing systems have been widely used in various scenarios such as smartphones, smart headphones, car audio systems, and smart homes. Traditional audio processing technology mainly uses fixed parameters or preset modes for sound effect processing. This method can provide basic audio enhancement effects in stable environments. However, in practical applications, the acoustic environment in which users are located is complex and varied, and the types of audio content are also different, which places higher demands on audio processing systems.

[0003] Current audio processing technologies face challenges such as insufficient environmental adaptability and limited content understanding capabilities. In noisy environments, important audio content is often masked by noise, and existing technologies struggle to accurately identify and enhance these critical components. Furthermore, different types of audio content (such as speech, music, and ambient sound) possess different characteristics and processing requirements, but current technologies lack a deep understanding of the semantic level of audio content, failing to provide differentiated processing strategies. Moreover, in rapidly changing environments, the coherence and smooth transition of sound effects processing also become urgent issues to be addressed.

[0004] To enhance the user's audio experience, a sound processing system is needed that can adapt to both environmental changes and content variations. This system should be able to perceive the user's acoustic environment in real time, deeply understand the characteristics of the audio content, and perform collaborative optimization across both environmental and content dimensions. This would provide a consistently high-quality audio experience across various complex scenarios, while also considering computational efficiency and protecting the user's hearing health. Summary of the Invention

[0005] This invention provides a method, system, and storage medium for enhancing audio effects, addressing the technical problems of insufficient environmental adaptability, limited content understanding capabilities, separation of environment and content processing, waste of computing resources, and inconsistent user experience in related technologies.

[0006] This invention provides a method for enhancing audio effects, comprising the following steps: Take environmental acoustic sampling data, extract environmental feature vectors including noise level, spectrum distribution, and reverberation characteristics, and perform time-frequency transformation on the input audio to obtain the audio time spectrum, extract audio content features, and determine the audio content type; Based on the acquired audio time spectrum, extracted audio content features and content type information, a spectrum attention algorithm is applied to generate an attention distribution map representing the perceived importance of each time frequency point; Based on environmental feature vectors and audio content features, a joint feature space of environment and content is constructed. A correlation analysis algorithm is applied to establish a balance between environmental adaptability and content fidelity, and a multi-objective optimization function for sound effect processing is constructed. Based on the constructed environmental content joint feature space and multi-objective optimization function, a time-frequency domain processing function is generated through an adaptive mapping function, and an adaptive filter bank is constructed to process the audio signal; Record user interaction feedback data on the processed audio, construct a user preference model using a Bayesian learning framework, optimize the adaptive mapping function, and achieve personalized sound effect enhancement.

[0007] In a preferred embodiment, the environmental feature vector further includes at least one of the following parameters: spatial characteristic parameters, motion state parameters, and location-related features; The steps for extracting environmental feature vectors include: An environmental acoustic feature extraction algorithm is applied to analyze the sampled data, extract environmental parameters, and generate an initial environmental feature vector; By integrating data from other auxiliary sensors, a multimodal data fusion algorithm is applied to construct a complete representation of the environmental state. The current environment is classified using an environmental classification model, and the environmental type identifier and environmental parameter set are output.

[0008] In a preferred embodiment, the step of generating an attention distribution map characterizing the perceived importance at each time frequency point includes: Based on spectral features and content type information, a spectral attention algorithm is applied to calculate the attention distribution map in the time-frequency domain. The spectrum attention algorithm consists of three modules: perceptual importance assessment, content relevance analysis, and environmental contrast calculation. The output is weighted and fused to form the final attention distribution map.

[0009] In a preferred embodiment, the step of constructing the joint feature space of environment content includes: By applying correlation analysis algorithms, the mutual influence relationships between environmental features and content features across various dimensions are calculated, and a correlation matrix is ​​generated. Based on the results of environmental content correlation analysis, a multi-objective optimization function for sound effect processing is constructed. The parameter dataset is processed using labeled environmental content, and an adaptive mapping function is trained.

[0010] In a preferred embodiment, the adaptive filter bank includes at least two of the following components: a gain control filter, a frequency selective filter, a dynamic range processor, a spatial enhancement processor, and a psychoacoustic optimization unit. Among them, the gain control filter applies different gain coefficients to different time-frequency points based on the gain parameter in the processing function; Frequency-selective filters construct adaptive bandpass, notch, or shelving filters based on attention distribution patterns and environmental characteristics. The dynamic range processor dynamically adjusts the compression ratio, threshold, and release time based on ambient noise levels and content type; The spatial enhancement processor adjusts the width, depth, and positioning accuracy of the stereo image based on the characteristics of the ambient space and the content requirements.

[0011] In a preferred embodiment, the multi-objective optimization function includes an environmental adaptability objective function, a content fidelity objective function, and a computational complexity objective function, and the three are balanced by weighting coefficients; The weighting coefficients are dynamically adjusted based on the device capabilities and application scenarios. On resource-constrained devices, the weight of the computational complexity objective function is increased to reduce computational resource consumption. In high-fidelity audio processing scenarios, the weight of the content fidelity objective function is increased to ensure content quality.

[0012] In a preferred embodiment, the steps of constructing a user preference model include: Using a Bayesian learning framework, user feedback is treated as an observation of the latent preference model; The user preference representation is gradually optimized through three core components: a priori preference model, a feedback interpreter, and a preference updater. A hierarchical structure is adopted to consider both general preferences and specific scenario preferences, balancing the model's generalization ability and personalization.

[0013] In a preferred embodiment, the adaptive mapping function is implemented using a deep neural network architecture, including an encoder-decoder architecture: The encoder part consists of a multi-layer convolutional network, used to extract high-dimensional representations of environmental and content features; The decoder consists of transposed convolutional layers and fully connected layers, used to generate time-frequency domain processing functions; Introducing an attention mechanism into the network enables the model to focus on key elements in the environmental and content features, thereby improving mapping accuracy.

[0014] In a preferred embodiment, an audio effect enhancement processing system is used to perform an audio effect enhancement processing method, including: The environmental content perception module is used to acquire environmental acoustic sampling data, extract environmental feature vectors including noise level, spectrum distribution, and reverberation characteristics, and perform time-frequency transformation on the input audio to obtain the audio time spectrum, extract audio content features, and determine the audio content type. The attention processing module is used to generate an attention distribution map representing the perceived importance of each time-frequency point based on the acquired audio time-frequency spectrum, extracted audio content features and content type information, using a spectral attention algorithm. The feature mapping module is used to construct a joint feature space of environment and content based on environmental feature vectors and audio content features, apply correlation analysis algorithms to establish a balance between environmental adaptability and content fidelity, and construct a multi-objective optimization function for sound effect processing. The audio processing module is used to generate time-frequency domain processing functions through an adaptive mapping function based on the constructed environmental content, the joint feature space, and the multi-objective optimization function, and to construct an adaptive filter bank to process the audio signal. The user feedback module records user interaction feedback data on the processed audio. It uses a Bayesian learning framework to build a user preference model and optimizes the adaptive mapping function to achieve personalized sound effect enhancement.

[0015] In a preferred embodiment, a computer-readable storage medium is provided for storing computer-readable instructions that, when read by a computer, enable the execution of an audio enhancement processing system.

[0016] The beneficial effects of this invention are as follows: It achieves true environmental adaptability, capable of sensing and analyzing the acoustic environment in which the user is located in real time, and dynamically adjusting the sound processing strategy to ensure that the audio output is always at its best. Whether in a noisy street, in a moving vehicle, or in a quiet indoor environment, the system can provide a consistent high-quality audio experience, solving the problem of unstable performance of traditional sound processing when the environment changes.

[0017] Through deep content understanding and spectral attention algorithms, this system can intelligently identify important components of audio content and adopt differentiated processing strategies for different types of audio content (such as speech, music, and ambient sound). This content-based adaptive processing mechanism significantly improves the clarity of speech, the expressiveness of music, and the realism of ambient sound, while effectively reducing computational resource consumption and improving system operating efficiency.

[0018] By establishing a joint feature space of environmental feature vectors and content feature functions, the system achieves synergistic optimization of environmental factors and content characteristics. This innovative two-dimensional adaptive mechanism not only overcomes the limitations of the separation between environment and content in traditional audio processing but also allows for personalized adjustments based on user preferences, providing an audio experience that better meets user expectations. The system's self-learning capability enables it to continuously optimize its processing strategies as usage time increases, thereby continuously improving user satisfaction.

[0019] The time-frequency domain processing function enables precise control of audio signals, effectively protecting users' hearing while ensuring sound quality and reducing auditory fatigue caused by prolonged use. This intelligent audio processing solution not only enhances the immediate user experience but also safeguards users' long-term hearing health, embodying the combination of technology and humanistic care. Attached Figure Description

[0020] Figure 1 This is a flowchart of a sound effect enhancement processing method according to the present invention; Figure 2 This is a bar chart comparing speech clarity scores under different environments according to the present invention; Figure 3 This is a line graph showing the change in auditory comfort during prolonged listening in a noisy environment, according to the present invention. Figure 4 This is a radar chart of the sound effect performance scores for each dimension of the music content in this invention; Figure 5 This is a dual-axis chart showing the relationship between computational resource consumption and sound quality score in this invention; Figure 6 This is a bar chart showing the system satisfaction scores for different user groups of this invention; Figure 7 This is a scatter plot showing the prediction accuracy analysis of the adaptive mapping function of this invention. Detailed Implementation

[0021] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0022] At least one embodiment of the present invention discloses a sound effect enhancement processing method, such as... Figure 1 As shown, it includes the following steps: Step 1: Acquire environmental acoustic sampling data, extract environmental feature vectors including noise level, spectral distribution, and reverberation characteristics, perform time-frequency transformation on the input audio, obtain the time spectrum of the audio, extract audio content features, and determine the audio content type; Specifically, it includes the following sub-steps: Step 1.1, Environmental acoustic sampling; The device uses its built-in microphone to collect ambient audio samples with a sampling duration of 200-500 milliseconds and a sampling frequency of 16kHz, forming raw ambient acoustic data.

[0023] Optionally, in some implementations, the sampling duration can be adaptively adjusted according to the complexity of the environment, and in complex environments, it can be extended to 800 milliseconds to obtain more complete acoustic features.

[0024] Step 1.2, Environmental Feature Extraction; An environmental acoustic feature extraction algorithm is applied to analyze the sampled data and extract key environmental parameters, including background noise level (dB value), noise spectrum distribution, and spatial reverberation characteristics, to generate an initial environmental feature vector.

[0025] It should be understood that this feature extraction process can be achieved using a method combining short-time Fourier transform with Mel frequency cepstral coefficients (MFCC) calculation.

[0026] like Figure 2 As shown, this paper presents a comparison of the clarity scores of our system and traditional methods in different environments. The clarity score ranges from 0 to 100, with higher scores indicating clearer speech. It can be seen that our system significantly outperforms traditional methods in all types of environments, especially in noisy environments (noisy streets, subway cars), demonstrating the system's advantage in environmental adaptability.

[0027] Step 1.3, sensor data fusion; By integrating data from other auxiliary sensors (such as accelerometers, gyroscopes, light sensors, GPS, etc.) and combining them with environmental acoustic characteristics, a multimodal data fusion algorithm is applied to construct a complete representation of the environmental state.

[0028] The algorithm employs a weight-based feature fusion method to output a unified environmental feature vector E. In some embodiments, the weights can be automatically adjusted using a Bayesian optimization method to adapt to different devices and application scenarios.

[0029] Step 1.4, Environment Type Identification; Based on the extracted environmental feature vector E, the current environment is classified using an environmental classification model, and it is categorized into predefined environmental types (such as quiet indoor, noisy outdoor, vehicle interior, outdoor open space, etc.), outputting environmental type identifiers and environmental parameter sets.

[0030] The environment classification model is a multi-layer neural network structure containing three fully connected layers, each with 64, 32, and 16 neurons respectively. It uses the ReLU activation function and finally outputs the probability distribution of each environment category through a softmax layer.

[0031] This model takes the environmental feature vector E as input and is trained on a large amount of labeled data through supervised learning, effectively distinguishing different acoustic environment types. The environmental classification model also possesses uncertainty estimation capabilities, providing confidence scores for classification results and employing a mixed-class strategy when environmental features are ambiguous.

[0032] It should be noted that, in some embodiments, the environment classification model may further include a recurrent neural network layer to capture the temporal variation patterns of environmental features and improve the ability to identify dynamic environments.

[0033] The environmental feature vector E can be represented as: ; in, This is a noise level parameter, representing the intensity of ambient noise. These represent noise spectrum characteristic parameters, describing the distribution of environmental noise across different frequency bands. Reverberation time is a parameter that reflects the acoustic reflection characteristics of the environment and represents the time required for sound to decay to a certain degree in space. These represent spatial characteristic parameters, describing the physical spatial attributes of the environment, such as spatial size and shape. These are motion state parameters, representing the motion state of the device or user, such as stationary, walking, or riding in a vehicle. This represents location-related features, environmental parameters related to the user's geographical location.

[0034] The aforementioned environmental feature vector E is a vector composed of multiple feature parameters of different dimensions, including: noise level parameters (representing the intensity of environmental noise, usually in decibels), noise spectrum feature parameters (describing the distribution of environmental noise across different frequency bands), reverberation time parameters (reflecting the acoustic reflection characteristics of the environment, representing the time required for sound to decay to a certain level in space), spatial characteristic parameters (describing the physical spatial attributes of the environment, such as space size and shape), motion state parameters (representing the motion state of the equipment or user, such as stationary, walking, or riding in a vehicle), and location-related features (environmental parameters related to the user's geographical location). These parameters together constitute a complete environmental representation, providing environmental context information for subsequent processing.

[0035] like Figure 3As shown, this diagram illustrates the trend of listener comfort over time during prolonged listening in a noisy environment. Comfort scores (0-100) were compared for the original audio, traditionally processed audio, and audio processed by this system. It can be seen that as the duration of ambient noise increases, the comfort levels of the original audio and traditional processing methods decrease significantly. However, the audio processed by this system maintains a high level of comfort even after prolonged listening, demonstrating the system's advantages in long-term use scenarios and its ability to prevent auditory fatigue.

[0036] Step 2: Based on the acquired audio time spectrum, extracted audio content features and content type information, apply the spectral attention algorithm to generate an attention distribution map representing the perceived importance of each time frequency point; Specifically, it includes the following sub-steps: Step 2.1, Time-frequency domain transformation; Perform a short-time Fourier transform (STFT) on the input audio signal to convert it from the time domain to the time-frequency domain representation, generating the time-frequency spectrum of the audio.

[0037] The transform uses a 25ms frame length, a 10ms frame shift, and a Hanning window function to form a complex time spectrum S(f,t).

[0038] Optionally, when processing low-frequency important audio content (such as certain music genres), the frame length can be increased to 50ms to obtain higher frequency resolution.

[0039] Step 2.2, Spectral Feature Extraction; Based on the time-spectrum diagram, a series of spectral characteristic parameters are calculated, including band energy distribution, spectral entropy, spectral centroid, spectral flux, etc., forming a multidimensional spectral characteristic matrix.

[0040] In addition, in some embodiments, modulation spectrum features can be calculated to capture the temporal dynamic characteristics of audio content.

[0041] Step 2.3, Content Type Identification; The content classification model is used to analyze spectral characteristics, determine the type of audio content (such as speech, music, ambient sound or a combination thereof), assign weight coefficients to different types, and output content type identifiers and type weight vectors.

[0042] The content classification model employs a convolutional neural network (CNN) structure, comprising three convolutional layers and two fully connected layers. The convolutional layers use kernels of different sizes to capture time-frequency domain features, max pooling layers reduce data dimensionality, and the fully connected layers integrate features and perform classification.

[0043] This model can simultaneously identify multiple content types and their mixtures, outputting multi-label classification results and corresponding confidence scores. The model is trained using a large-scale audio dataset, containing individual samples and mixtures of various speech, music, and ambient sounds, ensuring good generalization ability in practical applications.

[0044] It should be understood that, in some embodiments, the content classification model may employ transfer learning methods to transfer knowledge from a pre-trained audio classification model in order to reduce reliance on large-scale labeled data.

[0045] like Figure 4 The figure shows a performance comparison between our system and traditional methods across various sound effect dimensions when processing music content. The score ranges from 0 to 100, with higher scores indicating better performance. As can be seen from the figure, our system outperforms traditional methods in all sound effect dimensions, with particularly significant improvements in spatial localization, noise reduction, and dynamic range. This demonstrates our system's comprehensive optimization capabilities in music content processing and the adaptability of our attention algorithm to different audio content types.

[0046] Step 2.4: Generation of attention distribution map; Based on spectral features and content type information, a spectral attention algorithm is applied to calculate the attention distribution map in the time-frequency domain. This figure reflects the perceptual importance at each time and frequency point.

[0047] The attention algorithm combines a perceptual psychoacoustic model, taking into account the differences in human ear sensitivity to different frequencies and the masking effect, to generate an attention weight matrix ranging from 0 to 1.

[0048] The spectral attention algorithm consists of three main computational modules: perceptual importance assessment, content relevance analysis, and environmental contrast calculation.

[0049] Perceptual importance assessment is based on the human ear equal-loudness curve and masking threshold curve, calculating the basic perceptual importance at each time-frequency point.

[0050] Content relevance analysis uses semantic information to estimate the contribution of each time-frequency point to content understanding; Environmental contrast calculation and analysis are used to analyze the energy differences between environmental noise and audio signals in different frequency bands.

[0051] The outputs of the three modules are weighted and fused to form the final attention distribution map, with the weight coefficients dynamically adjusted according to the content type.

[0052] For speech content, the algorithm assigns higher attention weights to key frequency bands of the speech. Regarding the musical content, more emphasis is placed on maintaining spectral balance and timbre integrity.

[0053] Optionally, the attention algorithm can also integrate temporal information to identify transient events (such as percussion in music or plosives in speech) and assign higher attention weights to these brief but important sound events.

[0054] Content Feature Function It can be represented as: ; The above content feature function It is a multidimensional feature representation composed of three main components; it is a function of time and frequency, corresponding to each time-frequency point of the audio signal. The time-spectral representation of an audio signal describes the energy distribution of different frequency components at different points in time. It represents the content type characteristics, describing the audio content type (such as speech, music, or ambient sound) and its confidence level for each time frequency point; This represents the attention distribution map, which characterizes the perceptual importance at each time point, with values ​​ranging from 0 to 1.

[0055] These three components together provide a comprehensive description of the audio content in the time-frequency domain, providing content-related information for adaptive audio processing.

[0056] like Figure 5 The figure shows the relationship between the computational resource utilization (bar chart, left Y-axis) and the corresponding sound quality score ratio (line chart, right Y-axis) of the system under different frequency spectrum processing ranges. The sound quality score ratio is based on the score of full-band processing (1.0). It can be seen that as the processing range decreases, the computational resource utilization decreases significantly, while the sound quality score only decreases slightly. In particular, when processing 60% of the frequency band, the computational resource utilization decreases by 35%, but the sound quality score only decreases by about 23%. This verifies that the system's two-dimensional optimization model can concentrate computational resources in the perceptually important time-frequency region, achieving a good balance between computational efficiency and sound quality performance.

[0057] Step 3: Based on environmental feature vectors and audio content features, construct a joint feature space of environment and content, apply correlation analysis algorithm to establish a balance between environmental adaptability and content fidelity, and construct a multi-objective optimization function for sound effect processing; Specifically, it includes the following sub-steps: Step 3.1, Feature space construction; Construct a joint feature space for environmental content, and combine environmental feature vectors and content feature function Mapped into this space, a unified feature representation is formed.

[0058] It should be noted that this joint feature space adopts a multi-level structure, with the lower level representing physical acoustic features and the higher level representing semantic association features, thereby achieving comprehensive optimization at both the physical and perceptual levels.

[0059] Step 3.2, Feature Correlation Analysis; By applying correlation analysis algorithms, the mutual influence relationships between environmental features and content features across various dimensions are calculated, generating a correlation matrix. This matrix quantifies the degree of influence of environmental factors on different content components.

[0060] In some embodiments, the association analysis may employ canonical correlation analysis (CCA) to capture the direction of maximum correlation between two sets of features.

[0061] Step 3.3, optimize target construction; Based on the results of environmental content correlation analysis, a multi-objective optimization function for sound effect processing is constructed, which establishes a balance between environmental adaptability and content fidelity.

[0062] Optimize objective function It can be expressed as a weighted combination of the environmental adaptability objective function and the content fidelity objective function.

[0063] In addition, in some embodiments, the optimization objective may also include a computational complexity term, minimizing resource consumption while ensuring sound quality.

[0064] Optimize objective function The expression is: ; in, To comprehensively optimize the objective function, it represents the overall objective value that the system needs to minimize; This represents the environmental adaptability objective function, which measures the degree to which the treatment results adapt to the current acoustic environment. The objective function for content fidelity measures the degree to which semantic information of audio content is preserved. It represents the objective function of computational complexity, which measures the computational resource consumption of the algorithm; This represents the environmental adaptability weight coefficient, which controls the importance of environmental adaptability in the overall optimization. This represents the content fidelity weighting coefficient, which controls the importance of content fidelity in the overall optimization. This represents the computational complexity weight coefficient, controlling the importance of computational efficiency in the overall optimization; the weight coefficient satisfies the following condition: This ensures a relative balance among the various objectives.

[0065] The above optimization objective function expresses the three core objectives that the system needs to balance: environmental adaptability, content fidelity, and computational complexity.

[0066] First, the system obtains the environmental adaptability target value, which reflects the degree to which the processing results adapt to the current environmental conditions; At the same time, we need to obtain the content fidelity target value, which represents the degree to which the semantic information of the audio content is preserved; we also need to obtain the computational complexity target value, which measures the computational resource consumption of the algorithm.

[0067] Then, the system calculates the product of each of the three target values ​​and its corresponding weight coefficient, which reflects the relative importance of each target in the current scenario.

[0068] Finally, the three weighted objective values ​​are added together to obtain the comprehensive optimization objective value. The smaller this value, the better the processing solution.

[0069] The system selects the combination of processing parameters that minimizes the overall optimization objective value as the final solution. The sum of the weight coefficients equals 1, ensuring the relative balance of each objective.

[0070] Step 3.4, parameter mapping model training; Use the labeled environment content to process the parameter dataset and train an adaptive mapping function. This function can generate the optimal set of processing parameters based on the characteristics of the input environment and content.

[0071] The mapping function employs a non-linear model structure, which is able to capture the complex relationships between features.

[0072] Adaptive mapping function It is implemented using a deep neural network structure, which includes an encoder-decoder architecture.

[0073] The encoder part consists of a multi-layer convolutional network, used to extract high-dimensional representations of environmental and content features; The decoder consists of transposed convolutional layers and fully connected layers, used to generate time-frequency domain processing functions.

[0074] Introducing an attention mechanism into the network enables the model to focus on key elements in the environmental and content features, thereby improving mapping accuracy.

[0075] The training process employs an end-to-end supervised learning method. The loss function comprehensively considers the accuracy of the processing parameters and the perceptual quality of the generated audio, and is optimized using a large amount of environmental content parameter triplet data.

[0076] In addition, the mapping function includes an adaptive layer that can be personalized based on user feedback, improving the model's ability to adapt to specific user preferences.

[0077] Alternatively, on devices with limited computing resources, the mapping function can be simplified using knowledge distillation techniques, retaining core functionality while reducing computational requirements.

[0078] Adaptive mapping function The core computational expression is: ; in, Indicates time and frequency points The processing strategy includes multi-dimensional processing instructions such as gain adjustment, filter parameters, and dynamic range control parameters; This represents an adaptive mapping function used to map environmental and content features to the processing parameter space; These are frequency-dimensional coordinates, representing a specific frequency point; These are time-dimensional coordinates, representing specific points in time. It represents an environmental feature vector, which includes environmental information such as environmental noise characteristics and spatial acoustic characteristics; This represents the content feature function, describing the audio content at time and frequency points. Its characteristics, including time-spectrum representation Content type characteristics Attention distribution map .

[0079] The above function describes how the system determines the audio processing strategy based on environmental and content characteristics.

[0080] First, the system obtains the current time and frequency point. The coordinate information, along with the input environmental feature vector. and the content feature function value at that time and frequency point ; Then, through the adaptive mapping function The function processes these inputs, including multiple processing stages such as data normalization, feature extraction, correlation analysis, and parameter generation. Finally, the system outputs values ​​specific to time and frequency points. Specific handling strategies This strategy includes multi-dimensional processing instructions such as gain adjustment, filter parameters, and dynamic range control parameters.

[0081] The processing strategy is dynamically adjusted based on the environment and content characteristics, enabling audio processing to adapt to environmental changes while preserving the key features of the content.

[0082] like Figure 6The diagram shows a comparison of satisfaction ratings from different user groups regarding the traditional method, the average performance of this system, and the personalized adjustments made to this system. The rating ranges from 0 to 100, with higher scores indicating higher satisfaction. It can be seen that this system achieved higher satisfaction across all user groups compared to the traditional method, with the personalized adjustments demonstrating even more outstanding performance, particularly among audiophiles and commuters. This verifies that the system's user feedback learning mechanism can effectively adapt to different user preferences, providing a truly personalized audio enhancement experience.

[0083] Step 4: Based on the constructed environmental content joint feature space and multi-objective optimization function, a time-frequency domain processing function is generated through an adaptive mapping function, and an adaptive filter bank is constructed to process the audio signal; Specifically, it includes the following sub-steps: Step 4.1, process parameter generation; Using the trained adaptive mapping function Combined with the current environmental feature vector and content feature function Calculate the optimal time-frequency processing function This function specifies the specific processing strategy for each time-frequency point of the audio signal.

[0084] It should be understood that processing parameters include, but are not limited to, gain coefficients, filter parameters, dynamic range control parameters, and spatial processing parameters.

[0085] Step 4.2, Adaptive filter construction; Based on the generated processing function A time-varying adaptive filter bank is constructed, including gain control filters, frequency selective filters, dynamic range processors, etc., forming a complete processing chain.

[0086] An adaptive filter bank is a dynamically configurable collection of digital signal processing units, mainly including the following components: Gain-controlled filter: based on processing function The gain parameter in the filter allows for selective enhancement by applying different gain coefficients to different time-frequency points. The filter employs a parameterized structure, enabling adjustment of amplitude characteristics while maintaining phase characteristics.

[0087] Frequency-selective filter: based on attention distribution map Based on environmental characteristics, adaptive bandpass, notch, or shelving filters are constructed to perform fine processing on specific frequency bands. This filter employs a hybrid architecture of IIR (Infinite Impulse Response) and FIR (Finite Impulse Response) to balance processing performance and computational efficiency.

[0088] Dynamic Range Processor: Based on ambient noise levels and content type, this processor dynamically adjusts parameters such as compression ratio, threshold, and release time to optimize the dynamic range performance of audio. It achieves multi-band adaptive compression, avoiding the "pumping" effect of traditional single-band compression.

[0089] Spatial Enhancement Processor: Adjusts the width, depth, and positioning accuracy of the stereo image based on the characteristics of the ambient space and the requirements of the content, enhancing spatial expressiveness. This processor achieves spatial effect adjustment through precise control of the phase and level differences between channels.

[0090] Optionally, in some embodiments, the filter bank may also include a psychoacoustic optimization unit that dynamically adjusts processing parameters based on an auditory masking model to maximize the perceptual effect while minimizing auditory artifacts.

[0091] Step 4.3, Audio signal processing; The original audio signal is processed through an adaptive filter bank to achieve adaptive enhancement for different environments and content characteristics, and the processed audio signal is output.

[0092] In addition, in some embodiments, the processing can employ a parallel computing architecture, distributing processing tasks of different frequency bands to multiple processing cores to improve the real-time performance of the system.

[0093] Step 4.4, smooth transition processing; When environmental or content characteristics change, a smooth transition algorithm is applied to adjust processing parameters to avoid auditory discomfort caused by sudden changes and to ensure the continuity and natural transition of audio output.

[0094] Optionally, the transition smoothing algorithm can adaptively adjust the smoothing time constant according to the rate of change of the environment, using a shorter transition time in a rapidly changing environment and a longer transition time in a slowly changing environment.

[0095] like Figure 7 The figure shows the correlation between the system's adaptive mapping function's predicted user satisfaction and actual user satisfaction (range 0-1). Each point represents a different combination of processing parameters and its effect evaluation. As can be seen from the figure, the system's predicted values ​​and actual user satisfaction exhibit a strong positive correlation, with most points concentrated near the ideal diagonal, indicating that the adaptive mapping function Φ has high predictive accuracy. Especially in the high satisfaction range (0.7-0.95), the prediction accuracy is even higher, which is precisely the main operating range of the system during actual operation, proving that the system can accurately predict and meet users' expectations for sound effect processing.

[0096] Step 5: Record user interaction feedback data on the processed audio, construct a user preference model using a Bayesian learning framework, optimize the adaptive mapping function, and achieve personalized sound effect enhancement.

[0097] Specifically, it includes the following sub-steps: Step 5.1, User interaction data collection; Record users' direct adjustment behaviors to the system's processing results (such as changes to volume and equalizer settings) and usage pattern data (such as playback duration and skipping behavior) to form a user feedback dataset.

[0098] It should be noted that the data collection process follows privacy protection principles, is conducted only with user authorization, and all data is anonymized.

[0099] Step 5.2, Preference Model Construction; Based on collected user feedback data, a preference learning algorithm is applied to construct a personalized user preference model, which represents the user's sound effect preferences under different environments and content types.

[0100] The preference learning algorithm adopts a Bayesian learning framework, treating user feedback as an observation of the potential preference model, and gradually optimizing the user preference representation through iterative updates.

[0101] The algorithm comprises three core components: a prior preference model, a feedback interpreter, and a preference updater. The prior preference model provides basic preference estimates based on user demographic characteristics and initial usage behavior; The feedback interpreter analyzes the correlation between user adjustment behavior and the current environment and content, and extracts implicit preference information; The preference updater updates the distribution of user preference parameters across different combinations of content in various environments based on new observation data. The preference model adopts a hierarchical structure, considering both general preferences and scenario-specific preferences, thus balancing the model's generalization ability and personalization level.

[0102] Optionally, in some embodiments, the preference model may also integrate collaborative filtering technology to use preference data of similar user groups to assist in modeling and accelerate the personalization process for new users.

[0103] Step 5.3, Adaptive mapping function optimization; Using a user preference model, an adaptive mapping function is applied. Personalized adjustments are made to make the output processing strategy more in line with the preferences of specific users, forming a user-adaptive mapping function. .

[0104] In addition, in some embodiments, the system can provide different levels of personalization options, allowing users to choose the intensity of personalization and balance the relationship between automatic system optimization and manual user control.

[0105] Step 5.4, continuous learning and updates; The system periodically analyzes accumulated user feedback data and updates the environment classification model, content classification model, and adaptive mapping function to achieve long-term optimization of system performance.

[0106] It is understandable that the system can use incremental learning methods to continuously integrate newly collected data to improve the model and avoid the problem of model performance decaying over time.

[0107] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for enhancing audio effects, characterized in that, The process includes the following steps: acquiring environmental acoustic sampling data, extracting environmental feature vectors including noise level, spectral distribution, and reverberation characteristics, performing time-frequency transformation on the input audio to obtain the audio's time-frequency spectrum, extracting audio content features, and determining the audio content type; based on the acquired audio time-frequency spectrum, extracted audio content features, and content type information, applying a spectral attention algorithm to generate an attention distribution map representing the perceived importance of each time-frequency point; the spectral attention algorithm includes three modules: perceived importance assessment, content relevance analysis, and environmental contrast calculation, and the output is weighted and fused to form the final attention distribution map, and the audio time-frequency spectrum, content type features, and the attention distribution map are used to construct a content feature function; based on the environmental feature vector and content feature function, an environmental content joint feature space is constructed, and a correlation analysis algorithm is applied to build... This paper establishes a balance between environmental adaptability and content fidelity, and constructs a multi-objective optimization function for audio effect processing. Based on the constructed environmental-content joint feature space and the multi-objective optimization function, a time-frequency domain processing function is generated through an adaptive mapping function, and an adaptive filter bank is constructed to process the audio signal. User interaction feedback data on the processed audio is recorded, and a user preference model is constructed using a Bayesian learning framework to optimize the adaptive mapping function, achieving personalized audio effect enhancement. Specifically, the perceptual importance assessment module calculates the basic perceptual importance of each time-frequency point based on the human ear's equal-loudness curve and masking threshold curve; the content relevance analysis module estimates the contribution of each time-frequency point to content understanding using semantic information; the environmental contrast calculation module analyzes the energy difference between environmental noise and audio signals in each frequency band; and the content feature function is expressed as... ,in The time-spectral representation of an audio signal. Indicates content type characteristics, The attention distribution map is used to characterize the perceptual importance of each time-frequency point; the time-frequency domain processing function is expressed as follows: ,in This indicates that the adaptive mapping function is used to map environmental features and content features to the processing parameter space. For frequency dimension coordinates, Using time as the coordinate dimension, This represents an environmental feature vector, containing environmental information such as environmental noise characteristics and spatial acoustic characteristics. Indicates time and frequency points The processing strategy includes multi-dimensional processing instructions for gain adjustment, filter parameters, and dynamic range control parameters.

2. The sound effect enhancement processing method according to claim 1, characterized in that, The environmental feature vector also includes at least one of the following parameters: spatial characteristic parameters, motion state parameters, and location-related features; The steps for extracting environmental feature vectors include: analyzing the sampled data using an environmental acoustic feature extraction algorithm to extract environmental parameters and generate an initial environmental feature vector; integrating data from other auxiliary sensors and using a multimodal data fusion algorithm to construct a complete environmental state representation; and classifying the current environment using an environmental classification model to output an environmental type identifier and a set of environmental parameters.

3. The sound effect enhancement processing method according to claim 1, characterized in that, The steps for constructing the joint feature space of environment and content include: applying an association analysis algorithm to calculate the mutual influence relationship between the dimensions of environment features and content features, and generating an association matrix; constructing a multi-objective optimization function for sound effect processing based on the results of environment and content association analysis; and training an adaptive mapping function using the labeled environment and content processing parameter dataset.

4. The sound effect enhancement processing method according to claim 1, characterized in that, The adaptive filter bank includes at least two of the following components: gain-controlled filter, frequency-selective filter, dynamic range processor, spatial enhancement processor, and psychoacoustic optimization unit; wherein, the gain-controlled filter applies different gain coefficients to different time-frequency points based on the gain parameter in the processing function; the frequency-selective filter constructs adaptive bandpass, notch, or shelving filters based on the attention distribution map and environmental characteristics; the dynamic range processor dynamically adjusts the compression ratio, threshold, and release time based on the ambient noise level and content type; and the spatial enhancement processor adjusts the stereo image width, depth perception, and positioning accuracy based on the environmental spatial characteristics and content requirements.

5. The sound effect enhancement processing method according to claim 1, characterized in that, The multi-objective optimization function includes an environmental adaptability objective function, a content fidelity objective function, and a computational complexity objective function, and the three are balanced by weighting coefficients; The weighting coefficients are dynamically adjusted based on device capabilities and application scenarios. On resource-constrained devices, the weight of the computational complexity objective function is increased to reduce computational resource consumption. In high-fidelity audio processing scenarios, the weight of the content fidelity objective function is increased to ensure content quality.

6. The sound effect enhancement processing method according to claim 1, characterized in that, The steps to build a user preference model include: adopting a Bayesian learning framework to treat user feedback as an observation of the potential preference model; gradually optimizing the user preference representation through three core components: a prior preference model, a feedback interpreter, and a preference updater; and adopting a hierarchical structure to consider both general preferences and specific scenario preferences, balancing the model's generalization ability and personalization.

7. The sound effect enhancement processing method according to claim 1, characterized in that, The adaptive mapping function is implemented using a deep neural network structure, which includes an encoder-decoder architecture: the encoder part consists of a multi-layer convolutional network, used to extract high-dimensional representations of environmental and content features; The decoder consists of transposed convolutional layers and fully connected layers, used to generate time-frequency domain processing functions; an attention mechanism is introduced into the network, enabling the model to focus on key elements in environmental and content features, thereby improving mapping accuracy.

8. A sound effect enhancement processing system, used to execute any one of the sound effect enhancement processing methods described in 1-7, characterized in that, include: The environmental content perception module is used to acquire environmental acoustic sampling data, extract environmental feature vectors including noise level, spectrum distribution, and reverberation characteristics, and perform time-frequency transformation on the input audio to obtain the audio time spectrum, extract audio content features, and determine the audio content type. The attention processing module is used to generate an attention distribution map representing the perceived importance of each time-frequency point based on the acquired audio time-frequency spectrum, extracted audio content features and content type information, using a spectral attention algorithm. The feature mapping module is used to construct a joint feature space of environment and content based on environmental feature vectors and audio content features, apply correlation analysis algorithms to establish a balance between environmental adaptability and content fidelity, and construct a multi-objective optimization function for sound effects processing; the sound effects processing module is used to generate a time-frequency domain processing function through an adaptive mapping function based on the constructed joint feature space of environment and content and the multi-objective optimization function, and construct an adaptive filter bank to process the audio signal; The user feedback module records user interaction feedback data on the processed audio. It uses a Bayesian learning framework to build a user preference model and optimizes the adaptive mapping function to achieve personalized sound effect enhancement.

9. A computer-readable storage medium, characterized in that, It is used to store computer-readable instructions, which, when read by a computer, enable the execution of an audio enhancement processing system as described in claim 8.

Images