A sound effect processing method and system thereof

Abstract

The present invention provides a sound effect processing method and a system thereof, which relate to the technical field of sound effect processing, and include calculating the difference between a first comparison value and a second comparison value of a source audio file after global sound effect processing, and judging the difference based on a preset threshold value; when it is judged that the difference is less than or equal to the preset threshold value, inputting the audio features after global sound effect processing and the difference value into a pre-trained classification model, and adjusting the sound effect parameters in real time based on the output result of the classification model; when it is judged that the difference value is greater than the preset threshold value, comparing the original audio features of the source audio file with the audio features after global sound effect processing, and determining the type of audio problem; selecting a sound effect repair strategy according to the type of audio problem, and repairing the audio file after global sound effect processing; and outputting an audio file adjusted in real time or an audio file after repair processing. The present invention improves the personalization and intelligence level of sound effect processing while ensuring audio quality.

Description

Technical Field

[0001] The present invention relates to the technical field of sound effect processing, and in particular to a sound effect processing method and a system thereof. Background Art

[0002] In the current audio processing field, audio processing is an important means to improve audio quality and enhance user experience. With the rapid development of digital audio technology, audio processing has evolved from simple volume adjustment and equalizer adjustment to global audio processing containing complex algorithms, such as surround sound, stereo enhancement, dynamic range compression, etc. These global audio processing aims to optimize audio signals through algorithms so that they can achieve the best auditory effect in various playback environments.

[0003] However, while global audio processing improves audio quality, it may also introduce some unforeseen problems. For example, some audio processing may distort the audio signal or produce discordant auditory effects on certain types of audio. These problems are often difficult to solve by simple parameter adjustments and require more sophisticated audio analysis and processing strategies.

[0004] Therefore, it is necessary to provide a sound effect processing method and system thereof to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a sound effect processing method and system thereof, which can automatically adjust the sound effect parameters according to different global sound effect processing intensity levels and the perceptual characteristics of the audio content, thereby improving the personalization and intelligence level of sound effect processing while ensuring the audio quality.

[0006] The present invention provides a sound effect processing method, which comprises the following steps:

[0007] In response to an audio playback instruction, a difference between a first comparison value and a second comparison value of a source audio file is calculated after global sound effect processing, and the difference is judged based on a preset threshold, wherein the first comparison value and the second comparison value are respectively audio perception hash values ​​of the source audio file before and after global sound effect processing;

[0008] When it is determined that the difference is less than or equal to a preset threshold, the audio features processed by the global sound effect and the difference are input into a pre-trained classification model, and the sound effect parameters are adjusted in real time based on the output result of the classification model;

[0009] When it is determined that the difference is greater than a preset threshold, the original audio features of the source audio file are compared with the audio features after being processed by the global sound effect, and the type of audio problem is determined based on the comparison result;

[0010] Selecting a sound effect repair strategy according to the determined audio problem type, and using the selected sound effect repair strategy to repair the audio file after the global sound effect processing;

[0011] Output and play the audio files that have been adjusted in real time or the audio files that have been repaired.

[0012] Preferably, the preset threshold is determined by searching a preset mapping table according to the intensity level of global sound effect processing, wherein the preset mapping table stores a mapping relationship between different intensity levels of global sound effect processing and corresponding acceptable upper limits of audio perception hash value differences.

[0013] Preferably, in response to the audio playback instruction, calculating the difference between the first comparison value and the second comparison value of the source audio file after the global sound effect processing, and judging the difference based on a preset threshold, comprises:

[0014] Extracting original audio features of the source audio file, and generating the first comparison value using a perceptual hash algorithm;

[0015] Perform global sound effect processing on the source audio file and extract the audio features after the global sound effect processing;

[0016] Using the same perceptual hash algorithm, generating the second comparison value based on the audio features processed by the global sound effect;

[0017] The difference between the first comparison value and the second comparison value is calculated, and compared with the preset threshold value determined by searching, wherein the difference is the Hamming distance.

[0018] Preferably, before searching the preset mapping table to determine the preset threshold, the method further includes the following steps:

[0019] Determines the intensity level of global sound effect processing based on the configuration parameters of global sound effect processing.

[0020] Preferably, the training process of the classification model includes:

[0021] Collect audio samples containing various sound processing types and intensity levels;

[0022] Perform global sound effect processing on each audio sample and record the difference in audio perception hash values ​​before and after processing;

[0023] Obtaining the user's auditory evaluation data on the audio samples before and after processing, including positive and negative evaluations;

[0024] The neural network model is trained using the collected audio samples, corresponding difference values, and user auditory evaluation data.

[0025] Preferably, when it is determined that the difference is less than or equal to a preset threshold, the audio features processed by the global sound effect and the difference are input into a pre-trained classification model, and the sound effect parameters are adjusted in real time based on the output result of the classification model, including:

[0026] Combining the audio features processed by the global sound effect and the difference into an input vector;

[0027] Inputting the input vector into a pre-trained classification model, wherein the classification model is a neural network model trained based on audio samples and corresponding user auditory evaluation data;

[0028] Based on the output result of the classification model, determine the type of sound effect parameter that needs to be adjusted and the adjustment range;

[0029] According to the output result of the classification model, the sound effect parameters are adjusted in real time.

[0030] Preferably, when it is determined that the difference is greater than a preset threshold, the original audio features of the source audio file are compared with the audio features after being processed by the global sound effect, and the type of the audio problem is determined based on the comparison result, including:

[0031] Compare the original audio features of the source audio file with the audio features processed by the global sound effect item by item;

[0032] The feature similarity algorithm is used to calculate the change amount of each audio feature;

[0033] According to the change amount of each audio feature, the audio problem type is determined based on a predefined problem classification rule.

[0034] Preferably, the step of selecting a sound effect repair strategy according to the determined audio problem type, and using the selected sound effect repair strategy to repair the audio file after the global sound effect processing comprises:

[0035] Selecting an audio repair strategy from a library of predefined audio repair strategies based on the determined audio problem type;

[0036] Apply the selected sound effect repair strategy to repair the audio file after global sound effect processing.

[0037] The present invention also provides a sound effect processing system for executing a sound effect processing method, the processing system comprising:

[0038] an audio quality assessment module, configured to respond to an audio playback instruction, calculate a difference between a first comparison value and a second comparison value of a source audio file after global sound effect processing, and judge the difference based on a preset threshold, wherein the first comparison value and the second comparison value are respectively audio perception hash values ​​of the source audio file before and after global sound effect processing;

[0039] A sound effect parameter real-time adjustment module, for inputting the audio features processed by the global sound effect and the difference into a pre-trained classification model when it is determined that the difference is less than or equal to a preset threshold, and adjusting the sound effect parameters in real time based on the output result of the classification model;

[0040] An audio problem type determination module, for comparing the original audio features of the source audio file with the audio features processed by the global sound effect when it is determined that the difference is greater than a preset threshold, and determining the audio problem type based on the comparison result;

[0041] A sound effect repair module, used to select a sound effect repair strategy according to the determined audio problem type, and use the selected sound effect repair strategy to repair the audio file after the global sound effect processing;

[0042] The audio playback module is used to output and play the audio files that have been adjusted in real time or the audio files that have been repaired.

[0043] Compared with the related art, the sound effect processing method and system provided by the present invention have the following beneficial effects:

[0044] The present invention calculates the difference in the audio perception hash value of the source audio file before and after the global sound effect processing, and judges the difference based on a preset threshold, thereby realizing automatic evaluation of the sound effect processing effect. When the difference is less than or equal to the preset threshold, the method uses a pre-trained classification model to adjust the sound effect parameters in real time; when the difference is greater than the preset threshold, the method determines the type of audio problem by comparing the original audio features with the processed audio features, and applies the corresponding sound effect repair strategy to perform repair processing.

[0045] This audio processing method not only improves the automation of audio processing, but also significantly improves the efficiency and accuracy of audio processing. By analyzing and adjusting audio parameters in real time, this method can ensure that the audio signal can maintain the best auditory effect under various processing conditions, thereby meeting the user's personalized needs for audio quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flow chart of a sound effect processing method provided by the present invention;

[0047] Figure 2A module structure diagram of a sound effect processing system provided by the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0049] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0050] Embodiment 1

[0051] The present invention provides a sound effect processing method, referring to Figure 1 As shown, the processing method comprises the following steps:

[0052] S1: In response to an audio playback instruction, calculate a difference between a first comparison value and a second comparison value of a source audio file after global sound effect processing, and judge the difference based on a preset threshold, wherein the first comparison value and the second comparison value are respectively audio perception hash values ​​of the source audio file before and after global sound effect processing.

[0053] Specifically, step S1 includes the following steps:

[0054] S11: extracting original audio features of the source audio file, and using a perceptual hash algorithm to generate the first comparison value.

[0055] In this embodiment, in order to ensure that the effect of the sound effect processing can be quantified and evaluated, a method is needed to capture the key features of the audio file. Perceptual hashing is a fingerprint technology based on audio content that can generate a short binary string (hash value) that can reflect the auditory characteristics of the audio to a certain extent. Through the perceptual hashing algorithm, the audio file can be converted into a unique identifier for subsequent comparison and analysis.

[0056] When implementing:

[0057] First, feature extraction is performed on the source audio file, and the extracted features include but are not limited to spectrum distribution, dynamic range, channel balance, phase information, etc. These features can fully describe the auditory characteristics of the audio and ensure that the subsequent perceptual hash generation is representative.

[0058] Then, the extracted audio features are processed using a perceptual hash algorithm (including but not limited to pHash, aHash or dHash) to generate a fixed-length binary string as the first comparison value. The core idea of ​​the perceptual hash algorithm is to extract the main features that can reflect the audio content by transforming the audio signal in the frequency domain or time domain, while ignoring subtle differences, ensuring that even if the audio file is slightly modified, the generated hash value remains similar.

[0059] The generated first comparison value (ie, the perceptual hash value) is stored as a reference for comparison with the processed audio features in subsequent steps.

[0060] S12: Perform global sound effect processing on the source audio file, and extract audio features after the global sound effect processing.

[0061] In this embodiment, global sound effect processing refers to applying a series of preset sound effect adjustments to the entire audio file to enhance the auditory experience of the audio. The types of global sound effect processing include low frequency enhancement, high frequency compression, stereo expansion, and surround sound effects. In order to ensure that the effect of the sound effect processing can be quantified and evaluated, it is necessary to re-extract the audio features after processing so as to compare them with the features before processing.

[0062] When implementing:

[0063] First, according to the user's settings, the specific parameters of the global sound effect processing are determined. For example, the user can select different low-frequency enhancement intensity levels.

[0064] Then, according to the configured parameters, global audio processing is performed on the source audio file. During the processing, a variety of audio adjustments are applied, such as equalizer, compressor, expander, etc., to optimize the auditory effect of the audio.

[0065] Finally, after completing the global sound effect processing, feature extraction is performed on the processed audio file again. The extracted features are the same as those in step S11, including spectrum distribution, dynamic range, channel balance, phase information, etc. These features will be used for subsequent perceptual hash value generation and difference calculation.

[0066] S13: Using the same perceptual hash algorithm, generate the second comparison value based on the audio features processed by the global sound effect.

[0067] In this embodiment, in order to ensure that the effect of the sound effect processing can be accurately evaluated, it is necessary to generate two perceptual hash values ​​before and after the processing and compare the two values. Due to the characteristics of the perceptual hash algorithm, the generated hash values ​​are comparable only when the same algorithm is used. Therefore, the same perceptual hash algorithm as step S11 must be used to ensure that the subsequent difference calculation is scientific and accurate.

[0068] When implementing:

[0069] The processed audio features extracted in step S12 are processed using the same perceptual hash algorithm as in step S11 to generate a binary string of fixed length as the second comparison value.

[0070] S14: Calculate the difference between the first comparison value and the second comparison value, and compare the difference with the preset threshold determined by searching, wherein the difference is the Hamming distance.

[0071] In this embodiment, in order to evaluate the effect of the sound effect processing, a method is needed to quantify the difference before and after the processing. Hamming distance is a commonly used measurement method for measuring the difference between two binary strings. By calculating the Hamming distance of the perceptual hash value, the impact of the sound effect processing on the audio can be quantified, and it can be determined whether the processed audio has changed significantly. The preset threshold is determined by searching from a preset mapping table based on the intensity level of the global sound effect processing, ensuring that different types of sound effect processing can have reasonable evaluation standards.

[0072] When implementing:

[0073] Calculate the Hamming distance between the first comparison value (the perceptual hash value before processing) and the second comparison value (the perceptual hash value after processing). The Hamming distance is calculated by counting the number of different bits in the two binary strings. For example, if the two hash values ​​are 101010 and 100110, their Hamming distance is 2 because there are two different bits.

[0074] According to the intensity level of the global sound effect processing, the corresponding preset threshold is searched from the preset mapping table.

[0075] Compare the calculated Hamming distance with the preset threshold. If the Hamming distance is less than or equal to the preset threshold, it means that the sound effect processing has little effect on the audio, and you can proceed to the next step of adjusting the sound effect parameters; if the Hamming distance is greater than the preset threshold, it means that the sound effect processing has a significant impact on the audio, and further analysis and repair are needed.

[0076] Specifically, the preset threshold is determined by searching a preset mapping table according to the intensity level of global sound effect processing, wherein the preset mapping table stores a mapping relationship between intensity levels of different global sound effect processing and corresponding acceptable upper limits of audio perception hash value differences.

[0077] In this embodiment, in order to ensure that the audio quality after the sound effect processing meets the expected standard, when performing global sound effect processing, it is necessary to dynamically adjust the preset threshold according to the specific intensity level of the global sound effect processing. Specifically, the preset threshold is not fixed, but is determined by searching from a preset mapping table according to the intensity level of the global sound effect processing. The following is a detailed description:

[0078] The intensity level of global sound effect processing refers to the intensity or degree of processing parameters applied during sound effect processing. Different types of global sound effect processing (such as low frequency enhancement, high frequency compression, stereo expansion, surround sound effects, etc.) can have different intensity levels. For example, the intensity of low frequency enhancement can range from slight enhancement to strong enhancement, depending on the user's needs and the configuration of the device.

[0079] Before performing global sound effect processing, the intensity level of the global sound effect processing is determined according to the configuration parameters of the global sound effect processing. This can be achieved by, but not limited to, the following methods:

[0080] User input: The user can select or adjust the intensity level of the sound effects processing in the application interface.

[0081] Automatic detection: The appropriate intensity level can be automatically selected according to the type, content or playback environment of the audio file. For example, for music files, a higher low-frequency enhancement intensity is selected; and for voice files, a lower low-frequency enhancement intensity is selected.

[0082] The preset mapping table stores the mapping relationship between the intensity levels of different global sound effect processing and the corresponding acceptable upper limit of the audio perception hash value difference. The core function of this mapping table is to provide a reasonable threshold for each sound effect processing intensity level to ensure that the audio quality after sound effect processing does not change significantly.

[0083] The multiple intensity levels in the mapping table may be expressed in, but not limited to, numerical values ​​(such as 1-10) or text descriptions (such as “mild”, “medium”, “strong”).

[0084] For each intensity level, a corresponding acceptable audio perception hash value difference upper limit is specified in the mapping table. This upper limit is derived through a large number of experiments and user auditory evaluation data to ensure that within this range, the audio quality after sound effect processing is still considered acceptable.

[0085] When constructing the mapping table, it is necessary to collect a large number of different types of audio samples and perform global sound effects processing at multiple intensity levels on each sample. At the same time, the difference in the audio perception hash value before and after processing is recorded, and the user's auditory evaluation data on the processed audio is obtained; by analyzing the collected data, the maximum acceptable difference at different intensity levels is determined. Exemplarily, for low-frequency enhancement, when the intensity level is 5, the maximum acceptable difference may be 3 Hamming distance units; and when the intensity level is 8, the maximum acceptable difference may be 5 Hamming distance units.

[0086] When searching for a preset threshold, the intensity level of the global sound effect processing is first determined, then the corresponding acceptable difference upper limit is searched in the preset mapping table according to the determined intensity level, and finally the preset threshold is applied.

[0087] S2: When it is determined that the difference is less than or equal to a preset threshold, the audio features after global sound effect processing and the difference are input into a pre-trained classification model, and the sound effect parameters are adjusted in real time based on the output result of the classification model.

[0088] It should be noted that the pre-trained classification models include:

[0089] The input layer is used to receive the audio features processed by the global sound effect and the difference.

[0090] The feature extraction layer is used to extract key features from the received audio features and differences. The specific feature extraction layer includes a convolutional neural network (CNN) structure, which is used to extract local and global features from the audio features and differences to improve the accuracy and robustness of the classification model.

[0091] The hidden layer includes at least one neural network layer, which is used to perform nonlinear transformation and combination on the extracted key features.

[0092] The output layer is used to output sound effect parameter adjustment suggestions based on the learning results of the neural network on the audio samples and their corresponding user auditory evaluation data.

[0093] In step S2, the training process of the classification model includes:

[0094] a: Collect audio samples containing various sound effect processing types and intensity levels.

[0095] In this embodiment, in order to train a classification model that can accurately predict the impact of sound effect parameter adjustment on audio quality, a large number of high-quality audio samples are required as training data. These samples should not only cover different types of audio content (such as music, movies, speech, etc.), but also include a variety of sound effect processing types and intensity levels. In this way, the model can effectively learn and generalize in different audio scenarios, ensuring its robustness and accuracy in practical applications.

[0096] The sources of audio samples collected include but are not limited to the following:

[0097] Public databases: Get a large number of well-annotated audio files from public audio databases, such as MUSAN, ESC-50, FSD50K, etc. These databases provide a rich range of audio categories, including music, environmental sounds, speech, etc.

[0098] User upload: Allow users to upload their own audio files. The system will pre-process and annotate these files to ensure that they meet the training requirements.

[0099] Synthetic audio: Use audio synthesis tools to generate audio samples with specific characteristics, such as different frequency distributions, dynamic ranges, channel configurations, etc. These synthetic audios can be used to test the performance of the model under extreme conditions.

[0100] For the sound effect processing type and intensity level, low-frequency enhancement generates low-frequency enhancement audio samples with different intensity levels, from slight enhancement to strong enhancement; high-frequency compression generates high-frequency compression audio samples with different intensity levels, from slight compression to strong compression; stereo expansion generates stereo expansion audio samples with different intensity levels, from slight expansion to strong expansion; surround sound effect generates surround sound effect audio samples with different intensity levels, from slight simulation to strong simulation.

[0101] Label each audio sample with its original type (such as music, movie, speech, etc.) and intensity level; score the quality of each audio sample through methods including but not limited to manual labeling or automated tools to ensure the diversity of training data.

[0102] b: Perform global sound effect processing on each audio sample and record the difference in audio perception hash values ​​before and after processing.

[0103] In this embodiment, in order to evaluate the effect of the sound effect processing, it is necessary to perform global sound effect processing on each audio sample and record the difference in audio perceptual hash values ​​before and after the processing. The difference in perceptual hash values ​​can quantify the impact of the sound effect processing on the audio and help the model learn the specific impact of different sound effect processing types and intensity levels on audio quality. This process is a key step in training the classification model, ensuring that the model can make reasonable sound effect parameter adjustment recommendations based on changes in perceptual hash values.

[0104] According to the sound effect processing type and intensity level marked in step a, corresponding sound effect processing parameters are configured. For example, for low frequency enhancement, different gain values ​​are set; for high frequency compression, different compression ratios are set.

[0105] Apply global audio processing to each audio sample, using audio processing software to process it. During processing, make sure all audio samples go through the same processing pipeline to maintain consistency.

[0106] A perceptual hash algorithm is used to generate a first comparison value (i.e., the perceptual hash value before processing) for the original audio sample; the same perceptual hash algorithm is used again for the audio sample after global sound effect processing to generate a second comparison value (i.e., the perceptual hash value after processing).

[0107] The Hamming distance between the first comparison value and the second comparison value is calculated to quantify the effect of the sound effect processing on the audio. The Hamming distance is calculated by counting the number of different bits in the two binary strings.

[0108] The calculated Hamming distance is stored along with the intensity level of the audio sample as part of the training data.

[0109] c: Obtain the user's auditory evaluation data on the audio samples before and after processing, including positive and negative evaluations.

[0110] In this embodiment, in order to enable the classification model to better understand the user's auditory preferences, it is necessary to collect the user's auditory evaluation data on the audio samples before and after processing. These evaluation data include not only positive evaluations (such as "clear", "balanced", "comfortable", etc.), but also negative evaluations (such as "distorted", "harsh", "unnatural", etc.). By combining the user's subjective evaluation and the objective perceptual hash value difference, a more intelligent classification model can be trained to ensure that the sound effect parameter adjustment suggestions it outputs can meet the user's expectations.

[0111] Design a simple user interface that allows users to listen to and evaluate the audio samples before and after processing. The interface can include but is not limited to sliders, star ratings, text input boxes, etc. to facilitate users to express their feelings.

[0112] Users evaluate from multiple dimensions, such as clarity, balance, comfort, immersion, etc. The evaluation results of each dimension will be used as part of the training data to help the model understand the user's auditory preferences more comprehensively.

[0113] When the user plays the audio samples before and after the processing, the user's evaluation data is collected in real time. The user can submit the evaluation at any time during the playback process, and the evaluation time and the corresponding audio clip are automatically recorded.

[0114] The user's evaluation data is divided into two categories: positive evaluation and negative evaluation. For example, if the user's evaluation of an audio sample is "high clarity" or "good sound quality", it is marked as a positive evaluation; while if the evaluation is "serious distortion" or "harsh sound", it is marked as a negative evaluation.

[0115] d: Use the collected audio samples, corresponding difference values, and user auditory evaluation data to train the neural network model.

[0116] In this embodiment, in order to enable the classification model to accurately predict the impact of sound effect parameter adjustment on audio quality, it is necessary to use the collected audio samples, the corresponding perceptual hash value differences, and the user's auditory evaluation data to train the neural network model. In this way, the model can learn the specific impact of different sound effect processing types and intensity levels on audio quality, and make personalized sound effect parameter adjustments based on the user's auditory preferences. This process is the core of the entire sound effect processing method and determines the performance and intelligence level of the model.

[0117] Extract key features from each audio sample, such as spectral distribution, dynamic range, channel balance, phase information, etc. These features will be used as input data for the neural network model to help the model better understand the content and structure of the audio.

[0118] The Hamming distance of the perceptual hash value is normalized and converted into a value between 0 and 1. This helps the model better handle differences of different scales and avoids training instability caused by large numerical differences.

[0119] The user's auditory evaluation data (such as positive evaluation, negative evaluation) is converted into a label format to facilitate the model to perform classification tasks. For example, a positive evaluation can be encoded as 1 and a negative evaluation can be encoded as 0.

[0120] The model is trained using a loss function and a stochastic gradient descent algorithm. The choice of optimization algorithm should be adjusted according to the complexity of the model and the size of the training data to ensure that the model can converge quickly and achieve optimal performance. The model's hyperparameters (such as learning rate, batch size, regularization coefficient, etc.) are tuned using a grid search method to ensure that the model performs best on the validation set.

[0121] Specifically, step S2 includes the following steps:

[0122] S21: Combining the audio features processed by the global sound effect and the difference into an input vector.

[0123] In this embodiment, before making real-time adjustments to the sound effect parameters, it is necessary to combine the difference (Hamming distance) between the audio features after global sound effect processing and the perceptual hash value into an input vector. This input vector is the input data of the classification model, which is used to predict the adjustment suggestions for the sound effect parameters. In this way, the model can comprehensively consider the changes in audio content and the influence of user auditory preferences, and make more intelligent adjustment decisions.

[0124] In specific implementation, the audio features after the global sound effect processing extracted in step S12 include spectrum distribution, dynamic range, channel balance, phase information, etc. These features can fully describe the content and structure of the processed audio.

[0125] The Hamming distance between the first comparison value (perceptual hash value before processing) and the second comparison value (perceptual hash value after processing) calculated in step S14. The Hamming distance quantifies the impact of the sound effect processing on the audio and reflects the degree of change of the audio characteristics.

[0126] The processed audio features and the Hamming distance are concatenated into a high-dimensional vector. Specifically, the various dimensions of the audio features (such as spectral distribution, dynamic range, etc.) and the Hamming distance can be used as an additional feature dimension to combine into a unified input vector.

[0127] In order to ensure that the numerical ranges of different features are consistent and to avoid excessive influence of certain features on the model, each feature in the input vector can be normalized. For example, the Hamming distance is normalized to between 0 and 1, and other audio features can also be normalized accordingly according to their characteristics.

[0128] S22: Inputting the input vector into a pre-trained classification model, wherein the classification model is a neural network model trained based on audio samples and their corresponding user auditory evaluation data.

[0129] In this embodiment, in order to achieve real-time adjustment of sound effect parameters, a pre-trained classification model is used to predict the best sound effect parameter adjustment scheme. The classification model is a neural network model trained based on a large number of audio samples and their corresponding user auditory evaluation data. In this way, the model can learn the specific impact of different sound effect processing types and intensity levels on audio quality, and make personalized adjustments based on the user's auditory preferences.

[0130] In a specific implementation, the input vector combined in step S21 is received, including the processed audio features and the Hamming distance. The design of the input layer should ensure that audio features of different lengths and formats can be processed while maintaining data consistency.

[0131] The convolutional neural network (CNN) structure is used to extract local and global features from the input vector. CNN can effectively capture the spatiotemporal relationship in the audio signal and improve the accuracy and robustness of the model.

[0132] The hidden layer performs nonlinear transformation and combination on the extracted key features.

[0133] The output layer outputs the sound effect parameter adjustment suggestions based on the learning results of the neural network on the audio samples and their corresponding user auditory evaluation data. The output layer uses the softmax function for classification tasks.

[0134] The input vector is input into the pre-trained classification model. The model will perform forward propagation through the input layer, feature extraction layer, hidden layer, and output layer in sequence, and finally generate suggestions for adjusting the sound effect parameters.

[0135] The output layer will give suggestions for adjusting the sound effect parameters, including the type of sound effect parameters that need to be adjusted (such as low-frequency enhancement, high-frequency compression, stereo expansion, etc.) and the specific adjustment range. The model's prediction results are based on a large amount of training data and user auditory evaluation data, which can reflect the user's auditory preferences and audio quality requirements.

[0136] S23: Based on the output result of the classification model, determine the type of sound effect parameter that needs to be adjusted and the adjustment range.

[0137] In this embodiment, the output of the classification model is a suggestion for adjusting the sound effect parameters, including the type of sound effect parameters that need to be adjusted (such as low-frequency enhancement, high-frequency compression, stereo expansion, etc.) and the specific adjustment range. In order to achieve real-time adjustment of the sound effect parameters, the system needs to determine the specific adjustment plan based on the output of the model. This process is the core step of sound effect processing and determines the quality of the final output audio and user experience.

[0138] Specifically, the output of the classification model will indicate the type of sound effect parameter that needs to be adjusted. For example, the model recommends increasing the intensity of low-frequency enhancement, reducing the proportion of high-frequency compression, or expanding the stereo effect. Each type of sound effect parameter corresponds to a specific audio processing operation, and the corresponding processing module is selected according to the model's recommendation.

[0139] The model also gives the specific adjustment range for each sound effect parameter. For example, for low-frequency enhancement, the model recommends adjusting the gain value from 5% to 8%; for high-frequency compression, the model recommends adjusting the compression ratio from 2:1 to 3:1. The size of the adjustment range directly affects the final effect of the audio, and precise adjustments are made according to the model's recommendations.

[0140] If the model recommends adjusting multiple sound effect parameters, the different adjustment schemes are prioritized according to the user's auditory preferences and the characteristics of the audio content. For example, for music files, low-frequency enhancement and stereo expansion are prioritized, while for voice files, high-frequency compression and noise suppression are prioritized.

[0141] To ensure that the audio quality does not change significantly, the adjustment range of each sound effect parameter is limited. For example, the gain value of low-frequency enhancement cannot exceed a certain upper limit, and the compression ratio of high-frequency compression cannot be lower than a certain lower limit. These limits can be defined by a preset mapping table or user settings to ensure that the adjusted audio quality still meets the expected standards.

[0142] S24: Adjusting sound effect parameters in real time according to the output result of the classification model.

[0143] In this embodiment, after determining the type of sound effect parameters that need to be adjusted and the adjustment range, the sound effect parameters are adjusted in real time according to the output results of the classification model. This process is the final execution step of the sound effect processing, which directly determines the quality of the final output audio and the user experience. Real-time adjustment requires that the parameter update and audio reprocessing be completed in a short time to ensure that the user can immediately feel the effect of the sound effect change.

[0144] According to the adjustment scheme determined in step S23, the corresponding sound effect processing parameters are updated. For example, if the model suggests increasing the strength of low-frequency enhancement, the gain value of the low-frequency enhancement module is adjusted; if the model suggests reducing the proportion of high-frequency compression, the compression ratio of the high-frequency compression module is adjusted.

[0145] The updated sound effect parameters are passed to the module that controls the sound effect through the API interface or internal communication mechanism to ensure the real-time and accuracy of parameter updates.

[0146] After the module that controls the sound effects receives the updated parameters, it will immediately apply these parameters to reprocess the audio. During the processing, the audio's spectral distribution, dynamic range, channel balance and other characteristics are adjusted according to the new parameters to optimize the audio's auditory effect.

[0147] S3: When it is determined that the difference is greater than a preset threshold, the original audio features of the source audio file are compared with the audio features after being processed by the global sound effects, and the type of audio problem is determined based on the comparison result.

[0148] Specifically, step S3 includes the following steps:

[0149] S31: Compare the original audio features of the source audio file with the audio features processed by the global sound effect item by item.

[0150] In this embodiment, when the difference in the perceptual hash values ​​(Hamming distance) is greater than a preset threshold, it indicates that the sound effect processing has a significant impact on the audio. In order to further analyze the specific reasons for this impact, the original audio features of the source audio file are compared item by item with the audio features after global sound effect processing. This process is a key step in determining the type of audio problem, which can help the system identify which audio features have changed significantly, thereby providing a basis for subsequent repairs.

[0151] In specific implementation, the original audio features of the source audio file extracted from step S11 include spectrum distribution, dynamic range, channel balance, phase information, etc. These features can fully describe the initial state of the audio.

[0152] The audio features after the global sound effect processing extracted in step S12 also include spectrum distribution, dynamic range, channel balance, phase information, etc. These features reflect the changes of the audio after the global sound effect processing.

[0153] Compare the original audio features of the source audio file with the audio features after global sound effect processing, including:

[0154] Spectral distribution: Compare the spectral distribution of the original audio and the processed audio to analyze the changes in frequency components. For example, low-frequency enhancement may increase the energy of the low-frequency band, while high-frequency compression may reduce the energy of the high-frequency band.

[0155] Dynamic range: Compares the dynamic range of the original audio and the processed audio, analyzing the difference between the maximum and minimum amplitudes of the audio. For example, the application of a compressor may reduce the dynamic range, making the loudness of the audio more consistent.

[0156] Channel balance: Compare the channel balance of the original audio and the processed audio to analyze whether the energy distribution of the left and right channels has changed. For example, stereo expansion may increase the separation of the left and right channels, making the audio more spatial.

[0157] Phase information: Compare the phase information of the original audio and the processed audio to analyze whether the time delay and phase relationship of the audio signal have changed. For example, some audio processing may introduce phase distortion, causing the audio to sound unnatural.

[0158] S32: Calculate the change amount of each audio feature using a feature similarity algorithm.

[0159] In this embodiment, on the basis of comparing the audio features item by item, the change amount of each audio feature is further calculated to quantify the impact of the sound effect processing on the audio. The feature similarity algorithm is a commonly used measurement method that can measure the similarity between two audio features. By calculating the feature similarity, the change amount of each audio feature can be quantified, thereby providing more accurate data support for subsequent problem classification.

[0160] In specific implementation, the similarity algorithm can be implemented using Euclidean distance or cosine similarity, where Euclidean distance is used to measure the distance between two audio feature vectors. The smaller the Euclidean distance, the more similar the two features are; cosine similarity is used to measure the angle between two audio feature vectors. The closer the cosine similarity is to 1, the more similar the two features are.

[0161] Then, the difference between the spectrum distribution of the original audio and the processed audio is calculated using Euclidean distance or cosine similarity. Exemplarily, the energy change of each frequency segment can be calculated to obtain the overall change of the spectrum distribution.

[0162] Dynamic range: Use Euclidean distance or cosine similarity to calculate the difference between the dynamic range of the original audio and the processed audio. For example, the maximum and minimum amplitude changes can be calculated to obtain the overall change in dynamic range.

[0163] Channel balance: The difference between the channel balance of the original audio and the processed audio is calculated using Euclidean distance or cosine similarity. For example, the proportional change of the left and right channel energies can be calculated to obtain the overall change in channel balance.

[0164] Phase information: The difference between the phase information of the original audio and the processed audio is calculated using Euclidean distance or cosine similarity. For example, the change in the phase difference can be calculated to obtain the overall change in the phase information.

[0165] In order to ensure that the numerical ranges of different features are consistent and to avoid excessive influence of certain features on the model, the calculated changes can be normalized. For example, all changes are normalized to between 0 and 1, or corresponding normalization is performed according to the characteristics of the features.

[0166] S33: Determine the audio problem type according to the change amount of each audio feature and based on a predefined problem classification rule.

[0167] In this embodiment, after calculating the changes in each audio feature, the specific audio problem type is determined based on these changes and combined with predefined problem classification rules. This process is one of the core steps in sound effect processing and determines the selection and application of subsequent repair strategies. By accurately identifying the type of audio problem, the system can apply repair strategies in a targeted manner to ensure that the audio quality is effectively improved.

[0168] A set of problem classification rules are predefined, covering common audio problem types and their corresponding feature change patterns. For example, excessive low-frequency enhancement may lead to excessive low-frequency energy, excessive high-frequency compression may lead to high-frequency distortion, and excessive stereo expansion may lead to excessive channel separation.

[0169] According to the change of each audio feature, the system matches the predefined problem classification rules one by one to determine the audio problem type that best matches the current audio feature change. For example, if the energy of the low-frequency band of the spectrum distribution increases significantly, the system will judge it as "excessive low-frequency enhancement"; if the energy of the high-frequency band decreases significantly, it will judge it as "excessive high-frequency compression."

[0170] S4: Selecting a sound effect repair strategy according to the determined audio problem type, and using the selected sound effect repair strategy to repair the audio file after the global sound effect processing.

[0171] Specifically, step S4 includes the following steps:

[0172] S41: Select an audio repair strategy from a predefined audio repair strategy library according to the determined audio problem type.

[0173] In this embodiment, after the change amount of each audio feature is calculated by the feature similarity algorithm and the audio problem type is determined based on the predefined problem classification rules, the next step is to select a suitable sound effect repair strategy. The selection of the sound effect repair strategy is a key step to ensure that the audio quality is effectively restored. Different audio problem types require different repair methods, so the system needs to select the most appropriate repair strategy from the predefined sound effect repair strategy library according to the specific problem type.

[0174] In the specific implementation, a sound effect repair strategy library is pre-defined, covering common audio problem types and their corresponding repair methods. For example, for the problem of "excessive low-frequency enhancement", the "low-frequency reduction" strategy can be selected; for the problem of "high-frequency compression distortion", the "high-frequency restoration" strategy can be selected; for the problem of "excessive channel separation", the "channel balance adjustment" strategy can be selected.

[0175] According to the audio problem type determined in step S33, the predefined repair strategy library is matched one by one to find the repair strategy that best suits the current problem type. For example, if it is determined to be "excessive low-frequency enhancement", the "low-frequency reduction" strategy will be selected; if it is determined to be "high-frequency compression distortion", the "high-frequency restoration" strategy will be selected.

[0176] If multiple problem types exist at the same time, different repair strategies are prioritized according to the user's auditory preferences and the characteristics of the audio content. For example, for music files, low-frequency problems are repaired first, while for voice files, high-frequency problems are repaired first.

[0177] S42: Apply the selected sound effect repair strategy to repair the audio file after the global sound effect processing.

[0178] In this embodiment, after selecting a suitable sound effect repair strategy, the strategy is applied to the audio file after the global sound effect processing to perform specific repair processing. This process is the execution step of the sound effect processing and directly determines the quality of the final output audio. The goal of the repair processing is to eliminate or alleviate the problems in the audio, restore the original auditory characteristics of the audio, and ensure that the user can get the best audio experience.

[0179] The specific implementation is divided into the following situations:

[0180] Low frequency reduction: If the "Low frequency reduction" strategy is selected, a low-pass filter or equalizer will be applied to reduce the energy of the low frequency band. For example, the gain value of the low frequency band can be appropriately reduced according to the severity of the problem to avoid the turbidity caused by excessive low frequency.

[0181] High Frequency Restoration: If the "High Frequency Restoration" strategy is selected, a high-pass filter or reverse compressor will be applied to restore the clarity of the high frequency band. For example, depending on the severity of the problem, the energy of the high frequency band can be appropriately increased to avoid the harshness caused by high frequency distortion.

[0182] Channel balance adjustment: If the "Channel balance adjustment" strategy is selected, the stereo expander or channel mixer will be applied to adjust the energy distribution of the left and right channels. For example, the separation of the left and right channels can be appropriately reduced according to the severity of the problem to avoid the imbalance of the spatial sense caused by excessive channel separation.

[0183] Phase Correction: If the "Phase Correction" strategy is selected, a phase compensator or time delayer will be applied to correct the time delay and phase relationship of the audio signal. For example, the phase difference can be adjusted appropriately according to the severity of the problem to avoid the unnatural feeling caused by phase distortion.

[0184] At the same time, in the process of applying the repair strategy, it is necessary to dynamically adjust the repair parameters according to the specific situation of the audio. For example, for the low-frequency reduction strategy, the cutoff frequency or gain value of the low-pass filter is dynamically adjusted according to the energy size of the low-frequency band; for the high-frequency restoration strategy, the cutoff frequency or gain value of the high-pass filter is dynamically adjusted according to the degree of energy loss in the high-frequency band.

[0185] To ensure that the quality of the restored audio does not change significantly, each restoration parameter is limited. For example, the gain value of low-frequency reduction cannot be lower than a certain lower limit, and the gain value of high-frequency restoration cannot exceed a certain upper limit. These limits can be defined by a preset mapping table or user settings to ensure that the quality of the restored audio still meets the expected standards.

[0186] S5: Output and play the audio file that has been adjusted in real time or the audio file that has been repaired.

[0187] In this embodiment, in the sound effect processing method, step S5 is the last step of the entire process, which is responsible for outputting and playing the audio file after real-time adjustment or repair processing to the user. This process not only directly affects the user's final auditory experience, but also involves multiple aspects such as format conversion of audio files, selection of output devices, and playback control. The system needs to ensure that the processed audio files can be smoothly transmitted to the output device and played out with the best quality to meet the user's expectations.

[0188] Real-time adjusted audio file: If the difference in the perceptual hash value is less than or equal to the preset threshold in step S2, the sound effect parameters are adjusted in real time according to the output of the classification model. The adjusted audio file will be passed to step S5 for output and playback.

[0189] Repairing the audio file: If the difference in the perceptual hash value is greater than the preset threshold in step S3, the corresponding sound repair strategy is selected and applied according to the determined audio problem type. The repaired audio file will be passed to step S5 for output and playback.

[0190] In addition, in order to ensure that the audio files can be played normally on various output devices, the processed audio files will be converted into different formats and the sampling rate and bit depth of the audio files will be adjusted according to the characteristics of the output devices.

[0191] Embodiment 2

[0192] The present invention also provides a sound effect processing system for executing a sound effect processing method, referring to Figure 2 As shown, the processing system includes:

[0193] The audio quality assessment module 100 is used to respond to an audio playback instruction, calculate the difference between a first comparison value and a second comparison value of a source audio file after global sound effect processing, and judge the difference based on a preset threshold, wherein the first comparison value and the second comparison value are respectively the audio perception hash values ​​of the source audio file before and after global sound effect processing.

[0194] The sound effect parameter real-time adjustment module 200 is used to input the audio features after global sound effect processing and the difference into a pre-trained classification model when it is determined that the difference is less than or equal to a preset threshold, and to make real-time adjustments to the sound effect parameters based on the output results of the classification model.

[0195] The audio problem type determination module 300 is used to compare the original audio features of the source audio file with the audio features after global sound effect processing when it is determined that the difference is greater than a preset threshold, and determine the audio problem type based on the comparison result.

[0196] The sound effect repair module 400 is used to select a sound effect repair strategy according to the determined audio problem type, and use the selected sound effect repair strategy to repair the audio file after the global sound effect processing.

[0197] The audio playing module 500 is used to output and play the audio file that has been adjusted in real time or the audio file that has been repaired.

[0198] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0200] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A sound effect processing method, characterized in that: The processing method comprises the following steps: In response to an audio playback instruction, a difference between a first comparison value and a second comparison value of a source audio file is calculated after global sound effect processing, and the difference is judged based on a preset threshold, wherein the first comparison value and the second comparison value are respectively audio perception hash values ​​of the source audio file before and after global sound effect processing; When it is determined that the difference is less than or equal to a preset threshold, the audio features processed by the global sound effect and the difference are input into a pre-trained classification model, and the sound effect parameters are adjusted in real time based on the output result of the classification model; When it is determined that the difference is greater than a preset threshold, the original audio features of the source audio file are compared with the audio features after being processed by the global sound effect, and the type of audio problem is determined based on the comparison result; Selecting a sound effect repair strategy according to the determined audio problem type, and using the selected sound effect repair strategy to repair the audio file after the global sound effect processing; Output and play the audio files that have been adjusted in real time or the audio files that have been repaired.

2. A sound effect processing method according to claim 1, characterized in that: The preset threshold is determined by searching a preset mapping table according to the intensity level of the global sound effect processing, wherein the preset mapping table stores a mapping relationship between the intensity levels of different global sound effect processing and the corresponding acceptable upper limits of the audio perception hash value difference.

3. A sound effect processing method according to claim 2, characterized in that: The step of responding to the audio playback instruction, calculating the difference between the first comparison value and the second comparison value of the source audio file after the global sound effect processing, and judging the difference based on a preset threshold value, includes: Extracting original audio features of the source audio file, and generating the first comparison value using a perceptual hash algorithm; Perform global sound effect processing on the source audio file and extract the audio features after the global sound effect processing; Using the same perceptual hash algorithm, generating the second comparison value based on the audio features processed by the global sound effect; The difference between the first comparison value and the second comparison value is calculated, and compared with the preset threshold value determined by searching, wherein the difference is the Hamming distance.

4. A sound effect processing method according to claim 3, characterized in that: Before searching the preset mapping table to determine the preset threshold, the following steps are also included: Determines the intensity level of global sound effect processing based on the configuration parameters of global sound effect processing.

5. A sound effect processing method according to claim 4, characterized in that: The training process of the classification model includes: Collect audio samples containing various sound processing types and intensity levels; Perform global sound effect processing on each audio sample and record the difference in audio perception hash values ​​before and after processing; Obtaining the user's auditory evaluation data on the audio samples before and after processing, including positive and negative evaluations; The neural network model is trained using the collected audio samples, corresponding difference values, and user auditory evaluation data.

6. A sound effect processing method according to claim 5, characterized in that: When it is determined that the difference is less than or equal to a preset threshold, the audio features processed by the global sound effect and the difference are input into a pre-trained classification model, and the sound effect parameters are adjusted in real time based on the output result of the classification model, including: Combining the audio features processed by the global sound effect and the difference into an input vector; Inputting the input vector into a pre-trained classification model, wherein the classification model is a neural network model trained based on audio samples and corresponding user auditory evaluation data; Based on the output result of the classification model, determine the type of sound effect parameter that needs to be adjusted and the adjustment range; According to the output result of the classification model, the sound effect parameters are adjusted in real time.

7. A sound effect processing method according to claim 6, characterized in that: When it is determined that the difference is greater than a preset threshold, comparing the original audio features of the source audio file with the audio features processed by the global sound effect, and determining the type of the audio problem based on the comparison result, including: Compare the original audio features of the source audio file with the audio features processed by the global sound effect item by item; The feature similarity algorithm is used to calculate the change amount of each audio feature; According to the change amount of each audio feature, the audio problem type is determined based on a predefined problem classification rule.

8. The sound effect processing method according to claim 7, characterized in that: The step of selecting a sound effect repair strategy according to the determined audio problem type, and using the selected sound effect repair strategy to repair the audio file after the global sound effect processing, comprises: Selecting an audio repair strategy from a library of predefined audio repair strategies based on the determined audio problem type; Apply the selected sound effect repair strategy to repair the audio file after global sound effect processing.

9. A sound effect processing system, used to execute a sound effect processing method according to any one of claims 1 to 8, characterized in that: The processing system comprises: an audio quality assessment module, configured to respond to an audio playback instruction, calculate a difference between a first comparison value and a second comparison value of a source audio file after global sound effect processing, and judge the difference based on a preset threshold, wherein the first comparison value and the second comparison value are respectively audio perception hash values ​​of the source audio file before and after global sound effect processing; A sound effect parameter real-time adjustment module, for inputting the audio features processed by the global sound effect and the difference into a pre-trained classification model when it is determined that the difference is less than or equal to a preset threshold, and adjusting the sound effect parameters in real time based on the output result of the classification model; An audio problem type determination module, for comparing the original audio features of the source audio file with the audio features processed by the global sound effect when it is determined that the difference is greater than a preset threshold, and determining the audio problem type based on the comparison result; A sound effect repair module, used to select a sound effect repair strategy according to the determined audio problem type, and use the selected sound effect repair strategy to repair the audio file after the global sound effect processing; The audio playback module is used to output and play the audio files that have been adjusted in real time or the audio files that have been repaired.

Images