A method and system for enhancing audio effects in streaming media bitrates
By constructing an inference model and quantization parameter package, the streaming audio bitstream is transcoded and its features are extracted, and the sound effects are dynamically adjusted. This solves the problem of fixed audio sound effects in streaming media, realizes personalized sound effect enhancement and format adaptability, and improves the audio playback experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHEAST UNIV
- Filing Date
- 2023-02-22
- Publication Date
- 2026-05-26
AI Technical Summary
In existing streaming media scenarios, audio sound effects processing is relatively fixed and cannot be dynamically adjusted according to audio characteristics, resulting in a monotonous music style and difficulty in achieving personalized sound effect enhancement.
By receiving audio bitstream information from streaming media services, pre-defined sound effect information is obtained, an inference model is constructed and a quantization parameter package is loaded, the audio data is transcoded and preprocessed, features are extracted, the equalization and reverberation feature sequences are determined using the inference model, sound effects are enhanced, and the sound effects are dynamically adjusted through a digital audio processor.
It enables dynamic adjustment of streaming audio effects, enriches music styles, enhances the personalized audio playback experience, adapts to unified processing of different audio formats, and reduces the complexity of audio enhancement.
Smart Images
Figure CN116170424B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer technology, and in particular relates to a method and system for enhancing audio effects in streaming media bitstreams. Background Technology
[0002] With the development of science and technology, wireless speakers have entered home life. Wireless speakers are popular among users due to their portability and ease of use. Wireless speakers can acquire audio data from smart devices via wireless networks using streaming media protocols. To meet the needs of different users, some speakers also support audio effects functions to provide a better listening experience.
[0003] However, the sound effects processing used in some streaming music is generally relatively fixed. It's impossible to select appropriate audio effects for equalization and reverb based on audio characteristics. Furthermore, the audio formats transmitted in streaming media are not relatively fixed, which makes personalized audio adjustments difficult. This paper proposes a method for dynamically enhancing sound effects in streaming media. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for enhancing audio effects in streaming media bitstreams, in order to solve the technical problem that the use of audio effects in existing streaming media scenarios is relatively fixed and monotonous, unable to dynamically adjust audio effects according to the audio playback progress and current characteristics, and the music style and content are relatively monotonous.
[0005] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:
[0006] A method for enhancing audio effects in streaming media includes the following steps:
[0007] Step 110: Receive audio stream information transmitted by the streaming media service; the audio stream information includes encoding type information and audio information;
[0008] Step 120: Obtain preset sound effect information; the preset sound effect information includes equalization information and reverberation information;
[0009] Step 130: Construct an inference model and load the quantization parameter package corresponding to the preset sound effect information onto the inference model based on the preset sound effect information;
[0010] Step 140: Transcode the audio stream information to convert different audio formats into the same audio data format; based on the transcoded audio data, preprocess the audio data and pre-extract audio features.
[0011] Step 150: After loading the quantization parameter package, the inference model determines the equalization feature sequence based on the pre-extracted audio features.
[0012] Based on the pre-extracted audio features, the reverberation feature sequence is determined through the inference model; the equalization feature sequence and the reverberation feature sequence correspond to the quantization parameter package.
[0013] Step 160: Based on the equalization feature sequence and reverberation feature sequence, the digital audio processor enhances the audio bitstream information with sound effects;
[0014] Step 170: Play the enhanced audio stream information through a playback device.
[0015] Furthermore, in step 130, the inference model includes an equalization feature sequence and a reverberation feature sequence; the equalization feature sequence determines the optimal equalization method based on the audio information of the bitstream, while ensuring a smooth transition of equalization method changes; the reverberation feature layer determines the reverberation feature sequence based on the mixed reverberation information of the audio bitstream.
[0016] Furthermore, the inference model in step 130 is obtained through the following method:
[0017] Obtain the dataset corresponding to the preset sound effect information. The dataset source includes MP3, FLAC, DFF, and FLAC music format files. The dataset includes music files and sound effect information, and is divided into three parts: training set, test set, and validation set. Data augmentation is performed on the training set by pruning and pre-adding noise interference to enhance the data for information with fewer sound effect types, and enhancing the amount of information for each sound effect type to be equal. Information with fewer sound effect types refers to information where one sound effect type is much less than another sound effect type, such as more than twice the amount of information, and therefore needs to be augmented.
[0018] The parameters of the inference model are randomly initialized, the dataset is input into the initial inference model, the initial inference model is trained, and the trained inference model is obtained.
[0019] Quantize the inference model to reduce its computational load and size, thereby increasing inference speed while maintaining inference accuracy.
[0020] Furthermore, in step 160, the digital audio processor interacts with the control bus and the audio transmission bus, with the control bus transmitting instruction information and the audio transmission bus transmitting data information.
[0021] Furthermore, based on the encoding format information in step 110, the audio transcoding module will call the corresponding audio format decoder to decode the audio.
[0022] Furthermore, based on step 160, the digital audio processor performs corresponding gain on different frequency bands of the audio based on the equalization feature sequence, and smooths the bitstream equalization feature results of different frames.
[0023] Furthermore, in step 140, the audio transcoding module transcodes audio format streams that are not suitable for direct inference in the streaming media stream; the audio transcoding module obtains the audio format information of the streaming media transmission and loads different decoders, wherein the transcoder includes a software decoder and a hardware decoder; MP3 music, MQA and DSD encoded music are uniformly transcoded into PCM streams.
[0024] Furthermore, in step 170, the playback device includes a hardware playback sound card device and other wireless playback devices within the same LAN network.
[0025] Furthermore, in step 160, digital audio processing is performed on the audio bitstream to convert the audio into the frequency domain, and the audio frequency band is divided into 5 segments based on the equalization feature sequence.
[0026] Furthermore, the transcoded bitstream format in step 140 is a 16-bit PCM bitstream.
[0027] This invention also discloses a system for enhancing audio effects in streaming media bitstreams, the system comprising the following modules:
[0028] The audio stream data acquisition module is used to acquire audio stream data information; the audio transcoding module is used to transcode the audio stream data information from the audio stream acquisition module into a unified format; the audio stream parsing module is used to analyze the audio stream data information to obtain equalization effect information and reverb effect information; the sound effect processing module is used to add special effects to the audio based on the equalization effect information and reverb effect information; and the audio playback module plays the audio with the added sound effects.
[0029] The present invention provides a method and system for enhancing audio effects in streaming media, which has the following advantages:
[0030] 1. Step 140 of the audio enhancement method for streaming media bitstream described in this invention can reduce the difficulty of audio enhancement caused by the large number of audio formats in streaming media scenarios, and convert audio data of different formats into a unified format.
[0031] 2. In the system described in this invention, step 110 can acquire different audio bitstreams from numerous streaming media, and steps 150 and 160 can avoid the problem of monotonous sound effects. They can extract features from audio information to adopt the most suitable sound effect method, making the music playback sound effect more pleasant and richer. Attached Figure Description
[0032] Figure 1 This is a flowchart of a method for enhancing audio effects in streaming media bitstreams according to an embodiment of the present invention;
[0033] Figure 2This is a flowchart of the program running for the audio enhancement system for streaming media bitstreams in an embodiment of the present invention;
[0034] Figure 3 This is a flowchart of the inference model training process in an embodiment of the present invention;
[0035] Figure 4 This is a structural diagram of a device for enhancing audio effects in streaming media bitstreams, as described in an embodiment of the present invention. Detailed Implementation
[0036] To better understand the purpose, structure, and function of this invention, the following detailed description of a method and system for enhancing audio effects in streaming media bitstreams is provided in conjunction with the accompanying drawings.
[0037] Figure 1 This is a flowchart illustrating a method for enhancing audio effects in a streaming media bitstream, according to an embodiment of this application.
[0038] In this embodiment, process 100 includes:
[0039] Step 110: Obtain audio service bitstream information.
[0040] The bitstream information refers to the bitstream audio data that can be written to the sound card in the streaming media service transmission system. This data can be obtained through audio file decoding or during the streaming media transmission process. In this embodiment, this part also includes bitstream information obtained after transcoding other audio with different encodings.
[0041] Step 120: Obtain preset sound effect information, which includes music style-related information associated with the song.
[0042] Sound effect information refers to the sound effect settings determined before music is played. Preset sound effects are set as the basis for dynamically changing sound effects in real time. In this embodiment, they include equalization sound effects and reverb sound effects. There can be one sound effect or a combination of multiple sound effects.
[0043] Music style refers to the characteristics of audio during music playback, and can be pop, rock, jazz, etc. Style can be obtained through user preferences or through the original attribute information of the music.
[0044] Step 130: Load the corresponding quantization parameter package through the inference model based on the audio information.
[0045] In this embodiment, the inference model refers to the sound effect information obtained by inferring from the audio bitstream information. These inference models often contain a large number of parameters. In order to reduce the model size and inference speed without affecting the inference accuracy, the parameter package can be quantized to reduce the size of the parameter package and increase the real-time performance of the dizziness response.
[0046] Step 140: Perform the same transcoding on the audio bitstream, and preprocess the transcoded audio to adapt it to the inference model.
[0047] In streaming media scenarios, there may be multiple audio formats. The system may receive different music formats played by different devices, such as MP3, FLAC, WAV, etc. Different encoding methods have significant differences. Before using the inference model for inference, the audio bitstream needs to be transcoded first.
[0048] The transcoded bitstream needs to be preprocessed. This part is mainly to extract the feature information of a segment of audio in advance to help the inference model to reason better and improve the accuracy of the inference model. In the embodiment, audio preprocessing can use MFCC (Mel Frequency Cepstral Coefficients), but is not limited to this.
[0049] Step 150: Determine the equilibrium feature sequence and reverberation feature sequence based on the inference model.
[0050] In this embodiment, the inference model can be a neural network model, which can be an existing model capable of performing multiple feature processing tasks, such as CNN (Convolutional Neural Network) or RNN (Recurrent Convolutional Neural Network). The inference model can also be a custom model as needed. The neural network can contain multiple hidden layers, such as fully connected layers, pooling layers, and convolutional layers. The parameters of each layer of the neural network are obtained through a learning algorithm.
[0051] Step 160: Based on the feature sequence, the audio processing device enhances the audio effect of the bitstream.
[0052] In this embodiment, the equalization gain and reverberation effect of the bitstream are determined based on the equalization feature sequence and the reverberation feature sequence. The enhanced audio bitstream data can then be sent to a playback device for playback.
[0053] Step 170: Play the enhanced audio data through a playback device.
[0054] In this embodiment, the playback device can be a hardware speaker playback device or a wireless playback device on the same LAN network.
[0055] This invention also discloses a system for enhancing audio effects in streaming media bitstreams, the system comprising the following modules:
[0056] The audio stream data acquisition module is used to acquire audio stream data information; the audio transcoding module is used to transcode the audio stream data information from the audio stream acquisition module into a unified format; the audio stream parsing module is used to analyze the audio stream data information to obtain equalization effect information and reverb effect information; the sound effect processing module is used to add special effects to the audio based on the equalization effect information and reverb effect information; and the audio playback module plays the audio with the added sound effects.
[0057] Figure 2 This is a program flowchart for the runtime of the audio enhancement system for streaming media bitstreams, according to an embodiment of this application.
[0058] Step 201: After the system starts, it will first perform initialization, which mainly includes initializing the streaming media audio service, initializing the relevant audio encoding format decoder, initializing the audio preprocessing module, and initializing the inference model to load the corresponding quantization parameter package.
[0059] Step 202: Receive audio data via a streaming media service. The audio data includes not only the raw audio information but also audio encoding information. Store the received audio data in the raw buffer queue.
[0060] Step 203: Based on the encoding format information obtained in step 202, determine whether the current encoded audio is supported. If not, proceed to step 209; if supported, proceed to step 204.
[0061] Step 204: Based on the audio encoded data in the original cache queue, decode it using an audio decoder, perform audio preprocessing on the decoded data, and cache the decoded data in the inference queue.
[0062] Step 205: Determine whether the size of the data in the queue to be inferred meets the data size required for inference. If it does, proceed to step 206; otherwise, proceed to step 210.
[0063] Step 206: Based on the inference model, inference is performed to obtain the equilibrium feature sequence and the reverberation feature sequence.
[0064] Step 207: Based on the equalization feature sequence and reverberation feature sequence, the digital audio processing device performs sound effect processing on the original audio, and the processed audio data is cached to the playback queue.
[0065] Step 208: Play the video through a speaker or an airplay device within the same LAN.
[0066] Step 209: Provisional streaming service.
[0067] Step 210: The reasoning task is suspended, awaiting activation.
[0068] Figure 3This is based on the inference model training flowchart in the embodiments of this application, used to obtain the quantization parameter package in step 130.
[0069] Step 301: Obtain audio data and corresponding sound effect feature information tags.
[0070] Audio data can be obtained from audio files of different formats, but all different audio data need to be converted into the same format for unified training. Users can predefine the music style of each type of music or obtain it automatically through the system. For example, the music style can be judged by singer information and the sound effects can be tagged.
[0071] Step 302: Input the training set into the inference model for learning and training to obtain model parameter information. The model continuously learns from the dataset until the loss essentially stops decreasing.
[0072] Step 303: Quantization is used to reduce the model size. Most of the model runs on edge devices, and an excessively large model may burden the system's memory and inference time.
[0073] Figure 4 This is a structural diagram of a device for enhancing audio effects in streaming media bitrates, based on some embodiments of this application. The system mainly comprises a main controller, a hardware decoder, a digital audio processor, and a playback device. In this embodiment, the main controller includes an operating system to better support network protocols. The main controller is primarily responsible for receiving audio data from the smart device and coordinating the scheduling of tasks. The hardware decoder is responsible for decoding the audio data received by the main controller; the digital audio processor dynamically processes the audio data and interacts with the main controller through two buses: a control bus and an audio transmission bus. The audio processor converts the processed audio data into analog signals and sends them to the playback device for playback.
[0074] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.
Claims
1. A method for enhancing audio effects in streaming media bitstreams, characterized in that, The method includes the following steps: Step 110: Receive audio stream information transmitted by the streaming media service; the audio stream information includes encoding type information and audio information; Step 120: Obtain preset sound effect information; the preset sound effect information includes equalization information and reverberation information; Step 130: Construct an inference model and load the quantization parameter package corresponding to the preset sound effect information onto the inference model based on the preset sound effect information; Step 140: The audio stream information is transcoded using the audio transcoding module to convert different audio formats into audio data of the same format; based on the transcoded audio data, the audio data is preprocessed and audio features are pre-extracted. Step 150: After loading the quantization parameter package, the inference model determines the equalization feature sequence based on the pre-extracted audio features. Based on the pre-extracted audio features, the reverberation feature sequence is determined through the inference model; the equalization feature sequence and the reverberation feature sequence correspond to the quantization parameter package. Step 160: Based on the equalization feature sequence and reverberation feature sequence, the digital audio processor receives the audio bitstream information, the equalization feature sequence, and the reverberation feature sequence, and the digital audio processor enhances the audio bitstream information with sound effects. Step 170: Play the enhanced audio stream information through a playback device; The inference model in step 130 is obtained through the following method: Obtain the dataset corresponding to the preset sound effect information; the dataset includes music files and sound effect information, and the dataset is divided into three parts: training set, test set, and validation set; perform data augmentation on the training set, and enhance the data by pruning and pre-adding noise interference to information with few sound effect types, and enhance the amount of information of each sound effect type to be equal; The parameters of the inference model are randomly initialized, the dataset is input into the initial inference model, the initial inference model is trained, and the trained inference model is obtained. Quantize the inference model to reduce its computational load and size, thereby increasing inference speed while maintaining inference accuracy.
2. The audio enhancement method for streaming media bitstreams according to claim 1, characterized in that, In step 160, the digital audio processor interacts with the control bus and the audio transmission bus, with the control bus transmitting instruction information and the audio transmission bus transmitting data information.
3. The audio enhancement method for streaming media bitstreams according to claim 1, characterized in that, Based on the encoding type information in step 110, the audio transcoding module will call the corresponding audio format decoder to decode the audio.
4. The audio enhancement method for streaming media bitstreams according to claim 1, characterized in that, In step 160, the digital audio processor performs corresponding gain on different frequency bands of the audio based on the equalization feature sequence, and smooths the bitstream equalization feature results of different frames.
5. The audio enhancement method for streaming media bitstreams according to claim 1, characterized in that, The audio transcoding module in step 140 transcodes audio format streams in the streaming media stream that are not suitable for direct inference. The audio transcoding module acquires the audio format information of the streaming media transmission and loads different decoders, including software decoders and hardware decoders; it uniformly transcodes MP3 music, MQA and DSD encoded music into PCM streams.
6. The audio enhancement method for streaming media bitstreams according to claim 1, characterized in that, The playback devices in step 170 include hardware playback sound cards and other wireless playback devices within the same LAN network.
7. A system for enhancing audio effects in a streaming media bitstream according to any one of claims 1-6, characterized in that, The system includes the following modules: The audio stream data acquisition module is used to acquire audio stream data information; The audio transcoding module is used to transcode the audio stream data information from the audio stream acquisition module into a unified format; the audio stream parsing module is used to analyze the audio stream data information to obtain equalization and reverb information. The audio effects processing module is used to overlay audio effects based on equalization and reverberation information. The audio playback module plays audio with overlaid sound effects.