Audio data processing method and device, electronic equipment and storage medium
By processing the audio data with waveform and spectrogram, and building a sequential model for training, the problem of unsatisfactory vocal separation effect is solved, and efficient vocal separation and communication quality improvement is achieved.
Patent Information
- Application Number
- CN202510638563.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2025-08-12
AI Technical Summary
In the prior art, the vocal separation effect is not ideal and cannot meet the needs of high-quality audio and video communication. It is difficult to effectively separate and process the vocals of each person when multiple people speak at the same time.
By obtaining the audio data to be trained, converting it into waveform data and extracting the spectrum diagram, building a sequential model for training, obtaining an audio separation model, which is used to separate the human voice audio and background audio in real-time audio data.
It realizes efficient vocal separation, improves communication quality, and can effectively separate and process the vocals of each person while multiple people speak at the same time.
Smart Images
Figure CN120472923A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an audio data processing method, device, electronic device and storage medium. Background Art
[0002] During business operations, real-time audio and video communication scenarios such as conference calls are often conducted. Interference factors such as background noise and other speakers' voices need to be eliminated from the audio signal to improve communication quality. Especially when multiple people are speaking simultaneously, it is necessary to separate each person's voice and process and transmit them separately.
[0003] In the prior art, multi-person audio data is processed based on processing methods such as filters and noise reduction algorithms to achieve the purpose of voice separation.
[0004] However, the voice separation effect of the above methods is not ideal and cannot meet the growing demand for high-quality audio and video communications. Summary of the Invention
[0005] The embodiments of the present application provide an audio data processing method, device, electronic device, and storage medium, which can achieve efficient voice separation and improve communication quality.
[0006] In a first aspect, an embodiment of the present application provides an audio data processing method, comprising:
[0007] Acquire audio data to be trained, wherein the audio data to be trained includes audio signals of different types of music; and obtain a vocal track to be trained;
[0008] Performing conversion processing on the audio data to be trained to obtain waveform data corresponding to the audio data to be trained; and extracting a spectrogram corresponding to the waveform data; wherein the waveform data represents the amplitude variation of the audio signal over time, and the spectrogram represents the relative strength of the frequency components of the audio signal;
[0009] Constructing a sequential model, and training the sequential model based on the waveform data and spectrogram corresponding to the audio data to be trained, and the vocal track to be trained, to obtain an audio separation model;
[0010] Acquire real-time audio data; and based on the audio separation model, separate the human voice audio and background audio in the real-time audio data to obtain an audio separation result corresponding to the real-time audio data.
[0011] In a second aspect, an embodiment of the present application provides an audio data processing device, comprising:
[0012] An acquisition module is used to acquire audio data to be trained, wherein the audio data to be trained includes audio signals of different types of music; and acquire a vocal track to be trained;
[0013] a processing module, configured to convert the audio data to be trained to obtain waveform data corresponding to the audio data to be trained; and extract a spectrogram corresponding to the waveform data; wherein the waveform data represents the amplitude variation of the audio signal over time, and the spectrogram represents the relative strength of the frequency components of the audio signal;
[0014] A training module is used to construct a sequential model and train the sequential model based on the waveform data and spectrogram corresponding to the audio data to be trained and the vocal track to be trained to obtain an audio separation model;
[0015] The separation module is used to obtain real-time audio data; and based on the audio separation model, separate the human voice audio and background audio in the real-time audio data to obtain the audio separation result corresponding to the real-time audio data.
[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor;
[0017] The memory stores computer-executable instructions;
[0018] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.
[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.
[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.
[0021] The audio data processing method, device, electronic device, and storage medium provided in the embodiments of the present application process the audio data to be trained to obtain corresponding waveform data and spectrograms, and train a sequential model in combination with the human voice track to be trained to obtain an audio separation model, so as to separate the human voice audio and background audio in real-time audio data and obtain corresponding audio separation results, thereby achieving efficient human voice separation and improving communication quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0023] Figure 1 A flowchart of an audio data processing method provided in an embodiment of the present application;
[0024] Figure 2 A flow chart of voice separation in audio and video communication provided by an embodiment of the present application;
[0025] Figure 3 A flowchart of another audio data processing method provided in an embodiment of the present application;
[0026] Figure 4 A model training flow chart provided in an embodiment of the present application;
[0027] Figure 5 An audio acquisition flow chart provided in an embodiment of the present application;
[0028] Figure 6 A model reasoning flow chart provided in an embodiment of the present application;
[0029] Figure 7 An audio analysis waveform effect diagram provided in an embodiment of the present application;
[0030] Figure 8 A schematic structural diagram of an audio data processing device provided in an embodiment of the present application;
[0031] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0032] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION
[0033] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
[0034] First, let’s explain the terms involved in this application:
[0035] Convolutional Neural Networks (CNN): A type of feedforward neural network with a deep structure that includes convolution calculations. It is one of the representative algorithms of deep learning.
[0036] Recurrent Neural Network (RNN): A type of neural network used to process sequence data. Unlike traditional neural networks, RNNs introduce the concept of "memory," enabling them to process sequence data of arbitrary length and retain and utilize historical information in the sequence during processing.
[0037] PCM: PCM (Pulse-Code Modulation) is a coding method used in digital communications. It's not a specific network transmission technology, but rather an audio compression technique. Its full name is Pulse Code Modulation. It's widely used in communications, computers, and other fields.
[0038] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, processing, transmission, provision, disclosure and application of relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0039] In addition, this application involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and the use of artificial intelligence technology for automated decision-making, and a technical solution for making decisions that have a significant impact on personal rights and interests based on the results of automated decision-making. The application provides users with corresponding operation entrances for users to choose to agree or reject the results of automated decision-making; if the user chooses to reject, the expert decision-making process will be entered.
[0040] It should be noted that this application can be used in the field of artificial intelligence technology, and can also be used in any field other than artificial intelligence. The application field of this application is not limited.
[0041] During business operations, real-time audio and video communication scenarios such as conference calls are often conducted. Interference factors such as background noise and other speakers' voices need to be eliminated from the audio signal to improve communication quality. Especially when multiple people are speaking simultaneously, it is necessary to separate each person's voice and process and transmit them separately.
[0042] Based on the above scenarios, it can be seen that the processing methods based on filters, noise reduction algorithms, etc. have the technical problem that the human voice separation effect is not ideal and cannot meet the growing demand for high-quality audio and video communications.
[0043] The audio data processing method provided in this application processes the audio data to be trained to obtain corresponding waveform data and spectrograms, and trains the sequential model in combination with the human voice track to be trained to obtain an audio separation model, so as to separate the human voice audio and background audio in the real-time audio data and obtain the corresponding audio separation results, thereby solving the technical problem that the human voice separation effect is not ideal.
[0044] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0045] Figure 1 A flowchart of an audio data processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes:
[0046] 201. Acquire audio data to be trained, wherein the audio data to be trained includes audio signals of different types of music; and obtain a vocal track to be trained.
[0047] Exemplarily, the execution subject of this embodiment may be an electronic device, hereinafter referred to as the device. The device may obtain a preset training set and load the training set, which contains a variety of full-length music of different types and independent drums, bass, vocals, and other stems in the music that can be used as data for vocal separation. The data of the audio set is selected from the training set and stored. The data of the audio set is the audio data to be trained, and the audio data to be trained includes audio signals of different types of music. The separate vocal data in the audio set is selected and stored to obtain the vocal track to be trained.
[0048] For example, the audio and vocal data sets for loading and preprocessing use system functions to load audio files and convert them into mono audio signals. The load_train_data function is used to load and preprocess the audio data. The audio data set's package file path is used to load the audio files using the read file library to obtain the audio data set. The audio data set is loaded using the function to obtain the audio files and the corresponding vocal tracks, i.e., the vocal data to be trained. Then, the load and preprocess functions are called to preprocess the audio files to obtain the audio data to be trained. The audio data to be trained and the vocal tracks are then added to the dataset in the code for data training.
[0049] 202. Convert the audio data to be trained to obtain waveform data corresponding to the audio data to be trained; and extract a spectrogram corresponding to the waveform data; wherein the waveform data represents the amplitude variation of the audio signal over time, and the spectrogram represents the relative strength of the frequency components of the audio signal.
[0050] Exemplarily, the audio data to be trained is loaded, the audio PCM data from the audio data to be trained is extracted, and the audio PCM data is converted into waveform data to characterize the temporal amplitude variation of the audio signal in the audio data to be trained. The purpose of this step is to read the audio signal from the file and represent it in digital form for subsequent processing. According to the waveform data, a spectrogram is extracted. For example, the audio file is loaded using a file reading library function, and the sampling rate is specified. The Mel spectrum is extracted from the waveform data using a function to extract the Mel spectrum to characterize the frequency components and intensity distribution of the audio signal. The amplitude spectrum generation function is used to convert the Mel spectrum into a logarithmic amplitude spectrum, which is the final spectrogram to characterize the relative intensity of the frequency components of the audio signal, so as to better represent the frequency characteristics. Among them, the Mel spectrum is widely used in audio processing and audio classification tasks because it can provide better frequency resolution, is more in line with human ear perception, and is conducive to feature extraction and classification of audio signals.
[0051] 203. Construct a sequential model and train the sequential model based on the waveform data and spectrogram corresponding to the audio data to be trained and the human voice track to be trained to obtain an audio separation model.
[0052] For example, a sequential model, such as a convolutional neural network model, is created. The waveform data and spectrogram corresponding to the audio data to be trained, as well as the vocal track to be trained, are input into the sequential model to obtain a model processing result. Based on the model processing result, the sequential model is trained and optimized to obtain a trained model, namely the audio separation model. During the training process, it is necessary to use an appropriate loss function and optimizer, and set appropriate hyperparameters. At the same time, the model performance needs to be evaluated using appropriate evaluation metrics to ensure the effectiveness of the model.
[0053] 204. Acquire real-time audio data; and based on the audio separation model, separate the human voice audio and background audio in the real-time audio data to obtain an audio separation result corresponding to the real-time audio data.
[0054] Exemplarily, the device obtains real-time audio data and inputs the real-time audio data into an audio separation model, so that the audio separation model separates the human voice audio and background audio in the real-time audio data, and obtains an audio separation result corresponding to the real-time audio data to obtain high-quality human voice audio.
[0055] For example, Figure 2 The embodiment of the present application provides a flow chart of voice separation in audio and video communication, such as Figure 2 As shown, the goal of mobile-based real-time voice separation technology is to separate human voices from background sounds in audio signals, enabling better performance in applications such as speech recognition, audio analysis, and processing. This technology requires knowledge in fields such as audio signal processing, machine learning, and deep learning, and requires the use of appropriate tools and algorithms. The basic approach includes the following steps: Model preparation: Use the machine learning platform to train a model suitable for audio separation and export it to the model format of the mobile version of the machine learning platform; Model import: Import the exported machine learning platform model file into the app; Audio input: Capture an audio signal containing human voices and background sounds through a microphone or other audio input device; Audio preprocessing: Preprocess the input audio signal, such as by noise reduction or adding white noise for standardization, to prepare it for model input; Model inference: Input the preprocessed audio signal into the mobile version of the machine learning platform model and run the model for inference; Audio output: The separated human voice and background sounds are output separately, which can be played or further processed to enhance performance in applications such as speech recognition, audio analysis, and processing.
[0056] In this embodiment, an audio data processing method is provided, which processes the audio data to be trained to obtain corresponding waveform data and spectrogram, and trains a sequential model in combination with the human voice track to be trained to obtain an audio separation model, so as to separate the human voice audio and background audio in the real-time audio data and obtain the corresponding audio separation result, thereby achieving efficient human voice separation and improving communication quality.
[0057] Figure 3 A flowchart of another audio data processing method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method includes:
[0058] 301. Acquire audio data to be trained, wherein the audio data to be trained includes audio signals of different types of music; and acquire a vocal track to be trained.
[0059] For example, Figure 4 A model training flow chart provided in the embodiment of the present application is as follows: Figure 4 As shown, during training set loading and preprocessing, the training set is first loaded to obtain an audio file. The audio file is then loaded and preprocessed to obtain audio data and vocal data, thereby obtaining the audio data and vocal track to be trained. The audio data to be trained may include whistling audio captured by a simulated mobile phone. For example, the length of the whistling sound is adjusted to match the audio data. The execution code is: if len(whistle) > len(audio):, whistle = whistle[:len(audio)], else:. If the length of the whistle sound is shorter than the audio data, the whistle sound is repeated to match the length of the audio data. The execution code is: num_repeats = len(audio) / / len(whistle), whistle = np.tile(whistle, num_repeats), whistle = np.concatenate((whistle, whistle[:len(audio) % len(whistle)])). The whistle is mixed into the audio data according to the specified mixing ratio. The execution code is: mixed_audio = audio + mixing_ratio * whistle. The corresponding whistle is added to obtain the audio data to be trained.
[0060] 302. Convert the audio data to be trained to obtain waveform data corresponding to the audio data to be trained; and extract a spectrogram corresponding to the waveform data; wherein the waveform data represents the amplitude variation of the audio signal over time, and the spectrogram represents the relative strength of the frequency components of the audio signal.
[0061] For example, in combination Figure 4 , load the audio data to be trained, obtain the audio PCM data in the audio data to be trained, convert the audio PCM data into waveform data, and extract the spectrum diagram according to the waveform data.
[0062] 303. Construct a sequential model and perform enhancement processing on the waveform data corresponding to the audio data to be trained to obtain enhanced data corresponding to the audio data to be trained.
[0063] Exemplarily, the device uses code to create a timing model object for building a sequential model layer by layer, such as creating a sequential model, adding a convolutional layer and a pooling layer to the sequential model, adding a recurrent neural network layer and a fully connected layer to the sequential model.
[0064] Combine Figure 4 The device applies audio enhancement technology to enhance the waveform data corresponding to the audio data to be trained, and obtains enhanced data corresponding to the audio data to be trained.
[0065] For example, the device adds the "apply_audio_augmentation" function for applying audio enhancement technology. Within this function, you can add various audio enhancement techniques to the training audio data, such as adding white noise, speed change, and mixing, to make the training audio data richer and closer to real calls.
[0066] In one example, step 303 includes the following steps:
[0067] The first step of step 303 is to determine a target Gaussian distribution random number; wherein the target Gaussian distribution random number represents the intensity of the white noise to be added.
[0068] The second step of step 303 is to perform white noise addition processing on the waveform data corresponding to the audio data to be trained according to the target Gaussian distribution random number to obtain enhanced data corresponding to the audio data to be trained.
[0069] For example, audio enhancement techniques are applied through a function in which a target Gaussian distributed random number is determined, for example, by generating a Gaussian distributed random number with a mean of 0 and a standard deviation of 0.05, representing the intensity of white noise to be added. Based on this target Gaussian distributed random number, white noise is then added to the amplitude spectrum to obtain enhanced data corresponding to the audio data to be trained. Specifically, the white noise is added to the waveform data corresponding to the audio data to be trained to obtain an enhanced audio signal, i.e., the enhanced data corresponding to the audio data to be trained.
[0070] 304. Perform filtering processing on the enhanced data corresponding to the audio data to be trained to obtain filtered data corresponding to the audio data to be trained.
[0071] For example, in combination Figure 4 In the apply_audio_filter function, various filters, such as low-pass and high-pass filters, can be added to the enhanced data corresponding to the audio data to be trained. This filtering process generates filtered data corresponding to the audio data to be trained. This data is then combined with the vocal track and added to the training set for model training.
[0072] In one example, step 304 includes:
[0073] Step 1: Perform median filtering on the enhanced data corresponding to the audio data to be trained to obtain initial filtered data corresponding to the audio data to be trained.
[0074] Step 2: Perform frequency domain conversion on the audio signal and noise profile in the initial filtered data to obtain frequency domain data corresponding to the audio data to be trained.
[0075] Step 3: Perform noise reduction processing on the frequency domain data corresponding to the audio data to be trained to obtain noise reduction data corresponding to the audio data to be trained.
[0076] Step 4: Perform time domain conversion processing on the noise reduction data corresponding to the audio data to be trained to obtain filtered data corresponding to the audio data to be trained.
[0077] Specifically, we use the signal.medfilt function to apply a median filter to the enhanced data corresponding to the audio data to be trained, obtaining the initial filtered data corresponding to the audio data to be trained. For example, the signal.medfilt function performs a median filter on the audio signal, where the kernel_size parameter represents the filter window size. The audio signal and noise profile from this initial filtered data are then extracted. The apply_audio_noise_reduction function accepts these as input and returns the denoised audio signal. This function first converts the audio signal and noise profile to the frequency domain, obtaining the frequency domain data corresponding to the audio data to be trained. Then, based on the principle of spectral subtraction, it estimates the noise spectrum of this frequency domain data and subtracts it from the audio spectrum to perform noise reduction, obtaining the denoised data corresponding to the audio data to be trained. Finally, the denoised audio signal, i.e., the denoised data, is converted back to the time domain, obtaining the filtered data corresponding to the audio data to be trained. The result is returned for model training.
[0078] 305. Train the sequential model according to the filter data and spectrogram corresponding to the audio data to be trained, and the human voice track to be trained, to obtain an audio separation model.
[0079] Exemplarily, the device uses code to create a time series model object, which is used to construct a sequential model layer by layer. For example, a sequential model is created, and convolutional and pooling layers, recurrent neural network layers, and fully connected layers are added to the sequential model. The sequential model is compiled and trained using a training set. Specifically, the sequential model is trained based on the filter data and spectrogram corresponding to the audio data to be trained, as well as the vocal data to be trained, to obtain an audio separation model.
[0080] During loading and preprocessing, use the apply_audio_augmentation function to apply audio enhancement techniques, and use the apply_audio_filter function to apply filters. This can increase the diversity of audio data during preprocessing and improve the quality of training data, thereby improving the quality of model training.
[0081] In one example, the sequential model includes at least one convolutional layer, a pooling layer corresponding to the convolutional layer, a recurrent neural network layer, and a fully connected layer; step 305 includes:
[0082] Step 1: Based on each convolutional layer and the pooling layer corresponding to the convolutional layer, feature extraction is performed on the filter data and spectrum graph corresponding to the audio data to be trained to obtain local features corresponding to the audio data to be trained; wherein the local features represent the structure and pattern of the audio signal in the time domain and frequency domain.
[0083] Step 2: Based on the recurrent neural network layer, feature extraction is performed on the local features corresponding to the audio data to be trained to obtain the key features corresponding to the audio data to be trained; wherein the key features represent the context information of the audio signal.
[0084] Step 3: Based on the fully connected layer, perform binary classification on the local features and key features corresponding to the audio data to be trained to obtain the audio classification results.
[0085] Step 4: Based on the optimizer and loss function, the sequential model is optimized according to the audio classification results and the human voice track to be trained to obtain the audio separation model.
[0086] Specifically, combined Figure 4The device uses code to create a time series model object, which is used to build a sequential model layer by layer. For example, to create a CNN-RNN model, a Sequential model is first created. Convolutional and pooling layers are added, including at least one convolutional layer, such as three, followed by a max pooling layer. These layers extract features from the input spectrogram. Specifically, each convolutional layer and its corresponding pooling layer extracts features from the waveform data and spectrogram corresponding to the training audio data, and then reshapes the output. Because the following RNN layer requires a one-dimensional input, the two-dimensional convolutional output is converted to a one-dimensional vector to obtain local features corresponding to the training audio data. These local features represent the structure and patterns of the audio signal in the time and frequency domains. Next, an RNN layer is added. Two gated recurrent unit layers are added to the sequential model, each of which returns a sequence output. These RNN layers process time series data and capture long-term dependencies in the audio. Specifically, local features corresponding to the training audio data are extracted, resulting in key features corresponding to the training audio data. Each key feature represents the temporal dependencies and contextual information of the audio signal. The sequential model adds a fully connected layer and uses an activation function to perform binary classification (although it may be used for regression here because there is only one output unit). That is, it performs multi-scale feature fusion on the local features and key features corresponding to the audio data to be trained, and then performs binary classification on the fused features to obtain the audio classification results corresponding to the audio data to be trained, including the separated human voice data. The loss function is used to compare the separated human voice data in the audio classification results with the human voice track to be trained. The sequential model is optimized by the optimizer to obtain the audio separation model, that is, the trained model, which is saved in a file.
[0087] For example, based on the function for creating a model based on a convolutional neural network and a recurrent neural network, the created model accepts an input shape (input_shape) parameter and returns a sequential model. The structure of the sequential model is as follows: extract features by adding a 2D convolution layer, using 32 3x3 convolution kernels and ReLU activation function, and the input shape is specified by the input_shape parameter; reduce the size of the feature map by adding a 2D maximum pooling layer, using a 2x2 pooling window; repeat steps 1 and 2, add a convolution layer with 64 convolution kernels and a 2D maximum pooling layer; repeat steps 1 and 2, add a convolution layer with 128 convolution kernels and a 2D maximum pooling layer; adjust the shape of the feature map to (-1, output_height*output_width*num_channels), where output_height and output_width are the height and width of the feature map output by the last layer, and num_channels is the number of channels; add two GRU layers, each with 64 hidden units; add a fully connected layer, use the activation function, and output a scalar value (1 dimension) to obtain the audio classification result.
[0088] For example, run model.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy']) to compile the model. Here, the model uses the Adam optimizer, binary cross entropy as the loss function, and calculates accuracy as the evaluation metric. Then, run model.fit(train_data, train_labels, epochs = 10, batch_size = 32) to train the model. Here, the training data and corresponding labels are used for training. The training process will be carried out in 10 batches of 32. The trained model is saved to the file model.save('vocal_separation_model.h5').
[0089] CNNs have excellent feature extraction capabilities and can automatically learn local features in audio data. Audio signals typically exhibit structure and patterns in both the time and frequency domains. CNNs can effectively capture these features through convolution and pooling operations, extracting useful audio representations. Audio data is time-series in nature, and RNNs are capable of modeling sequential data. Through RNN layers, the model can capture the temporal dependencies and contextual information of audio signals. This is crucial for audio classification tasks, as some key features in audio may span multiple time steps. The CNN-RNN architecture can handle variable-length audio data, as RNN layers can flexibly handle sequences of varying lengths. This is particularly useful for audio classification in practical applications, where audio clips may vary in length. The CNN-RNN architecture improves the model's robustness and generalization capabilities. Through the feature extraction capabilities of the CNN layers, the model can learn representations that are robust to various variations in the input audio, such as audio quality and background noise. Furthermore, the sequence modeling capabilities of the RNN layers help the model better understand the contextual information in the audio data, improving its ability to distinguish between different audio categories. The CNN-RNN architecture extracts features at different scales through multiple convolutional and pooling layers. Earlier convolutional layers can capture lower-level local features, while deeper convolutional layers can capture higher-level abstract features. This multi-scale feature fusion can improve the model's ability to represent audio data.
[0090] 306. Obtain real-time audio data.
[0091] For example, this step may refer to step 204 and will not be described in detail here.
[0092] In one example, step 306 includes:
[0093] Step 1: Import the header file and create the audio component class.
[0094] Step 2: Get the audio acquisition unit instance according to the audio component class.
[0095] Step 3: Configure the preset audio format into the audio acquisition unit instance; and configure the input and output information of the audio acquisition unit instance.
[0096] Step 4: Based on the start function, start the audio acquisition unit instance; and based on the start acquisition function, configure the audio acquisition unit corresponding to the audio acquisition unit instance to obtain real-time audio data.
[0097] Specifically, Figure 5 An audio acquisition flow chart provided in an embodiment of the present application is as follows: Figure 5As shown, first, in the code, the audio acquisition library imports the header file, including the required configuration files, and creates an audio component class that conforms to the device, namely the Audio Component class, which is subsequently used to instantiate the Audio Unit audio acquisition unit instance. Through the AudioComponent audio component, obtain the Audio Unit instance, for example, execute Audio Component Instance audioUnit = Audio Component Instance New (component, NULL); create the format of the captured audio and obtain the preset audio format, for example, sampling rate: 44100.0; audio format: PCM; number of channels: 1; number of bytes per frame: audioFormat.m Bytes Per Frame = size of (int16_t) * number of channels; number of frames per packet: 1; number of bytes per packet: number of frames per packet * number of bytes per frame; configure the preset audio format to the Audio Unit audio acquisition unit instance, and configure the input and output information of the audio acquisition unit instance, including input parameters and output parameters. Call the Audio Unit audio acquisition unit's start function to start the audio acquisition unit instance. Call the system to start the collection function to configure the audio collection unit to start collecting audio. In the callback function of the audio collection unit, the system microphone audio is collected and processed to obtain real-time audio data.
[0098] 307. Based on the audio separation model, separate the human voice audio and the background audio in the real-time audio data to obtain an audio separation result corresponding to the real-time audio data.
[0099] For example, this step may refer to step 204 and will not be described in detail here.
[0100] In one example, step 307 includes:
[0101] Step 1: Perform echo cancellation processing on the real-time audio data to obtain processed real-time audio data.
[0102] Step 2: Based on each convolution layer and each pooling layer in the audio separation model, feature extraction is performed on the spectrum graph corresponding to the processed real-time audio data to obtain local features corresponding to the processed real-time audio data.
[0103] Step 3: Based on the recurrent neural network layer in the audio separation model, feature extraction is performed on the local features corresponding to the processed real-time audio data to obtain the key features corresponding to the processed real-time audio data.
[0104] Step 4: Based on the fully connected layer in the audio separation model, perform binary classification on the local features and key features corresponding to the processed real-time audio data to obtain the audio separation results.
[0105] For example, after we obtain the original data of the mobile phone audio, that is, the real-time audio data in the callback function, we can pre-process the real-time audio data, such as noise reduction, etc. Specifically, the real-time audio data can be echo-cancelled to eliminate the acquired audio echo. Echo cancellation is mainly achieved by identifying and eliminating the echo signal to obtain the processed real-time audio data. For example, the adaptive algorithm of the Audio Unit audio acquisition unit can be used to implement echo cancellation. The purpose of doing so is to improve the recognition readiness rate and obtain cleaner audio to improve the recognition rate. According to each convolutional layer and the pooling layer corresponding to the convolutional layer in the audio separation model, the spectrum corresponding to the processed real-time audio data is feature extracted and then reshaped and output. Since the next RNN layer requires a one-dimensional input, the two-dimensional convolution output is converted into a one-dimensional vector to obtain the local features corresponding to the processed real-time audio data; wherein, the local features characterize the structure and pattern of the audio signal in the time domain and frequency domain. Next, the RNN layer of the audio separation model extracts local features corresponding to the processed real-time audio data, obtaining key features corresponding to the processed real-time audio data. Key features characterize the temporal dependencies and contextual information of the audio signal. A fully connected layer of the audio separation model uses an activation function to perform binary classification. This involves performing multi-scale feature fusion on the local features and key features corresponding to the processed real-time audio data, and then performing binary classification on the fused features. This results in audio classification results corresponding to the processed real-time audio data, including the separated vocal data.
[0106] For example, create an Audio Unit instance and connect it to the captured real-time audio data, set the audio format to ensure that the real-time audio data can be processed correctly, and start the Audio Unit to start processing the real-time audio data. Figure 6 A model reasoning flow chart provided in the embodiment of this application is as follows: Figure 6 As shown, the audio separation model can be loaded on the mobile phone, and the audio separation model can be initialized after the model module is introduced in the code, for example, by executing _model = [[vocal_separation_model alloc]init]. The processed real-time audio data is input into the audio separation model, and the model is run to perform audio cropping and separation on the processed real-time audio data to obtain the audio separation result, and the processed audio in the audio separation result is output, and the processed audio is transmitted for subsequent use, for example, Figure 7An audio analysis waveform effect diagram provided in an embodiment of the present application, such as Figure 7 As shown, audio waveform analysis is performed on the processed audio.
[0107] In this embodiment, based on the above embodiment, a CNN-RNN structure is used to construct a model to improve the performance and effect of the audio classification model, which can effectively extract and model the key features and timing information in the audio data, thereby improving the robustness, generalization ability and classification accuracy of the model; the trained audio separation model is applied to the human voice separation technology in real-time audio and video communication, which can solve the problem of eliminating environmental audio in current mobile phone audio and video communication, highlighting the transmission of only human voice to ensure the quality of human voice in the call, and eliminating other sounds other than non-human voices, such as the environmental sound of mobile phone howling during the call, thereby improving the quality of communication.
[0108] Figure 8 A schematic diagram of the structure of an audio data processing device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the device includes:
[0109] An acquisition module 401 is configured to acquire audio data to be trained, wherein the audio data to be trained includes audio signals of different types of music; and acquire a vocal track to be trained;
[0110] Processing module 402 is configured to convert the audio data to be trained to obtain waveform data corresponding to the audio data to be trained; and extract a spectrogram corresponding to the waveform data; wherein the waveform data represents the amplitude variation of the audio signal over time, and the spectrogram represents the relative strength of the frequency components of the audio signal;
[0111] A training module 403 is used to construct a sequential model and train the sequential model based on the waveform data and spectrogram corresponding to the audio data to be trained, as well as the vocal track to be trained, to obtain an audio separation model;
[0112] The separation module 404 is used to obtain real-time audio data; and based on the audio separation model, separate the human voice audio and background audio in the real-time audio data to obtain an audio separation result corresponding to the real-time audio data.
[0113] In one example, the training module 403 is specifically used to: enhance the waveform data corresponding to the audio data to be trained to obtain enhanced data corresponding to the audio data to be trained; filter the enhanced data corresponding to the audio data to be trained to obtain filtered data corresponding to the audio data to be trained; train the sequential model according to the filtered data and spectrum corresponding to the audio data to be trained, and the human voice track to be trained to obtain an audio separation model.
[0114] In one example, the training module 403 is specifically used to: determine a target Gaussian distribution random number; wherein the target Gaussian distribution random number represents the intensity of white noise to be added; according to the target Gaussian distribution random number, perform white noise addition processing on the waveform data corresponding to the audio data to be trained to obtain enhanced data corresponding to the audio data to be trained.
[0115] In one example, the training module 403 is further specifically used to: perform median filtering on the enhanced data corresponding to the audio data to be trained to obtain initial filtered data corresponding to the audio data to be trained; perform frequency domain conversion on the audio signal and noise profile in the initial filtered data to obtain frequency domain data corresponding to the audio data to be trained; perform noise reduction on the frequency domain data corresponding to the audio data to be trained to obtain noise reduction data corresponding to the audio data to be trained; perform time domain conversion on the noise reduction data corresponding to the audio data to be trained to obtain filtered data corresponding to the audio data to be trained.
[0116] In one example, the sequential model includes at least one convolutional layer, a pooling layer corresponding to the convolutional layer, a recurrent neural network layer and a fully connected layer; the training module 403 is further specifically used to: based on each convolutional layer and the pooling layer corresponding to the convolutional layer, perform feature extraction on the filter data and spectrum graph corresponding to the audio data to be trained, and obtain local features corresponding to the audio data to be trained; wherein the local features represent the structure and pattern of the audio signal in the time domain and the frequency domain; based on the recurrent neural network layer, perform feature extraction on the local features corresponding to the audio data to be trained, and obtain key features corresponding to the audio data to be trained; wherein the key features represent the contextual information of the audio signal; based on the fully connected layer, perform binary classification processing on the local features and key features corresponding to the audio data to be trained, and obtain an audio classification result; based on the optimizer and the loss function, optimize the sequential model according to the audio classification result and the human voice track to be trained to obtain an audio separation model.
[0117] In one example, the separation module 404 is specifically used to: import a header file and create an audio component class; obtain an audio acquisition unit instance based on the audio component class; configure a preset audio format into the audio acquisition unit instance; and configure the input and output information of the audio acquisition unit instance; based on the start function, start the audio acquisition unit instance; and based on the start acquisition function, configure the audio acquisition unit corresponding to the audio acquisition unit instance to obtain real-time audio data.
[0118] In one example, the separation module 404 is further specifically used to: perform echo cancellation processing on the real-time audio data to obtain processed real-time audio data; based on each convolution layer and each pooling layer in the audio separation model, perform feature extraction on the spectrum graph corresponding to the processed real-time audio data to obtain local features corresponding to the processed real-time audio data; based on the recurrent neural network layer in the audio separation model, perform feature extraction on the local features corresponding to the processed real-time audio data to obtain key features corresponding to the processed real-time audio data; based on the fully connected layer in the audio separation model, perform binary classification processing on the local features and key features corresponding to the processed real-time audio data to obtain audio separation results.
[0119] The device of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles are the same and will not be repeated here.
[0120] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 9 As shown, the electronic device includes: a memory 501 and a processor 502; the memory 501 is a memory for storing instructions executable by the processor 502.
[0121] The processor 502 is configured to execute the method provided in the above embodiment.
[0122] The electronic device further includes a receiver 503 and a transmitter 504. The receiver 503 is used to receive instructions and data sent by other devices, and the transmitter 504 is used to send instructions and data to external devices.
[0123] The specific implementation process of the processor can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.
[0124] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.
[0125] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed on a computer, the computer executes the technical solution of the above embodiment.
[0126] The readable storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0127] An exemplary readable storage medium is coupled to a processor, such that the processor can read information from and write information to the readable storage medium. The readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may reside in an application-specific integrated circuit. The processor and the readable storage medium may also reside in a device as discrete components.
[0128] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when at least one processor executes the computer program, the technical solutions in the above embodiments can be implemented.
[0129] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.
[0130] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0132] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program code.
[0133] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0134] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.
Claims
1. A method for processing audio data, characterized in that: include: Acquire audio data to be trained, wherein the audio data to be trained includes audio signals of different types of music; and obtain a vocal track to be trained; Performing conversion processing on the audio data to be trained to obtain waveform data corresponding to the audio data to be trained; and extracting a spectrogram corresponding to the waveform data; wherein the waveform data represents the amplitude variation of the audio signal over time, and the spectrogram represents the relative strength of the frequency components of the audio signal; Constructing a sequential model, and training the sequential model based on the waveform data and spectrogram corresponding to the audio data to be trained, and the vocal track to be trained, to obtain an audio separation model; Acquire real-time audio data; and based on the audio separation model, separate the human voice audio and background audio in the real-time audio data to obtain an audio separation result corresponding to the real-time audio data.
2. The method according to claim 1, characterized in that The step of training the sequential model based on the waveform data and spectrogram corresponding to the audio data to be trained and the human voice track to be trained to obtain an audio separation model includes: Performing enhancement processing on the waveform data corresponding to the audio data to be trained to obtain enhanced data corresponding to the audio data to be trained; Performing filtering on the enhanced data corresponding to the audio data to be trained to obtain filtered data corresponding to the audio data to be trained; The sequential model is trained according to the filter data and the spectrogram corresponding to the audio data to be trained, and the human voice track to be trained, to obtain the audio separation model.
3. The method according to claim 2, characterized in that The enhancing the waveform data corresponding to the audio data to be trained to obtain enhanced data corresponding to the audio data to be trained includes: Determine a target Gaussian distribution random number; wherein the target Gaussian distribution random number represents the intensity of white noise to be added; According to the target Gaussian distribution random number, white noise addition processing is performed on the waveform data corresponding to the audio data to be trained to obtain enhanced data corresponding to the audio data to be trained.
4. The method according to claim 2, characterized in that The filtering process is performed on the enhanced data corresponding to the audio data to be trained to obtain filtered data corresponding to the audio data to be trained, including: Performing median filtering on the enhanced data corresponding to the audio data to be trained to obtain initial filtered data corresponding to the audio data to be trained; Performing frequency domain conversion processing on the audio signal and noise profile in the initial filtered data to obtain frequency domain data corresponding to the audio data to be trained; Performing noise reduction processing on the frequency domain data corresponding to the audio data to be trained to obtain noise reduction data corresponding to the audio data to be trained; Performing time domain conversion processing on the noise reduction data corresponding to the audio data to be trained to obtain filtered data corresponding to the audio data to be trained.
5. The method according to claim 2, characterized in that The sequential model includes at least one convolutional layer, a pooling layer corresponding to the convolutional layer, a recurrent neural network layer, and a fully connected layer; the sequential model is trained according to the filter data and spectrogram corresponding to the audio data to be trained, and the human voice track to be trained to obtain the audio separation model, including: Based on each of the convolutional layers and the pooling layer corresponding to the convolutional layer, feature extraction is performed on the filter data and the spectrogram corresponding to the audio data to be trained to obtain local features corresponding to the audio data to be trained; wherein the local features represent the structure and pattern of the audio signal in the time domain and the frequency domain; Based on the recurrent neural network layer, feature extraction is performed on local features corresponding to the audio data to be trained to obtain key features corresponding to the audio data to be trained; wherein the key features represent contextual information of the audio signal; Based on the fully connected layer, performing binary classification processing on the local features and key features corresponding to the audio data to be trained to obtain an audio classification result; Based on the optimizer and the loss function, the sequential model is optimized according to the audio classification result and the human voice track to be trained to obtain the audio separation model.
6. The method according to claim 1, characterized in that The acquiring of real-time audio data includes: Import the header file and create the audio component class; According to the audio component class, obtain an audio acquisition unit instance; Configuring a preset audio format into the audio acquisition unit instance; and configuring input and output information of the audio acquisition unit instance; Based on the start function, the audio acquisition unit instance is started; and based on the start acquisition function, the audio acquisition unit corresponding to the audio acquisition unit instance is configured to obtain the real-time audio data.
7. The method according to any one of claims 1 to 6, characterized in that The separating process of the human voice audio and the background audio in the real-time audio data based on the audio separation model to obtain an audio separation result corresponding to the real-time audio data includes: Performing echo cancellation processing on the real-time audio data to obtain processed real-time audio data; Based on each convolution layer and each pooling layer in the audio separation model, feature extraction is performed on the spectrogram corresponding to the processed real-time audio data to obtain local features corresponding to the processed real-time audio data; Based on the recurrent neural network layer in the audio separation model, feature extraction is performed on local features corresponding to the processed real-time audio data to obtain key features corresponding to the processed real-time audio data; Based on the fully connected layer in the audio separation model, binary classification processing is performed on the local features and key features corresponding to the processed real-time audio data to obtain the audio separation result.
8. An audio data processing device, characterized in that: include: An acquisition module is used to acquire audio data to be trained, wherein the audio data to be trained includes audio signals of different types of music; and acquire a vocal track to be trained; a processing module, configured to convert the audio data to be trained to obtain waveform data corresponding to the audio data to be trained; and extract a spectrogram corresponding to the waveform data; wherein the waveform data represents the amplitude variation of the audio signal over time, and the spectrogram represents the relative strength of the frequency components of the audio signal; A training module is used to construct a sequential model and train the sequential model based on the waveform data and spectrogram corresponding to the audio data to be trained and the vocal track to be trained to obtain an audio separation model; The separation module is used to obtain real-time audio data; and based on the audio separation model, separate the human voice audio and background audio in the real-time audio data to obtain the audio separation result corresponding to the real-time audio data.
9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.