High-fidelity parameterized audio equalizer control method and system based on deep learning

Abstract

The invention relates to a high-fidelity parameterization audio equalizer control method and system based on deep learning, and is applied to the technical field of audio equalizers, and the method comprises the steps: collecting and preprocessing audio samples, extracting audio features, and training a preset deep learning model according to the audio features; extracting time-frequency characteristics of the current audio signal, transmitting the time-frequency characteristics to the deep learning model, and outputting parameter information including frequency, gain and bandwidth parameters; and adjusting the parameterized audio equalizer in real time according to the parameter information, so that the parameterized audio equalizer outputs an optimized high-fidelity audio signal. According to the method, the preset deep learning model is trained according to the audio features, the time-frequency features are transmitted to the deep learning model, the parameter information is output, and then the parameterized audio equalizer is adjusted in real time according to the parameter information, so that the parameterized audio equalizer outputs the optimized high-fidelity audio signal, and the effect of intelligently optimizing the equalizer parameters is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of audio equalizers, and in particular to a high-fidelity parametric audio equalizer control method and system based on deep learning. Background Art

[0002] An audio equalizer is an electronic device that can adjust the volume of different frequency bands. It is widely used in music production, audio processing, and high-fidelity audio systems. Traditional audio equalizers usually use fixed parameters or manual adjustment methods, and cannot be intelligently optimized according to different audio environments and content. With the development of deep learning technology, its application in the field of audio processing has provided new possibilities for improving the performance of audio equalizers.

[0003] One limitation of traditional equalizers is that they often cannot automatically adjust parameters based on different audio content. This means that when playing different types of music or audio, the settings of the equalizer need to be changed manually to match the best listening experience for the specific audio content. Due to the lack of automatic adaptability, this may result in suboptimal sound quality in some cases, especially when the audio content changes frequently, manual adjustments may become impractical or time-consuming.

[0004] Regarding the above-mentioned related technologies, it is believed that the traditional equalizer cannot automatically adjust parameters according to different audio content, resulting in the sound quality not being able to achieve the best effect, affecting the user experience. Summary of the invention

[0005] In order to improve the problem that traditional equalizers cannot automatically adjust parameters according to different audio content, resulting in suboptimal sound quality and affecting user experience, the present application provides a high-fidelity parametric audio equalizer control method and system based on deep learning.

[0006] In a first aspect, the present application provides a high-fidelity parametric audio equalizer control method based on deep learning, which adopts the following technical solution: including: Collect and preprocess audio samples, extract audio features, and train a preset deep learning model based on the audio features; Extracting the time-frequency features of the current audio signal, transmitting the time-frequency features to the deep learning model, and outputting parameter information, wherein the parameter information includes: frequency, gain, and bandwidth parameters; The parametric audio equalizer is adjusted in real time according to the parameter information, so that the parametric audio equalizer outputs an optimized high-fidelity audio signal.

[0007] Optionally, after extracting the time-frequency features of the current audio signal, transmitting the time-frequency features to the deep learning model, and outputting parameter information, the method further includes: optimizing the deep learning model based on a preset feedback mechanism algorithm; The feedback mechanism algorithm is as follows: ; The audio signal is , the frequency domain characteristics are , the transfer function of the parametric equalizer is , deep learning model The predicted parameters are ; ; ; in, Indicates The gain function of the frequency band, parameter Including center frequency , Gain and bandwidth , deep learning model According to the input audio characteristics Predicting the optimal parameters .

[0008] Optionally, preprocessing is performed based on an unstructured reference face image to obtain face data; Performing fast network fine-tuning on the face data and the preset driving audio based on a deep neural network; Transmitting the fine-tuned audio data and the face data to a preset audio-face correspondence model for training; The audio signal is transmitted to the audio-to-face corresponding model, and the three-dimensional face key points corresponding to the audio signal are output to achieve the mapping of audio to dynamic video.

[0009] Optionally, based on a NeRF renderer, performing radiation field estimation on the three-dimensional facial key points and the reference facial image to obtain a radiation field estimation result; According to the radiation field estimation result, high-fidelity neural reconstruction and rendering are achieved to generate a dynamic face that is consistent with the audio signal and the reference face image.

[0010] Optionally, based on the reduced-state Viterbi algorithm, at each time step t, compute the Transition to each possible next state Probability ; For each state , retain the k state transition paths with the largest transition probability; Update each state The probability of is the probability of the state transition path with the highest probability among the retained paths; The state space calculation amount of the parametric audio equalizer is reduced based on the probability of the maximum state transition path.

[0011] Optionally, the entire path tracing process is divided into a plurality of sub-path segments based on a segmented path backtracking algorithm, each of the sub-path segments comprising a number of time steps t; At the end of each sub-path segment, the best path of all current states is stored, and starting from the last sub-path segment, the path is backtracked to determine the best path; Merge the optimal paths obtained by backtracing each of the sub-path segments to obtain a global optimal path; A parametric audio equalizer is adjusted based on the global optimal path.

[0012] In a second aspect, the present application provides a high-fidelity parametric audio equalizer control device based on deep learning, which adopts the following technical solution, including: An audio processing module is used to collect and preprocess audio samples, extract audio features, and train a preset deep learning model based on the audio features; A model output module, used to extract the time-frequency features of the current audio signal, transmit the time-frequency features to the deep learning model, and output parameter information, wherein the parameter information includes: frequency, gain and bandwidth parameters; The audio adjustment module is used to adjust the parametric audio equalizer in real time according to the parameter information so that the parametric audio equalizer outputs an optimized high-fidelity audio signal.

[0013] In a third aspect, the present application further provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the high-fidelity parametric audio equalizer control method based on deep learning as described above.

[0014] In a fourth aspect, the present application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute the high-fidelity parametric audio equalizer control method based on deep learning as described above.

[0015] In summary, the system in this application collects audio samples and preprocesses them, extracts audio features, and trains a preset deep learning model based on the audio features; extracts the time-frequency features of the current audio signal, transmits the time-frequency features to the deep learning model, outputs parameter information, and then adjusts the parametric audio equalizer in real time based on the parameter information, so that the parametric audio equalizer outputs an optimized high-fidelity audio signal, thereby automatically learning the optimal equalization parameters for different audio contents, and achieving the effect of intelligent optimization of the equalizer parameters. At the same time, the system combines the audio equalizer and uses the deep learning model to perform multi-speaker video synthesis. By analyzing the audio content, a high-fidelity video output matching the audio content is generated, which provides strong technical support for the efficient generation of any speaker's face and driving voice. Finally, in terms of hardware algorithm implementation, the system adopts a reduced-state Viterbi algorithm to reduce the computational complexity by reducing the branch state, effectively reducing the state space, and significantly reducing the required computing resources. The segmented path backtracking algorithm is adopted to reduce the number of paths that need to be processed in each clock cycle by segmenting the critical path, thereby improving the timing performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of a high-fidelity parametric audio equalizer control method based on deep learning.

[0017] Figure 2 It is a flowchart of the high-fidelity multi-speaker video synthesis method.

[0018] Figure 3 It is a structural block diagram of a high-fidelity parametric audio equalizer control device based on deep learning.

[0019] Explanation of reference numerals: 310, audio processing module; 320, model output module; 330, audio adjustment module. DETAILED DESCRIPTION

[0020] The following combination Figure 1 - Figure 3 This application is described in further detail.

[0021] An audio equalizer is a device or software tool used to adjust the different frequency components in an audio signal. It can enhance or attenuate the sound within a specific frequency range to improve the sound quality or adapt to different listening environments.

[0022] Defects, deficiencies or problems in the existing industry or products: Complex adjustment: Users need to manually adjust the parameters of multiple frequency bands, which is complex and time-consuming.

[0023] Lack of intelligent optimization: Traditional equalizers cannot automatically adjust parameters according to different audio content, resulting in suboptimal sound quality.

[0024] Hardware limitations: High-fidelity audio processing has high hardware requirements, and traditional equalizers are difficult to achieve efficient processing under limited resources.

[0025] Based on this, the embodiment of the present application discloses a high-fidelity parametric audio equalizer control method based on deep learning. The execution subject is a control system. The present invention adopts a deep learning algorithm to automatically learn the optimal equalization parameters of different audio contents by training audio data sets to achieve intelligent optimization. Specifically including: Automatic adjustment: using a deep learning algorithm to automatically adjust the equalizer parameters according to the real-time audio content to simplify the operation. High-fidelity processing: improving the accuracy and efficiency of audio processing by optimizing algorithms and hardware design. Multifunctional integration: combining text- and voice-driven high-fidelity multi-speaker video synthesis technology to achieve comprehensive optimization of multimedia content. Low-power design: low-power equalizer design and implementation to reduce power consumption and improve system stability.

[0026] Reference Figure 1 , the embodiment of the present application at least includes step S10 to step S30.

[0027] S10, collect and preprocess audio samples, extract audio features, and train a preset deep learning model based on the audio features.

[0028] S20, extracting the time-frequency features of the current audio signal, transmitting the time-frequency features to the deep learning model, and outputting parameter information.

[0029] The parameter information includes: frequency, gain and bandwidth parameters.

[0030] S30, adjusting the parametric audio equalizer in real time according to the parameter information, so that the parametric audio equalizer outputs an optimized high-fidelity audio signal.

[0031] Specifically, the system collects audio samples and preprocesses them, extracts audio features, and trains a preset deep learning model based on the audio features; extracts the time-frequency features of the current audio signal, feeds the time-frequency features into the deep learning model, outputs parameter information, and then adjusts the parametric audio equalizer in real time based on the parameter information, so that the parametric audio equalizer outputs an optimized high-fidelity audio signal, thereby automatically learning the optimal equalization parameters for different audio content and achieving the effect of intelligently optimizing the equalizer parameters.

[0032] In practice, this application includes the following steps: audio sample collection and preprocessing, deep learning model training, and real-time audio signal processing. The specific processing steps are as follows: 1. Audio sample collection and preprocessing The data sources of audio samples are: audio samples of different audio environments and contents, covering various music types, speech scenes, etc. Public audio datasets (such as LibriSpeech, GTZAN Music / Speech Collection) are combined with self-made samples to enhance the diversity and representativeness of samples. Recording is performed in different environments using high-quality recording equipment, and high-quality audio files are obtained from online resource libraries. The preprocessing of audio samples includes: denoising, normalization, and feature extraction. Denoising: Use classic denoising algorithms such as Wiener Filter to denoise audio samples. Normalization: Normalize the amplitude of audio signals to ensure that different audio samples are in the same amplitude range. Feature extraction: Extract time domain and frequency domain features, such as time domain waveform, spectrum, Mel-frequency cepstral coefficients (MFCC), instantaneous frequency, etc. These features will be used as inputs to the deep learning model. The specific steps of feature extraction are as follows: 1.1 Multi-channel convolutional neural network. Input layer: The input is the processed audio features (such as short-time Fourier transform (STFT) spectrogram). Multi-scale convolution layer: Use convolution kernels of different scales (such as 3×3, 5×5, 7×7) to extract features of different scales.

[0033] 1.2 Adaptive feature selection. Input layer: output features after multi-channel convolution. Feature weight calculation: calculate the weight of each feature through the adaptive weight mechanism, and weight the features according to the calculated weight. The formula is as follows: ; in, The weight of the i-th feature, is the weighted i-th feature, is the original i-th feature.

[0034] 1.3 Graph Convolutional Neural Network. Graph structure construction: The weighted features are regarded as nodes in the graph, and the graph structure between features is constructed. Graph convolution operation: The complex relationship between features is captured through graph convolution operation. The formula is as follows: ; in, It is The node feature matrix of the layer, is the adjacency matrix with self-loops added, yes The degree matrix of It is The weight matrix of the layer, is the activation matrix.

[0035] Summary: Multi-channel convolutional neural network (MC-CNN): extracts multi-scale features of audio signals through convolution kernels of different scales, enhancing the model's ability to understand audio details and overall structure.

[0036] Adaptive Feature Selection: Dynamically select important features through an adaptive weight mechanism to improve the effectiveness and robustness of feature extraction.

[0037] Graph Convolutional Network (GCN): Captures the complex relationships between features through graph structures, further improving the model's ability to understand audio signals.

[0038] Combined architecture: The combination of MC-CNN, adaptive feature selection and GCN can not only extract multi-scale spatial features of audio signals, but also capture long-range temporal dependencies, providing more comprehensive audio signal analysis and processing capabilities.

[0039] 2. Deep learning model training 2.1 Deep learning model architecture: We chose an innovative model architecture that combines a multi-channel convolutional neural network (MC-CNN) with a bidirectional long short-term memory network (Bi-LSTM). We use multi-channel convolution to capture the multi-scale features of audio signals, and use bidirectional LSTM to better capture time series information. This combination can more comprehensively understand the complex structure of audio signals and achieve higher-fidelity audio equalization control. The details are as follows: Multi-Channel Convolutional Neural Network (MC-CNN) is a method that extracts different scale features of input data through multiple convolution channels. In the field of audio signal processing, the use of MC-CNN can more comprehensively capture various features in audio signals, thereby improving the performance and robustness of the model.

[0040] Input layer: The input is processed audio features (such as short-time Fourier transform (STFT) spectrogram).

[0041] Multi-scale convolution layer: Use convolution kernels of different scales (such as 3×3, 5×5, 7×7) to extract features of different scales. The formula is as follows: ; in, The kth convolution is at position The output at represents the input feature matrix, is the input feature at position The value of is the convolution kernel weight matrix, The kth convolution kernel is at position The weight of Bias of the kth convolution kernel.

[0042] Activation function: Use the ReLU activation function, the formula is as follows: ; in, represents the output of the activation function, Represents the activation function input.

[0043] Pooling layer: Use the maximum pooling layer to reduce the feature dimension. The formula is as follows: ; in, Indicates that the pooling layer is at position The output, Indicates that the input feature is at position The value at .

[0044] Bidirectional LSTM unit: The forward and backward time series information is captured through bidirectional LSTM. The calculation formula of LSTM is as follows: Input Gate: ; in, The input gate output at time step t, is the Sigmoid activation function, Input gate weight matrix, The hidden state at time step t-1, The input at time step t, Bias for input gate.

[0045] Forget Gate: ; in, The output of the forget gate at time step t, Forget gate weight matrix, Forgot the location of the door.

[0046] Output Gate: ; in, The output gate output at time step t is, The output gate weight matrix, Bias for the output gate.

[0047] Candidate memory cells: ; in, The candidate memory cell output at time step t, Hyperbolic tangent activation function, Candidate memory cell weight matrix, Bias of candidate memory cells.

[0048] Current memory unit: ; in, The current memory cell at time t, The memory cell at time step t-1, The output of the forget gate at time step t, Output of the input gate at time step t.

[0049] Bidirectional processing: concatenate the outputs of the forward LSTM and the backward LSTM. The formula is as follows: ; in, The hidden state at time step t, The hidden state of the forward LSTM at time step t, The hidden state of the backward LSTM at time step t, Refers to the splicing operation.

[0050] 2.2 Data preprocessing. This includes feature segmentation and normalization. Feature segmentation: the audio features are segmented, each with a length of 25ms and an overlap of 10ms to obtain a short-time Fourier transform (STFT) spectrogram; Normalization: each feature segment is normalized to ensure that the feature input range is consistent.

[0051] 2.3 Model training. First, the loss function and optimizer: define the loss function as mean square error (MSE) and the optimization algorithm as Adam optimizer. Training process: use 80% of the data for training and 20% of the data for validation. Monitor the validation loss during training to avoid overfitting. Hyperparameter settings: learning rate 0.001, batch size 32, and training rounds 100.

[0052] 3. Real-time audio signal processing 3.1 Time-frequency feature extraction: Perform short-time Fourier transform (STFT) on the current audio signal to extract time-frequency features, and input the extracted time-frequency features into the trained deep learning model to obtain parameter information.

[0053] 3.2 Equalizer parameter adjustment. According to the output parameter information, the parametric audio equalizer is adjusted in real time to optimize the audio signal. The adjustment formula is as follows: ; in, is the transfer function, is the gain of the ith frequency band, is the center frequency, For broadband.

[0054] In some embodiments, considering the effect of continuous feedback optimization of the deep learning model, the corresponding steps are processed as follows: Optimize deep learning models based on preset feedback mechanism algorithms; The feedback mechanism algorithm is as follows: ; The audio signal is , the frequency domain characteristics are , the transfer function of the parametric equalizer is , deep learning model The predicted parameters are ; ; ; in, Indicates The gain function of the frequency band, parameter Including center frequency , Gain and bandwidth , deep learning model According to the input audio characteristics Predicting the optimal parameters .

[0055] Specifically, the algorithm includes the acquisition of audio signals, frequency domain feature extraction, and adjustment of the equalizer transfer function to ensure the robustness and adaptability of the model in different audio environments. The system can also combine reference face images and driving audio data to map audio to dynamic video, achieving high-fidelity neural reconstruction and rendering. The system uses the reduced-state Viterbi algorithm and the segmented path backtracking algorithm to reduce computational complexity and state space, significantly improving the system's timing performance and processing efficiency.

[0056] Reference Figure 2In some embodiments, considering the problem of combining audio with facial images to generate dynamic facial audio and video, the corresponding processing steps are as follows: pre-processing based on unstructured reference facial images to obtain facial data; fast network fine-tuning of facial data and preset driving audio based on deep neural networks; transmitting the fine-tuned audio data and facial data to the preset audio-to-face corresponding model for training; transmitting the audio signal to the audio-to-face corresponding model, outputting the three-dimensional facial key points corresponding to the audio signal, and realizing audio-to-dynamic mapping. Based on the NeRF renderer, the radiation field of the three-dimensional facial key points and the reference facial image is estimated to obtain the radiation field estimation result; based on the radiation field estimation result, high-fidelity neural reconstruction and rendering are realized to generate a dynamic face consistent with the audio signal and the reference facial image.

[0057] Figure 2 In the system, the system first receives the input driving speech signal, which is used as the initial data for system processing. The driving speech signal is extracted through the Deepspeech model. Deepspeech is a speech recognition model that can convert audio signals into feature representations. The extracted speech features enter the normalized stream prior module, which normalizes the features. Subsequently, the features are sent to the decoder for decoding to generate feature data for subsequent processing. The system uses the pre-collected reference face image as the base image of the target face. These images are pre-processed to ensure quality and consistency. The decoded speech features and reference face images are input to the target face domain transfer network. The network is responsible for mapping the speech features to the corresponding 3D face key points to achieve audio to dynamic video conversion. The 3D face key points output by the target face domain transfer network are processed by a renderer based on NeRF (neural radiant field). The NeRF renderer estimates the radiant field of the 3D face key points and the reference face image and generates high-fidelity dynamic face video. In the renderer, σ represents the estimated value of the radiation field, c represents the color information, and Ray Distance is the ray distance. The final rendering result is obtained by integration. The system finally outputs a high-fidelity dynamic face video corresponding to the input voice signal.

[0058] Specifically, reference face image acquisition and preprocessing: collect a small number of unstructured reference face images and perform preprocessing, including image alignment and normalization, to ensure the consistency and quality of the input data. Fast network fine-tuning: based on a deep neural network, fast fine-tuning is performed using a small number of reference face images and driving audio. Using the trained model weights, the network is fine-tuned to achieve efficient adaptation to new input data, avoiding the tedious process of frame-by-frame optimization. Audio to dynamic pre-training model: build a pre-training model of audio to 3D facial key points. The model is trained using a large-scale speaker video dataset to ensure its generalization ability and robustness. The specific steps are as follows: Input audio signal: predict the corresponding 3D facial key points through the pre-trained model to achieve audio to dynamic mapping. Efficient radiation field estimation: based on the NeRF method, the radiation field of the input 3D facial key points and reference face images is estimated, and the radiation field estimation results are used to achieve high-fidelity neural reconstruction and rendering, generating dynamic faces consistent with the input audio and reference images. Through innovative algorithm design and optimization, combined with an audio equalizer, and using a deep learning model, multi-speaker video synthesis is performed. By analyzing the audio content and generating high-fidelity video output that matches the audio content, high-fidelity, low-latency neural reconstruction and rendering are achieved, providing strong technical support for the efficient generation of arbitrary speaking faces and driving speech.

[0059] In some embodiments, considering that the equalizer needs to fit the hardware algorithm, the corresponding processing steps are as follows: Based on the reduced state Viterbi algorithm, at each time step t, calculate the Transition to each possible next state Probability ; For each state , retain the k state transition paths with the largest transition probabilities; update each state The probability of is the probability of the state transition path with the highest probability in the retained path; based on the probability of the largest state transition path, the state space calculation amount of the parametric audio equalizer is reduced. Based on the segmented path backtracking algorithm, the entire path tracing process is divided into multiple sub-path segments, each of which contains a number of time steps t; at the end of each sub-path segment, the best path of all current states is stored, and starting from the last sub-path segment, the path is backtracked to determine the best path; the best paths obtained by backtracking each sub-path segment are merged to obtain the global best path; the parametric audio equalizer is adjusted based on the global best path.

[0060] Specifically, in terms of hardware algorithm implementation, the system adopts the reduced-state Viterbi algorithm to reduce computational complexity by reducing branch states, effectively reducing the state space and significantly reducing the required computing resources. It also adopts a segmented path backtracking algorithm to split the critical path and reduce the number of paths that need to be processed in each clock cycle, thereby improving the timing performance of the system.

[0061] Furthermore, the system adopts a sliding block Viterbi decoding architecture. By cutting off the feedback loop, multi-path parallel processing is achieved, thereby further improving processing speed and efficiency. This architecture not only improves the system throughput, but also effectively reduces power consumption.

[0062] At the micro-architecture level, the present invention has made many optimizations. By optimizing the data path and the acquisition of common operators, the power consumption of the system is further reduced. For example, the present invention has redesigned the data path to make data transmission more efficient and reduce unnecessary power consumption. At the same time, the shared use of common operators also reduces the occupation of hardware resources, thereby reducing the total power consumption of the system.

[0063] The implementation principle of a high-fidelity parametric audio equalizer control method based on deep learning in the embodiment of the present application is as follows: the system collects audio samples and performs preprocessing, extracts audio features, and trains a preset deep learning model according to the audio features; extracts the time-frequency features of the current audio signal, transmits the time-frequency features to the deep learning model, outputs parameter information, and then adjusts the parametric audio equalizer in real time according to the parameter information, so that the parametric audio equalizer outputs an optimized high-fidelity audio signal, thereby automatically learning the optimal equalization parameters of different audio contents, and achieving the effect of intelligent optimization of equalizer parameters. At the same time, the system combines the audio equalizer and uses the deep learning model to perform multi-speaker video synthesis, and generates high-fidelity video output matching the audio content by analyzing the audio content, providing strong technical support for the efficient generation of any speaker face and driving voice. Finally, in terms of hardware algorithm implementation, the system adopts a reduced-state Viterbi algorithm to reduce the computational complexity by reducing the branch state, effectively reducing the state space, and significantly reducing the required computing resources. The system also adopts a segmented path backtracking algorithm to reduce the number of paths to be processed in each clock cycle by segmenting the critical path, thereby improving the timing performance of the system.

[0064] Figure 1 FIG. 1 is a flow chart of a high-fidelity parametric audio equalizer control method based on deep learning in one embodiment. It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0065] Based on the same technical concept, Figure 3 The embodiment of the present application also provides a high-fidelity parametric audio equalizer control device based on deep learning, which adopts the following technical solution. The device includes: An audio processing module 310 is used to collect and pre-process audio samples, extract audio features, and train a preset deep learning model based on the audio features; The model output module 320 is used to extract the time-frequency features of the current audio signal, transmit the time-frequency features to the deep learning model, and output parameter information, which includes: frequency, gain and bandwidth parameters; The audio adjustment module 330 is used to adjust the parametric audio equalizer in real time according to the parameter information so that the parametric audio equalizer outputs an optimized high-fidelity audio signal.

[0066] In some embodiments, the model output module 320 is also used to optimize the deep learning model based on a preset feedback mechanism algorithm; The feedback mechanism algorithm is as follows: ; The audio signal is , the frequency domain characteristics are , the transfer function of the parametric equalizer is , deep learning model The predicted parameters are ; ; ; in, Indicates The gain function of the frequency band, parameter Including center frequency , Gain and bandwidth , deep learning model According to the input audio characteristics Predicting the optimal parameters .

[0067] In some embodiments, the model output module 320 is further used to perform preprocessing based on the unstructured reference face image to obtain face data; Quickly fine-tune the network based on deep neural network for face data and preset driving audio; The fine-tuned audio data and face data are transmitted to the preset audio-face correspondence model for training; The audio signal is transmitted to the audio-face correspondence model, and the three-dimensional face key points corresponding to the audio signal are output to achieve the mapping of audio to dynamic video.

[0068] In some embodiments, the model output module 320 is further used to perform radiation field estimation on the three-dimensional face key points and the reference face image based on the NeRF renderer to obtain the radiation field estimation result; Based on the radiation field estimation results, high-fidelity neural reconstruction and rendering are achieved to generate dynamic faces consistent with the audio signal and reference face images.

[0069] In some embodiments, the model output module 320 is further configured to calculate, at each time step t, the number of states from each state based on the reduced state Viterbi algorithm. Transition to each possible next state Probability ; For each state , retain the k state transition paths with the largest transition probability; Update each state The probability of is the probability of the state transition path with the highest probability among the retained paths; The state space calculation amount of the parametric audio equalizer is reduced based on the probability of the maximum state transition path.

[0070] In some embodiments, the model output module 320 is further used to divide the entire path tracing process into a plurality of sub-path segments based on a segmented path backtracking algorithm, each sub-path segment comprising a number of time steps t; At the end of each sub-path segment, the best path of all current states is stored, and starting from the last sub-path segment, the path is backtracked to determine the best path; Merge the best paths obtained by backtracking each sub-path segment to obtain the global best path; A parametric audio equalizer is tuned based on a global optimal path.

[0071] The embodiment of the present application also discloses a control device.

[0072] Specifically, the control device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned high-fidelity parametric audio equalizer control method based on deep learning.

[0073] The embodiment of the present application also discloses a computer-readable storage medium.

[0074] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute the high-fidelity parametric audio equalizer control method based on deep learning as described above. The computer-readable storage medium includes, for example: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0075] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A high-fidelity parametric audio equalizer control method based on deep learning, characterized in that: include: Collect and preprocess audio samples, extract audio features, and train a preset deep learning model based on the audio features; Extracting the time-frequency features of the current audio signal, transmitting the time-frequency features to the deep learning model, and outputting parameter information, wherein the parameter information includes: frequency, gain, and bandwidth parameters; The parametric audio equalizer is adjusted in real time according to the parameter information, so that the parametric audio equalizer outputs an optimized high-fidelity audio signal.

2. According to claim 1, a high-fidelity parametric audio equalizer control method based on deep learning is characterized in that: After extracting the time-frequency features of the current audio signal, transmitting the time-frequency features to the deep learning model, and outputting parameter information, the method further includes: Optimizing the deep learning model based on a preset feedback mechanism algorithm; The feedback mechanism algorithm is as follows: ； The audio signal is , the frequency domain characteristics are , the transfer function of the parametric equalizer is , deep learning model The predicted parameters are ; ； in, Indicates The gain function of the frequency band, parameter Including center frequency , Gain and bandwidth , deep learning model According to the input audio characteristics Predicting the optimal parameters .

3. The high-fidelity parametric audio equalizer control method based on deep learning according to claim 1, characterized in that: The method further comprises: Preprocessing is performed based on unstructured reference face images to obtain face data; Performing fast network fine-tuning on the face data and the preset driving audio based on a deep neural network; Transmitting the fine-tuned audio data and the face data to a preset audio-face correspondence model for training; The audio signal is transmitted to the audio-to-face corresponding model, and the three-dimensional face key points corresponding to the audio signal are output to achieve the mapping of audio to dynamic video.

4. The high-fidelity parametric audio equalizer control method based on deep learning according to claim 3, characterized in that: After transmitting the audio signal to the audio-to-face correspondence model, outputting the three-dimensional face key points corresponding to the audio signal, and realizing the audio-to-dynamic mapping, the method further includes: Based on the NeRF renderer, performing radiation field estimation on the three-dimensional face key points and the reference face image to obtain a radiation field estimation result; According to the radiation field estimation result, high-fidelity neural reconstruction and rendering are achieved to generate a dynamic face that is consistent with the audio signal and the reference face image.

5. The high-fidelity parametric audio equalizer control method based on deep learning according to claim 4, characterized in that: The method further comprises: Based on the reduced-state Viterbi algorithm, at each time step t, calculate the Transition to each possible next state Probability ; For each state , retain the k state transition paths with the largest transition probability; Update each state The probability of is the probability of the state transition path with the highest probability among the retained paths; The state space calculation amount of the parametric audio equalizer is reduced based on the probability of the maximum state transition path.

6. The high-fidelity parametric audio equalizer control method based on deep learning according to claim 5, characterized in that: The method further comprises: The entire path tracing process is divided into a plurality of sub-path segments based on a segmented path backtracking algorithm, each of the sub-path segments comprising a number of time steps t; At the end of each sub-path segment, the best path of all current states is stored, and starting from the last sub-path segment, the path is backtracked to determine the best path; Merge the optimal paths obtained by backtracing each of the sub-path segments to obtain a global optimal path; A parametric audio equalizer is adjusted based on the global optimal path.

7. A high-fidelity parametric audio equalizer control device based on deep learning, characterized in that: The device comprises: An audio processing module is used to collect and preprocess audio samples, extract audio features, and train a preset deep learning model based on the audio features; A model output module, used to extract the time-frequency features of the current audio signal, transmit the time-frequency features to the deep learning model, and output parameter information, wherein the parameter information includes: frequency, gain and bandwidth parameters; The audio adjustment module is used to adjust the parametric audio equalizer in real time according to the parameter information so that the parametric audio equalizer outputs an optimized high-fidelity audio signal.

8. A control device, characterized in that: The device comprises: The method comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 6.

Images