Audio restoration method based on multi-track semantic separation and residual noise distribution and sound system

Abstract

The application relates to the field of audio processing and discloses an audio restoration method based on multi-track semantic separation and residual noise distribution and a sound system. The method comprises the following steps: separating original audio by using a pre-constructed audio separation model to obtain a plurality of semantic tracks and a residual track which is a residual component not explained by the semantic tracks; performing noise analysis on the residual track to obtain noise characteristics; according to the noise characteristics, distinguishing each semantic track into a clean semantic track or a non-clean semantic track; isolating the clean semantic track to a sandbox area, and performing special noise reduction and restoration processing on the non-clean semantic track; and superimposing the protected clean semantic track in the sandbox area and the noise reduction semantic track after the restoration processing to obtain a restored audio. The application separates and disassembles audio through multi-track semantic separation, estimates noise characteristics on the residual track, distinguishes clean tracks from non-clean tracks according to the noise characteristics, and specially reduces noise for the non-clean tracks, so that accurate noise reduction and signal fidelity are realized.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing, and in particular to an audio restoration method and audio system based on multi-track semantic separation and residual noise allocation. Background Technology

[0002] With the rapid development of digital audio technology, audio restoration has significant application value in music archiving, film and television post-production, and the digital preservation of historical recordings. Old music recordings, film optical sound tracks, and other historical audio materials often contain various types of noise, such as tape hissing, film optical acoustic background noise, and pulse pops, due to aging storage media and limitations of recording equipment, severely affecting the listening quality.

[0003] Audio noise reduction is a core technical step in audio restoration, music production, and film and television post-production. Traditional audio noise reduction methods mainly include the following categories: The first category is single-track overall noise reduction methods, such as spectral subtraction, Wiener filtering, and statistical model noise reduction. These methods directly estimate and reduce noise in mixed audio, based on the core assumption that noise has relatively stable statistical characteristics in the time-frequency domain. However, in real audio, the useful signal and noise highly overlap in the time-frequency domain, and overall noise reduction inevitably damages the useful signal, leading to problems such as loss of musical details and distortion of vocals. Furthermore, different semantic components (such as vocals, percussion, and strings) have different sensitivities to noise reduction, making it difficult to balance the protection needs of various components with uniform noise reduction intensity.

[0004] The second category is end-to-end noise reduction methods based on deep learning, such as noise reduction models based on convolutional neural networks or Transformers. These methods learn the mapping from noisy frequencies to clean audio through a large amount of paired data, achieving good results on specific noise types. However, these methods are usually trained on a single type of noise, and their noise reduction performance drops significantly when faced with multiple noises superimposed in real-world audio. Furthermore, end-to-end methods lack explicit modeling of the audio's semantic structure and cannot differentiate the processing of different semantic components, easily introducing artificial artifacts when restoring complex music or film audio.

[0005] Therefore, the existing technology has the following shortcomings: First, it cannot effectively protect the inherent characteristics of different semantic components during the noise reduction process, resulting in loss of detail and distortion in the restored audio; Second, it lacks the ability to specifically process multiple superimposed noises, making it difficult to meet the restoration needs in complex scenarios.

[0006] The disclosure of the above background technical content is only for the purpose of assisting in understanding the concept and technical solution of this application, and does not necessarily provide technical instruction. Summary of the Invention

[0007] The purpose of this invention is to provide an audio restoration method based on multi-track semantic separation and residual noise allocation. By separating the audio into multiple semantic tracks and residual tracks, noise analysis is performed on the residual tracks to obtain accurate noise features. Based on the signal-to-noise ratio of each semantic track, they are distinguished into clean tracks and unclean tracks. The unclean tracks are restored using a special noise reduction algorithm that matches the semantic type, while the clean tracks are isolated and protected. This effectively suppresses noise while preserving the original semantic details of the audio to the maximum extent, thereby improving the restoration quality.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An audio restoration method based on multi-track semantic separation and residual noise allocation includes the following steps: Using a pre-built audio separation model, the original audio to be repaired is separated to obtain multiple semantic tracks and a residual track. The residual track is the remaining component in the original audio that was not explained by the semantic track. Noise analysis was performed on the residual trajectory to obtain noise characteristics; Based on the noise characteristics, each semantic track is classified into a clean semantic track or a non-clean semantic track. The clean semantic track is isolated to the sandbox area, and the non-clean semantic track is subjected to corresponding special noise reduction and repair processing to obtain the noise-reduced semantic track. The protected clean semantic track in the sandbox area is recombined with the repaired denoised semantic track to obtain the repaired audio.

[0009] Furthermore, following any or a combination of the aforementioned technical solutions, the audio separation model adopts a deep learning-based encoder-decoder network architecture, which is configured with an encoder, a mask generation module, and a decoder, wherein: The encoder is configured to convert the time-domain signal of the original audio to be repaired into a time-frequency domain signal through a short-time Fourier transform, and extract high-level acoustic features through multi-layer convolution or Transformer coding. The mask generation module is configured to generate a corresponding time-frequency mask for each semantic track based on the high-level acoustic features. Each time-frequency unit of the time-frequency mask takes a value between [0,1], which represents the probability that the time-frequency unit belongs to the corresponding semantic track. The decoder multiplies the time-frequency mask of each semantic track with the time-frequency features of the original audio to obtain the time-frequency representation of each semantic track, and then converts it back to the time domain signal through inverse short-time Fourier transform, which serves as the audio waveform of each semantic track.

[0010] Furthermore, following any one or a combination of the aforementioned technical solutions, the time-frequency mask corresponding to the semantic track is generated using any of the following methods: Mask i ( m , k )= s ( W i · H ( m , k )+ b i ),in, Mask i ( m , k ) is the first i The first orbit m Frame, First k Time-frequency mask for each frequency point s ( ) is the Sigmoid activation function. H ( m , k ) represents the high-level acoustic features extracted by the encoder. W i , b i These are the learnable weights and biases corresponding to the i-th track, respectively; or, ,in, Mask i ( m , k ) is the first i The first orbit m Frame, First k Time-frequency mask for each frequency point N This represents the total number of semantic tracks. Indicates the first j The score function for each track; or, Mask i ( m , k )=Sigmoid( Attention i ( H ( m , k ))),in, Mask i ( m , k ) is the first i The first orbit m Frame, First k Time-frequency mask for each frequency point Attention i ( H ( m , k To focus on the first through attention mechanisms i The first orbit m Frame, First k The attention function for the time-frequency region of each frequency point, Sigmoid( ) is the activation function.

[0011] Furthermore, following any or a combination of the aforementioned technical solutions, the audio separation model is configured with an encoder-decoder model, a feature tagging module, a mapping module, and an audio track allocation module; wherein: The encoder-decoder model employs a convolutional neural network and a long short-term memory neural network structure. The encoder-decoder model is configured to: learn the temporal structure of music and parse the separated waveforms corresponding to multiple single sound sources in the target audio signal; identify the sound source attributes matched by the separated waveforms, where the sound source attributes include one or more of frequency, timbre, loudness, and pitch; and, based on the parsed separated waveforms of each single sound source and their matched sound source attributes, separate multiple audio elements from the target audio signal. The feature labeling module is configured to label the separated audio elements according to preset classification labels, which are multiple categories including human voice, piano, strings, guitar, bass, percussion, dialogue, motion effects, sound effects, and music. The mapping module is configured to pre-build the association mapping relationship between different tracks and different classification labels; The audio track allocation module is configured to assign the separated audio elements to tracks mapped to their labeled classification tags, thereby obtaining multiple semantic tracks.

[0012] Furthermore, following any one or a combination of the aforementioned technical solutions, the noise characteristics include a noise power spectrum; each semantic track is distinguished as a clean semantic track or a non-clean semantic track in the following manner: Multiplying the noise power spectrum by the time-frequency mask of the semantic track yields the noise power spectrum estimate corresponding to the semantic track; The noise power corresponding to each frequency point is summed or weighted averaged to obtain the overall noise power of the semantic track. The signal-to-noise ratio of this semantic track is calculated using the following formula: , where P noise P represents the overall noise power of this semantic track. signal This indicates the power of the semantic track itself; If the signal-to-noise ratio of the semantic track is higher than the preset signal-to-noise ratio threshold, the semantic track is determined to be a clean semantic track; otherwise, the semantic track is determined to be a non-clean semantic track.

[0013] Furthermore, following any one or a combination of the aforementioned technical solutions, the power P of the semantic track itself is calculated in the following manner. signal : If the overall noise power P noise To calculate the power of the semantic track itself using the summation of noise power, the following formula is used: ,in, X ( m , k This represents the time-frequency representation obtained by performing a short-time Fourier transform on the semantic orbit. m For frame index, k Frequency index; If the overall noise power P noise The power of the semantic track itself is calculated using the following formula, based on the noise power weighted average: ,in, X ( m , k This represents the time-frequency representation obtained by performing a short-time Fourier transform on the semantic orbit. m For frame index, k For frequency point index, M Total number of frames K This represents the total number of frequency points.

[0014] Furthermore, following any one or a combination of the aforementioned technical solutions, the noise analysis of the residual trajectory is performed as follows: The noise power estimate is updated using a soft-decision method based on the probability of speech presence, and the update formula is as follows: ,in, Indicates the current number m Frame, First k Noise power estimation at each frequency point m For frame index, k For frequency point index, Indicates the previous frame, the first frame k Noise power estimation at each frequency point Indicates the current number of residual trajectories. m Frame, First k The power spectrum of a noisy signal at each frequency point; Indicates by the first m Frame, First k Each frequency point of speech has a probability-determined adaptive smoothing factor, which is within a preset lower limit value α.min The adaptive smoothing factor is closer to 1 as the probability of speech is higher, and the linear interpolation between 1 and 1 is used.

[0015] Furthermore, following any one or a combination of the aforementioned technical solutions, the first... m Frame, First k The probability of voice at a specific frequency point It is calculated in the following way: Calculate the time-frequency representation of the current frame noisy signal of the residual orbit. Y ( m , k ); Calculate the posterior signal-to-noise ratio ; Calculate the prior signal-to-noise ratio Where α is a constant smoothing factor, This represents the historical state of the signal amplitude spectrum of the noise-reduced output in the previous frame. Indicates the posterior signal-to-noise ratio of the current frame. max [ ] indicates a function that takes the maximum value; and The first is calculated using the likelihood ratio or Sigmoid function mapping. m Frame, First k The probability of voice at a specific frequency point ; The adaptive smoothing factor is then calculated using the following formula. .

[0016] Furthermore, following any one or a combination of the aforementioned technical solutions, the noise analysis of the residual trajectory is performed as follows: The residual track is framed into audio signals; The time-frequency representation is obtained by performing a short-time Fourier transform on each frame. Y ( m , k ),in, m For frame index, k Frequency index; For each frequency point, the minimum power is tracked on the time axis characterized by the frame index, and used as the noise power estimate for that frequency point: ,in, Indicates the current number m Frame, First k Noise power estimation at each frequency point Indicates the previous frame, the first frame k Noise power estimation at each frequency point It is an increase factor greater than 1, and its set value is related to the allowable increase in noise estimation.

[0017] Furthermore, based on any one or a combination of the aforementioned technical solutions, and according to different audio scenarios, the original audio is separated to obtain corresponding semantic tracks: For music audio scenarios, the original audio is separated into multiple semantic tracks, including vocal track, piano track, string track, guitar track, bass track, and percussion track; For film and television audio scenarios, the original audio is separated into multiple semantic tracks, including dialogue track, motion effect track, sound effects track, and music track; And performing corresponding specific noise reduction and repair processing on the aforementioned unclean semantic tracks includes: For the unclean semantic tracks of the human voice track or the dialogue track, the first noise reduction algorithm is used for noise reduction and repair. For the unclean semantic tracks of the piano track, guitar track, or foley track, a second noise reduction algorithm is used for noise reduction and repair. For the unclean semantic track of the string track, a third noise reduction algorithm is used for noise reduction and repair; For the non-clean semantic tracks of the percussion track or motion effect track, the fourth noise reduction algorithm is used for noise reduction and repair. For the non-clean semantic tracks of the bass track, the fifth noise reduction algorithm is used for noise reduction and repair. For the non-clean semantic tracks of the music track, an adaptive noise reduction strategy is used to dynamically select one or more combinations of the first noise reduction algorithm, the second noise reduction algorithm, the third noise reduction algorithm, the fourth noise reduction algorithm, and the fifth noise reduction algorithm for noise reduction and repair.

[0018] Furthermore, following any one or a combination of the aforementioned technical solutions, the first noise reduction algorithm is a speech enhancement model based on deep learning, which is used to suppress noise while protecting the consonants and aerophones of speech. The second noise reduction algorithm is a transient protection noise reduction algorithm, which is used to detect transient regions in the signal and reduce the noise reduction intensity in the transient regions; The third noise reduction algorithm is a continuous tone protection noise reduction algorithm, which is used to protect continuous harmonic components and vibrato modulation during noise reduction. The fourth noise reduction algorithm is a pulse protection noise reduction algorithm, which is used to distinguish between real pulses and noise pulses, and then only attenuates the noise pulses while retaining the real pulses. The fifth noise reduction algorithm is a low-frequency dedicated noise reduction algorithm, which is used to attenuate low-frequency noise while protecting the low-frequency fundamental frequency and low-order harmonics of the bass track.

[0019] Furthermore, following any one or a combination of the aforementioned technical solutions, the noise reduction parameters of the first, second, third, fourth, and fifth noise reduction algorithms are set with reference to the noise characteristics corresponding to the residual track, and the noise reduction parameters include noise reduction intensity; the higher the noise power spectrum in the noise characteristics, the stronger the noise reduction intensity is set.

[0020] Furthermore, following any one or a combination of the aforementioned technical solutions, the step of recombining the clean semantic track and the denoised semantic track includes: The denoised semantic tracks after repair are phase-aligned with the protected clean semantic tracks in the sandbox area; The phase-aligned denoised semantic track is added to the clean semantic track to obtain the initial resynthesized signal; The spectral envelope difference between the preliminary resynthesized signal and the original audio to be repaired is calculated to identify the spectral hole bands of the preliminary resynthesized signal; Energy compensation is applied to the frequency band of the spectral holes in the preliminary reconstituted signal to obtain the restored audio.

[0021] Furthermore, following any one or a combination of the aforementioned technical solutions, the spectral envelope difference between the preliminary reconstituted signal and the original audio to be repaired is calculated in the following manner: Short-time Fourier transforms are performed on the preliminary reconstituted signal and the original audio to be repaired, respectively, to obtain the time-frequency representation of the preliminary reconstituted signal. Y recon ( m , k ) and the time-frequency representation of the original audio to be repaired. Y orig ( m , k ),in, m For frame index, k Frequency index; The average power spectrum of each frequency point is obtained by averaging the frame indexes.

[0022] in, P recon ( k The ) represents the average power spectrum of the initially reconstituted signal. P orig ( k () represents the average power spectrum of the original audio to be restored. M Total number of frames; The average power spectrum is smoothed to obtain the spectral envelope: , Where, smooth[ ] represents the smoothing operation function; Calculate the difference in spectral envelope: ,or, ; If the spectral envelope difference D ( k )or D dB ( k If the frequency band exceeds the preset hole threshold, it is determined to be a spectral hole band, and the restored audio after energy compensation is calculated using the following formula. ,or, ,in, β 1( k ), β 2( k ) represents a compensation coefficient between 0 and 1.

[0023] Furthermore, based on any one or a combination of the aforementioned technical solutions, the compensation coefficient is determined using the following adaptive strategy: β ( k )=

[0024] in, D For spectral envelope differences D ( k )or D dB ( k ), D low The hole threshold, D high The threshold for significant holes, β min For conservative compensation constant coefficients, β max To enhance the compensation constant coefficient, β max > β min ; or, β ( k )=min[1, ] ,in, D For spectral envelope differences D ( k )or D dB ( k ), D max This is the preset maximum allowable difference value. β baseThe reference compensation constant coefficient.

[0025] According to another aspect of the present invention, an audio system is provided, including a speaker, an audio processor, and a memory, wherein the memory is used to store program instructions, the audio processor is configured to run the program instructions to perform the steps of the method described above, and output the processed repaired audio to the speaker.

[0026] The beneficial effects of the technical solution provided by this invention are as follows: a. First, the original audio is separated into multiple semantic tracks and residual tracks. The residual tracks contain noise components in the original audio that are not interpreted by the semantic tracks. Noise analysis on the residual tracks can avoid interference from useful signals on noise estimation, thereby obtaining more accurate noise features than estimating directly on the mixed audio, and providing reliable parameter basis for subsequent noise reduction processing. b. Based on noise characteristics, each semantic track is divided into clean tracks and non-clean tracks. The clean tracks are isolated to the sandbox area without noise reduction processing, which fundamentally eliminates the negative impact of noise reduction algorithms on clean signals and ensures the complete preservation of the original high-quality components in the repaired audio. c. Different semantic tracks (such as vocals, percussion, and strings) have different acoustic characteristics and varying sensitivities to noise reduction. This invention employs specific noise reduction algorithms for the semantic types of non-clean tracks, effectively suppressing noise while preserving the unique acoustic details of each track. This avoids the loss of detail and distortion caused by uniform noise reduction, significantly improving the naturalness and fidelity of the restored audio. d. The noise characteristic parameters in the residual track provide a unified reference for the noise reduction of each semantic track, making the noise reduction parameter settings of each track targeted and consistent, and effectively dealing with the complex scene of multiple noise superposition in actual audio. This invention supports the identification of specific noise types (film optical acoustic background noise, tape hiss, pulse pop) in old film and television restoration and the adjustment of noise reduction parameters. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating an audio restoration method based on multi-track semantic separation and residual noise allocation, provided as an exemplary embodiment of the present invention; Figure 2This is a schematic diagram illustrating the process of recombining a clean semantic track with a denoised semantic track after repair processing, as provided in an exemplary embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that terms such as "first" and "second" in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such uses of the objects can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0031] In one embodiment of the present invention, an audio restoration method based on multi-track semantic separation and residual noise allocation is provided, such as... Figure 1 As shown, the audio repair method includes the following steps: S100: Using a pre-built audio separation model, the original audio to be repaired is separated to obtain multiple semantic tracks and a residual track. In one embodiment, the audio separation model adopts a deep learning-based encoder-decoder network architecture, which is configured with an encoder, a mask generation module, and a decoder, wherein: The encoder is configured to convert the time-domain signal of the original audio to be repaired into a time-frequency domain signal via a short-time Fourier transform, and extract high-level acoustic features through multi-layer convolution or Transformer coding. m Frame, First k The high-frequency acoustic characteristics of a single frequency point are defined as H ( m , k ); The mask generation module is configured to generate a corresponding time-frequency mask for each semantic track based on the high-level acoustic features. Mask i Each time-frequency unit of the time-frequency mask (the first...) m Frame, Firstk The frequency point (each frequency point) takes a value between [0,1], which represents the probability that the time-frequency unit belongs to the corresponding semantic orbit; The decoder multiplies the time-frequency mask of each semantic track with the time-frequency features of the original audio to obtain the time-frequency representation of each semantic track: X i ( m , k )= Mask i ( m , k )· Y ( m , k ),in, X i ( m , k ) is the first i Time-frequency representation of a semantic track Mask i ( m , k ) is the first i A set of time-frequency masks for each semantic track, Y ( m , k ) represents the time-frequency characteristics of the original audio to be repaired, and then the th i Time-frequency representation of semantic orbits X i ( m , k Transform back to the first digit using the inverse short-time Fourier transform. i Temporal signals of semantic tracks x i ( t )=ISTFT[ X i ( m , k )], which serves as the audio waveform for each semantic track.

[0032] Thus, after the original audio is input into the audio source separation model, the output consists of N semantic tracks and 1 residual track. The residual track is the remaining component in the original audio that has not been interpreted by the semantic track, i.e., residual track = original audio - semantic track 1 - semantic track 2 - ... - semantic track N; In this embodiment, the mask generation module is the core of multi-track separation. It generates time-frequency masks for each semantic track. Its input is the high-level acoustic features extracted by the encoder, and its output is N sets of time-frequency masks, each set of masks corresponding to one semantic track. Mask i Each time-frequency unit (the first) m Frame, Firstk The value of the frequency point (0, 1) is between [0, 1], indicating that the time-frequency unit belongs to the [1, 1]th frequency point. i The probability of each semantic track. The time-frequency mask corresponding to the semantic track is generated using any of the following methods: Method 1: Mask i ( m , k )= s ( W i · H ( m , k )+ b i ),in, Mask i ( m , k ) is the first i The first orbit m Frame, First k Time-frequency mask for each frequency point s ( ) is the Sigmoid activation function. H ( m , k ) represents the high-level acoustic features extracted by the encoder. W i , b i These are the learnable weights and biases corresponding to the i-th track, respectively; Method 2: ,in, Mask i ( m , k ) is the first i The first orbit m Frame, First k Time-frequency mask for each frequency point N This represents the total number of semantic tracks. Indicates the first j The score function for each track; Method 3: Mask i ( m , k )=Sigmoid( Attention i ( H ( m , k ))),in, Mask i ( m , k ) is the first i The first orbitm Frame, First k Time-frequency mask for each frequency point Attention i ( H ( m , k To focus on the first through attention mechanisms i The first orbit m Frame, First k The attention function for the time-frequency region of each frequency point, Sigmoid( ) is the activation function.

[0033] In Method Two, the scoring function can be a learnable neural network layer, such as a fully connected layer or a convolutional layer, whose internal parameters, weights w and bias b, are automatically adjusted during training (continuously optimized on the training data through backpropagation). The training data contains a large amount of known mixed audio and its corresponding clean tracks (such as vocals, piano, strings, etc.). For each time-frequency unit ( m , k The model manually labels the predicted orbital based on its actual orbital position. The goal of the model is to make the predicted mask... As close as possible to the real mask The loss function can be cross-entropy or mean squared error, for example:

[0034] By minimizing the loss function, the model gradually learns to assign the correct score to each time-frequency unit: When a certain time-frequency unit mainly belongs to human voice, the score corresponding to the human voice track. It should be higher; When a certain time-frequency unit primarily belongs to the piano, the score corresponding to the vocal track... It should be higher; Ultimately, we learned how to assign scores to each track based on the acoustic characteristics of higher altitudes: , where w is the weight of the learnable neural network layer according to the scoring function, and b is the bias of the neural network layer.

[0035] Besides generating time-frequency masks for each semantic track, another embodiment achieves track separation through audio source attribute recognition and mapping: see Chinese patent application CN117295004A, which is cited in its entirety to explain this embodiment, with key points as follows: The audio separation model is configured with an encoder-decoder model, a feature tagging module, a mapping module, and an audio track allocation module; wherein: The encoder-decoder model employs a convolutional neural network and a long short-term memory neural network structure. The encoder-decoder model is configured to: learn the temporal structure of music and parse the separated waveforms corresponding to multiple single sound sources in the target audio signal; identify the sound source attributes matched by the separated waveforms, where the sound source attributes include one or more of frequency, timbre, loudness, and pitch; and, based on the parsed separated waveforms of each single sound source and their matched sound source attributes, separate multiple audio elements from the target audio signal. The feature labeling module is configured to label the separated audio elements according to preset classification labels, which are multiple categories including human voice, piano, strings, guitar, bass, percussion, dialogue, motion effects, sound effects, and music. The mapping module is configured to pre-build the association mapping relationship between different tracks and different classification labels; The audio track allocation module is configured to assign the separated audio elements to the tracks mapped to their labeled classification tags. The separated audio elements have timestamps. Based on the timestamps of the audio elements assigned to each track, the single-track audio corresponding to each track is obtained, that is, multiple semantic tracks are obtained.

[0036] The encoder-decoder model is configured to be constructed as follows: An initial model is built based on the convolutional neural network and long short-term memory neural network, the initial model having multiple CNN layers and multiple LSTM layers. The CNN layers are used to extract features from each segment of the input signal, generating CNN features for each segment, and the execution space is modeled for each segment of the input signal based on the CNN features. The LSTM layers are used to process the CNN features to generate LSTM features for each segment, and the CNN features and the execution time of each segment are modeled based on the LSTM features. Separate waveforms of human voice and / or multiple types of musical instruments, as well as the full waveform of mixed audio, are collected, and the separated waveforms and the full waveform are manually labeled with classification tags as learning sample sets for model training and validation. For example, for various tracks, audio engineers manually separate the waveforms of human voice, background music, and various musical instruments, and label the waveforms, such as labeling one waveform as piano and another as human voice, etc. For example, the full waveform of a song contains separate waveforms of five single sound sources: Vocals, Drums, Guitar, Piano, and Bass. These separate waveforms are obtained through manual identification and separation. The learning sample set is then input into an initial model that completes spatial and temporal modeling. This initial model learns to extract the time-frequency features of vocals, background music, and / or instruments; that is, the model learns to distinguish various waveforms from the full waveform. Multi-objective learning of the initial model is achieved through time-frequency masking to obtain the encoder-decoder model capable of predicting the separate waveforms.

[0037] At this point, step S100 is complete, resulting in N semantic tracks and one residual track. For example, for old film and television audio, specific noise types such as film optical acoustic background noise, tape hiss, and pulse pops in the residual track are identified, and the noise reduction parameters of the corresponding track are adjusted according to the steps below.

[0038] S200: Perform noise analysis on the residual trajectory to obtain noise characteristics, the noise characteristics including a noise power spectrum; Specifically, the first method for noise analysis of the residual trajectory is to update the noise power estimate based on soft decision based on the probability of speech presence, with the update formula being: ,in, Indicates the current number m Frame, First k Noise power estimation at each frequency point m For frame index, k For frequency point index, Indicates the previous frame, the first frame k Noise power estimation at each frequency point Indicates the current number of residual trajectories.m Frame, First k The power spectrum of a noisy signal at each frequency point; Indicates by the first m Frame, First k Each frequency point of speech has a probability-determined adaptive smoothing factor, which is within a preset lower limit value α. min Linear interpolation between and 1, the first m Frame, First k The probability of voice at a specific frequency point The higher the value, the closer the adaptive smoothing factor is to 1. The calculation method is as follows: Calculate the time-frequency representation of the current frame noisy signal of the residual orbit. Y ( m , k ); Calculate the posterior signal-to-noise ratio ; Calculate the prior signal-to-noise ratio Where α is a constant smoothing factor, This represents the historical state of the signal amplitude spectrum of the noise-reduced output in the previous frame. Indicates the posterior signal-to-noise ratio of the current frame. max [ ] indicates a function that takes the maximum value; and The first is calculated using the likelihood ratio or Sigmoid function mapping. m Frame, First k The probability of voice at a specific frequency point ; The adaptive smoothing factor is then calculated using the following formula. .

[0039] The process of calculating the probability of speech presence using the likelihood ratio is as follows: First, the posterior signal-to-noise ratio is used... and prior signal-to-noise ratio Calculate the likelihood ratio This indicates the strength of evidence that the current time-frequency unit is more likely to contain speech:

[0040] Likelihood ratio A large signal-to-noise ratio indicates a high prior / posterior signal-to-noise ratio, suggesting the presence of speech. A likelihood ratio approaching 1 indicates the presence of speech; If the signal-to-noise ratio is very small, then the prior / posterior signal-to-noise ratio is low, and the probability of speech presence is low. If the value approaches 0, it is determined to be pure noise; the specific calculation formula is as follows:

[0041] in, This indicates the probability that there is no prior speech, i.e., the prior probability that the current time-frequency unit is pure noise. It can take a fixed value (e.g., 0.2 to 0.5) or be adaptively adjusted according to the speech activity detection (VAD) results.

[0042] Another approach is to calculate the probability of speech presence using a Sigmoid function mapping. :

[0043] Where a is a constant controlling the steepness of the mapping. The prior signal-to-noise ratio (SNR) threshold directly affects the tendency to determine the probability of speech presence. Physically, it represents the prior SNR corresponding to a speech presence probability of 0.5. This threshold can be set according to the noise type. For example, for stationary noise, because the noise power is relatively stable and the SNR estimation is reliable, the prior SNR threshold... One can be chosen; for non-stationary noise, It can be appropriately reduced to 0.5 to 0.8 to avoid rapidly changing noise causing speech to be misjudged as noise.

[0044] Besides the soft-decision update method based on the probability of speech presence, another way to perform noise analysis on the residual track is the minimum tracking method. The residual track is framed into audio signals; The time-frequency representation is obtained by performing a short-time Fourier transform on each frame. Y ( m , k ),in, m For frame index, k Frequency index; For each frequency point, the minimum power is tracked on the time axis characterized by the frame index, and used as the noise power estimate for that frequency point: ,in, Indicates the current number m Frame, First k Noise power estimation at each frequency point Indicates the previous frame, the first frame k Noise power estimation at each frequency point It is an increase factor greater than 1, and its set value is related to the allowable increase in noise estimation.

[0045] At this point, step S200 is complete, and the first [item] on the residual orbit is obtained. m Frame, First k Noise power estimation at each frequency point This yields the noise power spectrum of the residual track. Noise analysis is performed on the residual track (rather than the original mixed audio or individual semantic tracks), taking advantage of the high noise concentration in the residual track to obtain more accurate noise characteristic parameters.

[0046] S300: Based on the noise characteristics, each semantic track is classified into a clean semantic track or a non-clean semantic track: Multiplying the noise power spectrum by the time-frequency mask of the semantic track yields the noise power spectrum estimate corresponding to the semantic track; The noise power corresponding to each frequency point is summed or weighted averaged to obtain the overall noise power of the semantic track. The signal-to-noise ratio of this semantic track is calculated using the following formula: , where P noise P represents the overall noise power of this semantic track. signal The power of the semantic track itself is represented as follows: if the overall noise power P noise To calculate the power of the semantic track itself using the summation of noise power, the following formula is used: ,in, X ( m , k This represents the time-frequency representation obtained by performing a short-time Fourier transform on the semantic orbit. m For frame index, k For frequency point index, i.e., the first m Frame, First k Time-frequency representation of the time-frequency region to which each frequency point belongs; If the overall noise power P noise The power of the semantic track itself is calculated using the following formula, based on the noise power weighted average: ,in, X ( m , k This represents the time-frequency representation obtained by performing a short-time Fourier transform on the semantic orbit. m For frame index, k For frequency point index, M Total number of frames K This represents the total number of frequency points.

[0047] If the signal-to-noise ratio of the semantic track is higher than the preset signal-to-noise ratio threshold, the semantic track is determined to be a clean semantic track; otherwise, the semantic track is determined to be a non-clean semantic track.

[0048] S400: The clean semantic track is isolated to the sandbox area, and the non-clean semantic track is subjected to corresponding special noise reduction and repair processing to obtain the noise-reduced semantic track. The sandbox area is a logically isolated storage / processing area. Clean tracks placed in the sandbox area do not undergo any noise reduction processing and remain in their original state. The purpose of this is to avoid the noise reduction algorithm introducing additional distortion or artifacts (such as reduced musicality, smoothed transients, phase distortion, spectral coloration, etc.) into the already clean tracks, thereby preserving the sound details in the original audio that are not contaminated by noise to the greatest extent. For example, in a music restoration scenario, if the piano track and the string track have a high signal-to-noise ratio and are judged to be clean tracks, these two tracks are placed in the sandbox area for protection and no noise reduction operation is performed.

[0049] The specific noise reduction and repair processing performed on the aforementioned unclean semantic tracks means that different semantic track types have their own corresponding noise reduction and repair methods, rather than using a uniform noise reduction and repair method. Specifically, based on different audio scenarios, the original audio is separated into corresponding semantic tracks. For example, for music audio scenarios, the original audio is separated into multiple semantic tracks such as vocal tracks, piano tracks, string tracks, guitar tracks, bass tracks, and percussion tracks; for film and television audio scenarios, the original audio is separated into multiple semantic tracks such as dialogue tracks, motion effect tracks, foley tracks, and music tracks; different noise reduction algorithms are configured for the characteristics of each semantic track. For the non-clean semantic tracks of the vocal or dialogue tracks, a deep learning-based speech enhancement model is used for noise reduction and restoration. This model suppresses noise while protecting consonants and aerophones, such as publicly available models like Conv-TasNet and Demucs. These models are known speech enhancement techniques in the field. They are trained using a large amount of paired noisy and clean speech data to learn the mapping relationship from the time-frequency representation of noisy speech to the time-frequency representation of clean speech. This allows them to suppress background noise while preserving key acoustic features such as the fundamental frequency, formant structure, and consonant transients. This invention applies such known models to the noise reduction and restoration of vocal or dialogue tracks, utilizing their speech enhancement capabilities to suppress noise components in the tracks while simultaneously protecting vulnerable details such as consonants and aerophones through the model's ability to model speech structure. In specific applications, the noise reduction intensity parameters (such as gain upper limit and noise attenuation coefficient) of the speech enhancement model are set with reference to the noise feature parameters obtained in step S200 to adapt to the restoration requirements of different noise types and intensities.

[0050] For the non-clean semantic tracks of the piano track, guitar track, or foley track, a transient protection denoising algorithm is used for noise reduction and repair. This algorithm detects transient regions in the signal and reduces the denoising intensity in these regions. Transient detection can be achieved through time-domain energy surge detection, high-frequency band energy change rate analysis, or a deep learning-based transient detection model. This invention applies this known transient protection concept to the noise reduction and repair of piano tracks, guitar tracks, or foley tracks. Specifically, the following steps are taken: First, transient detection is performed on the track signal to identify the transient region where the attack of the note is located. Within the transient region, the denoising intensity is reduced or the denoising process is skipped to avoid damage to the impact and clarity of the attack of the note by the denoising algorithm. In non-transient regions (such as attenuation sections and sustained sections), conventional denoising processing is performed to suppress noise components. The threshold for determining the transient region and the reduction magnitude of the denoising intensity are set with reference to the noise characteristic parameters obtained in step S200.

[0051] For the non-clean semantic track of the string track, a continuous tone protection denoising algorithm is used for denoising and restoration. This algorithm is used to protect continuous harmonic components and vibrato modulation during denoising. The concept of harmonic structure preservation and continuous component protection is a known technique in the field of music signal processing. This invention applies this known continuous tone protection concept to the denoising and restoration of the string track. Specifically, the track signal is subjected to harmonic analysis to identify continuous harmonic components and their vibrato modulation characteristics. During denoising, a higher lower gain limit is set for the time-frequency unit where the continuous harmonic components are located to avoid the string sound becoming thin or unnatural due to excessive attenuation. A smaller denoising intensity is used for the frequency band where the vibrato modulation is located to protect the periodic amplitude changes and frequency perturbations of the vibrato. The harmonic protection range and vibrato protection frequency band are determined with reference to the noise characteristic parameters obtained in step S200.

[0052] For the non-clean semantic tracks of the percussion or motion effect tracks, a pulse protection denoising algorithm is used for noise reduction and restoration. This algorithm distinguishes between real pulses and noise pulses, and then attenuates only the noise pulses while preserving the real pulses. Existing tools such as iZotope RX and Audacity are examples. The idea of ​​distinguishing between real and noise pulses is a known technique in the field of audio restoration. Pulse detection can be achieved through time-domain amplitude threshold detection, spectral flatness analysis, or a classifier-based pulse recognition model. This invention applies this known pulse protection idea to the noise reduction and restoration of percussion or motion effect tracks. Specifically, the following steps are taken: pulse detection is performed on the track signal to identify pulse events; real pulses (drum beats, motion effects) and noise pulses (pops, clicks) are distinguished from noise pulses (cracks, clicks) by using the pulse's time-domain waveform characteristics (such as rise time, decay time), spectral characteristics (such as spectral flatness, bandwidth), or contextual characteristics (such as pulse interval regularity); only the noise pulses are attenuated, while the real pulses are fully preserved. The pulse classification threshold is set with reference to the noise feature parameters obtained in step S200.

[0053] For the non-clean semantic track of the bass track, a low-frequency dedicated noise reduction algorithm is used for noise reduction and repair. This algorithm attenuates low-frequency noise while protecting the low-frequency fundamental frequency and low-order harmonics of the bass track. Frequency-selective noise reduction is a known technique in the field of audio noise reduction. This invention applies this known frequency-selective noise reduction technique to the noise reduction and repair of the bass track. Specifically, the track signal is divided into frequency bands to identify noise components in the low-frequency band (e.g., 20Hz~200Hz); independent noise reduction parameters are set for the low-frequency band to attenuate the low-frequency noise; and a low-frequency protection mechanism is set to prevent excessive attenuation of the low-frequency fundamental frequency and low-order harmonics of the bass track. The frequency range, noise reduction intensity, and fundamental frequency protection range of the low-frequency band are set with reference to the noise characteristic parameters obtained in step S200.

[0054] For the non-clean semantic tracks of the music track, an adaptive noise reduction strategy is used to dynamically select one or more combinations of deep learning-based speech enhancement model noise reduction algorithms, transient protection noise reduction algorithms, continuous tone protection noise reduction algorithms, impulse protection noise reduction algorithms, and low-frequency dedicated noise reduction algorithms for noise reduction and restoration. The content characteristics of the music track can be obtained through spectral feature analysis of the track signal (such as harmonic density, transient density, and impulse density) or through identification by a pre-classifier. When multiple content characteristics are mixed, corresponding algorithms are combined to process the corresponding frequency bands or time-frequency units.

[0055] Compared to existing technologies that perform noise reduction on all separated tracks, this embodiment only performs noise reduction on non-clean tracks, fundamentally avoiding the introduction of additional distortion into clean tracks by the noise reduction algorithm.

[0056] Instead of using the same noise reduction algorithm for all tracks, select the appropriate noise reduction algorithm based on the sound type of each non-clean track (voice / dialogue, piano / guitar, strings, percussion / motion effects, bass).

[0057] In addition to music and film audio restoration, it can also be applied to podcast / interview audio restoration, meeting recording restoration, live recording restoration, voice memo restoration, and digital restoration of old records / tapes.

[0058] S500: Recombine the protected clean semantic track in the sandbox area with the repaired denoised semantic track to obtain the repaired audio.

[0059] The steps of resynthesis are as follows Figure 2 As shown: S510: Perform phase alignment between each denoised semantic track after repair and the protected clean semantic track in the sandbox area; S520: The phase-aligned denoised semantic track is added to the clean semantic track to obtain the initial resynthesized signal; S530: Calculate the spectral envelope difference between the preliminary reconstituted signal and the original audio to be repaired, in order to identify the spectral hole frequency band of the preliminary reconstituted signal; specifically, the spectral envelope difference is calculated as follows: Short-time Fourier transforms are performed on the preliminary reconstituted signal and the original audio to be repaired, respectively, to obtain the time-frequency representation of the preliminary reconstituted signal. Y recon ( m , k ) and the time-frequency representation of the original audio to be repaired. Y orig ( m , k ),in, m For frame index, k Frequency index; The average power spectrum of each frequency point is obtained by averaging the frame indexes.

[0060] in, P recon ( k The ) represents the average power spectrum of the initially reconstituted signal. P orig ( k () represents the average power spectrum of the original audio to be restored. M Total number of frames; The average power spectrum is smoothed to obtain the spectral envelope: , Where smooth[ ] represents the smoothing operation function; Calculate the difference in spectral envelope: ,or, ; If the spectral envelope difference D ( k )or D dB ( k If the frequency band exceeds the preset hole threshold, it is determined to be a spectral hole band, and the restored audio after energy compensation is calculated using the following formula. ,or, ,in, β 1( k ), β 2( k ) represents a compensation coefficient between 0 and 1.

[0061] compensation coefficient β 1( k ), β 2( k This can be a constant set through debugging, or in a specific embodiment, it can be determined through the following adaptive strategy (not distinguished below). β 1( k ), β 2( k ), uniformly defined as β ( k )): β ( k )=

[0062] in, D For spectral envelope differences D ( k )or D dB ( k ), D low The hole threshold, D high The threshold for significant holes, β min For conservative compensation constant coefficients, β max To enhance the compensation constant coefficient, β max > β min ; or, β( k )=min[1, ] ,in, D For spectral envelope differences D ( k )or D dB ( k ), D max This is the preset maximum allowable difference value. β base The reference compensation constant coefficient.

[0063] S540: Perform energy compensation on the frequency band of the spectral hole of the preliminary resynthesized signal to obtain the repaired audio.

[0064] Performing cross-track phase alignment and spectral hole compensation on the denoised tracks during the resynthesis stage can reduce comb filtering effects and spectral holes, reduce phase inconsistencies and spectral energy loss problems introduced by the separation-denoising-resynthesis pipeline, and help improve the overall naturalness of the restored audio.

[0065] If the spectral envelope differences calculated in step S530 do not exceed the preset hole threshold, it indicates that the preliminary resynthesized signal does not have any spectral hole frequency bands that require energy compensation. In this case, the preliminary resynthesized signal is directly used as the repaired audio.

[0066] According to another aspect of the present invention, an audio system is provided, including a speaker, an audio processor, and a memory, wherein the memory is used to store program instructions, the audio processor is configured to run the program instructions to perform the steps of the method described above, and output the processed repaired audio to the speaker.

[0067] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0068] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An audio restoration method based on multi-track semantic separation and residual noise allocation, characterized in that, Includes the following steps: Using a pre-built audio separation model, the original audio to be repaired is separated to obtain multiple semantic tracks and a residual track. The residual track is the remaining component in the original audio that was not explained by the semantic track. Noise analysis was performed on the residual trajectory to obtain noise characteristics; Based on the noise characteristics, each semantic track is classified into a clean semantic track or a non-clean semantic track. The clean semantic track is isolated to the sandbox area, and the non-clean semantic track is subjected to corresponding special noise reduction and repair processing to obtain the noise-reduced semantic track. The protected clean semantic track in the sandbox area is recombined with the repaired denoised semantic track to obtain the repaired audio.

2. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 1, characterized in that, The audio separation model employs a deep learning-based encoder-decoder network architecture, which includes an encoder, a mask generation module, and a decoder, wherein: The encoder is configured to convert the time-domain signal of the original audio to be repaired into a time-frequency domain signal through a short-time Fourier transform, and extract high-level acoustic features through multi-layer convolution or Transformer coding. The mask generation module is configured to generate a corresponding time-frequency mask for each semantic track based on the high-level acoustic features. Each time-frequency unit of the time-frequency mask takes a value between [0,1], which represents the probability that the time-frequency unit belongs to the corresponding semantic track. The decoder multiplies the time-frequency mask of each semantic track with the time-frequency features of the original audio to obtain the time-frequency representation of each semantic track, and then converts it back to the time domain signal through inverse short-time Fourier transform, which serves as the audio waveform of each semantic track.

3. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 2, characterized in that, Generate the time-frequency mask corresponding to the semantic track using any of the following methods: Mask i ( m , k )= σ ( W i · H ( m , k )+ b i ),in, Mask i ( m , k ) is the first i The first orbit m Frame, First k Time-frequency mask for each frequency point σ () is the Sigmoid activation function. H ( m , k ) represents the high-level acoustic features extracted by the encoder. W i , b i These are the learnable weights and biases corresponding to the i-th track, respectively; or, ,in, Mask i ( m , k ) is the first i The first orbit m Frame, First k Time-frequency mask for each frequency point N This represents the total number of semantic tracks. Indicates the first j The score function for each track; or, Mask i ( m , k )=Sigmoid( Attention i ( H ( m , k ))),in, Mask i ( m , k ) is the first i The first orbit m Frame, First k Time-frequency mask for each frequency point Attention i ( H ( m , k To focus on the first through attention mechanisms i The first orbit m Frame, First k The attention function for the time-frequency region of each frequency point, with Sigmoid() as the activation function.

4. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 1, characterized in that, The audio separation model is configured with an encoder-decoder model, a feature tagging module, a mapping module, and an audio track allocation module; wherein: The encoder-decoder model employs a convolutional neural network and a long short-term memory neural network structure. The encoder-decoder model is configured to: learn the temporal structure of music and parse the separated waveforms corresponding to multiple single sound sources in the target audio signal; identify the sound source attributes matched by the separated waveforms, where the sound source attributes include one or more of frequency, timbre, loudness, and pitch; and, based on the parsed separated waveforms of each single sound source and their matched sound source attributes, separate multiple audio elements from the target audio signal. The feature labeling module is configured to label the separated audio elements according to preset classification labels, which are multiple categories including human voice, piano, strings, guitar, bass, percussion, dialogue, motion effects, sound effects, and music. The mapping module is configured to pre-build the association mapping relationship between different tracks and different classification labels; The audio track allocation module is configured to assign the separated audio elements to tracks mapped to their labeled classification tags, thereby obtaining multiple semantic tracks.

5. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 1, characterized in that, The noise characteristics include the noise power spectrum; The semantic tracks are distinguished as clean or unclean semantic tracks using the following method: Multiplying the noise power spectrum by the time-frequency mask of the semantic track yields the noise power spectrum estimate corresponding to the semantic track; The noise power corresponding to each frequency point is summed or weighted averaged to obtain the overall noise power of the semantic track. The signal-to-noise ratio of this semantic track is calculated using the following formula: , where P noise P represents the overall noise power of this semantic track. signal This indicates the power of the semantic track itself; If the signal-to-noise ratio of the semantic track is higher than the preset signal-to-noise ratio threshold, the semantic track is determined to be a clean semantic track; otherwise, the semantic track is determined to be a non-clean semantic track.

6. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 5, characterized in that, The power P of the semantic track itself is calculated in the following way. signal : If the overall noise power P noise To calculate the power of the semantic track itself using the summation of noise power, the following formula is used: ,in, X ( m , k This represents the time-frequency representation obtained by performing a short-time Fourier transform on the semantic orbit. m For frame index, k Frequency index; If the overall noise power P noise The power of the semantic track itself is calculated using the following formula, based on the noise power weighted average: ,in, X ( m , k This represents the time-frequency representation obtained by performing a short-time Fourier transform on the semantic orbit. m For frame index, k For frequency point index, M Total number of frames K This represents the total number of frequency points.

7. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 1, characterized in that, The noise analysis of the residual trajectory is performed as follows: The noise power estimate is updated using a soft-decision method based on the probability of speech presence, and the update formula is as follows: ,in, Indicates the current number m Frame, First k Noise power estimation at each frequency point m For frame index, k For frequency point index, Indicates the previous frame, the first frame k Noise power estimation at each frequency point Indicates the current number of residual trajectories. m Frame, First k The power spectrum of a noisy signal at each frequency point; Indicates by the first m Frame, First k Each frequency point of speech has a probability-determined adaptive smoothing factor, which is within a preset lower limit value α. min The adaptive smoothing factor is closer to 1 as the probability of speech is higher, and the linear interpolation between 1 and 1 is used.

8. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 7, characterized in that, The first m Frame, First k The probability of voice at a specific frequency point It is calculated in the following way: Calculate the time-frequency representation of the current frame noisy signal of the residual orbit. Y ( m , k ); Calculate the posterior signal-to-noise ratio ; Calculate the prior signal-to-noise ratio Where α is a constant smoothing factor, This represents the historical state of the signal amplitude spectrum of the noise-reduced output in the previous frame. Indicates the posterior signal-to-noise ratio of the current frame. max [ ] indicates a function that takes the maximum value; and The first is calculated using the likelihood ratio or Sigmoid function mapping. m Frame, First k The probability of voice at a specific frequency point ; The adaptive smoothing factor is then calculated using the following formula. .

9. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 1, characterized in that, The noise analysis of the residual trajectory is performed as follows: The residual track is framed into audio signals; The time-frequency representation is obtained by performing a short-time Fourier transform on each frame. Y ( m , k ),in, m For frame index, k Frequency index; For each frequency point, the minimum power is tracked on the time axis characterized by the frame index, and used as the noise power estimate for that frequency point: ,in, Indicates the current number m Frame, First k Noise power estimation at each frequency point Indicates the previous frame, the first frame k Noise power estimation at each frequency point It is an increase factor greater than 1, and its set value is related to the allowable increase in noise estimation.

10. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 1, characterized in that, Based on different audio scenarios, the original audio is separated into corresponding semantic tracks: For music audio scenarios, the original audio is separated into multiple semantic tracks, including vocal track, piano track, string track, guitar track, bass track, and percussion track; For film and television audio scenarios, the original audio is separated into multiple semantic tracks, including dialogue track, motion effect track, sound effects track, and music track; And performing corresponding specific noise reduction and repair processing on the aforementioned unclean semantic tracks includes: For the unclean semantic tracks of the human voice track or the dialogue track, the first noise reduction algorithm is used for noise reduction and repair. For the unclean semantic tracks of the piano track, guitar track, or foley track, a second noise reduction algorithm is used for noise reduction and repair. For the unclean semantic track of the string track, a third noise reduction algorithm is used for noise reduction and repair; For the non-clean semantic tracks of the percussion track or motion effect track, the fourth noise reduction algorithm is used for noise reduction and repair. For the non-clean semantic tracks of the bass track, the fifth noise reduction algorithm is used for noise reduction and repair. For the non-clean semantic tracks of the music track, an adaptive noise reduction strategy is used to dynamically select one or more combinations of the first noise reduction algorithm, the second noise reduction algorithm, the third noise reduction algorithm, the fourth noise reduction algorithm, and the fifth noise reduction algorithm for noise reduction and repair.

11. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 10, characterized in that, The first noise reduction algorithm is a speech enhancement model based on deep learning, which is used to suppress noise while protecting the consonants and aerophones of speech; The second noise reduction algorithm is a transient protection noise reduction algorithm, which is used to detect transient regions in the signal and reduce the noise reduction intensity in the transient regions; The third noise reduction algorithm is a continuous tone protection noise reduction algorithm, which is used to protect continuous harmonic components and vibrato modulation during noise reduction. The fourth noise reduction algorithm is a pulse protection noise reduction algorithm, which is used to distinguish between real pulses and noise pulses, and then only attenuates the noise pulses while retaining the real pulses. The fifth noise reduction algorithm is a low-frequency dedicated noise reduction algorithm, which is used to attenuate low-frequency noise while protecting the low-frequency fundamental frequency and low-order harmonics of the bass track.

12. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 10, characterized in that, The noise reduction parameters of the first, second, third, fourth, and fifth noise reduction algorithms are set with reference to the noise characteristics corresponding to the residual track. The noise reduction parameters include noise reduction intensity. The higher the noise power spectrum in the noise characteristics, the stronger the noise reduction intensity is set.

13. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 1, characterized in that, The step of recombining the clean semantic track with the denoised semantic track includes: The denoised semantic tracks after repair are phase-aligned with the protected clean semantic tracks in the sandbox area; The phase-aligned denoised semantic track is added to the clean semantic track to obtain the initial resynthesized signal; The spectral envelope difference between the preliminary resynthesized signal and the original audio to be repaired is calculated to identify the spectral hole bands of the preliminary resynthesized signal; Energy compensation is applied to the frequency band of the spectral holes in the preliminary reconstituted signal to obtain the restored audio.

14. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 13, characterized in that, The difference in spectral envelope between the preliminary reconstituted signal and the original audio to be repaired is calculated using the following method: Short-time Fourier transforms are performed on the preliminary reconstituted signal and the original audio to be repaired, respectively, to obtain the time-frequency representation of the preliminary reconstituted signal. Y recon ( m , k ) and the time-frequency representation of the original audio to be repaired. Y orig ( m , k ),in, m For frame index, k Frequency index; The average power spectrum of each frequency point is obtained by averaging the frame indexes. ； in, P recon ( k The ) represents the average power spectrum of the initially reconstituted signal. P orig ( k () represents the average power spectrum of the original audio to be restored. M Total number of frames; The average power spectrum is smoothed to obtain the spectral envelope: , Where, smooth[ ] represents the smoothing operation function; Calculate the difference in spectral envelope: ,or, ; If the spectral envelope difference D ( k )or D dB ( k If the frequency band exceeds the preset hole threshold, it is determined to be a spectral hole band, and the restored audio after energy compensation is calculated using the following formula. ,or, ,in, β 1( k ), β 2( k ) represents a compensation coefficient between 0 and 1.

15. The audio restoration method based on multi-track semantic separation and residual noise allocation according to claim 14, characterized in that, The compensation coefficient is determined using the following adaptive strategy: β ( k )= ； in, D For spectral envelope differences D ( k )or D dB ( k ), D low The hole threshold, D high The threshold for significant holes, β min For conservative compensation constant coefficients, β max To enhance the compensation constant coefficient, β max > β min ; or, β ( k )=min[1, ] ,in, D For spectral envelope differences D ( k )or D dB ( k ), D max This is the preset maximum allowable difference value. β base The reference compensation constant coefficient.

16. A sound system, characterized in that, The device includes a speaker, an audio processor, and a memory, wherein the memory is used to store program instructions, the audio processor is configured to run the program instructions to perform the steps of the method as described in any one of claims 1 to 15, and output the processed repaired audio to the speaker.

Images