Sea voice missing repairing method combining flow matching and auxiliary repairing module

Abstract

The invention provides an offshore voice missing repair method combining stream matching and an auxiliary repair module, and the method comprises the following steps: S1, processing a damaged offshore audio signal, and generating a damaged Mel spectrum; s2, inputting the damaged Mel spectrum into a flow matching repair model; the flow matching repair model is based on a diffusion converter architecture and is trained through a conditional flow matching loss function to output a flow matching repair Mel spectrum; s3, converting the stream matching repair Mel spectrum into a preliminarily repaired offshore audio signal by using a vocoder; s4, adding the preliminarily repaired offshore audio signal and the damaged offshore audio signal, and inputting the added signals into an auxiliary repairing module; and the auxiliary repair module is based on an encoder-decoder structure, and the auxiliary repair module executes short-time Fourier transform operation on the damaged signal after preliminary repair, extracts an STFT amplitude spectrum from the damaged signal, and outputs a finally repaired offshore audio signal. According to the invention, key guarantee can be provided for the stability, transmission efficiency and reliability of maritime communication.

Description

Technical Field

[0001] This invention relates to the field of audio restoration technology, and more particularly to a method for restoring missing voice at sea that combines stream matching and auxiliary restoration modules. Background Technology

[0002] Audio segment restoration technology aims to reconstruct incomplete speech signals due to transmission or environmental interference. By mining contextual information, it generates semantically coherent complete content, thereby improving the fluency and intelligibility of the speech. This technology has significant value in many fields, especially in the complex environment of maritime VHF voice communication, where signals are often partially lost due to factors such as wind, waves, vibration, and electromagnetic radiation, leading to the loss of critical commands and directly threatening navigational safety. Therefore, developing efficient speech restoration methods is crucial for ensuring clear, stable, and reliable maritime communication.

[0003] Existing speech restoration methods can be mainly divided into traditional methods and deep learning-based methods. Traditional methods, such as linear prediction and spectral interpolation, are computationally simple and have low latency, but they struggle to effectively capture high-order temporal dependencies in speech signals when faced with complex and varied contexts, resulting in limited reconstruction quality. With the development of deep learning, data-driven methods have made significant progress. Among them, autoregressive models, while possessing strong semantic modeling capabilities, are prone to introducing errors during speech feature quantization due to their frame-by-frame generation mechanism, producing artificial artifacts and affecting the naturalness of the speech. Diffusion probability models can generate highly natural speech, but their multi-step iterative denoising inference process is computationally intensive and time-consuming, making it difficult to meet the low-latency requirements of real-time communication.

[0004] In recent years, flow matching, as an emerging generative model, has emerged by establishing a deterministic mapping from noise to target data, enabling the generation of high-fidelity speech in just one or a few steps, thus achieving a good balance between quality and efficiency. However, this method still has significant shortcomings in practical applications: First, the model may over-rely on the learned global speech prior distribution during the generation process, resulting in "illusionary" content that does not match the original input context; second, its restoration process does not adequately utilize local residual information, leading to a lack of smoothness and consistency in the restoration results, limiting its practical application in scenarios with stringent reliability requirements. Therefore, how to effectively suppress "illusions" and improve the accuracy and smoothness of restoration results while maintaining the high-speed inference advantage of flow matching has become a pressing technical challenge in this field. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a method for repairing missing speech at sea that combines stream matching and auxiliary repair modules, so as to solve the technical problem that the repair results of existing methods are insufficient in accuracy and smoothness.

[0006] The technical means employed in this invention are as follows: A method for restoring missing speech at sea, combining stream matching and auxiliary repair modules, includes the following steps: S1. Process the damaged maritime audio signal to generate a damaged Mel spectrum; S2. Input the damaged Mel spectrum into the flow matching repair model; the flow matching repair model is based on the diffusion transformer architecture and is trained by the conditional flow matching loss function to output the flow matching repaired Mel spectrum; S3. Use a vocoder to convert the stream-matched repaired Mel spectrum into a preliminarily repaired marine audio signal; S4. The initially repaired marine audio signal is added to the damaged marine audio signal and then input together into the auxiliary repair module. The auxiliary repair module is based on an encoder-decoder structure. The auxiliary repair module performs a short-time Fourier transform operation on the initially repaired damaged signal to extract the STFT amplitude spectrum. Then, the frequency filter bank is used to scale the STFT amplitude spectrum and output the finally repaired marine audio signal.

[0007] Furthermore, S1 specifically includes the following steps: A short-time Fourier transform (STFT) operation is performed on the damaged maritime audio signal to extract the STFT amplitude spectrum. A frequency filter bank is then used to scale the STFT amplitude spectrum to map it to the Mel scale space, thus obtaining the Mel spectrum corresponding to the damaged signal.

[0008] Furthermore, S2 specifically includes the following steps: The input to the stream matching model is the Mel spectrogram of the missing speech, its mixed spectrogram with Gaussian noise, and optional text encoding features; The flow matching model is built on the Diffusion Transformer architecture, and the flow matching model models the global dependencies of the input spectral blocks through a self-attention mechanism; The flow matching model uses the input as a joint condition to perform a probabilistic flow prediction task and outputs a flow matching repaired Mel spectrum.

[0009] Furthermore, the construction and training steps of the flow matching model are as follows: Gradually adding noise to real data Convert to Gaussian noise , No. Step state satisfy:

[0010] in, , , For the preset first Step noise coefficient, It is standard Gaussian noise; Modeling the inverse: from Predicting the original noise during noise addition Then, iterative denoising is obtained. Constructed using stacked Transformer blocks By capturing global dependencies of image patches through self-attention, the local receptive field of convolution is replaced to improve modeling capabilities; noise prediction network Meaning:

[0011] in, For network parameters, Will The image is segmented into patches and mapped to feature vectors. For time sequence Location encoding; The training objective is to minimize the MSE loss between the predicted noise and the actual noise. ; The flow matching framework uses a DiT architecture to model a continuous flow mapping from noise distribution to data distribution, and the evolution of the flow is determined by a time-dependent vector field. The driving force satisfies the ordinary differential equation:

[0012]

[0013] By capturing spatial-temporal information through image patch embedding and temporal coding, a vector field driving the flow evolution is output.

[0014] Optimize parameters by matching the target flow with the model prediction flow. Target Flow Defined as the derivative of the data and noise interpolation, the loss function is: .

[0015] Furthermore, in S3: The vocoder is a BigVGAN vocoder, which employs a fully convolutional architecture, including multiple upsampling blocks and residual dilated convolutional layers. It uses a multi-period discriminator to capture the periodic structure of the waveform from multiple periodic perspectives. The training objective of the vocoder is to minimize the difference between the generated waveform and the real waveform, and the loss function is as follows: The least-squares adversarial loss forces the generator G and discriminator D to engage in an adversarial game, making the generated waveform approximate the real data in terms of distribution.

[0016]

[0017] in, The latent variables representing the generator input, Representing authentic audio, Representing the L1 norm, the Mel spectrum L1 loss is used to force the generation of the Mel spectrum of the waveform. Mel spectrum of real audio Alignment; .

[0018] Furthermore, the auxiliary repair module is an encoder-decoder structure. The input signal of the auxiliary repair module is converted into a spectrum and then downsampled. After feature extraction via a two-dimensional convolution and residual network, it is upsampled. The loss function is MSE loss, which predicts the true value. Compared with the predicted value The error is calculated using the following formula: .

[0019] The present invention also provides a storage medium comprising a stored program, wherein, when the program is executed, it performs any of the above-described methods for repairing missing voice data at sea in combination with the stream matching and auxiliary repair modules.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described method for repairing lost speech at sea by means of the combination of a stream matching and auxiliary repair module.

[0021] Compared with the prior art, the present invention has the following advantages: This invention addresses the challenges of complex phase processing and high task difficulty in damaged audio restoration. It employs the Mel spectrum of maritime audio signals as the core input and output features, avoiding the need to repair phase information of missing signal segments and significantly simplifying the overall complexity of the audio restoration task. A novel generative speech restoration framework combining stream matching restoration and auxiliary restoration modules is adopted, ensuring high-quality speech reconstruction while maintaining real-time performance and higher restoration efficiency. An auxiliary restoration mechanism is introduced to effectively reduce the "illusion" risk in stream matching generation, enhancing feature consistency and reconstruction stability. A dual-mode inference strategy is designed to support flexible application in multiple scenarios. System experiments verify the superiority of the proposed method in terms of speech quality and intelligibility. This research provides an efficient and practical solution for generative speech restoration, improving the subjective perceived quality and objective intelligibility of the final restored signal, and has significant theoretical and practical value for ensuring smooth and stable reception of maritime communications. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the method of the present invention.

[0024] Figure 2 This is a diagram showing the audio restoration effect of the present invention with a 10% missing ratio.

[0025] Figure 3 This is a diagram showing the audio restoration effect of the present invention with a 20% missing ratio.

[0026] Figure 4 This is a diagram showing the audio restoration effect of the present invention with a 30% missing ratio.

[0027] Figure 5 This is a diagram showing the audio restoration effect of the present invention with a 40% missing ratio.

[0028] Figure 6 This is a diagram illustrating the restoration effect of the present invention on real-world maritime communication audio with missing audio data. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] like Figure 1 As shown, this invention provides a method for repairing missing speech segments at sea, combining stream matching and an auxiliary repair module. This effectively addresses the "illusion" problem where content generated by generative models appears reasonable and fluent but does not conform to objective facts, input information, or real-world patterns. The method repairs missing speech segments based on the Mel spectrum of the input audio and combines a stream matching model with an auxiliary repair module, including a high-fidelity vocoder and selectable input text. This achieves natural and fluent repair of missing speech segments and improves the "illusion" problem generated by generative models. The method includes the following steps: S1. Perform a short-time Fourier transform on the damaged signal to extract the STFT amplitude spectrum. Then, use a frequency filter bank to perform a scale transformation on the amplitude spectrum and map it to the Mel scale space to finally obtain the Mel spectrum corresponding to the damaged signal. Specifically, firstly, a Short-Time Fourier Transform (STFT) is performed on the input speech segment using a window length of 1024 points and a step size of 256 points. The STFT amplitude spectrum is then calculated using the Hanning window function. Next, a filter bank containing 100 channels is constructed, covering a frequency range matching the target sampling rate of 24000Hz. This filter bank is then used to convert the STFT amplitude spectrum to a Mel frequency scale to generate a Mel spectrogram. Finally, the resulting Mel spectrogram is logarithmically normalized to form a data format that meets the model input requirements.

[0032] S2. Input the damaged Mel spectrogram and the selected text encoding features into the diffusion transformer. After the diffusion model gradually denoises, predict the probability flow of the model. This probability flow is trained by gradient update to reconstruct a clean speech Mel spectrogram. Specifically, this invention employs a two-stage training approach. The first stage, stream matching, aims to learn how to progressively recover a complete Mel spectrogram from a Mel spectrogram of missing speech. The model inputs include the Mel spectrogram of the missing speech, the Mel spectrogram of the missing speech mixed with Gaussian noise, and selectively transcribed text. The text is converted into text embeddings by a text encoder, and these text embeddings are concatenated with the audio embeddings as additional conditional inputs to the model. DiT (DiffusionTransformer) is an architecture combining a diffusion model and a Transformer to model the data distribution. The stream model predicts a probability stream describing how to progressively repair the missing speech spectrum to approximate the complete spectrum. The training objective is to minimize the difference between the predicted stream and the true stream. The stream model is optimized through backpropagation gradient updates to more accurately predict the repair path. In the inference stage, the input speech is progressively repaired, generating a repaired Mel spectrogram, which is ultimately converted into a waveform. The input is the missing speech Mel spectrogram and Gaussian noise, used to guide the repair process. The stream model predicts the repair path. If text conditions are used, text encoding also affects prediction. The predicted stream is fed into an ordinary differential equation solver. Numerical integration is used to solve the repair trajectory. The solver gradually adjusts the missing speech spectrum to approximate the Mel spectrogram of the complete speech. The final output is the repaired Mel spectrogram. A pre-trained vocoder is used to convert the repaired Mel spectrogram into a waveform, yielding the clear speech.

[0033] The diffusion model uses progressively added noise to obscure the true data. Convert to Gaussian noise , No. Step state satisfy: (1) in ( , For the preset first (step noise coefficient) It is standard Gaussian noise.

[0034] The core task of DiT is to model the inverse process: from Predicting the original noise during noise addition Then, iterative denoising is obtained. Constructed using stacked Transformer blocks. Its noise prediction network improves modeling capabilities by capturing global dependencies of image patches through self-attention, replacing the local receptive field of convolution. Meaning: (2) in, For network parameters, Will The image is segmented into patches and mapped to feature vectors. For time sequence Location encoding.

[0035] The training objective is to minimize the MSE loss between the predicted noise and the actual noise. (3) The core of the flow matching framework is to model a continuous flow mapping from noise distribution to data distribution using a DiT architecture. The evolution of the flow is determined by a time-dependent vector field. Driven by, satisfying ordinary differential equations (ODEs). (4) (5) Spatial-temporal information is captured through image patch embedding and temporal encoding (PosEmbed), and the output vector field drives the evolution of the stream.

[0036] Optimize parameters by matching the "target flow" with the model's predicted flow. Target Flow Defined as the derivative of the data and noise interpolation, the loss function is: (6) S3. The predicted probability flow guides the ordinary differential equation solver to generate the repaired Mel spectrum, and then the high-fidelity BigVGAN vocoder is used to restore the Mel spectrum, and the output stream matches the repaired speech signal. Specifically, BigVGAN is a high-efficiency vocoder based on Generative Adversarial Networks (GANs). Its core design goal is to generate high-fidelity audio waveforms from Mel spectrograms. The generator is an improvement on HiFi-GAN, employing a fully convolutional architecture with multiple upsampling blocks and residual dilated convolutional layers. A multi-period discriminator captures the periodic structure of the waveform from multiple periodic perspectives, enhancing its ability to model complex audio. The training objective of BigVGAN is to minimize the difference between the generated waveform and the real waveform. The main loss function is as follows: Least squares adversarial loss forces the generator (G) and discriminator (D) into an adversarial game, making the generated waveform approximate the real data in terms of distribution: (7) (8) The Mel spectrum of the generated waveform is aligned with the Mel spectrum of the real audio by using Mel spectrum L1 loss to force alignment. (9) S4. The stream-matched repaired speech signal and the original damaged speech signal are added together and input into the encoder-decoder auxiliary repair module, which outputs the final repaired speech signal. A short-time Fourier transform is performed on the damaged signal to extract the STFT amplitude spectrum. Then, a frequency filter bank is used to scale the amplitude spectrum to output the final repaired speech signal.

[0037] Specifically, the initially repaired speech is input into the auxiliary repair module to reduce the "illusion" problem of generative models. The auxiliary repair module consists of a classic encoder-decoder structure. The input signal is converted into a spectrum and then downsampled. After feature extraction through a two-dimensional convolution and residual network, it is upsampled. The loss function is MSE loss, and the error between the predicted and actual values ​​is used. (10) Regarding the dataset, the training set consisted of missing speech pairs and their corresponding clean speech pairs, with audio length uniformly truncated to 10 seconds. The initial learning rate was 7.5 × 10⁻⁶, and training data was loaded in batches of 10. Data was pinned to memory, and samples from the last insufficient batch were discarded. The validation set consisted of independent missing and complete speech pairs, loaded in batches of 1. During training, a total of 120 epochs were performed, with a model checkpoint saved at the end of each epoch. The gradient pruning threshold was set to 3.0 to ensure training stability. Experimental results and model weights were stored in a designated experimental directory, and no breakpoint resumption was enabled during training.

[0038] Figure 2-5 These are audio restoration effect diagrams based on the missing percentage of the present invention. From left to right, each diagram represents the lost spectrogram, the restored spectrogram, and the ground truth spectrogram. It can be seen that the method of the present invention demonstrates effective restoration capabilities under different missing percentages. The generated spectrograms are smooth and coherent, effectively restoring the missing information in the original spectrogram.

[0039] Figure 6 This is a diagram showing the restoration effect of the present invention on real-world maritime communication audio with missing segments. From left to right, the diagrams are the lost spectrogram, the restored spectrogram, and the ground truth (complete) spectrogram. It can be seen that the method of the present invention also demonstrates effective restoration capabilities in real-world maritime communication voice scenarios, effectively restoring the missing information in the original spectrogram.

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for restoring missing speech at sea by combining stream matching and auxiliary repair modules, characterized in that, Includes the following steps: S1. Process the damaged maritime audio signal to generate a damaged Mel spectrum; S2. Input the damaged Mel spectrum into the flow matching repair model; the flow matching repair model is based on the diffusion transformer architecture and is trained by the conditional flow matching loss function to output the flow matching repaired Mel spectrum; S3. Use a vocoder to convert the stream-matched repaired Mel spectrum into a preliminarily repaired marine audio signal; S4. Add the initially repaired marine audio signal to the damaged marine audio signal and input them together into the auxiliary repair module; The auxiliary repair module is based on an encoder-decoder structure. The auxiliary repair module performs a short-time Fourier transform operation on the damaged signal after preliminary repair, extracts the STFT amplitude spectrum, and then uses a frequency filter bank to perform scale transformation on the STFT amplitude spectrum to output the finally repaired marine audio signal.

2. The maritime speech loss repair method combining stream matching and auxiliary repair modules according to claim 1, characterized in that, S1 specifically includes the following steps: A short-time Fourier transform (STFT) operation is performed on the damaged maritime audio signal to extract the STFT amplitude spectrum. A frequency filter bank is then used to scale the STFT amplitude spectrum to map it to the Mel scale space, thus obtaining the Mel spectrum corresponding to the damaged signal.

3. The maritime speech loss repair method combining stream matching and auxiliary repair modules according to claim 1, characterized in that, S2 specifically includes the following steps: The input to the stream matching model is the Mel spectrogram of the missing speech, its mixed spectrogram with Gaussian noise, and optional text encoding features; The flow matching model is built on the Diffusion Transformer architecture, and the flow matching model models the global dependencies of the input spectral blocks through a self-attention mechanism; The flow matching model uses the input as a joint condition to perform a probabilistic flow prediction task and outputs a flow matching repaired Mel spectrum.

4. The maritime speech loss repair method combining stream matching and auxiliary repair modules according to claim 1, characterized in that, The construction and training steps of the flow matching model are as follows: Gradually adding noise to real data Convert to Gaussian noise , No. Step state satisfy: in, , , For the preset first Step noise coefficient, It is standard Gaussian noise; Modeling the inverse: from Predicting the original noise during noise addition Then, iterative denoising is obtained. Constructed using stacked Transformer blocks By capturing global dependencies of image patches through self-attention, the local receptive field of convolution is replaced to improve modeling capabilities; noise prediction network Meaning: in, For network parameters, Will The image is segmented into patches and mapped to feature vectors. For time sequence Location encoding; The training objective is to minimize the MSE loss between the predicted noise and the actual noise. ； The flow matching framework uses a DiT architecture to model a continuous flow mapping from noise distribution to data distribution, and the evolution of the flow is determined by a time-dependent vector field. The driving force satisfies the ordinary differential equation: By capturing spatial-temporal information through image patch embedding and temporal coding, a vector field driving the flow evolution is output. Optimize parameters by matching the target flow with the model prediction flow. Target Flow Defined as the derivative of the data and noise interpolation, the loss function is: 。 5. The maritime speech loss repair method combining stream matching and auxiliary repair modules according to claim 1, characterized in that, In S3: The vocoder is a BigVGAN vocoder, which employs a fully convolutional architecture, including multiple upsampling blocks and residual dilated convolutional layers. It uses a multi-period discriminator to capture the periodic structure of the waveform from multiple periodic perspectives. The training objective of the vocoder is to minimize the difference between the generated waveform and the real waveform, and the loss function is as follows: The least-squares adversarial loss forces the generator G and discriminator D to engage in an adversarial game, making the generated waveform approximate the real data in terms of distribution. in, The latent variables representing the generator input, Representing authentic audio, Representing the L1 norm, the Mel spectrum L1 loss is used to force the generation of the Mel spectrum of the waveform. Mel spectrum of real audio Alignment; 。 6. The maritime speech loss repair method combining stream matching and auxiliary repair modules according to claim 1, characterized in that, The auxiliary repair module is an encoder-decoder structure. The input signal is converted into a spectrum and then downsampled. After feature extraction via a two-dimensional convolution and residual network, it is upsampled. The loss function is MSE loss, used to predict the true value. Compared with the predicted value The error is calculated using the following formula: 。 7. A storage medium, characterized in that, The storage medium includes a stored program, wherein when the program is executed, it performs the maritime voice loss repair method combining a stream matching and auxiliary repair module as described in any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the maritime speech loss repair method according to any one of claims 1 to 6, which combines stream matching and auxiliary repair modules, through the computer program.

Images