Speech enhancement method and device based on stream matching, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a speech enhancement method, device and equipment based on stream matching and a medium, which can be applied to the financial and medical fields, and the method comprises the following steps: obtaining noise speech data, and carrying out time-frequency conversion processing and standardization processing to obtain standardized time-frequency representation data; performing stream matching input construction processing based on the standardized time-frequency representation data, and inputting the data to a stream matching model for processing to obtain enhanced time-frequency representation data; performing anti-standardization processing on the enhanced time-frequency representation data to obtain anti-standardized time-frequency representation data; and performing time domain reconstruction processing on the destandardized time-frequency representation data to obtain enhanced voice waveform data. In the invention, aiming at the ubiquitous problems of poor fidelity and large reasoning time delay caused by discrete quantization in the existing speech enhancement method, enhanced speech waveform data can be obtained by introducing a stream matching modeling and time frequency-time domain consistent processing chain, so that the speech detail fidelity is improved and the time delay is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a speech enhancement method and device based on flow matching, equipment and medium. BACKGROUND

[0002] The existing speech enhancement technology route mainly includes three categories: first, the discrete modeling method based on the language model (LM), which maps the speech into discrete labels and then models, but the discrete quantization process inevitably introduces quantization loss, causing artifacts and audio quality degradation, and thus affecting the speaker similarity and intelligibility; second, the method based on the diffusion model (such as CDiffuSE, SGMSE, StoRM), which usually relies on a large number of step-by-step inversion iterations to complete the restoration from noise to data distribution, with many inference steps and long time consumption, which is difficult to meet the real-time scene requirements; third, the generative method based on VAE or GAN, which has improved perceptual quality, but the model structure is complex, the training stability is poor, and it is easy to have problems such as mode collapse or over-smoothed synthetic results. In summary, the existing technology has the problems of poor fidelity and long inference delay in speech representation, such as the application in the financial and medical fields. SUMMARY

[0003] The present application provides a speech enhancement method and device based on flow matching, equipment and medium to solve the technical problems of poor fidelity and long inference delay caused by discrete quantization in the existing speech enhancement method.

[0004] In a first aspect, a speech enhancement method based on flow matching is provided, comprising:

[0005] Obtaining noise speech data and performing time-frequency transformation processing to obtain noise time-frequency representation data;

[0006] Standardizing the noise time-frequency representation data to obtain standardized time-frequency representation data;

[0007] Performing flow matching input construction processing based on the standardized time-frequency representation data to obtain flow matching input representation data;

[0008] inputting the flow matching input representation data into a feature encoding submodule of a flow matching model to obtain flow matching feature representation data; performing context modeling processing on the flow matching feature representation data to obtain context enhanced feature representation data; inputting the context enhanced feature representation data into a velocity field estimation submodule to output velocity field estimation data; calculating a time-frequency state increment based on the velocity field estimation data and a preset numerical step length parameter to obtain time-frequency state increment data; performing numerical solution of an ordinary differential equation on an intermediate state time-frequency tensor in the flow matching input representation data using the time-frequency state increment data to obtain an updated intermediate state time-frequency tensor; inputting the updated intermediate state time-frequency tensor as input into the flow matching model for updating until a target path parameter is reached to obtain enhanced time-frequency representation data;

[0009] performing inverse normalization processing on the enhanced time-frequency representation data to obtain inverse normalized time-frequency representation data;

[0010] performing time domain reconstruction processing on the inverse normalized time-frequency representation data to obtain enhanced speech waveform data.

[0011] In a second aspect, a speech enhancement device based on flow matching is provided, comprising:

[0012] a noise acquisition module configured to acquire noise speech data and perform time-frequency transform processing to obtain noise time-frequency representation data;

[0013] a noise processing module configured to perform normalization processing on the noise time-frequency representation data to obtain normalized time-frequency representation data;

[0014] a data processing module configured to perform flow matching input construction processing based on the normalized time-frequency representation data to obtain flow matching input representation data;

[0015] a model optimization module configured to input the flow matching input representation data into a feature encoding submodule of a flow matching model to obtain flow matching feature representation data; perform context modeling processing on the flow matching feature representation data to obtain context enhanced feature representation data; input the context enhanced feature representation data into a velocity field estimation submodule to output velocity field estimation data; calculate a time-frequency state increment based on the velocity field estimation data and a preset numerical step length parameter to obtain time-frequency state increment data; perform numerical solution of an ordinary differential equation on an intermediate state time-frequency tensor in the flow matching input representation data using the time-frequency state increment data to obtain an updated intermediate state time-frequency tensor; input the updated intermediate state time-frequency tensor as input into the flow matching model for updating until a target path parameter is reached to obtain enhanced time-frequency representation data;

[0016] a data back-propagation module, configured to perform inverse normalization processing on the enhanced time-frequency representation data to obtain inverse normalized time-frequency representation data;

[0017] a reconstruction output module, configured to perform time-domain reconstruction processing on the inverse normalized time-frequency representation data to obtain enhanced speech waveform data.

[0018] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the speech enhancement method based on stream matching when executing the computer program.

[0019] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the speech enhancement method based on stream matching when executed by a processor.

[0020] In the speech enhancement method based on stream matching, noise speech data can be obtained and time-frequency transformation processing is performed to obtain noise time-frequency representation data. The noise time-frequency representation data is normalized to obtain normalized time-frequency representation data. Stream matching input construction processing is performed based on the normalized time-frequency representation data to obtain stream matching input representation data. The stream matching input representation data is input into a stream matching model to perform velocity field estimation and numerical solution of ordinary differential equation processing, respectively, to obtain enhanced time-frequency representation data. The enhanced time-frequency representation data is inversely normalized to obtain inverse normalized time-frequency representation data. Time-domain reconstruction processing is performed on the inverse normalized time-frequency representation data to obtain enhanced speech waveform data. In the present application, the problem of poor fidelity and long inference delay caused by discrete quantization in existing speech enhancement methods can be solved by introducing stream matching modeling and time-frequency-time domain consistent processing chain to obtain enhanced speech waveform data, thereby improving speech detail fidelity and reducing time delay. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is a flowchart of a speech enhancement method based on stream matching in an embodiment of the present application;

[0023] Figure 2 is Figure 1 a specific implementation of step S10 inFigure 1 ;

[0024] Figure 3 yes Figure 1 A detailed implementation flow diagram of step S20 Figure 2 ;

[0025] Figure 4 yes Figure 1 A detailed implementation flow diagram of step S30 Figure 3 ;

[0026] Figure 5 yes Figure 1 A detailed implementation flow diagram of step S40 Figure 4 ;

[0027] Figure 6 yes Figure 1 A detailed implementation flow diagram of step S50 Figure 5 ;

[0028] Figure 7 yes Figure 1 A detailed implementation flow diagram of step S60 Figure 6 ;

[0029] Figure 8 This is a schematic diagram of a speech enhancement device based on stream matching in one embodiment of the present invention;

[0030] Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0031] Figure 10 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Please see Figure 1 As shown, Figure 1 A flowchart illustrating a stream-matching-based speech enhancement method provided in this embodiment of the invention includes the following steps:

[0034] S10: Acquire noisy speech data and perform time-frequency transformation processing to obtain noise time-frequency representation data.

[0035] The audio input stream can be received at a medical or financial collection terminal, the file header and codec parameters are parsed to obtain noise speech data; the noise speech data is uniformly sampled and channel mapped (for example, medical speech is uniformly 16 kHz monaural, and financial counter recording can be uniformly 8 / 16 kHz monaural), and after time-frequency conversion processing, noise time-frequency representation data can be obtained, and the service domain identifier (such as medical / financial) is attached to facilitate subsequent selection of appropriate engineering parameters.

[0036] In combination Figure 2 As shown, the step S10 specifically includes the following steps:

[0037] S101: receiving an audio input stream and parsing file header information to obtain original noise speech data.

[0038] The continuous audio input stream is received from the collection terminal, and the container and file header information are parsed to obtain the original noise speech data. The parsed content includes: sampling rate, bit depth, channel number, encoding format (such as PCM / WAV, FLAC, AAC, Opus), time length, timestamp, and optional service domain identifier and session identifier (such as department / equipment identifier in a medical scene or agent / work order number in a financial scene).

[0039] S102: uniformly sampling and channel converting the original noise speech data respectively to obtain formatted noise speech data.

[0040] The sampling rate uniformization can adopt a band-limited interpolation or a polyphase filter resampling strategy, for example, medical scene uniformization to 16 kHz monaural, and financial scene uniformization to 8 kHz or 16 kHz monaural; the channel conversion supports selecting a main talk path from stereo or merging into a single channel according to a weight. After uniformization, the samples are normalized to a uniform numerical domain (such as floating point [-1, 1]), to obtain formatted noise speech data, to ensure the numerical consistency of subsequent time-frequency processing.

[0041] S103: de-biasing and pre-emphasis processing the formatted noise speech data respectively to obtain preprocessed noise speech data.

[0042] De-biasing is completed by calculating and deducting the mean value of the whole or segmented section, and pre-emphasis can adopt a first-order high-pass pre-emphasis filter, whose difference form is: the current sample minus the weighted value of the previous sample, and the weight coefficient a can be between 0.95-0.98 (for example, a=0.97), and finally the preprocessed noise speech data is obtained.

[0043] S104: frame processing the preprocessed noise speech data based on a preset window length and frame shift to obtain a noise speech frame sequence.

[0044] The window length and frame shift are determined according to the service domain and the identified link configuration, for example, the window length is 20-32 ms (commonly 25 ms), and the frame shift is 8-12 ms (commonly 10 ms). Zero padding is used for samples less than one frame at the end to ensure that the lengths of the frames are consistent, and a noise speech frame sequence arranged in time sequence is formed.

[0045] S105: A preset window function is applied to the noise speech frame sequence frame by frame to obtain a windowed noise speech frame sequence.

[0046] The window function can be selected from a Hamming window, a Hanning window or other equivalent smooth windows, and the window length is consistent with the frame length in step S104. In order to avoid energy mismatch, a window energy normalization strategy can also be configured in the implementation to maintain the same energy before and after windowing, and finally obtain the windowed noise speech frame sequence.

[0047] S106: The windowed noise speech frame sequence is subjected to short-time Fourier transform to obtain a complex spectrum matrix.

[0048] Each frame of the windowed noise speech frame sequence is subjected to N-point discrete Fourier transform (N is not less than the frame length, and is preferably a power of 2, such as 512 / 1024), and zero padding is used in the time domain or frequency domain as necessary to meet the frequency resolution requirement. The complex spectrum of each frame is stacked in the order of “time x frequency” to form a complex spectrum matrix, and the elements thereof include real and imaginary parts.

[0049] S107: The complex spectrum matrix is subjected to amplitude spectrum and phase spectrum calculation and combined in a preset time-frequency feature format to obtain noise time-frequency representation data.

[0050] The amplitude spectrum can be a complex modulus or a logarithmic form thereof, and the phase spectrum is obtained by calculating the complex argument; both are arranged in the order of frequency from low to high and time from early to late. The specific encoding format of the feature combination can include: a “amplitude-phase” two-channel tensor representation, or a split structure representation of “amplitude tensor + phase tensor”; in the medical and financial scenarios, the frequency band number, frequency point truncation range and quantization precision can be configured at the engineering level without changing the mathematical definition. The final output noise time-frequency representation data is directly input for subsequent standardization processing and flow matching input construction.

[0051] S20: The noise time-frequency representation data is subjected to standardization processing to obtain standardized time-frequency representation data.

[0052] The amplitude-related features are extracted from the noise time-frequency representation data, and the standardized time-frequency representation data can be obtained by standardization processing. In the medical and financial scenarios, independent statistical parameter caches can be maintained respectively to improve the cross-domain adaptability, but the standardization calculation process is not changed.

[0053] In combination Figure 3As shown, the step S20 specifically includes the following steps:

[0054] S201: Extracting amplitude spectrum matrix and phase spectrum matrix from the noise time-frequency representation data respectively, to obtain the to-be-normalized amplitude spectrum matrix and phase spectrum matrix.

[0055] The to-be-normalized amplitude spectrum matrix can be in the form of modulus of complex spectrum or square root of power spectrum; and the phase spectrum matrix is the argument of complex spectrum. For the convenience of subsequent statistics, the two-dimensional tensor storage is performed according to “time frame × frequency point (or mel band)”, and the business domain identifier (medical / financial) and the acquisition channel information are retained for selecting the subsequent statistical parameter source.

[0056] S202: Numerical stabilization processing is performed on the to-be-normalized amplitude spectrum matrix to obtain a stabilized amplitude spectrum matrix.

[0057] Preferably, a combination strategy of lower limit truncation and small constant addition is adopted: the amplitude value is clipped at a lower limit to avoid zero value, and a small constant is added before the calculation of logarithm or ratio to suppress numerical divergence, and finally the stabilized amplitude spectrum matrix is obtained. In the medical scenario, considering that the device noise floor and low-energy components such as respiratory sound are more common, the lower limit and small constant can be taken as a more conservative value than the financial scenario; in the financial scenario (such as 8 kHz voice audio), the lower limit can be correspondingly reduced to avoid excessive lifting of the noise floor.

[0058] S203: Dynamic range compression processing is performed based on the stabilized amplitude spectrum matrix to obtain a compressed amplitude spectrum matrix.

[0059] The compression strategy can be logarithmic amplitude mapping or power-law compression, and the compressed amplitude spectrum matrix is obtained after processing. Logarithmic amplitude mapping: performing logarithmic operation on the stabilized amplitude to compress the high-energy difference and improve the distinguishability of low-energy speech details; power-law compression: performing nonlinear mapping on the stabilized amplitude according to an index less than 1 to realize dynamic range compression similar to logarithm. In the medical scenario, in order to take care of low-energy details such as heart sound / respiratory sound, logarithmic amplitude mapping can be preferred; in the financial scenario, in order to adapt to the spectrum distribution dominated by voice, power-law compression or logarithmic mapping can be adopted and the maximum dynamic range is limited.

[0060] S204: According to a preset dimension, the mean and standard deviation parameters of the compressed amplitude spectrum matrix are obtained to obtain a normalized statistical parameter.

[0061] The compressed amplitude spectrum matrix is statistically averaged and standard deviation in the time dimension according to the preset dimension (for example, independent by frequency point or independent by frequency band) to obtain normalized statistical parameters. Statistics can adopt two strategies of sliding time window or whole section statistics: the first kind, medical scene: long-term statistics are preferably maintained with department / equipment granularity (device characteristics are stable), and sliding window statistics of the current session are supplemented to adapt to scene changes; the second kind, financial scene: long-term statistics are preferably maintained with seat / circuit granularity, and short window update is performed according to the call turn to adapt to the rate and speaker change. When the current session length is not enough to support stable statistics, the global prior statistics (obtained by pre-training) of the corresponding business domain are used as a fallback to ensure that the normalized parameters can be obtained.

[0062] S205: The compressed amplitude spectrum matrix is mean-variance normalized using the normalized statistical parameters to obtain a normalized amplitude spectrum matrix.

[0063] To avoid the amplification effect caused by extremely small variance, a lower threshold can be set for the standard deviation. The normalized matrix and the original matrix are consistent in the “time frame × frequency point (or mel band)” dimension, and the time index consistent with the phase spectrum matrix is retained, so that the normalized amplitude spectrum matrix can be obtained to ensure the consistency of the subsequent encoding combination. If necessary, the statistical parameters (mean, standard deviation) and the business domain identifier are cached for subsequent reverse normalization and cross-section consistency maintenance.

[0064] S206: The normalized amplitude spectrum matrix and the phase spectrum matrix are encoded and combined based on a preset time-frequency feature format to obtain normalized time-frequency representation data.

[0065] According to the preset time-frequency feature format, the normalized amplitude spectrum matrix and the phase spectrum matrix are encoded and combined to obtain normalized time-frequency representation data. The preferred encoding format includes:

[0066] Dual-channel superposition format: superimposed in the channel dimension to form a tensor with a size of “time frame × frequency point × 2”, wherein channel 0 is the normalized amplitude spectrum, and channel 1 is the phase spectrum;

[0067] Split type format: maintain “amplitude tensor” and “phase tensor” respectively, and record the alignment index of the two in the metadata;

[0068] Phase encoding extension: under the premise of keeping the phase information unchanged, the phase spectrum can be replaced by its sine / cosine pair according to the preset format to improve the numerical continuity.

[0069] In the medical scene, the dual-channel superposition format can be preferentially used to simplify the end-side reasoning deployment; in the financial scene, if it is necessary to be connected with the existing identification front end, the split type format can be used and splicing can be completed in the interface layer. The normalized time-frequency representation data finally output serves as the input of the input constructed by the subsequent stream matching.

[0070] S30: performing a stream matching input construction processing based on the standardized time-frequency representation data to obtain stream matching input representation data.

[0071] The stream matching input construction processing is performed on part of the data in the standardized time-frequency representation data to obtain the stream matching input representation data.

[0072] In combination Figure 4 As shown in the figure, the step S30 specifically comprises the following steps:

[0073] S301: extracting a feature tensor corresponding to a model input size from the standardized time-frequency representation data to obtain a standardized time-frequency tensor.

[0074] The feature tensor consistent with the stream matching model input size is extracted from the standardized time-frequency representation data to obtain the standardized time-frequency tensor. The input size is represented in a three-dimensional structure of "time frame number x frequency point number x channel number", wherein the channel dimension contains normalized amplitude and phase correlation coding. For samples that do not meet the target size, time dimension zero padding or center cropping and frequency dimension alignment strategy are used to complete the shaping; medical scenarios (such as 16 kHz sampling) can configure a longer time window, and financial scenarios (such as 8 / 16 kHz voice audio) can configure a more compact time window, but they are consistent with the model static input size.

[0075] S302: generating a random noise tensor with zero mean and unit variance based on the standardized time-frequency tensor to obtain a noise time-frequency initial tensor.

[0076] The random number generator sets a deterministic seed at the session level to ensure repeatability; the tensor elements follow an independent and identically distributed Gaussian distribution, and the tensor dimension is strictly consistent with the standardized time-frequency tensor without any scaling or offset. The noise time-frequency initial tensor is obtained after processing.

[0077] S303: sampling a path parameter within a preset interval and performing linear interpolation operation on the noise time-frequency initial tensor and the standardized time-frequency tensor to obtain an intermediate state time-frequency tensor.

[0078] The path parameter t is sampled within a preset interval (preferably [0, 1]); the path parameter can be sampled according to a uniform distribution or according to a preset density function (such as Beta family) to form different interpolation covers. Linear interpolation is performed on the noise time-frequency initial tensor Z and the standardized time-frequency tensor X to obtain the intermediate state time-frequency tensor Xt, wherein Xt=(1-t)·Z+t·X. To ensure numerical stability, the interpolation is completed in the floating point domain, and the same data type and memory layout as the input tensor are used.

[0079] The step S303 specifically comprises the following steps:

[0080] S3031: Analyze the start and end boundaries of the preset interval and configure the sampling strategy to obtain the path parameters.

[0081] The processing unit reads the start and end values of the path parameter space from the configuration storage, checks the data types and value ranges of the two values, confirms that the two values meet the sequential relationship and are located within the allowed interval. Then, according to the task configuration, determine the sampling strategy, which can be equal-interval sampling, segmented encryption sampling, or adaptive sampling based on historical distribution, and set a random seed in the case of repetitive scenarios to ensure the reproducibility of sampling. After completing the strategy initialization, the processing unit performs sampling in the preset interval composed of the start and end values to obtain at least one target path parameter; when only single interpolation is required, a single path parameter is directly output; when batch processing is required, a sequence of path parameters arranged in generation order is output, along with an index for subsequent steps.

[0082] S3032: Perform relative position conversion and range constraint on the target path parameters respectively to obtain the interpolation weights.

[0083] Receive the path parameters (or path parameter sequence) output by step S3031, and based on the relative position of the path parameters between the start and end boundaries, calculate the corresponding proportion coefficient; the proportion coefficient is used to describe whether the parameter is closer to the start value or the end value. To avoid out-of-range, the processing unit performs upper and lower limit constraint on the proportion coefficient, limiting it to the closed interval of zero to one; when the proportion coefficient is less than the lower limit, take the lower limit, and when it is greater than the upper limit, take the upper limit. In the batch scenario, each item in the sequence is converted and constrained one by one, and a one-to-one mapping relationship is established with the original path parameters to form interpolation weights, ensuring consistent tensor processing and synthesis according to the weights in the subsequent steps.

[0084] S3033: Based on the interpolation weights, perform size, channel, and step alignment processing on the noise time-frequency initial tensor and the standardized time-frequency tensor to obtain an aligned tensor set.

[0085] Read two types of tensors to be processed: noise time-frequency initial tensor and normalized time-frequency tensor, and check the dimension description (batch size, channel number, time step number and frequency point number) and whether the memory layout is consistent with the model input specification; if there is inconsistency, alignment is performed in the following order: first, size alignment, when the time or frequency dimension is inconsistent, the time and frequency grid consistent with the model is preferred, and the smaller side is filled with no information or boundary repetition to complete, and the larger side is truncated according to the center or head-tail cropping strategy; second, channel alignment, when the channel number is different, channel selection, channel replication or linear mapping is performed according to the model channel definition to ensure one-to-one correspondence of the two tensors in the channel dimension; third, step alignment, when the time step or frequency resolution is inconsistent, resampling is performed according to the reference step of the model input to align the two tensors on the same time and frequency grid. In the above alignment process, the interpolation weight is used to determine the priority order and bias selection in boundary cropping and filling to reduce the consistency influence on subsequent synthesis. After the above processing, an aligned tensor set composed of two sets of aligned tensors is output.

[0086] S3034: Linearly weighted synthesis of the aligned tensor set to obtain an intermediate state time-frequency tensor.

[0087] According to the mapping relationship of the interpolation weight, weighted addition is performed on the corresponding element positions of the two types of aligned tensors: the complementary part of the weight is used to measure the share of the noise time-frequency initial tensor, and the weight itself is used to measure the share of the normalized time-frequency tensor; in the batch scenario, the weight list is traversed item by item to generate a corresponding set of intermediate state results. To ensure numerical stability, the processing unit truncates outliers during synthesis, replaces missing data with zero or nearest neighbor, and performs a consistency check and data type normalization after synthesis is complete, finally outputting an intermediate state time-frequency tensor consistent with the model input size; when there are multiple path parameters, the intermediate state time-frequency tensors are arranged in order according to the path parameter index to be directly called by the subsequent time encoding module and velocity field estimation submodule.

[0088] S304: Frame block division and position index coding processing are performed on the intermediate state time-frequency tensor to obtain a position-coded intermediate state time-frequency tensor.

[0089] The frame block division is performed on the intermediate state time-frequency tensor along the time dimension and the frequency dimension to obtain a regular frame block grid; the frame block size and the step size are fixed in the model configuration (for example, the time dimension Bt x the frequency dimension Bf, the step size St, Sf), and zero padding is used to fill in the insufficient boundary area. Then, a two-dimensional position index is generated according to the starting index and the center index of the frame block in the global “time-frequency” plane. The above index is mapped into a position encoding vector (two-dimensional sine-cosine position encoding or equivalent learnable encoding can be used) through a position encoding function, and is aligned with the tensor segment of the corresponding frame block to obtain the intermediate state time-frequency tensor with position encoding. The medical and financial domains can use consistent encoding functions, and only differ in the configuration of the frame block size and the step size to adapt to different speech time-frequency resolutions.

[0090] S305: inputting the intermediate state time-frequency tensor with position encoding and the path parameter t into a time encoding module to output a condition vector.

[0091] The intermediate state time-frequency tensor with position encoding and the path parameter t are input into the time encoding module to output a condition vector. The time encoding module maps the path parameter t into a fixed-dimensional time embedding (for example, a time embedding realized by a sine-cosine time-frequency basis or a multi-layer perception), and aligns with the position encoding statistics (such as the normalized statistics of the frame block index) when needed, so that the condition vector is compatible with the subsequent network input in the channel dimension. The medical and financial domains can adjust the time embedding dimension and the frequency coverage range according to different real-time requirements, but do not change the functional relationship of “t→time embedding→condition vector”.

[0092] S306: performing feature splicing and channel mapping processing on the condition vector and the intermediate state time-frequency tensor with position encoding to obtain stream matching input representation data.

[0093] The condition vector and the intermediate state time-frequency tensor with position encoding are spliced in the channel dimension to form an extended feature tensor; then the extended feature tensor is mapped into the input channel number expected by the stream matching model through channel mapping processing to obtain the stream matching input representation data. The channel mapping can use point-by-point convolution (1x1 convolution) or equivalent linear mapping layer, which keeps the time and frequency dimensions unchanged and only adjusts the channel dimension. To ensure cross-domain consistency, the weights and normalization strategies of the channel mapping layer are uniformly fitted in the training stage, and the same atlas is executed in the inference stage, and the mathematical form does not change due to the switching of business domains.

[0094] S40: inputting the stream matching input representation data into the stream matching model to perform velocity field estimation and numerical solution processing of ordinary differential equations, respectively, to obtain enhanced time-frequency representation data.

[0095] The flow matching input representation data is input into the feature encoding and context modeling sub-module of the flow matching model (for example, using a time-frequency Transformer backbone) for processing to obtain enhanced time-frequency representation data. To meet the real-time processing needs of medical consultations and financial agents, the corresponding numerical step size and iteration round can be configured according to the end-side computing power.

[0096] In combination Figure 5 As shown, the step S40 specifically includes the following steps:

[0097] S401: input the flow matching input representation data into the feature encoding sub-module of the flow matching model to obtain flow matching feature representation data.

[0098] The generated flow matching input representation data (which contains the intermediate state time-frequency tensor M t , position, time encoding and path parameter t) is input into the feature encoding sub-module of the flow matching model. Preferably, a two-dimensional time-frequency encoding backbone based on DiT (Diffusion Transformer) is used:

[0099] First, the intermediate state time-frequency tensor is linearly projected (or 1x1 convolution) according to the frame block to obtain an initial embedding;

[0100] Second, the time encoding (generated by the path parameter t) and the two-dimensional position encoding (from the start and end indexes of the frame block) are added or spliced and fused with the initial embedding;

[0101] Third, the flow matching feature representation data is obtained via several Transformer blocks (multi-head self-attention + feedforward network + layer normalization + residual connection).

[0102] In the medical scenario, a larger time attention window can be configured to cover long sentences; in the financial scenario, the window can be tightened to reduce the end-side delay, but the encoding process remains unchanged.

[0103] S402: perform context modeling processing on the flow matching feature representation data to obtain context-enhanced feature representation data.

[0104] The context modeling preferably considers the correlation of the time axis and the frequency axis: the time axis self-attention aggregates the dependencies across frame speech units (syllables / words); the frequency axis self-attention aggregates the harmonic and formant structures; and the optional local convolution feedforward (Conv-FFN) enhances the neighborhood robustness. To reduce the differences in different recording environments, a domain embedding (a learnable vector identified by the medical / financial business domain) can be introduced in this stage to perform light adjustment through gating (such as FiLM / gated residual), but the operator form of the self-attention remains unchanged, and finally the context-enhanced feature representation data is obtained.

[0105] S403: Input the context-enhanced feature representation data into the velocity field estimation submodule and output the velocity field estimation data.

[0106] Input the context-enhanced feature representation data into the velocity field estimation submodule, and output the velocity field estimation data v. θ (M t ,t,C). Where: v θ The head network consists of several layers of 1×1 convolutional MLPs, and its output is related to M... t A time-frequency velocity field of the same shape; t is the path parameter; C is an optional condition input.

[0107] When the scenario provides textual conditions, a text encoder (such as a ConvNeXtV2 text branch or an equivalent semantic encoder) can be used to generate a semantic vector T(C). This vector is then fused with the feature backbone through cross-attention or channel modulation to enhance semantic consistency under extremely low signal-to-noise ratios. Technical terminology readings in medical scenarios and key information verification statements in financial scenarios can both serve as sources of C. If no textual conditions are provided, the branch is left empty without affecting the solution chain.

[0108] S404: Calculate the time-frequency state increment based on the velocity field estimation data and the preset numerical step size parameter to obtain the time-frequency state increment data.

[0109] The time-frequency state increment ΔM is calculated based on the velocity field estimation data and the preset numerical step size parameter h. t Finally, the time-frequency state increment data is obtained:

[0110] ΔM t =h·v θ (M t ,t,C)

[0111] h can adopt a fixed step size (such as 1 / K) or a segmented step size strategy (larger in the early stage and smaller in the later stage). To balance real-time performance and stability, K≈10–16 can be used in medical scenarios and K≈6–12 can be used in financial scenarios. When the input energy is detected to be too low or the speech rate changes abruptly, h can be pruned to control numerical oscillation.

[0112] S405: Using the time-frequency state increment data, numerically solve the ordinary differential equation for the intermediate state time-frequency tensor in the stream-matched input representation data to obtain the updated intermediate state time-frequency tensor.

[0113] The intermediate state time-frequency tensor is updated by performing ODE numerical solution using the time-frequency state increment, resulting in the updated intermediate state time-frequency tensor, as follows: Explicit Euler: M t+h =M t +ΔM tOr improve Euler Heyn: predict first, then correct to reduce truncation error. RK4 can also be used on the server side if the computing power permits, but Euler or Heun is preferred on mobile devices / edge devices. During the solving process, the time and frequency dimensions of the tensor remain unchanged, and only the channel values are updated; and the generated M t is consistent with the memory layout of the M

[0114] S406: input the updated intermediate state time-frequency tensor as input into the flow matching model until the target path parameter is reached to obtain enhanced time-frequency representation data.

[0115] The updated intermediate state time-frequency tensor is used as new input, and steps S401-S405 are repeated until the path parameter reaches the target endpoint (t→1), and enhanced time-frequency representation data is output. During training, the linear interpolation relationship is:

[0116] M t =(1-t)·M y +t·M x

[0117] Sample the intermediate state and minimize the velocity field supervision:

[0118]

[0119] During inference, K-step forward numerical integration is performed from the noise end (or initial value end) to the target end, and the typical step number is less than 20 steps. Medical and financial scenarios only differ in the engineering values of K and h, and whether to enable text condition C, without changing the unified processing flow of “feature encoding-context modeling-velocity field estimation-numerical integration-iterative convergence”.

[0120] S50: perform inverse normalization processing on the enhanced time-frequency representation data to obtain inverse normalized time-frequency representation data.

[0121] Inverse normalization processing is performed on the enhanced time-frequency representation data to obtain inverse normalized time-frequency representation data.

[0122] In combination with Figure 6 , the step S50 specifically comprises the following steps:

[0123] S501: extract enhanced normalized amplitude spectrum matrix and enhanced phase spectrum matrix from the enhanced time-frequency representation data, respectively.

[0124] The enhanced normalized amplitude spectrum matrix is a normalized amplitude channel, and the enhanced phase spectrum matrix is a phase channel corresponding to each time frame-frequency point of the enhanced normalized amplitude spectrum matrix. To ensure consistency of subsequent reverse normalization, the standardized metadata bound to the current session (including dimensions for calculating mean and standard deviation, sliding window length, and service domain identifier) are read and verified at the same time; if the session-level metadata is missing, the global prior metadata of the corresponding service domain (medical / financial) is used as a fallback.

[0125] S502: inverse transform calculation is performed on the enhanced normalized amplitude spectrum matrix to obtain an inverse normalized amplitude spectrum matrix.

[0126] Preferably, it is performed band by band (or point by point) to ensure consistency with the statistical dimensions of the normalization stage; the lower limit is truncated for the minimum standard deviation to avoid numerical amplification, and the inverse normalized amplitude spectrum matrix is obtained after processing. In the medical scenario, department / equipment-level long-term statistical parameters can be used to improve cross-period consistency; in the financial scenario, agent / circuit-level statistical parameters can be used to adapt to traffic characteristics.

[0127] S503: reverse compression processing is performed on the inverse normalized amplitude spectrum matrix according to the preset dynamic range compression configuration parameters to obtain a reverse compressed amplitude spectrum matrix.

[0128] If logarithmic amplitude compression is used, exponential mapping and bias rollback are performed; if power-law compression is used, inverse transformation of the corresponding exponent is performed, and the reverse compressed amplitude spectrum matrix is obtained after processing. To avoid overflow of high-energy points after inverse transformation, an upper limit is set for the output and the clipping ratio is recorded for quality monitoring. The medical scenario usually retains a wider low-energy resolution interval to cover details such as breathing and heart sounds; in the financial scenario, the maximum dynamic range can be limited under the condition of voice dominance to suppress line noise playback.

[0129] S504: numerical restoration processing is performed on the reverse compressed amplitude spectrum matrix to obtain a restored amplitude spectrum matrix.

[0130] The numerical restoration processing is performed on the reverse compressed amplitude spectrum matrix to obtain a restored amplitude spectrum matrix. This process includes: removing the small stable constant introduced in the normalization stage; zero threshold correction is performed on the negative values; unit consistency is performed according to the system agreed numerical domain (such as linear amplitude or power square root amplitude); band-limited smoothing is performed when needed to suppress isolated spikes introduced by inverse transformation. The medical end can enable light frequency band regularization to smooth the device noise floor, and the financial end can perform out-of-band suppression according to the standard of the telecommunication circuit.

[0131] S505: the restored amplitude spectrum matrix and the enhanced phase spectrum matrix are combined to obtain a candidate reverse standardized time-frequency representation data.

[0132] The reduced magnitude spectrum matrix and the enhanced phase spectrum matrix are combined to obtain candidate anti-standardized time-frequency representation data. The encoding format is consistent with the time-frequency feature format of step S20: that is, the amplitude tensor and the phase tensor are output respectively and the alignment index is retained in the metadata. In addition, without changing the phase value, the value continuity can be improved by storing in the form of (sinφ, cosφ) (whether to enable is determined according to the business domain strategy). The time frame and the frequency point are strictly aligned during the combination to ensure that the subsequent inverse short-time Fourier transform can be directly consumed.

[0133] S506: Dimension rearrangement is performed on the candidate anti-standardized time-frequency representation data to obtain anti-standardized time-frequency representation data.

[0134] The target of the dimension rearrangement is consistent with the input of the time domain reconstruction module (for example, the data is rearranged into a continuous memory layout of “frequency × time × channel” or “time × frequency × channel”), and the spectrum symmetry and the boundary frame required for reconstruction are supplemented: if the real number FFT is used in steps S10 and S20 and only the positive frequency is retained, the conjugate frequency band is reconstructed according to the convention at this time; the zero padding part of the last frame is explicitly labeled so that the window function energy compensation can be performed in the reconstruction stage. The medical scene usually outputs a data layout compatible with the 16kHz / 16bit reconstruction link; the financial scene can be aligned with the ISTFT implementation of the 8 / 16kHz traffic link. The final anti-standardized time-frequency representation data is used for complex synthesis and inverse short-time Fourier transform.

[0135] S60: Time domain reconstruction processing is performed on the anti-standardized time-frequency representation data to obtain enhanced speech waveform data.

[0136] The time domain reconstruction processing is performed on the anti-standardized time-frequency representation data to generate enhanced speech waveform data for archiving, quality inspection or downstream identification.

[0137] In combination Figure 7 As shown in the figure, the step S60 specifically includes the following steps:

[0138] S601: Anti-standardized magnitude spectrum matrix and anti-standardized phase spectrum matrix are extracted from the anti-standardized time-frequency representation data respectively.

[0139] In order to ensure consistency with the previous link, the meta information (such as the number of FFT points, the window function type, the frame shift, whether to save only the positive frequency, etc.) carried with the anti-standardized time-frequency representation data is extracted and verified. The medical domain preferentially uses the same mel / spectral configuration as the front end for 16kHz monaural data; the financial domain can be executed according to the 8 / 16kHz spectral configuration, and the anti-standardized magnitude spectrum matrix and the anti-standardized phase spectrum matrix are extracted.

[0140] S602: Complex synthesis processing is performed on the inverse normalized amplitude spectrum matrix and the inverse normalized phase spectrum matrix to obtain a reconstructed complex spectrum matrix.

[0141] The inverse normalized amplitude spectrum matrix and the inverse normalized phase spectrum matrix are complex synthesized to obtain a reconstructed complex spectrum matrix. Specifically, the amplitude and the phase can be mapped into a complex plane representation according to frames and frequency points, and the negative frequency band can be filled according to the conjugate symmetry principle in the case of only saving positive frequencies; the direct current and the Nyquist frequency point are processed as real numbers.

[0142] S603: Inverse short-time Fourier transform processing is performed on the reconstructed complex spectrum matrix to obtain a windowed time domain frame sequence.

[0143] The inverse short-time Fourier transform can use the FFT point number and the window function (such as the Hamming window / Hanning window), and the real number IFFT optimization consistent with the platform is enabled in the implementation. For the boundary frequency points caused by frequency truncation or zero padding, consistent complex symmetry is maintained to avoid time domain leakage.

[0144] S604: The windowed time domain frame sequence is respectively subjected to de-windowing compensation and overlap-add processing to obtain a continuous time domain speech sequence.

[0145] The windowed time domain frame sequence is subjected to de-windowing compensation and overlap-add to obtain a continuous time domain speech sequence. Preferably, the WOLA (Weighted Overlap-Add) strategy is adopted:

[0146] First, de-windowing compensation: amplitude correction is performed on each frame according to window power weighting to ensure that the reconstructed energy is consistent with the frequency domain;

[0147] Second, overlap-add: adjacent frames are added according to the consistent frame shift of the front end, and the last insufficient frame is processed according to the zero tail strategy;

[0148] Third, boundary processing: smoothing is performed on the transition area at the beginning and the end to suppress the amplitude step at the splicing position.

[0149] The medical domain can configure a longer window (such as 25 ms, 10 ms frame shift) to maintain smooth details; the financial domain can use a more compact window shift combination to reduce the end side delay, but the WOLA process is not changed.

[0150] S605: Direct current bias correction is performed on the continuous time domain speech sequence to obtain corrected time domain speech data.

[0151] The correction method can be to calculate the sliding mean value and deduct, and if necessary, a first-order high-pass micro filter is used to suppress the extremely low frequency drift. To prevent amplitude clipping, subsequent safety normalization in the amplitude domain is performed (without changing the relative dynamic range), and the peak margin matching the subsequent encoding bit depth is maintained to obtain the corrected time domain speech data.

[0152] S606: Perform sampling rate and bit depth output encoding processing on the corrected time-domain speech data to obtain enhanced speech waveform data.

[0153] The medical domain prioritizes outputting 16kHz / 16bit PCM (WAV container) for archiving and subsequent ASR; the financial domain can output 8kHz / 16bit PCM or 16kHz / 16bit PCM according to the quality inspection platform specifications, and write metadata such as call identifiers and timestamps when necessary. To avoid quantization noise concentration, low-amplitude unbiased jitter can be used before shaping to the integer bit depth to obtain enhanced speech waveform data.

[0154] It should be noted that because step S40 uses numerical integration of ordinary differential equations with a small step size to obtain the enhanced time-frequency representation during the inference stage, step S60 can complete the reconstruction output under low latency conditions. While keeping the ISTFT reconstruction link unchanged, if the upstream uses Mel spectrum as the enhancement target and enables a high-fidelity vocoder according to system specifications, steps S603-S605 can be replaced with a vocoder reconstruction sub-process in the engineering implementation.

[0155] As can be seen, this invention organically integrates generative modeling with flow matching modeling based on ordinary differential equations, constructing a consistent enhancement link of "standardization—velocity field estimation and numerical integration—destandardization—temporal reconstruction": replacing high-step diffusion inversion with velocity field estimation combined with numerical integration significantly reduces the number of inference steps and the real-time factor (RTF), achieving rapid inference; it does not rely on adversarial training and variational constraints, avoiding GAN training instability and VAE over-smoothing, thus improving the stability of training and inference; with the synergy of contextual modeling and optional semantic conditions, it can better maintain time-frequency details and phase consistency, improve speech intelligibility, naturalness, and speaker similarity in complex noise scenarios, and achieve better performance on objective evaluation metrics such as DNSMOS. Furthermore, the method is decoupled from the high-fidelity vocoder, facilitating engineering integration and stable deployment in edge and online systems.

[0156] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0157] In one embodiment, a stream-matching-based speech enhancement device is provided, which corresponds one-to-one with the stream-matching-based speech enhancement methods described in the above embodiments. For example... Figure 8 As shown, the stream matching-based speech enhancement device includes: a noise acquisition module 100, a noise processing module 200, a data processing module 300, a model optimization module 400, a data inversion module 500, and a reconstruction output module 600. Detailed descriptions of each functional module are as follows:

[0158] The noise acquisition module 100 is configured to acquire noise speech data and perform time-frequency conversion processing to obtain noise time-frequency representation data;

[0159] The noise processing module 200 is configured to perform standardization processing on the noise time-frequency representation data to obtain standardized time-frequency representation data;

[0160] The data processing module 300 is configured to perform stream matching input construction processing based on the standardized time-frequency representation data to obtain stream matching input representation data;

[0161] The model optimization module 400 is configured to input the stream matching input representation data into a feature encoding submodule of a stream matching model to obtain stream matching feature representation data; perform context modeling processing on the stream matching feature representation data to obtain context-enhanced feature representation data; input the context-enhanced feature representation data into a velocity field estimation submodule to output velocity field estimation data; calculate a time-frequency state increment based on the velocity field estimation data and a preset numerical step length parameter to obtain time-frequency state increment data; perform numerical solution of an ordinary differential equation on an intermediate state time-frequency tensor in the stream matching input representation data using the time-frequency state increment data to obtain an updated intermediate state time-frequency tensor; input the updated intermediate state time-frequency tensor as input into the stream matching model to update until a target path parameter is reached to obtain enhanced time-frequency representation data;

[0162] The data backstepping module 500 is configured to perform inverse standardization processing on the enhanced time-frequency representation data to obtain inverse standardized time-frequency representation data;

[0163] The reconstruction output module 600 is configured to perform time-domain reconstruction processing on the inverse standardized time-frequency representation data to obtain enhanced speech waveform data.

[0164] In an embodiment, the noise acquisition module 100 is specifically configured to:

[0165] receive an audio input stream and parse file header information to obtain original noise speech data;

[0166] perform sampling rate unification and channel conversion processing on the original noise speech data respectively to obtain formatted noise speech data;

[0167] perform DC offset removal and pre-emphasis processing on the formatted noise speech data respectively to obtain preprocessed noise speech data;

[0168] perform frame processing on the preprocessed noise speech data based on a preset window length and a frame shift to obtain a noise speech frame sequence;

[0169] apply a preset window function on the noise speech frame sequence frame by frame to obtain a windowed noise speech frame sequence;

[0170] perform short-time Fourier transform on the windowed noise speech frame sequence to obtain a complex spectrum matrix;

[0171] perform amplitude spectrum and phase spectrum calculation on the complex spectrum matrix and combine according to a preset time-frequency feature format to obtain noise time-frequency representation data.

[0172] In an embodiment, the noise processing module 200 is specifically configured to:

[0173] extract an amplitude spectrum matrix and a phase spectrum matrix from the noise time-frequency representation data respectively to obtain a to-be-normalized amplitude spectrum matrix and a to-be-normalized phase spectrum matrix;

[0174] perform numerical stabilization processing on the to-be-normalized amplitude spectrum matrix to obtain a stabilized amplitude spectrum matrix;

[0175] perform dynamic range compression processing based on the stabilized amplitude spectrum matrix to obtain a compressed amplitude spectrum matrix;

[0176] statistically obtain mean and standard deviation parameters of the compressed amplitude spectrum matrix according to a preset dimension to obtain normalization statistical parameters;

[0177] perform mean and variance normalization processing on the compressed amplitude spectrum matrix by using the normalization statistical parameters to obtain a normalized amplitude spectrum matrix;

[0178] perform encoding combination on the normalized amplitude spectrum matrix and the phase spectrum matrix based on a preset time-frequency feature format to obtain normalized time-frequency representation data.

[0179] In an embodiment, the data processing module 300 is specifically configured to:

[0180] extract a feature tensor corresponding to a model input size from the normalized time-frequency representation data to obtain a normalized time-frequency tensor;

[0181] generate a random noise tensor with zero mean and unit variance based on the normalized time-frequency tensor to obtain a noise time-frequency initial tensor;

[0182] sample a path parameter within a preset interval and perform linear interpolation operation on the noise time-frequency initial tensor and the normalized time-frequency tensor to obtain an intermediate state time-frequency tensor;

[0183] perform frame block division and position index encoding processing on the intermediate state time-frequency tensor respectively to obtain an intermediate state time-frequency tensor with position encoding;

[0184] inputting the position-encoding intermediate state time-frequency tensor and the path parameter into a time encoding module to output a condition vector;

[0185] concatenating the condition vector and the position-encoding intermediate state time-frequency tensor and performing channel mapping processing to obtain flow matching input representation data.

[0186] In an embodiment, the data processing module 300 is further specifically configured to:

[0187] analyzing start and end boundaries of a preset interval and configuring a sampling strategy to obtain a path parameter;

[0188] performing relative position conversion and range constraint on the target path parameter to obtain an interpolation weight;

[0189] performing size, channel and step alignment processing on the noise time-frequency initial tensor and the standardized time-frequency tensor based on the interpolation weight to obtain an aligned tensor set;

[0190] performing linear weighted synthesis on the aligned tensor set to obtain an intermediate state time-frequency tensor.

[0191] In an embodiment, the data reverse calculation module 500 is specifically configured to:

[0192] extracting an enhanced normalized amplitude spectrum matrix and an enhanced phase spectrum matrix from the enhanced time-frequency representation data;

[0193] performing inverse transform calculation on the enhanced normalized amplitude spectrum matrix to obtain an inverse normalized amplitude spectrum matrix;

[0194] performing inverse compression processing on the inverse normalized amplitude spectrum matrix according to a preset dynamic range compression configuration parameter to obtain an inverse compressed amplitude spectrum matrix;

[0195] performing numerical restoration processing on the inverse compressed amplitude spectrum matrix to obtain a restored amplitude spectrum matrix;

[0196] performing encoding combination on the restored amplitude spectrum matrix and the enhanced phase spectrum matrix to obtain candidate inverse standardized time-frequency representation data;

[0197] performing dimension rearrangement on the candidate inverse standardized time-frequency representation data to obtain inverse standardized time-frequency representation data.

[0198] In an embodiment, the reconstruction output module 600 is specifically configured to:

[0199] extracting an inverse standardized amplitude spectrum matrix and an inverse standardized phase spectrum matrix from the inverse standardized time-frequency representation data;

[0200] complexly combine the inverse normalized amplitude spectrum matrix with the inverse normalized phase spectrum matrix to obtain a reconstructed complex spectrum matrix;

[0201] perform inverse short-time Fourier transform processing on the reconstructed complex spectrum matrix to obtain a windowed time-domain frame sequence;

[0202] perform de-windowing compensation and overlap-add processing on the windowed time-domain frame sequence respectively to obtain a continuous time-domain speech sequence;

[0203] perform direct current bias correction on the continuous time-domain speech sequence to obtain corrected time-domain speech data;

[0204] perform sampling rate and bit depth output coding processing on the corrected time-domain speech data to obtain enhanced speech waveform data.

[0205] The specific limitations of the speech enhancement device based on stream matching can refer to the limitations of the speech enhancement method based on stream matching in the foregoing, which will not be described here. Each module in the speech enhancement device based on stream matching can be realized by software, hardware, and combinations thereof, in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0206] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in Figure 9 The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external client through a network connection. The computer program is executed by the processor to implement the functions or steps of the speech enhancement method based on stream matching on the server side.

[0207] In one embodiment, a computer device is provided, which can be a client, and an internal structure diagram thereof can be as shown in Figure 10As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to implement the functions or steps of the client side of the flow matching based voice enhancement method.

[0208] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program which can implement the steps provided by the above-mentioned embodiments when executed.

[0209] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the related descriptions of the server side and the client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0210] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM) and the like.

[0211] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0212] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A speech enhancement method based on stream matching, characterized by, The method comprises the following steps: obtain noise speech data and perform time-frequency transformation processing to obtain noise time-frequency representation data; standardize the noise time-frequency representation data to obtain standardized time-frequency representation data; perform flow matching input construction processing based on the standardized time-frequency representation data to obtain flow matching input representation data; input the flow matching input representation data into the feature encoding submodule of the flow matching model to obtain flow matching feature representation data; perform context modeling processing on the flow matching feature representation data to obtain context enhanced feature representation data; input the context enhanced feature representation data into the velocity field estimation submodule to output velocity field estimation data; calculate the time-frequency state increment based on the velocity field estimation data and a preset numerical step length parameter to obtain time-frequency state increment data; use the time-frequency state increment data to perform numerical solution of ordinary differential equations on the intermediate state time-frequency tensor in the flow matching input representation data to obtain an updated intermediate state time-frequency tensor; input the updated intermediate state time-frequency tensor as input into the flow matching model to update until the target path parameter is reached to obtain enhanced time-frequency representation data; perform inverse standardization processing on the enhanced time-frequency representation data to obtain inverse standardized time-frequency representation data; perform time domain reconstruction processing on the inverse standardized time-frequency representation data to obtain enhanced speech waveform data.

2. The speech enhancement method based on stream matching according to claim 1, characterized in that, The method comprises the following steps: receive an audio input stream and parse the file header information to obtain original noise speech data; perform sampling rate unification and channel conversion processing on the original noise speech data respectively to obtain formatted noise speech data; perform DC offset removal and pre-emphasis processing on the formatted noise speech data respectively to obtain preprocessed noise speech data; perform frame processing on the preprocessed noise speech data based on a preset window length and frame shift to obtain a noise speech frame sequence; apply a preset window function to the noise speech frame sequence frame by frame to obtain a windowed noise speech frame sequence; perform short-time Fourier transform on the windowed noise speech frame sequence to obtain a complex spectrum matrix; perform amplitude spectrum and phase spectrum calculation on the complex spectrum matrix and combine according to a preset time-frequency feature format to obtain noise time-frequency representation data.

3. The speech enhancement method based on stream matching according to claim 1, characterized in that, The method comprises the following steps: extract the amplitude spectrum matrix and the phase spectrum matrix from the noise time-frequency representation data respectively to obtain the to-be-standardized amplitude spectrum matrix and the to-be-standardized phase spectrum matrix; perform numerical stabilization processing on the to-be-standardized amplitude spectrum matrix to obtain a stabilized amplitude spectrum matrix; perform dynamic range compression processing on the stabilized amplitude spectrum matrix to obtain a compressed amplitude spectrum matrix; statistically obtain the mean and standard deviation parameters of the compressed amplitude spectrum matrix according to a preset dimension to obtain standardization statistical parameters; perform mean and variance normalization processing on the compressed amplitude spectrum matrix using the standardization statistical parameters to obtain a normalized amplitude spectrum matrix; The normalized amplitude spectrum matrix and the phase spectrum matrix are encoded and combined based on a preset time-frequency feature format, to obtain standardized time-frequency representation data.

4. The speech enhancement method based on stream matching according to claim 1, characterized in that, The standardized time-frequency representation data are subjected to stream matching input construction processing, to obtain stream matching input representation data, including: A feature tensor corresponding to a model input size is extracted from the standardized time-frequency representation data, to obtain a standardized time-frequency tensor; A random noise tensor with zero mean and unit variance is generated based on the standardized time-frequency tensor, to obtain a noise time-frequency initial tensor; A path parameter is sampled in a preset interval, and linear interpolation operation is performed on the noise time-frequency initial tensor and the standardized time-frequency tensor, to obtain an intermediate state time-frequency tensor; The intermediate state time-frequency tensor is subjected to frame block division and position index encoding processing respectively, to obtain an intermediate state time-frequency tensor with position encoding; The intermediate state time-frequency tensor with position encoding and the path parameter are jointly input into a time encoding module, to output a condition vector; The condition vector and the intermediate state time-frequency tensor with position encoding are subjected to feature splicing and channel mapping processing, to obtain stream matching input representation data.

5. The speech enhancement method based on stream matching according to claim 4, characterized in that, The path parameter is sampled in a preset interval, and linear interpolation operation is performed on the noise time-frequency initial tensor and the standardized time-frequency tensor, to obtain an intermediate state time-frequency tensor, including: The start and end boundaries of the preset interval are analyzed and a sampling strategy is configured, to obtain a path parameter; The target path parameter is subjected to relative position conversion and range constraint respectively, to obtain an interpolation weight; The noise time-frequency initial tensor and the standardized time-frequency tensor are subjected to size, channel and step alignment processing based on the interpolation weight, to obtain an aligned tensor set; The aligned tensor set is subjected to linear weighted synthesis, to obtain an intermediate state time-frequency tensor.

6. The voice enhancement method based on stream matching according to claim 1, characterized in that, The enhanced time-frequency representation data are subjected to reverse standardization processing, to obtain reverse standardized time-frequency representation data, including: An enhanced normalized amplitude spectrum matrix and an enhanced phase spectrum matrix are extracted from the enhanced time-frequency representation data respectively; Inverse transform calculation is performed on the enhanced normalized amplitude spectrum matrix, to obtain a reverse normalized amplitude spectrum matrix; Reverse compression processing is performed on the reverse normalized amplitude spectrum matrix according to a preset dynamic range compression configuration parameter, to obtain a reverse compressed amplitude spectrum matrix; Numerical restoration processing is performed on the reverse compressed amplitude spectrum matrix, to obtain a restored amplitude spectrum matrix; The restored amplitude spectrum matrix and the enhanced phase spectrum matrix are encoded and combined, to obtain candidate reverse standardized time-frequency representation data; The candidate reverse standardized time-frequency representation data are subjected to dimension rearrangement, to obtain reverse standardized time-frequency representation data.

7. The voice enhancement method based on stream matching according to claim 1, characterized in that, The reverse standardized time-frequency representation data are subjected to time domain reconstruction processing, to obtain enhanced speech waveform data, including: A reverse standardized amplitude spectrum matrix and a reverse standardized phase spectrum matrix are extracted from the reverse standardized time-frequency representation data respectively; Complex synthesis processing is performed on the reverse standardized amplitude spectrum matrix and the reverse standardized phase spectrum matrix, to obtain a reconstructed complex spectrum matrix; Inverse short-time Fourier transform processing is performed on the reconstructed complex spectrum matrix, to obtain a windowed time domain frame sequence; The windowed time domain frame sequence is respectively subjected to de-windowing compensation and overlap-add processing to obtain a continuous time domain speech sequence; The continuous time domain speech sequence is subjected to direct current bias correction to obtain corrected time domain speech data; The corrected time domain speech data is subjected to sampling rate and bit depth output coding processing to obtain enhanced speech waveform data.

8. A speech enhancement device based on stream matching, characterized by It comprises: a noise acquisition module for acquiring noise speech data and performing time-frequency transform processing to obtain noise time-frequency representation data; a noise processing module for standardizing the noise time-frequency representation data to obtain standardized time-frequency representation data; a data processing module for performing flow matching input construction processing based on the standardized time-frequency representation data to obtain flow matching input representation data a model optimization module for inputting the flow matching input representation data into a feature encoding submodule of a flow matching model to obtain flow matching feature representation data; performing context modeling processing on the flow matching feature representation data to obtain context enhanced feature representation data; inputting the context enhanced feature representation data into a velocity field estimation submodule to output velocity field estimation data; based on the velocity field estimation data and a preset numerical step parameter to calculate a time-frequency state increment to obtain time-frequency state increment data; using the time-frequency state increment data to perform numerical solution of ordinary differential equations on the intermediate state time-frequency tensor in the flow matching input representation data to obtain an updated intermediate state time-frequency tensor; inputting the updated intermediate state time-frequency tensor as input into the flow matching model to update until the target path parameter is reached to obtain enhanced time-frequency representation data; a data backstepping module for performing inverse standardization processing on the enhanced time-frequency representation data to obtain inverse standardized time-frequency representation data; a reconstruction output module for performing time domain reconstruction processing on the inverse standardized time-frequency representation data to obtain enhanced speech waveform data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the flow matching based speech enhancement method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to realize the steps of the flow matching based speech enhancement method according to any one of claims 1 to 7.

Images