Audio generation method and device based on conditional stream matching, equipment and medium

Abstract

The invention relates to the technical field of voice processing, can be applied to the fields of financial science and technology and medical health, and discloses an audio generation method, device and equipment based on conditional stream matching and a medium, and the method comprises the steps: obtaining an original audio signal, and processing the original audio signal to generate a voice Me < l > spectrum; encoding the voice Me < l > spectrum to obtain a low-dimensional compression feature, and processing the low-dimensional compression feature to generate a quantized Token sequence; and decoding the quantized Token sequence to obtain a reconstructed Me < l > spectrum, and inputting the reconstructed Me < l > spectrum into a conditional stream matching model to perform audio reconstruction so as to generate a target audio signal. And the audio generation quality under the low code rate is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, which can be applied to the fields of financial technology and medical health, and in particular relates to an audio generation method and device based on conditional flow matching, equipment and medium. BACKGROUND

[0002] Although the traditional audio encoding method can guarantee good sound quality and compression efficiency at high code rate, when it is applied to the fields of financial technology (such as telephone bank customer service recording quality inspection, high-risk transaction instruction voiceprint authentication, and remote account opening voice audit) and medical health (such as remote heart sound / pulmonary sound diagnosis transmission, voice-based old people mental health assessment, and voice interaction of wearable medical devices), when its code rate is compressed to below 1.5 kbps, i.e. at low code rate, it faces significant problems: 1. Distortion is obvious, and sound quality is reduced: the traditional method is based on perceptual modeling, and the limited spectral information retained at low code rate makes it difficult to reconstruct high-quality audio, resulting in "metallic" or blurred speech; 2. Encoding process lacks generative ability: existing compression schemes lack flexible data-driven mechanisms, especially when facing complex speech signals (such as multi-speaker, emotional speech); 3. Compression-reconstruction end mismatch problem: the compression end and the reconstruction end are generally driven by different strategies, such as the vocoder structure based on linear predictive coding, which is often disconnected from end-to-end speech synthesis, resulting in low coordination efficiency and limited sound quality; 4. Insufficient model generative ability, difficult to generalize across languages and scenarios: traditional methods often need to manually adjust parameters for different languages, channels or noise conditions, making it difficult to generalize. SUMMARY

[0003] The present application provides an audio generation method and device based on conditional flow matching, and a computer device and medium, to solve the technical problem of poor quality of existing audio generation at low code rate.

[0004] In a first aspect, an audio generation method based on conditional flow matching is provided, comprising:

[0005] An original audio signal is obtained, and the original audio signal is processed to generate a speech Mel spectrum;

[0006] The speech Mel spectrum is encoded to obtain low-dimensional compression features, and the low-dimensional compression features are processed to generate a quantized Token sequence;

[0007] The quantized Token sequence is decoded to obtain a reconstructed Mel spectrum, and the reconstructed Mel spectrum is input into a conditional flow matching model for audio reconstruction to generate a target audio signal.

[0008] In a second aspect, an audio generation device based on conditional flow matching is provided, comprising:

[0009] An acquisition processing unit is configured to acquire an original audio signal and process the original audio signal to generate a speech Mel spectrum.

[0010] An encoding processing unit is configured to encode the speech Mel spectrum to obtain low-dimensional compressed features, and process the low-dimensional compressed features to generate a quantized Token sequence.

[0011] A decoding generation unit is configured to decode the quantized Token sequence to obtain a reconstructed Mel spectrum, input the reconstructed Mel spectrum into a conditional flow matching model for audio reconstruction, and generate a target audio signal.

[0012] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the audio generation method based on conditional flow matching when executing the computer program.

[0013] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the audio generation method based on conditional flow matching when executed by a processor.

[0014] The audio generation method based on conditional flow matching, the device, the computer device, and the storage medium can acquire an original audio signal and process the original audio signal to generate a speech Mel spectrum, encode the speech Mel spectrum to obtain low-dimensional compressed features, process the low-dimensional compressed features to generate a quantized Token sequence, decode the quantized Token sequence to obtain a reconstructed Mel spectrum, input the reconstructed Mel spectrum into a conditional flow matching model for audio reconstruction, and generate a target audio signal. In the present application, the original audio signal is first processed to generate a speech Mel spectrum, then the speech Mel spectrum is sequentially encoded, quantized, and decoded to generate a reconstructed Mel spectrum, and finally the reconstructed Mel spectrum is input into a conditional flow matching model for audio reconstruction to generate a target audio signal, thereby improving the quality of audio generation at a low code rate. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 is a flowchart of an audio generation method based on conditional flow matching in an embodiment of the present application;

[0017] Figure 2 is Figure 1 a whole schematic diagram of a tool with low code rate audio encoding function in the embodiment;

[0018] Figure 3 is Figure 1 a specific embodiment flowchart of step S110 in the embodiment;

[0019] Figure 4 is Figure 1 a specific embodiment flowchart of step S120 in the embodiment;

[0020] Figure 5 is Figure 1 a specific embodiment flowchart of step S130 in the embodiment;

[0021] Figure 6 is a schematic block diagram of an audio generation device based on conditional flow matching in an embodiment of the present application;

[0022] Figure 7 is a structural schematic diagram of a computer device in an embodiment of the present application;

[0023] Figure 8 is another structural schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0025] The audio generation method based on conditional flow matching provided by the embodiments of the present application can be applied to a client or a server. The client can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. At present, in the field of financial technology and medical health, the quality of existing audio generation is poor at low code rate. In view of the above problems, the present application provides an audio generation method based on conditional flow matching. The method first processes the original audio signal to generate a speech Mel spectrum; then encodes, quantizes and decodes the speech Mel spectrum in turn to generate a reconstructed Mel spectrum; finally, the reconstructed Mel spectrum is input into a conditional flow matching model to generate a target audio signal, thereby improving the quality of audio generation at low code rate. The present application will be described in detail through specific embodiments.

[0026] Please refer to Figure 1 as shown, Figure 1 A flowchart of an audio generation method based on conditional flow matching provided by an embodiment of the present application is shown, including the following steps: S110-S130.

[0027] S110, an original audio signal is obtained, and the original audio signal is processed to generate a speech Mel spectrum.

[0028] Specifically, for the convenience of understanding, a tool with low-bit-rate audio coding function is introduced, such as Figure 2 As shown, the tool with low-bit-rate audio coding function includes a speech processing module, an encoder, a quantizer, a decoder, and a conditional flow matching module, wherein the speech processing module is used to process the original audio signal to generate a speech Mel spectrum, the encoder is used to encode the speech Mel spectrum to obtain low-dimensional compressed features, the quantizer is used to process the low-dimensional compressed features to generate a quantized Token sequence, the decoder is used to decode the quantized Token sequence to obtain a reconstructed Mel spectrum, and the conditional flow matching module includes a conditional flow matching model, which is used to reconstruct the reconstructed Mel spectrum to generate a target audio signal. The original audio signal is obtained, wherein the original audio signal can be an identity authentication signal, a transaction instruction signal, and a risk control monitoring signal in the field of financial technology, or a pathological diagnosis signal, a diagnosis and treatment interaction signal, and a rehabilitation monitoring signal in the field of medical health. The original audio signal is processed to generate a speech Mel spectrum. By implementing the above step S110, the original audio signal is converted into a speech Mel spectrum, so that the semantic and emotional features of the original audio signal are more sensitive.

[0029] As shown in Figure 3 In step S110, the following steps are included: S111-S112.

[0030] S111, the original audio signal is pre-emphasized, and the original audio signal after pre-emphasis is divided and windowed to obtain a plurality of windowed signals;

[0031] S112, the amplitude spectrum of each windowed signal is obtained by fast Fourier transform, and the amplitude spectrum is compressed to obtain the speech Mel spectrum.

[0032] Specifically, pre-emphasis processing enhances the high-frequency components of the original audio signal using a first-order high-pass filter to compensate for high-frequency attenuation during glottal excitation transmission. This also flattens the spectrum, preventing subsequent processing from masking crucial information due to excessive low-frequency energy. The pre-emphasis-processed original audio signal is divided into short-time overlapping frames (frame length 20-40ms, frame shift 10ms) to capture the quasi-steady-state characteristics of speech. A Hamming window function is applied to each frame, its core purpose being to smooth the truncation effect at frame edges and suppress spectral leakage. The windowed output signal is continuous in the time domain and reduces spurious components in the frequency domain, ensuring that the spectral resolution of the Fourier transform meets the requirements of the time-varying characteristics of speech. A Fast Fourier Transform is performed on each windowed frame, converting the time-domain waveform into a complex frequency-domain spectrum, and its magnitude is taken to output a linear amplitude spectrum. Fast Fourier Transform (FFT) of each windowed signal frame reveals the frequency distribution of the signal, but retaining all frequency points leads to data redundancy. Therefore, amplitude spectrum compression is necessary, which includes nonlinear Mel compression and logarithmic compression. Nonlinear Mel compression uses a Mel filter bank to nonlinearly map the amplitude spectrum. The filter bank is designed based on human auditory characteristics (Mel scale: dense at low frequencies, sparse at high frequencies), compressing linear frequencies into a perceptually compliant Mel band (e.g., 128 dimensions). Each filter weights and sums the band energy, outputting the Mel band energy value. Logarithmic compression takes the natural logarithm of the Mel energy value, simulating the nonlinear response of the human ear to sound intensity, while compressing the dynamic range to enhance the visibility of low-energy components. The final output is a logarithmic Mel spectrum, which serves as the speech Mel spectrum, preserving key auditory features while significantly reducing data dimensionality.

[0033] S120. Encode the speech mel spectrum to obtain low-dimensional compressed features, and process the low-dimensional compressed features to generate a quantized token sequence.

[0034] Specifically, in tools with low-bitrate audio encoding capabilities, the core objective of the encoder in encoding the speech Mel spectrum is to address the problem of high-dimensional data redundancy. It efficiently transforms the input Mel spectrum (a frame-structured time-frequency matrix, e.g., 40 frames per second × 128-dimensional coefficients) into a compact, transmittable feature representation at extremely low bitrates (e.g., ≤1.5kbps), supporting subsequent high-quality reconstruction. This process is performed by the encoder: First, the encoder receives the Mel spectrum as input and extracts intrinsic features in the time-frequency domain through a multi-layer neural network (e.g., convolutional or Transformer architecture), capturing long-term dependencies and local patterns in the speech to generate low-dimensional compressed features.

[0035] Among them, such as Figure 4 As shown, step S120 includes the following steps: S121-S122:

[0036] S121, calculate the distance between the low-dimensional compressed feature and a pre-defined Token set, wherein the pre-defined Token set comprises a plurality of numerical representation vectors;

[0037] S122, select the Token sequence corresponding to the numerical representation vector with the smallest distance as the quantized Token sequence.

[0038] Specifically, in a tool with low-bitrate audio coding function, the quantizer undertakes the key dimension reduction task: converting the continuous low-dimensional compressed features (such as 128-dimensional vectors) output by the encoder into discrete, efficiently transmissible symbol sequences, while minimizing information loss. Its core solves the spectral distortion problem caused by manual quantization in traditional compression, providing accurate input for generative reconstruction, i.e., providing accurate input for the decoder. Construction of the pre-defined Token set: generate a numerical representation vector library (such as embedding vectors corresponding to numbers 1, 2, and 3) through end-to-end training, and each numerical representation vector represents a typical pattern in the compressed feature space. This Token set covers the diversity of speech spectrum (such as phonemes, fundamental frequency, formants), ensuring generalization ability. The vectorizer inputs the low-dimensional compressed features generated by the encoder, calculates the distance between the low-dimensional compressed features and each numerical representation vector in the Token set, wherein the distance uses Euclidean distance or cosine similarity to measure the spatial proximity between the low-dimensional compressed features and the numerical representation vector; select the Token sequence corresponding to the numerical representation vector with the smallest distance as the quantized Token sequence, to ensure that the error between the original feature and the symbol representation is minimized, and to avoid reconstruction distortion. It should be noted that the Token sequence retains the spectral structure and temporal correlation, providing a structured conditional input for the conditional stream matching model. It should also be noted that the implementation of the above steps S121-S122 ensures that the compressed features accurately retain key spectral information, and the quantization standard is optimized through end-to-end training, minimizing information loss and cooperating with the entire system (e.g., the conditional stream matching model) to improve compression efficiency and robustness. This step achieves significant dimension reduction (e.g., simplifying the 24000 sampling points of the original 24kHz audio to dozens of Tokens), reducing storage and transmission overhead, while providing structured input for the decoder, solving the distortion problem of traditional methods at low bitrates, and adapting to complex scenarios such as multiple speakers and noisy environments. Ultimately, it balances audio fidelity and resource constraints in financial technology (such as voiceprint authentication audio compression) and medical health (such as remote diagnosis speech transmission) applications.

[0039] S130, decoding the quantized Token sequence to obtain a reconstructed Mel spectrum, and inputting the reconstructed Mel spectrum into a conditional stream matching model for audio reconstruction to generate a target audio signal.

[0040] Specifically, the quantized token sequence is first restored to a reconstructed Mel spectrum by the decoder, which retains the spectral structure and timing characteristics of the speech (such as the fundamental frequency trajectory and the formant envelope), but may have phase ambiguity or detail distortion due to quantization loss; then, the reconstructed Mel spectrum is input as a conditional vector into the conditional flow matching model, which generates a controlled flow path based on the flow matching theory, uses the dense conditional vector to regulate the transformation direction of noise to the target, directly synthesizes high-fidelity time-domain waveforms through single-step ODE solving, synchronously integrates a phase correction mechanism to compensate for spectral defects, and finally outputs natural and smooth target audio signals. Understandably, by implementing the above step 130, the quality of audio generation at low code rate is improved. It should be noted that in this embodiment, ODE (Ordinary Differential Equation) is a common differential equation used in the conditional flow matching model to describe the continuous transformation path from noise distribution to target audio signals. Single-step ODE refers to directly solving the equation through a single calculation to output the target audio signal. It should also be noted that the conditional flow matching model is obtained by training the flow matching model using a sample training set, wherein the sample training set includes a sample noise signal, a conditional vector, and a time step, the sample noise signal is a signal obtained by adding Gaussian noise to a sample audio signal, the conditional vector is the reconstructed Mel spectrum, and the time step takes a value between 0 and 1. The specific process of training the flow matching model is as follows: add Gaussian noise to the sample audio signal to obtain the sample noise signal, simulate the starting point of the generated path; use the reconstructed Mel spectrum (with quantization error) as the control signal to bind the acoustic properties (such as fundamental frequency and formant); locate the path interpolation point through the time step; path derivative fitting: the flow matching model predicts the vector field to approximate the true derivative; loss function: minimize the mean square error between the predicted field and the true field: the reconstructed Mel spectrum (conditional vector) is jointly trained with the encoder and quantizer to form a compression-generation closed loop; the time step controls the nonlinearity of the path to ensure that the model covers the complete transformation space from noise to target, and finally obtains the conditional flow matching model.

[0041] wherein, as shown in Figure 5 the step S130 includes the following steps: S131-S134.

[0042] S131, input the reconstructed Mel spectrum into the conditional flow matching model to extract key acoustic properties and generate a dense conditional vector;

[0043] S132, obtain standard Gaussian noise, and generate a controlled flow path based on the standard Gaussian noise, the dense conditional vector, and the target audio signal;

[0044] S133, predict a vector field according to the controlled flow path and the dense conditional vector to output an initial audio signal;

[0045] S134, supplementing the missing phase information in the initial audio signal to generate the target audio signal.

[0046] Specifically, the conditional flow matching model includes a shallow convolution, a deep hole convolution, a cross-channel attention mechanism, and a fully connected layer. Step S131 includes: inputting the reconstructed Mel spectrum into the conditional flow matching model to generate a multi-granularity feature tensor by capturing local spectral patterns through the shallow convolution and modeling long-time dependence through the deep hole convolution; separating the multi-granularity feature tensor to obtain content features and attribute features, and weighting and aggregating the attribute features through the cross-channel attention mechanism to obtain aggregated attribute features; and compressing and dimension reducing the aggregated attribute features and the content features through the fully connected layer to generate the dense condition vector. Step S132 includes: mapping the dense condition vector to path parameters to modulate the standard Gaussian noise to obtain attribute-modulated noise; defining a time coefficient, and constructing a linear interpolation path according to the time coefficient, the attribute-modulated noise, and the target audio signal to obtain the controlled flow path. The specific process of inputting the reconstructed Mel spectrum into the conditional flow matching model for audio reconstruction to generate the target audio signal is as follows: after the reconstructed Mel spectrum is input into the conditional flow matching model, the local area of the spectrum is first scanned through the shallow convolution network (kernel size 3*3) to accurately capture transient acoustic events, including the burst energy peak of the burst sound, the micro disturbance of the formant, and the pathological short-time clutter in the medical heart sound, and a high spatiotemporal resolution feature map is output; then the deep hole convolution is stacked to expand the receptive field to more than 200ms, and the macro-rhythm structure of the speech is modeled: the rise and fall slope of the fundamental frequency trajectory depicts the tone credibility of the financial voiceprint, the continuous change of the formant envelope simulates the shape of the sound channel (the 400Hz sustained peak of the asthma wheezing sound in the medical scene), and the energy attenuation trend quantifies the pathological expiratory phase extension feature. The generated multi-granularity feature tensor is decoupled by a double-branch gating mechanism: the content branch strips off the speaker-independent information such as phoneme sequence and fundamental frequency profile, and the attribute branch extracts individualized features such as timbre fingerprint and emotional intensity; the latter is weighted and aggregated through four-head cross-channel attention to calculate a channel similarity matrix, strengthen the medical pathological frequency band and the financial voiceprint sensitive area, and suppress environmental noise and redundant spectrum. Finally, the mixed features are compressed through a two-layer fully connected network (512 dimensions→64 dimensions): the ReLU activation selects effective parameters, and the linear mapping avoids distortion to generate a 64-dimensional dense condition vector. After the dense condition vector is generated, the dense condition vector is interpreted into path parameters through an affine transformation layer: the scaling factor α controls the energy range of the noise distribution, and the bias β aligns the statistical mean of the target signal to generate the attribute-modulated noise z cond = α * z + β, so that the statistical characteristics of the Gaussian noise are essentially matched with the target audio signal. Define a continuous time coefficient t∈

[0047] [0,1] serves as the path evolution parameter: t=0 corresponds to the target signal end, and t=1 corresponds to the modulation noise end. The linear interpolation formula based on t is x. t = (1-t)*x target +t*z cond Constructing a deterministic flow path yields a controlled flow path. Inputting the controlled flow path points and dense conditional vectors, the conditional flow matching model predicts a deterministic vector field to fit the derivative of the true path. By minimizing the mean square error between the predicted and true fields, the flow matching model learns a direct mapping rule from noise to the target. In the inference phase, modulation noise and conditional vectors are input into the training converged vector field, and a single-step ODE solver outputs the initial audio signal. Although the initial audio signal retains 90% of its spectral structure, the missing phase information in the Mel spectrum may cause spatiotemporal misalignment of the waveform (metallization distortion). Through phase correction using the Hilbert transform basis, the instantaneous phase angle of the analytic signal is calculated, and orthogonal components are reconstructed to fill spectral gaps, yielding the target audio signal. This is achieved through the implementation steps.

[0048] S131-S134 can still reconstruct speech with high fidelity and naturalness at low bit rates, solving the problem of poor sound quality and obvious distortion in existing audio coding at low bit rates.

[0049] The audio generation method based on conditional flow matching in this invention first processes the original audio signal to generate a speech mel spectrum; then, the speech mel spectrum is sequentially encoded, quantized, and decoded to generate a reconstructed mel spectrum; finally, the reconstructed mel spectrum is input into a conditional flow matching model to reconstruct the audio signal and generate the target audio signal, thereby improving the quality of audio generation at low bit rates.

[0050] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0051] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0052] In one embodiment, an audio generation apparatus 200 based on conditional stream matching is provided, which corresponds one-to-one with the audio generation method based on conditional stream matching in the above embodiments. For example... Figure 6 As shown, the audio generation device based on conditional stream matching includes an acquisition processing unit 201, an encoding processing unit 202, and a decoding generation unit 203. Detailed descriptions of each functional module are as follows:

[0053] The acquisition and processing unit 201 is used to acquire the original audio signal and process the original audio signal to generate a speech mel spectrum;

[0054] The encoding processing unit 202 is configured to encode the speech Mel spectrum to obtain low-dimensional compressed features, and process the low-dimensional compressed features to generate a quantized Token sequence.

[0055] The decoding generation unit 203 is configured to decode the quantized Token sequence to obtain a reconstructed Mel spectrum, input the reconstructed Mel spectrum into a conditional flow matching model for audio reconstruction, and generate a target audio signal.

[0056] In an embodiment, the obtaining processing unit 201 is specifically configured to:

[0057] perform pre-emphasis processing on the original audio signal, and split and window the original audio signal after pre-emphasis processing to obtain a plurality of windowed signals;

[0058] perform fast Fourier transform on each windowed signal to obtain an amplitude spectrum, and compress the amplitude spectrum to obtain the speech Mel spectrum.

[0059] In an embodiment, the encoding processing unit 202 is specifically configured to:

[0060] calculate a distance between the low-dimensional compressed features and a predefined Token set, wherein the predefined Token set includes a plurality of numerical representation vectors;

[0061] select a Token sequence corresponding to a numerical representation vector with the smallest distance as the quantized Token sequence.

[0062] In an embodiment, the decoding generation unit 203 is specifically configured to:

[0063] input the reconstructed Mel spectrum into the conditional flow matching model to extract key acoustic attributes and generate a dense condition vector;

[0064] obtain standard Gaussian noise, and generate a controlled flow path based on the standard Gaussian noise, the dense condition vector, and the target audio signal;

[0065] predict a vector field according to the controlled flow path and the dense condition vector to output an initial audio signal;

[0066] supplement missing phase information in the initial audio signal to generate the target audio signal.

[0067] In an embodiment, the decoding generation unit 203 is further configured to:

[0068] input the reconstructed Mel spectrum into the conditional flow matching model to capture local spectral patterns through the shallow convolution and model long-time dependence through the deep dilated convolution, and generate a multi-granularity feature tensor.

[0069] The multi-granularity feature tensor is separated to obtain content features and attribute features, and the attribute features are weighted and aggregated through the cross-channel attention mechanism to obtain aggregated attribute features;

[0070] The dense condition vector is generated by compressing and reducing the dimensionality of the aggregated attribute features and the content features through the fully connected layer.

[0071] In one embodiment, the decoding generation unit 203 is further configured to:

[0072] The dense conditional vector is mapped to path parameters to modulate the standard Gaussian noise to obtain attribute-modulated noise.

[0073] Define a time coefficient, and construct a linear interpolation path based on the time coefficient, the attribute modulation noise, and the target audio signal to obtain the controlled flow path.

[0074] In one embodiment, the decoding generation unit 203 is further configured to:

[0075] The conditional flow matching model is obtained by training the flow matching model using a sample training set. The sample training set includes sample noise signals, conditional vectors, and time steps. The sample noise signals are obtained by adding Gaussian noise to the sample audio signals. The conditional vectors are the reconstructed Mel spectrum. The time steps take values ​​between 0 and 1.

[0076] The audio generation device based on conditional flow matching in this invention first processes the original audio signal to generate a speech mel spectrum; then, the speech mel spectrum is sequentially encoded, quantized, and decoded to generate a reconstructed mel spectrum; finally, the reconstructed mel spectrum is input into the conditional flow matching model to reconstruct the audio signal and generate the target audio signal, thereby improving the quality of audio generation at low bit rates.

[0077] Specific limitations regarding the audio generation device based on conditional stream matching can be found in the limitations of the audio generation method based on conditional stream matching above, and will not be repeated here. Each unit in the aforementioned audio generation device based on conditional stream matching can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0078] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media, internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external client through the network connection. The computer program is executed by the processor to realize the functions or steps of the server side of the condition stream matching based audio generation method.

[0079] In one embodiment, a computer device is provided, which can be a client, and its internal structure diagram can be as shown in the figure. Figure 8 As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media, internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the functions or steps of the client side of the condition stream matching based audio generation method.

[0080] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the steps of the condition stream matching based audio generation method.

[0081] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to realize the steps of the condition stream matching based audio generation method.

[0082] It should be noted that the functions or steps that the computer readable storage medium or the computer device can realize are described above, which can be referred to the related description of the server side and the client side in the foregoing method embodiment. To avoid repetition, they will not be described one by one here.

[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchl ink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0085] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not limit it. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features. The modification or replacement does not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An audio generation method based on conditional stream matching, characterized in that, include: The original audio signal is acquired and processed to generate a speech mel spectrum; The speech mel spectrum is encoded to obtain low-dimensional compressed features, and the low-dimensional compressed features are processed to generate a quantized token sequence. The quantized token sequence is decoded to obtain the reconstructed Mel spectrum, which is then input into a conditional flow matching model for audio reconstruction to generate the target audio signal.

2. The audio generation method based on conditional stream matching as described in claim 1, characterized in that, The step of inputting the reconstructed Mel spectrum into the conditional flow matching model for audio reconstruction to generate the target audio signal includes: The reconstructed Mel spectrum is input into the conditional flow matching model to extract key acoustic properties and generate a dense conditional vector; Obtain standard Gaussian noise, and generate a controlled flow path based on the standard Gaussian noise, the dense condition vector, and the target audio signal; Predict a vector field based on the controlled flow path and the dense conditional vector to output an initial audio signal; The missing phase information in the initial audio signal is supplemented to generate the target audio signal.

3. The audio generation method based on conditional stream matching as described in claim 2, characterized in that, The conditional flow matching model includes shallow convolution, deep dilated convolution, cross-channel attention mechanism, and fully connected layers. The step of inputting the reconstructed Mel spectrum into the conditional flow matching model to extract key acoustic properties and generate dense conditional vectors includes: The reconstructed Mel spectrum is input into the conditional flow matching model to capture local spectral patterns through shallow convolution and model long-term dependencies through deep dilated convolution, generating multi-granularity feature tensors. The multi-granularity feature tensor is separated to obtain content features and attribute features, and the attribute features are weighted and aggregated through the cross-channel attention mechanism to obtain aggregated attribute features; The dense condition vector is generated by compressing and reducing the dimensionality of the aggregated attribute features and the content features through the fully connected layer.

4. The audio generation method based on conditional stream matching as described in claim 2, characterized in that, The step of generating a controlled flow path based on the standard Gaussian noise, the dense conditional vector, and the target audio signal includes: The dense conditional vector is mapped to path parameters to modulate the standard Gaussian noise to obtain attribute-modulated noise. Define a time coefficient, and construct a linear interpolation path based on the time coefficient, the attribute modulation noise, and the target audio signal to obtain the controlled flow path.

5. The audio generation method based on conditional stream matching as described in any one of claims 1-4, characterized in that, The conditional flow matching model is obtained by training the flow matching model using a sample training set. The sample training set includes sample noise signals, conditional vectors, and time steps. The sample noise signals are obtained by adding Gaussian noise to the sample audio signals. The conditional vectors are the reconstructed Mel spectrum. The time steps take values ​​between 0 and 1.

6. The audio generation method based on conditional stream matching as described in any one of claims 1-4, characterized in that, The step of processing the low-dimensional compressed features to generate a quantized token sequence includes: Calculate the distance between the low-dimensional compressed feature and the predefined token set, wherein the predefined token set includes multiple digital representation vectors; The token sequence corresponding to the digital representation vector with the smallest distance is selected as the quantized token sequence.

7. The audio generation method based on conditional stream matching as described in any one of claims 1-4, characterized in that, The step of processing the original audio signal to generate a speech mel spectrum includes: The original audio signal is pre-emphasized, and the pre-emphasized original audio signal is segmented and windowed to obtain multi-frame windowed signals. The amplitude spectrum is obtained by performing a fast Fourier transform on each windowed signal frame, and the amplitude spectrum is then compressed to obtain the speech mel spectrum.

8. An audio generation device based on conditional stream matching, characterized in that, include: An acquisition and processing unit is used to acquire the original audio signal and process the original audio signal to generate a speech mel spectrum. The encoding processing unit is used to encode the speech mel spectrum to obtain low-dimensional compressed features, and to process the low-dimensional compressed features to generate a quantized token sequence. The decoding and generation unit is used to decode the quantized token sequence to obtain the reconstructed Mel spectrum, and input the reconstructed Mel spectrum into the conditional flow matching model for audio reconstruction to generate the target audio signal.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the audio generation method based on conditional stream matching as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the audio generation method based on conditional stream matching as described in any one of claims 1 to 7.

Images