Audio coding and decoding method and device based on stream matching, equipment and medium

Abstract

The invention relates to the technical field of voice processing, and discloses an audio encoding and decoding method and device based on stream matching, equipment and a medium, which can be applied to financial service or digital medical service scenes, and can be used for obtaining an initial voice waveform corresponding to a target audio according to a target audio receiving instruction in response to the target audio receiving instruction; inputting the initial voice waveform into a preset encoder for encoding to obtain high-dimensional voice information; inputting the high-dimensional voice information into a preset quantizer for processing to obtain discrete voice information; inputting the discrete voice information into a preset decoder for decoding to obtain a first voice waveform; and performing voice enhancement on the first voice waveform by using the target stream matching model to generate an enhanced target voice, thereby realizing audio reconstruction with high perception quality at a low bit rate. And meanwhile, through the target stream matching model, the calculation overhead can be remarkably reduced, the reasoning speed is improved, and the method is suitable for full-band universal audio and is high in universality.

Description

Technical Field

[0001] The present invention relates to the fields of speech processing and medical technology, and in particular to an audio encoding and decoding method, apparatus, device and medium based on stream matching. Background Art

[0002] With the rapid development of information technology, audio data is increasingly used in key areas such as medical diagnosis, financial services, and intelligent interaction. Whether it is remote medical auscultation, medical equipment alarm monitoring, or voice authentication in financial scenarios, intelligent customer service interaction, etc., they are all highly dependent on efficient audio encoding and decoding technology.

[0003] In recent years, neural network-driven end-to-end (E2E) audio codecs, such as ScoreDec, have begun to emerge. These models use technologies such as generative adversarial networks (GANs) to achieve high audio quality at low bit rates. However, these methods still have some limitations. For example, the training process of GANs is complex and prone to instability, which may make model training difficult. Taking ScoreDec as an example, its post-processing step requires approximately 60 deep neural network (DNN) evaluations, resulting in a real-time factor (RTF) of 1.7, which is difficult to meet the needs of real-time applications. In addition, many existing neural codecs are mainly optimized for speech signals and lack the ability to generalize to general audio (such as music and ambient sound), limiting their use in a wider range of application scenarios. Therefore, existing audio codec technologies still have problems such as difficulty in achieving high-perceptual-quality audio reconstruction at low bit rates, slow inference speed, and poor versatility. Summary of the Invention

[0004] The embodiments of the present invention provide a stream matching-based audio encoding and decoding method, apparatus, device, and medium, aiming to solve the problems of the prior art in achieving high-perceptual-quality audio reconstruction at low bit rates, slow inference speed, and poor versatility.

[0005] In a first aspect, an embodiment of the present invention provides an audio encoding and decoding method based on stream matching, the method comprising:

[0006] Responding to a target audio receiving instruction, and acquiring an initial speech waveform corresponding to the target audio according to the target audio receiving instruction;

[0007] Inputting the initial speech waveform into a preset encoder for encoding to obtain encoded high-dimensional speech information;

[0008] Inputting the high-dimensional speech information into a preset quantizer for processing to obtain compressed discrete speech information;

[0009] Inputting the discrete voice information into a preset decoder for decoding to obtain a decoded first voice waveform;

[0010] The target stream matching model is used to perform speech enhancement on the first speech waveform to generate an enhanced target speech.

[0011] In a second aspect, an embodiment of the present invention further provides an audio encoding and decoding device based on stream matching, the device comprising:

[0012] a response unit, configured to respond to a target audio receiving instruction and acquire an initial speech waveform corresponding to the target audio according to the target audio receiving instruction;

[0013] An encoding unit, configured to input the initial speech waveform into a preset encoder for encoding to obtain encoded high-dimensional speech information;

[0014] a quantization unit, configured to input the high-dimensional speech information into a preset quantizer for processing to obtain compressed discrete speech information;

[0015] a decoding unit, configured to input the discrete voice information into a preset decoder for decoding to obtain a decoded first voice waveform;

[0016] The speech enhancement unit is used to perform speech enhancement on the first speech waveform using a target stream matching model to generate an enhanced target speech.

[0017] In a third aspect, an embodiment of the present invention further provides a computer device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the audio encoding and decoding method based on stream matching as described in the first aspect is implemented.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the audio encoding and decoding method based on stream matching described in the first aspect is implemented.

[0019] The embodiments of the present invention provide an audio encoding and decoding method, apparatus, device and medium based on stream matching, which responds to a target audio receiving instruction and obtains an initial speech waveform corresponding to the target audio according to the target audio receiving instruction; inputs the initial speech waveform into a preset encoder for encoding to obtain high-dimensional speech information; inputs the high-dimensional speech information into a preset quantizer for processing to obtain discrete speech information; inputs the discrete speech information into a preset decoder for decoding to obtain a first speech waveform; and uses a target stream matching model to perform speech enhancement on the first speech waveform to generate an enhanced target speech, thereby achieving audio reconstruction with high perceptual quality at a low bit rate; at the same time, the target stream matching model can significantly reduce computational overhead, improve inference speed, and is applicable to general audio in the full frequency band, with strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of a flow chart of an audio encoding and decoding method based on stream matching provided by an embodiment of the present invention;

[0022] Figure 2 A schematic block diagram of an audio encoding and decoding device based on stream matching provided by an embodiment of the present invention;

[0023] Figure 3 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0026] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0027] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] The embodiment of the present invention provides an audio encoding and decoding method based on stream matching, which can be specifically applied to various application scenarios, such as telemedicine auscultation, voice identity authentication, intelligent customer service interaction, etc., and the audio encoding and decoding method based on stream matching can be applied to a server, wherein the server receives an instruction in response to a target audio and obtains an initial speech waveform corresponding to the target audio according to the target audio reception instruction; inputs the initial speech waveform into a preset encoder for encoding to obtain encoded high-dimensional speech information; inputs the high-dimensional speech information into a preset quantizer for processing to obtain compressed discrete speech information; inputs the discrete speech information into a preset decoder for decoding to obtain a decoded first speech waveform; and uses a target stream matching model to perform speech enhancement on the first speech waveform to generate an enhanced target speech. The server can be an independent server, a server cluster, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0029] Among them, Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0030] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0031] The present invention is described in detail below through specific examples.

[0032] See also Figure 1 , Figure 1 FIG is a flow chart of an audio encoding and decoding method based on stream matching provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110-S150.

[0033] S110 , responding to a target audio receiving instruction and acquiring an initial speech waveform corresponding to the target audio according to the target audio receiving instruction.

[0034] In this embodiment, upon receiving a target audio reception instruction, the server responds to the target audio reception instruction and, in accordance with the target audio reception instruction, acquires the target audio from an audio capture device and generates an initial speech waveform. The audio capture device may be a microphone, a pickup, or an integrated audio sensor, and the target audio reception instruction may be triggered by a user or configured to be triggered periodically, without further limitation.

[0035] For example, in a telemedicine consultation scenario, when a patient initiates a consultation request, the patient describes the symptoms: "Doctor, I have been coughing a lot recently, especially at night, and I also have chest tightness." At this time, the audio acquisition device collects this voice and generates an initial voice waveform; among them, the initial voice waveform exists in the form of an analog signal, which contains information such as the patient's voice characteristics and environmental noise.

[0036] S120: Input the initial speech waveform into a preset encoder for encoding to obtain encoded high-dimensional speech information.

[0037] In this embodiment, after obtaining the initial speech waveform, the initial speech waveform is input into a preset encoder for encoding, wherein the preset encoder uses a deep learning model, such as a convolutional neural network or a Transformer architecture, to extract and transform key features of the initial speech waveform to remove redundant information in the initial speech signal, thereby generating high-dimensional speech information to facilitate subsequent quantization processing operations; wherein the high-dimensional speech information includes detailed features of the user's voice, such as pitch, timbre, speaking speed, emotional color, etc.

[0038] For example, in a telemedicine consultation scenario, accurate speech reconstruction is crucial for doctors to understand the patient's description of their symptoms. Therefore, by encoding the speech waveform input by the patient, silent segments or background noise in the patient's speech waveform can be removed to restore the naturalness and clarity of the speech, thereby reducing speech distortion.

[0039] In one embodiment, step S120 includes:

[0040] Performing a first preprocessing on the initial speech waveform using a preprocessing strategy to obtain a preprocessed speech waveform;

[0041] Extracting features from the preprocessed speech waveform based on a preset feature extraction network to obtain acoustic features corresponding to the preprocessed speech waveform;

[0042] The acoustic features are mapped to a high-dimensional latent space to obtain the high-dimensional speech information.

[0043] In this embodiment, since the initial speech waveform may contain some environmental noise, such as conversations in the user's surrounding environment, after obtaining the initial speech waveform, this embodiment first preprocesses the initial speech waveform using a preprocessing strategy to obtain a preprocessed speech waveform, thereby reducing noise interference. The preprocessing operation includes filtering, normalization, and other operations, which are not specifically limited here. In addition, in this embodiment, a convolutional neural network (CNN) is preferably used as the preset feature extraction network, so that key features of the preprocessed speech waveform are extracted based on the preset feature extraction network to obtain acoustic features that can comprehensively describe the acoustic characteristics of the user's speech. Finally, the acoustic features (e.g., 40 dimensions) are mapped to a high-dimensional latent space to obtain high-dimensional speech information (e.g., 1024 dimensions) containing rich speech information (e.g., emotional tone of the speech, speaking habits, etc.).

[0044] S130: Input the high-dimensional speech information into a preset quantizer for processing to obtain compressed discrete speech information.

[0045] In this embodiment, to reduce data volume and facilitate transmission within limited bandwidth, high-dimensional speech information is input into a pre-defined quantizer for processing. The pre-defined quantizer then quantizes and compresses the high-dimensional speech information to produce discrete speech information. For example, a segment of high-dimensional speech information that might have originally taken several minutes to transmit might only take tens of seconds after quantization and compression, enabling faster transmission of the speech information.

[0046] In one embodiment, the step S130 includes:

[0047] The high-dimensional voice information is quantized and compressed using a preset quantization algorithm to obtain the discrete voice information.

[0048] In this embodiment, the high-dimensional speech information is input into a preset quantizer for processing. Specifically, the high-dimensional speech information can be quantized and compressed by a preset quantization algorithm to obtain the discrete speech information; wherein, the preset quantization algorithm can be a scalar quantization algorithm, a vector quantization algorithm, etc., which is not specifically limited here.

[0049] For example, scalar quantization is performed on 1024-dimensional high-dimensional speech information. Assuming that the value range of a dimension in the 1024-dimensional high-dimensional speech information is 0 to 10, it is divided into 8 small intervals, such as [0, 1.25), [1.25, 2.5), ..., [8.75, 10]. Each small interval corresponds to a quantization level, such as 0.625, 1.875, ..., 9.375. When the speech information value of this dimension is 4.1, it falls in the fuzzy region between the intervals [2.5, 3.75) and [3.75, 5). At this time, according to the nearest principle, it is closer to the quantization level of the interval [3.75, 5) (for example, 4.375). Therefore, 4.375 is used to approximate the value of 4.1. Similarly, the discrete speech information of the 1024-dimensional high-dimensional speech information can be obtained.

[0050] Therefore, this embodiment can quickly compress the high-dimensional voice information through a preset quantization algorithm, thereby ensuring timely transmission of voice, such as remote medical consultation voice.

[0051] S140: Input the discrete voice information into a preset decoder for decoding to obtain a decoded first voice waveform.

[0052] In this embodiment, after obtaining discrete voice information, in order to restore the discrete voice information to a continuous voice waveform, this embodiment inputs the quantized and compressed discrete voice information into a preset decoder, and decodes the discrete voice information through the preset decoder to restore the continuity and partial features of the voice, thereby obtaining a first voice waveform. For example, in a telemedicine consultation scenario, the patient's voice describing symptoms (e.g., "Doctor, I have been coughing a lot recently, especially at night, and I have chest tightness") is encoded and quantized, and the decoder must restore it to a voice waveform close to the original so that the doctor can accurately understand the patient's description and make an accurate diagnosis.

[0053] In one embodiment, the step S140 includes:

[0054] Performing inverse quantization processing on the discrete speech information to convert it into continuous high-dimensional speech information;

[0055] Based on an inverse mapping network corresponding to the preset feature extraction network, the continuous high-dimensional speech information is mapped to a speech waveform space to obtain the first speech waveform.

[0056] In this embodiment, the discrete speech information is first dequantized to convert the discrete values ​​into continuous high-dimensional speech information, thereby restoring the continuity and some features of the original speech signal, such as pitch, timbre, speaking speed, etc., as much as possible; then, based on the inverse mapping network corresponding to the preset feature extraction network, the continuous high-dimensional speech information is mapped to the speech waveform space to obtain a reconstructed first speech waveform; wherein the inverse mapping network is the inverse process of the preset feature extraction network. Thus, the time domain characteristics of the speech signal, such as the start, end, amplitude change, etc. of the speech, can be restored through the inverse mapping network. For example, when processing the speech of a patient describing symptoms, the inverse mapping network can restore the sharp start and clear end of the cough sound, making the reconstructed first speech waveform closer to the initial speech waveform.

[0057] S150: Perform speech enhancement on the first speech waveform using a target stream matching model to generate an enhanced target speech.

[0058] In this embodiment, the first speech waveform obtained by encoding, quantizing, and decoding alone still has some deficiencies, such as noise interference from the surrounding environment during speech collection and transmission, and distortion during the encoding and decoding process. For example, a patient may describe their symptoms as follows: "Doctor, I've been coughing a lot lately, especially at night, and I have chest tightness." However, due to noise interference, the first speech waveform obtained after encoding, quantizing, and decoding may sound like "Doctor, I've been coughing a lot lately, especially at night, and I have chest tightness." The cough is partially obscured by the noise, and the overall speech is not clear or fluent.

[0059] Therefore, after obtaining the first speech waveform, to further improve the speech quality of the first speech waveform, this embodiment inputs the first speech waveform into a pre-trained target stream matching model to identify and remove random Gaussian noise components in the first speech waveform. The model also repairs and enhances the noise and distortion in the first speech waveform, thereby obtaining an enhanced speech representation that approximates the characteristics of real speech. The enhanced speech representation is then used as the target speech. The target stream matching model has pre-learned the relationship between random Gaussian noise and real speech.

[0060] In one embodiment, before step S150, the method further includes:

[0061] Acquire pre-stored historical speech waveforms from a preset database as a data training set according to a preset data acquisition strategy, and perform a second preprocessing on the data training set to obtain a second speech waveform set;

[0062] Generate a random Gaussian noise set with the same number and matching dimension as the historical speech waveform according to a preset noise strategy;

[0063] Constructing a stream matching model to be trained; wherein the stream matching model to be trained includes a condition input module, the condition input module is used to receive the speech waveform generated by the preset decoder as condition information;

[0064] The random Gaussian noise set and the second speech waveform set are used to perform model training on the stream matching model to be trained until the loss function in the stream matching model to be trained meets a preset training stop condition, then the model training is stopped and the target stream matching model is obtained.

[0065] In this embodiment, to obtain a target stream matching model, pre-stored historical speech waveforms are first obtained from a preset database according to a preset data collection strategy as a data training set. For example, historical speech waveforms within a certain time period are collected from the preset database according to the preset data collection strategy. These historical speech waveforms include speech from different users, covering different speakers, speech styles, and possible background noise. For example, some speech is recorded in a quiet environment, while others contain some ambient noise. Furthermore, the collected data training set is subjected to a second preprocessing to obtain a second speech waveform set. The second preprocessing includes operations such as encoding, quantization, and decoding. The second speech waveform set is a reconstructed speech waveform set obtained after encoding, quantization, and decoding each training set in the data training set. The second speech waveform set includes at least one speech waveform. Then, random Gaussian noise is generated according to a preset noise strategy, with the same number and matching dimensions as the historical speech waveforms. A stream matching model to be trained is constructed. The stream matching model to be trained includes a conditional input module, which is used to provide additional information guidance during model training and inference, enabling the model to generate speech waveforms that better meet expectations based on given conditions. Finally, the random Gaussian noise set and the second speech waveform set serving as conditional information are input into the stream matching model to be trained for processing to generate predicted speech. The predicted speech waveform is then compared with the actual speech to obtain a loss function value, which reflects the difference between the predicted and actual speech. The parameters of the stream matching model to be trained are then updated using a backpropagation algorithm based on the gradient of the loss function, ensuring that the speech generated by the model is closer to the actual speech. Furthermore, the aforementioned steps are repeated for multiple rounds of iterative training until the loss function of the stream matching model to be trained meets a preset model training stopping condition. At this point, model training is terminated and the target stream matching model is obtained.

[0066] In one embodiment, the step S150 further includes:

[0067] Obtaining a random Gaussian noise vector that matches the first speech waveform dimension;

[0068] Inputting the first speech waveform as a conditional variable into a target condition input module in the target stream matching model to extract a feature representation of the conditional variable;

[0069] Concatenating the feature representation with the random Gaussian noise vector to obtain a concatenated vector;

[0070] The concatenated vector is converted according to a preset number of iterative mapping times to obtain an enhanced speech representation, and the enhanced speech representation is used as the target speech.

[0071] In this embodiment, to improve the speech quality of a first speech waveform, this embodiment specifically generates a random Gaussian noise vector that matches the dimensions of the first speech waveform through a preset noise strategy, and inputs the first speech waveform as a conditional variable into the target stream matching model to extract a feature representation of the conditional variable. This feature representation includes speech frequency features, timbre features, intonation variation features, etc., which can comprehensively describe the essential attributes of the first speech waveform and help the model better understand the characteristics and contextual information of the current speech. For example, the pronunciation frequency and timbre variation of words such as "cough", "night", and "chest tightness" in the first speech waveform are analyzed to extract feature representations related to these semantics. The feature representation is then concatenated with the random Gaussian noise vector, allowing the model to simultaneously consider the characteristics of the original speech and the influence of noise, extracting effective speech information from the noise. For example, assuming that the feature representation is a 64-dimensional vector and the random Gaussian noise vector is a 128-dimensional vector, concatenating them yields a 192-dimensional concatenated vector. Finally, the concatenated vector is transformed according to a preset number of iterative mappings. In each iteration, the concatenated vector is nonlinearly transformed to gradually remove noise and enhance speech features. As a result, after multiple iterations, the speech details in the first speech waveform are more clearly presented, ultimately yielding a target speech with optimal speech quality. In this embodiment, the preset number of iterative mappings is preferably six, which significantly reduces computational overhead and improves inference speed.

[0072] In one embodiment, after step S150, the method further includes:

[0073] The residual noise in the target speech is suppressed using a preset noise reduction algorithm to obtain noise-reduced speech.

[0074] In this embodiment, after generating the enhanced target speech, although the target stream matching model can perform speech enhancement on the reconstructed first speech waveform to achieve the effect of improving speech quality, it is not specifically designed for various complex noises, resulting in some noise still existing in the target speech. Therefore, in order to further optimize the target speech, this embodiment also suppresses the residual noise in the target speech by using a preset noise reduction algorithm, thereby significantly improving the clarity of the speech. For example, in a telemedicine consultation scenario, after the noise reduction processing, when the patient describes the symptoms, the pronunciation of key words such as "coughing badly", "at night", and "chest tightness" is clearer, allowing the doctor to understand the patient's condition more accurately, thereby avoiding misdiagnosis due to noise interference.

[0075] It can be seen from the above technical solution that the present invention responds to the target audio receiving instruction and obtains the initial speech waveform corresponding to the target audio according to the target audio receiving instruction; inputs the initial speech waveform into a preset encoder for encoding to obtain high-dimensional speech information; inputs the high-dimensional speech information into a preset quantizer for processing to obtain discrete speech information; inputs the discrete speech information into a preset decoder for decoding to obtain a first speech waveform; and uses the target stream matching model to perform speech enhancement on the first speech waveform to generate an enhanced target speech, thereby achieving audio reconstruction with high perceptual quality at a low bit rate; at the same time, the target stream matching model can significantly reduce the computational overhead, improve the inference speed, and is suitable for general audio in the full frequency band, with strong versatility.

[0076] Figure 2 Schematic block diagram of an audio codec device based on stream matching provided by an embodiment of the present invention. Figure 2 As shown, the present invention also provides an audio codec device based on stream matching; the audio codec device based on stream matching includes a unit for executing the above-mentioned audio codec method based on stream matching, specifically, refer to Figure 2 The stream matching-based audio codec device 100 includes a response unit 110 , an encoding unit 120 , a quantization unit 130 , a decoding unit 140 and a speech enhancement unit 150 .

[0077] The responding unit 110 is configured to respond to a target audio receiving instruction and obtain an initial speech waveform corresponding to the target audio according to the target audio receiving instruction.

[0078] In this embodiment, upon receiving a target audio reception instruction, the server responds to the target audio reception instruction and, in accordance with the target audio reception instruction, acquires the target audio from an audio capture device and generates an initial speech waveform. The audio capture device may be a microphone, a pickup, or an integrated audio sensor, and the target audio reception instruction may be triggered by a user or configured to be triggered periodically, without further limitation.

[0079] For example, in a telemedicine consultation scenario, when a patient initiates a consultation request, the patient describes the symptoms: "Doctor, I have been coughing a lot recently, especially at night, and I also have chest tightness." At this time, the audio acquisition device collects this voice and generates an initial voice waveform; among them, the initial voice waveform exists in the form of an analog signal, which contains information such as the patient's voice characteristics and environmental noise.

[0080] The encoding unit 120 is configured to input the initial speech waveform into a preset encoder for encoding to obtain encoded high-dimensional speech information.

[0081] In this embodiment, after obtaining the initial speech waveform, the initial speech waveform is input into a preset encoder for encoding, wherein the preset encoder uses a deep learning model, such as a convolutional neural network or a Transformer architecture, to extract and transform key features of the initial speech waveform to remove redundant information in the initial speech signal, thereby generating high-dimensional speech information to facilitate subsequent quantization processing operations; wherein the high-dimensional speech information includes detailed features of the user's voice, such as pitch, timbre, speaking speed, emotional color, etc.

[0082] For example, in a telemedicine consultation scenario, accurate speech reconstruction is crucial for doctors to understand the patient's description of their symptoms. Therefore, by encoding the speech waveform input by the patient, silent segments or background noise in the patient's speech waveform can be removed to restore the naturalness and clarity of the speech, thereby reducing speech distortion.

[0083] In one embodiment, the encoding unit 120 is specifically configured to:

[0084] Performing a first preprocessing on the initial speech waveform using a preprocessing strategy to obtain a preprocessed speech waveform;

[0085] Extracting features from the preprocessed speech waveform based on a preset feature extraction network to obtain acoustic features corresponding to the preprocessed speech waveform;

[0086] The acoustic features are mapped to a high-dimensional latent space to obtain the high-dimensional speech information.

[0087] In this embodiment, since the initial speech waveform may contain some environmental noise, such as conversations in the user's surrounding environment, after obtaining the initial speech waveform, this embodiment first preprocesses the initial speech waveform using a preprocessing strategy to obtain a preprocessed speech waveform, thereby reducing noise interference. The preprocessing operation includes filtering, normalization, and other operations, which are not specifically limited here. In addition, in this embodiment, a convolutional neural network (CNN) is preferably used as the preset feature extraction network, so that key features of the preprocessed speech waveform are extracted based on the preset feature extraction network to obtain acoustic features that can comprehensively describe the acoustic characteristics of the user's speech. Finally, the acoustic features (e.g., 40 dimensions) are mapped to a high-dimensional latent space to obtain high-dimensional speech information (e.g., 1024 dimensions) containing rich speech information (e.g., emotional tone of the speech, speaking habits, etc.).

[0088] The quantization unit 130 is configured to input the high-dimensional speech information into a preset quantizer for processing to obtain compressed discrete speech information.

[0089] In this embodiment, to reduce data volume and facilitate transmission within limited bandwidth, high-dimensional speech information is input into a pre-defined quantizer for processing. The pre-defined quantizer then quantizes and compresses the high-dimensional speech information to produce discrete speech information. For example, a segment of high-dimensional speech information that might have originally taken several minutes to transmit might only take tens of seconds after quantization and compression, enabling faster transmission of the speech information.

[0090] In one embodiment, the quantization unit 130 is specifically configured to:

[0091] The high-dimensional voice information is quantized and compressed using a preset quantization algorithm to obtain the discrete voice information.

[0092] In this embodiment, the high-dimensional speech information is input into a preset quantizer for processing. Specifically, the high-dimensional speech information can be quantized and compressed by a preset quantization algorithm to obtain the discrete speech information; wherein, the preset quantization algorithm can be a scalar quantization algorithm, a vector quantization algorithm, etc., which is not specifically limited here.

[0093] For example, scalar quantization is performed on 1024-dimensional high-dimensional speech information. Assuming that the value range of a dimension in the 1024-dimensional high-dimensional speech information is 0 to 10, it is divided into 8 small intervals, such as [0, 1.25), [1.25, 2.5), ..., [8.75, 10]. Each small interval corresponds to a quantization level, such as 0.625, 1.875, ..., 9.375. When the speech information value of this dimension is 4.1, it falls in the fuzzy region between the intervals [2.5, 3.75) and [3.75, 5). At this time, according to the nearest principle, it is closer to the quantization level of the interval [3.75, 5) (for example, 4.375). Therefore, 4.375 is used to approximate the value of 4.1. Similarly, the discrete speech information of the 1024-dimensional high-dimensional speech information can be obtained.

[0094] Therefore, this embodiment can quickly compress the high-dimensional voice information through a preset quantization algorithm, thereby ensuring timely transmission of voice, such as remote medical consultation voice.

[0095] The decoding unit 140 is configured to input the discrete speech information into a preset decoder for decoding to obtain a decoded first speech waveform.

[0096] In this embodiment, after obtaining discrete voice information, in order to restore the discrete voice information to a continuous voice waveform, this embodiment inputs the quantized and compressed discrete voice information into a preset decoder, and decodes the discrete voice information through the preset decoder to restore the continuity and partial features of the voice, thereby obtaining a first voice waveform. For example, in a telemedicine consultation scenario, the patient's voice describing symptoms (e.g., "Doctor, I have been coughing a lot recently, especially at night, and I have chest tightness") is encoded and quantized, and the decoder must restore it to a voice waveform close to the original so that the doctor can accurately understand the patient's description and make an accurate diagnosis.

[0097] In one embodiment, the decoding unit 140 is specifically configured to:

[0098] Performing inverse quantization processing on the discrete speech information to convert it into continuous high-dimensional speech information;

[0099] Based on an inverse mapping network corresponding to the preset feature extraction network, the continuous high-dimensional speech information is mapped to a speech waveform space to obtain the first speech waveform.

[0100] In this embodiment, the discrete speech information is first dequantized to remap the discrete values ​​back to continuous high-dimensional speech information, thereby restoring the continuity and some features of the original speech signal, such as pitch, timbre, and speaking speed, as much as possible; then, based on an inverse mapping network corresponding to the preset feature extraction network, the continuous high-dimensional speech information is mapped to the speech waveform space to obtain a reconstructed first speech waveform; wherein the inverse mapping network is the inverse process of the preset feature extraction network. Thus, the time domain characteristics of the speech signal, such as the start, end, and amplitude change of the speech, can be restored through the inverse mapping network. For example, when processing the speech of a patient describing symptoms, the inverse mapping network can restore the sharp start and clear end of the cough sound, making the reconstructed first speech waveform closer to the initial speech waveform.

[0101] The speech enhancement unit 150 is configured to perform speech enhancement on the first speech waveform using a target stream matching model to generate an enhanced target speech.

[0102] In this embodiment, the first speech waveform obtained by encoding, quantizing, and decoding alone still has some deficiencies, such as noise interference from the surrounding environment during speech collection and transmission, and distortion during the encoding and decoding process. For example, a patient may describe their symptoms as follows: "Doctor, I've been coughing a lot lately, especially at night, and I have chest tightness." However, due to noise interference, the first speech waveform obtained after encoding, quantizing, and decoding may sound like "Doctor, I've been coughing a lot lately, especially at night, and I have chest tightness." The cough is partially obscured by the noise, and the overall speech is not clear or fluent.

[0103] Therefore, after obtaining the first speech waveform, to further improve the speech quality of the first speech waveform, this embodiment inputs the first speech waveform into a pre-trained target stream matching model to identify and remove random Gaussian noise components in the first speech waveform. The model also repairs and enhances the noise and distortion in the first speech waveform, thereby obtaining an enhanced speech representation that approximates the characteristics of real speech. The enhanced speech representation is then used as the target speech. The target stream matching model has pre-learned the relationship between random Gaussian noise and real speech.

[0104] In one embodiment, the speech enhancement unit 150 is further configured to:

[0105] Acquire pre-stored historical speech waveforms from a preset database as a data training set according to a preset data acquisition strategy, and perform a second preprocessing on the data training set to obtain a second speech waveform set;

[0106] Generate a random Gaussian noise set with the same number and matching dimension as the historical speech waveform according to a preset noise strategy;

[0107] Constructing a stream matching model to be trained; wherein the stream matching model to be trained includes a condition input module, the condition input module is used to receive the speech waveform generated by the preset decoder as condition information;

[0108] The random Gaussian noise set and the second speech waveform set are used to perform model training on the stream matching model to be trained until the loss function in the stream matching model to be trained meets a preset training stop condition, then the model training is stopped and the target stream matching model is obtained.

[0109] In this embodiment, to obtain a target stream matching model, pre-stored historical speech waveforms are first obtained from a preset database according to a preset data collection strategy as a data training set. For example, historical speech waveforms within a certain time period are collected from the preset database according to the preset data collection strategy. These historical speech waveforms include speech from different users, covering different speakers, speech styles, and possible background noise. For example, some speech is recorded in a quiet environment, while others contain some ambient noise. Furthermore, the collected data training set is subjected to a second preprocessing to obtain a second speech waveform set. The second preprocessing includes operations such as encoding, quantization, and decoding. The second speech waveform set is a reconstructed speech waveform set obtained after encoding, quantization, and decoding each training set in the data training set. The second speech waveform set includes at least one speech waveform. Then, random Gaussian noise is generated according to a preset noise strategy, with the same number and matching dimensions as the historical speech waveforms. A stream matching model to be trained is constructed. The stream matching model to be trained includes a conditional input module, which is used to provide additional information guidance during model training and inference, enabling the model to generate speech waveforms that better meet expectations based on given conditions. Finally, the random Gaussian noise set and the second speech waveform set serving as conditional information are input into the stream matching model to be trained for processing to generate predicted speech. The predicted speech waveform is then compared with the actual speech to obtain a loss function value, which reflects the difference between the predicted and actual speech. The parameters of the stream matching model to be trained are then updated using a backpropagation algorithm based on the gradient of the loss function, ensuring that the speech generated by the model is closer to the actual speech. Furthermore, the aforementioned steps are repeated for multiple rounds of iterative training until the loss function of the stream matching model to be trained meets a preset model training stopping condition. At this point, model training is terminated and the target stream matching model is obtained.

[0110] In one embodiment, the speech enhancement unit 150 is further configured to:

[0111] Obtaining a random Gaussian noise vector that matches the first speech waveform dimension;

[0112] Inputting the first speech waveform as a conditional variable into a target condition input module in the target stream matching model to extract a feature representation of the conditional variable;

[0113] Concatenating the feature representation with the random Gaussian noise vector to obtain a concatenated vector;

[0114] The concatenated vector is converted according to a preset number of iterative mapping times to obtain an enhanced speech representation, and the enhanced speech representation is used as the target speech.

[0115] In this embodiment, to improve the speech quality of a first speech waveform, this embodiment specifically generates a random Gaussian noise vector that matches the dimensions of the first speech waveform through a preset noise strategy, and inputs the first speech waveform as a conditional variable into the target stream matching model to extract a feature representation of the conditional variable. This feature representation includes speech frequency features, timbre features, intonation variation features, etc., which can comprehensively describe the essential attributes of the first speech waveform and help the model better understand the characteristics and contextual information of the current speech. For example, the pronunciation frequency and timbre variation of words such as "cough", "night", and "chest tightness" in the first speech waveform are analyzed to extract feature representations related to these semantics. The feature representation is then concatenated with the random Gaussian noise vector, allowing the model to simultaneously consider the characteristics of the original speech and the influence of noise, extracting effective speech information from the noise. For example, assuming that the feature representation is a 64-dimensional vector and the random Gaussian noise vector is a 128-dimensional vector, concatenating them yields a 192-dimensional concatenated vector. Finally, the concatenated vector is transformed according to a preset number of iterative mappings. In each iteration, the concatenated vector is nonlinearly transformed to gradually remove noise and enhance speech features. As a result, after multiple iterations, the speech details in the first speech waveform are more clearly presented, ultimately yielding a target speech with optimal speech quality. In this embodiment, the preset number of iterative mappings is preferably six, which significantly reduces computational overhead and improves inference speed.

[0116] In one embodiment, the speech enhancement unit 150 is further configured to:

[0117] The residual noise in the target speech is suppressed using a preset noise reduction algorithm to obtain noise-reduced speech.

[0118] In this embodiment, after generating the enhanced target speech, although the target stream matching model can perform speech enhancement on the reconstructed first speech waveform to achieve the effect of improving speech quality, it is not specifically designed for various complex noises, resulting in some noise still existing in the target speech. Therefore, in order to further optimize the target speech, this embodiment also suppresses the residual noise in the target speech by using a preset noise reduction algorithm, thereby significantly improving the clarity of the speech. For example, in a telemedicine consultation scenario, after the noise reduction processing, when the patient describes the symptoms, the pronunciation of key words such as "coughing badly", "at night", and "chest tightness" is clearer, allowing the doctor to understand the patient's condition more accurately, thereby avoiding misdiagnosis due to noise interference.

[0119] It can be seen from the above technical solution that the present invention responds to the target audio receiving instruction and obtains the initial speech waveform corresponding to the target audio according to the target audio receiving instruction; inputs the initial speech waveform into a preset encoder for encoding to obtain high-dimensional speech information; inputs the high-dimensional speech information into a preset quantizer for processing to obtain discrete speech information; inputs the discrete speech information into a preset decoder for decoding to obtain a first speech waveform; and uses the target stream matching model to perform speech enhancement on the first speech waveform to generate an enhanced target speech, thereby achieving audio reconstruction with high perceptual quality at a low bit rate; at the same time, the target stream matching model can significantly reduce the computational overhead, improve the inference speed, and is suitable for general audio in the full frequency band, with strong versatility.

[0120] The above-mentioned audio codec device based on stream matching can be implemented in the form of a computer program. The computer program can be used in Figure 3 Runs on the computer equipment shown.

[0121] See also Figure 3 , Figure 3 is a schematic block diagram of a computer device provided in an embodiment of the present invention. Computer device 500 is a server, which can be a standalone server or a server cluster consisting of multiple servers. Computer device 500 can also be a transmitter, which can be a communication-capable electronic device such as a smartphone, tablet computer, laptop computer, desktop computer, personal digital assistant, or wearable device.

[0122] See Figure 3 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .

[0123] The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute an audio encoding and decoding method based on stream matching.

[0124] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0125] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the audio encoding and decoding method based on stream matching.

[0126] The network interface 505 is used for network communication, such as providing data information transmission. Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0127] The processor 502 is configured to run a computer program 5032 stored in the memory to implement the stream matching-based audio encoding and decoding method disclosed in the embodiment of the present invention.

[0128] Those skilled in the art will understand that Figure 3 The embodiment of the computer device shown in the figure does not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and processor are the same as those in the figure. Figure 3 The embodiments shown are consistent and will not be described again here.

[0129] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0130] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the stream matching-based audio encoding and decoding method disclosed in an embodiment of the present invention is implemented.

[0131] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the devices, systems and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0132] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, or units with the same function may be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, systems or units, or may be an electrical, mechanical or other form of connection.

[0133] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the embodiments of the present invention.

[0134] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0136] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An audio encoding and decoding method based on stream matching, characterized in that: The method comprises: Responding to a target audio receiving instruction, and acquiring an initial speech waveform corresponding to the target audio according to the target audio receiving instruction; Inputting the initial speech waveform into a preset encoder for encoding to obtain encoded high-dimensional speech information; Inputting the high-dimensional speech information into a preset quantizer for processing to obtain compressed discrete speech information; Inputting the discrete voice information into a preset decoder for decoding to obtain a decoded first voice waveform; The target stream matching model is used to perform speech enhancement on the first speech waveform to generate an enhanced target speech.

2. The audio encoding and decoding method based on stream matching according to claim 1, characterized in that The step of inputting the initial speech waveform into a preset encoder for encoding to obtain encoded high-dimensional speech information includes: Performing a first preprocessing on the initial speech waveform using a preprocessing strategy to obtain a preprocessed speech waveform; Extracting features from the preprocessed speech waveform based on a preset feature extraction network to obtain acoustic features corresponding to the preprocessed speech waveform; The acoustic features are mapped to a high-dimensional latent space to obtain the high-dimensional speech information.

3. The audio encoding and decoding method based on stream matching according to claim 1, characterized in that The step of inputting the high-dimensional speech information into a preset quantizer for processing to obtain compressed discrete speech information includes: The high-dimensional voice information is quantized and compressed using a preset quantization algorithm to obtain the discrete voice information.

4. The audio encoding and decoding method based on stream matching according to claim 2, characterized in that The step of inputting the discrete voice information into a preset decoder for decoding to obtain a decoded first voice waveform includes: Performing inverse quantization processing on the discrete speech information to convert it into continuous high-dimensional speech information; Based on an inverse mapping network corresponding to the preset feature extraction network, the continuous high-dimensional speech information is mapped to a speech waveform space to obtain the first speech waveform.

5. The audio encoding and decoding method based on stream matching according to claim 1, characterized in that Before performing speech enhancement on the first speech waveform using the target stream matching model to generate the enhanced target speech, the method further includes: Acquire pre-stored historical speech waveforms from a preset database as a data training set according to a preset data acquisition strategy, and perform a second preprocessing on the data training set to obtain a second speech waveform set; Generate a random Gaussian noise set with the same number and matching dimension as the historical speech waveform according to a preset noise strategy; Constructing a stream matching model to be trained; wherein the stream matching model to be trained includes a condition input module, the condition input module is used to receive the speech waveform generated by the preset decoder as condition information; The random Gaussian noise set and the second speech waveform set are used to perform model training on the stream matching model to be trained until the loss function in the stream matching model to be trained meets a preset training stop condition, then the model training is stopped and the target stream matching model is obtained.

6. The audio encoding and decoding method based on stream matching according to claim 5, characterized in that The performing speech enhancement on the first speech waveform by using the target stream matching model to generate an enhanced target speech includes: Obtaining a random Gaussian noise vector that matches the first speech waveform dimension; Inputting the first speech waveform as a conditional variable into a target condition input module in the target stream matching model to extract a feature representation of the conditional variable; Concatenating the feature representation with the random Gaussian noise vector to obtain a concatenated vector; The concatenated vector is converted according to a preset number of iterative mapping times to obtain an enhanced speech representation, and the enhanced speech representation is used as the target speech.

7. The audio encoding and decoding method based on stream matching according to claim 5, characterized in that: After performing speech enhancement on the first speech waveform using the target stream matching model to generate an enhanced target speech, the method further includes: The residual noise in the target speech is suppressed using a preset noise reduction algorithm to obtain noise-reduced speech.

8. An audio codec device based on stream matching, characterized in that: The device comprises: a response unit, configured to respond to a target audio receiving instruction and acquire an initial speech waveform corresponding to the target audio according to the target audio receiving instruction; An encoding unit, configured to input the initial speech waveform into a preset encoder for encoding to obtain encoded high-dimensional speech information; a quantization unit, configured to input the high-dimensional speech information into a preset quantizer for processing to obtain compressed discrete speech information; a decoding unit, configured to input the discrete voice information into a preset decoder for decoding to obtain a decoded first voice waveform; The speech enhancement unit is used to perform speech enhancement on the first speech waveform using a target stream matching model to generate an enhanced target speech.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the stream matching-based audio encoding and decoding method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the audio encoding and decoding method based on stream matching according to any one of claims 1 to 7 can be implemented.

Images