Speech processing methods, devices, systems, equipment and storage media

Abstract

This application provides a speech processing method, apparatus, system, device, and storage medium. The method includes: extracting features from speech data to be encoded to generate first speech features; determining encoded speech data based on the first speech features; and training a generative adversarial network (GAN) based on the encoded speech data, wherein the GAN is used to decode the speech data. This solves the problem that existing technologies cannot guarantee high-fidelity audio quality while achieving high-efficiency, low-consumption audio encoding and decoding.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a voice processing method, apparatus, system, device, and storage medium. Background Technology

[0002] With the development of information technology, people's demand for real-time communication is constantly increasing and has gradually become an indispensable part of work and life. However, the massive amount of audio and video call minutes poses a huge challenge to the Internet infrastructure.

[0003] Currently, weak network conditions still exist in the network environment. Therefore, in order to provide a high-quality audio experience with limited bandwidth, autoregressive models or end-to-end neural audio codecs are generally used. However, autoregressive models cannot be parallelized, resulting in low efficiency and high overhead. Although end-to-end neural audio codecs use codecs that can be parallelized, they directly encode the input speech, leading to relatively high overall overhead.

[0004] Therefore, existing technologies cannot guarantee high-fidelity audio quality while achieving high-efficiency, low-consumption audio encoding and decoding. Summary of the Invention

[0005] This application provides a voice processing method, apparatus, system, device, and storage medium to solve the problem that existing technologies cannot guarantee high-fidelity audio quality while achieving high-efficiency and low-consumption audio encoding and decoding.

[0006] In a first aspect, embodiments of this application provide a voice processing method, the method comprising:

[0007] Feature extraction is performed on the speech data to be encoded to generate the first speech feature;

[0008] Based on the first speech feature, the encoded speech data is determined;

[0009] A generative adversarial network is trained based on the encoded speech data, and the generative adversarial network is used to decode the speech data.

[0010] Optionally, the step of extracting features from the speech data to be encoded to generate first speech features includes:

[0011] If the speech data to be encoded is a frequency domain speech signal, then the frequency domain speech signal is converted into a time domain speech signal, and features are extracted from the time domain speech signal to obtain the Mel spectrum;

[0012] If the speech data to be encoded is the time-domain speech signal, then feature extraction is performed on the time-domain speech signal to obtain the Mel spectrum;

[0013] The Mel spectrum is used to represent the first speech feature.

[0014] Optionally, determining the encoded speech data based on the first speech feature includes:

[0015] Scene detection is performed on the first speech feature to determine the target scene that matches the first speech feature, and a second speech feature corresponding to the target scene is obtained.

[0016] Based on the second speech feature, the encoded speech data is determined.

[0017] Optionally, determining the encoded speech data based on the second speech feature includes:

[0018] The second speech feature is subjected to residual quantization to obtain the quantized speech feature, as well as a codebook and the index value corresponding to the codebook;

[0019] The quantized speech features are used as encoded speech data;

[0020] The codebook is used to look up the quantized speech features using index values.

[0021] Optionally, the generative adversarial network includes a generator network and a discriminator network. The generator network is used to decode the encoded speech data, and the discriminator network is used to distinguish the decoded speech data from the speech data to be encoded, so as to update the generator network. The speech data to be encoded is multiple, and correspondingly, the encoded speech data is multiple. Training the generative adversarial network based on the encoded speech data includes:

[0022] The encoded speech data is divided into multiple training sample sets, and the following steps are repeated for each training sample set until the training stop condition is met to stop the training of the generator network and the discriminator network:

[0023] The training sample set is input into the generator network to train and obtain the decoded speech data corresponding to each encoded speech data in the training sample set;

[0024] The decoded speech data and the speech data to be encoded corresponding to each encoded speech data in the training sample set are input into the discriminator network to calculate the loss value of the generative adversarial network.

[0025] The generator network and the discriminator network are updated using the loss value.

[0026] Secondly, embodiments of this application provide a voice processing method applied at a transmitting end; the method includes:

[0027] The target speech data to be encoded is determined, and features are extracted from the target speech data to be encoded to obtain the first target speech features;

[0028] The first target speech features are encoded to obtain encoded target speech data;

[0029] Based on the encoded target speech data, an instruction message is sent to the receiving end. The instruction message is used to instruct the receiving end to determine the encoded target speech data according to the encapsulation information in the instruction message, and to obtain the decoded target speech data through a generative adversarial network based on the target speech data.

[0030] Thirdly, embodiments of this application provide a voice processing method applied at a receiving end; the method includes:

[0031] Receive the instruction message sent by the sender;

[0032] The encapsulation information in the instruction message is parsed to obtain the encoded target speech data. The encoded target speech data is obtained by the receiving end through feature extraction of the target speech data to be encoded to obtain the first target speech feature, and then encoding the first target speech feature.

[0033] Based on the encoded target speech data, the decoded target speech data is obtained through a generative adversarial network.

[0034] Fourthly, embodiments of this application provide a voice processing apparatus, the apparatus comprising:

[0035] The feature extraction module is used to extract features from the speech data to be encoded and generate the first speech features;

[0036] The encoding module is used to determine the encoded speech data based on the first speech feature;

[0037] The training module is used to train a generative adversarial network based on the encoded speech data, and the generative adversarial network is used to decode the speech data.

[0038] Fifthly, embodiments of this application provide a voice processing device applied at a transmitting end, the device comprising:

[0039] The processing module is used to determine the target speech data to be encoded and to extract features from the target speech data to be encoded to obtain the first target speech features;

[0040] The encoding module is used to encode the first target speech features to obtain encoded target speech data;

[0041] The sending module is used to send an indication message to the receiving end based on the encoded target speech data. The indication message is used to instruct the receiving end to determine the encoded target speech data based on the encapsulation information in the indication message, and to obtain the decoded target speech data through a generative adversarial network based on the target speech data.

[0042] Sixthly, embodiments of this application provide a voice processing device applied at a receiving end, the device comprising:

[0043] The receiving module is used to receive indication messages sent by the sending end;

[0044] The processing module is used to parse the encapsulation information in the indication message to obtain the encoded target speech data, which is obtained by the receiving end through encoding the target speech data to be encoded.

[0045] The decoding module is used to obtain decoded target speech data from the encoded target speech data through a generative adversarial network.

[0046] Seventhly, embodiments of this application provide a voice processing system, the system comprising:

[0047] The transmitting end and the receiving end are communicatively connected to achieve end-to-end transmission;

[0048] The transmitting end is used to implement the method as described in any of the second aspects;

[0049] The receiving end is used to implement the method as described in any of the third aspects;

[0050] The sending end is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments.

[0051] The receiving end is equipped with a packet loss compensation post-processing function, which records the current state and predicts the audio frames with lost features when voice features are lost, thereby determining the lost features.

[0052] Eighthly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0053] The memory stores computer-executed instructions;

[0054] The processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects.

[0055] Ninthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the first aspects.

[0056] In a tenth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the first aspects.

[0057] The speech processing method, apparatus, system, device, and storage medium provided in this application extract features from the speech data to be encoded, encode the extracted features, and then train a generative adversarial network (GAN) with the encoded speech data to achieve decoding. By extracting speech features and then encoding them, unlike existing technologies that directly encode speech, the method reduces the feature dimensionality, avoids excessive resource consumption, and provides a foundation for supporting ultra-low bitrate transmission. The GAN is then trained using a parallel processing approach, improving operational efficiency. Furthermore, adversarial training is more conducive to generating natural and realistic decoded speech with higher accuracy, thus achieving high-fidelity speech quality. Therefore, while ensuring high-fidelity audio quality, it achieves high-efficiency and low-resource audio encoding and decoding. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1A This is a schematic diagram of a system architecture in which the technical solution of this application is applied;

[0060] Figure 1B A schematic diagram of a scenario for the speech processing method provided in an embodiment of this application;

[0061] Figure 2 A schematic flowchart illustrating the speech processing method provided in an embodiment of this application;

[0062] Figure 3 A schematic flowchart of a speech processing method provided in another embodiment of this application;

[0063] Figure 4 A schematic flowchart of a speech processing method provided in another embodiment of this application;

[0064] Figure 5 A schematic flowchart of a speech processing method provided in another embodiment of this application;

[0065] Figure 6 A schematic flowchart of a speech processing method provided in another embodiment of this application;

[0066] Figure 7 This is a schematic diagram of the structure of the voice processing device provided in the embodiments of this application;

[0067] Figure 8 This is a schematic diagram of the structure of a voice processing device provided in another embodiment of this application;

[0068] Figure 9 This is a schematic diagram of the structure of a voice processing device provided in another embodiment of this application;

[0069] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0071] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can also include other sequential examples besides those illustrated or described. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] Currently, weak network conditions still exist in the network environment. Therefore, in order to provide a high-quality audio experience with limited bandwidth, autoregressive models or end-to-end neural audio codecs are generally used. However, existing technologies cannot guarantee high-fidelity audio quality while achieving high-efficiency, low-power audio encoding and decoding.

[0073] To address the aforementioned issues, the inventive concept of this application is as follows: First, feature extraction is performed on the speech data to be encoded to reduce the data dimensionality. Then, the extracted features are encoded. By reducing the feature dimensionality, the problem of excessive resource consumption is avoided, while supporting ultra-low bitrate transmission and adapting to various network environments (such as weak network environments). Next, the network is trained using generative adversarial training, which supports parallel processing and thus improves operating efficiency. At the same time, adversarial training is also more conducive to the generation of natural and realistic decoded speech with higher accuracy, thereby achieving high-fidelity speech quality.

[0074] The technical solution of this application is applied to communication scenarios, and in a practical application, it is particularly suitable for communication scenarios based on RTC (Real-Time Communication) technology.

[0075] RTC technology refers to a communication technology capable of sending and receiving text, audio, and video in real time. It is suitable for scenarios such as live streaming, video-on-demand, video conferencing, online classrooms, online chat rooms, and interactive games, enabling real-time transmission of pure audio and video data. The technical solution of this application can be specifically applied to communication scenarios such as live streaming, video-on-demand, video conferencing, online classrooms, online chat rooms, and interactive games based on RTC.

[0076] See Figure 1A This diagram illustrates a system architecture in which the technical solution of this application can be applied. The system may include a server 100 and multiple clients 200. Multiple clients 200 can establish communication connections through the server 100. In an RTC scenario, the server 100 provides RTC services between the multiple clients 200, and each client 200 can act as a sender or receiver, achieving real-time communication through the server 100.

[0077] Users can interact with server 100 through client 200 to receive data sent by other clients 200, or send data to other clients 200. In RTC scenarios, users can publish data streams to server 100 through client 200, and server 200 can push the data stream to clients subscribed to that data stream. Data streams can be media data such as audio or video streams. For example, in a live streaming scenario, broadcasters can collect media data in real time through their clients and send it to the server. Media data from different broadcasters is differentiated by live streaming room, and the server can push the media data of a particular broadcaster to viewers who have entered that broadcaster's corresponding live streaming room. Similarly, in a conference scenario, participants can collect media data in real time through their clients and send it to the server, and the server can push the media data sent by each client to the clients of other participants.

[0078] The data transmitted by the client 200 may need to be processed such as encoding, transcoding, and compression before being published to the server 100. The data may also be subjected to interference processing in accordance with the technical solutions of this application embodiment, which will be described in detail below.

[0079] In this setup, client 200 and server 100 establish a connection via a network. The network provides the medium for communication between the client and server. The network can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0080] The client 200 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The client 200 can be developed based on the corresponding service SDK (Software Development Kit) provided by the server, such as the RTC SDK. The client 200 can be deployed on electronic devices and depends on the device or certain apps on the device to run. Electronic devices can have displays and support information browsing, such as personal mobile terminals like mobile phones, tablets, and personal computers. Various other types of applications can also be configured on electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platforms.

[0081] Server 100 may include servers that provide various services, such as servers that provide communication services for multiple clients, servers that support the models used on the clients for background training, and servers that process the data sent by the clients.

[0082] It should be noted that the server-side 100 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0083] It should be noted that the voice processing method provided in the embodiments of this application is generally executed by the server, and the corresponding voice processing device is generally located in the server. However, in other embodiments of this application, the client may also have similar functions to the server, thereby executing the voice processing method provided in the embodiments of this application. In other embodiments, the voice processing method provided in the embodiments of this application may also be executed jointly by the client and the server.

[0084] The technical solutions of this application will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0085] In this embodiment, speech processing can include two parts: a training process and a deployment process (application process). The training process is performed on a server, meaning the server is the execution entity. On the server, operations such as feature extraction, encoding, and generative adversarial network (GAN) training can be executed. During training, a trained generator network and discriminator network, a trained codebook, and the corresponding index values ​​are obtained. The deployment process supports end-to-end audio encoding and decoding. The "end" can include a transmitter and a receiver, both of which are equipped with encoding and decoding functions (including feature extraction modules, scene detection modules, residual quantization modules, and a generator network; capable of encoding and decoding). Specifically, they are configured with a trained codebook, corresponding index values, and a generator network, thus realizing the model trained in the above training process and its corresponding functions. Both the transmitter and receiver can be terminals, servers, etc. Terminals can be mobile phones, tablets, computers, etc., and the specific carriers of the transmitter and receiver are not specifically limited here.

[0086] For example, taking the deployment process as an example, Figure 1B This is a schematic diagram illustrating a scenario of the voice processing method provided in this application embodiment. Terminal 1 and Terminal 2 can function as either receivers or senders, depending on the specific call scenario. The call scenario here can be a real-time communication scenario such as a voice conference. Real-time communication scenario: A real-time communication system that allows two or more people to exchange text messages, files, voice, and video messages in real time over a network. Here, it may specifically refer to voice or video communication.

[0087] For example, in a voice conferencing scenario, taking terminal 1 as the sender and terminal 2 as the receiver, the encoding and decoding process can be implemented as follows:

[0088] User 1 inputs or generates speech data (i.e., speech signal to be encoded) using terminal 1. Terminal 1 then extracts features from the speech signal using a feature extraction module to obtain the first speech feature, which can be a Mel spectrum (i.e., a mel spectrum) supporting ultra-low bitrate transmission. The extracted features (i.e., the first speech feature, such as the mel spectrum) are then encoded. For example, a scene detection module distinguishes between human voice (single voice, multi-voice) scenes, music scenes, and reverberation scenes to obtain the analyzed speech features. Finally, a residual quantization module, such as a Residual Vector Quantizer (RVQ), performs residual quantization on the analyzed speech features to obtain the quantized speech features, i.e., the encoded speech data (or encoded speech features). The network model (i.e., the feature extraction module, scene detection module, and residual quantization module) for the above feature extraction and encoding processes supports adaptive training during training, continuously updating the network model parameters.

[0089] Then, terminal 1 uses the codebook to look up the quantized speech features, determines the matching index value, and packages the index value into a bit stream for transmission over the network to terminal 2. Terminal 2 receives the bit stream, unpacks it to recover the index value, and, combined with the codebook, re-obtains the speech features analyzed by the scene detection module and / or the quantized speech features. The quantized speech features are then fed into the generator network to obtain the decoded speech data (the decoded speech signal), which is then played back to user 2 using terminal 2. In this way, terminal 1 and terminal 2 complete one speech encoding / decoding cycle.

[0090] Therefore, in this embodiment, feature extraction is first performed on the speech to obtain speech features. Unlike existing technologies that directly encode speech, this reduces the feature dimensionality and avoids excessive resource consumption. Furthermore, the significantly reduced feature dimensionality provides a foundation for supporting ultra-low bitrate transmissions (e.g., below 6kbps), making it suitable for various network environments (e.g., weak network environments). Then, generative adversarial training is used to train the network, supporting parallel processing and improving operational efficiency. Adversarial training also facilitates the generation of more natural and realistic decoded speech with higher accuracy, thus achieving high-fidelity speech quality.

[0091] Figure 2 This is a flowchart illustrating the speech processing method provided in an embodiment of this application. The method in this embodiment can be executed by a server. Figure 2 As shown, the method in this embodiment may include:

[0092] S101. Extract features from the speech data to be encoded to generate the first speech feature.

[0093] The first speech feature is used to support ultra-low bitrate transmission. Ultra-low bitrate generally refers to audio encoding bitrates below 6kbps in real-time communication scenarios.

[0094] In this embodiment, during training, a feature extraction network layer is first trained to perform feature extraction on the speech data to be encoded. This feature extraction network layer can serve as a feature extraction module, and the speech data can be either time-domain or frequency-domain speech. Extracting features from the speech data using this module reduces the processing dimensionality of the encoding, thereby supporting ultra-low bitrate transmission and ensuring high-fidelity speech quality even in weak network environments. High-fidelity audio quality (i.e., high-fidelity speech quality) refers to the decoded sound signal being infinitely close to the original sound signal after network transmission, to the point where the difference is imperceptible to the human ear.

[0095] S102. Determine the encoded speech data based on the first speech feature.

[0096] In this embodiment, the extracted first speech features undergo a series of processing steps, such as scene detection and residual quantization; this processing is the encoding process. The encoded speech data obtained after processing the first speech features is used as the input for training the generative adversarial network (GAN) to perform training operations.

[0097] During the encoding process, a codebook and its corresponding index value can be generated. In the application process, the index value can be directly encapsulated in a bit stream and sent to the receiving end. The receiving end obtains the index value based on the unpacking and, combined with the codebook, can find the corresponding encoded speech features, reducing resource consumption in the network environment. Especially in weak network scenarios, it can support ultra-low bit rate transmission.

[0098] S103. Based on the encoded speech data, a generative adversarial network is trained, and the generative adversarial network is used to decode the speech data.

[0099] Generative adversarial networks (GANs) are a training strategy that can be used for unsupervised learning, semi-supervised learning, supervised learning, and reinforcement learning.

[0100] In this embodiment, there are multiple speech data to be encoded, and correspondingly, multiple encoded speech data. The generative adversarial network (GAN) can be trained using these multiple encoded speech data. The GAN includes a generator network and a discriminator network. The generator network decodes the encoded speech data, and the discriminator network distinguishes the decoded speech data from the speech data to be encoded, thereby updating the generator network. Training is iterated continuously until a training stopping condition is met.

[0101] The speech processing method provided in this application extracts features from the speech data to be encoded, encodes the extracted features, and then trains a generative adversarial network (GAN) with the encoded speech data to achieve decoding. By extracting speech features and then encoding them, unlike existing technologies that directly encode speech, the feature dimensionality is reduced, avoiding excessive resource consumption. Furthermore, the significantly reduced feature dimensionality provides a foundation for supporting ultra-low bitrate transmission. The GAN is then trained in parallel, improving operational efficiency. Adversarial training also facilitates the generation of more natural and realistic decoded speech with higher accuracy, thus achieving high-fidelity speech quality. Therefore, while ensuring high-fidelity audio quality, it achieves high-efficiency, low-resource audio encoding and decoding.

[0102] Optionally, the step of extracting features from the speech data to be encoded to generate the first speech features can be achieved through the following steps:

[0103] Step a1: If the speech data to be encoded is a frequency domain speech signal, then the frequency domain speech signal is converted into a time domain speech signal, and features are extracted from the time domain speech signal to obtain the Mel spectrum.

[0104] Step a2: If the speech data to be encoded is the time-domain speech signal, then feature extraction is performed on the time-domain speech signal to obtain the Mel spectrum.

[0105] The Mel spectrum is used to represent the first speech feature.

[0106] In this embodiment, if the speech data to be encoded is a time-domain speech signal, the feature extraction module can be used to directly extract features from the time-domain speech signal to obtain the mel spectrum; if the speech data to be encoded is a frequency-domain speech signal, the frequency-domain speech signal needs to be converted into a time-domain speech signal before the feature extraction module is used to extract features from the time-domain speech signal to obtain the mel spectrum.

[0107] Specifically, speech features are extracted to obtain speech features, namely the MEL spectrum. Based on the characteristics of the MEL spectrum, it is used as the feature to be encoded. This differs from existing technologies that directly encode speech, reducing the feature dimension and avoiding excessive consumption. At the same time, the greatly reduced feature dimension provides a basis for supporting ultra-low bit rate transmission, thus making it suitable for various network environments (such as weak network environments).

[0108] Optionally, determining the encoded speech data based on the first speech feature can be achieved through the following steps:

[0109] Step b1: Perform scene detection on the first speech feature, determine the target scene that matches the first speech feature, and obtain the second speech feature corresponding to the target scene.

[0110] Step b2: Determine the encoded speech data based on the second speech feature.

[0111] In this embodiment, during training, a scene detection network layer can be trained to perform scene discrimination operations on the first speech feature. This scene detection network layer can function as a scene detection module, inputting the first speech feature into the scene detection module to obtain the analyzed speech feature, i.e., the second speech feature. Then, the second speech feature is further encoded.

[0112] The scene detection module can distinguish between human voices (single voice, multi-voice voice), music scenes, and reverberation scenes. Therefore, this embodiment supports encoding and decoding in music, human voice, and reverberation scenes, and supports mixed encoding of multiple scenes. No specific scene is limited here.

[0113] Specifically, since the speech sent by the transmitting end in actual applications may include not only human voice scenes, but also other scenes, in order to enable the trained model to accurately decode all kinds of features, scene detection has been added. It can support the differentiation of human voice (single voice, multi-voice) scenes, music scenes, reverberation scenes, etc., and supports automatic switching mode for single scene or multiple scenes. The analyzed speech features are more in line with the actual application scenarios.

[0114] Optionally, determining the encoded speech data based on the second speech feature can be achieved through the following steps:

[0115] Step c1: Perform residual quantization on the second speech feature to obtain the quantized speech feature, and generate a codebook and the index value corresponding to the codebook;

[0116] Step c2: Use the quantized speech features as encoded speech data;

[0117] The codebook is used to look up the quantized speech features using index values.

[0118] In this embodiment, during the training process, in order to represent floating-point information using bitstreams, speech features need to be quantized. At the same time, in order to represent the same amount of information using fewer bits, residual quantization is used to generate encoded data.

[0119] Specifically, during training, a residual quantization network layer can be trained to perform quantization operations on the second speech feature. This residual quantization network layer can act as a residual quantization module, inputting the second speech feature into it to obtain the quantized speech feature, which forms the encoded speech data.

[0120] In subsequent applications, the index values ​​corresponding to the quantized speech features need to be encapsulated into a bitstream and transmitted over the network to the receiving end. The receiving end obtains the encoded speech data based on the index values ​​transmitted over the network and combined with the codebook, which eliminates the need to transmit the quantized speech features over the network, thus reducing power consumption.

[0121] Optionally, the generative adversarial network includes a generator network and a discriminator network. The generator network is used to decode the encoded speech data, and the discriminator network is used to distinguish the decoded speech data from the speech data to be encoded, so as to update the generator network. The speech data to be encoded is multiple, and correspondingly, the encoded speech data is multiple. Training the generative adversarial network based on the encoded speech data can be achieved through the following steps:

[0122] The encoded speech data is divided into multiple training sample sets, and the following steps are repeated for each training sample set until the training stop condition is met to stop the training of the generator network and the discriminator network:

[0123] Step d1: Input the training sample set into the generator network to train and obtain the decoded speech data corresponding to each encoded speech data in the training sample set.

[0124] Step d2: Input the decoded speech data and the speech data to be encoded corresponding to each encoded speech data in the training sample set into the discriminator network, and calculate the loss value of the generative adversarial network.

[0125] Step d3: Update the generator network and the discriminator network using the loss value.

[0126] In this embodiment, the quantized speech features are used as input to the generator network to train the generator network. The output of the generator network is to decode the quantized speech features and generate decoded speech data. Then, the decoded speech data and the speech data to be encoded are input into the discriminator network. By distinguishing real speech data, the parameters of the generator network are updated until training stops.

[0127] One training sample set corresponds to one round of training. This training sample set is used as input to the generator network to train the generator network and output decoded speech data. Then, the decoded speech data and the speech data to be encoded corresponding to the training sample set are input into the discriminator network. The parameters of the generator network and the discriminator network are updated by calculating the loss value to complete one round of training and start the next round of training. The training stops when the loss value reaches the preset threshold range, indicating that the stopping condition is met. At this time, the training of the generative adversarial network is stopped, that is, the training of the generator network and the discriminator network is stopped.

[0128] By employing generative adversarial networks (GANs), the generator network generates fake data to try to deceive the discriminator network, while the discriminator network continuously distinguishes the fake data and engages in a continuous battle against it. This makes the output of the generator network closer to the real data, rendering the discriminator network unable to distinguish between real and fake data. This not only results in higher decoding accuracy and ensures high speech fidelity, but also allows the GAN to support parallel processing and achieve higher generation efficiency.

[0129] Specifically, in combination Figure 3 As shown, Figure 3 This is a schematic flowchart illustrating a speech processing method provided in another embodiment of this application. Wherein, Figure 3This is a flowchart illustrating the training process. Taking the speech data to be encoded as a time-domain speech signal as an example, the time-domain speech signal to be encoded is first processed by the feature extraction module to obtain the mel spectrum (i.e., the first speech feature). Then, it is sent to the scene detection module to distinguish between human voices (single voice, multi-voice), music scenes, and reverberation scenes, obtaining the analyzed speech features (i.e., the second speech feature). Next, it enters the residual quantization (RVQ) module to obtain the quantized features (i.e., the encoded speech data), and its codebook and corresponding index values ​​are saved. To support streaming processing, the generator network can be designed as a causal generator network, that is, its input is the quantized speech features, and its output is the decoded time-domain speech signal (i.e., the decoded speech data). At the same time, in order to increase the naturalness of the decoded speech, adversarial network training is added. That is, both generated speech and real speech are sent to the discriminator network for training, continuously updating the parameters of the generator network and the discriminator network until the training is completed.

[0130] Optionally, the generator network includes multiple influencing factors, including at least one of the following: receptive field, frequency spatial consistency, and multi-scale frequency resolution.

[0131] In this embodiment, a lightweight generative model based on 1D convolution is proposed, taking into account factors such as receptive field, frequency spatial consistency, and multi-scale frequency resolution.

[0132] Specifically, because the generator network is designed with consideration of receptive field, frequency spatial consistency, and multi-scale frequency resolution, it can achieve lightweight processing and obtain better audio quality after decoding with lower performance consumption.

[0133] Among them, the high-fidelity audio codec based on generative adversarial networks can be deployed in real time on mobile devices, supporting encoding bitrates of 1.6kbps to 12kbps.

[0134] Therefore, while ensuring a certain level of sound quality, using the fewest possible bits to represent high-fidelity audio in digital audio can effectively reduce the bandwidth and storage space required for audio file transmission. This technology enables audio encoding and decoding that supports ultra-low bitrates and high-fidelity sound quality. Simultaneously, it features low power consumption, parallel processing capabilities, and the ability to handle multiple scenarios including music, vocals, and reverb. It can provide natural and clear voice chat even with network bandwidth of 3-6kb per second, greatly enhancing the audio experience during real-time communication.

[0135] Figure 4 This is a flowchart illustrating a voice processing method according to another embodiment of this application. The method in this embodiment can be executed by a sending end, which can be a server or a terminal, etc. No specific limitations are placed on the carrier of the sending end here. Figure 4As shown, the method in this embodiment may include:

[0136] S401. Determine the target speech data to be encoded, and extract features from the target speech data to be encoded to obtain the first target speech features.

[0137] In this embodiment, the transmitting end extracts first target speech feature data from the acquired target speech data to be encoded using a feature extraction module. This feature extraction process is similar to... Figure 1B The feature extraction process for the speech data to be encoded is similar in the embodiments shown, and will not be described again here.

[0138] S402. Encode the first target speech features to obtain encoded target speech data.

[0139] In this embodiment, scene detection is performed on the first target speech feature to determine the actual target scene that matches the first target speech feature, and a second target speech feature corresponding to the actual target scene is obtained. Based on the second target speech feature, the encoded target speech data is determined, that is, residual quantization is performed on the second target speech feature to obtain the quantized target speech feature.

[0140] Specifically, firstly, the first target speech features are processed by a scene detection module to obtain the target speech features after analysis of the second target speech feature set. Then, the second target speech features are processed by a residual quantization module to obtain the quantized target speech features, which are used as the encoded target speech data. This encoding process is similar to... Figure 1B The process of encoding the first voice data in the illustrated embodiment is similar and will not be described again here.

[0141] S403. Based on the encoded target speech data, send an instruction message to the receiving end. The instruction message is used to instruct the receiving end to determine the encoded target speech data according to the encapsulation information in the instruction message, and to obtain the decoded target speech data through a generative adversarial network based on the target speech data.

[0142] Optionally, sending an indication message to the receiving end based on the encoded target speech data may include the following steps:

[0143] Step e1: Based on the encoded target speech data, determine the target index value corresponding to the encoded target speech data through the codebook;

[0144] Step e2: Pack the target index value into a bit stream and transmit it to the receiving end via the network; the bit stream serves as an indication message.

[0145] Specifically, based on the encoded target speech data, the target index value corresponding to the encoded target speech data is determined through the codebook; the target index value is packaged and encapsulated into a bit stream and transmitted to the receiving end through the network, so that the receiving end can obtain the encoded target speech data by unpacking the target index value and the configured codebook, and obtain the decoded target speech data through the generator network based on the target speech data.

[0146] The information encapsulated in the sent message can be the target index value in the bitstream. Using a trained codebook, the encoded target speech data corresponding to that target index value can be found, and then the encoded target speech data is input into the trained generator network to obtain the decoded target speech data.

[0147] In practical applications, the sending end encodes the data (i.e., quantized speech features), packages the index values ​​corresponding to the codebook into a bit stream for transmission over the network, and then decodes it at the receiving end.

[0148] In this embodiment, combined with Figure 1B and Figure 5 As shown, where, Figure 5 This is a flowchart illustrating a speech processing method provided in another embodiment of this application. Speech features are extracted from the speech, and then encoded. Unlike existing technologies that directly encode speech, this method reduces the feature dimensionality, avoiding excessive resource consumption. Furthermore, the significantly reduced feature dimensionality provides a foundation for supporting ultra-low bitrate transmission. The target index value matching the encoded data is then packaged and encapsulated into a bitstream for network transmission. The receiving end unpacks the target index value and combines it with a codebook to obtain the encoded data. A generator network is trained using a generative adversarial network (GAN) to perform decoding, supporting parallel processing and improving operational efficiency. The adversarial training also facilitates the generation of more natural and realistic decoded speech with higher accuracy, thus achieving high-fidelity speech quality. Therefore, while ensuring high-fidelity audio quality, high-efficiency and low-resource audio encoding and decoding are achieved.

[0149] Therefore, while ensuring a certain level of sound quality, using the fewest possible bits to represent high-fidelity audio in digital audio can effectively reduce the bandwidth and storage space required for audio file transmission. This technology enables audio encoding and decoding that supports ultra-low bitrates and high-fidelity sound quality. Simultaneously, it features low power consumption, parallel processing capabilities, and the ability to handle multiple scenarios including music, vocals, and reverb. It can provide natural and clear voice chat even with network bandwidth of 3-6kb per second, greatly enhancing the audio experience during real-time communication.

[0150] Figure 6This is a flowchart illustrating a speech processing method according to another embodiment of this application. The method in this embodiment can be executed by a receiving end, which can be a server or a terminal, etc. No specific limitations are placed on the carrier of the receiving end here. Figure 6 As shown, the method in this embodiment may include:

[0151] S601, Receive the instruction message sent by the sending end.

[0152] Optionally, receive the bit stream sent by the sender, where the bit stream can serve as an indication message.

[0153] In this process, after determining the target index value, the sending end packages and encapsulates the target index value into a bitstream, which is then transmitted to the receiving end via the network. Specifically, the encapsulated information in the transmitted message can be the target index value from the bitstream. Using a pre-trained codebook, the encoded target speech data corresponding to the target index value can be found, and then the encoded target speech data is input into a pre-trained generator network to obtain the decoded target speech data.

[0154] In this embodiment, the instruction message is used to instruct the receiving end to determine the encoded target speech data according to the encapsulation information in the instruction message, and to obtain the decoded target speech data through a generative adversarial network based on the target speech data.

[0155] For details on the generation or determination of the instruction message, please refer to [link / reference]. Figure 4 The execution process of the instruction message in the illustrated embodiment will not be described again here.

[0156] S602. Parse the encapsulation information in the instruction message to obtain encoded target speech data. The encoded target speech data is obtained by the receiving end through feature extraction of the target speech data to be encoded to obtain the first target speech feature, and then encoding the first target speech feature.

[0157] Optionally, parsing the encapsulation information in the indication message to obtain the encoded target speech data may include the following steps:

[0158] Step f1: By parsing the bit stream, the target index value is obtained. The target index value is the index value corresponding to the encoded target speech data.

[0159] Step f2: Obtain the encoded target speech data according to the configured codebook and the target index value.

[0160] S603. Based on the encoded target speech data, the decoded target speech data is obtained through a generative adversarial network.

[0161] In this embodiment, the receiving end uses a codebook combined with the target index value parsed through network transmission to find the encoded data (i.e., the quantized speech features), and then inputs it into the generator network for decoding. Specifically, after receiving the bit stream, the receiving end unpacks it to recover the index value, and simultaneously combines it with the codebook to obtain the speech features analyzed by the scene detection module, which are then sent to the generator network to obtain the decoded speech. Furthermore, no training is required during application; therefore, it is unnecessary to deploy a discriminator network at either the receiving or transmitting end.

[0162] In this embodiment, since the received indication information has already undergone dimensionality reduction, excessive resource consumption is avoided. Furthermore, the significantly reduced feature dimensions provide a foundation for supporting ultra-low bitrate transmission, making it suitable for various network environments (such as weak network environments). The network trained using generative adversarial training supports parallel processing, thus improving operational efficiency. Adversarial training also facilitates the generation of more natural and realistic decoded speech with higher accuracy, thereby achieving high-fidelity speech quality. Therefore, while ensuring high-fidelity audio quality, high-efficiency and low-resource-consumption audio encoding and decoding are achieved.

[0163] Therefore, while ensuring a certain level of sound quality, using the fewest possible bits to represent high-fidelity audio in digital audio can effectively reduce the bandwidth and storage space required for audio file transmission. This technology enables audio encoding and decoding that supports ultra-low bitrates and high-fidelity sound quality. Simultaneously, it features low power consumption, parallel processing capabilities, and the ability to handle multiple scenarios including music, vocals, and reverb. It can provide natural and clear voice chat even with network bandwidth of 3-6kb per second, greatly enhancing the audio experience during real-time communication.

[0164] Based on the same idea, this application also provides an apparatus corresponding to the above method, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of a voice processing device provided in an embodiment of this application. The voice processing device may include:

[0165] The feature extraction module 701 is used to extract features from the speech data to be encoded and generate the first speech features;

[0166] The encoding module 702 is used to determine the encoded speech data based on the first speech feature;

[0167] The training module 703 is used to train a generative adversarial network based on the encoded speech data, and the generative adversarial network is used to decode the speech data.

[0168] In this embodiment, a feature extraction module 701, an encoding module 702, and a training module 703 are configured. Features are extracted from the speech data to be encoded, and these extracted features are encoded. The encoded speech data is then used to train a generative adversarial network (GAN) for decoding. By extracting speech features and then encoding them, unlike existing technologies that directly encode speech, the feature dimensionality is reduced, avoiding excessive resource consumption. Furthermore, the significantly reduced feature dimensionality provides a foundation for supporting ultra-low bitrate transmission. The GAN is then trained using a parallel processing approach, improving operational efficiency. Adversarial training also facilitates the generation of more natural and realistic decoded speech with higher accuracy, thus achieving high-fidelity speech quality. Therefore, while ensuring high-fidelity audio quality, high-efficiency and low-resource audio encoding and decoding are achieved.

[0169] Optional, feature extraction module, specifically used for:

[0170] When the speech data to be encoded is a frequency domain speech signal, the frequency domain speech signal is converted into a time domain speech signal, and features are extracted from the time domain speech signal to obtain the Mel spectrum;

[0171] When the speech data to be encoded is the time-domain speech signal, feature extraction is performed on the time-domain speech signal to obtain the Mel spectrum;

[0172] The Mel spectrum is used to represent the first speech feature.

[0173] Optionally, the encoding module includes a first encoding unit and a second encoding unit;

[0174] The first encoding unit is used to perform scene detection on the first speech feature, determine the target scene that matches the first speech feature, and obtain the second speech feature corresponding to the target scene.

[0175] The second encoding unit is used to determine the encoded speech data based on the second speech feature.

[0176] Optional, the second coding unit is specifically used for:

[0177] The second speech feature is subjected to residual quantization to obtain the quantized speech feature, as well as a codebook and the index value corresponding to the codebook;

[0178] The quantized speech features are used as encoded speech data;

[0179] The codebook is used to look up the quantized speech features using index values.

[0180] Optionally, the generative adversarial network includes a generator network and a discriminator network. The generator network is used to decode the encoded speech data, and the discriminator network is used to distinguish the decoded speech data from the speech data to be encoded, so as to update the generator network. The speech data to be encoded is multiple, and correspondingly, the encoded speech data is multiple. The training module is specifically used for:

[0181] The encoded speech data is divided into multiple training sample sets, and the following steps are repeated for each training sample set until the training stop condition is met to stop the training of the generator network and the discriminator network:

[0182] The training sample set is input into the generator network to train and obtain the decoded speech data corresponding to each encoded speech data in the training sample set;

[0183] The decoded speech data and the speech data to be encoded corresponding to each encoded speech data in the training sample set are input into the discriminator network to calculate the loss value of the generative adversarial network.

[0184] The generator network and the discriminator network are updated using the loss value.

[0185] This embodiment, while ensuring a certain level of sound quality, uses the fewest possible bits to represent high-fidelity audio in digital audio, effectively reducing the bandwidth and storage space required for audio file transmission. It achieves audio encoding and decoding that supports ultra-low bitrates and high-fidelity sound quality. Simultaneously, it features low power consumption, parallel processing capabilities, and the ability to handle multiple scenarios including music, vocals, and reverb. It can provide natural and clear voice chat even with a network bandwidth of 3-6kb per second, greatly enhancing the audio experience during real-time communication.

[0186] The apparatus provided in this application can implement the methods of the embodiments shown in Figures 1-3 above. The implementation principle and technical effects are similar, and will not be repeated here.

[0187] Based on the same idea, this application also provides an apparatus corresponding to the above method, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of a voice processing apparatus according to another embodiment of this application. The voice processing apparatus may include:

[0188] The processing module 801 is used to determine the target speech data to be encoded and to extract features from the target speech data to be encoded to obtain the first target speech features.

[0189] Encoding module 802 is used to encode the first target speech features to obtain encoded target speech data;

[0190] The sending module 803 is used to send an indication message to the receiving end based on the encoded target speech data. The indication message is used to instruct the receiving end to determine the encoded target speech data based on the encapsulation information in the indication message, and to obtain the decoded target speech data through a generative adversarial network based on the target speech data.

[0191] Optional, the sending module, specifically used for:

[0192] Based on the encoded target speech data, the target index value corresponding to the encoded target speech data is determined through the codebook;

[0193] The target index value is packaged and encapsulated into a bit stream and transmitted to the receiving end via the network; the bit stream serves as an indication message.

[0194] Specifically, the sending module is configured to: determine the target index value corresponding to the encoded target speech data according to the codebook; package and encapsulate the target index value into a bit stream and transmit it to the receiving end via the network, so that the receiving end can obtain the encoded target speech data by unpacking the target index value and the configured codebook, and obtain the decoded target speech data through the generator network according to the target speech data.

[0195] In this embodiment, by setting up a processing module 801, an encoding module 802, and a sending module 803, speech features are extracted and then encoded. This differs from existing technologies that directly encode speech, reducing the feature dimensionality and avoiding excessive resource consumption. Furthermore, the significantly reduced feature dimensionality provides a foundation for supporting ultra-low bitrate transmission. The target index values ​​that match the encoded data are then packaged into a bitstream for network transmission. The receiving end unpacks the target index values ​​and combines them with a codebook to obtain the encoded data. A generative adversarial network (GAN) is used to train a generator network for decoding, supporting parallel processing and improving operational efficiency. The adversarial training also facilitates the generation of more natural and realistic decoded speech with higher accuracy, thus achieving high-fidelity speech quality. Therefore, while ensuring high-fidelity audio quality, high-efficiency and low-resource audio encoding and decoding are achieved.

[0196] Therefore, while ensuring a certain level of sound quality, using the fewest possible bits to represent high-fidelity audio in digital audio can effectively reduce the bandwidth and storage space required for audio file transmission. This technology enables audio encoding and decoding that supports ultra-low bitrates and high-fidelity sound quality. Simultaneously, it features low power consumption, parallel processing capabilities, and the ability to handle multiple scenarios including music, vocals, and reverb. It can provide natural and clear voice chat even with network bandwidth of 3-6kb per second, greatly enhancing the audio experience during real-time communication.

[0197] The apparatus provided in this application embodiment can achieve the above-mentioned... Figure 4 and 5 The methods in the embodiments shown are similar in principle and technical effect, and will not be described again here.

[0198] Based on the same idea, this application also provides an apparatus corresponding to the above method, such as... Figure 9 As shown, Figure 9 This is a schematic diagram of a voice processing apparatus provided in another embodiment of this application. The voice processing apparatus may include:

[0199] The receiving module 901 is used to receive the indication message sent by the sending end;

[0200] Processing module 902 is used to parse the encapsulation information in the indication message to obtain encoded target speech data, wherein the encoded target speech data is obtained by the receiving end through encoding the target speech data to be encoded.

[0201] The decoding module 903 is used to obtain decoded target speech data by generating adversarial network based on the encoded target speech data.

[0202] Optional, processing module, specifically used for:

[0203] By parsing the bitstream, the target index value is obtained, which is the index value corresponding to the encoded target speech data;

[0204] The encoded target speech data is obtained based on the configured codebook and the target index value.

[0205] In this embodiment, by setting up a receiving module 901, a processing module 902, and a decoding module 903, the encoded data (i.e., the quantized speech features) is found by combining the codebook with the target index value parsed through network transmission. This data is then input into the generator network for decoding. Specifically, after receiving the bit stream, the receiving end unpacks it to recover the index value. Simultaneously, combined with the codebook, it re-obtains the speech features analyzed by the scene detection module and sends them to the generator network to obtain the decoded speech. Furthermore, no training is required during application; therefore, it is unnecessary to deploy a discriminator network at either the receiving or transmitting end.

[0206] Since the received instruction information has already undergone dimensionality reduction, excessive resource consumption is avoided. Furthermore, the significantly reduced feature dimensions provide a foundation for supporting ultra-low bitrate transmission, making it suitable for various network environments (such as weak network environments). The network trained using generative adversarial training supports parallel processing, thus improving operational efficiency. Adversarial training also facilitates the generation of more natural and realistic decoded speech with higher accuracy, thereby achieving high-fidelity speech quality. Therefore, while ensuring high-fidelity audio quality, high-efficiency and low-resource-consumption audio encoding and decoding are achieved.

[0207] Therefore, this application, while ensuring a certain level of sound quality, uses the fewest possible bits to represent high-fidelity audio in digital audio, effectively reducing the bandwidth and storage space required for audio file transmission. It achieves audio encoding and decoding that supports ultra-low bitrates and high-fidelity sound quality, while also possessing low-power, parallel processing capabilities and the ability to handle multiple scenarios including music, vocals, and reverb. It can provide natural and clear voice chat even with network bandwidth of 3-6kb per second, greatly enhancing the audio experience during real-time communication.

[0208] Therefore, while ensuring a certain level of sound quality, using the fewest possible bits to represent high-fidelity audio in digital audio can effectively reduce the bandwidth and storage space required for audio file transmission. This technology enables audio encoding and decoding that supports ultra-low bitrates and high-fidelity sound quality. Simultaneously, it features low power consumption, parallel processing capabilities, and the ability to handle multiple scenarios including music, vocals, and reverb. It can provide natural and clear voice chat even with network bandwidth of 3-6kb per second, greatly enhancing the audio experience during real-time communication.

[0209] The apparatus provided in this application embodiment can achieve the above-mentioned... Figure 6 The methods in the illustrated embodiments are similar in principle and technical effect, and will not be described again here. Based on the same idea, this application also provides a system corresponding to the above methods, combined with... Figure 5 As shown, the speech processing system includes:

[0210] The transmitting end and the receiving end are communicatively connected to achieve end-to-end transmission;

[0211] The transmitting end is used to implement the method as described in any of the second aspects;

[0212] The receiving end is used to implement the method as described in any of the third aspects;

[0213] The sending end is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments.

[0214] The receiving end is equipped with a packet loss compensation post-processing function, which records the current state and predicts the audio frames with lost features when voice features are lost, thereby determining the lost features.

[0215] In practical applications, deploying in-band FEC functionality, based on a reasonable balance of codebook dimensions and quantity, can add a certain amount of redundancy during transmission, ensuring the correct transmission of encoded features under extremely weak network conditions. Deploying packet loss compensation post-processing functionality, at the decoding end (i.e., the receiving end), when audio features are lost, the current state is recorded and entered into the post-processing module (i.e., the packet loss compensation post-processing function module), which predicts the audio frames with lost features to maximize the audio experience at the receiving end.

[0216] In this embodiment, the transmitting end extracts speech features to obtain speech features, and then encodes these features. This differs from existing technologies that directly encode speech, reducing the feature dimensionality and avoiding excessive resource consumption. Furthermore, the significantly reduced feature dimensionality provides a foundation for supporting ultra-low bitrate transmission. The target index values ​​that match the encoded data are then packaged and encapsulated into a bitstream for network transmission. The receiving end unpacks the target index values ​​and combines them with a codebook to obtain the encoded data. A generator network trained using a generative adversarial network is then used for decoding, supporting parallel processing and improving operational efficiency. Simultaneously, adversarial training is more conducive to generating natural and realistic decoded speech with higher accuracy, thus achieving high-fidelity speech quality. Therefore, while ensuring high-fidelity audio quality, high-efficiency and low-resource-consumption audio encoding and decoding are achieved.

[0217] Therefore, while ensuring a certain level of sound quality, using the fewest possible bits to represent high-fidelity audio in digital audio can effectively reduce the bandwidth and storage space required for audio file transmission. This technology enables audio encoding and decoding that supports ultra-low bitrates and high-fidelity sound quality. Simultaneously, it features low power consumption, parallel processing capabilities, and the ability to handle multiple scenarios including music, vocals, and reverb. It can provide natural and clear voice chat even with network bandwidth of 3-6kb per second, greatly enhancing the audio experience during real-time communication.

[0218] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application. Figure 10 As shown, the electronic device 1000 provided in this embodiment includes: a processor 1001 and a memory communicatively connected to the processor. The processor 1001 and the memory 1002 are connected via a bus 1003.

[0219] In a specific implementation, the processor 1001 executes the computer execution instructions stored in the memory 1002, causing the processor 1001 to execute the method in the above method embodiment.

[0220] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0221] In the above Figure 10 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0222] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0223] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0224] This application also provides a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the speech processing method described in the above method embodiments.

[0225] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the speech processing method described above.

[0226] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0227] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0228] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A speech processing method, characterized in that, The method includes: Feature extraction is performed on the speech data to be encoded to generate the first speech feature; Scene detection is performed on the first speech feature to determine the target scene that matches the first speech feature, and a second speech feature corresponding to the target scene is obtained. The target scene is one of single voice, multi-voice, music scene, or reverberation scene. The second speech feature is subjected to residual quantization to obtain the quantized speech feature, as well as a codebook and the index value corresponding to the codebook; The quantized speech features are used as encoded speech data; wherein, the codebook is used to look up the quantized speech features by index value; Based on the encoded speech data, a generative adversarial network is trained. The generative adversarial network is used to decode the speech data. The generator network includes multiple influencing factors, including at least one of the following: receptive field, frequency spatial consistency, and multi-scale frequency resolution. The voice processing system includes a transmitter and a receiver. The transmitter is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments. The receiver is equipped with a packet loss compensation post-processing function to record the current state and predict the audio frames with lost features when voice features are lost, thereby determining the lost features.

2. The method according to claim 1, characterized in that, The step of extracting features from the speech data to be encoded to generate first speech features includes: If the speech data to be encoded is a frequency domain speech signal, then the frequency domain speech signal is converted into a time domain speech signal, and features are extracted from the time domain speech signal to obtain the Mel spectrum; If the speech data to be encoded is the time-domain speech signal, then feature extraction is performed on the time-domain speech signal to obtain the Mel spectrum; The Mel spectrum is used to represent the first speech feature.

3. The method according to claim 1 or 2, characterized in that, The generative adversarial network (GAN) includes a generator network and a discriminator network. The generator network decodes the encoded speech data, and the discriminator network distinguishes the decoded speech data from the speech data to be encoded, thereby updating the generator network. The speech data to be encoded is multiple, and correspondingly, the encoded speech data is multiple. Training the GAN based on the encoded speech data includes: The encoded speech data is divided into multiple training sample sets, and the following steps are repeated for each training sample set until the training stop condition is met to stop the training of the generator network and the discriminator network: The training sample set is input into the generator network to train and obtain the decoded speech data corresponding to each encoded speech data in the training sample set; The decoded speech data and the speech data to be encoded corresponding to each encoded speech data in the training sample set are input into the discriminator network to calculate the loss value of the generative adversarial network. The generator network and the discriminator network are updated using the loss value.

4. A speech processing method, characterized in that, Applied to the sending end; the method includes: The target speech data to be encoded is determined, and features are extracted from the target speech data to be encoded to obtain the first target speech features; Scene detection is performed on the first target speech feature to determine the target scene that matches the first target speech feature, and a second speech feature corresponding to the target scene is obtained. The target scene is one of single voice, multi-voice, music scene, or reverberation scene. The second speech feature is subjected to residual quantization to obtain the quantized speech feature, as well as a codebook and the index value corresponding to the codebook; The quantized speech features are used as encoded speech data; wherein, the codebook is used to look up the quantized speech features by index value; Based on the encoded target speech data, an instruction message is sent to the receiving end. The instruction message is used to instruct the receiving end to determine the encoded target speech data according to the encapsulation information in the instruction message, and to obtain the decoded target speech data through a generative adversarial network based on the target speech data. The generator network includes multiple influencing factors, including at least one of the following: receptive field, frequency spatial consistency, and multi-scale frequency resolution. The voice processing system includes a transmitter and a receiver. The transmitter is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments. The receiver is equipped with a packet loss compensation post-processing function to record the current state and predict the audio frames with lost features when voice features are lost, thereby determining the lost features.

5. A speech processing method, characterized in that, Applied to the receiving end; the method includes: Receive the instruction message sent by the sender; The encapsulation information in the instruction message is parsed to obtain encoded target speech data. The encoded target speech data is obtained by the receiving end through feature extraction of the target speech data to be encoded to obtain a first target speech feature, and scene detection is performed on the first target speech feature to determine a target scene matching the first target speech feature, and a second speech feature corresponding to the target scene is obtained. The second speech feature is residual quantized to obtain quantized speech features, and a codebook and the index value corresponding to the codebook are generated. The quantized speech features are used as encoded speech data. The target scene is one of single voice, multi-voice, music scene, and reverberation scene. The codebook is used to look up the quantized speech features by the index value. Based on the encoded target speech data, the decoded target speech data is obtained through a generative adversarial network. The generator network includes multiple influencing factors, including at least one of the following: receptive field, frequency spatial consistency, and multi-scale frequency resolution. The voice processing system includes a transmitter and a receiver. The transmitter is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments. The receiver is equipped with a packet loss compensation post-processing function to record the current state and predict the audio frames with lost features when voice features are lost, thereby determining the lost features.

6. A voice processing device, characterized in that, The device includes: The feature extraction module is used to extract features from the speech data to be encoded and generate the first speech features; The encoding module includes a first encoding unit and a second encoding unit; The first encoding unit is used to perform scene detection on the first speech feature, determine the target scene that matches the first speech feature, and obtain the second speech feature corresponding to the target scene. The target scene is one of single voice, multi-voice, music scene, and reverberation scene. The second encoding unit is used to perform residual quantization on the second speech feature to obtain the quantized speech feature, and to generate a codebook and an index value corresponding to the codebook; the quantized speech feature is used as the encoded speech data; wherein, the codebook is used to look up the quantized speech feature by the index value; The training module is used to train a generative adversarial network based on the encoded speech data. The generative adversarial network is used to decode the speech data. The generator network includes multiple influencing factors, including at least one of the following: receptive field, frequency spatial consistency, and multi-scale frequency resolution. The voice processing system includes a transmitter and a receiver. The transmitter is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments. The receiver is equipped with a packet loss compensation post-processing function to record the current state and predict the audio frames with lost features when voice features are lost, thereby determining the lost features.

7. A voice processing device, characterized in that, Applied to the transmitting end, the device includes: The processing module is used to determine the target speech data to be encoded and to extract features from the target speech data to be encoded to obtain the first target speech features; The encoding module includes a first encoding unit and a second encoding unit; The first encoding unit is used to perform scene detection on the first target speech features, determine the target scene that matches the first target speech features, and obtain the second speech features corresponding to the target scene. The target scene is one of single voice, multi-voice, music scene, and reverberation scene. The second encoding unit is used to perform residual quantization on the second speech feature to obtain the quantized speech feature, and to generate a codebook and an index value corresponding to the codebook; the quantized speech feature is used as the encoded speech data; wherein, the codebook is used to look up the quantized speech feature by the index value; The sending module is used to send an indication message to the receiving end based on the encoded target speech data. The indication message is used to instruct the receiving end to determine the encoded target speech data based on the encapsulation information in the indication message, and to obtain the decoded target speech data through a generative adversarial network based on the target speech data. The generator network includes multiple influencing factors, including at least one of the following: receptive field, frequency spatial consistency, and multi-scale frequency resolution. The voice processing system includes a transmitter and a receiver. The transmitter is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments. The receiver is equipped with a packet loss compensation post-processing function to record the current state and predict the audio frames with lost features when voice features are lost, thereby determining the lost features.

8. A voice processing device, characterized in that, Applied to the receiving end, the device includes: The receiving module is used to receive indication messages sent by the sending end; The processing module is used to parse the encapsulation information in the instruction message to obtain encoded target speech data. The encoded target speech data is obtained by the receiving end through feature extraction of the target speech data to be encoded to obtain a first target speech feature, and scene detection is performed on the first target speech feature to determine a target scene matching the first target speech feature, and a second speech feature corresponding to the target scene is obtained. The second speech feature is residual quantized to obtain quantized speech features, and a codebook and the index value corresponding to the codebook are generated. The quantized speech features are used as encoded speech data. The target scene is one of single voice, multi-voice, music scene, and reverberation scene. The codebook is used to look up the quantized speech features by the index value. The decoding module is used to obtain decoded target speech data from the encoded target speech data through a generative adversarial network. The generator network includes multiple influencing factors, including at least one of the following: receptive field, frequency spatial consistency, and multi-scale frequency resolution. The voice processing system includes a transmitter and a receiver. The transmitter is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments. The receiver is equipped with a packet loss compensation post-processing function to record the current state and predict the audio frames with lost features when voice features are lost, thereby determining the lost features.

9. A voice processing system, characterized in that, The system includes a transmitter and a receiver, which are communicatively connected to enable end-to-end transmission. The transmitting end is used to implement the method as described in claim 4; The receiving end is used to implement the method as described in claim 5; The sending end is equipped with a forward error correction function to add redundant information during network transmission and support correct transmission in various network environments. The receiving end is equipped with a packet loss compensation post-processing function, which records the current state and predicts the audio frames with lost features when voice features are lost, thereby determining the lost features.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the speech processing method as described in any one of claims 1-3.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by the processor, implements the speech processing method as described in any one of claims 1 to 3.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech processing method according to any one of claims 1-3.

Images