Artificial intelligence-based speech generation method and apparatus, computer device, and medium

Abstract

The application belongs to the technical field of artificial intelligence, and relates to a voice generation method and device based on artificial intelligence, computer equipment and a storage medium, which comprises the following steps: receiving an input original audio signal; calling a voice processing model; performing encoding processing on the original audio signal based on an encoder in the voice processing model to obtain voice features; performing separation processing on the voice features to obtain semantic features and acoustic features; performing discrete processing on the semantic features based on a semantic quantization module to obtain a semantic codebook; performing multilayer residual vector quantization processing on the acoustic features based on a filter bank to obtain an acoustic residual codebook sequence; screening a target acoustic residual codebook sequence from the acoustic residual codebook sequence; and performing conversion processing on the semantic codebook and the target acoustic residual codebook sequence based on a neural decoder to generate voice data and output the voice data. The application can be applied to voice generation scenes in the fields of financial technology and medical health, and can effectively improve voice generation quality.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to fields such as fintech and healthcare, particularly to artificial intelligence-based speech generation methods, devices, computer equipment, and storage media. Background Technology

[0002] Against the backdrop of the rapid development of speech and audio generation technologies driven by neural networks, neural audio codecs, as core components, undertake the crucial task of compressing high-dimensional continuous audio signals into low-dimensional discrete representations while maintaining the fidelity of the original speech. Their performance directly affects the effectiveness of applications such as speech synthesis and large-scale speech models. However, while existing audio coding schemes have made progress in medium-to-high fidelity speech reconstruction, they still have significant drawbacks: traditional multi-codebook structures cannot effectively separate semantic and acoustic features of speech during compression, resulting in severe aliasing of the two types of features in the compressed low-dimensional representation. This aliasing directly leads to blurred details and reduced clarity in the reconstructed speech, especially in complex speech scenarios, where the quality of speech generation is significantly reduced, making it difficult to meet the requirements of high-precision speech interaction.

[0003] For example, in the intelligent voice customer service scenario in the financial insurance field, traditional audio encoding schemes suffer from semantic and acoustic feature overlap, leading to an increased semantic parsing error rate for customer voice commands (such as "check the cash value of the policy"). At the same time, the voice responses generated by the system appear mechanical and monotonous due to the loss of acoustic features, reducing the customer interaction experience. In the voice-assisted medical record entry scenario in the healthcare field, when doctors dictate medical records by compressing audio, traditional schemes may lead to errors or omissions in the entry of key information because they cannot clearly separate pathological descriptions (such as "persistent chest pain") from background noise, affecting the accuracy and completeness of the medical records.

[0004] Therefore, there is an urgent need for a neural audio encoding and decoding technology that can improve the quality of speech reconstruction in order to support key applications such as highly reliable voice interaction and medical information technology. Summary of the Invention

[0005] The purpose of this application is to propose a speech generation method, apparatus, computer device, and storage medium based on artificial intelligence, so as to solve the technical problem of low speech generation quality in existing audio coding schemes.

[0006] Firstly, an artificial intelligence-based speech generation method is provided, including: Receive the raw audio signal input; A preset speech processing model is invoked; wherein, the speech processing model includes an encoder, a semantic quantization module, a filter bank, and a neural decoder; The encoder is used to encode the original audio signal to obtain the corresponding speech features. Based on a preset separation strategy, speech features are separated to obtain corresponding semantic and acoustic features. Based on the semantic quantization module, the semantic features are discretized to obtain the corresponding semantic codebook; Based on the filter bank, the acoustic features are subjected to multi-layer residual vector quantization to obtain the corresponding acoustic residual codebook sequence. Based on a preset adaptive discarding strategy, a target acoustic residual codebook sequence is selected from the acoustic residual codebook sequence. The neural decoder performs conversion processing on the semantic codebook and the target acoustic residual codebook sequence to generate corresponding speech data, and then outputs the speech data.

[0007] Secondly, an artificial intelligence-based speech generation device is provided, comprising: The receiving module is used to receive the input raw audio signal; The calling module is used to call a preset speech processing model; wherein, the speech processing model includes an encoder, a semantic quantization module, a filter bank, and a neural decoder; The encoding module is used to encode the original audio signal based on the encoder to obtain corresponding speech features; The separation module is used to separate speech features based on a preset separation strategy to obtain corresponding semantic and acoustic features. The discrete module is used to perform discrete processing on the semantic features based on the semantic quantization module to obtain the corresponding semantic codebook; The quantization module is used to perform multi-layer residual vector quantization on the acoustic features based on the filter bank to obtain the corresponding acoustic residual codebook sequence. The filtering module is used to filter out the target acoustic residual codebook sequence from the acoustic residual codebook sequence based on a preset adaptive discarding strategy. The processing module is used to perform conversion processing on the semantic codebook and the target acoustic residual codebook sequence based on the neural decoder to generate corresponding speech data, and to perform output processing on the speech data.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described artificial intelligence-based speech generation method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described artificial intelligence-based speech generation method.

[0010] In the aforementioned scheme implemented by the artificial intelligence-based speech generation method, device, computer equipment, and storage medium, the input raw audio signal is first received; then a preset speech processing model is invoked; wherein, the speech processing model includes an encoder, a semantic quantization module, a filter bank, and a neural decoder; then, the raw audio signal is encoded based on the encoder to obtain corresponding speech features; and the speech features are separated based on a preset separation strategy to obtain corresponding semantic features and acoustic features; subsequently, the semantic features are discretized based on the semantic quantization module to obtain a corresponding semantic codebook; and the acoustic features are subjected to multi-layer residual vector quantization based on the filter bank to obtain a corresponding acoustic residual codebook sequence; further, a target acoustic residual codebook sequence is selected from the acoustic residual codebook sequence based on a preset adaptive discarding strategy; finally, the semantic codebook and the target acoustic residual codebook sequence are converted based on the neural decoder to generate corresponding speech data, and the speech data is output. Based on the above automated processing flow, unlike existing audio coding schemes, this application uses a speech processing model that includes an encoder, semantic quantization module, filter bank and neural decoder. By introducing a separation strategy, a multi-codebook hierarchical quantization structure and an adaptive discard strategy, it can quickly and accurately perform speech generation processing on the original audio signal, achieve natural speech reconstruction processing at low bit rates, and achieve near-lossless speech reconstruction results, thereby effectively improving the quality of speech generation. Attached Figure Description

[0011] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of an embodiment of the artificial intelligence-based speech generation method according to this application; Figure 3 This is a schematic diagram of the structure of an embodiment of the artificial intelligence-based speech generation device according to this application; Figure 4This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0017] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0018] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0019] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0020] It should be noted that the AI-based speech generation method provided in this application is generally executed by a server / terminal device, and correspondingly, the AI-based speech generation device is generally located in the server / terminal device.

[0021] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0022] Continue to refer to Figure 2 This document illustrates a flowchart of an embodiment of the AI-based speech generation method according to this application. The order of steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The AI-based speech generation method provided in this application can be applied to any scenario requiring speech generation, and thus can be applied to products in these scenarios, such as speech generation products in the financial insurance and healthcare fields. The AI-based speech generation method includes the following steps: Step S201: Receive the input raw audio signal.

[0023] In this embodiment, the AI-based speech generation method runs on an electronic device (e.g., Figure 1The server / terminal device shown can acquire the input raw audio signal via a wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods. The implementing entity of this application is specifically a voice generation system, which can be simply referred to as the system.

[0024] During the acquisition process, the original audio signal will have various sampling rates due to different acquisition devices, acquisition scenarios, and acquisition requirements. Sampling rate refers to the number of times the sound signal is sampled per second, measured in Hertz (Hz), with common rates such as 8kHz, 16kHz, and 44.1kHz. To ensure that the neural encoder can process the audio signal uniformly and effectively, these audio signals with different sampling rates need to be uniformly adjusted to a waveform sequence with a 24kHz sampling rate. This adjustment process can be accomplished using specialized audio processing libraries or tools, the core of which is the application of resampling algorithms. The basic principle of resampling algorithms is based on the sampling theorem, which states that when the sampling frequency is greater than twice the highest frequency of the signal, the original signal can be recovered from the sampled signal without distortion.

[0025] Furthermore, this application can be applied to voice generation scenarios in the financial insurance and healthcare sectors. For example, in a telephone consultation scenario within the financial insurance sector, the aforementioned raw audio signal can be used as a recording of a customer's telephone consultation, including content such as: a customer calling the customer service hotline of an insurance company or financial institution to inquire about insurance products, claims procedures, investment plans, and other related issues.

[0026] Alternatively, in the context of teleconferences in the financial and insurance sector, the aforementioned raw audio signal can be a recording of a meeting or training session. The corresponding content includes: internal meetings of insurance companies, training courses, or teleconference recordings of sales teams, covering topics such as product introductions, market strategies, and sales techniques.

[0027] Furthermore, in the medical consultation scenario within the healthcare field, the aforementioned original audio signal can be used as a recording of a remote medical consultation. The corresponding content includes: patients conducting remote medical consultations with doctors via telephone or video, describing symptoms, medical history, or inquiring about treatment plans.

[0028] Alternatively, in the field of healthcare, the aforementioned raw audio signals can also be used for medical equipment operation or patient monitoring audio, including audio signals generated by medical equipment (such as ventilators or electrocardiogram monitors) during operation, or breathing sounds, heartbeat sounds, etc. recorded during patient monitoring.

[0029] Step S202: Invoke a preset speech processing model; wherein the speech processing model includes an encoder, a semantic quantization module, a filter bank, and a neural decoder.

[0030] In this embodiment, the speech processing model includes an encoder, a semantic quantization module, a filter bank, and a neural decoder. The model training process for the speech processing model will be described in further detail in subsequent embodiments, and will not be elaborated upon here.

[0031] Step S203: Encode the original audio signal based on the encoder to obtain the corresponding speech features.

[0032] In this embodiment, the encoder consists of convolutional blocks and self-attention blocks, used to compress the temporal waveform and extract the intermediate latent vector representation h, i.e., speech features. Where x is the original audio signal, This is the encoding function. By using an encoder to map long-time waveforms into continuous embedding vectors at low frame rates (e.g., 50Hz), the downstream quantization module can model both time and frequency.

[0033] In addition, the core objective of the encoding process is to compress the time dimension and preserve the perceptually important speech features, which is equivalent to completing the combined function of the traditional "feature extractor" and "downsampling encoder" in an end-to-end framework.

[0034] Step S204: Based on a preset separation strategy, the speech features are separated to obtain the corresponding semantic features and acoustic features.

[0035] In this embodiment, semantically relevant features can be identified and separated by pre-designing specific network structures or algorithms. Specifically, attention mechanisms can be used to focus on the parts of intermediate latent vectors that are closely related to semantic information, or specific convolutional kernels can be used to extract patterns with semantic features.

[0036] In this context, the convolutional kernel is a set of learnable weight parameters. When designing a convolutional kernel for extracting semantic features, its initial values ​​are set based on the understanding of semantic feature patterns and prior knowledge. For example, if certain specific frequency ranges or temporal patterns are known to be related to semantic information, the convolutional kernel can be designed to be more sensitive to these features. When the intermediate latent vector h, i.e., the aforementioned speech features (if it has a structure suitable for convolution operations, such as being viewed as a two-dimensional feature map), is input into the convolutional layer, the convolutional kernel slides across h. At each sliding position, the convolutional kernel performs element-wise multiplication with a local region of h and then sums the results to obtain an output value. By sliding the convolutional kernel across the entire h and performing calculations, a new feature map is obtained. Each element in this feature map reflects the degree of matching between the corresponding local region in h and the semantic feature pattern represented by the convolutional kernel. Then, through further processing, such as pooling operations on the feature map (e.g., max pooling, taking the maximum value in a local region as the output to enhance the saliency of the features) or using fully connected layers for integration, the semantically relevant part is finally obtained and used as the corresponding semantic feature.

[0037] Furthermore, the aforementioned acoustic features are semantically relevant parts extracted from the intermediate latent vector h. After semantic features, the remaining features are those that are not semantic features. They include acoustic information other than semantic information in the speech signal, such as pitch, timbre, volume changes, and other information related to the physical properties of sound.

[0038] Step S205: Discretize the semantic features based on the semantic quantization module to obtain the corresponding semantic codebook.

[0039] In this embodiment, the semantic quantizer is a module that discretizes continuous semantic features through vector quantization (VQ) mechanism.

[0040] The specific implementation process of discretizing the semantic features based on the semantic quantization module to obtain the corresponding semantic codebook will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0041] Step S206: Perform multi-layer residual vector quantization on the acoustic features based on the filter bank to obtain the corresponding acoustic residual codebook sequence.

[0042] In this embodiment, the specific implementation process of performing multi-layer residual vector quantization on the acoustic features based on the filter bank to obtain the corresponding acoustic residual codebook sequence will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0043] Step S207: Based on a preset adaptive discarding strategy, select the target acoustic residual codebook sequence from the acoustic residual codebook sequence.

[0044] In this embodiment, the specific implementation process of filtering the target acoustic residual codebook sequence from the acoustic residual codebook sequence based on the preset adaptive discarding strategy will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0045] Step S208: Based on the neural decoder, the semantic codebook and the target acoustic residual codebook sequence are converted to generate corresponding speech data, and the speech data is output.

[0046] In this embodiment, the specific implementation process of converting the semantic codebook and the target acoustic residual codebook sequence based on the neural decoder to generate corresponding speech data, and outputting the speech data, will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0047] This application first receives the input raw audio signal; then it calls a preset speech processing model, which includes an encoder, a semantic quantization module, a filter bank, and a neural decoder; subsequently, the raw audio signal is encoded based on the encoder to obtain corresponding speech features; and the speech features are separated based on a preset separation strategy to obtain corresponding semantic features and acoustic features; the semantic features are then discretized based on the semantic quantization module to obtain a corresponding semantic codebook; and the acoustic features are subjected to multi-layer residual vector quantization based on the filter bank to obtain a corresponding acoustic residual codebook sequence; further, a target acoustic residual codebook sequence is selected from the acoustic residual codebook sequence based on a preset adaptive discarding strategy; finally, the semantic codebook and the target acoustic residual codebook sequence are converted based on the neural decoder to generate corresponding speech data, and the speech data is output. Based on the above automated processing flow, unlike existing audio coding schemes, this application uses a speech processing model that includes an encoder, semantic quantization module, filter bank and neural decoder. By introducing a separation strategy, a multi-codebook hierarchical quantization structure and an adaptive discard strategy, it can quickly and accurately perform speech generation processing on the original audio signal, achieve natural speech reconstruction processing at low bit rates, and achieve near-lossless speech reconstruction results, thereby effectively improving the quality of speech generation.

[0048] In some alternative implementations, step S205 includes the following steps: Step S2051: Call the preset discrete codebook space.

[0049] In this embodiment, vector quantization requires a predefined discrete codebook space, which consists of a set of discrete codewords, denoted as C={ , , , }, where m is the number of codewords in the codebook, and each codeword It is also a vector, with its dimension and semantic features They are the same (or semantic feature vectors).

[0050] Step S2052: Obtain the preset vector quantization strategy.

[0051] In this embodiment, the above-mentioned vector quantization strategy can specifically refer to the quantization method.

[0052] Step S2053: Based on the vector quantization strategy, the semantic quantization module is used to transform the semantic features in the discrete codebook space to obtain the corresponding target codeword.

[0053] In this embodiment, when the semantic feature vector When inputting into the semantic quantization module, it needs to find the corresponding value in the codebook space C. The closest codeword is used as the discrete semantic codebook. In order to measure The proximity between codewords in the codebook can be measured using distance metrics, such as Euclidean distance. The semantic quantization module will calculate this. With each codeword in the codebook space C The Euclidean distance d between them , Then find the codeword with the smallest distance from it. The codeword with the smallest distance. That is, semantic features The closest codeword (i.e., the target codeword) is used as the discrete semantic codebook. ,Right now = In this way, continuous semantic features are... It was converted into a discrete semantic codebook. This enables the discretization of semantic information, facilitating subsequent storage, transmission, and processing, and also provides a foundation for the semantic reconstruction of speech.

[0054] Step S2054: Use the target codeword as the semantic codebook.

[0055] This application calls a preset discrete codebook space; then obtains a preset vector quantization strategy; subsequently, based on the vector quantization strategy, a semantic quantization module is used to transform semantic features in the discrete codebook space to obtain the corresponding target codeword; the target codeword is then used as the semantic codebook. Based on the above processing flow, this application discretizes the semantic features separated from speech features by using the semantic quantization module, extracts semantically relevant parts through a specific method, and then uses quantization technology to convert them into a discrete form, which facilitates subsequent storage, transmission, and processing, and also provides an accurate data foundation for the reconstruction of the semantic level of speech.

[0056] In some optional implementations of this embodiment, step S206 includes the following steps: Step S2061: Decompose the acoustic features into multiple sub-band signals based on the filter bank.

[0057] In this embodiment, after stripping away the semantic information, the remaining acoustic features are obtained, which are a comprehensive vector containing broadband signal information (such as timbre, pitch, formants, and high-frequency texture). To process these acoustic features more finely, a multi-rate signal processing technique called PQMF (pseudo-quadruple matrix filter bank, i.e., the filter bank mentioned above) is used to decompose them into multiple sub-band signals.

[0058] The core of PQMF lies in its filter bank design, which consists of a carefully designed set of filters with specific frequency response characteristics. For the input residual acoustic feature signal, each filter has its unique frequency passband and stopband characteristics, enabling the separation of the acoustic feature signal across different frequency ranges. Furthermore, during the decomposition process, the input signal undergoes convolution operations within each of these filters.

[0059] However, signals obtained solely through filter convolution suffer from excessively high sampling rates and potential aliasing between different sub-band signals. To address these issues, PQMF also performs downsampling on the output of each filter. Downsampling involves taking a sample at regular intervals, resulting in multiple sub-band signals. Each sub-band signal is a narrowband signal that contains information from the original wideband signal across different frequency ranges, and the reduced sampling rate facilitates subsequent quantization processing.

[0060] Step S2062: Invoke the preset hierarchical quantization method.

[0061] In this embodiment, the aforementioned hierarchical quantization method can specifically employ RVQ (Residual Vector Quantization). The purpose of RVQ is to gradually approximate the original signal through multi-layer quantization, thereby more accurately representing acoustic features and achieving a certain degree of data compression. RVQ operations will then be performed on each sub-band signal.

[0062] Step S2063: Based on the hierarchical quantization method, perform residual vector quantization on each of the sub-band signals to obtain the initial acoustic residual codebook sequence corresponding to each of the sub-band signals.

[0063] In this embodiment, the implementation process of residual vector quantization for each sub-band signal includes: 1) First-level quantization. In the first-level quantization, the sub-band signals are first quantized... This is considered as the raw signal to be quantized. The quantization process requires a predefined codebook C1={ , , , },in It is the number of codewords in the first-level codebook, and each codeword It is a vector whose dimension is the same as the subband signal. The dimension is related (if the subband signal is a discrete sequence, the codeword dimension can correspond to the number of samples within a time window of the sequence). To find the dimension related to the subband signal... The closest codeword is found using a distance metric (such as Euclidean distance). Subband signals are then calculated. With each codeword in codebook C1 The Euclidean distance between them. Then find the specific codeword with the smallest distance. And use this specific codeword as the first-level quantization result. Next, the current sub-band signal is calculated. Compared with the first-level quantization result residual = This residual This reflects the difference between the first-level quantized signal and the original subband signal.

[0064] 2) Subsequent layer quantization. For the second and subsequent layers of quantization, the process is similar to the first layer, but the input signal becomes the residual of the previous layer, and the residual of the second layer is calculated. This process is repeated for multiple layers of quantization until the preset number of quantization layers is reached. After multi-level quantization, a solution for the sub-band signal was obtained. The initial acoustic residual codebook sequence { , , , }, each of which This represents the quantization codebook of the subband signal in the i-th layer.

[0065] Step S2064: Integrate all the initial acoustic residual codebook sequences to obtain the corresponding integrated sequence.

[0066] In this embodiment, after performing the RVQ operation on all K sub-band signals, the initial acoustic residual codebook sequences corresponding to all sub-band signals are integrated to obtain the acoustic residual codebook sequence of the entire acoustic feature. , , , }, each of which It is a combination of all subband signals in the i-th layer quantized codebook (e.g., it can be arranged in subband order into a larger codebook vector). This sequence can more accurately represent the acoustic features after stripping semantic information and provides rich acoustic information for subsequent rate control and speech reconstruction.

[0067] Step S2065: The integrated sequence is used as the acoustic residual codebook sequence.

[0068] This application decomposes acoustic features into multiple sub-band signals using a filter bank approach. Then, a pre-defined hierarchical quantization method is invoked, and residual vector quantization is performed on each sub-band signal using this method to obtain an initial acoustic residual codebook sequence corresponding to each sub-band signal. These initial acoustic residual codebook sequences are then integrated to obtain a corresponding integrated sequence, which is subsequently used as the acoustic residual codebook sequence. Based on this processing flow, this application decomposes complex acoustic features into multiple sub-band signals using a filter bank approach, allowing for more refined quantization operations on signals within different frequency ranges. Furthermore, the hierarchical quantization method gradually approximates the original acoustic signal, and the combination of multiple quantized codebooks more accurately represents the acoustic features. Simultaneously, data compression is achieved to a certain extent, providing rich acoustic information for subsequent rate control and speech reconstruction.

[0069] In some alternative implementations, step S207 includes the following steps: Step S2071: Obtain the current network bandwidth.

[0070] In this embodiment, the aforementioned network bandwidth refers to the currently available network bandwidth. Real-time data can be obtained through an interface with the network service provider or through the system's built-in network monitoring tools, such as obtaining the current network transmission rate (the unit could be kbps, etc.).

[0071] Step S2072: Obtain the current computing resources.

[0072] In this embodiment, the aforementioned computing resources refer to currently available computing resources. Regarding computing resources, the main considerations are processor (CPU) load and available memory size. This information can be obtained by calling system call functions provided by the operating system or using a dedicated performance monitoring library. For example, CPU utilization can be expressed as a percentage, and available memory can be measured in bytes. Assuming the obtained network bandwidth is B (in kbps), the comprehensive evaluation value of computing resources is R. The comprehensive evaluation value R can be calculated based on factors such as CPU utilization and available memory, for example, R = α × CPU_usage + β × (available_memory / total_memory), where α and β are weighting coefficients set according to the actual situation.

[0073] Step S2073: Based on a preset codebook filtering strategy, select a first acoustic residual codebook sequence that corresponds to the network bandwidth and the computing resources from the acoustic residual codebook sequence.

[0074] In this embodiment, the specific implementation process of filtering out the first acoustic residual codebook sequence corresponding to the network bandwidth and the computing resources from the acoustic residual codebook sequence based on the preset codebook filtering strategy will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0075] Step S2074: Use the first acoustic residual codebook sequence as the target acoustic residual codebook sequence.

[0076] This application obtains the current network bandwidth and computing resources, and then, based on a preset codebook filtering strategy, selects a first acoustic residual codebook sequence corresponding to the network bandwidth and computing resources from the acoustic residual codebook sequence; subsequently, this first acoustic residual codebook sequence is used as the target acoustic residual codebook sequence. Based on the above processing flow, this application can dynamically control the bitrate of speech processing according to the actual application scenario and resource conditions. By adaptively discarding part of the acoustic residual codebook layer, the amount of data can be flexibly adjusted according to the limitations of network bandwidth and computing resources while ensuring a certain level of speech quality, thereby achieving efficient speech transmission and processing, and effectively enhancing the adaptability and practicality of the speech processing model in different environments.

[0077] In some alternative implementations, step S2073 includes the following steps: Step S20731: Obtain the preset mapping relationship.

[0078] In this embodiment, based on the design of the speech generation system, a pre-defined correspondence (i.e., mapping relationship) between different bitrates and the number of quantization layers used is established. During the acoustic residual hierarchical quantization stage, the acoustic residual codebook sequence { has already been obtained} , , , },in This refers to the total number of quantization layers. Each quantization layer corresponds to a certain amount of data. As the number of layers increases, the accumulated data volume increases, and the bitrate also increases accordingly. For example, when using layer 1, the bitrate is set to be... =2.2kbps, with a bitrate of 2.2kbps when using the first two layers. =4.4kbps, using all Layer time code rate =8.8kbps. This correspondence can be determined through experiments and experience to ensure a good balance between speech quality and bitrate at different layer levels.

[0079] Step S20732: Based on the mapping relationship, determine the specified quantization layer number corresponding to the network bandwidth and the computing resources.

[0080] In this embodiment, the specified quantization layer n is selected based on the obtained network bandwidth B and the comprehensive evaluation value of computing resources R, combined with a pre-set correspondence (mapping relationship) between bitrate and layer number. If the network bandwidth is narrow, for example, B < (That is, the current bandwidth is less than the bitrate when using the first two layers), and computing resources are also limited, for example, R is less than a certain preset threshold. To ensure smooth voice transmission and prevent system lag due to excessive computation, a smaller number of layers is chosen, such as n=1, corresponding to a bitrate of r1. If network bandwidth is sufficient, for example, B≥... And computational resources also allow it, i.e., R ≥ Then you can choose to use more layers, such as n= To obtain higher speech quality and richer acoustic information, the corresponding bitrate is... If network bandwidth and computing resources are in an intermediate state, for example... ≤B< Then, the number of intermediate layers can be selected according to the actual situation, such as choosing n=2 or other suitable intermediate layers, with a corresponding bitrate of Or the corresponding intermediate bitrate.

[0081] Step S20733: Determine the specified layer corresponding to the specified quantization layer number from the acoustic residual codebook sequence.

[0082] In this embodiment, after determining the specified number of quantization layers n to be used, the specified layers that have not been selected in the acoustic residual codebook sequence can be further determined, i.e., from the (n+1)th layer to the nth layer. The codebook of the layers.

[0083] Step S20734: Discard the specified layer in the acoustic residual codebook sequence to obtain the processed second acoustic residual codebook sequence.

[0084] In this embodiment, an adaptive discarding operation is performed on specified layers in the aforementioned acoustic residual codebook sequence. The discarding can be achieved by releasing the codebook data of these unselected layers from memory within the system, or by marking them as invalid data so that their information is no longer considered in subsequent speech reconstruction and other processing. This adaptive discarding of unselected layers allows for flexible adjustment of the data volume based on actual resource availability, thereby achieving the goal of controlling the bitrate.

[0085] Step S20735: Use the second acoustic residual codebook sequence as the first acoustic residual codebook sequence.

[0086] In this embodiment, after adaptive dropping and bitrate control are completed, the system uses the selected first n layers of quantization codebook for subsequent speech reconstruction and other processing. Based on these quantization codebooks and other necessary information (i.e., semantic information) retained during previous processing, the speech signal is gradually recovered according to the speech reconstruction algorithm and process. Since the number of quantization layers used is dynamically selected based on actual resource conditions, it can adapt to different network bandwidth and computing resource environments while ensuring a certain level of speech quality, thus achieving efficient speech transmission and processing.

[0087] This application obtains a preset mapping relationship; based on the mapping relationship, it determines a specified number of quantization layers corresponding to network bandwidth and computing resources; then, it determines a specified layer corresponding to the specified number of quantization layers from the acoustic residual codebook sequence; subsequently, it discards the specified layer from the acoustic residual codebook sequence to obtain a processed second acoustic residual codebook sequence; this second acoustic residual codebook sequence is then used as the first acoustic residual codebook sequence. Based on the above processing flow, this application, by adaptively discarding some acoustic residual codebook layers, can flexibly adjust the data volume according to the limitations of network bandwidth and computing resources while ensuring a certain level of voice quality. This allows it to adapt to different network bandwidth and computing resource environments while maintaining a certain level of voice quality, thereby achieving efficient voice transmission and processing.

[0088] In some optional implementations of this embodiment, step S208 includes the following steps: Step S2081: Based on a preset splicing strategy, the semantic codebook and the target acoustic residual codebook sequence are spliced ​​to obtain the corresponding splicing vector.

[0089] In this embodiment, the aforementioned semantic codebook is a discrete representation of speech semantic information, containing key information such as the content and meaning conveyed by the speech. The aforementioned acoustic residual codebook sequence is a hierarchical quantization representation of the residual information in the speech acoustic features, excluding the semantically related parts. Different layers represent acoustic residual information of different levels of refinement. The splicing method needs to be determined according to specific design requirements; for example, it can be used to connect them into a longer vector in a certain order.

[0090] Step S2082: Based on the neural decoder, perform frame-by-frame prediction processing on the spliced ​​vector to obtain multiple corresponding waveform signals.

[0091] In this embodiment, the aforementioned neural decoder is a meticulously designed complex neural network structure composed of multiple layers of different types. These layers cooperate with each other and possess powerful nonlinear mapping capabilities. Layer types may include fully connected layers, convolutional layers, and recurrent neural network layers (such as LSTM or GRU). Fully connected layers can perform linear transformations on the input vector and introduce nonlinear factors through activation functions to enhance the network's expressive power; convolutional layers can extract local features from the input vector, which is very effective for locally correlated data such as speech signals; recurrent neural network layers can process sequential data and capture temporal information in the input vector, which is crucial for reconstructing time-continuous signals like speech. The input to the neural decoder is the concatenated vector, which serves as the initial information entering the first layer of the neural network and is then processed step-by-step through each subsequent layer.

[0092] Inside the neural decoder, it begins predicting waveform signals frame by frame based on the input concatenation vector and the complex mapping relationships learned during training with large amounts of speech data. Each frame of waveform signal represents the amplitude changes of speech within a very short time interval and is the basic unit that constitutes the continuous speech waveform. When processing each frame, the neural decoder comprehensively considers all information in the concatenation vector, including semantic information and acoustic residual information. For example, in the processing of a certain layer, it may determine the approximate content category of the speech in this frame based on semantic information (such as whether it is a vowel or a consonant), and then further refine the specific shape of the waveform of this frame by combining acoustic residual information, such as the amplitude and frequency. Through continuous processing and optimization across multiple layers, the neural decoder gradually generates accurate waveform signals for each frame.

[0093] Step S2083: Combine all the waveform signals to obtain the corresponding speech signal sequence.

[0094] In this embodiment, after the neural decoder has completed the prediction of all T frames of waveform signals, the final step is to combine these frames of waveform signals to form a complete speech signal. Since each frame of waveform signal is generated sequentially in chronological order, the combination process simply involves connecting them according to their generation time order to obtain a continuous speech waveform signal sequence. This speech signal sequence completes the reconstruction process from discrete semantic and acoustic codebooks to continuous speech waveforms, achieving complete speech restoration and output, and presenting the speech to the user in a natural and clear manner.

[0095] Step S2084: Optimize the speech signal sequence based on a preset optimization strategy to obtain the corresponding speech data.

[0096] In this embodiment, the specific implementation process of optimizing the speech signal sequence based on the preset optimization strategy to obtain the corresponding speech data will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0097] Step S2085: Output the voice data.

[0098] This application concatenates the semantic codebook and the target acoustic residual codebook sequence based on a preset concatenation strategy to obtain a corresponding concatenation vector. Then, a neural decoder performs frame-by-frame waveform signal prediction processing on the concatenation vector to obtain multiple waveform signals. All waveform signals are then combined to obtain a corresponding speech signal sequence. Subsequently, a preset optimization strategy is used to optimize the speech signal sequence to obtain the corresponding speech data. Finally, the speech data is processed for output. Based on this processing flow, this application, by using a neural decoder, can convert the processed discrete codebook back into a continuous speech waveform signal. By learning from a large amount of speech data, the neural decoder masters the mapping relationship from discrete representation to continuous waveform, thus accurately reconstructing a high-quality speech signal from the input semantic codebook and the target acoustic residual codebook sequence, intelligently and accurately achieving complete speech restoration and output.

[0099] In some optional implementations of this embodiment, step S2084 includes the following steps: Step S20841: Perform echo cancellation processing on the speech signal sequence to obtain the corresponding first speech signal.

[0100] In this embodiment, in voice communication systems such as teleconferences and hands-free calls, the sound from the local speaker may be picked up again by the microphone, creating an echo and affecting voice quality. Echo cancellation technology estimates the echo path and generates a corresponding echo signal, then subtracts the echo signal from the received signal to eliminate the echo.

[0101] An adaptive filter can be used to automatically adjust the filter parameters based on the input and output signals to better estimate the echo path and eliminate the echo, thus avoiding howling and interference. In this way, by eliminating the echo, the clarity and comfort of voice communication are improved, and the interference of echoes on speech understanding and communication is avoided.

[0102] Step S20842: Optimize the encoding parameters of the first speech signal to obtain the corresponding second speech signal.

[0103] In this embodiment, audio encoding is the process of converting speech signals into digital codes. The selection of encoding parameters directly affects the quality and data volume of the encoded speech. By optimizing the encoding parameters, it is possible to reduce the data volume while maintaining a certain level of speech quality, or to improve speech quality without increasing the data volume.

[0104] In Adaptive Multi-Rate (AMR) coding, the coding rate is dynamically adjusted based on network bandwidth and voice quality requirements. When network bandwidth is limited, a lower coding rate is selected to reduce data transmission; when network bandwidth is sufficient, a higher coding rate is selected to improve voice quality. Simultaneously, parameters such as quantization parameters and frame length can be optimized to further improve coding performance. In this way, an optimal balance between voice quality and data volume is achieved in different application scenarios, improving the efficiency and reliability of voice transmission.

[0105] Step S20843: Perform equalization processing on the second speech signal to obtain the corresponding third speech signal.

[0106] In this embodiment, the energy distribution of the speech signal varies across different frequency bands, which affects the timbre and clarity of the speech. Equalization processing improves the frequency response characteristics of the speech by adjusting the gain of the speech signal in different frequency bands, making the timbre of the speech more natural and clear.

[0107] Equalization is achieved through the use of either a graphic equalizer or a parametric equalizer. A graphic equalizer divides the audio frequency range into multiple bands, allowing users to adjust the gain of each band to change the frequency response of the speech. A parametric equalizer, on the other hand, precisely adjusts the frequency characteristics of the speech by setting parameters such as center frequency, bandwidth, and gain. For example, increasing the gain of high-frequency bands can enhance the clarity and brightness of the speech; decreasing the gain of low-frequency bands can reduce the muddiness in the speech.

[0108] Step S20844: Perform environmental adaptation processing on the third speech signal to obtain the corresponding fourth speech signal.

[0109] In this embodiment, different environments have different acoustic characteristics, such as room reverberation time and reflection characteristics, which affect the auditory effect of speech. By simulating the effect of a specific environment, speech can be made more consistent with the characteristics of that environment, enhancing the realism and immersion of the speech.

[0110] This method employs a convolutional reverberation algorithm to simulate the reverberation effect of the environment. This algorithm performs a convolution operation between the speech signal and the impulse response of the environment to obtain a speech signal with the reverberation characteristics of that environment. For example, in music production, the reverberation effect of a concert hall can be simulated to make the music more spatial and layered; in speech synthesis, different scenarios, such as classrooms and conference rooms, can be simulated to make the synthesized speech more natural. In this way, the realism and immersion of the speech are enhanced, making the speech more suitable for specific environments and increasing its application value.

[0111] Step S20845: Use the fourth voice signal as the voice data.

[0112] This application obtains a first speech signal by performing echo cancellation on a speech signal sequence; then, it optimizes the encoding parameters of the first speech signal to obtain a second speech signal; next, it performs equalization on the second speech signal to obtain a third speech signal; subsequently, it performs environment adaptation processing on the third speech signal to obtain a fourth speech signal; finally, it uses the fourth speech signal as speech data. Based on the above processing flow, this application optimizes the speech signal from multiple dimensions by performing echo cancellation, encoding parameter optimization, equalization processing, and environment adaptation processing on the speech signal sequence, thereby effectively improving speech quality.

[0113] In some optional implementations, the training process of the above speech processing model includes: 1. Audio input and feature encoding stage. 1) Input signal preparation: The original audio signal is adjusted to a waveform sequence with a sampling rate of 24kHz to unify the sampling standard and provide a basis for subsequent processing. 2) Neural encoder processing: The adjusted waveform sequence is input into the neural encoder, which consists of convolutional blocks and self-attention blocks. The convolutional blocks initially extract local audio features, and the self-attention blocks capture long-distance dependencies in the audio sequence, mapping long-time waveforms to low-frame-rate continuous embedding vectors (e.g., 50Hz), compressing the time dimension, retaining perceptually important speech features, and outputting intermediate latent vector representations. Where x is the original audio, This is the encoding function. This stage learns audio feature representations through a neural encoder, preparing for subsequent semantic and acoustic feature extraction.

[0114] 2. Semantic Channel Modeling and Discretization Stage. 1) Semantic Feature Extraction: Extracting semantically relevant parts directly related to the language content from the features h output by the encoder. 1) Excluding acoustic factors such as timbre and intonation. 2) Semantic quantization module processing: A semantic quantization module is introduced, which discretizes continuous semantic features through vector quantization (VQ) mechanism to obtain... 3) External teacher model supervision: An external teacher model supervision mechanism is adopted. The pre-trained Hubert model is used to extract the semantic embedding S corresponding to the speech, and the output is quantized by semantic similarity loss constraint. The loss function is: By minimizing this loss function, the semantic codebook representation is aligned with the language text embedding space, achieving content consistency and cross-speaker generalization. The goal of this stage is to enable the model to learn accurate semantic information representations.

[0115] 3. Acoustic Residual Layered Quantization Stage (RVQ+PQMF Joint). 1) Signal Processing After Semantic Information Removal: After removing semantic information, the remaining signal contains acoustic features such as timbre, pitch, formants, and high-frequency texture. 2) Multilayer Residual Vector Quantization (RVQ) Modeling: Multilayer residual vector quantization is used to model the acoustic features in layers. First, the initial residual is calculated. Then for each layer i, we have , ,in For the i-th layer codebook, each layer is responsible for encoding the residual information that the previous layer failed to represent. 3) Introduction of PQMF subband supervision mechanism: A PQMF subband supervision mechanism is introduced into RVQ training. A filter bank is used to decompose the audio into multi-subband signals (such as low frequency, mid frequency, and high frequency), specifying specific frequency band learning targets for different codebook layers. For example, layers 1-4 correspond to low frequency (fundamental frequency and formant region); layers 5-8 correspond to mid frequency (channel resonance and energy dynamics); and layers 9-16 correspond to high frequency (clarity and breathiness). Each layer's RVQ output... Its corresponding sub-band signal The two sides are constrained by cosine similarity, and the loss function is... ,in, , To align the projection matrices and unify different spatial dimensions, the goal of this stage is to enable the model to learn accurate acoustic feature representations.

[0116] 4. Adaptive Dropout and Code Rate Control Stage: Training Phase. Adaptive Dropout Strategy: During the training phase, an adaptive dropout strategy is employed, randomly dropping a portion of the codebook layers to enhance model robustness and inter-layer independence. Specifically, a semi-Gaussian distribution is used for sampling, with the probability distribution being... ,in, This indicates the number of active codebook layers, with μ and σ controlling the distribution of active layers. This stage involves randomly discarding some layers, ensuring the model learns effective features across different layer combinations and improving its generalization ability.

[0117] 5. Neural Decoding and Waveform Reconstruction Stage. 1) Input Concatenation at the Decoding End: At the decoding end, the semantic codebook zsem and the acoustic residual codebook sequence { , , , The concatenated data is then input into the neural decoder. 2) Neural decoder processing: The decoder employs a generative network based on Conv-TDF blocks and a self-attention structure to predict waveform signals frame by frame. Its goal is to minimize reconstruction error and perceptual loss; the loss function is... ,in, To counteract loss in the discriminator and improve subjective sound quality; This stage involves perceptual metric for speech quality evaluation. By continuously optimizing the loss function, the decoder can accurately reconstruct high-fidelity speech signals, completing model training and obtaining a well-trained speech processing model.

[0118] Furthermore, the innovation of this application lies in theoretically modeling the semantic space and acoustic space as orthogonal. The latent space generated by the encoder can be regarded as a joint distribution P(h|x), in which traditional RVQ is directly quantized. It is decomposed into the following by semantic teacher constraints: The former corresponds to linguistic information, while the latter represents conditional acoustic features. The recursive distribution of the RVQ layers achieves layer-by-layer approximation of the residual space. Therefore, this application actually establishes a hierarchical probabilistic approximation model, which enables semantic, acoustic, and high-frequency features to be modeled independently layer by layer, achieving a balance between high compression rate and high-fidelity reconstruction.

[0119] In some alternative implementations, the user information obtained is subject to user consent and complies with relevant laws and policies.

[0120] Furthermore, any software tools or components not belonging to our company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0121] Furthermore, the proposed Multi-Codebook Codec is a high-fidelity neural audio compression system based on multi-codebook residual quantization. Unlike traditional coding schemes, it not only pursues data compression efficiency but also emphasizes the separation and modeling of semantic (content) and acoustic (timbre, frequency, formants) features. The overall system flow can be summarized as follows: input audio -- feature extraction and encoding -- semantic and acoustic dual-channel quantization -- adaptive codebook activation and compression -- decoding and reconstruction -- output high-fidelity speech. This design of "semantic-acoustic dual-stream decoupling + adaptive residual quantization" overcomes the limitations of traditional neural codecs such as semantic and acoustic aliasing, codebook coupling, and unadjustable bit rate.

[0122] Furthermore, this application offers the following improvements: 1. High fidelity and ultra-low bitrate coexist. Speech can be reconstructed almost losslessly at an extremely low bitrate of 2.2kbps, achieving a compression ratio of 170×. Through semantic-acoustic dual-channel decoupling, the model avoids detail loss and high-frequency collapse, achieving natural sound quality reconstruction at low bitrates.

[0123] 2. Significantly improved interpretability and controllability. Through PQMF subband supervision, each codebook layer corresponds to a specific frequency range, making the model's internal structure interpretable and facilitating parameter tuning. The adaptive dropout strategy supports flexible bit rate switching, dynamically adjusting the compression level according to application needs, making it suitable for various scenarios such as TTS, voice storage, and transmission.

[0124] 3. Highly efficient training and strong generalization ability. The adaptive training mechanism using non-uniform sampling reduces unnecessary computation, improves convergence speed, and prevents overfitting. Experiments show that it maintains stable performance under different languages, speakers, and recording conditions, providing a unified coding interface for multilingual speech synthesis and speech LLM.

[0125] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0126] It should be emphasized that, to further ensure the privacy and security of the aforementioned voice data, the voice data can also be stored in a blockchain node.

[0127] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0128] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0129] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0130] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0131] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of an artificial intelligence-based speech generation device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0132] like Figure 3 As shown, the AI-based speech generation device 300 described in this embodiment includes: a receiving module 301, a calling module 302, an encoding module 303, a separation module 304, a discretization module 305, a quantization module 306, a filtering module 307, and a processing module 308. Wherein: The receiving module 301 is used to receive the input raw audio signal; The calling module 302 is used to call a preset speech processing model; wherein, the speech processing model includes an encoder, a semantic quantization module, a filter bank, and a neural decoder; Encoding module 303 is used to encode the original audio signal based on the encoder to obtain corresponding speech features; The separation module 304 is used to separate speech features based on a preset separation strategy to obtain corresponding semantic features and acoustic features. Discrete module 305 is used to perform discrete processing on the semantic features based on the semantic quantization module to obtain the corresponding semantic codebook; Quantization module 306 is used to perform multi-layer residual vector quantization processing on the acoustic features based on the filter bank to obtain the corresponding acoustic residual codebook sequence; The filtering module 307 is used to filter out the target acoustic residual codebook sequence from the acoustic residual codebook sequence based on a preset adaptive discarding strategy. The processing module 308 is used to perform conversion processing on the semantic codebook and the target acoustic residual codebook sequence based on the neural decoder to generate corresponding speech data, and to perform output processing on the speech data.

[0133] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the artificial intelligence-based speech generation method in the aforementioned implementation method, and will not be repeated here.

[0134] In some optional implementations of this embodiment, the discrete module 305 includes: The first calling submodule is used to call the preset discrete codebook space; The first acquisition submodule is used to acquire the preset vector quantization strategy; The conversion submodule is used to convert the semantic features in the discrete codebook space based on the vector quantization strategy, using the semantic quantization module to obtain the corresponding target codeword. The first determining submodule is used to use the target codeword as the semantic codebook.

[0135] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the artificial intelligence-based speech generation method in the aforementioned implementation method, and will not be repeated here.

[0136] In some optional implementations of this embodiment, the quantization module 306 includes: A decomposition submodule is used to decompose the acoustic features into multiple sub-band signals based on the filter bank; The second calling submodule is used to call the preset layered quantization method; The quantization submodule is used to perform residual vector quantization on each of the sub-band signals based on the hierarchical quantization method to obtain an initial acoustic residual codebook sequence corresponding to each of the sub-band signals. The integration submodule is used to integrate all the initial acoustic residual codebook sequences to obtain the corresponding integrated sequence; The second determining submodule is used to use the integrated sequence as the acoustic residual codebook sequence.

[0137] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the artificial intelligence-based speech generation method in the aforementioned implementation method, and will not be repeated here.

[0138] In some optional implementations of this embodiment, the filtering module 307 includes: The second acquisition submodule is used to acquire the current network bandwidth; The third acquisition submodule is used to acquire the current computing resources; The filtering submodule is used to filter out the first acoustic residual codebook sequence corresponding to the network bandwidth and the computing resources from the acoustic residual codebook sequence based on a preset codebook filtering strategy. The third determining submodule is used to use the first acoustic residual codebook sequence as the target acoustic residual codebook sequence.

[0139] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the artificial intelligence-based speech generation method in the aforementioned implementation method, and will not be repeated here.

[0140] In some optional implementations of this embodiment, the filtering submodule includes: The acquisition unit is used to acquire a preset mapping relationship; The first determining unit is used to determine, based on the mapping relationship, the specified quantization layer number corresponding to the network bandwidth and the computing resources; The second determining unit is used to determine the specified layer corresponding to the specified quantization layer number from the acoustic residual codebook sequence; An operation unit is used to perform a discard operation on the specified layer in the acoustic residual codebook sequence to obtain a processed second acoustic residual codebook sequence; The third determining unit is used to use the second acoustic residual codebook sequence as the first acoustic residual codebook sequence.

[0141] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the artificial intelligence-based speech generation method in the aforementioned implementation method, and will not be repeated here.

[0142] In some optional implementations of this embodiment, the processing module 308 includes: The first processing submodule is used to perform echo cancellation processing on the speech signal sequence to obtain the corresponding first speech signal; The second processing submodule is used to perform encoding parameter optimization processing on the first speech signal to obtain the corresponding second speech signal; The third processing submodule is used to perform equalization processing on the second speech signal to obtain the corresponding third speech signal; The fourth processing submodule is used to perform environmental adaptation processing on the third speech signal to obtain the corresponding fourth speech signal; The fourth determining submodule is used to use the fourth voice signal as the voice data.

[0143] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the artificial intelligence-based speech generation method in the aforementioned implementation method, and will not be repeated here.

[0144] Some optional implementations of this embodiment include: The judgment module is used to determine whether the user-triggered business behavior operation on the target page has been received; The determination module is used to determine the URL of the redirect link page corresponding to the target page based on the business behavior operation if the condition is met. The acquisition module is used to acquire the position parameters corresponding to the resource positions in the target page based on the resource position embedding rules; The splicing module is used to splice the location parameters into the URL of the redirect link page.

[0145] In this embodiment, the operations performed by the above modules or units correspond one-to-one with the steps of the artificial intelligence-based speech generation method in the aforementioned implementation method, and will not be repeated here.

[0146] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed] for details. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0147] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0148] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0149] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for speech generation methods based on artificial intelligence. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0150] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the artificial intelligence-based speech generation method.

[0151] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0152] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the artificial intelligence-based speech generation method described above.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0154] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

[0155] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. A speech generation method based on artificial intelligence, characterized in that, Includes the following steps: Receive the raw audio signal input; A preset speech processing model is invoked; wherein, the speech processing model includes an encoder, a semantic quantization module, a filter bank, and a neural decoder; The encoder is used to encode the original audio signal to obtain the corresponding speech features. Based on a preset separation strategy, speech features are separated to obtain corresponding semantic and acoustic features. Based on the semantic quantization module, the semantic features are discretized to obtain the corresponding semantic codebook; Based on the filter bank, the acoustic features are subjected to multi-layer residual vector quantization to obtain the corresponding acoustic residual codebook sequence. Based on a preset adaptive discarding strategy, a target acoustic residual codebook sequence is selected from the acoustic residual codebook sequence. The neural decoder performs conversion processing on the semantic codebook and the target acoustic residual codebook sequence to generate corresponding speech data, and then outputs the speech data.

2. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of discretizing the semantic features based on the semantic quantization module to obtain the corresponding semantic codebook specifically includes: Call the preset discrete codebook space; Obtain the preset vector quantization strategy; Based on the vector quantization strategy, the semantic quantization module is used to transform the semantic features in the discrete codebook space to obtain the corresponding target codeword. The target codeword is used as the semantic codebook.

3. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of performing multi-layer residual vector quantization on the acoustic features based on the filter bank to obtain the corresponding acoustic residual codebook sequence specifically includes: The acoustic features are decomposed into multiple sub-band signals based on the filter bank; Invoke the preset layered quantization method; Based on the hierarchical quantization method, residual vector quantization is performed on each of the sub-band signals to obtain the initial acoustic residual codebook sequence corresponding to each of the sub-band signals. All the initial acoustic residual codebook sequences are integrated to obtain the corresponding integrated sequence; The integrated sequence is used as the acoustic residual codebook sequence.

4. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of selecting the target acoustic residual codebook sequence from the acoustic residual codebook sequence based on the preset adaptive discarding strategy specifically includes: Get the current network bandwidth; Get the current computing resources; Based on a preset codebook filtering strategy, a first acoustic residual codebook sequence corresponding to the network bandwidth and the computing resources is filtered out from the acoustic residual codebook sequence; The first acoustic residual codebook sequence is used as the target acoustic residual codebook sequence.

5. The artificial intelligence-based speech generation method according to claim 4, characterized in that, The step of selecting the first acoustic residual codebook sequence corresponding to the network bandwidth and the computing resources from the acoustic residual codebook sequence based on the preset codebook filtering strategy specifically includes: Obtain the preset mapping relationship; Based on the mapping relationship, the specified number of quantization layers corresponding to the network bandwidth and the computing resources is determined; Determine the specified layer corresponding to the specified number of quantization layers from the acoustic residual codebook sequence; The specified layer is discarded in the acoustic residual codebook sequence to obtain the processed second acoustic residual codebook sequence; The second acoustic residual codebook sequence is used as the first acoustic residual codebook sequence.

6. The artificial intelligence-based speech generation method according to claim 1, characterized in that, The step of converting the semantic codebook and the target acoustic residual codebook sequence based on the neural decoder to generate corresponding speech data, and then outputting the speech data, specifically includes: The semantic codebook and the target acoustic residual codebook sequence are concatenated based on a preset concatenation strategy to obtain the corresponding concatenation vector. The neural decoder performs frame-by-frame waveform prediction processing on the spliced ​​vector to obtain multiple corresponding waveform signals. All the waveform signals are combined to obtain the corresponding speech signal sequence; The speech signal sequence is optimized based on a preset optimization strategy to obtain the corresponding speech data. The voice data is then processed for output.

7. The artificial intelligence-based speech generation method according to claim 6, characterized in that, The step of optimizing the speech signal sequence based on a preset optimization strategy to obtain the corresponding speech data specifically includes: The speech signal sequence is subjected to echo cancellation processing to obtain the corresponding first speech signal; The encoding parameters of the first speech signal are optimized to obtain the corresponding second speech signal; The second speech signal is subjected to equalization processing to obtain the corresponding third speech signal; The third speech signal is subjected to environmental adaptation processing to obtain the corresponding fourth speech signal; The fourth speech signal is used as the speech data.

8. A speech generation device based on artificial intelligence, characterized in that, include: The receiving module is used to receive the input raw audio signal; The calling module is used to call a preset speech processing model; wherein, the speech processing model includes an encoder, a semantic quantization module, a filter bank, and a neural decoder; The encoding module is used to encode the original audio signal based on the encoder to obtain corresponding speech features; The separation module is used to separate speech features based on a preset separation strategy to obtain corresponding semantic and acoustic features. The discrete module is used to perform discrete processing on the semantic features based on the semantic quantization module to obtain the corresponding semantic codebook; The quantization module is used to perform multi-layer residual vector quantization on the acoustic features based on the filter bank to obtain the corresponding acoustic residual codebook sequence. The filtering module is used to filter out the target acoustic residual codebook sequence from the acoustic residual codebook sequence based on a preset adaptive discarding strategy. The processing module is used to perform conversion processing on the semantic codebook and the target acoustic residual codebook sequence based on the neural decoder to generate corresponding speech data, and to perform output processing on the speech data.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the artificial intelligence-based speech generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the artificial intelligence-based speech generation method as described in any one of claims 1 to 7.

Images