Audio coding method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an audio coding method and device, electronic equipment and a readable storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: extracting a first Mel spectrum feature of a first audio; dividing the first Mel spectrum feature into N sub-bands in different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges are not overlapped with each other; performing quantization coding on each sub-band feature to obtain a quantization coding value of each sub-band feature; determining an index value of each sub-band feature according to a preset quantization code table and the quantization coding value of each sub-band feature; wherein the quantization code table comprises a mapping relation between quantization coding values and index values, the index values are integers, the index values of the N sub-band features form a group of training data corresponding to the first audio, and the training data are used for training a text voice model.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an audio encoding method, device, electronic device and readable storage medium. Background Art

[0002] Audio quantization coding, also known as audio coding, refers to the collection of audio signals in digital form for storage and transmission, and the use of a fixed number of bits to represent each sample value. The audio quantization coding results can be used to train models, for example, training text to generate a large speech model, which can be referred to as a "text-generated speech model". This coding method can reduce the size and computing requirements of the model, thereby reducing storage and computing costs, and helping the model run on devices with limited resources. However, in related technologies, the quality of audio quantization coding results is low, resulting in greater loss of the reconstructed audio of the audio quantization coding results compared to the original audio. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an audio encoding method, device, electronic device and readable storage medium, which can improve the quality of audio quantization encoding results, thereby reducing the loss between the reconstructed audio of the audio quantization encoding results and the original audio.

[0004] In a first aspect, an embodiment of the present application provides an audio encoding method, the method comprising:

[0005] Extracting a first Mel-spectrogram feature of the first audio;

[0006] Dividing the first Mel-spectrum frequency feature into N sub-bands with different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other;

[0007] Performing quantization encoding on each of the sub-band features to obtain a quantization encoding value of each of the sub-band features;

[0008] According to a preset quantization code table and the quantization coding value of each sub-band feature, an index value of each sub-band feature is determined; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of the N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

[0009] In a second aspect, an embodiment of the present application provides an audio encoding device, the device comprising:

[0010] An extraction module, used for extracting a first Mel frequency spectrum feature of a first audio;

[0011] A division module, used for dividing the first Mel frequency spectrum feature into N sub-bands with different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other;

[0012] An encoding module, used for performing quantization encoding on each of the sub-band features to obtain a quantization encoding value of each of the sub-band features;

[0013] A determination module is used to determine the index value of each sub-band feature according to a preset quantization code table and the quantization coding value of each sub-band feature; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the audio encoding method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the audio encoding method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the steps of the audio encoding method as described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the audio encoding method as described in the first aspect.

[0018] In an embodiment of the present application, a first Mel-spectrogram feature of a first audio is extracted; the first Mel-spectrogram feature is divided into N sub-bands of different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other; each sub-band feature is quantized and encoded to obtain a quantized coding value of each sub-band feature; an index value of each sub-band feature is determined according to a preset quantization code table and the quantization coding value of each sub-band feature; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of the N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

[0019] It can be seen that in the embodiment of the present application, since the Mel spectrum feature of the audio is more in line with the auditory characteristics of the human ear and has a good characterization ability for the audio signal, the restoration effect of the audio based on the Mel spectrum feature is better. Therefore, the Mel spectrum feature of the audio is divided into more fine-grained sub-band features based on the frequency range, and each sub-band feature is quantized and encoded to obtain a quantized coding value, and the quantized coding value with a longer code length and a decimal type is converted into an index value with a shorter code length and an integer type. On the one hand, the quality of the audio quantization coding result can be improved, thereby reducing the loss between the reconstructed audio and the original audio of the audio quantization coding result; on the other hand, the training data composed of the index values ​​of the N sub-band features has a relatively small data volume and an integer type, which is easier to process. Therefore, when the Wensheng speech model is trained based on the training data, the performance and speed of the model training can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of an audio encoding method provided by some embodiments of the present application;

[0021] Figure 2 is an example diagram of a Mel frequency spectrum feature map provided by some embodiments of the present application;

[0022] Figure 3 is a structural block diagram of an audio encoding device provided by some embodiments of the present application;

[0023] Figure 4 is a schematic diagram of the structure of an electronic device provided by some embodiments of the present application;

[0024] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings of the embodiments of the present application to clearly describe the technical solutions of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present application belong to the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the target object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0027] To facilitate understanding, some relevant concepts and application scenarios involved in the embodiments of the present application are first introduced below.

[0028] 1. Related concepts

[0029] The Vincent Speech Model, also known as the "text-generated speech model or text-generated speech model", is a technology that uses artificial intelligence technology to convert text into speech. This model trains a large amount of text and speech data through a deep learning algorithm, and can generate speech that is difficult to distinguish from human speech. The working principle of the Vincent Speech Model mainly includes the following steps: 1) Data preparation: Collect various types of text and speech data, and pre-process and annotate the data. 2) Model training: Use a deep learning model (such as recurrent neural network RNN, long short-term memory network LSTM, Transformer or diffusion model) to train the data, and generate grammatical and semantic text by learning language patterns and rules. 3) Speech synthesis: The model generates corresponding speech based on the input text, including acoustic information such as timbre, tone, rhythm and emotion.

[0030] Mel Spectrogram is a spectrum representation method that converts the frequency of an audio signal into a Mel scale, which is designed to better simulate human perception of sound frequency. The Mel scale is a frequency unit based on human auditory perception. It is similar to the linear frequency in the low-frequency area and has a logarithmic relationship with the linear frequency in the high-frequency area. This conversion can better simulate the human ear's ability to perceive sounds of different frequencies.

[0031] The high-fidelity generative adversarial network residual module is mainly used to solve the "degradation problem" in deep neural network training. As the number of network layers increases, traditional deep neural networks are prone to gradient vanishing or gradient exploding problems, making model training difficult. The residual module solves this problem by introducing skip connections, allowing gradients to propagate directly, allowing the network to be trained more stably.

[0032] Audio quantization coding refers to the storage and transmission of speech, music or other audio signals in digital form, and the use of a fixed number of bits to represent each sample value. This quantization can reduce the model size and computing requirements, thereby reducing storage and computing costs, and helping the model run on devices with limited resources, such as mobile devices and embedded devices. The quality of quantization coding affects the size of the loss from the quantized value to the restored value.

[0033] 2. Application Scenarios

[0034] Large language models have achieved remarkable success in various natural language processing tasks using self-supervised pre-training on large amounts of text data. Inspired by this success, research has extended language modeling methods to speech processing, aiming to build a unified model that can handle both speech understanding and generation tasks, such as a large text-to-speech model, which can be referred to as a "text-to-speech model".

[0035] At present, the application scenarios of Wensheng speech models are very wide, such as voice assistants and intelligent customer service, education and training, information broadcasting and message notifications, entertainment and media, assisted reading and hearing impairment, multilingual support and other scenarios.

[0036] Among them, in the scenarios of voice assistants and intelligent customer service, the Vincent voice model can enable voice assistants and intelligent customer service to interact with users more naturally and smoothly. By generating natural speech, these systems can provide a better user experience, such as in the fields of smart speakers, virtual personal assistants, etc.

[0037] In education and training scenarios, the Wensheng speech model can be used to generate personalized teaching content and exercises, providing an intelligent online learning experience; in addition, it can also serve as a teaching assistant to assist teachers in their teaching work.

[0038] In the scenarios of information broadcasting and message notification, the Wensheng speech model can be used to automatically generate voice broadcast content, such as news broadcasts, weather forecasts, traffic information, etc., to improve the efficiency and accuracy of information transmission.

[0039] In entertainment and media scenarios, the Wensheng speech model can be used to generate audiobooks, dubbing, game character dialogues, etc., to enhance user experience and content richness.

[0040] In scenarios of assisting reading and hearing impairment, for people with limited vision or hearing impairment, the Wensheng speech model can convert text content into speech to help them better obtain information.

[0041] In the multi-language support scenario, the Wensheng speech model can support multi-language generation, meet the language requirements of different countries and regions, and further expand its scope of application.

[0042] However, the key challenge in training a Vincent speech model is the continuity of speech signals, which requires an effective tokenization method to discretize the audio input, that is, to quantize and encode the audio so that it can become the training data of the Vincent speech model, and the Vincent speech model is trained based on the training data. In the related art, audio quantization coding can be roughly divided into two categories: semantic labeling and acoustic labeling. Semantic labels are extracted from self-supervised pre-trained speech models, in which speech signals are first encoded into speech representations and then clustered into semantic labels using the k-means method; however, since the speaker identity and other details of the original speech are lost during training, this self-supervised pre-trained model is useless for high-fidelity speech synthesis. In contrast, acoustic labels can be obtained from audio compression models, which prioritize acoustic reconstruction but lose semantic information, resulting in poor results in generated audio.

[0043] It can be seen that in the related art, the quality of the audio quantization encoding result is low, resulting in a greater loss of the reconstructed audio of the audio quantization encoding result compared to the original audio. At this time, if the audio quantization encoding result generated in the related art is used as the training data of the Wensheng speech model, the speech generation ability of the trained Wensheng speech model is poor.

[0044] In order to solve the above-mentioned technical problems, the embodiments of the present application provide an audio encoding method, device, electronic device and readable storage medium, which can improve the quality of the audio quantization encoding results, so that based on the audio quantization encoding results as training data of the Wensheng speech model, a Wensheng speech model with strong speech generation ability can be efficiently trained.

[0045] An audio encoding method provided in an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0046] It should be noted that the audio encoding method provided in the embodiment of the present application is applicable to electronic devices. In practical applications, the electronic devices include but are not limited to: mobile terminals such as mobile phones, tablet computers, laptops, PDAs, and computer devices such as servers and desktops. The embodiment of the present application does not limit this.

[0047] Figure 1 is a flowchart of an audio encoding method provided by some embodiments of the present application, such as Figure 1 As shown, the method includes the following steps: step 101, step 102, step 103 and step 104;

[0048] In step 101, a first mel-spectrogram feature of a first audio is extracted.

[0049] At present, if you want to train a Wensheng speech model, you first need to obtain the training data required for model training. In addition to descriptive text, the training data also needs to include the audio corresponding to the descriptive text. However, since the audio is used directly for model training, the model training cost is relatively high (including storage cost, computing cost, etc.). Therefore, it is necessary to quantize and encode the audio and use the quantized encoding result of the audio to train the Wensheng speech model to reduce the training cost of the model.

[0050] In the embodiment of the present application, the first audio is the training data of the Wensheng speech model. Since the first audio is directly used for model training, the training cost of the model is relatively high. Therefore, it is necessary to quantize and encode the first audio, and use the quantized encoding result of the first audio to train the Wensheng speech model to reduce the training cost of the Wensheng speech model. The process of quantizing and encoding the first audio is described in detail in detail. Figure 1 The contents of steps 101 to 104 in the illustrated embodiment.

[0051] For example, if the audio required for the Wensheng speech model training is K, then use Figure 1 The audio quantization encoding method of step 101 to step 104 in the illustrated embodiment can perform quantization encoding processing on the K audios to obtain quantization encoding results of the K audios. The quantization encoding results of the K audios and the description texts of the K audios are used to train the Wensheng speech model.

[0052] In the embodiment of the present application, the first audio includes but is not limited to: audio recorded by various recording devices, audio from video files, audio provided by various online platforms, voice input by users, etc.

[0053] In the embodiment of the present application, considering that the Mel spectrum is a spectrum representation method that converts the frequency of an audio signal into the Mel scale, this conversion can better simulate the human ear's perception of sounds of different frequencies. Therefore, the first audio can be converted into a spectrum representation of the Mel scale to obtain a first Mel spectrum feature of the first audio.

[0054] In an embodiment of the present application, the calculation process of the first Mel spectrum may include the following steps: 1) Performing a short-time Fourier transform (STFT) on the first audio, converting the audio signal of the first audio from the time domain to the frequency domain, and obtaining a spectrum diagram. 2) Using a Mel filter group to process the spectrum diagram obtained by STFT, wherein each Mel filter corresponds to a specific Mel frequency band, and the output of the Mel filter represents the energy of the frequency band. For example, if Q Mel frequency bands are required, Q different Mel filters are required, where Q is an integer greater than 1. 3) Performing a logarithmic transformation on the output of the Mel filter group, usually using decibels (dB) as the unit, to compress the dynamic range of the high-frequency part so that it is more suitable for the first Mel spectrum characteristics of the human auditory system.

[0055] In the embodiment of the present application, considering that the larger the frequency range of the Mel spectrum feature, the more comprehensive the audio information contained in the Mel spectrum feature, and the smaller the loss of audio information, but the larger the frequency range of the Mel spectrum feature, the greater the amount of calculation in the subsequent quantization encoding processing, so in order to balance the relationship between the two, the frequency range of the first Mel spectrum feature can be set according to actual needs.

[0056] For example, for the first audio with a sampling rate of 44.1kHz, the frequency range of the first Mel spectrum feature can be set to [0,80]Hz, that is, the frequency dimension is 80. If the Mel bandwidth of the Mel filter is 10Hz, 8 different Mel filters are required. These 8 different Mel filters constitute a Mel filter bank. The basic parameters of each Mel filter can be: the window size is 2048 samples, and the jump length is 512. If the duration of the first audio is T, the first Mel spectrum feature of the first audio can be recorded as [1,80,T], where 1 represents 1 part, 80 represents the frequency dimension (i.e., frequency range), and T represents the duration of the first audio.

[0057] In step 102, the first Mel-spectrum frequency feature is divided into N sub-bands in different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other.

[0058] In the embodiment of the present application, the bandwidths of the N different frequency ranges may be the same or different.

[0059] In some embodiments, the frequency range of the first mel-spectrum frequency feature may be divided into N equal parts to obtain N different frequency ranges. In this case, the bandwidths of the N different frequency ranges are the same.

[0060] For example, Figure 2The first Mel spectrum feature Z0 shown in the figure has a coordinate system where the horizontal coordinate of the first Mel spectrum feature Z0 is time and the vertical coordinate is frequency. The frequency range of the first Mel spectrum feature Z0 is [0,80], that is, the first Mel spectrum feature Z0 is an 80-dimensional Mel spectrum feature. If N is 8, 8 different frequency ranges are set (such as Figure 2 The first Mel spectrum feature Z0 is divided into 8 sub-band features of 8 different frequency ranges, namely: sub-band feature Z1 corresponding to the frequency range [0,10], sub-band feature Z2 corresponding to the frequency range (10,20], sub-band feature Z3 corresponding to the frequency range (20,30], sub-band feature Z4 corresponding to the frequency range (30,40], sub-band feature Z5 corresponding to the frequency range (40,50], sub-band feature Z6 corresponding to the frequency range (50,60], sub-band feature Z7 corresponding to the frequency range (60,70], and sub-band feature Z8 corresponding to the frequency range (70,80]. If the first Mel spectrum feature is recorded as [1,80,T], each sub-band feature Z i It can be recorded as [1,10,T], 1≤i≤8, where 1 represents 1 part, 10 represents the frequency dimension, and T represents the duration of the first audio. The sub-band features Z1 to Z8 are composed of [8,10,T], where 8 represents 8 parts, 10 represents the frequency dimension, and T represents the duration of the first audio.

[0061] In some embodiments, the frequency range of the first mel-spectrum frequency feature may be randomly divided into N different frequency ranges. In this case, the bandwidths of the N different frequency ranges may be different.

[0062] In the embodiment of the present application, N can be any positive integer value. Generally, the larger the value of N, the higher the quantization coding quality. However, considering that if the value of N is too large, it will bring greater pressure on the computing power and storage capacity of the device, N is usually taken between [3,20]. Preferably, N is 8.

[0063] In step 103, each sub-band feature is quantized and encoded to obtain a quantized encoding value of each sub-band feature.

[0064] In the embodiment of the present application, each sub-band feature is quantized and encoded to convert the sub-band feature from a frequency domain feature into a numerical value, thereby obtaining a quantized encoding value of each sub-band feature.

[0065] It can be understood that, for example, the duration of the first audio is T, each sub-band feature has a set of frequency domain features at each time point within the duration of T of the first audio, and a set of frequency domain features at each time point corresponds to a set of quantized coding values. Therefore, taking a sub-band feature as an example, the quantized coding value of this sub-band feature is actually a set of quantized coding values.

[0066] Exemplarily, N=8, and 8 sub-band features are quantized and encoded to obtain 8 groups of quantized encoding values ​​of the 8 sub-band features respectively.

[0067] In an embodiment of the present application, the quantized coding value of each sub-band feature includes various numerical values ​​obtained by converting each sub-band feature from the frequency domain feature. Each numerical value is usually a value within a certain value range and retains one decimal place. For example, the value range is [-1.0, 1.0], and the value of the quantized coding value can be -1.0, -0.5, 0.5 or 1.0, etc.

[0068] In some embodiments, in order to improve the accuracy of the quantization encoding result, the above step 103 may include the following steps: step 1031;

[0069] In step 1031, each sub-band feature is quantized and encoded through an encoder to obtain a quantized encoding value of each sub-band feature; wherein the encoder includes M high-fidelity generative adversarial network residual modules, and M is an integer greater than 1.

[0070] For example, when M is 6, the hidden dimension of each high-fidelity generative adversarial network residual module is 256 and the residual channels are 1024.

[0071] In an embodiment of the present application, the high-fidelity generative adversarial network residual module alternately uses holey convolution and ordinary convolution to increase the receptive field, and the residual block captures patterns of different lengths in parallel and returns the sum of the outputs of multiple residual blocks. Each residual block has a different kernel size and expansion rate, thereby forming various receptive field patterns. Therefore, it is possible to improve the encoding processing speed while ensuring the accuracy of the encoding results.

[0072] In some embodiments, the following steps can be used to pre-generate an encoder: construct an initial backbone network; wherein the initial backbone network includes a coding structure network and a decoding structure network, and the coding structure network includes M high-fidelity generative adversarial network residual modules; obtain a data set; wherein the data set includes multiple sample audios; extract the second Mel spectrum feature of each sample audio; input the second Mel spectrum feature into the initial backbone network for processing, and output the predicted audio; extract the third Mel spectrum feature of the predicted audio, calculate the cross entropy loss value between the second Mel spectrum feature and the third Mel spectrum feature, and update the network parameters of the initial backbone network according to the cross entropy loss value until the model converges to obtain the target model; determine the coding structure network in the target model as the encoder.

[0073] In the embodiment of the present application, in order to ensure the accuracy of the training results, the data set may contain a large amount of sample audio.

[0074] In the embodiment of the present application, the network parameters of the initial backbone network can be updated by back propagation.

[0075] In the embodiment of the present application, the encoder is trained with a large amount of sample audio data, and can learn the encoding prior knowledge of the Mel spectrum features of a large amount of sample audio data, so that it can achieve accurate quantization encoding of each sub-band feature.

[0076] In some embodiments, in order to reduce the amount of calculation and increase the quantization coding speed, the above step 1031 may include the following steps: step 10311 and step 10312;

[0077] In step 10311, feature dimension reduction processing is performed on each sub-band feature through the feature encoding network of the encoder to obtain the reduced dimension feature of each sub-band feature.

[0078] For example, the feature encoding network includes 6 high-fidelity generative adversarial network residual modules with a hidden dimension of 256 and a residual channel of 1024, and the output is a reduced-dimensional feature with a size of [1,3,T]. If N is 8, the first Mel-spectrum feature is recorded as [1,80,T], and each sub-band feature Z i Let it be [1,10,T], 1≤i≤8, then Z i The reduced-dimensional features obtained after the dimensionality reduction processing of the feature encoding network of the encoder are recorded as [1,3,T], where 1 represents 1 copy, 3 represents the frequency dimension, and T represents the duration of the first audio. By comparing [1,10,T] with [1,3,T], it can be seen that the frequency dimension is reduced from 10 to 3.

[0079] In the embodiment of the present application, since the dimensionality reduction feature of each sub-band feature has much less data than the sub-band feature, the amount of calculation can be reduced and the quantization coding speed can be increased during quantization coding.

[0080] In step 10312, the dimension reduction features of each sub-band feature are quantized and encoded through the quantization coding network of the encoder to obtain the quantization coding value of each sub-band feature.

[0081] In some embodiments, in order to eliminate the dimensional differences between different features and make the different features comparable, thereby improving the performance and accuracy of quantization coding, the above step 10312 may include the following steps: step 103121 and step 103122;

[0082] In step 103121, each dimensionality reduction feature is normalized through the quantization coding network of the encoder to obtain the normalized features of the dimensionality reduction features of each sub-band feature.

[0083] In the embodiment of the present application, the dimension reduction features of each sub-band feature are normalized to eliminate the dimensional differences between different features so that different features are comparable, thereby obtaining the normalized features of the dimension reduction features of each sub-band feature.

[0084] In step 103122, based on the hyperbolic tangent function, the preset offset and the preset translation, the value of the normalized feature of the reduced dimension feature of each sub-band feature is constrained to a preset numerical range to obtain the quantized coding value of each sub-band feature.

[0085] In the embodiment of the present application, when the normalized features of the dimension reduction features of each sub-band feature are quantized and encoded, some offsets and translations are first added to the normalized features by using a hyperbolic tangent function, and then rounded to retain one decimal place. Specifically, the following formulas (1) and (2) are first used to add some offsets and translations to the normalized features:

[0086] x1=(x0+shift)·tanh()*half_1-offset (1)

[0087] half_1=(_levels-1)*(1+eps) / 2 (2)

[0088] Among them, x1 is the value after adding the offset and shift, x0 is the normalized feature, half_l is the half range of the transformed tensor value determined by eps and _levels, eps is a small positive number, usually indicating tolerance; _levels represents the number of levels that can be learned, that is, the codebook level. For example, when the frequency dimension of the normalized feature is 3, the value of _levels is [5,5,5]; shift is the preset shift, and offset is the preset offset, which is used to balance the distribution of values ​​in the half range. This method uses the hyperbolic tangent (tanh) function to transform x0 plus the shift shift to the interval from -half_l to half_l, and then subtracts the offset offset from the result to complete the value restriction. This transformation has the characteristics of stabilizing the gradient and can increase the robustness of learning. Finally, x1 is rounded to obtain a value with one decimal place, which is the quantized code value. For example, the normalized feature is recorded as [1,3,T], and the range of the quantized code value is limited to [-1,1].

[0089] In an embodiment of the present application, the quantization coding network may include multiple codebooks and codebook levels. For example, when N is 8, the quantization coding network may include 8 codebooks and codebook levels [5,5,5], each codebook has 125*3 codes, and each code has 5 dimensions.

[0090] It can be seen that in the embodiment of the present application, by normalizing the dimensionality reduction features of each sub-band feature and performing quantization coding on the features obtained by the normalization, the dimensional differences between different features can be eliminated, making different features comparable, thereby improving the performance and accuracy of quantization coding.

[0091] In step 104, the index value of each sub-band feature is determined according to a preset quantization code table and the quantization coding value of each sub-band feature; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of the N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

[0092] In the embodiment of the present application, considering that the quantized coding value is usually a decimal with a long code length, the data amount is relatively large when it is directly used as training data, which is not conducive to model training. Therefore, the quantized coding value of each sub-band feature is converted into an index value with a shorter code length. Using the index value as training data can reduce the data amount and improve the performance and speed of model training.

[0093] In the embodiment of the present application, an adaptive quantization code table can be pre-constructed according to the dimension of the quantization code value of the sub-band feature. For example, the duration of the first audio is T, and the quantization code value of each sub-band feature at each time point within the duration of T of the first audio is 3-dimensional, then the size of the quantization code table is [125,3], where 125 is the total number of index values ​​in the quantization code table, 3 is the dimension of the quantization code value, and the value range of the quantization code value in the quantization code table is [-1,1], and the interval of the value is 0.5, that is, the possible values ​​at each position are -1, -0.5, 0, 0.5, 1, as shown in Table 1 below.

[0094] Index value Quantized Coded Value 0 [-1.0000，-1.0000，-1.0000] 1 [-0.5000，-1.0000，-1.0000] 2 [0.0000，-1.0000，-1.0000] 3 [0.5000，-1.0000，-1.0000] ...... ...... 123 [0.5000，1.0000，1.0000] 124 [1.0000，1.0000，1.0000]

[0095] Table 1

[0096] Exemplarily, the duration of the first audio is T. Taking a sub-band feature as an example, the quantized coding value of the sub-band feature at a time point within the duration of T of the first audio is [0.5000, 1.0000, 1.0000]. By looking up Table 1, the corresponding index value is 123.

[0097] In an embodiment of the present application, for the quantized coding value of each time point in each group of quantized coding values ​​of N sub-band features, by looking up a table, it is possible to obtain the index value of each group of N sub-band features, and by merging the index value of each group of N sub-band features, it is possible to obtain the complete index value of the first audio, and the complete index value is used for training the Wensheng speech model.

[0098] In some embodiments, decoding may be performed based on the complete index value of the first audio to reconstruct the first audio, wherein the decoding process is the inverse process of the above encoding process.

[0099] Exemplarily, the complete index value of size [8, 1, T] is first split into 8 groups of index values ​​of [1, T]; then, the 8 groups of index values ​​of [1, T] are used to look up the preset quantization code table to obtain the quantization code value after the look-up, which is [1, T, 3]; then, the decoder is used to obtain an output of 8 sub-band features of size [1, T, 10]; then, these 8 sub-band features are spliced ​​together to obtain an output matrix of size [8, T, 10], and the restored Mel spectrum feature of size [1, 80, T] can be obtained; finally, the final first audio is obtained by upsampling and vocoder decoding.

[0100] It can be seen from the above embodiments that in this embodiment, the first Mel-spectrogram feature of the first audio is extracted; the first Mel-spectrogram feature is divided into N sub-bands of different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other; each sub-band feature is quantized and encoded to obtain a quantized coding value of each sub-band feature; and the index value of each sub-band feature is determined according to a preset quantization code table and the quantization coding value of each sub-band feature; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of the N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

[0101] It can be seen that in the embodiment of the present application, since the Mel spectrum feature of the audio is more in line with the auditory characteristics of the human ear and has a good characterization ability for the audio signal, the restoration effect of the audio based on the Mel spectrum feature is better. Therefore, the Mel spectrum feature of the audio is divided into more fine-grained sub-band features based on the frequency range, and each sub-band feature is quantized and encoded to obtain a quantized coding value, and the quantized coding value with a longer code length and a decimal type is converted into an index value with a shorter code length and an integer type. On the one hand, the quality of the audio quantization coding result can be improved, thereby reducing the loss between the reconstructed audio and the original audio of the audio quantization coding result; on the other hand, the training data composed of the index values ​​of the N sub-band features has a relatively small data volume and an integer type, which is easier to process. Therefore, when the Wensheng speech model is trained based on the training data, the performance and speed of the model training can be improved.

[0102] In some other embodiments provided in this application, using Figure 1 The quantization encoding method of steps 101 to 104 in the illustrated embodiment can obtain multiple groups of training data after processing multiple different audios. A pre-constructed initial network for literary speech is trained based on each group of training data and the corresponding description text. After multiple iterations of training until the model converges, a literary speech model is obtained.

[0103] In the actual use stage, the user can input a descriptive text into the text-based speech model, and the text-based speech model will output an audio corresponding to the descriptive text.

[0104] For example, if the description text input by the user is "Please generate a real-time news audio read by XXX announcer", the text-based speech model will output an audio corresponding to the description text.

[0105] It can be seen that in the embodiment of the present application, due to Figure 1The quantization coding method of steps 101 to 104 in the illustrated embodiment can improve the quality of the audio quantization coding results. Therefore, based on the audio quantization coding results as training data for the Wensheng speech model, a Wensheng speech model with strong speech generation capability can be trained.

[0106] The audio encoding method provided in the embodiment of the present application may be executed by an audio encoding device. In the embodiment of the present application, an audio encoding device executing the audio encoding method is taken as an example to illustrate the audio encoding device provided in the embodiment of the present application.

[0107] Figure 3 is a structural block diagram of an audio encoding device provided by some embodiments of the present application, such as Figure 3 As shown, the audio encoding device 300 may include: an extraction module 301, a division module 302, an encoding module 303 and a determination module 304;

[0108] The extraction module 301 is used to extract a first Mel-frequency spectrum feature of a first audio;

[0109] The division module 302 is used to divide the first Mel-frequency spectrum feature into N sub-bands with different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other;

[0110] The encoding module 303 is used to perform quantization encoding on each of the sub-band features to obtain a quantization encoding value of each of the sub-band features;

[0111] The determination module 304 is used to determine the index value of each sub-band feature according to a preset quantization code table and the quantization coding value of each sub-band feature; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of the N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

[0112] It can be seen from the above embodiments that in this embodiment, the first Mel-spectrogram feature of the first audio is extracted; the first Mel-spectrogram feature is divided into N sub-bands of different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other; each sub-band feature is quantized and encoded to obtain a quantized coding value of each sub-band feature; and the index value of each sub-band feature is determined according to a preset quantization code table and the quantization coding value of each sub-band feature; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of the N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

[0113] It can be seen that in the embodiment of the present application, since the Mel spectrum feature of the audio is more in line with the auditory characteristics of the human ear and has a good characterization ability for the audio signal, the restoration effect of the audio based on the Mel spectrum feature is better. Therefore, the Mel spectrum feature of the audio is divided into more fine-grained sub-band features based on the frequency range, and each sub-band feature is quantized and encoded to obtain a quantized coding value, and the quantized coding value with a longer code length and a decimal type is converted into an index value with a shorter code length and an integer type. On the one hand, the quality of the audio quantization coding result can be improved, thereby reducing the loss between the reconstructed audio and the original audio of the audio quantization coding result; on the other hand, the training data composed of the index values ​​of the N sub-band features has a relatively small data volume and an integer type, which is easier to process. Therefore, when the Wensheng speech model is trained based on the training data, the performance and speed of the model training can be improved.

[0114] Optionally, as an embodiment, the encoding module 303 may include:

[0115] The encoding submodule is used to quantize and encode each of the sub-band features through an encoder to obtain a quantized encoding value of each of the sub-band features; wherein the encoder includes M high-fidelity generative adversarial network residual modules, and M is an integer greater than 1.

[0116] Optionally, as an embodiment, the encoding submodule may include:

[0117] A dimensionality reduction processing unit, used to perform feature dimensionality reduction processing on each of the sub-band features through a feature encoding network of an encoder to obtain a reduced dimensionality feature of each of the sub-band features;

[0118] The quantization coding unit is used to perform quantization coding on the reduced-dimensional features of each of the sub-band features through the quantization coding network of the encoder to obtain a quantization coding value of each of the sub-band features.

[0119] Optionally, as an embodiment, the quantization coding unit may include:

[0120] A normalization processing subunit, used for performing normalization processing on the reduced dimension features of each of the sub-band features through a quantization coding network of the encoder to obtain normalized features of the reduced dimension features of each of the sub-band features;

[0121] The calculation subunit is used to constrain the value of the normalized feature of the dimensionality reduction feature of each sub-band feature within a preset numerical range based on the hyperbolic tangent function, the preset offset and the preset translation, so as to obtain the quantized coding value of each sub-band feature.

[0122] Optionally, as an embodiment, the audio encoding device 300 may further include: a generating module;

[0123] The generating module may include:

[0124] A construction submodule is used to construct an initial backbone network; wherein the initial backbone network includes a coding structure network and a decoding structure network, and the coding structure network includes the M high-fidelity generative adversarial network residual modules;

[0125] The acquisition submodule is used to acquire a data set; wherein the data set includes a plurality of sample audios; and extract the second Mel frequency spectrum feature of each of the sample audios;

[0126] A training submodule is used to input the second Mel spectrum feature into the initial backbone network for processing and output a predicted audio; extract a third Mel spectrum feature of the predicted audio, calculate a cross entropy loss value between the second Mel spectrum feature and the third Mel spectrum feature, and update the network parameters of the initial backbone network according to the cross entropy loss value until the model converges to obtain a target model;

[0127] A determination submodule is used to determine the encoding structure network in the target model as the encoder.

[0128] The device in the embodiment of the present application can be a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0129] The device of the embodiment of the present application may be a device having an action system. The action system may be an Android action system, an IOS action system, or other possible action systems, which are not specifically limited in the embodiment of the present application.

[0130] The device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0131] Alternatively, if Figure 4 As shown, an embodiment of the present application also provides an electronic device 400, including a processor 401 and a memory 402, a program or instruction stored in the memory 402 and executable on the processor 401, and when the program or instruction is executed by the processor 401, the various steps of the above-mentioned audio encoding method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0132] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0133] Figure 5 is a schematic diagram of the hardware structure of an electronic device provided by some embodiments of the present application, such as Figure 5 As shown, the electronic device 500 includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510 and other components.

[0134] Those skilled in the art will appreciate that the electronic device 500 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0135] Processor 510 is used to extract a first Mel-spectrogram feature of a first audio; divide the first Mel-spectrogram feature into N sub-bands of different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other; perform quantization encoding on each of the sub-band features to obtain a quantization encoding value of each of the sub-band features; determine an index value of each of the sub-band features according to a preset quantization code table and the quantization encoding value of each of the sub-band features; wherein the quantization code table contains a mapping relationship between quantization encoding values ​​and index values, the index value is an integer, and the index values ​​of the N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train a Wensheng speech model.

[0136] It can be seen that in the embodiment of the present application, since the Mel spectrum feature of the audio is more in line with the auditory characteristics of the human ear and has a good characterization ability for the audio signal, the restoration effect of the audio based on the Mel spectrum feature is better. Therefore, the Mel spectrum feature of the audio is divided into more fine-grained sub-band features based on the frequency range, and each sub-band feature is quantized and encoded to obtain a quantized coding value, and the quantized coding value with a longer code length and a decimal type is converted into an index value with a shorter code length and an integer type. On the one hand, the quality of the audio quantization coding result can be improved, thereby reducing the loss between the reconstructed audio and the original audio of the audio quantization coding result; on the other hand, the training data composed of the index values ​​of the N sub-band features has a relatively small data volume and an integer type, which is easier to process. Therefore, when the Wensheng speech model is trained based on the training data, the performance and speed of the model training can be improved.

[0137] Optionally, as an embodiment, the processor 510 is specifically used to perform quantization encoding on each of the sub-band features through an encoder to obtain a quantization encoding value of each of the sub-band features; wherein the encoder includes M high-fidelity generative adversarial network residual modules, and M is an integer greater than 1.

[0138] Optionally, as an embodiment, the processor 510 is specifically used to perform feature dimensionality reduction processing on each of the sub-band features through the feature encoding network of the encoder to obtain the reduced dimensionality features of each of the sub-band features; and to perform quantization encoding on the reduced dimensionality features of each of the sub-band features through the quantization encoding network of the encoder to obtain the quantization encoding value of each of the sub-band features.

[0139] Optionally, as an embodiment, the processor 510 is specifically used to normalize the reduced dimension features of each of the sub-band features through the quantization coding network of the encoder to obtain the normalized features of the reduced dimension features of each of the sub-band features; based on the hyperbolic tangent function, a preset offset and a preset translation, constrain the value of the normalized feature of the reduced dimension features of each of the sub-band features to a preset numerical range to obtain the quantized coding value of each of the sub-band features.

[0140] Optionally, as an embodiment, the processor 510 is also used to construct an initial backbone network; wherein the initial backbone network includes a coding structure network and a decoding structure network, and the coding structure network includes the M high-fidelity generative adversarial network residual modules; obtain a data set; wherein the data set includes multiple sample audios; extract the second Mel spectrum feature of each of the sample audios; input the second Mel spectrum feature into the initial backbone network for processing, and output predicted audio; extract the third Mel spectrum feature of the predicted audio, calculate the cross entropy loss value between the second Mel spectrum feature and the third Mel spectrum feature, and update the network parameters of the initial backbone network according to the cross entropy loss value until the model converges to obtain the target model; determine the coding structure network in the target model as the encoder.

[0141] It should be understood that in the embodiment of the present application, the input unit 504 may include a graphics processor (Graphics Processing Unit, GPU) 5041 and a microphone 5042, and the graphics processor 5041 processes the image data of the static picture or video image obtained by the image capture device (such as a camera) in the video image capture mode or the image capture mode. The display unit 506 may include a display panel 5061, and the display panel 5061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 507 includes at least one of a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include two parts: a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and an action rod, which will not be repeated here. The memory 509 can be used to store software programs and various data, including but not limited to applications and action systems. The processor 510 may integrate an application processor and a modem processor, wherein the application processor mainly processes the action system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 510.

[0142] The memory 509 can be used to store software programs and various data. The memory 509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 509 may include a volatile memory or a non-volatile memory, or the memory 509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0143] The processor 510 may include one or more processing units; optionally, the processor 510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 510.

[0144] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, each process of the above-mentioned audio encoding method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the processor is the processor in the electronic device in the above-mentioned embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a disk or an optical disk, etc.

[0145] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio encoding method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0146] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0147] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned audio encoding method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0148] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0149] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0150] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An audio encoding method, characterized in that: The method comprises: Extracting a first Mel-spectrogram feature of the first audio; Dividing the first Mel-spectrum frequency feature into N sub-bands with different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other; Performing quantization encoding on each of the sub-band features to obtain a quantization encoding value of each of the sub-band features; According to a preset quantization code table and the quantization coding value of each sub-band feature, an index value of each sub-band feature is determined; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of the N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

2. The method according to claim 1, characterized in that The step of quantizing and encoding each of the sub-band features to obtain a quantized encoding value of each of the sub-band features includes: Through the encoder, each of the sub-band features is quantized and encoded to obtain a quantized encoding value of each of the sub-band features; wherein the encoder includes M high-fidelity generative adversarial network residual modules, and M is an integer greater than 1.

3. The method according to claim 2, characterized in that The step of performing quantization encoding on each of the sub-band features through an encoder to obtain a quantization encoding value of each of the sub-band features includes: Performing feature dimensionality reduction processing on each of the sub-band features through a feature encoding network of an encoder to obtain a reduced dimensionality feature of each of the sub-band features; The dimension reduction features of each of the sub-band features are quantized and encoded through the quantization encoding network of the encoder to obtain the quantization encoding value of each of the sub-band features.

4. The method according to claim 3, characterized in that The step of quantizing and encoding the dimension reduction feature of each sub-band feature through the quantization coding network of the encoder to obtain a quantization coding value of each sub-band feature includes: Normalizing the dimension reduction features of each sub-band feature through the quantization coding network of the encoder to obtain normalized features of the dimension reduction features of each sub-band feature; Based on the hyperbolic tangent function, the preset offset and the preset translation, the value of the normalized feature of the dimension reduction feature of each sub-band feature is constrained to a preset numerical range to obtain a quantized coding value of each sub-band feature.

5. The method according to any one of claims 2 to 4, characterized in that: Before the step of quantizing and encoding each of the sub-band features by an encoder to obtain a quantized coded value of each of the sub-band features, the method further includes: Constructing an initial backbone network; wherein the initial backbone network includes a coding structure network and a decoding structure network, and the coding structure network includes the M high-fidelity generative adversarial network residual modules; Acquire a data set; wherein the data set includes a plurality of sample audios; Extracting a second Mel-spectrogram feature of each of the sample audios; Inputting the second Mel spectrum feature into the initial backbone network for processing and outputting predicted audio; extracting the third Mel spectrum feature of the predicted audio, calculating the cross entropy loss value between the second Mel spectrum feature and the third Mel spectrum feature, and updating the network parameters of the initial backbone network according to the cross entropy loss value until the model converges to obtain the target model; The encoding structure network in the target model is determined as the encoder.

6. An audio coding generation device, characterized in that: The device comprises: An extraction module, used for extracting a first Mel frequency spectrum feature of a first audio; A division module, used for dividing the first Mel frequency spectrum feature into N sub-bands with different frequency ranges to obtain N sub-band features; wherein the N different frequency ranges do not overlap with each other; An encoding module, used for performing quantization encoding on each of the sub-band features to obtain a quantization encoding value of each of the sub-band features; A determination module is used to determine the index value of each sub-band feature according to a preset quantization code table and the quantization coding value of each sub-band feature; wherein the quantization code table contains a mapping relationship between the quantization coding value and the index value, the index value is an integer, and the index values ​​of N sub-band features constitute a set of training data corresponding to the first audio, and the training data is used to train the Wensheng speech model.

7. The device according to claim 6, characterized in that The encoding module comprises: The encoding submodule is used to quantize and encode each of the sub-band features through an encoder to obtain a quantized encoding value of each of the sub-band features; wherein the encoder includes M high-fidelity generative adversarial network residual modules, and M is an integer greater than 1.

8. The device according to claim 7, characterized in that The encoding submodule includes: A dimensionality reduction processing unit, used to perform feature dimensionality reduction processing on each of the sub-band features through a feature encoding network of an encoder to obtain a reduced dimensionality feature of each of the sub-band features; The quantization coding unit is used to perform quantization coding on the reduced-dimensional features of each of the sub-band features through the quantization coding network of the encoder to obtain a quantization coding value of each of the sub-band features.

9. The device according to claim 8, characterized in that The quantization coding unit comprises: A normalization processing subunit, used for performing normalization processing on the reduced dimension features of each of the sub-band features through a quantization coding network of the encoder to obtain normalized features of the reduced dimension features of each of the sub-band features; The calculation subunit is used to constrain the value of the normalized feature of the dimensionality reduction feature of each sub-band feature within a preset numerical range based on the hyperbolic tangent function, the preset offset and the preset translation, so as to obtain the quantized coding value of each sub-band feature.

10. The device according to any one of claims 7 to 9, characterized in that: The device also includes: a generating module; The generating module comprises: A construction submodule is used to construct an initial backbone network; wherein the initial backbone network includes a coding structure network and a decoding structure network, and the coding structure network includes the M high-fidelity generative adversarial network residual modules; The acquisition submodule is used to acquire a data set; wherein the data set includes a plurality of sample audios; and extract the second Mel frequency spectrum feature of each of the sample audios; A training submodule is used to input the second Mel spectrum feature into the initial backbone network for processing and output a predicted audio; extract a third Mel spectrum feature of the predicted audio, calculate a cross entropy loss value between the second Mel spectrum feature and the third Mel spectrum feature, and update the network parameters of the initial backbone network according to the cross entropy loss value until the model converges to obtain a target model; A determination submodule is used to determine the encoding structure network in the target model as the encoder.

11. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the audio encoding method as described in any one of claims 1 to 5 are implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the audio encoding method according to any one of claims 1 to 5 are implemented.

Images