Signal processing method and related apparatus

Abstract

The present application provides a signal processing method and a related apparatus. The method comprises: acquiring N quantized audio features of raw non-speech audio, wherein N is a positive integer greater than or equal to 1; determining a first predicted silence audio feature based on the N quantized audio features and reference noise; and determining, based on the first predicted silence audio feature and a first neural network, first non-speech audio corresponding to the raw non-speech audio.

Description

Method of processing a signal and related apparatus

[0001] This application claims priority to the Russian patent application No. 2024129150, filed on October 01, 2024, with the Russian Federal Service for Intellectual Property, and having the title “Method of processing a signal and related apparatus”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of signal processing, and more particularly, to a method of processing a signal and related apparatus. BACKGROUND

[0003] In voice communication, although the speech content of the speaker is the main information, if the environmental background noise is not transmitted, a completely silent state will occur in the communication process, which may make the user mistakenly think that the communication is interrupted or has a fault. Transmitting certain environmental background noise can make the user still feel the continuity of the communication during the non-speech period. In addition, the environmental background noise can also provide some information about the environment of the sender to the receiver. For example, if there is the sound of a vehicle driving in the background, it can be roughly judged that the other party is probably on the street outdoors. Therefore, transmitting background noise in voice communication helps to improve the user experience.

[0004] However, if the environmental background noise is normally encoded and transmitted to the receiving end through the link, unnecessary communication bandwidth will be occupied. If the environmental background noise can be transmitted discontinuously, and the non-speech period audio is approximately recovered at the receiving end using discontinuous frames, the communication link pressure will be greatly reduced. Therefore, how to transmit environmental background noise is a problem that the industry is concerned about. SUMMARY

[0005] The present application provides a method of processing a signal and related apparatus, which can generate more natural noise.

[0006] The first aspect of the present application provides a method of processing a signal. The method comprises: obtaining N quantized audio features of an original non-speech audio, N being a positive integer greater than or equal to 1; determining a first predicted silence audio feature according to the N quantized audio features and a reference noise; determining a first non-speech audio corresponding to the original non-speech audio according to the first predicted silence audio feature and a first neural network.

[0007] The technical solution of the present application proposes a noise generation scheme based on a neural network (i.e., the first non-speech audio). Compared with the traditional scheme, the noise generated by the neural network is more natural. In addition, the above technical solution can predict the subsequent silence audio feature based on the obtained quantized audio features, and noise is introduced in the process of predicting the silence audio feature, so that the noise generated by the neural network is more natural.

[0008] With reference to the first aspect, in a possible implementation of the first aspect, the N quantized audio features, the reference noise and the first predicted silence audio feature satisfy the following relationship: wherein SD N+1 represents the first predicted silence audio feature, SD i represents the i-th quantized audio feature in the N quantized audio features, a i represents the i-th preset parameter in the N preset parameters, noise represents the reference noise, β represents a preset parameter corresponding to the reference noise, and i = 1, …, N.

[0009] With reference to the first aspect, in a possible implementation of the first aspect, a i and β satisfy the following relationship: wherein Pst_V is a number greater than or equal to 0.9 and less than or equal to 1.1.

[0010] With reference to the first aspect, in a possible implementation of the first aspect, the reference noise is random noise.

[0011] In the case where the introduced noise is random noise, the non-speech audio generated by the neural network has better randomness. In this way, the noise generated by the neural network is more natural.

[0012] With reference to the first aspect, in a possible implementation of the first aspect, N is a positive integer less than or equal to 3.

[0013] The greater the value of N is, the more silence information needs to be transmitted through the link, and the greater the communication overhead is. Therefore, N less than or equal to 3 can save the communication overhead.

[0014] With reference to the first aspect, in a possible implementation of the first aspect, the method further includes:

[0015] determining a second predicted silence audio feature according to the N reference audio features and the reference noise, wherein in the case where N is equal to 1, the N reference audio features are the first predicted silence audio feature, and in the case where N is a positive integer greater than 1, the N reference audio features include the first predicted silence audio feature and the last N-1 quantized audio features in the N quantized audio features; and determining a second non-speech audio according to the second predicted silence audio feature and the first neural network.

[0016] In a possible implementation manner of the first aspect, the obtaining the N quantized audio features comprises: receiving a code stream through a low-code-rate audio communication channel, the low-code-rate audio communication channel at least comprising a satellite communication channel, the code stream containing the N quantized audio features extracted from the original non-speech audio, and the N quantized audio features being quantized and coded; and decoding and dequantizing the code stream to obtain the N quantized audio features, the N quantized audio features corresponding to the N audio features one by one.

[0017] The low-code-rate audio communication channel has limited communication capability, and therefore it is desirable to further compress the non-speech audio (i.e., noise) with a relatively small amount of information contained. The technical solution described above can reduce the non-speech information to be transmitted by predicting the silent audio features. In this way, the occupation of the low-code-rate audio communication channel can be reduced.

[0018] In a possible implementation manner of the first aspect, the N audio features are mel-frequency spectrograms of the original non-speech audio or mel-frequency cepstral coefficients of the original non-speech audio, or the N audio features are features extracted by a second neural network from the original non-speech audio.

[0019] Optionally, the first neural network and the second neural network can be jointly trained.

[0020] The audio features in the technical solution described above are based on mel-frequency spectrum or neural network extraction. The mel-frequency spectrum or neural network can extract audio features that are better perceived by humans. Therefore, the predicted audio features predicted based on the quantized audio features are also better perceived by humans. In this way, the effect and naturalness of the noise generated by the neural network can be more in line with the hearing habits of the human ear.

[0021] In a possible implementation manner of the first aspect, the method further comprises: determining N non-speech audios according to the N quantized audio features and the first neural network, wherein the reconstructed audio of the original non-speech audio comprises the N non-speech audios and the first non-speech audio.

[0022] In a second aspect, an electronic device is provided, which comprises units for implementing the first aspect or any possible implementation manner of the first aspect.

[0023] In a third aspect, an electronic device is provided, which comprises a processor configured to couple with a memory, read and execute instructions and / or program codes in the memory, to perform the first aspect or any possible implementation manner of the first aspect.

[0024] In a fourth aspect, an embodiment of the present application provides a chip system, which comprises a logic circuit configured to be coupled with an input / output interface through which data is transmitted to perform the first aspect or any possible implementation manner of the first aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium storing program codes, which, when executed on an electronic device, cause the electronic device to perform the first aspect or any possible implementation manner of the first aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a computer program product, which comprises computer program codes, which, when executed on an electronic device, cause the electronic device to perform the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0027] FIG. 1 is a schematic diagram of a voice communication encoding process;

[0028] FIG. 2 is a schematic diagram of a satellite communication system;

[0029] FIG. 3 is a schematic diagram of a process for processing a signal according to an embodiment of the present application;

[0030] FIG. 4 is a schematic flowchart of a method for processing a signal according to an embodiment of the present application;

[0031] FIG. 5 is a schematic diagram of a process for extracting audio features;

[0032] FIG. 6 is a schematic diagram of a system architecture according to an embodiment of the present application;

[0033] FIG. 7 is a schematic diagram of a convolutional neural network;

[0034] FIG. 8 is a chip hardware structure according to an embodiment of the present application;

[0035] FIG. 9 is a schematic flowchart of another method for processing a signal according to an embodiment of the present application;

[0036] FIG. 10 is a schematic diagram of another system architecture according to an embodiment of the present application;

[0037] FIG. 11 is a schematic structural block diagram of an electronic device according to an embodiment of the present application;

[0038] FIG. 12 is a schematic structural block diagram of another electronic device according to an embodiment of the present application;

[0039] FIG. 13 is a schematic structural block diagram of still another electronic device according to an embodiment of the present application;

[0040] FIG. 14 is a schematic diagram of a chip system according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the present application will be described below with reference to the accompanying drawings.

[0042] FIG. 1 is a schematic diagram of a speech communication encoding process.

[0043] As shown in FIG. 1, a voice activity detection (VAD) module can divide an audio signal into non-silence segments and silence segments. For example, the non-silence segments and the silence segments can be determined based on the energy, the zero-crossing rate, the spectral features, etc. of the audio signal. For the non-silence segments, a continuous encoding manner can be used to encode the non-silence segments, and a non-silence segment signal (which can also be referred to as a non-silence bitstream) is obtained. For the silence segments, a discontinuous encoding manner can be used to encode the silence segments, and a silence segment signal (which can also be referred to as a silence bitstream) is obtained.

[0044] The non-silence segment signal and the silence segment signal are transmitted to a receiving end through a communication link. The receiving end continuously decodes the received non-silence segment signal to obtain an audio signal of the non-silence segments. A comfort noise generation module obtains the received silence segment signal, and restores the audio signal of the silence segments by using a comfort noise generation technique. Then, the audio signal of the non-silence segments and the audio signal of the silence segments are combined to obtain a final decoding result.

[0045] The technical solutions provided in the embodiments of the present application can be applied to a low-code-rate audio communication system. As shown in FIG. 2, a satellite communication system is a typical low-code-rate audio communication system.

[0046] A terminal device 1 (for example, a mobile phone) transmits the encoded bitstream to a satellite. The satellite sends the received bitstream to a satellite ground station. The satellite ground station transcodes the bitstream from the satellite, and sends the transcoded bitstream to a terminal device 2 through a network device.

[0047] FIG. 3 is a schematic diagram of a process of processing a signal according to an embodiment of the present application.

[0048] As shown in FIG. 3, a feature extraction module of a sending end can extract audio features of non-speech audio. For ease of description, the non-speech audio obtained by the feature extraction module can be referred to as original non-speech audio. In other words, the feature extraction module can extract audio features of the original non-speech audio. After the audio features of the original non-speech audio are extracted, the sending end can obtain a bitstream by quantizing and encoding the audio features of the original non-speech audio, and then send the bitstream to a receiving end through a transmission link.

[0049] Correspondingly, the receiving end receives the bitstream, decodes and inverse-quantizes the bitstream to obtain the quantized audio features. The non-speech audio generation module performs frame prediction based on the quantized audio features to obtain the predicted silent audio features, and then generates the target non-speech audio according to the predicted silent audio features. This target non-speech audio is the speech audio that the receiving-end user can hear.

[0050] FIG. 4 is a schematic flowchart of a method for processing signals provided by an embodiment of the present application. The method shown in FIG. 4 can be executed by a sending-end device, or can be executed by components (such as a chip, a system on chip (SoC), etc.) in the sending-end device. The sending-end device can be a terminal device capable of communication, such as a mobile phone, a tablet computer, a smart watch, etc. For the sake of description, it is assumed that the method shown in FIG. 4 is executed by the sending-end device, and this method includes steps 401-403.

[0051] 401. The sending-end device extracts the audio features of the original non-speech audio to obtain N audio features.

[0052] 402. The sending-end device quantizes and encodes the N audio features to obtain a bitstream containing the N audio features.

[0053] 403. The sending-end device sends this bitstream. The implementation process of step 401 will be described in detail below with reference to FIGS. 5-8.

[0054] The original non-speech audio is the non-speech audio for which audio features need to be extracted. The original non-speech audio can be a part of the non-speech audio in the silent segment output by the VAD module. Taking FIG. 5 as an example, FIG. 5 is a schematic diagram of the process of extracting audio features. Assume that the duration T total =t5 - t1 of the silent segment output by the VAD module, then the original non-speech audio can be the non-speech audio from time t1 to time t4. As shown in FIG. 5, the original non-speech audio can be divided into multiple original non-speech frames. The lengths of different original non-speech frames can be the same or different. However, for the sake of description, it is assumed below that the lengths of the original reference non-speech frames are the same. FIG. 5 shows two original non-speech frames, original non-speech frame 1 and original non-speech frame 2. The start time of original non-speech frame 1 is t1 and the end time is t3; the start time of original non-speech frame 2 is t2 and the end time is t4, where t3 - t1 = t4 - t2. As shown in FIG. 5, t1 < t2 < t3 < t4. In other words, there is an overlap between original non-speech frame 1 and original non-speech frame 2, that is, there is an overlap between adjacent original non-speech frames as shown in FIG. 5.

[0055] It can be understood that FIG. 5 is only an example of two original non-speech frames, and is not a limitation on the original non-speech frames. Embodiments of the present application do not limit whether there is an overlap between the two adjacent original non-speech frames. In other words, in some embodiments, there is an overlap between the adjacent original non-speech frames. In other embodiments, there can also be no overlap between the adjacent original non-speech frames. In addition, embodiments of the present application do not limit the number of silence segments output by the VAD module. For example, in some embodiments, the VAD module can only output one silence segment, while in other embodiments, the VAD module can output two or more silence segments.

[0056] The sending end device can extract audio features of the N original non-speech frames respectively to obtain N audio features. The N audio features correspond one-to-one to the N original non-speech frames, and each of the N audio features is an audio feature extracted from a corresponding original non-speech frame.

[0057] Optionally, in some embodiments, the sending end device can extract a mel spectrogram of the original non-speech frame as the audio feature of the original non-speech frame.

[0058] Optionally, in other embodiments, the sending end device can extract mel-frequency cepstral coefficients (MFCC) of the original non-speech frame as the audio feature of the original non-speech frame.

[0059] Optionally, in other embodiments, the sending end device can extract the audio feature of the original non-speech frame by using an artificial intelligence (AI) model. Embodiments of the present application involve multiple neural networks, in order to facilitate differentiation, the neural network used to extract the audio feature can be referred to as a feature extraction model.

[0060] The feature extraction model can be a neural network, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a Transformer model, etc.

[0061] In some embodiments, the input signal of the feature extraction model can be the original non-speech frame. The output of the feature extraction model is the extracted T audio features. The dimension of the audio feature extracted by the feature extraction model can be represented by the letter D, and the number of extracted features can be represented by the letter T, T is a positive integer greater than or equal to 1. The number of features T can satisfy the following relationship:

[0062] T = signal_length / hop_size, (Equation 1.1)

[0063] wherein signal_length represents the time length of the original non-speech audio, and hop_size represents the time length between the starting time of two adjacent frames. For example, as shown in FIG. 5, hop_size is the time length between t1 and t2.

[0064] In some other embodiments, the frame operation can also be implemented by the feature extraction model. In this case, the input signal of the feature extraction model can be the original non-speech audio. The feature extraction model can frame the input original non-speech audio to obtain T original non-speech frames, and then determine the audio features of each original non-speech frame.

[0065] For ease of description, it is assumed in the following embodiments that the input of the feature extraction model is the original non-speech frame.

[0066] The following describes how to train the feature extraction model by taking a convolutional neural network as an example.

[0067] The convolutional neural network is a deep convolutional neural network with a convolutional structure. The convolutional neural network has better feature learning ability and can automatically extract useful information from non-speech frames. The core feature of the convolutional neural network is the convolution operation, which uses filters (also known as convolution kernels) and pooling layers to extract features of non-speech frames. The convolution operation can effectively reduce the number of weights and reduce the amount of calculation. The pooling layer can reduce the amount of calculation and improve the robustness of the model without changing the dimension of the feature map.

[0068] The convolution kernel can be initialized in the form of a random size vector or matrix, and the convolution kernel can obtain reasonable weights through learning in the training process of the convolutional neural network. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0069] As an unsupervised learning algorithm, the autoencoder (AE) is widely used in data dimensionality reduction, feature extraction, and other aspects. By combining the convolutional neural network and the autoencoder, the feature extraction model can be obtained.

[0070] The CNN autoencoder is composed of an encoder and a decoder. The encoder is responsible for feature extraction and compression of the input data through convolutional layers, pooling layers, and other structures to generate a low-dimensional latent representation. The decoder is responsible for reconstructing the original input data through convolutional layers and up-sampling layers.

[0071] In the encoding stage, the input data is gradually extracted high-level feature representation through the alternating action of convolutional layers and pooling layers, and the high-level feature representation is compressed into a low-dimensional encoding representation. In this process, the redundant information in the data is removed, and the important features are retained.

[0072] In the decoding stage, the encoding representation gradually recovers the spatial dimension and detail information of the data through the alternating action of convolutional layers and up-sampling layers, and finally reconstructs the data close to the original input. In this process, the decoder needs to learn how to reconstruct the original data according to the encoding representation.

[0073] In the process of training the CNN autoencoder, because we want the output of the convolutional neural network to be as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the convolutional neural network according to the difference between the two. (Of course, before the first update, there is usually an initialization process, that is, to pre-configure the parameters of each layer in the convolutional neural network), for example, if the network's predicted value is high, adjust the weight vector to make it predict lower, and keep adjusting until the convolutional neural network can predict the target value we really want or a value very close to it. For the CNN autoencoder, the target value is the input training sample of the encoder, and the predicted value is the output value of the decoder, which is determined based on the input training sample. For example, the input training sample of the encoder is x, and the output of the encoder is the feature of the training sample x, denoted as z. The input of the decoder is the feature z, and the output is the predicted value x'. By comparing the difference between the training sample x and the predicted value x', the weight vector of the convolutional neural network can be adjusted.

[0074] In order to achieve the above purpose, it is necessary to define "how to compare the difference between the predicted value and the target value" in advance, which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the convolutional neural network becomes a process of trying to minimize this loss.

[0075] The convolutional neural network can adopt a back propagation (BP) algorithm to correct the size of the parameters in the convolutional neural network during the training process, so that the error loss between the predicted value output by the convolutional neural network and the target value that is truly desired becomes smaller and smaller. Specifically, forward propagation of the input signal until the output generates an error loss, and the initial parameters in the convolutional neural network are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by the error loss, aiming to obtain optimal parameters of the convolutional neural network, such as a weight vector or a matrix, that is, a convolution kernel of a convolution layer.

[0076] After the training of the CNN autoencoder is completed, the encoder in the trained CNN autoencoder can be used as a feature extraction model.

[0077] The system architecture provided in the embodiments of the present application is described below.

[0078] Referring to FIG. 6, the system architecture 600 provided in the embodiments of the present application is shown. As shown in the system architecture 600, the data collection device 640 is configured to collect training data, and the training data can be stored in the database 630. The training data in the embodiments of the present application includes one or more non-speech frames. The training device 620 can train the target model / rule 601 (601 is the model trained in the training stage as described above, which can be a convolutional neural network for extracting audio features of reference non-speech audio) based on the training data maintained in the database 630. The target model / rule 601 is a feature extraction model for extracting audio features of original non-speech audio, that is, the original non-speech frame or the original non-speech audio is input into the target model / rule 601, and the audio features can be obtained. The target model / rule 601 in the embodiments of the present application can be a trained convolutional neural network, which is obtained by training the initialized convolutional neural network in the embodiments provided in the present application. It should be noted that in actual applications, the training data maintained in the database 630 can not all be collected by the data collection device 640, but can also be received from other devices. In addition, it should be noted that the training device 620 can not completely train the target model / rule 601 based on the training data maintained in the database 630, but can also obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation on the embodiments of the present application. The training data can be noise collected in public scenes, traffic scenes, work scenes, etc.

[0079] The target model / rule 601 trained by the training device 620 can be applied to an execution device 610 shown in FIG. 6, which can be the sender device mentioned above. In FIG. 6, the execution device 610 is configured with an I / O interface 612 for data interaction with external devices. A user can input data to the I / O interface 612, which in the embodiments of the present application can include an audio signal collected by a microphone. The microphone can be a microphone built in the execution device 610, or a microphone of a wired earphone connected to the execution device 610 through a wired interface (for example, a headset interface, a universal serial bus (USB) interface, etc.), or a microphone of a wireless earphone connected to the execution device 610 through a wireless communication technology (for example, Wi-Fi).

[0080] The preprocessing module 613 is configured to perform preprocessing on the input data (such as the audio signal) received by the I / O interface 612. In the embodiments of the present application, the preprocessing module 613 can be configured to implement voice activity detection to determine a silent segment signal (i.e., original non-speech audio), and can also be configured to segment the original non-speech audio to obtain a plurality of original non-speech frames. The preprocessing module 613 can also be configured to implement other preprocessing operations, which are not limited in the present application.

[0081] During the preprocessing of the input data by the execution device 610, or during the processing of the calculation module 611 of the execution device 610, the modules (such as the calculation module 611 and the preprocessing module 613) in the execution device 610 can call the data, code, etc. in the storage module 615 to implement corresponding processing, and can also store the data, instructions, etc. obtained by corresponding processing in the storage module 615.

[0082] After obtaining the audio features of the original non-speech frames based on the target model / rule 601, the calculation module 611 can quantize and encode the audio features of the original non-speech frames to obtain a code stream containing the audio features. The communication module 614 is responsible for sending the code stream.

[0083] It is worth noting that FIG. 6 is only a schematic diagram of a system architecture provided by the embodiments of the present application, and the positional relationship between the devices, components, modules, etc. shown in FIG. 6 does not constitute any limitation. For example, in FIG. 6, the storage module 615 is placed in the execution device 610. In other cases, the storage module 615 can also be an external memory of the execution device 610.

[0084] As mentioned previously, a convolutional neural network is a deep learning architecture with convolutional structure, where deep learning architecture refers to a multi-level learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial convolutional neural network, in which each neuron can respond to non-speech frames in the input.

[0085] As shown in FIG. 7, the convolutional neural network (CNN) 700 can include an input layer 710, a convolutional layer / down-sampling layer 720, a convolutional layer / up-sampling layer 730, and an output layer 740. The convolutional layer / down-sampling layer 720 is configured to implement the function of an encoder, and the convolutional layer / up-sampling layer 730 is configured to implement the function of a decoder. As shown in FIG. 7, the convolutional layer / down-sampling layer 720 can include layers such as 721-726, for example: in one implementation, the 721 layer is a convolutional layer, the 722 layer is a down-sampling layer, the 723 layer is a convolutional layer, the 724 layer is a down-sampling layer, the 725 layer is a convolutional layer, and the 726 layer is a down-sampling layer; in another implementation, the 721 and 722 layers are convolutional layers, the 723 layer is a down-sampling layer, the 724 and 725 layers are convolutional layers, and the 726 layer is a down-sampling layer. That is, the output of a convolutional layer can be used as the input of a subsequent down-sampling layer, or as the input of another convolutional layer for further convolution operation.

[0086] The core idea of a convolutional layer is to map the input features to the output feature space through convolution operation. In a convolutional neural network, the convolution operator (which can be referred to as a convolution kernel) is a learnable parameter, and the parameters of the convolution kernel are adjusted through training to optimize the performance of the model. The convolution kernel can be one-dimensional or multi-dimensional, corresponding to one-dimensional or multi-dimensional input signals. The convolution operator can essentially be a weight vector or a matrix, and the weight values in the weight vector or matrix need to be obtained through a large amount of training in practical applications. The weight vectors or matrices formed by the weight values obtained through training can be used to extract information from the input, so that the convolutional neural network 700 can make correct predictions.

[0087] When the convolutional neural network 700 has multiple convolutional layers, the initial convolutional layer (e.g., 721) tends to extract more general features, which can also be referred to as low-level features, corresponding to high-resolution feature maps. As the depth of the convolutional neural network 700 increases, the convolutional layers further back (e.g., 726) extract more complex features, such as high-level semantic features, corresponding to low-resolution feature maps. The higher the semantic features, the more suitable they are for the problem to be solved.

[0088] Because it is often necessary to reduce the number of training parameters, a periodic down-sampling layer is often needed after the convolution layer. In each layer 721-726 as exemplified in 720 in FIG. 7, it can be a convolution layer followed by a down-sampling layer, or a plurality of convolution layers followed by one or more down-sampling layers. The core idea of the down-sampling layer is to map the input features to a lower resolution space through a down-sampling operation.

[0089] As shown in FIG. 7, the convolution layer / up-sampling layer 730 can include layers as exemplified in 731-736, for example: in one implementation, 731 is an up-sampling layer, 732 is a convolution layer, 723 is an up-sampling layer, 724 is a convolution layer, 725 is an up-sampling layer, and 726 is a convolution layer; in another implementation, 731 is an up-sampling layer, 732 and 733 are convolution layers, 734 is an up-sampling layer, and 735 and 736 are convolution layers.

[0090] The convolution layer in the convolution layer / up-sampling layer 730 can also be referred to as a transposed convolution layer, which is used to up-sample the encoded representation and gradually recover the spatial dimensions of the data.

[0091] The up-sampling layer:

[0092] The up-sampling layer corresponds to the down-sampling layer and is used to increase the spatial resolution of the data.

[0093] Similarly, as shown in 731-732 in 730 in FIG. 7, it can be an up-sampling layer followed by a convolution layer, or one or more up-sampling layers followed by a plurality of convolution layers.

[0094] The output layer 740 has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation (as shown in FIG. 7, the propagation from 710 to 740 is forward propagation) of the entire convolutional neural network 700 is completed, the back propagation (as shown in FIG. 7, the propagation from 740 to 710 is back propagation) will start to update the weight values and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 700, and the error between the result output by the output layer of the convolutional neural network 700 and the ideal result.

[0095] The input layer 710 and the convolution layer / down-sampling layer 720 in the trained convolutional neural network 700 can be used as a feature extraction model.

[0096] It should be noted that the convolutional neural network 700 as shown in FIG. 7 is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models.

[0097] Next, a chip hardware structure provided by an embodiment of the present application is introduced.

[0098] Figure 8 is a chip hardware structure provided by an embodiment of the present application, which includes a convolutional neural network processor 80. The chip can be arranged in the execution device 610 shown in Figure 6 to complete the computing work of the computing module 611. The chip can also be arranged in the training device 620 shown in Figure 6 to complete the training work of the training device 620 and output the target model / rule 601. The algorithms of each layer in the convolutional neural network shown in Figure 7 can be implemented in the chip shown in Figure 8.

[0099] The convolutional neural network processor 80 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), or any other processor suitable for large-scale XOR operation processing. Taking the NPU as an example: the NPU can be mounted as a coprocessor to the central processing unit (CPU), i.e., the Host CPU, and tasks are allocated to it by the Host CPU. The core part of the NPU is the operation circuit 803, which extracts matrix data in the memory (801 and 802) and performs multiplication and addition operations under the control of the controller 804.

[0100] In some implementations, the operation circuit 803 includes a plurality of processing engines (PEs) inside. In some implementations, the operation circuit 803 is a two-dimensional systolic array. The operation circuit 803 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 803 is a general matrix processor.

[0101] For example, assuming there are input data A, weight data B, and output data C. The operation circuit 803 takes the weight values of data B from the weight memory 802 and caches them on each PE in the operation circuit 803. The operation circuit 803 takes the input data of data A from the input memory 801, performs operations according to the input data of data A and the weight values of data B, and saves the partial results or final results in the accumulator 808. The input data can be original non-speech frames, and the weight data can be a weight vector or matrix, i.e., a convolution kernel.

[0102] The unified memory 806 is used to store input data and output data. The weight vector or matrix is carried into the weight memory 802 by the direct memory access controller (DMAC) 805. The input data is also carried into the unified memory 806 by the direct memory access controller 805.

[0103] The bus interface unit (BIU) 810 is used for the interaction between the direct memory access controller 805 and the instruction fetch buffer 809; the bus interface unit 810 is also used for the instruction fetch buffer 809 to obtain instructions from an external memory; the bus interface unit 810 is also used for the direct memory access controller 805 to obtain original data of input data A or weight data B from an external memory.

[0104] The vector calculation unit 807 can include a plurality of operation processing units, which further process the output of the operation circuit 803 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. The vector calculation unit 807 is mainly used for the calculation of non-convolution layers or fully connected layers (FC) in the convolutional neural network, and can specifically process the calculation of pooling, normalization, etc. For example, the vector calculation unit 807 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 807 generates normalized values, merged values, or both.

[0105] In some implementations, the vector calculation unit 807 stores the processed vector to the unified memory 806. In some implementations, the vector processed by the vector calculation unit 807 can be used as an activation input of the operation circuit 803, for example, for use in a subsequent layer in the convolutional neural network, such as the current processing layer 731 in FIG. 7 is hidden layer 1, and the vector processed by the vector calculation unit 807 can also be used for calculation in hidden layer 2 (such as 732).

[0106] The instruction fetch buffer 809 connected to the controller 804 is used to store instructions used by the controller 804.

[0107] The unified memory 806, the input memory 801, the weight memory 802, and the instruction fetch buffer 809 are all on-chip memories.

[0108] In the convolutional neural network shown in FIG. 7, the operations of each layer can be performed by the operation circuit 803 or the vector calculation unit 807.

[0109] Optionally, in some embodiments, the sending device can further extract a linear prediction coding (LPC) of the original non-speech frame as the audio feature of the original non-speech frame.

[0110] FIG. 9 is a schematic flowchart of another method for processing a signal according to an embodiment of the present application. The method shown in FIG. 9 can be performed by a receiving device, or can be performed by a component (e.g., a chip, a system on chip (SoC), etc.) in the receiving device. The receiving device can be a mobile phone, a tablet computer, a smart watch, or the like terminal device capable of communication. For ease of description, it is assumed that the method shown in FIG. 9 is performed by the receiving device, and the method includes steps 901-903.

[0111] 901, the receiving device obtains N quantized audio features of original non-speech audio.

[0112] The receiving device can receive a bitstream from the sending device. The received bitstream is decoded and dequantized to obtain the N quantized audio features. The N quantized audio features correspond to the N audio features one-to-one. Each of the N quantized audio features is obtained by quantizing, encoding, decoding, and dequantizing the corresponding audio feature.

[0113] The audio features extracted from the non-speech audio can also be referred to as silence audio features. Correspondingly, the corresponding quantized audio features can also be referred to as quantized silence audio features.

[0114] 902, the receiving device determines a first predicted silence audio feature according to the N quantized audio features and a reference noise.

[0115] Optionally, in some embodiments, the N quantized audio features, the reference noise, and the first predicted silence audio feature satisfy the following relationship:

[0116] wherein SD N+1 represents the first predicted silence audio feature, SD i represents the i-th quantized audio feature in the N quantized audio features, ai represents the i-th preset parameter in the N preset parameters, noise represents the reference noise, β represents a preset parameter corresponding to the reference noise, and i = 1, …, N.

[0117] Optionally, in some embodiments, N is an integer less than or equal to 3 and greater than or equal to 1.

[0118] For example, if N equals 1, the first predicted silence audio feature SD2 = a1SD1 + bnoise.

[0119] For example, if N equals 2, the first predicted silence audio feature SD3 = a1SD1 + a2SD2 + bnoise.

[0120] For example, if N equals 3, the first predicted silence audio feature SD4 = a1SD1 + a2SD2 + a3SD3 + bnoise.

[0121] Optionally, in some embodiments, a i and b satisfy the following relationship:

[0122] In some embodiments, Pst_V is a number greater than or equal to 0.9 and less than or equal to 1.1. For example, Pst_V can equal 0.9, 0.95, 0.99, 1, 1.01, 1.001, 1.0001, 1.05, 1.1, etc.

[0123] In other embodiments, Pst_V is a number greater than or equal to 1 and less than or equal to 1.1. For example, Pst_V can equal 1, 1.01, 1.001, 1.0001, 1.05, 1.1, etc.

[0124] In some embodiments, Pst_V is a number greater than or equal to 0.99 and less than or equal to 1.01. For example, Pst_V can equal 0.99, 1, 1.01, 1.001, 1.0001, etc.

[0125] In other embodiments, Pst_V is a number greater than or equal to 1 and less than or equal to 1.01. For example, Pst_V can equal 1, 1.01, 1.001, 1.0001, etc.

[0126] Pst_V can be a preset value, a value determined by the sending end device, or a value determined by the sending end device and the receiving end device in negotiation.

[0127] The reference noise can be random noise, white noise, Gaussian noise, or noise with certain specific frequencies.

[0128] 903, the receiving end device determines, according to the first predicted silence audio feature and the neural network, a first non-speech audio corresponding to the original non-speech audio.

[0129] Optionally, in some embodiments, after determining the first non-speech audio corresponding to the original non-speech audio, the receiving end device can further predict a silence audio feature, and determine a second non-speech audio corresponding to the original non-speech audio according to the predicted silence audio feature.

[0130] For example, in some embodiments, in the case where N is equal to 1, the receiving end device can determine a second predicted silence audio feature according to the first predicted silence audio feature and the reference noise.

[0131] As described above, in the case where N is equal to 1, the first predicted silence audio feature SD2 = a1SD1 + bnoise. Accordingly, the second predicted silence audio feature SD3 = a1SD2 + bnoise.

[0132] For another example, in some embodiments, in the case where N is a positive integer greater than 1, the receiving end device can determine a second predicted silence audio feature according to the first predicted silence audio feature and the last N-1 quantized audio features of the N quantized audio features and the reference noise.

[0133] For example, in the case where N is equal to 2, the first predicted silence audio feature SD3 = a1SD1 + a2SD2 + bnoise. Accordingly, the second predicted silence audio feature SD4 = a1SD2 + a2SD3 + bnoise.

[0134] For another example, in the case where N is equal to 3, the first predicted silence audio feature SD4 = a1SD1 + a2SD2 + a3SD3 + bnoise. Accordingly, the second predicted silence audio feature SD5 = a1SD2 + a2SD3 + a3SD4 + bnoise.

[0135] In the case where the second predicted silence audio feature is determined, a second non-speech audio can be determined according to the second predicted silence audio feature and the non-speech audio determination model. The determination process of the second non-speech audio is similar to that of the first non-speech audio, and thus will not be described here for brevity.

[0136] Optionally, in some embodiments, the receiving end device can further determine N non-speech audios according to the N quantized audio features and the non-speech audio determination model.

[0137] For example, in some embodiments, the quantized audio features can be directly input to the non-speech audio determination model. The non-speech audio determination model can determine a non-speech audio corresponding to the quantized audio features based on the input quantized audio features.

[0138] For example, in some embodiments, the N quantized audio features can be processed before the audio features are quantized. For example, the N audio features can be smoothed, and then the smoothed N audio features can be quantized. The non-speech audio determination model can determine the non-speech audio corresponding to the quantized N audio features based on the smoothed N quantized audio features.

[0139] For the jth quantized audio feature of the N quantized audio features, the smoothing operation of the jth quantized audio feature can be determined based on P frames before the jth quantized audio feature, where P is a positive integer greater than or equal to 2. If j < P, the P frames can include language frames before the quantized audio features. The relationship between the jth quantized audio feature and the P frames can satisfy the following formula:

[0140] where Fj is the result of the jth quantized audio feature after smoothing, f i represents the ith frame of the P frames, f j represents the jth quantized audio feature, a i represents the jth preset parameter of the P-1 preset parameters, a p represents the pth preset parameter of the P preset parameters, noise represents the reference noise, β represents the preset parameter corresponding to the reference noise, i = 1, …, N.

[0141] The reconstructed audio of the original non-speech audio includes the N non-speech audios and the non-speech audio predicted from the N non-speech audios, for example, the first non-speech audio.

[0142] For example, in FIG. 5, the length T total of the silence segment output by the VAD module is T total , where the original non-speech audio is the segment of non-speech audio from time t1 to time t4. The encoding end device extracts N audio features from the segment of non-speech audio, then quantizes, encodes the N extracted audio features, and transmits the code stream to the receiving end device. However, the segment of non-speech audio from time t4 to time t5 can not need to go through the above process. The receiving end device can obtain N non-speech audios based on the N quantized audio features obtained from the code stream and a neural network, and the length of the N non-speech audios is t4-t1. Then, the receiving end device can predict the silence audio features using the N quantized audio features, and then obtain the predicted non-speech audio based on the predicted silence audio features and the neural network, and the length of the predicted non-speech audio is t5-t4.

[0143] The non-speech audio corresponding to the original non-speech audio determined by the receiving end device is the target non-speech audio mentioned above.

[0144] The neural network used to determine the first non-speech audio can be referred to as a non-speech audio determination model.

[0145] The input of the non-speech audio determination model is the first predicted silence audio feature, and the output of the non-speech audio determination model is the first non-speech audio. The non-speech audio determination model can be a neural network, a DNN, a CNN, an RNN, a Transformer model, etc.

[0146] The role of the non-speech audio determination model is to output a non-speech audio as much as possible. The training samples used to train the non-speech audio determination model can include a plurality of non-speech audios and silence audio features determined according to the plurality of non-speech audios. In the process of training the non-speech audio determination model, the difference between the non-speech audio output by the current model and the non-speech audio in the training sample can be compared. Then, according to the difference between the two, the parameters in the current model are updated. For example, if the non-speech audio determination model is a CNN, then the weight vector of each layer of the convolutional neural network can be updated according to the difference between the non-speech audio output by the current model and the non-speech audio in the training sample (of course, before the first update, there is usually an initialization process, that is, the parameters of each layer in the convolutional neural network are pre-configured), until the convolutional neural network can restore the non-speech audio or basically restore the non-speech audio. The convolutional neural network can use the back propagation (BP) algorithm to correct the size of the parameters in the convolutional neural network during the training process, so that the error loss between the predicted value output by the convolutional neural network and the target value that is actually wanted becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output produces an error loss, and the error loss information is propagated backward to update the parameters in the initial convolutional neural network, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the convolutional neural network, such as weight vectors or matrices, that is, the convolution kernel of the convolutional layer.

[0147] In some embodiments, the training samples for training the non-speech audio determination model can be determined according to the feature extraction model. As described above, the training samples for training the non-speech audio determination model can include a plurality of non-speech audios and the silence audio features determined according to the plurality of non-speech audios, and the silence audio features in the training samples can be determined by the feature extraction model according to the plurality of non-speech audios. Assuming that there is a data set X, the data set X includes a plurality of non-speech frames. The non-speech frames in the data set X are respectively input into the trained feature extraction model to obtain a feature set Z, and the feature set Z includes a plurality of silence audio features, which correspond to the plurality of non-speech frames in a one-to-one manner. The obtained audio features are quantized and encoded to obtain a code stream containing the plurality of silence audio features, and the code stream is obtained through a transmission link or an analog transmission link. The code stream is decoded and dequantized to obtain a plurality of quantized silence audio features. The plurality of quantized silence audio features and the plurality of non-speech frames in the data set X can be used as training samples of the non-speech audio determination model. The non-speech frames are target values, and the output obtained by the non-speech audio determination model based on the quantized audio features is a predicted value. By comparing the difference between the predicted value and the target value, the weight vector of the convolutional neural network can be adjusted.

[0148] FIG. 10 is a system architecture 1000 provided according to an embodiment of the present application.

[0149] As shown in the system architecture 1000, the data acquisition device 1040 is configured to acquire training data and store the training data in the database 1030. In an embodiment of the present application, the training data includes one or more non-speech audios and silence audio features corresponding to the one or more non-speech audios. The training device 1020 can train a target model / rule 1001 (i.e., a non-speech audio determination model) based on the training data maintained in the database 1030. The target model / rule 1001 can determine non-speech audios based on silence audio features, i.e., the silence audio features are input into the target model / rule 1001 to obtain the non-speech audios.

[0150] It should be noted that in actual applications, the training data maintained in the database 1030 can not all be acquired by the data acquisition device 1040, but can also be received from other devices. In addition, it should be noted that the training device 1020 can not completely train the target model / rule 1001 based on the training data maintained in the database 1030, but can also obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation on the embodiments of the present application.

[0151] The target model / rule 1001 trained by the training device 1020 can be applied to an execution device 1010 shown in FIG. 10, which can be the receiving device mentioned above. In FIG. 10, the execution device 1010 is configured with a communication module 1012. The communication module 1012 is configured to receive a bitstream. The execution device 1010 is further configured with a calculation module 1011. The calculation module 1011 is configured to decode and dequantize the bitstream to obtain N quantized audio features. Then, the calculation module 1011 is configured to determine non-speech audio based on the target model / rule 1001 and the N quantized audio features. The non-speech audio can then be played through a speaker or a headphone connected to the execution device 1010.

[0152] During the process of pre-processing the input data by the execution device 1010 or the process of calculation performed by the calculation module 1011, the modules (e.g., the calculation module 1011) in the execution device 1010 can call the data, code, etc. in the storage module 1013 to perform the corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the storage module 1013.

[0153] It is worth noting that FIG. 10 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in FIG. 10 does not constitute any limitation. For example, in FIG. 10, the storage module 1013 is placed in the execution device 1010. In other cases, the storage module 1013 can also be an external memory of the execution device 1010.

[0154] If the target model / rule 1001 is a convolutional neural network, the convolutional neural network processor 80 shown in FIG. 8 can also be arranged in the execution device 1010 shown in FIG. 10 to complete the calculation work of the calculation module 1011, or can also be arranged in the training device 1020 shown in FIG. 10 to complete the training work of the training device 1020 and output the target model / rule 1001.

[0155] FIG. 11 is a schematic structural block diagram of an electronic device according to an embodiment of the present application. The electronic device 1100 shown in FIG. 11 can be used as the sending device in the above embodiments. As shown in FIG. 11, the electronic device 1100 includes a processing unit 1101 and a sending unit 1102.

[0156] The processing unit 1101 is configured to extract audio features of original non-speech audio to obtain N audio features.

[0157] The processing unit 1101 is further configured to quantize and encode the N audio features to obtain a bitstream containing the N audio features.

[0158] The sending unit 1102 is configured to send the code stream obtained by the encoding of the processing unit 1101.

[0159] The specific functions and advantages of the processing unit 1101 and the sending unit 1102 can refer to the above embodiments, and will not be repeated here for brevity.

[0160] The processing unit 1101 can be implemented by a processor, and the sending unit 1102 can be implemented by a transmitter.

[0161] FIG. 12 is a schematic structural block diagram of another electronic device according to an embodiment of the present application. The electronic device 1200 shown in FIG. 12 can serve as the receiving end device in the above embodiments. As shown in FIG. 12, the electronic device 1200 includes an obtaining unit 1201 and a processing unit 1202.

[0162] The obtaining unit 1201 is configured to obtain N quantized audio features of an original non-speech audio, N being a positive integer greater than or equal to 1.

[0163] The processing unit 1202 is configured to determine a first predicted mute audio feature according to the N quantized audio features and a reference noise.

[0164] The processing unit 1202 is further configured to determine a first non-speech audio corresponding to the original non-speech audio according to the first predicted mute audio feature and a first neural network.

[0165] The specific functions and advantages of the obtaining unit 1201 and the processing unit 1202 can refer to the above embodiments, and will not be repeated here for brevity.

[0166] The obtaining unit 1201 can be implemented by a processor and a receiver. For example, the receiver receives a code stream, and the processor decodes and dequantizes the code stream to obtain the N quantized audio features.

[0167] The processing unit 1202 can be implemented by a processor.

[0168] FIG. 13 is a schematic diagram of another electronic device according to an embodiment of the present application. The electronic device 1300 shown in FIG. 13 includes a processor 1301, which is configured to execute computer programs or instructions stored in a memory 1302, or read data / signaling stored in the memory 1302, to perform the methods in the above method embodiments. Alternatively, the processor 1301 is one or more.

[0169] The memory 1302 is configured to store computer programs or instructions and / or data. The memory 1302 can be integrated with the processor 1301, or can be separately arranged. Alternatively, the memory 1302 is one or more.

[0170] Optionally, as shown in FIG. 13, the electronic device 1300 further includes a transceiver 1303 configured to receive and / or transmit signals. For example, the processor 1301 is configured to control the transceiver 1303 to receive and / or transmit signals.

[0171] As an option, the electronic device 1300 is configured to implement operations performed by a sending-end device in the various method embodiments above.

[0172] As another option, the electronic device 1300 is configured to implement operations performed by a receiving-end device in the various method embodiments above.

[0173] It should be understood that the processor mentioned in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0174] It should also be understood that the memory mentioned in the embodiments of the present application can be a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM). For example, the RAM can be used as an external cache. As an example but not limitation, the RAM includes the following various forms: static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0175] It should be noted that when the processor is a general processor, a DSP, an ASIC, a FPGA or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, the memory (storage module) can be integrated in the processor.

[0176] It should also be noted that the memory described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0177] FIG. 14 is a schematic diagram of a chip system 1400 provided by an embodiment of the present application. The chip system 1400 (or also can be referred to as a processing system) includes a logic circuit 1401 and an input / output interface 1402.

[0178] The logic circuit 1401 can be a processing circuit in the chip system 1400. The logic circuit 1401 can be coupled to a storage unit, and invoke instructions in the storage unit, so that the chip system 1400 can implement the methods and functions of the embodiments of the present application. The input / output interface 1402 can be an input / output circuit in the chip system 1400, and output information processed by the chip system 1400, or input data or signaling information to be processed by the chip system 1400.

[0179] As an option, the chip system 1400 is configured to implement operations performed by the sending device or the receiving device in the above method embodiments.

[0180] The embodiments of the present application also provide a computer readable storage medium, which stores computer instructions for implementing the method performed by the sending device or the receiving device in the above method embodiments.

[0181] For example, the computer program is executed by an electronic device, so that the electronic device can implement the method performed by the sending device or the receiving device in the above method embodiments.

[0182] The embodiments of the present application also provide a computer program product, which contains instructions, and the instructions are executed by a computer to implement the method performed by the sending device or the receiving device in the above method embodiments.

[0183] The embodiments of the present application also provide a communication system, which includes the sending device and the receiving device as described above.

[0184] The above provides any one of the devices related to the explanation and beneficial effects of the above-mentioned corresponding method embodiments, and will not be repeated here.

[0185] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of processing a signal, characterized by, The method comprises: obtaining N quantized audio features of original non-speech audio, N being a positive integer greater than or equal to 1; determining a first predicted mute audio feature according to the N quantized audio features and reference noise; determining a first non-speech audio corresponding to the original non-speech audio according to the first predicted mute audio feature and a first neural network.

2. The method of claim 1, wherein, The N quantized audio features, the reference noise and the first predicted silence audio feature satisfy the following relationship: wherein SD N+1 represents the first predicted silence audio feature, SD i represents the i-th quantized audio feature in the N quantized audio features, a i represents the i-th preset parameter in the N preset parameters, noise represents the reference noise, β represents a preset parameter corresponding to the reference noise, and i = 1, …, N.

3. The method of claim 2, wherein, α i and β satisfy the following relationship: Wherein Pst_V is a number greater than or equal to 0.9 and less than or equal to 1.

1.

4. The method according to any one of claims 1 to 3, characterized in that, The reference noise is random noise.

5. The method according to any one of claims 1 to 4, characterized in that, N is a positive integer less than or equal to 3.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: determining a second predicted mute audio feature according to N reference audio features and the reference noise, wherein in the case of N equal to 1, the N reference audio features are the first predicted mute audio feature, and in the case of N being a positive integer greater than 1, the N reference audio features comprise the first predicted mute audio feature and the last N-1 quantized audio features among the N quantized audio features; determining a second non-speech audio according to the second predicted mute audio feature and the first neural network.

7. The method according to any one of claims 1 to 6, characterized in that, The method of obtaining N quantized audio features of original non-speech audio comprises: receiving a code stream through a low-code-rate audio communication channel, the low-code-rate audio communication channel at least comprising a satellite communication channel, the code stream containing N audio features quantized and encoded, the N audio features being extracted from the original non-speech audio; decoding and dequantizing the code stream to obtain the N quantized audio features, the N quantized audio features corresponding one-to-one to the N audio features.

8. The method of claim 7, wherein, The N audio features are mel-frequency spectrograms of the original non-speech audio or mel-frequency cepstral coefficients of the original non-speech audio, or the N audio features are features extracted by a second neural network from the original non-speech audio.

9. The method according to any one of claims 1 to 8, characterized in that, The method further comprises: determining N non-speech audios according to the N quantized audio features and the first neural network, wherein a reconstructed audio of the original non-speech audio comprises the N non-speech audios and the first non-speech audio.

10. An electronic device, comprising: The method comprises: an obtaining unit configured to obtain N quantized audio features of original non-speech audio, N being a positive integer greater than or equal to 1; a processing unit configured to determine a first predicted mute audio feature according to the N quantized audio features and reference noise; the processing unit is further configured to determine a first non-speech audio corresponding to the original non-speech audio according to the first predicted mute audio feature and a first neural network.

11. The electronic device of claim 10, wherein, The N quantized audio features, the reference noise and the first predicted silence audio feature satisfy the following relationship: wherein SD N+1 represents the first predicted silence audio feature, SD i represents the i-th quantized audio feature in the N quantized audio features, a i represents the i-th preset parameter in the N preset parameters, noise represents the reference noise, β represents a preset parameter corresponding to the reference noise, and i = 1, …, N.

12. The electronic device of claim 11, wherein, α i and β satisfy the following relationship: Wherein Pst_V is a number greater than or equal to 0.9 and less than or equal to 1.

1.

13. The electronic device of any of claims 10-12, wherein, The reference noise is random noise.

14. The electronic device of any of claims 10-13, wherein, N is a positive integer less than or equal to 3.

15. The electronic device of any of claims 10-14, wherein, The processing unit is further configured to determine a second predicted mute audio feature according to N reference audio features and the reference noise, wherein in the case of N equal to 1, the N reference audio features are the first predicted mute audio feature, and in the case of N being a positive integer greater than 1, the N reference audio features comprise the first predicted mute audio feature and the last N-1 quantized audio features among the N quantized audio features. The processing unit is further configured to determine, according to the second predicted non-speech audio feature and the first neural network, a second non-speech audio.

16. The electronic device of any of claims 10 to 15, wherein, The obtaining unit is specifically configured to receive a code stream through a low-code-rate audio communication channel, decode and dequantize the code stream to obtain the N quantized audio features, the low-code-rate audio communication channel at least including a satellite communication channel, the code stream containing N quantized and encoded audio features, the N audio features being extracted from the original non-speech audio, and the N quantized audio features corresponding to the N audio features one by one.

17. The electronic device of claim 16, wherein, The N audio features are mel-frequency spectrum graphs of the original non-speech audio or mel-frequency cepstral coefficients of the original non-speech audio, or the N audio features are features extracted by a second neural network from the original non-speech audio.

18. The electronic device of any of claims 10-17, wherein, The processing unit is further configured to determine, according to the N quantized audio features and the first neural network, N non-speech audios, wherein the reconstructed audio of the original non-speech audio includes the N non-speech audios and the first non-speech audio.

19. An electronic device, comprising: The computer readable medium stores program codes, when the program codes run on the electronic device, make the electronic device execute the method in any one of claims 1 to 9. The computer readable medium stores program codes, when the program codes run on the electronic device, make the electronic device execute the method in any one of claims 1 to 9.

20. A chip system, characterized by The logic circuit is coupled with the input / output interface, and transmits data through the input / output interface to execute the method in any one of claims 1 to 9. The computer readable medium stores program codes, when the program codes run on the electronic device, make the electronic device execute the method in any one of claims 1 to 9.

21. A computer readable medium characterized by ​

Images