Audio processing method, device, electronic device and storage medium

Abstract

The present disclosure provides an audio processing method, device, electronic device and storage medium, the audio processing method comprising: encoding an input audio signal through a self-learning network for encoding; dividing the encoded audio signal into N encoding streams, wherein N is an integer greater than 1; selecting n encoding streams from the N encoding streams in a preset manner and setting them to zero, wherein n is any integer from 0 to N-1; quantizing and integrating the remaining encoding streams after setting them to zero in the N encoding streams to generate an integrated encoding stream; decoding the integrated encoding stream through a self-learning network for decoding to obtain a decoded audio signal; comparing the decoded audio signal with the input audio signal and adjusting the self-learning network for encoding and the self-learning network for decoding according to the comparison result. The audio processing method of the present disclosure can achieve high sound quality at a low bit rate and can cope with poor transmission channel conditions.

Description

Technical Field

[0001] The present disclosure relates to the field of audio technology, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] Audio processing has always been a hot topic in online audio services. The self-learning-based audio codec in the related art uses an encoder with a self-learning network to compress the audio signal during training, transmits the compressed audio frame signal through the transmission channel, and then restores the original audio signal from the compressed information through the corresponding self-learning network at the decoding end. The audio encoder based on deep learning can improve the performance of traditional audio encoders at low bit rates and achieve higher sound quality. However, due to the influence of the transmission channel bandwidth and packet loss in actual scenarios, when jitter and large packet loss occur in the transmission channel, the self-learning-based audio decoder cannot decode because the received information is too little or completely missing. Summary of the invention

[0003] The present disclosure provides an audio processing method, device, electronic device and storage medium to at least solve the problem of packet loss recovery in the related art, and may not solve any of the above problems.

[0004] According to a first aspect of the present disclosure, there is provided a method for audio processing, comprising: encoding an input audio signal by a self-learning network for encoding;

[0005] Dividing the encoded audio signal into N encoded streams, where N is an integer greater than 1;

[0006] Select n coding streams from the N coding streams in a preset manner and set them to zero, wherein n is any integer from 0 to N-1; quantize and integrate the remaining coding streams after the N coding streams are set to zero to generate an integrated coding stream; decode the integrated coding stream by a self-learning network for decoding to obtain a decoded audio signal; compare the decoded audio signal with the input audio signal and adjust the self-learning network for encoding and the self-learning network for decoding according to the comparison result.

[0007] According to the first aspect of the present disclosure, encoding the input audio signal through the self-learning network for encoding includes: reducing the number of samples of the audio frame of the input audio signal to a predetermined dimension through the self-learning network for encoding, wherein the dimension of the reduced audio signal is such that the deviation between the signal obtained after decoding the reduced audio signal and the input audio signal is less than or equal to a preset threshold.

[0008] According to the first aspect of the present disclosure, the dividing of the encoded audio signal into N encoding streams includes dividing the reduced-dimensional audio signal in one of the following ways: allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams evenly and without overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams evenly and with overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams unevenly and without overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams unevenly and with overlapping.

[0009] According to a first aspect of the present disclosure, a self-learning network for decoding includes a first decoder and a second decoder, wherein decoding an integrated coded stream to obtain a decoded audio signal includes: determining the number of coded streams included in the integrated coded stream; in response to determining that the number of included coded streams is the N, using the first decoder to decode the integrated coded stream; in response to determining that the number of included coded streams is less than the N, using the second decoder to decode the integrated coded stream.

[0010] According to the first aspect of the present disclosure, the self-learning network for encoding and the self-learning network for decoding have corresponding network structures, and the process of comparing the decoded audio signal with the input audio signal and adjusting the audio encoder and the self-learning network for decoding according to the comparison result includes: adjusting the audio encoder and the self-learning network for decoding through a loss function, wherein the loss function includes N deviation values ​​between the audio signal obtained by decoding and the input audio signal when 1, 2, ..., N encoded streams are received respectively, and a weight corresponding to each deviation value.

[0011] According to the first aspect of the present disclosure, the deviation value between the audio signal obtained by decoding and the input audio signal includes one of the following indicator values: mean square error MSE in time domain or frequency domain, mean absolute error MAE in time domain or frequency domain, mean square logarithmic error MSLE in time domain or frequency domain, objective speech quality assessment PESQ or objective perceptual listening quality assessment POLQA.

[0012] According to a second aspect of the present disclosure, there is provided an audio processing method, comprising: encoding an input audio signal using a self-learning network for encoding; dividing the encoded audio signal into N encoding streams, wherein N is an integer greater than 1; quantizing the N encoding streams to generate N audio encoding streams, wherein the self-learning network is a network obtained by the method described above.

[0013] According to a second aspect of the present disclosure, encoding an audio signal using an encoder of a self-learning network for encoding includes: reducing the number of samples of an audio frame of the audio signal to a predetermined dimension, wherein the dimension of the reduced audio signal is such that a deviation between a signal obtained after decoding the reduced audio signal and the input audio signal is less than or equal to a preset threshold.

[0014] According to the second aspect of the present disclosure, dividing the encoded audio signal into N encoding streams includes dividing the reduced-dimensional audio signal in one of the following ways: allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams evenly and without overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams evenly and with overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams unevenly and without overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams unevenly and with overlapping.

[0015] According to a third aspect of the present disclosure, there is provided an audio processing method, comprising: receiving an audio coding stream from a transmitting end; integrating the received audio coding streams; and decoding the integrated audio coding streams by a self-learning network for decoding, wherein the received audio coding stream is at least one of N audio coding streams sent from the transmitting end, N being an integer greater than 1, and the N audio coding streams are generated by encoding an audio signal and segmenting the encoded audio signal by a self-learning network for encoding corresponding to the self-learning network for decoding, and the self-learning network for encoding and the self-learning network for decoding are networks obtained by the method described above.

[0016] According to a third aspect of the present disclosure, a self-learning network for decoding includes a first decoder and a second decoder, and decoding an integrated audio coding stream includes: determining the number of coding streams included in the integrated coding stream; in response to determining that the integrated audio coding stream includes N audio coding streams, using the first decoder to decode the integrated audio coding stream; in response to determining that the integrated audio coding stream includes less than N audio coding streams, using the second decoder to decode the integrated audio coding stream.

[0017] According to a fourth aspect of the present disclosure, an audio processing device is provided, comprising: an encoding unit, configured to encode an input audio signal through a self-learning network for encoding; a division unit, configured to divide the encoded audio signal into N encoding streams, wherein N is an integer greater than 1; a selection unit, configured to select n encoding streams from the N encoding streams in a preset manner and set them to zero, wherein n is any integer from 0 to N-1; an integration unit, configured to quantize and integrate the remaining encoding streams after the N encoding streams are set to zero to generate an integrated encoding stream; a decoding unit, configured to decode the integrated encoding stream through the self-learning network for decoding to obtain a decoded audio signal; and a training unit, configured to compare the decoded audio signal with the input audio signal and adjust the self-learning network for encoding in the encoding unit and the self-learning network for decoding in the decoding unit according to the comparison result.

[0018] According to a fourth aspect of the present disclosure, the encoding unit is configured to reduce the number of samples of an audio frame of an input audio signal to a predetermined dimension through a self-learning network for encoding, wherein the dimension of the reduced audio signal is such that a deviation between a signal obtained after decoding the reduced audio signal and the input audio signal is less than or equal to a preset threshold.

[0019] According to a fourth aspect of the present disclosure, the division unit is configured to divide the audio signal after dimensionality reduction in one of the following ways: allocating the samples of the audio frames of the audio signal after dimensionality reduction to the N coding streams evenly and without overlapping; allocating the samples of the audio frames of the audio signal after dimensionality reduction to the N coding streams evenly and with overlapping; allocating the samples of the audio frames of the audio signal after dimensionality reduction to the N coding streams unevenly and without overlapping; allocating the samples of the audio frames of the audio signal after dimensionality reduction to the N coding streams unevenly and with overlapping.

[0020] According to a fourth aspect of the present disclosure, the decoding unit includes a first decoder and a second decoder, and the decoding unit is configured to: determine the number of encoding streams included in the integrated encoding stream; in response to determining that the number of encoding streams included is the N, use the first decoder to decode the integrated encoding stream; in response to determining that the number of encoding streams included is less than the N, use the second decoder to decode the integrated encoding stream.

[0021] According to a fourth aspect of the present disclosure, the self-learning network for encoding and the self-learning network for decoding have corresponding network structures, and the training unit is configured to adjust the self-learning network for encoding and the self-learning network for decoding through a loss function, wherein the loss function includes N deviation values ​​between the decoded audio signal and the input audio signal when the decoding unit receives 1, 2, ..., N encoded streams respectively, and a weight corresponding to each value deviation.

[0022] According to the fourth aspect of the present disclosure, the deviation value between the audio signal obtained by decoding and the input audio signal includes one of the following indicator values: mean square error MSE in time domain or frequency domain, mean absolute error MAE in time domain or frequency domain, mean square logarithmic error MSLE in time domain or frequency domain, objective speech quality assessment PESQ or objective perceptual listening quality assessment POLQA.

[0023] According to a fifth aspect of the present disclosure, an audio processing device is provided, comprising: an encoding unit configured to encode an audio signal using a self-learning network for encoding; a division unit configured to divide the encoded audio signal into N encoding streams, where N is an integer greater than 1; and a quantization unit, which quantizes the N encoding streams to generate N audio encoding streams, wherein the self-learning network for encoding is a network obtained by the method described above.

[0024] According to a fifth aspect of the present disclosure, the encoding unit is configured to reduce the number of samples of the audio frame of the input audio signal to a predetermined dimension through the self-learning network for encoding.

[0025] According to a fifth aspect of the present disclosure, the division unit is configured to divide the audio signal after dimensionality reduction in one of the following ways: allocating the samples of the audio frames of the audio signal after dimensionality reduction to the N coding streams evenly and without overlapping; allocating the samples of the audio frames of the audio signal after dimensionality reduction to the N coding streams evenly and with overlapping; allocating the samples of the audio frames of the audio signal after dimensionality reduction to the N coding streams unevenly and without overlapping; allocating the samples of the audio frames of the audio signal after dimensionality reduction to the N coding streams unevenly and with overlapping.

[0026] According to a sixth aspect of the present disclosure, an audio processing device is provided, comprising: a receiving unit configured to receive an audio coding stream from a transmitting end; an integrating unit configured to integrate the received audio coding stream; and a decoding unit configured to decode the integrated audio coding stream through a self-learning network for decoding, wherein the received audio coding stream is at least one of N audio coding streams sent from the transmitting end, and the N audio coding streams are generated by encoding an audio signal and segmenting the encoded audio signal by a self-learning network for encoding corresponding to the self-learning network for decoding, and the self-learning network for encoding and the self-learning network for decoding are networks obtained by the method described above.

[0027] According to a sixth aspect of the present disclosure, the decoding unit includes a first decoder and a second decoder, and the decoding unit is configured to: determine the number of coding streams included in the integrated coding stream; in response to determining that the number of included coding streams is the N, use the first decoder to decode the integrated coding stream; in response to determining that the number of included coding streams is less than the N, use the second decoder to decode the integrated coding stream.

[0028] According to the seventh aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the audio processing method as described above.

[0029] According to an eighth aspect of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is enabled to execute the audio processing method as described above.

[0030] According to a ninth aspect of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor to perform the audio processing method as described above.

[0031] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: it avoids the disadvantages of the related technology's self-learning network coding that cannot cope well with network jitter and packet loss and the related technology's multiple description coding that has low sound quality at low bit rates. By dividing the coded signal generated by the self-learning network coding into multiple coded streams, the redundancy of the coded signal can be improved. At the same time, the original audio signal information contained in the multiple coded streams is constrained through the training of the self-learning network, thereby being able to cope with the jitter and packet loss of the transmission channel and achieving higher sound quality at a lower bit rate.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0034] Figure 1 is a diagram illustrating a system environment in which an audio processing method according to an exemplary embodiment of the present disclosure is implemented.

[0035] Figure 2 is a flowchart illustrating a method for audio processing according to an exemplary embodiment of the present disclosure.

[0036] Figure 3 is a schematic diagram illustrating a training process for an audio codec according to an exemplary embodiment of the present disclosure.

[0037] Figure 4 is a schematic diagram showing the structure of a self-learning network according to an exemplary embodiment of the present disclosure.

[0038] Figure 5 is a schematic diagram illustrating a training process for an audio codec according to another exemplary embodiment of the present disclosure.

[0039] Figure 6 2 is a schematic diagram illustrating an encoding process of an audio processing method and a decoding process of the audio processing method according to an exemplary embodiment of the present disclosure.

[0040] Figure 7 A schematic diagram illustrating an encoding process of an audio processing method and a decoding process of an audio processing method according to another exemplary embodiment of the present disclosure.

[0041] Figure 8 is a flowchart illustrating an audio processing method according to an exemplary embodiment of the present disclosure.

[0042] Fig. 9 is a flowchart illustrating an audio processing method according to an exemplary embodiment of the present disclosure.

[0043] Fig.10 is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment of the present disclosure.

[0044] Fig.11 is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment of the present disclosure.

[0045] Fig.12is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment of the present disclosure.

[0046] Fig.13 is a schematic diagram illustrating an electronic device for audio processing according to an exemplary embodiment of the present disclosure.

[0047] Fig.14 is a schematic diagram illustrating an electronic device for audio processing according to another exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0050] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0051] Before describing the embodiments of the present disclosure in detail, some terms or abbreviations that may be involved in the embodiments of the present disclosure are first described.

[0052] Figure 1 1 is a diagram showing a system environment for implementing an audio processing method according to an exemplary embodiment of the present disclosure. The environment may be composed of, for example, a terminal device 100 and a server 200 used by a user and a network connecting the terminal device and the server.

[0053] like Figure 1As shown, the terminal device 100 may be a terminal device with an audio and video processing function. For example, the terminal device 100 in the embodiment of the present disclosure may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a netbook, a personal digital assistant (PDA), an augmented reality (AR) / virtual reality (VR) device. Various audio and video applications, such as live broadcast applications, social applications, video conferencing applications, etc., may be run on the terminal device 100. When the terminal device 100 is running these applications, it may collect the voice generated by the user, and after pre-processing the collected voice, encode and generate an audio encoding stream including an audio packet, and then connect and communicate with the server 200 through a network, so that the audio encoding stream is provided to other terminal devices 100 by the server 200. The terminal device 100 according to the exemplary embodiment of the present disclosure may include an audio codec for audio encoding and decoding, and the audio codec method based on a self-learning network according to the exemplary embodiment of the present disclosure may be implemented in the audio codec. Here, the self-learning network of the audio encoder may be first trained in a simulation environment, and then the trained self-learning network may be used to perform encoding and decoding in the audio codec.

[0054] The following will refer to Figure 2-Figure 4 To illustrate an audio processing method according to various exemplary embodiments of the present disclosure.

[0055] Figure 2 is a flowchart illustrating an audio processing method according to an exemplary embodiment of the present disclosure. Figure 3 A schematic diagram of a training process for an audio codec according to an exemplary embodiment of the present disclosure is shown. Figure 4 A structural schematic diagram of a self-learning network according to an exemplary embodiment of the present disclosure is shown.

[0056] like Figure 2 As shown, first, in step S210, the original audio signal is encoded by a self-learning network for encoding. Figure 3 As shown, the encoder side may have a self-learning network based on artificial intelligence. The structure of the self-learning network can be as follows Figure 4 As shown on the left side of . The self-learning network according to an exemplary embodiment of the present disclosure may include multiple two-dimensional convolutional layers and multiple pooling layers. For example, the two-dimensional convolutional layer may have hyperparameters as shown in Table 1 below.

[0057] Table 1

[0058] layer Number of filters core Step Length Convolutional layer 1 32 (6,2) (2,1) Convolutional Layer 2 32 (6,2) (2,1) Convolutional layer 3 64 (7,2) (2,1) Convolutional layer 4 64 (6,2) (2,1) Convolutional layer 5 96 (6,2) (2,1) Convolutional layer 6 96 (6,2) (2,1) Convolutional layer 7 128 (2,2) (2,1) Convolutional layer 8 256 (2,2) (1,1)

[0059] Through the self-learning network as described above, the encoder can reduce the dimension of each frame of the audio signal to a predetermined dimension, thereby achieving compression of the audio signal. For example, assuming that the length of an audio frame of the audio signal is 1024 samples, after the dimension is reduced by the self-learning network of the encoder, the output audio signal after the dimension reduction is a vector smaller than 1024 points.

[0060] According to an exemplary embodiment of the present disclosure, the dimension of the audio signal after dimensionality reduction is such that the deviation between the signal obtained after the dimensionality reduction audio signal is decoded and the original audio signal is less than or equal to a preset threshold. For example, the ratio between the difference between the sample value of the original signal and the sample value of the audio signal after dimensionality reduction and the sample value of the original signal can be set to be less than or equal to a preset threshold. The dimension of the audio signal after dimensionality reduction can be determined by experiments. That is, the vector dimension of the audio signal output by the encoder is gradually reduced until the deviation between the output audio signal obtained by decoding by the decoder and the audio signal input to the encoder exceeds a preset threshold. The deviation between the output audio signal and the input audio signal can be calculated by a loss function, and the loss function will be described in more detail later. Here, assuming that it is determined experimentally that the output of the encoder is 128 dimensions and can meet the accuracy requirements after decoding, a 1024-dimensional signal can be compressed to 128 dimensions by the encoder.

[0061] Next, in step S220, the encoded audio signal is divided into N encoded streams, where N is an integer greater than 1. Here, the compressed audio signal can be divided into a predetermined number of N parts by a splitter, that is, a Multiple Description Code (MDC) stream is generated.

[0062] According to an exemplary embodiment of the present disclosure, a variety of forms can be used to divide the audio signal. For example, the audio signal after dimensionality reduction is divided in one of the following four ways: 1. The sample points of the audio frame of the audio signal after dimensionality reduction are evenly and non-overlappingly allocated to the N coding streams; 2. The sample points of the audio frame of the audio signal after dimensionality reduction are evenly and overlappedly allocated to the N coding streams; 3. The sample points of the audio frame of the audio signal after dimensionality reduction are unevenly and non-overlappingly allocated to the N coding streams; 4. The sample points of the audio frame of the audio signal after dimensionality reduction are unevenly and overlappedly allocated to the N coding streams. The first two methods are balanced divisions, and the last two methods are unbalanced divisions.

[0063] Taking the division of the 128-dimensional audio signal after dimensionality reduction into two MDC streams as an example, in the first method, the 1st to 64th dimensions of the audio signal can be assigned to MDC1, and the 65th to 128th dimensions can be assigned to MDC2. In the second method, the 1st to 96th dimensions can be assigned to MDC1, and the 33rd to 128th dimensions can be assigned to MDC2. In the third method, the 1st to 96th dimensions can be assigned to MDC1, and the 97th to 128th dimensions can be assigned to MDC2. In the fourth method, the 1st to 96th dimensions can be assigned to MDC1, and the 65th to 128th dimensions can be assigned to MDC2. The method of dividing the audio signal is not limited here.

[0064] Then, in step S230, n coded streams are selected from the N coded streams in a preset manner and set to zero, where n is any integer from 0 to N-1.

[0065] According to an exemplary embodiment of the present disclosure, a selection module may be introduced to perform zeroing operations on a predetermined number of coded streams among the N coded streams in a preset mode (e.g., a random mode). In other words, different zeroing operations may be performed on a predetermined number of MDC streams of each audio frame.

[0066] Next, in step S240, the remaining coded streams after the N coded streams are set to zero are quantized and integrated to generate an integrated coded stream.

[0067] like Figure 3 As shown, after the segmenter divides the encoded audio signal into N coding streams (e.g., multiple description coding stream MDC 1-multiple description coding stream MDC N), the above operations S230 and S240 can be performed by a random selector, a quantizer and an integrator to generate an integrated coding stream to be input to the decoder end.

[0068] Then, in step S250, the integrated encoded stream is decoded by a self-learning network for decoding to obtain a decoded audio signal.

[0069] The self-learning network used for decoding can have a structure corresponding to the self-learning network in the encoder, so as to ensure that the decoder can completely restore the encoded signal compressed by the self-learning network at the encoder. In other words, the two networks can have corresponding hierarchical structures and can also use the same network parameters. For example, Figure 4 As shown, the self-learning network at the decoder end may have the same number of two-dimensional convolutional layers (Conv2DTranspose) and upsampling layers corresponding to the pooling layers as the self-learning network at the encoder end, and use the Softmax function as the output.

[0070] Finally, in step S260, the decoded audio signal is compared with the input audio signal and the self-learning network for encoding and the self-learning network for decoding are adjusted according to the comparison result. According to an exemplary embodiment of the present disclosure, various parameters of the audio encoder and the self-learning network for decoding can be adjusted by a pre-designed loss function until the loss function converges. According to an exemplary embodiment of the present disclosure, the loss function may include N deviation values ​​between the audio signal obtained by decoding and the input audio signal when 1, 2, ..., N coded streams are received respectively, and a weight corresponding to each deviation value. For example, the loss function L can be calculated by the following equation:

[0071] L=w1·Loss1+…+w N ·Loss N

[0072] Among them, w is the weight, Loss is the loss function, and the subscripts 1 to N are the difference between the restored audio signal and the original audio signal after receiving n (1≤n≤N) MDC streams.

[0073] Here, the loss function can be expressed in any of the following ways:

[0074] (1) Mean square error (MSE) (time domain or frequency domain)

[0075]

[0076] Among them, S ori (n) represents the original audio signal sequence, S decoded (n) represents the audio signal sequence after decoding and recovery, MSE time Represents the loss in the time domain, MSE frequency Represents the loss in the frequency domain, S(i, j)=STFT(S(n)), i is the frame sequence index, and j is the frequency sequence index.

[0077] (2) Mean square error (MAE) (time domain or frequency domain)

[0078]

[0079] Among them, S ori (n) represents the original audio signal sequence, S decoded (n) represents the audio signal sequence after decoding and recovery, MAE time Represents the loss in the time domain, MAE frequency Represents the loss in the frequency domain, S(i, j)=STFT(S(n)), i is the frame sequence index, and j is the frequency sequence index.

[0080] (3) Mean square logarithmic error (MSLE) (time domain or frequency domain)

[0081]

[0082] Among them, S ori (n) represents the original audio signal sequence, S decoded (n) represents the audio signal sequence after decoding and recovery, MSLE time Represents the loss in the time domain, MSLE frequency Represents the loss in the frequency domain, S(i, j)=STFT(S(n)), i is the frame sequence index, and j is the frequency sequence index.

[0083] (4) PESQ (Perceptual Evaluation of Speech Quality)

[0084] ITU-T P862 Recommendation provides an objective MOS value evaluation method.

[0085] (5) POLQA (Perceptual Objective Listening Quality Assessment)

[0086] The POLQA algorithm is a new generation of voice quality assessment standard, which is applicable to voice quality assessment in fixed networks, mobile communication networks and IP networks. POLQA has been identified by ITU-T as the recommended specification P.863, which can be used for voice quality assessment of HD voice, 3G, 4G / VoLTE, and 5G networks.

[0087] According to the loss function designed in the above manner, the audio quality when receiving different numbers of packets can be flexibly adjusted according to application requirements or network packet loss statistics. For example, in the case of two-way MDC, if two signals are received, the original signal can be perfectly restored, and if any one is received, the low-quality original signal is restored. Here, since the actual network output cannot be perfect, the possible choices are: if the design needs to prioritize the original signal when both channels are received, then w1=0.2, w2=0.8; if the design needs to prioritize the optimal value when any one channel is received, then w1=0.8, w2=0.2.

[0088] It should be understood that the above loss functions are only illustrative, and those skilled in the art may adopt other types of loss functions as needed.

[0089] Through the above training process, a multi-description deep speech codec can be realized. By self-learning to constrain the original audio signal information contained in multiple descriptions, the advantages of self-learning network coding and multi-description coding can be fully utilized, and the disadvantages of related technology self-learning network coding that cannot cope well with network jitter and packet loss and related technology multi-description coding that has low sound quality at low bit rates are avoided, thereby achieving higher quality and efficient audio coding.

[0090] Figure 5 is a schematic diagram illustrating a training process for an audio codec according to another exemplary embodiment of the present disclosure. Figure 5 The training process and Figure 3 The training process is similar to that of , except that two decoders are used, one decoder (central decoder) processes the integrated coded stream when receiving the full coded stream, and the other decoder (bypass decoder) processes the integrated coded stream when receiving a partial coded stream. Figure 5 As shown, all the coding streams can be directly input into the quantizer and the integrator, and then the integrated coding stream can be decoded by the central decoder. A part of the coding streams can be selected in a predetermined manner through the random selection module of the whole coding stream for quantization and integration, and the integrated coding stream can be decoded by the bypass decoder.

[0091] That is, according to an exemplary embodiment of the present disclosure, the self-learning network for decoding includes a first decoder and a second decoder, and step S250 may include: determining the number of coding streams included in the integrated coding stream; in response to determining that the number of included coding streams is the N (i.e., including all the coding streams generated by the encoder end), using the first decoder to decode the integrated coding stream; in response to determining that the number of included coding streams is less than the N (i.e., only including part of the generated coding streams), using the second decoder to decode the integrated coding stream.

[0092] Using two decoders can make the decoder input form and the amount of information covered more uniform. When using multiple decoders, since the number of coded streams input to each decoder is the same (the amount of information covered is the same), the optimal decoding effect can be achieved for different numbers of coded streams received.

[0093] In such Figure 3 and Figure 5 After the self-learning networks in the encoder and decoder are trained, the trained codec can be used to encode and decode audio signals.

[0094] Figure 6 and Figure 7 2 is a schematic diagram illustrating an encoding process of an audio processing method and a decoding process of the audio processing method according to an exemplary embodiment of the present disclosure.

[0095] and Figure 3 and Figure 5 The training process shown is similar. In the inference stage (i.e., the encoding and decoding stage), at the transmitting end, the audio signal is first encoded by the self-learning network in the encoder, and the encoded audio signal is divided into multiple multiple description coding (MDC) streams and quantized by the segmenter. The divided and quantized MDC streams are sent to the receiving end through different transmission channels. At the receiving end, the received MDC streams are integrated by the integrator and provided to the decoder. The decoder uses the trained self-learning network to decode the integrated coded stream to obtain a decoded audio signal. Here, due to changes in the communication conditions of the transmission channel, at least a part or all of the multiple MDC streams transmitted through the transmission channel may be received by the receiving end and integrated by the integrator to form an integrated coded stream. The self-learning network of the trained decoder decodes the integrated coded stream. Refer to the above Figure 3 and Figure 4 As described, since the codec is trained for different integrated coding streams under the same constraints, the decoder can restore the audio signal more accurately under different channel conditions, thereby realizing the advantage of multi-description coding in coping with harsh channel conditions. Figure 6 The decoder shown is Figure 3 Corresponding, that is, using one decoder to decode the integrated coded stream in all cases, and Figure 7 The decoder shown is Figure 5 Correspondingly, that is, two decoders (ie, a central decoder and a bypass decoder) are used, one decoder processes the integrated coded stream when the entire coded stream is received, and the other decoder processes the integrated coded stream when the partial coded stream is received.

[0096] Figure 8 A flowchart of an audio processing method according to an exemplary embodiment of the present disclosure is shown.

[0097] like Figure 8 As shown, first, in step S810, the audio signal is encoded using a self-learning network. Here, the self-learning network can be as described above with reference to Figure 2-Figure 5 The self-learning network obtained by training the training method described above.

[0098] Next, in step S820, the encoded audio signal is divided into N encoding streams, where N is an integer greater than 1. That is, N may be the number of divided encoding streams used by the trained self-learning network.

[0099] Then, in step S830, the N coded streams are quantized to generate N audio coded streams, and the generated N audio coded streams can be respectively sent to the receiving end through a transmission channel for decoding.

[0100] According to an exemplary embodiment of the present disclosure, encoding an audio signal using a self-learning network-based encoder may include reducing the number of samples of an audio frame of the audio signal to a predetermined dimension.

[0101] According to an exemplary embodiment of the present disclosure, dividing the encoded audio signal into N encoding streams may include dividing the reduced-dimensional audio signal in one of the following ways: allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams evenly and without overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams evenly and with overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams unevenly and without overlapping; allocating the samples of the audio frames of the reduced-dimensional audio signal to the N encoding streams unevenly and with overlapping.

[0102] The dimensionality reduction process of step S820 is similar to Figure 2 The dimensionality reduction process of S220 is basically the same, so it will not be repeated here.

[0103] Fig. 9 A flowchart of an audio processing method according to an exemplary embodiment of the present disclosure is shown.

[0104] like Fig. 9 As shown, first, in step S910, an audio coding stream is received from a transmitting end. Here, the received audio coding stream may be at least one of N audio coding streams transmitted from the transmitting end. Here, the N audio coding streams may be generated by encoding an audio signal by a self-learning network for encoding and segmenting the encoded audio signal, where N is an integer greater than 1. Figure 2-Figure 5 The encoding and segmentation processes have been described and will not be repeated here.

[0105] Next, in step S920, the received audio coding stream is integrated. Figure 2-Figure 5 The integration process has been explained and will not be repeated here.

[0106] Finally, in step S930, the integrated audio coding stream can be decoded by the self-learning network for decoding, wherein the self-learning network for encoding and the self-learning network for decoding described above are corresponding deep neural networks and are obtained by referring to the above. Figure 2-Figure 5 The network is trained using the described training method.

[0107] According to an exemplary embodiment of the present disclosure, a self-learning network for decoding includes a first decoder and a second decoder, and the decoding of the integrated audio coding stream includes: in response to determining that the number of included coding streams is the N, using the first decoder to decode the integrated audio coding stream; in response to determining that the number of included coding streams is less than the N, using the second decoder to decode the integrated audio coding stream.

[0108] Fig.10 is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment of the present disclosure.

[0109] like Fig.10 As shown, the audio processing device 1000 includes: an encoding unit 1010, a dividing unit 1020, a selecting unit 1030, an integrating unit 1040, a decoding unit 1050 and a training unit 1060.

[0110] The encoding unit 1010 is configured to encode the input audio signal through a self-learning network for encoding.

[0111] The dividing unit 1020 is configured to divide the encoded audio signal into N encoded streams, where N is an integer greater than 1.

[0112] The selection unit 1030 is configured to select n coded streams from the N coded streams in a predetermined manner and set them to zero, where n is any integer from 0 to N-1. For example, a random selection module can be introduced to randomly set the N coded streams to zero using a preset random pattern.

[0113] The integration unit 1040 is configured to quantize and integrate the remaining coding streams after the N coding streams are set to zero to generate an integrated coding stream.

[0114] The decoding unit 1050 is configured to decode the integrated encoded stream through a self-learning network for decoding to obtain a decoded audio signal.

[0115] The training unit 1060 is configured to compare the decoded audio signal with the input audio signal and adjust the self-learning network for encoding in the encoding unit and the self-learning network for decoding in the decoding unit according to the comparison result.

[0116] According to an exemplary embodiment of the present disclosure, the encoding unit 1010 is configured to reduce the number of samples of an audio frame of an input audio signal to a predetermined dimension through a self-learning network for encoding, wherein the dimension of the reduced audio signal is such that a deviation between a signal obtained after decoding the reduced audio signal and the input audio signal is less than or equal to a preset threshold.

[0117] According to an exemplary embodiment of the present disclosure, the division unit 1020 is configured to divide the audio signal after dimensionality reduction in one of the following ways: allocating the samples of the audio frame of the audio signal after dimensionality reduction to the N coding streams evenly and without overlapping; allocating the samples of the audio frame of the audio signal after dimensionality reduction to the N coding streams evenly and with overlapping; allocating the samples of the audio frame of the audio signal after dimensionality reduction to the N coding streams unevenly and without overlapping; allocating the samples of the audio frame of the audio signal after dimensionality reduction to the N coding streams unevenly and with overlapping.

[0118] According to an exemplary embodiment of the present disclosure, the decoding unit 1050 may include a first decoder and a second decoder, wherein the decoding unit 1050 may be configured to: determine the number of coding streams included in the integrated coding stream; in response to determining that the number of coding streams included in the integrated coding stream is the N, the first decoder decodes the integrated coding stream; in response to determining that the number of coding streams included in the integrated coding stream is less than the N, the second decoder decodes the integrated coding stream.

[0119] According to an exemplary embodiment of the present disclosure, the self-learning network for encoding and the self-learning network for decoding have corresponding network structures, and the training unit 1060 is configured to adjust the self-learning network for encoding and the self-learning network for decoding through a loss function, wherein the loss function includes N deviation values ​​between the decoded audio signal and the input audio signal and a weight corresponding to each deviation value when the decoding unit receives 1, 2, ..., N encoding streams respectively.

[0120] According to an exemplary embodiment of the present disclosure, the deviation value between the decoded audio signal and the input audio signal includes one of the following indicator values: mean square error MSE in the time domain or frequency domain, mean absolute error MAE in the time domain or frequency domain, mean square logarithmic error MSLE in the time domain or frequency domain, objective speech quality assessment PESQ or objective perceptual listening quality assessment POLQA.

[0121] The above has been referred to Figure 2-Figure 5 The operation of each unit of the training device has been described in detail and will not be repeated here.

[0122] Fig.11 is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment of the present disclosure.

[0123] like Fig.11 As shown, the audio processing device 1100 includes: an encoding unit 1110 , a dividing unit 1120 and a quantization unit 1130 .

[0124] The encoding unit 1110 is configured to encode the audio signal based on a self-learning network.

[0125] The dividing unit 1120 is configured to divide the encoded audio signal into N encoded streams, where N is an integer greater than 1.

[0126] The quantization unit 1130 is configured to quantize the N coded streams to generate N audio coded streams, wherein the self-learning network is obtained by referring to the above Figure 2-Figure 5 The network is trained using the training method.

[0127] According to an exemplary embodiment of the present disclosure, the encoding unit 1110 is configured to reduce the number of samples of an audio frame of an input audio signal to a predetermined dimension through a self-learning network.

[0128] According to an exemplary embodiment of the present disclosure, the division unit 1120 is configured to divide the audio signal after dimensionality reduction in one of the following ways: allocating the samples of the audio frame of the audio signal after dimensionality reduction to the N coding streams evenly and without overlapping; allocating the samples of the audio frame of the audio signal after dimensionality reduction to the N coding streams evenly and with overlapping; allocating the samples of the audio frame of the audio signal after dimensionality reduction to the N coding streams unevenly and without overlapping; allocating the samples of the audio frame of the audio signal after dimensionality reduction to the N coding streams unevenly and with overlapping.

[0129] The operations performed by the encoding unit 1110, the dividing unit 1120 and the quantizing unit 1130 are the same as those in the above reference. Figure 2-Figure 5 The encoding, segmentation and quantization processes described are basically the same and will not be repeated here.

[0130] Fig.12 is a block diagram illustrating an audio processing apparatus according to an exemplary embodiment of the present disclosure.

[0131] like Fig.12 As shown, the audio processing device 1200 includes: a receiving unit 1210 , an integrating unit 1220 and a decoding unit 1230 .

[0132] The receiving unit 1210 is configured to receive an audio encoding stream from a transmitting end.

[0133] The integration unit 1220 is configured to integrate the received audio encoding stream.

[0134] The decoding unit 1230 is configured to decode the integrated audio encoding stream through a self-learning network for decoding.

[0135] Here, the received audio coding stream is at least one of the N audio coding streams sent from the sending end. The sending end may adopt the above reference Figure 2-Figure 5The N coded streams are generated in the manner described above, that is, the N audio coded streams may be generated by encoding an audio signal by a self-learning network for encoding corresponding to a self-learning network for decoding and segmenting the encoded audio signal. According to an exemplary embodiment of the present disclosure, the self-learning network for encoding and the self-learning network for decoding are generated by referring to Figure 2-Figure 5 The network is trained using the training method described above.

[0136] According to an exemplary embodiment of the present disclosure, the decoding unit 1230 may include a first audio decoder and a second audio decoder, and the decoding unit 1230 may be configured to: determine the number of coding streams included in the integrated coding stream; in response to determining that the number of audio coding streams included in the integrated audio coding stream is the N, use the first audio decoder to decode the integrated audio coding stream; in response to determining that the number of audio coding streams included in the integrated audio coding stream is less than the N, use the second audio decoder to decode the integrated audio coding stream.

[0137] Fig.13 1 is a block diagram showing a structure of an electronic device 1300 for training an audio codec and / or for audio processing according to an exemplary embodiment of the present disclosure. The electronic device 1300 may be, for example, a smart phone, a tablet computer, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. The electronic device 1300 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0138] Typically, the electronic device 1300 includes a processor 1301 and a memory 1302 .

[0139] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0140] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one instruction, which is used to be executed by the processor 1301 to implement the method of the present disclosure.

[0141] In some embodiments, the electronic device 1300 may further optionally include: a peripheral device interface 1303 and at least one peripheral device. The processor 1301, the memory 1302 and the peripheral device interface 1303 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1304, a touch display screen 1305, a camera 1306, an audio circuit 1307, a positioning component 1308 and a power supply 1309.

[0142] The peripheral device interface 1303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302, and the peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302, and the peripheral device interface 1303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0143] The radio frequency circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1304 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1304 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1304 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0144] The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1305 is a touch display screen, the display screen 1305 also has the ability to collect touch signals on the surface or above the surface of the display screen 1305. The touch signal can be input to the processor 1301 as a control signal for processing. At this time, the display screen 1305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1305 can be one, arranged on the front panel of the electronic device 1300; in other embodiments, the display screen 1305 can be at least two, respectively arranged on different surfaces of the electronic device 1300 or in a folding design; in some further embodiments, the display screen 1305 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device 1300. Even, the display screen 1305 can also be arranged as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1305 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0145] The camera assembly 1306 is used to capture images or videos. Optionally, the camera assembly 1306 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1306 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0146] The audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1301 for processing, or input them into the radio frequency circuit 1304 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 1300. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1307 may also include a headphone jack.

[0147] The positioning component 1308 is used to locate the current geographic location of the electronic device 1300 to implement navigation or LBS (Location Based Service). The positioning component 1308 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system, or the European Union's Galileo system.

[0148] The power supply 1309 is used to power various components in the electronic device 1300. The power supply 1309 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1309 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0149] In some embodiments, the electronic device 1300 further includes one or more sensors 910. The one or more sensors 1310 include, but are not limited to: an acceleration sensor 1311, a gyroscope sensor 1312, a pressure sensor 1313, a fingerprint sensor 1314, an optical sensor 1315, and a proximity sensor 1316.

[0150] The acceleration sensor 1311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the electronic device 1300. For example, the acceleration sensor 1311 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1301 can control the touch display screen 1305 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1311. The acceleration sensor 1311 can also be used for collecting game or user motion data.

[0151] The gyro sensor 1312 can detect the body direction and rotation angle of the electronic device 1300, and the gyro sensor 1312 can cooperate with the acceleration sensor 1311 to collect the user's 3D actions on the electronic device 1300. The processor 1301 can implement the following functions based on the data collected by the gyro sensor 1312: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0152] The pressure sensor 1313 can be set on the side frame of the electronic device 1300 and / or the lower layer of the touch display screen 1305. When the pressure sensor 1313 is set on the side frame of the electronic device 1300, it can detect the user's holding signal of the electronic device 1300, and the processor 1301 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1313. When the pressure sensor 1313 is set on the lower layer of the touch display screen 1305, the processor 1301 controls the operability controls on the UI according to the user's pressure operation on the touch display screen 1305. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0153] The fingerprint sensor 1314 is used to collect the user's fingerprint, and the processor 1301 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1314, or the fingerprint sensor 1314 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1301 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1314 can be set on the front, back, or side of the electronic device 1300. When a physical button or a manufacturer logo is set on the electronic device 1300, the fingerprint sensor 1314 can be integrated with the physical button or the manufacturer logo.

[0154] The optical sensor 1315 is used to collect the ambient light intensity. In one embodiment, the processor 1301 can control the display brightness of the touch display screen 1305 according to the ambient light intensity collected by the optical sensor 1315. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1305 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1305 is reduced. In another embodiment, the processor 1301 can also dynamically adjust the shooting parameters of the camera assembly 1306 according to the ambient light intensity collected by the optical sensor 1315.

[0155] The proximity sensor 1316, also called a distance sensor, is usually arranged on the front panel of the electronic device 1300. The proximity sensor 1316 is used to collect the distance between the user and the front of the electronic device 1300. In one embodiment, when the proximity sensor 1316 detects that the distance between the user and the front of the electronic device 1300 is gradually decreasing, the processor 1301 controls the touch display screen 1305 to switch from the screen-on state to the screen-off state; when the proximity sensor 1316 detects that the distance between the user and the front of the electronic device 1300 is gradually increasing, the processor 1301 controls the touch display screen 1305 to switch from the screen-off state to the screen-on state.

[0156] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the electronic device 1300, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0157] Fig.14 FIG. 1 is a block diagram of another electronic device 1400. For example, the electronic device 1400 may be provided as a server. Fig.14 , the electronic device 1400 includes one or more processing processors 1410 and a memory 1420. The memory 1420 may include one or more programs for executing the above audio codec training method and / or audio processing method. The electronic device 1400 may also include a power supply component 1430 configured to perform power management of the electronic device 1400, a wired or wireless network interface 1440 configured to connect the electronic device 1400 to a network and an input / output (I / O) interface 1450. The electronic device 1400 may operate based on an operating system stored in the memory 1420, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM or the like.

[0158] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the audio codec training method and / or the audio processing method according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0159] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to implement the above-mentioned audio codec training method and / or audio processing method.

[0160] According to the audio processing method, device, corresponding electronic device and storage medium disclosed in the present invention, a self-learning speech codec based on multiple descriptions is provided, which can constrain the original audio signal information contained in multiple coding streams through self-learning, can cope with the jitter and packet loss of the transmission channel, and can achieve higher sound quality at a lower bit rate.

[0161] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0162] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for audio processing, characterized in that: include: encoding an input audio signal by a self-learning network for encoding; Dividing the encoded audio signal into N encoded streams, where N is an integer greater than 1; Selecting n coded streams from the N coded streams in a preset manner and setting them to zero, wherein n is any integer from 0 to N-1; quantizing and integrating the remaining coded streams after the N coded streams are set to zero to generate an integrated coded stream; Decoding the integrated coded stream by a self-learning network for decoding to obtain a decoded audio signal; The decoded audio signal is compared with the input audio signal and the self-learning network for encoding and the self-learning network for decoding are adjusted according to the comparison result. The step of comparing the decoded audio signal with the input audio signal and adjusting the audio encoder and the self-learning network for decoding according to the comparison result includes: The loss function is used to adjust the audio encoder and the self-learning network for decoding. The loss function includes N deviation values ​​between the audio signal obtained by decoding and the input audio signal when 1, 2, ..., N coded streams are received respectively, and a weight corresponding to each deviation value.

2. The method according to claim 1, characterized in that The step of encoding the input audio signal by using the self-learning network for encoding comprises: reducing the number of samples of the audio frame of the input audio signal to a predetermined dimension by using the self-learning network for encoding, The dimension of the audio signal after dimension reduction is such that a deviation between a signal obtained after decoding the audio signal after dimension reduction and the input audio signal is less than or equal to a preset threshold.

3. The method according to claim 2, characterized in that The dividing the encoded audio signal into N encoded streams includes dividing the audio signal after dimension reduction in one of the following ways: Allocating the sample points of the audio frame of the audio signal after dimension reduction to the N coded streams evenly and without overlap; Allocating the sample points of the audio frame of the audio signal after the dimension reduction to the N coded streams in an even and overlapping manner; Allocating the sample points of the audio frame of the audio signal after the dimension reduction to the N coding streams unequally and without overlap; The sample points of the audio frame of the audio signal after the dimension reduction are allocated to the N encoding streams unevenly and with overlap.

4. The method according to claim 1, characterized in that The self-learning network for decoding includes a first decoder and a second decoder, wherein decoding the integrated coded stream to obtain a decoded audio signal includes: Determining the number of code streams included in the integrated code stream; In response to determining that the number of the included coded streams is the N, decoding the integrated coded stream using a first decoder; In response to determining that the number of included coded streams is less than the N, the integrated coded stream is decoded using a second decoder.

5. The method according to claim 1, characterized in that The self-learning network used for encoding and the self-learning network used for decoding have corresponding network structures.

6. The method according to claim 5, characterized in that The deviation value between the audio signal obtained by decoding and the input audio signal includes one of the following index values: Mean square error MSE in time domain or frequency domain, mean absolute error MAE in time domain or frequency domain, mean square logarithmic error MSLE in time domain or frequency domain, objective speech quality assessment PESQ or objective perceptual listening quality assessment POLQA.

7. An audio processing method, characterized in that: include: encoding an input audio signal using a self-learning network for encoding; Dividing the encoded audio signal into N encoded streams, where N is an integer greater than 1; quantizing the N coded streams to generate N audio coded streams, The self-learning network is a network obtained by the method described in any one of claims 1 to 6.

8. The audio processing method according to claim 7, characterized in that: Encoding an audio signal using an encoder of a self-learning network for encoding includes: reducing the number of samples of an audio frame of the audio signal to a predetermined dimension, The dimension of the audio signal after dimension reduction is such that a deviation between a signal obtained after decoding the audio signal after dimension reduction and the input audio signal is less than or equal to a preset threshold.

9. The audio processing method according to claim 8, characterized in that: Dividing the encoded audio signal into N encoded streams includes dividing the reduced-dimensional audio signal in one of the following ways: Allocating the sample points of the audio frame of the audio signal after dimension reduction to the N coded streams evenly and without overlap; Allocating the sample points of the audio frame of the audio signal after the dimension reduction to the N coded streams in an even and overlapping manner; Allocating the sample points of the audio frame of the audio signal after the dimension reduction to the N coding streams unequally and without overlap; The sample points of the audio frame of the audio signal after the dimension reduction are allocated to the N encoding streams unevenly and with overlap.

10. An audio processing method, characterized in that: include: Receive an audio encoding stream from a transmitter; Integrate the received audio encoding stream; Decode the combined audio codec stream using a self-learning network for decoding. The received audio coding stream is at least one of N audio coding streams sent from the sending end, where N is an integer greater than 1. The N audio coding streams are generated by encoding an audio signal using a self-learning network for encoding corresponding to the self-learning network for decoding and segmenting the encoded audio signal, The self-learning network for encoding and the self-learning network for decoding are networks obtained by the method according to any one of claims 1 to 6.

11. The audio processing method according to claim 10, characterized in that: The self-learning network for decoding includes a first decoder and a second decoder, and decoding the integrated audio coding stream includes: Determining the number of code streams included in the integrated code stream; In response to determining that the number of the included coded streams is the N, decoding the integrated audio coded stream using a first decoder; In response to determining that the number of included coded streams is less than the N, the integrated audio coded stream is decoded using a second decoder.

12. An audio processing device, characterized in that: include: an encoding unit configured to encode an input audio signal through a self-learning network for encoding; A dividing unit, configured to divide the encoded audio signal into N encoded streams, where N is an integer greater than 1; A selection unit is configured to select n coded streams from the N coded streams in a preset manner and set them to zero, where n is any integer from 0 to N-1; an integration unit, configured to quantize and integrate the remaining code streams after the N code streams are set to zero to generate an integrated code stream; A decoding unit configured to decode the integrated coded stream through a self-learning network for decoding to obtain a decoded audio signal; a training unit configured to compare the decoded audio signal with the input audio signal and adjust the self-learning network for encoding in the encoding unit and the self-learning network for decoding in the decoding unit according to the comparison result, The training unit is configured to adjust the self-learning network for encoding and the self-learning network for decoding through a loss function, The loss function includes N deviation values ​​between the audio signal decoded and the input audio signal when the decoding unit receives 1, 2, ..., N coded streams respectively, and a weight corresponding to each value deviation.

13. The device according to claim 12, characterized in that The encoding unit is configured to reduce the number of samples of the audio frame of the input audio signal to a predetermined dimension through a self-learning network for encoding, The dimension of the audio signal after dimension reduction is such that a deviation between a signal obtained after decoding the audio signal after dimension reduction and the input audio signal is less than or equal to a preset threshold.

14. The device according to claim 12, characterized in that The division unit is configured to divide the audio signal after dimension reduction in one of the following ways: Allocating the sample points of the audio frame of the audio signal after dimension reduction to the N coded streams evenly and without overlap; Allocating the sample points of the audio frame of the audio signal after the dimension reduction to the N coded streams in an even and overlapping manner; Allocating the sample points of the audio frame of the audio signal after the dimension reduction to the N coding streams unequally and without overlap; The sample points of the audio frame of the audio signal after the dimension reduction are allocated to the N encoding streams unevenly and with overlap.

15. The device according to claim 12, characterized in that The decoding unit includes a first decoder and a second decoder, and the decoding unit is configured to: Determining the number of code streams included in the integrated code stream; In response to determining that the number of the included coded streams is the N, decoding the integrated coded stream using a first decoder; In response to determining that the number of included coded streams is less than the N, the integrated coded stream is decoded using a second decoder.

16. The device according to claim 12, characterized in that The self-learning network used for encoding and the self-learning network used for decoding have corresponding network structures.

17. The device according to claim 16, characterized in that The deviation value between the audio signal obtained by decoding and the input audio signal includes one of the following index values: Mean square error MSE in time domain or frequency domain, mean absolute error MAE in time domain or frequency domain, mean square logarithmic error MSLE in time domain or frequency domain, objective speech quality assessment PESQ or objective perceptual listening quality assessment POLQA.

18. An audio processing device, characterized in that: include: An encoding unit configured to encode the audio signal using a self-learning network based on encoding; A division unit, configured to divide the encoded audio signal into N encoded streams, where N is an integer greater than 1; a quantization unit, quantizing the N coded streams to generate N audio coded streams, Wherein, the self-learning network for encoding is a network obtained by the method according to any one of claims 1-6.

19. The audio processing device according to claim 18, characterized in that: The encoding unit is configured to reduce the number of samples of the audio frame of the input audio signal to a predetermined dimension through the self-learning network for encoding.

20. The audio processing device according to claim 18, characterized in that: The division unit is configured to divide the audio signal after dimensionality reduction in one of the following ways: Allocating the sample points of the audio frame of the audio signal after dimension reduction to the N coded streams evenly and without overlap; Allocating the sample points of the audio frame of the audio signal after the dimension reduction to the N coded streams in an even and overlapping manner; Allocating the sample points of the audio frame of the audio signal after the dimension reduction to the N coding streams unequally and without overlap; The sample points of the audio frame of the audio signal after the dimension reduction are allocated to the N encoding streams unevenly and with overlap.

21. An audio processing device, characterized in that: include: A receiving unit, configured to receive an audio encoding stream from a transmitting end; An integration unit, configured to integrate the received audio encoding stream; A decoding unit configured to decode the integrated audio coding stream by a self-learning network for decoding, The received audio coding stream is at least one of N audio coding streams sent from a transmitter, and the N audio coding streams are generated by encoding an audio signal and segmenting the encoded audio signal by a self-learning network for encoding corresponding to the self-learning network for decoding, and the self-learning network for encoding and the self-learning network for decoding are networks obtained by the method described in any one of claims 1 to 6.

22. The audio processing device according to claim 21, characterized in that: The decoding unit includes a first decoder and a second decoder, and the decoding unit is configured to: Determining the number of code streams included in the integrated code stream; In response to determining that the number of the included coded streams is the N, decoding the integrated coded stream using a first decoder; In response to determining that the number of included coded streams is less than the N, the integrated coded stream is decoded using a second decoder.

23. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, they cause the at least one processor to execute the audio processing method described in any one of claims 1 to 6, the audio processing method described in any one of claims 7-9 and / or the audio processing method described in any one of claims 10-11.

24. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by at least one processor, enables the at least one processor to execute the audio processing method described in any one of claims 1 to 6, the audio processing method described in any one of claims 7-9 and / or the audio processing method described in any one of claims 10-11.

25. A computer program product, characterized in that The instructions in the computer program product are executed by at least one processor to perform the audio processing method according to any one of claims 1 to 6, the audio processing method according to any one of claims 7-9 and / or the audio processing method according to any one of claims 10-11.

Images