Adaptive block switching with deep neural networks

Abstract

The invention relates to a method for predicting transform coefficients representing frequency content of an adaptive block length media signal by receiving a frame and receiving block length information indicating a number of quantized transform coefficients per block in the frame, the number of quantized transform coefficients being one of a first number or a second number, wherein the first number is larger than the second number; determining that a first block has the second number of quantized transform coefficients; converting the first block into a converted block having the first number of quantized transform coefficients; adjusting a main neural network trained to predict at least one output variable given at least one adjustment variable, the at least one adjustment variable being based on information about the converted block and the block length information of the first block; providing at least one predicted transform coefficient from an output stage of the main neural network.

Description

[0001] Cross-references to related applications

[0002] This application claims the priority of U.S. Provisional Patent Application No. 63 / 092,685, filed October 16, 2020, and European Patent Application No. 20206462.2, filed November 9, 2020, both of which are incorporated herein by reference. Technical Field

[0003] This invention relates to combining generative models with existing efficient encoding and decoding schemes for media signals. Specifically, this invention relates to a method for predicting transform coefficients of adaptive block-length media signals using a trained neural network. Background Technology

[0004] In low-rate adaptive block-length encoding and decoding, the encoder is configured to optimize the trade-off between frequency and time resolution. This is achieved by having the encoder select the transform length for each signal sample block. Typically, the encoder will select long blocks with a higher number of transform coefficients for signal sample blocks representing signals with slow-evolving time characteristics, and a set of short blocks with a lower number of transform coefficients for signal sample blocks representing signals with fast-evolving time characteristics.

[0005] The problem with encoding and decoding adaptive block-length signals is that the blocks to be decoded may include different numbers of transform coefficients, which represent the frequency content of the media signal at different durations. Therefore, adaptive block lengths are incompatible with conventional decoding schemes developed for fixed block-length signals. Furthermore, it is beneficial to obtain a more accurate representation of the original media signal in the decoder, which has already been sampled in the encoder to form signal sample blocks and adaptively divided into blocks with different numbers of transform coefficients. Summary of the Invention

[0006] Based on the above, one object of the present invention is to provide a method for predicting transform coefficients of adaptive block-length media signals, and in particular adaptive block-length general audio signals, using a neural network.

[0007] According to a first aspect of the present invention, a method is provided for predicting transform coefficients representing frequency content of a media signal with an adaptive block length using a computer-implemented neural network system. The method includes: receiving blocks of a frame, each block of the frame including at least one quantized transform coefficient (or a set of quantized transform coefficients) representing a partial time segment of the media signal; receiving block length information indicating the number of quantized transform coefficients in each block of the frame, the number of quantized transform coefficients being one of a first number or a second number, wherein the first number is greater than the second number; determining that at least a first block of the frame has a second number of quantized transform coefficients; converting at least the first block into a converted block having a first number of quantized transform coefficients; adjusting a main neural network trained to predict at least one output variable given at least one adjustment variable based on adjustment information including a representation of the converted block and a representation of the block length information of the first block; and providing the at least one output variable to an output stage (output neural network) configured to provide at least one predicted transform coefficient from the at least one output variable.

[0008] As an alternative to quantized transform coefficients, the transform coefficients may be distorted or impaired. The transform coefficients output by the output stage (output neural network) are enhanced in a sense, meaning they more closely resemble a set of original transform coefficients, and / or the enhanced transform coefficients, inversely transformed to the time domain, describe the media signal, which is perceived as a higher quality media signal compared to the time-domain media signal described by the quantized transform coefficients. Furthermore, a frame as mentioned herein may include one or more blocks (e.g., a set of blocks).

[0009] This invention is at least partly based on the understanding that the generative properties of a trained main neural network can be incorporated into variable block switching decoding by transforming a (short) first block into a (long) transformed block with a first number of transform coefficients. Since the neural network has a fixed dimension in its output layer, it is incompatible with adaptive length blocks. By transforming a first block with quantized transform coefficients into a transformed block, and by conditioning the main neural network using the representation of the transformed block and the representation of block length information, the neural network can dynamically predict at least one (enhanced or unquantized) transform coefficient based on the block length. That is, since the representation of block length information is included in the conditioning information (based on at least one conditioning variable), the main neural network will be trained to appropriately respond to blocks that have been transformed into blocks including the first number of transform coefficients.

[0010] Additionally, it can be determined that a block of a frame includes a first number of quantized transform coefficients. Such a (long) block may not be converted into a transformed block, but rather a representation of a block having the first number of quantized transform coefficients is included in the adjustment information. Long blocks can be processed similarly to the determined short blocks, except that they are not converted. The transform coefficients output by the output stage include a first number of transform coefficients, which represent either a quantized transform coefficient block having the first number of transform coefficients or a transformed block having the first number of quantized transform coefficients, which in turn represent at least one quantized transform coefficient block having a second number of transform coefficients.

[0011] Because the main neural network can predict at least one transform coefficient for each variable-length block sequentially, it takes into account time and / or frequency dependencies. The main neural network can have memory capabilities, allowing previous inputs to influence current processing and ensuring that the prediction of the current (enhanced) at least one transform coefficient is influenced by previous transform coefficients.

[0012] Adaptive length blocks represent a trade-off between frequency and time. Longer blocks include more transform coefficients and will represent a longer duration of the media signal, while shorter blocks include fewer transform coefficients and will represent a shorter duration of the media signal.

[0013] According to a second aspect of the invention, a method is provided for obtaining at least one training block for training a computer-implemented neural network system to predict at least one transform coefficient of an adaptive block-length media signal. The method includes: obtaining a set of transform blocks, each transform block including a plurality of transform coefficients representing frequency content of the media signal, the number of transform coefficients in each block being a first number or a second number, wherein the first number is greater than the second number; determining that a first block includes a second number of transform coefficients; converting the first block into a converted block having a first number of transform coefficients; obtaining a target prediction block from the converted block; quantizing the converted block; and obtaining a training block from the quantized converted block.

[0014] The resulting set of transform blocks can be further represented as a series of associated time-domain window functions (short, long, bridged input, or bridged output).

[0015] According to a third aspect of the invention, a computer-implemented neural network system is provided for predicting at least one transform coefficient representing the frequency content of an adaptive block-length media signal. The neural network system includes an adaptive block preprocessing unit configured to: receive a frame comprising a set of quantized transform coefficients representing partial time segments of a media signal; receive block length information indicating the number of quantized transform coefficients for each block in the frame, the number of quantized transform coefficients being one of a first number or a second number, wherein the first number is greater than the second number; determine that at least a first block has a second number of transform coefficients; and convert the at least first block into a converted block having a first number of quantized transform coefficients. The neural network system further includes: a main neural network trained to predict at least one output variable given at least one modulating variable based on modulating information, the modulating information including a representation of the converted block and a representation of the block length information of the first block; and an output stage configured to provide at least one predicted transform coefficient from the at least one output variable.

[0016] In some implementations, the neural network system described above has been trained using a set of target prediction blocks and a set of training blocks. The set of training blocks is a damaged representation of the target prediction blocks, and each training block includes at least one training block with a first number of transform coefficients and at least one training block with a second number of transform coefficients. The set of training blocks is provided to an adaptive block preprocessing unit of the neural network system, and a set of prediction blocks is obtained from the output stage of the neural network system based on the set of training blocks. A metric of the prediction blocks relative to the set of target prediction blocks is calculated, and the weights of the neural network system are modified to reduce the metric.

[0017] By modifying the weights of the neural network system in response to a metric of the predicted block, training enables the neural network system to learn to predict (generate) at least one transform coefficient from at least one quantized transform coefficient. The training also enables the neural network system to learn to correctly identify at least one modulating variable representing (or a plurality of) short blocks and to process the modulating variable in such a way that the resulting at least one predicted transform coefficient closely resembles at least one transform coefficient of the media signal.

[0018] It should be understood that, based on the obtained metric, the neural network system can be trained by modifying the parameters (e.g., weights) of each neural network, preferably iteratively trained until a satisfactory small metric is obtained.

[0019] The inventions according to the second and third aspects are characterized by embodiments and benefits identical or equivalent to those according to the first aspect. Furthermore, any functionality described with respect to the method may have corresponding structural features in a system or code used to perform these functions in a computer program product.

[0020] Experiments have been conducted on encoding and decoding reference media signals using fixed block length and adaptive block length. In the case of fixed block length, a fixed-length neural network system is implemented in the decoder, and in the case of adaptive block length, a neural network system according to an embodiment of the present invention is implemented in the decoder. Fixed-length encoding uses 256 MDCT coefficient blocks, and adaptive block length encoding uses adaptive 256 / 128 MDCT coefficient blocks. When comparing the decoded signals, the adaptive block length switching using the neural network system of the present invention in the decoder shows a reduction in front echo distortion compared to the corresponding portion with fixed block length. Attached Figure Description

[0021] The invention will be described in more detail with reference to the accompanying drawings, which illustrate the presently preferred embodiments of the invention.

[0022] Figure 1 An adaptive block length encoder and decoder for implementing a neural network system according to an embodiment of the present invention are shown.

[0023] Figure 2 A neural network system according to an embodiment of the present invention is shown.

[0024] Figures 3a to 3b The process of merging time window functions is shown.

[0025] Figure 4 A flowchart illustrating a method for predicting at least one transform coefficient from quantized transform coefficients according to an embodiment of the present invention is shown.

[0026] Figure 5 A flowchart illustrating a method for obtaining training blocks for training a neural network system according to an embodiment of the present invention is shown.

[0027] Figure 6 A flowchart illustrating a method for obtaining training blocks for training a neural network system according to an embodiment of the present invention is shown. Detailed Implementation

[0028] Figure 1An adaptive block-length encoder / decoder system comprising encoder 1 and decoder 2 is depicted. A media signal is received at the input port of transient detector 101. The media signal can be divided into a series of time-domain frames and further divided into multiple time-domain segments, each segment comprising multiple media signal samples. For example, a time-domain frame comprises 16,000 signal samples and is divided into four segments with 4,000 samples each. The number of signal samples in the time-domain frames and segments (and thus the number of segments in the time-domain frames) is merely exemplary and can be any number. Transient detector 101 is configured to optimize the trade-off between frequency and time resolution for each segment by selecting a transform length. Generally, transient detector 101 selects a long transform length for segments containing signals with slowly evolving or stable time characteristics and a shorter transform length for segments containing signals with rapidly evolving time characteristics. This approach provides a fundamental advantage over encoding with time-invariant transform lengths by optimizing the "perceptual coding gain" of short and long signal classes.

[0029] Depending on the temporal characteristics of the media signal segment, the transient detector 101 may choose to request that the segment be represented by a transform domain block having a first number of transform coefficients (for slowly evolving time signal segments) or by multiple transform domain blocks each including a second number of transform coefficients (for rapidly evolving time signal segments), wherein the first number is greater than the second number. For example, the transient detector 101 may request representation using 256 transform coefficients X. k To represent slowly evolving segments, two sets (transform domain blocks) of 128 transform coefficients X are used. k Or four groups of 64 transformation coefficients X k This is used to represent rapidly evolving segments. The number of transform coefficients selected is not limited to the included examples and can be any number. The transient detector 101 can request multiple transform coefficients from a set of block lengths, wherein the set of block lengths includes at least two lengths, such as 256 / 128. In some embodiments, the set of block lengths includes at least three or more lengths, such as 256 / 128 / 64, from which the transient detector 101 can select an appropriate length for the block. For example, the transient detector 101 can request that a segment be represented by a combination of short blocks of different lengths. For example, a slowly evolving segment is represented by 256 transform coefficients X. k This indicates that the next rapidly evolving segment is composed of a segment with 128 transformation coefficients X. k The block and two X with 64 transformation coefficients k The block representation. The transient detector 101 generates block length information, which indicates the number of requested transform domain blocks (and / or the transform coefficients X of each block). kThe number of time-domain segments (i.e., the number of transform-domain blocks) should be used to represent the time-domain segments. The block length information is transmitted to the decoder 2. The transient detector 101 transmits the block length information to the transform unit 102.

[0030] Transformation unit 102 transforms the segment according to the block length information and includes the transformation coefficient X. k The adaptive length transform block is output to quantizer 103. For the example mentioned above, the 16000-sample time frame, which has been divided into four 4000-sample segments, is transformed into a series of transform coefficients X with 256, 256, 128, 128, and 256 respectively. k The transform blocks are then used to form transform-domain frames (frames) in encoder 1 and / or decoder 2. In other words, a frame can be referred to as a group of one or more transform blocks and / or one or more segments. In parts of encoder 1 and decoder 2, the frame to which a transform block belongs may not be explicitly indicated or considered, as it is sufficient to process the transform blocks continuously without considering their corresponding temporal or transform-domain frames.

[0031] The received media signal is further received by a perception model 111 that calculates a masking threshold. The masking threshold is then passed to a bit allocation unit 112.

[0032] In bit allocation unit 112, bits are assigned to the transform coefficients to be quantized based on the received perceptual masking threshold information received from perceptual model 111. Bit allocation unit 112 can allocate bits to reduce or minimize quantization noise. Bit allocation unit 112 then passes the bit allocation information to quantizer 103.

[0033] Quantizer 103 quantizes the transform coefficients X of each block in the adaptive block length block by allocating bits to each transform coefficient according to the received bit allocation information. k To form quantized transformation coefficients Block. Quantizer 103 will include quantized transform coefficients. The adaptive block length is transmitted to decoder 2.

[0034] In decoder 2, neural network (NN) system 201 receives frames from quantizer 103 of encoder 1, wherein each block of the frame includes at least one quantized transform coefficient. And block length information is received from the transient detector 101 of the encoder. The neural network system 201 includes a main neural network and an output stage (e.g., an output neural network), the output stage being trained to receive data from quantized transform coefficients. Predict at least one transformation coefficient (at least one predicted transformation coefficient) The transformation stage of the neural network system 201 will have a second number of quantized transformation coefficients. The block is converted into a first number of quantized transform coefficients. The transformed block. In some embodiments, the transform-level neural network system 201 only transmits a first number of quantized transform coefficients. Therefore, the output stage of the neural network system 201 can output a sequence of static-length blocks (e.g., each static-length block includes a first number of predicted transform coefficients). ), wherein some blocks represent quantized blocks of the same length, and wherein some blocks represent at least one (and in some embodiments more than one) short blocks of different (shorter) lengths.

[0035] At least one predicted transform coefficient is received at the inverse transform unit 202. The inverse transform unit is configured to transform at least one predicted transform coefficient of each transform domain block. The transformation is converted into a time-domain segment (i.e., a predicted time-domain segment). In some embodiments, the inverse transformation unit 202 may receive block length information from the transient detector 101 of the encoder 1.

[0036] As described above, at least one predicted transformation coefficient is used to reach the inverse transformation unit 202 from the block. It can have a statically predetermined length, although some blocks represent one or more original (before conversion) shorter quantized blocks. When the inverse transform unit 202 receives information about the original transform domain block length in the form of block length information, the inverse transform unit 202 can take necessary pre-inverse transform processing steps. For example, in response to the association of a predicted long block with an original short block that has been upsampled in the transform unit to form a transformed block, the inverse transform unit 202 can downsample the predicted long block into a predicted short block before inverse transforming the short block to the time domain. In another example, at least two blocks have quantized transform coefficients. The short blocks are transformed into a single transformed block in the transformation unit and predicted by the neural network system to have at least one predicted transformation coefficient. The single long block. In this case, the inverse transform unit 202 can determine, based on the block length information, that the predicted long block is actually a prediction based on at least two short blocks (which have been combined), and in response, perform a pre-inverse transform processing step, such as segmentation or performing an inverse transform process, i.e., the inverse of the transform performed in the neural network system 201, to obtain a predicted block of the same length as determined by the transient detector 101 in the encoder 1. The pre-inverse transform processing step can be performed by a separate unit (not shown) preceding the inverse transform unit of some pre-existing coding scheme for adaptive block length media signals. For example, the neural network system (along with the pre-inverse transform processing) can be implemented with any existing codec, for example to refine AC-4 transform coefficients, or used with a new codec designed for decoding with the neural network system 201.

[0037] In yet another embodiment, the inverse transform unit 202 transforms each predicted block (having a static length) to the time domain, as if the set of predicted blocks came from a static-length media signal. In this embodiment, the inverse transform unit does not need to consider block length information, and the neural network system efficiently converts the adaptive block-switching media signal into a static block-length media signal. The neural network system 201 receives blocks of different lengths and is trained to output fixed-length blocks. The inverse transform unit 202 transforms the static-length blocks into a time-domain media signal.

[0038] The inverse transform unit 202 outputs a time-domain media signal (or a sequence of time-domain media signal blocks) suitable for playback by a playback device (not shown). The neural network system 201 is configured to receive at least one quantized transform coefficient in the block and predict at least one transform coefficient.

[0039] refer to Figure 2 It describes in more detail Figure 1 An embodiment of a neural network system 201 implemented in a computer. The neural network system 201 is configured to receive a set of adaptive length blocks 20, each adaptive length block including a set of quantized transform coefficients representing the frequency content of a partial time segment of a media signal. The block length information 21 indicates the number of quantized transform coefficients in each block of frame 20, wherein the number of quantized transform coefficients is one of a first number or a second number. The computer-implemented neural network system 201 further includes a conversion stage 11 configured to determine that at least a first block has a second number of quantized transform coefficients and to convert the at least first block into a converted block having a first number of quantized transform coefficients. From frame 20 to conversion stage 11, wherein frame 20 has at least one block having a second number of quantized transform coefficients, the conversion stage generates an output frame 20', wherein all output blocks in the output frame have a first number of quantized transform coefficients.

[0040] The neural network system 201 further receives block length information 21 indicating the number of quantized transform coefficients in each block of frame 20. The block length information 21 thereby indicates a sequence of blocks including a first number or a second number of transform coefficients. The block length information 21 can be a sequence of integers or symbols, each integer or symbol representing a block, and the value of each integer (or the type of the symbol) representing the quantized transform coefficients of that block. The quantity.

[0041] Block length information 21 may include more than two alternative block lengths. In some embodiments, this includes a second number of transformation coefficients X. k The block preceding the first number of transformation coefficients X k The block can be labeled as a bridged input block, and has a second number of transform coefficients X. k The block following the first number of transformation coefficients X k The block can be marked as a bridged output block. Therefore, the block length information 21 can be a sequence of four (or more) distinct integers, one integer for the long block (the first number of transform coefficients X). k ), short blocks (with a second number of transformation coefficients X) k Each of the bridged input block and bridged output block.

[0042] The neural network system 201 forms at least one regulation variable 15 based on regulation information, wherein the regulation information includes at least two components: (i) information representing the transformed block (or a block including a first number of quantized transform coefficients), and (ii) information representing block length information 21. In a simple case, the information representing the transformed block is the quantized transform coefficients. The block length information is represented as an integer. At least one modulating variable 15 and the main neural network 16 can be characterized by a separate dimension for each modulating information or by a single dimension on which each modulating information is projected.

[0043] At least one adjustment variable 15 is used to adjust the main neural network 16. The main neural network 16 is trained to predict at least one output variable given at least one adjustment variable 15, and said at least one output variable is provided to an output neural network 17, which is trained to make a final prediction of at least one transform coefficient given at least one output variable from the main neural network 16 (i.e., output at least one predicted transform coefficient). The output neural network 17 may include one or more hidden layers.

[0044] The main neural network 16 can be any type of neural network, such as a deep neural network, a recurrent neural network, or any neural network system. The main neural network 16 can be a regression model. The media signal can be any type of media signal, including audio or video signals. In the case where the media signal is an audio signal, the main neural network 16, in a preferred embodiment, acts as a general audio generation model in the transform domain. The main neural network 16 is configured to operate in the transform domain and is trained to predict at least one output variable given at least one modulating variable. The at least one output variable can be considered a hidden state and is provided to an output neural network 17, wherein the output neural network 17 is configured (e.g., trained) to output at least one predicted transform coefficient given the at least one output variable. The output neural network 17 can be implemented as a single unit together with the main neural network 16, for example, as the output stage of the main neural network 16 or as a separate neural network. In any case, the output neural network 17 and the main neural network 16 exchange hidden state information.

[0045] Therefore, the main neural network 16 and the output neural network 17 capture the temporal and / or frequency correlations of the representation of the quantized transform coefficients from at least one quantized transform coefficient. Predict at least one transformation coefficient In other words, the main neural network 16 and the output neural network 17 can be trained such that previous representations of the transform coefficients already processed by the main neural network 16 can influence the prediction of at least one transform coefficient. Alternatively or additionally, the main neural network 16 and the output neural network 17 are trained to consider the interdependencies between transform coefficients in the current block and past blocks. Because transform coefficients represent frequency content, the main neural network 16 and the output neural network 17 can be trained to predict at least one transform coefficient by learning how the frequency content of the first frequency band (represented in the transform coefficients) affects the frequency content of the second frequency band.

[0046] In some embodiments, the neural network system 201 further includes an additional neural network, such as a conditioning neural network 12, which is connected to receive the output from the conversion unit 11 and block length information from the block length information neural network 14. The conditioning neural network 12 and the block length information neural network 14 are used to predict the corresponding conditioning information and can be any type of neural network, such as a convolutional layer, and using one type does not require the other.

[0047] The regulation neural network 12 and / or the block length information neural network 14 can be trained to predict at least one corresponding output variable, wherein the at least one regulation variable 15 is then obtained as the sum of the at least one predicted output variable. Further, the at least one regulation variable 15 passed to the main neural network 16 (e.g., the sum of the at least one corresponding output variable from the regulation neural network 12 and the block length information neural network 14) can be considered as a hidden neural network layer. The neural network system 201 can be operated (and trained) without any constraints on the interpretability of the hidden layers, except for establishing an internal dimension (as a hyperparameter) for the hidden layers that matches the input dimension of the main neural network 16. For example, the representations of regulation information and block length information representing quantized transform coefficients can each be at least one output variable, which is in the shape of a matrix having a dimension matching the internal dimension. Then, the at least one regulation variable 15 can be the sum of at least one matrix output variable. In another example, the matrix is ​​two-dimensional and includes a single row or column (i.e., a vector).

[0048] The neural network 12 is trained to adjust the quantized transform coefficients of a given block. In the case of predicting the representation of the block from the output frame 20', by predicting the representation of the quantized transform coefficients of the transformed block, and utilizing the adjustment neural network 12 trained to predict the representation, a representation that further facilitates the main neural network 16 in making predictions can be achieved. The quantized transform coefficients are... The assignment transforms them into representations of quantized transformation coefficients. In contrast to the static transformation function of the information, the regulating neural network 15 can be trained to predict the representation that facilitates the main neural network 16 and the output neural network 17 to make the final prediction.

[0049] Similarly, the block length information neural network 14 is trained to predict a representation of the block length information given block length information 21. By implementing the block length neural network 14 trained to predict a representation of the block length information given block length information 21 of at least the first block, the conditioning information for adjusting the main neural network 16 will facilitate the prediction of at least one transform coefficient by both the main neural network 16 and the output neural network 17. The format carries an indication of the quantized transform coefficients in the first block. The information pertains to the quantity of transform coefficients X. In one example, the block length neural network 14 outputs a representation of block length information, which indicates the number of transform coefficients X. k The block. Therefore, when the represented quantized transform coefficients From the transformed block or from the transform coefficients having a first number During the quantization of the blocks, the main neural network 16 is differentially tuned and will respond differentially. Since the main neural network 16 and the output neural network 17 have been trained together with the conversion unit 11 to represent the quantized transform coefficients... The information can be used to predict at least one transform coefficient, so that the prediction of at least one transform coefficient can be completed regardless of how the transform block is constructed from at least the first block.

[0050] In contrast to using integers from, for example, a sequence of integers to regulate a block-length neural network, some implementations of the neural network system 201 include a one-hot encoder 13 that converts block-length information 21 into one-hot vectors, which are then used to regulate the block-length neural network 14. The block-length information is categorized and indicates a separate state for each block (e.g., long, short, bridging input, or bridging output). Using one-hot encoding, these categories are divided into separate vector elements that facilitate the training and prediction of the block-length neural network 14 by clearly distinguishing the different possible states. For example, one-hot encoding facilitates a strong spatial dependency between at least one predicted output variable and the input elements of the input layer of the block-length neural network that receives the one-hot (on-state) vector elements.

[0051] In some implementations, the neural network system 201 further receives the perceptual model coefficients pEnvQ and / or spectral envelope for each block. Therefore, the regulation information may further include additional information as a representation of the perceptual model coefficient pEnvQ information and / or spectral envelope information. The perceptual model coefficients pEnvQ and / or spectral envelope may be processed in parallel with block length information and quantized transform coefficients, and combined with other information in at least one regulation variable 15, or provided to the main neural network 16 as auxiliary information in a separate dimension.

[0052] The set of sensing model coefficients pEnvQ can be obtained from a sensing model, such as the sensing model that appears in the encoder. The sensing model coefficients pEnvQ are calculated for each frequency band and are preferably mapped to the same resolution as the frequency coefficients of the block for ease of processing.

[0053] In implementations where a single short block has been converted into a transformed block, the pEnvQ coefficients are converted into an equivalent long block representation through a similar conversion process and used as conditioning information. For example, if a short block is upsampled, the pEnvQ coefficients are upsampled in the same way.

[0054] Note that for the neural network system 201 "trained" in an implementation featuring more than one neural network, all neural networks in the system are trained together during at least a portion of the training period. For example, block-length neural network 14 may be trained together with main neural network 16, wherein the internal parameters (e.g., weights) of each neural network 14, 16 are modified to match the original non-quantized transform coefficients X. k Compared to at least one transformation coefficient of some target predictions, optimize at least one transformation coefficient of the prediction. A certain metric. Then, the block-length neural network 14 is trained to output at least one adjustment variable 15, said at least one adjustment variable making the predicted at least one transform coefficient of the main neural network 16 and the output neural network 17 as close as possible to the original transform coefficients. The main neural network 16 and the output neural network 17 are simultaneously trained to predict as close as possible to the original transform coefficients X. k at least one transformation coefficient

[0055] Transforming a block having a second number of transform coefficients in transformation unit 11 may involve transforming a block having a first number of quantized transform coefficients. The block upsampling is a transformed block. Upsampling may include linear or polynomial interpolation (and optionally extrapolation) of a second number of quantized transform coefficients to a first number of quantized transform coefficients. Alternatively, upsampling to form a transformed block may include one of the following: repeating each quantized transform coefficient a predetermined number of times, adding zero elements between non-zero elements, or interleaving the quantized transform coefficients. Alternatively, any other suitable upsampling, expansion, or interpolation technique is applicable. In some implementations, the conversion unit 11 only converts the quantized transform coefficients of the block. The parameters are forwarded to the main neural network 16, which is trained to predict at least one output parameter of the output neural network 17. In this case, the main neural network 16 will learn to recognize parameters with a second number of quantized transform coefficients. The blocks are absorbed by training the converter's functionality.

[0056] The conversion unit 11 will include a second number of quantized transformation coefficients. The first block is transformed into at least two blocks (a first block and a second block, each block including a second number of quantized transform coefficients). An alternative to this is to convert the first and second blocks together into a system containing a first number of quantized transformation coefficients. The transformed blocks. Therefore, the main neural network 16 and the output neural network 17 can be trained to operate on a given set of quantized transform coefficients. Predict at least one transform coefficient in the case of the representation of the transformed block. Among them, the quantized transformation coefficients of the transformation block Quantized transformation coefficients derived from at least the first and second blocks

[0057] Typically, it has a second number of quantized transformation coefficients. At least the first and second blocks can have a second number of quantized transformation coefficients. N consecutive blocks are given, where the first number is N times the second number. These N consecutive blocks can then be transformed into blocks with the first number of quantized transform coefficients. The converted block. The adaptive block switching media signal may, for example, include a first number of quantized transform coefficients equal to 256. The second quantity, summing to 128, is N = 2. The first quantity, summing to 256 and N = 4, will result in four short blocks being converted into one transformed block, each containing 64 quantized transform coefficients. In another example, N=8, when the first number of transform coefficients is 1024, the quantized transform coefficients... The second number is 128.

[0058] Converting at least the first block and the second block into a converted block may include linking at least the first block and the second block into a converted block. Linking is an efficient and easy-to-implement method for converting at least the first block and the second block into a converted block.

[0059] In some implementations, the conversion unit 11 receives a representation of the corresponding time-domain window function for each block, wherein the window functions of the first block and the second block partially overlap.

[0060] Window functions can be used with quantized transform coefficients Alternatively, the block length information 21 (passed to the conversion unit 11) may be received together. Or, the window function may be constructed based on the block length information 21 (passed to the conversion unit 11). Alternatively, it may be constructed using the quantized transform coefficients in the block. The number of quantized transform coefficients in blocks within transformation unit 11 is determined by the correlation between the number of quantized transform coefficients and the sequence of blocks in each block having at least a first number and a second number of quantized transform coefficients, in order to construct the window function. For example, the first number of quantized transform coefficients... The block is associated with a long window function and has a second number of transform quantized coefficients. The block is associated with a short window function. In another example, if there is a first number of quantized transform coefficients... The block has a second number of quantized transform coefficients If a block is specified before another block, then that block can be associated with a bridging input window function.

[0061] exist Figure 2 In this context, all functions and units described upstream of the (optional) adjustment neural network 12 and (optional) block length information neural network 14 can be referred to as preprocessing units or adaptive block preprocessing units. The preprocessing unit can therefore be a multiple-input multiple / single-output unit, for example, receiving block length information 12 and quantized transform coefficients. The output represents the quantized transformation coefficients. The information, and the block length information 12 is represented as a single piece of information (at least one variable) or a combination of information (at least one variable).

[0062] Further reference Figure 6 The flowchart depicts a method for training a neural network system, for example. Figure 2 The embodiment depicted in the diagram. At S311, a set of adaptive-length target prediction (true) blocks is provided. This occurs in conjunction with providing a set of training blocks at S321 as a damaged representation (e.g., a quantized representation) of the target prediction blocks. The target prediction blocks comprise a set of non-quantized transform coefficients X. k Training blocks are provided to the neural network system 201 and processed to obtain a set of prediction blocks at S331. This is achieved by using transform coefficients that include at least one prediction. The output prediction block is compared with the target prediction block, and a similarity metric is obtained at S332. This metric can be an error metric, where a low error metric indicates a high level of similarity. The metric can be a negative likelihood, such as negative log-likelihood (NLL), where a low metric indicates a high level of similarity. The metric can also be mean absolute error (MAE) or mean squared error (MSE), where a high level of similarity will be indicated by a low MAE or MSE. At S333, the metric is used to modify the weights of the neural network system 201 to reduce or minimize the metric.

[0063] In one example, the metric is referred to as a loss function or "loss" because it is directly calculated as NLL, i.e.

[0064]

[0065] When calculating NLL loss, at least one transformation coefficient is predicted. Transformation coefficients predicted by at least one It is represented by at least one distribution parameter. Therefore, the NLL function is applied to represent at least one transformation coefficient of the prediction. At least one distribution parameter. The transformation coefficient of at least one distribution parameter for at least one prediction. The probability distribution is parameterized.

[0066] In other implementations, the loss is calculated as MSE according to the following:

[0067]

[0068] Alternatively, the loss can be calculated as MAE based on the following:

[0069]

[0070] When calculating the MSE and MAE losses, at least one predicted transformation coefficient is used. This is how it is used.

[0071] In some cases, a prediction block can be represented by a single predicted transformed block to represent more than one training block (and associated target prediction block). In this case, the prediction block can be inversely transformed into blocks corresponding to the training block (and associated target prediction block) respectively, so that the metric can be computed.

[0072] refer to Figure 3a The figure illustrates a series of time-domain window functions 31, 32a, 32b, and 33. Figure 3a The diagram illustrates a typical 2:1 block length switching window sequence. A first long window 31 is followed by two short windows 32a and 32b, then a second long window 33. The short time-domain window functions 32a and 32b can overlap by 50%, where the sum of the squares of the short window functions results in a value of one for the overlapping portion. Furthermore, the sum of the squares of each window function 31, 32a, 32b, and 33 results in a value of one for each overlap.

[0073] In some implementations, long windows 31 and 33 may further bridge the input window 31 and the output window 33, respectively, particularly suitable before and after the short windows 32a and 32b. The window functions 31, 32a, 32b, and 33 overlap at least partially in time. Each window function 31, 32a, 32b, and 33 is associated with a set of transform coefficient blocks, namely a long transform coefficient block with long window functions 31 and 33 and a short transform coefficient block with short window functions 32a and 32b.

[0074] In some additional implementations, where the number of transformation coefficients in each block is more than one of the two alternatives (e.g., one of 256, 128, and 64 coefficients as described above), the bridging input window 31 and bridging output window 33 functions may include more than two bridging window functions, for example, one for each type of transformation between variable-length blocks. If the block length is one of 256, 128, and 64, the input and output bridging window functions can be defined for each of the following: 256 to 128, 256 to 64, and 128 to 64.

[0075] Further reference Figure 3b The diagram illustrates a long transformation window 32 (with associated long transformation blocks), which is the transformation result of two short window functions 32a and 32b (and two short transformation coefficient blocks).

[0076] By quantizing the transform coefficients of the first and second (short) blocks (the window functions corresponding to these transform coefficients in...) Figure 3a (Shown as 32a and 32b) the inverse transform is performed back to a windowed time-domain representation, and these transform coefficients can be combined into a long transform block. This can be achieved by overlapping and adding the windowed time-domain representations of the first and second blocks and transforming the overlapping and added time-domain representations of the first and second blocks into a transformed block with a first number of quantized transform coefficients.

[0077] For example, if the transform coefficients are modified Discrete Cosine Transform (MDCT) coefficients, the intermediate short blocks (associated with window functions 32a, 32b) can be merged into a single long block by transforming the MDCT into short time-domain segments and overlapping these segments. The transform coefficients of the equivalent transformed long block 32 with a flat-top window can then be computed using DCT type 4. The window sequence following this merge / transformation operation is... Figure 3b As shown in the figure. It is further noted that this conversion process can be completed while maintaining the perfect reconstruction properties of the transform coefficients (in the absence of quantization).

[0078] refer to Figure 4The diagram illustrates a flowchart of a method for predicting at least one transform coefficient from quantized transform coefficients according to an embodiment of the present invention. At S111, a neural network system receives a frame including quantized transform coefficients. At S112, the neural network system determines that at least one block of the frame includes a second number of transform coefficients, and at S113 continues by converting the block having at least the second number of transform coefficients into a converted block having a first number of transform coefficients. Information representing the quantized transform coefficients of the converted block is a piece of information used to adjust at least one regulation variable of the main neural network at S131 based on this information. Optionally, the method involves adjusting the regulation neural network at S114 using the information representing the quantized transform coefficients of the converted block, and adjusting the main neural network at S131 using at least one output variable of the regulation neural network.

[0079] Furthermore, the method involves receiving block length information at S121. The representation of the block length information is used as a piece of information for regulating the main neural network at S131. Optionally, the block length information is used to first regulate the block length neural network at S123, wherein at least one output variable predicted by the block length neural network is used to regulate the main neural network at S131. Moreover, some embodiments include one-hot encoding of the block length information at S122, wherein the one-hot encoded block length information is used to regulate the block length neural network at S123, or as part of the information used to regulate the main neural network at S131.

[0080] At S131, the main neural network predicts at least one output variable given at least one adjustment variable, and wherein, at S132, at least one output variable is provided to the output stage (e.g., the output neural network). The output stage predicts at least one transformation coefficient at S132.

[0081] Figure 5 A flowchart illustrating a method for obtaining training blocks (training blocks for input and target prediction blocks for comparison with output) according to an embodiment of the present invention is shown, the training blocks being used to train a neural network system for predicting transform coefficients of media signals with adaptive block lengths. At S211, a set of transform blocks is obtained. For example, a batch of waveforms or media signals has been divided into a set of time-domain segments (e.g., forming time-domain frames), and each time-domain segment has been transformed into a set of transform blocks of different lengths (e.g., transform-domain frames). Alternatively, a batch of waveforms or media signals has been processed using a transient detector as described above to determine the length of each block. At S212, a first block is determined to include a second number of transform coefficients, and this block is converted into a transformed block having a first number of transform coefficients at S213. At S221, a target prediction block is obtained. The target prediction block obtained at S221 may be the transformed block itself.

[0082] At S231, the transformed block is quantized to form a quantized block. That is, the quantized block does not represent the complete information initially present in the determined first block; therefore, the quantized block can be referred to as a damaged block, and the neural network should learn to use the damaged block to predict undamaged blocks. At S232, a training block is obtained from the quantized block obtained at S231. The training block can be such a quantized block. In some implementations, additional steps are included, such as using a target training block as input to the neural network during training and using a target prediction block as training data.

[0083] The block identified as including the first number of transformation coefficients can be processed similarly to obtain the training block and the target prediction block, wherein step S213 is omitted.

[0084] In some implementations, the media signal or a batch of waveforms is processed by a transient detector that determines the transform length as described above. Therefore, the set of transform blocks will contain all different types of block and window functions.

[0085] ***

[0086] The foregoing has described possible methods for training and operating a deep learning-based system for determining an indication of the audio quality of an input audio sample, as well as possible implementations of such a system. Additionally, this disclosure relates to apparatus for implementing these methods. Examples of such apparatus may include a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof) and memory coupled to the processor. The processor may be adapted to perform some or all of the steps of the methods described throughout the disclosure.

[0087] The apparatus may be a server computer, client computer, personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, smartphone, web device, network router, switch or bridge, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by said apparatus. Furthermore, this disclosure will relate to any collection of apparatuses that individually or in combination execute instructions to perform any one or more methods discussed herein.

[0088] This disclosure further relates to a program (e.g., a computer program) that includes instructions that, when executed by a processor, cause the processor to perform some or all of the steps of the methods described herein.

[0089] Furthermore, this disclosure relates to computer-readable (or machine-readable) storage media for storing the aforementioned programs. Here, the term "computer-readable storage medium" includes, but is not limited to, data storage libraries in the form of, for example, solid-state memory, optical media, and magnetic media.

[0090] Unless otherwise specifically stated, it is obvious from the following discussion that, throughout this public discussion, terms such as “processing,” “computing,” “calculating,” “determining,” and “analyzing” are used to refer to the actions and / or processes by which data represented as physical (e.g., electronic) quantities are manipulated and / or transformed into other data similarly represented as physical quantities by a computer or computing system or similar electronic computing device.

[0091] In a similar manner, the term "processor" can refer to any device or part of a device that processes electronic data, for example, from registers and / or memory, to transform that electronic data into other electronic data, for example, that can be stored in registers and / or memory. "Computer," "computing machine," or "computing platform" can include one or more processors.

[0092] In one example embodiment, the methods described herein can be executed by one or more processors that accept computer-readable (also known as machine-readable) code containing a set of instructions that, when executed by the one or more processors, perform at least one of the methods described herein. This includes any processor capable of executing a set of instructions (sequential or otherwise) specifying an action to be taken. Thus, an example is a typical processing system comprising one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem comprising main RAM and / or static RAM and / or ROM. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system in which processors are coupled together via a network. If the processing system requires a display, it may include such a display, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data input is required, the processing system also includes one or more input devices, such as alphanumeric input units (e.g., a keyboard), pointing control devices (e.g., a mouse), etc. The processing system may also encompass storage systems such as disk drive units. In some configurations, the processing system may include sound output devices and network interface devices. The memory subsystem therefore includes a computer-readable carrier medium carrying computer-readable code (e.g., software) comprising a set of instructions that, when executed by one or more processors, cause one or more of the methods described herein to be performed. It should be noted that when the methods comprise several elements (e.g., several steps), no particular order of these elements is implied unless specifically stated otherwise. During the execution of software by a computer system, the software may reside on a hard disk, or it may reside wholly or at least partially in RAM and / or a processor. Therefore, the memory and processor also constitute a computer-readable carrier medium carrying computer-readable code. Furthermore, the computer-readable carrier medium may be formed or included in a computer program product.

[0093] In alternative example embodiments, one or more processors may operate as standalone devices or may be connected to (e.g., networked to) other processors in a networked deployment. These processors may operate as server or user machines in a server-user network environment, or as peer-to-peer machines in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), cellular phone, web facility, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) specifying the actions to be taken by that machine.

[0094] It should be noted that the term "machine" should also be considered to include any set of machines that individually or jointly execute a set (or more sets) of instructions to perform any or more of the methods discussed herein.

[0095] Therefore, an example embodiment of each method described herein takes the form of a computer-readable carrier medium carrying a set of instructions, such as a computer program for execution on one or more processors (e.g., one or more processors as part of a web server arrangement). Thus, as those skilled in the art will recognize, example embodiments of this disclosure can be embodied as methods, apparatus such as dedicated devices, apparatus such as data processing systems, or computer-readable carrier media (e.g., computer program products). A computer-readable carrier medium carries computer-readable code comprising a set of instructions that, when executed on one or more processors, cause one or more processors to implement the method. Therefore, aspects of this disclosure can take the form of methods, entirely hardware example embodiments, entirely software example embodiments, or example embodiments combining software and hardware aspects. Furthermore, this disclosure can take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

[0096] Software can be further sent or received over a network via a network interface device. While the carrier medium is a single medium in the example embodiment, the term "carrier medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of instructions. The term "carrier medium" should also be considered to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more processors and causing one or more processors to perform any one or more methods of this disclosure. The carrier medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs, magnetic disks, and magneto-optical discs. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing a bus subsystem. Transmission media can also take the form of acoustic or optical waves, such as those generated during radio wave and infrared data communication. For example, the term "carrier medium" should therefore be considered to include, but is not limited to, solid-state storage; computer products embodied in optical and magnetic media; media carrying propagation signals that can be detected by at least one or more processors and represent a set of instructions, which, when executed, implement a method; and transmission media in a network that carry propagation signals that can be detected by at least one of one or more processors and represent the set of instructions.

[0097] It will be understood that, in one exemplary embodiment, the steps of the method under discussion are performed by a suitable processor (or processors) in a processing (e.g., a computer) system that executes instructions (computer-readable code) stored in a storage device. It will also be understood that this disclosure is not limited to any particular implementation or programming technique, and that this disclosure can be implemented using any suitable technique for implementing the functions described herein. This disclosure is not limited to any particular programming language or operating system.

[0098] Throughout this disclosure, references to “one example embodiment,” “some example embodiments,” or “example embodiment” mean that a particular feature, structure, or characteristic described in connection with an example embodiment is included in at least one example embodiment of this disclosure. Therefore, the phrases “in one example embodiment,” “in some example embodiments,” or “in an example embodiment” appearing throughout this disclosure do not necessarily refer to the same example embodiment. Furthermore, in one or more example embodiments, particular features, structures, or characteristics may be combined in any suitable manner, as will be apparent to those skilled in the art based on this disclosure.

[0099] As used herein, unless otherwise specified, ordinal adjectives such as “first,” “second,” “third,” etc., are used to describe common objects only to indicate different instances of similar objects and are not intended to imply that the objects described must be in a given order in time, space, hierarchy, or any other way.

[0100] In the claims below and in the description herein, the terms *comprising*, *comprised of*, or *which comprises* are open-ended terms meaning that at least the following element / feature is included, but not excluding other elements / features. Therefore, when the term *comprising* is used in a claim, it should not be construed as limited to the means, elements, or steps listed thereafter. For example, the expression of a device including A and B should not be limited to a device that includes only elements A and B. As used herein, the terms *including*, *which includes*, or *that includes* are also open-ended terms meaning that at least the element / feature following the term is included, but not excluding other elements / features. Therefore, *including* is synonymous with *comprising* and means *comprising*.

[0101] It should be recognized that in the foregoing description of exemplary embodiments of this disclosure, various features of this disclosure are sometimes combined in a single exemplary embodiment, figure, or description thereof in order to simplify the disclosure and aid in understanding one or more of the inventive aspects. However, the approach of this disclosure should not be construed as reflecting an intention to require more features than expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in fewer than all features of a single foregoingly disclosed exemplary embodiment. Therefore, the claims following this specification are hereby expressly incorporated, wherein each claim is an independent, separate exemplary embodiment of this disclosure.

[0102] Furthermore, while some of the exemplary embodiments described herein include features that are included in other exemplary embodiments but not others, as those skilled in the art will understand, combinations of features from different exemplary embodiments are intended to be within the scope of this disclosure and to form different exemplary embodiments. For example, any exemplary embodiment of the claimed embodiments in the appended claims can be used in any combination.

[0103] Numerous specific details are set forth in the description provided herein. However, it should be understood that exemplary embodiments of this disclosure may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail to avoid obscuring the understanding of this specification.

[0104] Therefore, although the mode considered to be the best mode of this disclosure has been described, those skilled in the art will recognize that other and further modifications can be made thereto without departing from the spirit of this disclosure, and all such changes and modifications falling within the scope of this disclosure are intended to be claimed. For example, any formulas given above merely represent processes that can be used. Functions can be added or removed from the block diagram, and operations can be interchanged between functional blocks. Steps can be added or removed from the methods described within the scope of this disclosure.

[0105] Various aspects of the invention can be understood from the following enumerated list of example embodiments (EEE):

[0106] EEE1. A method for predicting at least one transform coefficient representing the frequency content of an adaptive block-length media signal using a computer-implemented neural network system, the method comprising the steps of:

[0107] The received frame is a block, each block of which includes a set of quantized transform coefficients representing a partial time segment of the media signal.

[0108] Receive block length information indicating the number of quantized transform coefficients in each block of the frame, wherein the number of quantized transform coefficients is one of a first number or a second number, wherein the first number is greater than the second number.

[0109] It is determined that at least a first block of the frame has the second number of quantized transform coefficients.

[0110] At least the first block is converted into a transformed block having the first number of quantized transformation coefficients.

[0111] A main neural network is modulated, the main neural network being trained to predict at least one output variable given at least one modulated variable, the at least one modulated variable being based on modulating information, the modulating information including a representation of the transformed block and a representation of the block length information of the first block.

[0112] The at least one output variable is provided to an output stage, which is configured to provide at least one predicted transformation coefficient from the at least one output variable.

[0113] EEE2. The method as described in EEE1, further comprising receiving a set of perceptual model coefficients for each block of the frame, wherein the adjustment information further comprises the set of perceptual model coefficients.

[0114] EEE3. The method as described in EEE1, further comprising receiving the spectral envelope of each block in the frame, wherein the adjustment information further comprises the spectral envelope.

[0115] EEE4. The method as described in EEE 1, further comprising:

[0116] The block length neural network is adjusted using the representation of the block length information of the first block, and the block length neural network is trained to output the representation of the block length information of the first block given the block length information.

[0117] EEE5. The method as described in EEE 4, wherein adjusting the block length neural network with the block length information includes encoding the block length information into a one-hot vector and adjusting the block length neural network with the one-hot vector.

[0118] EEE6. The method as described in EEE1, further comprising the following steps:

[0119] The adjustment neural network is adjusted using the quantized transform coefficients of the transformed block, wherein the adjustment neural network is trained to output the representation of the transformed block given the quantized transform coefficients.

[0120] EEE7. The method as described in EEE 1, wherein converting at least the first block into the converted block includes upsampling the first block.

[0121] EEE8. The method as described in EEE 1, further comprising determining that at least the first block and the subsequent second block have the second number of transformation coefficients, and wherein converting at least the first block into the converted block comprises converting at least the first block and the second block into the converted block.

[0122] EEE9. The method according to any of the preceding EEEs, wherein the quantized transform coefficients representing the frequency content are discrete cosine transform (DCT) coefficients.

[0123] EEE10. The method according to any of the foregoing EEEs further comprises:

[0124] The inverse transform unit receives the predicted transform coefficients and the block length information.

[0125] The predicted transform coefficients are then transformed into time-domain signals.

[0126] EEE11. The method according to EEE 8, wherein the first quantity is N times the second quantity, and determining that at least the first block and the subsequent second block have the second quantity of quantized transformation coefficients includes

[0127] The N consecutive blocks of the frame are determined to have the second number of quantized transform coefficients.

[0128] EEE12. The method according to EEE 8, wherein converting at least the first block and the second block into the converted block includes linking at least the first block and the second block into the converted block.

[0129] EEE13. The method according to EEE 8, wherein receiving the block length information includes:

[0130] For each block of the frame, a representation of the corresponding temporal window function is received, wherein the window functions of the first block and the second block partially overlap.

[0131] EEE14. The method according to EEE 13, wherein converting at least the first block and the second block into the converted block comprises:

[0132] The quantized transform coefficients are inversely transformed into windowed time-domain representations of the first and second blocks.

[0133] The windowed temporal representations of the first and second blocks are overlapped and added together.

[0134] The time-domain representation of the overlapped summation of the first block and the second block is transformed into a transformed block having the first number of quantized transformation coefficients.

[0135] EEE15. A method for obtaining at least one training block, said at least one training block for training a computer-implemented neural network system to predict at least one transform coefficient of an adaptive block-length media signal, said method comprising:

[0136] A set of transform blocks is obtained, each transform block including multiple transform coefficients representing the frequency content of the media signal. The number of transform coefficients in each block is a first number or a second number, wherein the first number is greater than the second number.

[0137] The first block is determined to include the second number of transformation coefficients.

[0138] The first block is converted into a transformed block having the first number of transformation coefficients.

[0139] The target prediction block is obtained from the transformed block.

[0140] Quantize the converted block, and

[0141] The training block is obtained from the quantized block.

[0142] EEE16. A computer-implemented neural network system for predicting transform coefficients representing the frequency content of an adaptive block-length media signal, the neural network system comprising:

[0143] An adaptive block preprocessing unit, the adaptive block preprocessing unit being configured to:

[0144] - Receive a frame, the frame comprising a set of quantized transform coefficients representing a partial time segment of a media signal.

[0145] - Receive block length information indicating the number of quantized transform coefficients in each block of the frame, wherein the number of quantized transform coefficients is one of a first number or a second number, wherein the first number is greater than the second number.

[0146] - Determine that at least the first block has the second number of transformation coefficients, and

[0147] - Convert at least the first block into a transformed block having the first number of quantized transformation coefficients.

[0148] A main neural network, wherein the main neural network is trained to predict at least one output variable given at least one modulating variable based on modulating information, the modulating information including a representation of the transformed block and a representation of the block length information of the first block, and

[0149] An output stage, configured to provide at least one predicted transformation coefficient from the at least one output variable.

[0150] EEE17. A neural network decoder comprising a computer-implemented neural network system according to EEE 16.

[0151] EEE18. The neural network decoder according to EEE 17 further includes an inverse transform unit.

[0152] The inverse transform unit is configured to:

[0153] - Receive the at least one predicted transform coefficient and block length information, and

[0154] - Transform the at least one predicted transform coefficient into a time-domain signal.

[0155] EEE 19. A neural network system according to EEE 16, wherein the neural network system has been trained by the following operations:

[0156] Provide a set of target prediction blocks,

[0157] A set of training blocks is provided to the adaptive block preprocessing unit. This set of training blocks includes at least one training block with the first number of transform coefficients and at least one training block with the second number of transform coefficients. The set of training blocks is a damaged representation of the set of target prediction blocks.

[0158] A set of prediction blocks is obtained from the output stage based on the set of training blocks.

[0159] Calculate the metric of the set of predicted blocks relative to the set of target predicted blocks.

[0160] Modify the weights of the neural network system to reduce the metric.

[0161] EEE20. A neural network system according to EEE 19, wherein the metric is one of negative likelihood, mean squared error, or absolute error.

Claims

1. A method for predicting at least one transform coefficient representing the frequency content of an adaptive block-length media signal using a computer-implemented neural network system (201), the method comprising the steps of: Receive a frame comprising one or more blocks, each block of the frame comprising a set of quantized transform coefficients representing a portion of a time segment of the media signal. Receive block length information indicating the number of quantized transform coefficients in each block of the frame, wherein the number of quantized transform coefficients is one of a first number or a second number, wherein the first number is greater than the second number. It is determined that at least a first block of the frame has the second number of quantized transform coefficients. At least the first block is converted into a transformed block having the first number of quantized transformation coefficients. A modulated main neural network (16) is trained to predict at least one output variable given at least one modulated variable, the at least one modulated variable being based on modulated information including a representation of the transformed block and a representation of the block length information of the first block. The at least one output variable is provided to the output stage (16), which is configured to provide at least one predicted transformation coefficient from the at least one output variable.

2. The method of claim 1, further comprising receiving a set of perceptual model coefficients for each block of the frame, wherein, The adjustment information further includes the set of perception model coefficients.

3. The method of claim 1, further comprising receiving the spectral envelope of each block in the frame, wherein, The adjustment information further includes the spectral envelope.

4. The method according to any one of claims 1-3, further comprising: The block length neural network (14) is adjusted using the representation of the block length information of the first block, the block length neural network being trained to output the representation of the block length information of the first block given the block length information.

5. The method according to claim 4, wherein, Adjusting the block length neural network (14) with the block length information includes encoding the block length information into a one-hot vector and adjusting the block length neural network (14) with the one-hot vector.

6. The method according to any one of claims 1-3, further comprising the following steps: The adjustment neural network (12) is adjusted using the quantized transform coefficients of the transformed block, wherein the adjustment neural network (12) is trained to output the representation of the transformed block given the quantized transform coefficients.

7. The method according to any one of claims 1-3, wherein, Converting at least the first block into the converted block includes upsampling the first block.

8. The method according to any one of claims 1-3, wherein, The quantized transform coefficients representing the frequency content are Discrete Cosine Transform (DCT) coefficients.

9. The method according to any one of claims 1-3, further comprising: The inverse transform unit receives the predicted transform coefficients and the block length information. The predicted transform coefficients are then transformed into time-domain signals.

10. The method according to any one of claims 1-3, further comprising: Determine that at least the first block and the subsequent second block have the second number of transformation coefficients, and wherein converting at least the first block into the converted block includes converting at least the first block and the second block into the converted block.

11. The method according to claim 10, wherein, The first quantity is N times the second quantity, and determining that at least the first block and the subsequent second block have the second quantity of quantized transform coefficients includes: The N consecutive blocks of the frame are determined to have the second number of quantized transform coefficients.

12. The method according to claim 10, wherein, Converting at least the first block and the second block into the converted block includes linking at least the first block and the second block into a converted block.

13. The method according to claim 10, wherein, Receiving the block length information includes: For each block of the frame, a representation of the corresponding temporal window function (31, 32a, 32b) is received, wherein the window functions of the first block and the second block partially overlap.

14. The method according to claim 13, wherein, Converting at least the first block and the second block into the converted block includes: The quantized transform coefficients are inversely transformed into windowed time-domain representations of the first and second blocks. The windowed temporal representations of the first and second blocks are overlapped and added together. The time-domain representation of the overlapped summation of the first block and the second block is transformed into a transformed block having the first number of quantized transformation coefficients.

15. A computer-implemented neural network system (201) for predicting transform coefficients representing the frequency content of an adaptive block-length media signal, the neural network system (201) comprising: An adaptive block preprocessing unit, the adaptive block preprocessing unit being configured to: - Receive frames comprising one or more blocks, each block of the frame comprising a set of quantized transform coefficients representing a partial time segment of a media signal. - Receive block length information indicating the number of quantized transform coefficients in each block of the frame, wherein the number of quantized transform coefficients is one of a first number or a second number, wherein the first number is greater than the second number. - Determine that at least the first block has the second number of transformation coefficients, and - Convert at least the first block into a transformed block having the first number of quantized transformation coefficients. A main neural network (16), wherein the main neural network (16) is trained to predict at least one output variable given at least one modulating variable based on modulating information, the modulating information including a representation of the transformed block and a representation of the block length information of the first block, and Output stage (17), which is configured to provide at least one predicted transformation coefficient from the at least one output variable.

16. The neural network system (201) according to claim 15, wherein, The neural network system (201) has been trained through the following operations: Provide a set of target prediction blocks, A set of training blocks is provided to the adaptive block preprocessing unit. This set of training blocks includes at least one training block with the first number of transform coefficients and at least one training block with the second number of transform coefficients. The set of training blocks is a damaged representation of the set of target prediction blocks. A set of prediction blocks is obtained from the output stage based on the set of training blocks. Calculate the metric of the set of predicted blocks relative to the set of target predicted blocks. Modify the weights of the neural network system to reduce the metric.

17. The neural network system according to claim 16, wherein, The metric is one of negative likelihood, mean squared error, or absolute error.

18. A neural network decoder comprising a computer-implemented neural network system (201) according to any one of claims 15 to 17.

19. The neural network decoder of claim 18, further comprising an inverse transform unit, The inverse transform unit is configured to: - Receive the at least one predicted transform coefficient and block length information, and - Transform the at least one predicted transform coefficient into a time-domain signal.

20. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1-14.

Images