Adaptive block switching using deep neural networks

Abstract

To provide a method for predicting, with a neural network, transform coefficients of an adaptive block length media signal, in particular, an adaptive block length general audio signal.SOLUTION: The method includes the steps of: receiving a frame including a quantized transform coefficient; converting at least one block having a second number of transform coefficients into a converted block having a first number of transform coefficients, the first number being larger than the second number; conditioning a conditioning neural network using information representing quantized transform coefficients of the converted block; conditioning a main neural network using at least one output variable of the conditioning neural network; and predicting at least one transform coefficient using at least one output variable of the main neural network.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims priority to U.S. Provisional Patent Application No. 63 / 092,685, filed on 16 October 2020, and European Patent Application No. 20206462.2, filed on 9 November 2020, which are incorporated herein by reference.

[0002] [Technical field to which the review belongs] This invention relates to combining a generative model with existing highly efficient coding schemes for media signals. Specifically, this invention relates to a method for predicting conversion coefficients of adaptive block-length media signals using a trained neural network. [Background technology]

[0003] In low-rate adaptive block-length coding and decoding, the encoder is configured to optimize the trade-off between frequency and time resolution. This can be achieved by the encoder selecting a conversion length for each signal sample block. Generally, the encoder selects longer blocks with more conversion factors for signal sample blocks representing signals with slowly evolving time characteristics, and selects a set of shorter blocks, each with fewer conversion factors, for signal sample blocks representing signals with rapidly evolving time characteristics.

[0004] The problem with encoding and decoding adaptive block-length signals lies in the fact that the blocks to be decoded may contain a varying number of transformation coefficients representing the frequency content of the media signal over varying durations. Therefore, adaptive block lengths are incompatible with conventional decoding methods developed for fixed-block-length signals. Furthermore, it would be beneficial to obtain a more accurate representation in the decoder of the original media signal, which has been sampled in the encoder to form signal sample blocks and adaptively divided into blocks of varying numbers of transformation coefficients. [Overview of the Initiative]

[0005] Accordingly, based on the above, the object of the present invention is to provide a method for predicting the conversion coefficients of adaptive block length media signals, particularly adaptive block length general-purpose audio signals, using a neural network.

[0006] According to a first aspect of the present invention, a method is provided for predicting conversion coefficients representing the frequency content of an adaptive block-length media signal using a computer-implemented neural network system. The method includes: receiving a block of frames, wherein each block of the frame includes at least one quantized conversion coefficient (or set of quantized conversion coefficients) representing a partial time segment of a media signal; receiving block-length information indicating the number of quantized conversion coefficients for each block of the frame, wherein the number of quantized conversion coefficients is one of a first number or a second number, and the first number is greater than the second number; determining that at least a first block of the frame has a second number of quantized conversion coefficients; converting at least a first block into a converted block having a first number of quantized conversion coefficients; and conditioning a main neural network trained to predict at least one output variable given at least one conditioning variable, wherein the at least one conditioning variable is determined based on the conditioning information. The information includes a step of including a representation of the transformed block and a representation of block length information for the first block, and a step of providing at least one output variable to an output stage (output neural network) configured to provide at least one predicted transformation coefficient from at least one output variable.

[0007] As an alternative to quantized transformation coefficients, the transformation coefficients may be distorted or compromised. The transformation coefficients output by the output stage (output neural network) are enhanced in the sense that they closely resemble the original set of transformation coefficients, and / or that the enhanced transformation coefficients, when inversely transformed into the time domain, describe a media signal that is perceived as a higher quality media signal compared to the time domain media signal described by the quantized transformation coefficients. Furthermore, the frames referred to herein may consist of one or more blocks (e.g., a set of blocks).

[0008] The present invention is based, at least in part, on the understanding that the generative properties of a trained main neural network can be introduced into variable block switching decoding by transforming a (short) first block into a (long) transformed block with a first number of transformation coefficients. Since neural networks have a fixed dimension in their output layer, they are incompatible with adaptive length blocks. By transforming a first block of quantized transformation coefficients into a transformed block, and conditioning the main neural network using the representation of the transformed block and the representation of the block length information, the neural network can predict at least one (enhanced or unquantized) transformation coefficient in a dynamic manner based on the block length. That is, since the representation of the block length information is included in the conditioning information (on which at least one conditioning variable is based), the main neural network will be trained to respond appropriately to a block transformed to contain a first number of transformation coefficients.

[0009] Additionally, it may be further determined that a block of a frame contains a first number of quantized transformation coefficients. Such (long) blocks do not need to be converted into a converted block; instead, a representation of the block with the first number of quantized transformation coefficients is included in the conditional information. In addition to not converting long blocks, long blocks may be treated similarly to determined short blocks. The transformation coefficients output by the output stage contain a first number of transformation coefficients representing either a quantized transformation coefficient block with the first number of transformation coefficients, or a converted block of the first number of quantized transformation coefficients, which represents at least one quantized transformation coefficient block with a second number of transformation coefficients.

[0010] Since the main neural network can sequentially predict at least one transformation coefficient for each variable-length block, the main neural network takes into account time-dependent and / or frequency-dependent factors. The main neural network may have memory functions such that previous inputs influence the current processing, and that the prediction of the current (enhanced) at least one transformation coefficient is influenced by previous transformation coefficients.

[0011] The adaptive length block represents the trade-off between frequency and time. Longer blocks contain more conversion factors and represent longer durations of the media signal, while shorter blocks contain fewer conversion factors and represent shorter durations of the media signal.

[0012] A second aspect of the present invention provides a method for obtaining at least one training block for training a computer-implemented neural network system to predict at least one transformation coefficient of an adaptive block-length media signal. The method includes the steps of: obtaining a set of transformation blocks, each containing a certain number of transformation coefficients representing the frequency content of a media signal, wherein the number of transformation coefficients in each block is a first number or a second number, where the first number is greater than the second number; determining that the first block contains a second number of transformation coefficients; transforming the first block into a transformed block having a first number of transformation coefficients; obtaining a target prediction block from the transformed block; quantizing the transformed block; and obtaining a training block from the quantized transformed block.

[0013] The acquired set of transformation blocks may further represent a sequence of associated time-domain window functions (short, long, bridge-in, or bridge-out).

[0014] A third aspect of the present invention provides a computer-implemented neural network system for predicting at least one transformation coefficient representing the frequency content of an adaptive block-length media signal. The neural network system includes an adaptive block preprocessing unit configured to receive a frame containing a set of quantized transformation coefficients representing a partial time segment of a media signal, and to receive block-length information indicating the number of quantized transformation coefficients for each block in the frame, wherein the number of quantized transformation coefficients is one of a first number or a second number, the first number being greater than the second number, to determine that at least a first block has a second number of transformation coefficients, and to convert at least a first block into a transformed block having a first number of quantized transformation coefficients. The neural network system comprises a main neural network, which is trained to predict at least one output variable given at least one conditioned variable based on conditioned information, wherein the conditioned information includes a representation of a transformed block and a representation of block length information for a first block; and an output stage configured to provide at least one predicted transformation coefficient from at least one output variable.

[0015] In some implementations, the neural network system described above is trained using a set of target prediction blocks and a set of training blocks. The set of training blocks is a degraded representation of the target prediction blocks, and the training blocks comprise at least one training block with a first number of transformation coefficients and at least one training block with a second number of transformation coefficients. The set of training blocks is provided to the adaptive block preprocessing unit of the neural network system, and a set of prediction blocks from the set of training blocks is obtained from the output stage of the neural network system. A scale of the prediction blocks relative to the target prediction block set is calculated, and the weights of the neural network system are modified to reduce the scale.

[0016] By adjusting the weights of the neural network system in response to the scale of the predicted block, during training, the neural network system learns to predict (generate) at least one transformation coefficient from at least one quantized transformation coefficient. Through training, the neural network system learns to appropriately recognize at least one conditioning variable representing a short block(s) and to process it in such a way that the resulting at least one predicted transformation coefficient closely resembles at least one transformation coefficient of the media signal.

[0017] It should be understood that, based on obtaining a scale, a neural network system can be trained, preferably iteratively, by modifying the parameters (e.g., weights) of each neural network until a satisfactory small scale is achieved.

[0018] The inventions according to the second and third embodiments feature the same or equivalent embodiments and advantages as those of the invention according to the first embodiment. Furthermore, any function described in relation to the method may have corresponding structural features in a system or code for performing such a function in a computer program product.

[0019] Experiments were conducted to encode and decode reference media signals with fixed and adaptive block lengths. In the case of fixed block lengths, a fixed-length neural network system was implemented in the decoder, and in the case of adaptive block lengths, a neural network system according to the implementation form of the present invention was implemented in the decoder. Fixed-block length encoding used 256 MDCT coefficient blocks, and adaptive block length encoding used adaptive 256 / 128 MDCT coefficient blocks. Comparing the decoded signals, it was shown that adaptive block length switching using the neural network system of the present invention in the decoder reduced pre-echo distortion compared to the fixed-block length counterpart. [Brief explanation of the drawing]

[0020] The present invention will be described in more detail with reference to the accompanying drawings illustrating currently preferred embodiments of the invention. [Figure 1] This document shows an adaptive block-length encoder and decoder that implement a neural network system according to an embodiment of the present invention. [Figure 2] This document illustrates a neural network system according to an embodiment of the present invention. [Figure 3a] This shows the merging process of time window functions. [Figure 3b] This shows the merging process of time window functions. [Figure 4] A flowchart illustrating a method for predicting at least one transformation coefficient from quantized transformation coefficients, according to one embodiment of the present invention, is shown. [Figure 5] A flowchart illustrating a method for obtaining training blocks for training a neural network system according to embodiments of the present invention is shown. [Figure 6] A flowchart illustrating a method for obtaining training blocks for training a neural network system according to embodiments of the present invention is shown. [Modes for carrying out the invention]

[0021] Figure 1 shows an adaptive block-length encoder / decoder system including encoder 1 and decoder 2. The media signal is received at the input port in the transient detector 101. The media signal may be divided into a series of time-domain frames, which may be subdivided into multiple time-domain segments, each segment containing a number of media signal samples. For example, a time-domain frame may contain 16,000 signal samples and be divided into four segments of 4,000 samples each. The number of signal samples in a time-domain frame and segment (and therefore the number of segments in a time-domain frame) are illustrative and may be any number. The transient detector 101 is configured to optimize the trade-off between frequency and time resolution by selecting a conversion length for each segment. Generally, the transient detector 101 selects a longer conversion length for segments containing signals with slowly evolving or stationary time characteristics, and a shorter conversion length for segments containing signals with rapidly evolving time characteristics. By optimizing the "perceived coding gain" for both short and long signal classes, this method offers a fundamental advantage over coding using time-invariant conversion lengths.

[0022] Depending on the temporal characteristics of the media signal segment, the transient detector 101 may choose the requirement that the segment be represented by a conversion region block having a first number of conversion coefficients (for slowly evolving temporal signal segments) or by multiple conversion region blocks, each containing a second number of conversion coefficients (for rapidly evolving temporal signal segments), where the first number is greater than the second number. For example, the transient detector 101 may have a slowly evolving segment with 256 conversion coefficients X krepresented by 128 conversion coefficients X in a segment that is rapidly developing k in two sets (conversion region blocks) of or 64 conversion coefficients X k in four sets of. The number of conversion coefficients selected is not limited to the examples included, and any number can be selected. The transition detector 101 may request the number of conversion coefficients from a set of block lengths, and the set of block lengths includes at least two lengths such as 256 / 128. In some implementations, the set of block lengths includes at least three or more lengths such as 256 / 128 / 64, and the transition detector 101 may select a length suitable for the block from among them. For example, the transition detector 101 may request that the segment be represented by a combination of short blocks of various lengths. For example, a slowly developing segment is represented by 256 conversion coefficients X k and a subsequent rapidly developing segment is represented by 128 conversion coefficients X k in one block and 64 conversion coefficients X k in two blocks. The transition detector 101 generates block length information representing the required number of conversion region blocks (and / or the number of conversion coefficients X k for each block) in which the time domain segment is to be represented. The block length information is transmitted to the decoder 2. The transition detector 101 passes the block length information to the conversion unit 102.

[0023] The conversion unit 102 converts the segment according to the block length information and outputs an adaptive length conversion block including the conversion coefficients X k to the quantizer 103. In the case of the above example, a 16000-sample time frame divided into four segments of 4000 samples each has 256, 256, 128, 128, and 256 conversion coefficients X kIt is converted into a series of conversion blocks having . These conversion blocks can then form conversion region frames (frames) in encoder 1 and / or decoder 2. In other words, a frame may be called a set of one or more conversion blocks and / or one or more segments. In some parts of encoder 1 and decoder 2, the frames to which the conversion blocks belong are sufficient to handle the conversion blocks sequentially regardless of their respective time or conversion region frames, and therefore may not be explicitly shown or considered.

[0024] The received media signal is further received by a perception model 111 that calculates a masking threshold. The masking threshold is passed to a bit allocation unit 112.

[0025] In the bit allocation unit 112, bit allocations for conversion coefficients to be quantized are made based on received perceptual masking threshold information received from the perceptual model 111. The bit allocation unit 112 may allocate bits to reduce or minimize quantization noise. The bit allocation unit 112 passes the bit allocation information to the quantizer 103.

[0026] The quantizer 103 assigns bits to each conversion coefficient according to the received bit assignment information, thereby converting the conversion coefficient X of each block in the adaptive block length block. k Quantize the quantized transformation coefficients

number

[0027] In decoder 2, the neural network (NN) system 201 receives a frame, and each block of the frame is converted by at least one quantized transformation coefficient X from the quantizer 103 of encoder 1. ~k This includes block length information from the encoder's transient detector 101. The neural network system 201 consists of a main neural network and quantized conversion coefficients X ~ k from at least one transformation coefficient (at least one predicted transformation coefficient)

number

[0028] At least one predicted transformation coefficient X ― k This is the predicted transformation coefficient X of at least one transformation region block. ― k The data is received by an inverse transform unit 202 configured to convert it into a time-domain segment (i.e., a predicted time-domain segment). In some implementations, the inverse transform unit 202 may receive block length information from the transient detector 101 of the encoder 1.

[0029] As explained above, at least one predicted transformation coefficient X arrives as a block in the inverse transformation unit 202. ― k The blocks may be static and of a predetermined length, even though some blocks represent one or more quantized blocks of a shorter length than the original (pre-transformation) blocks. The inverse transform unit 202 receives this information of the original transformed domain block length in the form of block length information, so that the inverse transform unit 202 can perform the necessary inverse transform preprocessing steps. For example, in response to the predicted long blocks being associated with the original short blocks that were upsampled in the transform unit to form the transformed blocks, the inverse transform unit 202 may downsample the predicted long blocks to predicted short blocks before inverse transforming the short blocks back into the time domain. In another example, the quantized transformation coefficient X ~ k At least two short blocks having are transformed into a single transformed block in the transformation unit, and at least one predicted transformation coefficient X ― k The neural network system predicts the longest block as a single long block. In such a case, the inverse transform unit 202 determines from the block length information that the predicted longest block is actually a prediction based on at least two shorter blocks (combined), and in response, may perform an inverse transform preprocessing step, such as splitting to obtain a predicted block of the same length as determined by the transient detector 101 in the encoder 1, or performing an inverse transform procedure, i.e., the reverse of the transform performed in the neural network system 201. The inverse transform preprocessing step may be performed by a separate (not shown) unit preceding the inverse transform unit for some existing coding scheme for adaptive block length media signals. For example, the neural network system (along with the inverse transform preprocessing) may be implemented with any existing codec to refine AC-4 transform coefficients, for example, or it may be used with a novel codec designed for decoding by the neural network system 201.

[0030] In a further implementation, the inverse unit 202 converts each predicted block (which is of static length) into a time-domain signal, for example, if the set of predicted blocks is from a media signal of static length. In such an implementation, the inverse unit does not need to consider block length information, and the neural network system effectively converts the adaptive block-switching media signal into a media signal of static block length. The neural network system 201 is trained to receive blocks of various lengths and output fixed-length blocks. The inverse unit 202 converts the static-length blocks into a time-domain media signal.

[0031] The inverse transform unit 202 outputs a time-domain media signal (or a sequence of time-domain media signal blocks) suitable for playback by a playback device (not shown). The neural network system 201 is configured to receive at least one quantized transform coefficient in the block and to predict at least one transform coefficient.

[0032] Referring to Figure 2, one embodiment of the computer-implemented neural network system 201 of Figure 1 is shown in more detail. The neural network system 201 uses quantized transformation coefficients X to represent the frequency content of a partial time segment of a media signal. ~ kThe system is configured to receive a set of adaptive length blocks 20, each containing a set of quantized transformation coefficients, and block length information 21 indicating the number of quantized transformation coefficients for each block in the frame 20, wherein the number of quantized transformation coefficients is one of a first number or a second number. The computer-implemented neural network system 201 further comprises a transformation stage 11 configured to determine that at least a first block has a second number of quantized transformation coefficients, and to transform at least a first block into a transformed block having a first number of quantized transformation coefficients. From frame 20 to transformation stage 11, if frame 20 has at least one block having a second number of quantized transformation coefficients, the transformation stage generates an output frame 20', in which all output blocks in the output frame have a first number of quantized transformation coefficients.

[0033] The neural network system 201 further receives block length information 21 indicating the number of quantized transformation coefficients for each block in frame 20. Thus, the block length information 21 indicates a sequence of blocks containing a first or second number of transformation coefficients. The block length information 21 may be a sequence of integers or symbols, where each integer or symbol represents a block, and the value of each integer (or type of symbol) is the quantized transformation coefficient X of that block. ~ k It represents the number.

[0034] The block length information 21 may include three or more alternative block lengths. In some implementations, the second number is a conversion coefficient X. k The transformation coefficient X of the first number preceding the block that has k A block having a second number transformation coefficient X may be labeled as a bridge-in block. k The transformation coefficient X of the first number that follows the block having k Blocks that have this can be labeled as bridge-out blocks. Thus, the block length information 21 can be a sequence of four (or more) different integers, and the longest block (the conversion coefficient X of the first number) k ), short block (conversion coefficient X of the second number) kThere is one for each of the bridge-in block and bridge-out block.

[0035] The neural network system 201 forms at least one conditional variable 15 based on conditional information, the conditional information includes at least two components: (i) information representing a transformed block (or a block containing a first number of quantized transformation coefficients), and (ii) information representing block length information 21. In a simple case, the information representing a transformed block is itself a quantized transformation coefficient X ~ k The block length information representation is an integer. At least one conditioning variable 15 and the main neural network 16 may feature a separate dimension for each conditioning information, or they may feature a single dimension on which each conditioning information is projected.

[0036] At least one conditioning variable 15 is used to condition the main neural network 16. The main neural network 16 is trained to predict at least one output variable given at least one conditioning variable 15, and at least one output variable is then used to make a final prediction of at least one transformation coefficient (i.e., at least one predicted transformation coefficient X) from the main neural network 16, given at least one output variable. ― k The output neural network 17 is provided with the output (which is trained to output). The output neural network 17 may have one or more hidden layers.

[0037] The main neural network 16 can be any type of neural network, e.g., a deep neural network, a recurrent neural network, or any neural network system. The main neural network 16 can be a regression model. The media signal can be any type of media signal, including an audio signal or a video signal. If the media signal is an audio signal, the main neural network 16, in a preferred embodiment, functions as a general audio generation model in the conversion domain. The main neural network 16 is configured to operate in the conversion domain and is trained to predict at least one output variable given at least one conditioning variable. The at least one output variable may be considered a hidden state and is provided to an output neural network 17, which is configured (e.g., trained) to output at least one predicted conversion coefficient given at least one output variable. The output neural network 17 can be implemented as a single unit, e.g., as an output stage of the main neural network 16, or as a separate neural network, together with the main neural network 16. In any case, the output neural network 17 and the main neural network 16 exchange hidden state information. Therefore, at least one conversion coefficient X ― k This involves understanding the time and / or frequency dependence of the representation of the quantized transformation coefficients, thereby enabling the main neural network 16 and the output neural network 17 to obtain at least one quantized transformation coefficient X ~ kThis is predicted from the above. That is, the main neural network 16 and the output neural network 17 may be trained so that previous representations of the transformation coefficients processed by the main neural network 16 can influence the prediction of at least one current transformation coefficient. Additionally or alternatively, the main neural network 16 and the output neural network 17 may be trained so that the interdependence between the transformation coefficients in the current block and the transformation coefficients in past blocks is taken into account. Since the transformation coefficients represent frequency content, the main neural network 16 and the output neural network 17 may be trained to predict at least one transformation coefficient by learning how the frequency content (represented in the transformation coefficients) of a first frequency band affects the frequency content of a second frequency band.

[0038] In some implementations, the neural network system 201 further comprises additional neural networks, such as a conditioning neural network 12 connected to receive output from the conversion unit 11 and block length information from the block length information neural network 14. The conditioning neural network 12 and the block length information neural network 14 are used to predict the respective conditioning information and may be any type of neural network, such as a convolutional layer, and the use of one type does not require the use of the other.

[0039] The conditioned neural network 12 and / or the block-length information neural network 14 may be trained to predict at least one output variable, where at least one conditioned variable 15 is then obtained as the sum of at least one predicted output variable. Furthermore, at least one conditioned variable 15 passed to the main neural network 16 (e.g., the sum of at least one output variable from the conditioned neural network 12 and the block-length information neural network 14) may be considered a hidden neural network layer. In addition to establishing an inner dimension (as a hyperparameter) for the hidden layer that matches the input dimension of the main neural network 16, the neural network system 201 may operate (and train) without any constraints on the interpretability of the hidden layer. For example, the conditioned information and block-length information representations representing quantized transformation coefficients may each be at least one output variable in the form of a matrix of dimensions matching the inner dimension. At least one conditioned variable 15 may be the sum of at least one matrix output variable. In a further example, a matrix is ​​two-dimensional and contains a single row or column (i.e., a vector).

[0040] The conditioned neural network 12 uses the quantized transformation coefficient X of the block. ~ k Given X, the conditioned neural network 12 is trained to predict the representation of the block from the output frame 20'. Using the conditioned neural network 12 trained to predict the representation, a representation that further facilitates prediction by the main neural network 16 can be achieved by predicting the representation of the quantized transformation coefficients of the transformed block. Quantized transformation coefficients X ~ k A static transformation function that converts information representing the quantized transformation coefficient X ~ kIn contrast to assigning to a specific neural network, the conditioned neural network 15 can be trained to predict representations that facilitate the final prediction by the main neural network 16 and the output neural network 17.

[0041] Similarly, the block length information neural network 14 is trained to predict a representation of block length information given block length information 21. The conditioning information used to condition the main neural network 16 is the block length neural network 14, which is trained to predict a representation of block length information given block length information 21 for at least the first block, by implementing the block length neural network 14, which is trained to predict a representation of block length information given block length information 21 for at least the first block, and the conversion coefficient X by the main neural network 16 and the output neural network 17. ― k The transformation coefficients X are quantized in the first block in a format that facilitates prediction. ~ k This will carry information indicating the number of [X]. In one example, the block length neural network 14 will carry the first number conversion coefficient X k It outputs a representation of block length information indicating a block that has the following characteristics. Therefore, the main neural network 16 converts the first number X ~ k Quantized transformation coefficients X represented from a transformed or quantized block having ~ k When is 1, it will be conditioned differently and respond differently. The main neural network 16 and the output neural network 17, together with the conversion unit 11, quantized the conversion coefficient X ~ k Since it is trained to predict at least one transformation coefficient from the information representing, the prediction of at least one transformation coefficient can be achieved regardless of how the transformed block was constructed from at least the first block.

[0042] For example, in contrast to conditioning a block-length neural network using integers from a sequence of integers, some implementations of the neural network system 201 include a one-hot encoder 13, which converts block-length information 21 into a one-hot vector, and the one-hot vector is used to condition the block-length neural network 14. The block-length information is categorical, indicating a distinct state for each block (e.g., long, short, bridged in, or bridged out). One-hot coding separates these categories into individual vector elements, which facilitates training and prediction of the block-length neural network 14 by clearly distinguishing different possible states. For example, one-hot coding promotes a strong spatial dependency between at least one predicted output variable and which input element in the input layer of the block-length neural network received the one-hot (on-state) vector element.

[0043] In some implementations, the neural network system 201 further receives perceptual model coefficients pEnvQ and / or spectral envelopes for each block. Thus, the conditioning information may further include additional information that is a representation of the perceptual model coefficients pEnvQ and / or spectral envelope information. The perceptual model coefficients pEnvQ and / or spectral envelopes may be processed in parallel with the block length information and quantized transformation coefficients and combined with other information in at least one conditioning variable 15, or provided to the main neural network 16 as side information in another dimension.

[0044] A set of perceptual model coefficients, pEnvQ, can be derived from a perceptual model that occurs in an encoder. The perceptual model coefficients pEnvQ are calculated for each frequency band and, for ease of processing, are preferably mapped to the same resolution as the frequency coefficients of the block.

[0045] In implementations where a single short block is converted to a converted block, the pEnvQ coefficients are converted to an equivalent long block representation by a similar conversion procedure and used as conditional information. For example, if a short block is upsampled, the pEnvQ coefficients are upsampled in the same manner.

[0046] In an implementation featuring two or more neural networks, it should be noted that in a "trained" neural network system 201, all neural networks within the system are trained together for at least part of the training. For example, a block-length neural network 14 may be trained together with the main neural network 16, and the internal parameters (e.g., weights) of each neural network 14, 16 are the original dequantized transformation coefficients X k Compared to several targets, including at least one predicted transformation coefficient, the predicted at least one transformation coefficient X ― k It is modified to optimize some measure of. Next, the block-length neural network 14 is trained to output at least one conditioning variable 15 that approximates as closely as possible the predicted at least one transformation coefficient of the main neural network 16 and the output neural network 17 to the original transformation coefficient. The main neural network 16 and the output neural network 17 are modified to optimize some measure of the original transformation coefficient X k At least one transformation coefficient X that approximates as closely as possible ― k They are simultaneously trained to make predictions.

[0047] The transformation in the transformation unit 11 of the block having a second number transformation coefficient is the quantized transformation coefficient X of the first number. ~ kIt may include upsampling a block having [the coefficient] to a transformed block. Upsampling may include linear interpolation or polynomial interpolation (and optionally extrapolation) of a second number of quantized transform coefficients to a first number of quantized transform coefficients. Alternatively, upsampling to form a transformed block may include repeating each quantized transform coefficient a predetermined number of times, adding zero elements between non-zero elements, or interleaving the quantized transform coefficients X ~ k It may include one of the following. Alternatively, any other suitable upsampling, expansion, or interpolation technique may be applicable. In some implementations, the transformation unit 11 simply transfers the quantized transform coefficients X ~ k of a block to the main neural network 16, and the main neural network 16 is trained to predict at least one output parameter of the output neural network 17. In this case, the main neural network 16 learns to recognize a block having a second number of quantized transform coefficients X ~ k and absorbs it by training the function of the transformer.

[0048] In the transformation unit 11, instead of transforming a first block including a second number of quantized transform coefficients X ~ k into at least two blocks, namely a first block and a second block each including a second number of quantized transform coefficients X ~ k the first block and the second block may be transformed together into a transformed block including a first number of quantized transform coefficients X ~ k Thus, when given a representation of a transformed block including a first number of quantized transform coefficients X ~ k the main neural network 16 and the output neural network 17 [output] at least one transform coefficient X ― kcan be trained to predict the quantized transform coefficients X of the transformed block ~ k is derived from the quantized transform coefficients X of at least the first block and the second block ~ k .

[0049] Generally, at least the first block and the second block having a second number of quantized transform coefficients X ~ k are N consecutive blocks having a second number of quantized transform coefficients X ~ k , where the first number is a multiple N of the second number. Then, the N consecutive blocks can be converted into a transformed block having a first number of quantized transform coefficients X ~ k . An adaptive block switching media signal can include, for example, a first number of quantized transform coefficients X equal to 256 ~ k and a second number equal to 128, i.e., N = 2. When the first number is equal to 256 and N = 4, four short blocks each including 64 quantized transform coefficients X ~ k will be converted into one transformed block. In a further example, when N = 8 and the first number of transform coefficients is 1024, the second number of quantized transform coefficients X ~ k is 128.

[0050] Converting at least the first block and the second block into a transformed block can include concatenating at least the first and second blocks into the transformed block. Concatenation is an efficient and easily implemented way to convert at least the first block and the second block into the transformed block.

[0051] In some implementations, the transform unit 11 receives, for each block, a representation of its respective time domain window function, where the window functions of the first block and the second block partially overlap.

[0052] The window function is the quantized transformation coefficient X ~ k It may be received together with or with block length information 21 (passed to the conversion unit 11). Alternatively, the window function may be constructed from block length information 21 (passed to the conversion unit 11). Or, the window function may be constructed in the conversion unit 11 by utilizing the correlation between the number of quantized conversion coefficients in a block and a sequence of blocks having at least a first number and a second number of quantized conversion coefficients in each block, thereby obtaining the quantized conversion coefficients X for each block. ~ k This can be constructed by determining the number of . For example, the quantized transformation coefficient X of the first number. ~ k Blocks with a long window function are associated with the transformation quantized coefficient X of a second number. ~ k Blocks with are associated with a short window function. In further examples, the quantized transformation coefficient X of the first number ~ k A block having a second number of quantized transformation coefficients X ~ k If it precedes a block containing a bridge-in window function, it can be associated with a bridge-in window function.

[0053] In Figure 2, all functions and units described as operating upstream of the (optional) conditioned neural network 12 and the (optional) block length information neural network 14 may be referred to as preprocessing units or adaptive block preprocessing units. Thus, the preprocessing units, for example, handle block length information 12 and quantized transformation coefficients X. ~ k It may be a multi-input multi / single-output unit that receives quantized transformation coefficients X as separate information (at least one variable) or combined information (at least one variable). ~ k It represents and outputs information representing block length information 12.

[0054] Referring further to Figure 6, a flowchart is shown illustrating a method for training a neural network system, for example, the embodiment shown in Figure 2. In S311, a set of adaptive length target prediction (true) blocks is provided. This is done in parallel with providing a set of training blocks in S321, which are degraded representations (e.g., quantized representations) of the target prediction blocks. The target prediction blocks are transformed by the X k The training blocks are provided to the neural network system 201 and processed so that a set of prediction blocks is obtained in S331. At least one predicted transformation coefficient X ― k In S332, a measure of similarity is obtained, for example, by comparing the output prediction block containing the target prediction block. The measure can be an error measure, with a low error measure indicating a high level of similarity. The measure can be a negative likelihood, such as the negative log-likelihood (NLL), with a low measure indicating a high level of similarity. The measure can be the mean absolute error (MAE) or mean squared error (MSE), with a low MAE or MSE indicating a high level of similarity. In S333, the measure is used to modify the weights of the neural network system 201 to reduce or minimize the measure.

[0055] In one example, the measure is calculated directly as an NLL as follows, and is therefore called the loss function or "loss":

number

[0056] When calculating the NLL loss, at least one predicted transformation coefficient X ― k This is at least one predicted transformation coefficient X ― k It is represented by at least one distribution parameter. Thus, the NLL function is represented by at least one predicted transformation coefficient X ― kThis is applied to at least one distribution parameter that represents the following: The at least one distribution parameter is at least one predicted transformation coefficient X ― k Parameterize the probability distribution for at least one of these.

[0057] In other implementations, the loss is calculated as MSE according to the following formula,

number

number

[0058] When calculating MSE and MAE losses, at least one predicted transformation coefficient X ― k It will be used as is.

[0059] In some cases, a prediction block may represent two or more training blocks (and associated target prediction blocks) with a single predicted transformed block, in which case the prediction block may be inversely transformed into blocks that individually correspond to the training block (and associated target prediction block) so that a scale can be calculated.

[0060] Referring to Figure 3a, the sequence of time-domain window functions 31, 32a, 32b, and 33 is shown. Figure 3a shows a typical window sequence for a 2:1 block-length switch. A first long window 31 is followed by two short windows 32a and 32b, and these windows are followed by a second long window 33. The short time-domain window functions 32a and 32b may overlap by 50%, where the sum of the squared short window functions equals 1 for the overlapping portion. Additionally, the sum of the squares of each window function 31, 32a, 32b, and 33 equals 1 for all overlaps.

[0061] In some implementations, the long windows 31 and 33 may further be bridge-in windows 31 and bridge-out windows 33, respectively, which are specifically adapted to precede and follow the short windows 32a and 32b. The window functions 31, 32a, 32b, and 33 overlap at least partially in time. Each window function 31, 32a, 32b, and 33 is associated with a set of transformation coefficient blocks, a long transformation coefficient block with the long window functions 31 and 33, and a short transformation coefficient block with the short window functions 32a and 32b.

[0062] In some additional implementations, if the number of transformation coefficients in each block is one of three or more options (e.g., one of the coefficients 256, 128, and 64 as described above), the functionality of the bridge-in window 31 and bridge-out window 33 may include three or more bridging window functions, e.g., one for each type of transition between variable-length blocks. If the block has a length of one of 256, 128, and 64, then in and out bridging window functions can be defined for each of 256 to 128, 256 to 64, and 128 to 64.

[0063] Referring further to Figure 3b, we see the long transformed window 32 (with the associated long transformed block), which is the result of the transformation of the two short window functions 32a, 32b (and the two short transformed coefficient blocks).

[0064] The quantized transformation coefficients of the first and second (short) blocks (their respective window functions are shown as 32a and 32b in Figure 3a) can be merged into the longer transformed block by inversely transforming them into their windowed time-domain representations. This can be achieved by double-summing the windowed time-domain representations of the first and second blocks and then transforming the double-summed time-domain representations of the first and second blocks into a transformed block with a first number of quantized transformation coefficients.

[0065] For example, if the transformation coefficients are modified discrete cosine transform (MDCT) coefficients, the intervening short blocks (associated with window functions 32a, 32b) can be merged into a single long block by inverting the MDCT into short time-domain segments and then doubling the short time-domain segments. The transformation coefficients of the equivalent transformed long block 32 with a flat-top window can then be computed using DCT type 4. The window sequence after this merge / transform operation is shown in Figure 3b. It should be further noted that this transformation procedure can be achieved while maintaining the complete reconstruction properties of the transformation coefficients (in the absence of quantization).

[0066] Referring to Figure 4, a flowchart is shown illustrating a method for predicting at least one transformation coefficient from quantized transformation coefficients according to one embodiment of the present invention. In S111, the neural network system receives a frame containing quantized transformation coefficients. In S112, the neural network system determines that at least one block of the frame contains a second number of transformation coefficients, and then in S113, transforms the at least one block with the second number of transformation coefficients into a transformed block with a first number of transformation coefficients. The information representing the quantized transformation coefficients of the transformed block is one piece of information on which at least one conditioning variable used to condition the main neural network in S131 is based. Optionally, the method includes the steps of: in S114, conditioning a conditioning neural network with the information representing the quantized transformation coefficients of the transformed block; and in S131, conditioning a main neural network with at least one output variable of the conditioning neural network.

[0067] Furthermore, the method includes the step of receiving block length information in S121. The representation of the block length information is used as one piece of information for conditioning the main neural network in S131. Optionally, the block length information is used to first condition the block length neural network in S123, and at least one predicted output variable of the block length neural network is used to condition the main neural network in S131. Also, some embodiments include one-hot coding of the block length information in S122, and the one-hot coded block length information is used to condition the block length neural network in S123 or as part of the information used to condition the main neural network in S131.

[0068] In S131, the main neural network, given at least one conditioning variable, predicts at least one output variable, and at least one output variable is provided to the output stage (e.g., the output neural network) in S132. The output stage in S132 predicts at least one transformation coefficient.

[0069] Figure 5 shows a flowchart illustrating a method for obtaining training blocks (training blocks for input and target prediction blocks for comparison with the output) for training a neural network system for predicting conversion coefficients of an adaptive block-length media signal according to an embodiment of the present invention. In S211, a set of conversion blocks is obtained. For example, a batch of waveforms or a media signal is divided into a set of time-domain segments (e.g., forming a time-domain frame), and each time-domain segment is converted into a set of conversion blocks of varying lengths (e.g., conversion-domain frames). Alternatively, a batch of waveforms or a media signal is processed using a transient detector as described above to determine the length of each block. In S212, it is determined that a first block contains a second number of conversion coefficients, and this block is converted into a converted block with the first number of conversion coefficients in S213. In S221, a target prediction block is obtained. The target prediction block obtained in S221 may be the converted block itself.

[0070] In S231, the transformed block is quantized to form a quantized block. That is, the quantized block does not represent the complete information originally present in the determined first block, and therefore, the quantized block is sometimes called a damaged block that the neural network should learn to use for predicting the undamaged block. In S232, a training block is obtained from the quantized block obtained in S231. The training block may be such a quantized block. Some implementations further include steps of using a target training block as input to the neural network during training, and using a target prediction block as training.

[0071] Blocks determined to contain a first number of conversion coefficients may be processed similarly to obtain training blocks and target prediction blocks, and step S213 is omitted.

[0072] In some implementations, batches of media signals or waveforms are processed using transient detectors that determine the conversion length, as described above. Therefore, the set of conversion blocks will include all different types of blocks and window functions.

[0073] The above describes possible methods for training and operating a deep learning-based system for determining audio quality instructions for input audio samples, as well as possible implementations of such a system. In addition, this disclosure also relates to apparatus for performing these methods. Examples of such apparatus may comprise a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof) and memory coupled to the processor. The processor may be adapted to perform some or all of the steps of the methods described throughout this disclosure.

[0074] The device may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing (sequential or otherwise) instructions that specify actions to be performed by the device. Furthermore, this disclosure relates to any set of devices that individually or collectively execute instructions to perform any one or more of the methods described herein.

[0075] This disclosure further relates to a program (e.g., a computer program) which, when executed by a processor, comprises instructions causing the processor to perform some or all of the steps of the method described herein.

[0076] The present invention also relates to a computer-readable (or machine-readable) recording medium on which the above-mentioned program is recorded. Here, the term "computer-readable storage medium" includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0077] Unless otherwise specified, as will be apparent from the following description, throughout this disclosure, any use of terms such as “processing,” “computing,” “calculating,” “determining,” and “analyzing” is understood to refer to the operation and / or process of a computer or computing system, or similar electronic computing device, that manipulates and / or converts data expressed as physical quantities, such as electron quantities, into other data similarly expressed as physical quantities.

[0078] Similarly, the term “processor” can refer to any device or part of a device that processes electronic data from, for example, registers and / or memory and converts that electronic data into other electronic data that can be stored, for example, in registers and / or memory. A “computer,” “computing machine,” or “computing platform” may include one or more processors.

[0079] The methods described herein are executable in one exemplary embodiment by one or more processors that accept computer-readable (also called machine-readable) code, which includes a set of instructions that, when executed by one or more of the processors, perform at least one of the methods described herein. This includes any processor capable of executing a set of instructions (sequential or otherwise) that specify the actions to be performed. Thus, an example is a typical processing system comprising one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem including main RAM and / or static RAM, and / or ROM. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system having processors connected by a network. If the processing system requires a display, such a display may include, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes one or more input devices, for example, an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, etc. The processing system may also include a storage system such as a disk drive unit. In some configurations, the processing system may include a sound output device and a network interface device. Thus, the memory subsystem includes a computer-readable carrier medium that carries computer-readable code (e.g., software) containing a set of instructions for performing one or more of the methods described herein when executed by one or more processors. Note that if a method involves several elements, e.g., several steps, the order of such elements is not implied unless otherwise specified. The software may reside in the hard disk, or it may reside entirely or at least partially in RAM and / or in the processor during its execution by the computer system.Therefore, memory and processors also constitute computer-readable carrier media for carrying computer-readable code. Furthermore, computer-readable carrier media may form or be included in computer program products.

[0080] In alternative exemplary embodiments, one or more processors may operate as standalone devices or be connected, for example, in a networked deployment, and one or more processors may operate as a server or user machine in a server-user network environment or as a peer machine in a peer-to-peer or distributed network environment. One or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify the actions that the machine should perform.

[0081] It should be noted that the term “machine” should also be interpreted as including any set of machines that individually or collectively execute one or more sets of instructions for executing any one or more of the methods discussed herein.

[0082] Accordingly, each exemplary embodiment of the methods described herein is in the form of a computer-readable carrier medium that carries a set of instructions, for example, a computer program to run on one or more processors, for example, one or more processors that are part of a web server configuration. Accordingly, as will be understood by those skilled in the art, exemplary embodiments of the disclosure may be embodied as a method, an apparatus such as a dedicated device, an apparatus such as a data processing system, or a computer-readable carrier medium, for example, a computer program product. The computer-readable carrier medium carries computer-readable code, which, when executed on one or more processors, causes one or more processors to execute the method. Accordingly, embodiments of the disclosure may be in the form of a method, an exemplary embodiment that is entirely hardware, an exemplary embodiment that is entirely software, or an exemplary embodiment that combines software and hardware embodiments. Furthermore, the disclosure may be in the form of a carrier medium (for example, a computer program product on a computer-readable storage medium) that carries computer-readable program code to be embodied in the medium.

[0083] The software may further be transmitted or received over a network via a network interface device. While the carrier medium is a single medium in exemplary embodiments, the term “carrier medium” should be interpreted to include a single or multiple mediums (e.g., centralized or distributed databases and / or associated caches and servers) that store one or more sets of instructions. The term “carrier medium” should also be interpreted to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more processors, causing one or more processors to execute one or more of the methods of this disclosure. The carrier medium can take many forms, but is not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute a bus subsystem. Transmission media can also take the form of sound waves or light waves, such as those generated during radio and infrared data communications. Consequently, the term “carrier medium” shall be interpreted to include, but not limited to, computer products embodied in solid memory, optical media and magnetic media, media carrying a propagation signal detectable by at least one processor or one or more processors and representing a set of instructions that perform a method when executed, and transmission media in a network carrying a propagation signal detectable by at least one of the one or more processors and representing a set of instructions.

[0084] It will be understood that, in one exemplary embodiment, the steps of the method described are performed by a suitable processor (or more processors) of a processing (e.g., a computer) system that executes instructions (computer-readable code) stored in storage. It will also be understood that this disclosure is not limited to any particular implementation or programming technique, and may be implemented using any suitable technique for implementing the functionality described herein. This disclosure is not limited to any particular programming language or operating system.

[0085] Throughout this disclosure, any reference to “one example embodiment,” “some example embodiments,” or “an example embodiment” means that any particular feature, structure, or characteristic described in relation to an exemplary embodiment is included in at least one exemplary embodiment of this disclosure. Therefore, the appearance of the phrases “one example embodiment,” “some example embodiments,” or “an exemplary embodiment” in various places throughout this disclosure does not necessarily all refer to the same exemplary embodiment. Furthermore, any particular feature, structure, or characteristic may be combined in any suitable manner in one or more exemplary embodiments, as will be apparent to those skilled in the art from this disclosure.

[0086] Where used herein, unless otherwise specified, the use of ordinal adjectives such as “first,” “second,” and “third” to describe common objects merely indicates that different examples of similar objects are being referred to, and is not intended to imply that the objects described in this way must be in a given order, temporally, spatially, in ranking, or in any other way.

[0087] In the following claims and description herein, any one of the terms “comprising,” “comprised of,” or “which comprises” is an open term meaning that it includes at least the elements / features that follow it, but does not exclude others. Therefore, when used in the claims, the term “comprising” should not be interpreted as being limited to the means, elements, or steps listed thereafter. For example, the expression “a device comprising A and B” should not be limited to a device consisting only of elements A and B. Also, as used herein, the terms “including,” “which includes,” or “that includes” are all open terms meaning that they include at least the elements / features that follow them, but do not exclude others. Therefore, “including” is synonymous with and means the same as “comprising.”

[0088] In the above description of exemplary embodiments of the Disclosure, it should be understood that various features of the Disclosure may be grouped together in a single exemplary embodiment, figure, or description in order to streamline the Disclosure and to aid in understanding one or more of the various embodiments of the Invention. However, this method of disclosure should not be construed as reflecting an intention that the claims require more features than those expressly described in each claim. Rather, as reflected in the following claims, the embodiments of the Invention have fewer features than all the features of a single, aforementioned exemplary embodiment. Accordingly, the claims following this Specification are expressly incorporated herein, and each claim stands alone as a separate exemplary embodiment of the Disclosure.

[0089] Furthermore, some exemplary embodiments described herein include some features included in other exemplary embodiments, but do not include other features, and as will be understood by those skilled in the art, combinations of features of different exemplary embodiments are within the scope of this disclosure and are intended to form different exemplary embodiments. For example, in the following claims, any of the exemplary embodiments described in the claims may be used in any combination.

[0090] The descriptions provided herein include numerous specific details. However, it should be understood that exemplary embodiments of this disclosure may be implemented without these specific details. In other instances, well-known methods, structures, and techniques are not described in detail so as not to obscure the understanding of this description.

[0091] Accordingly, while what is considered to be the best form of this disclosure has been described, those skilled in the art will recognize that other and further modifications may be made without departing from the spirit of this disclosure, and it is intended that all changes and modifications that fall within the scope of this disclosure be requested. For example, any formula given above is merely representative of the procedures that may be used. Functions may be added to or removed from the block diagram, and operations may be interchanged between function blocks. Steps may be added to or removed in the manner described within the scope of this disclosure.

[0092] Various aspects of the present invention can be understood from the following list of exemplary embodiments (EEEs):

[0093] EEE1. A method for predicting at least one conversion coefficient representing the frequency content of an adaptive block-length media signal using a computer-implemented neural network system, A step of receiving a block of frames, wherein each block of frames includes a set of quantized transformation coefficients representing a partial time segment of the media signal, A step of receiving block length information indicating the number of quantized transformation coefficients for each block of a frame, wherein the number of quantized transformation coefficients is one of a first number or a second number, and the first number is greater than the second number. The steps include determining that at least a first block of the frame has the second number of quantized transformation coefficients, The steps include: converting at least the first block into a converted block having the first number of quantized conversion coefficients; A step of conditioning a main neural network trained to predict at least one output variable given at least one conditioning variable, wherein the at least one conditioning variable is based on conditioning information, the conditioning information includes a representation of the transformed block and a representation of block length information for the first block. An output stage configured to provide at least one predicted conversion coefficient from the above at least one output variable, comprising the steps of providing the above at least one output variable and A method that includes this.

[0094] EEE2. The method of EEE1, further comprising the step of receiving a set of perceptual model coefficients for each block of the frame, wherein the conditioning information further comprises the set of perceptual model coefficients.

[0095] EEE3. The method of EEE1, further comprising the step of receiving the spectral envelope for each block of the above frame, wherein the conditioning information further comprises the spectral envelope.

[0096] EEE4. A step of conditioning a block length neural network using the above-mentioned representation of block length information for the first block, wherein the block length neural network is trained to output the above-mentioned representation of block length information for the first block when given block length information. The EEE1 method, which further includes the above.

[0097] EEE5. The method of EEE4, wherein the step of conditioning a block length neural network using the block length information includes the steps of encoding the block length information as a one-hot vector and conditioning the block length neural network using the one-hot vector.

[0098] EEE6. A step of conditioning a conditioned neural network using the quantized conversion coefficients of the converted block, wherein the conditioned neural network is trained to output the representation of the converted block when given the quantized conversion coefficients. The EEE1 method, which further includes the above.

[0099] EEE7. The method of EEE1, wherein the step of converting at least the first block to the converted block includes the step of upsampling the first block.

[0100] EEE8. A method of EEE1, further comprising the step of determining that at least the first block and the subsequent second block have a conversion coefficient of the second number, wherein the step of converting at least the first block into the converted block includes the step of converting at least the first block and the second block into the converted block.

[0101] EEE9. The quantized transformation coefficients representing frequency content are discrete cosine transform (DCT) coefficients, one of the preceding EEE methods.

[0102] EEE10. The inverse conversion unit receives the predicted conversion coefficient and the block length information, The steps include converting the above predicted conversion coefficients into a time-domain signal, and Any of the preceding EEE methods, including further.

[0103] EEE11. The step of determining that the first number is a multiple N of the second number, and that at least the first block and the subsequent second block have quantized transformation coefficients of the second number is: The step of determining that N consecutive blocks of the frame have the second number of quantized transformation coefficients mentioned above. The EEE8 method, including the method described above.

[0104] EEE12. The method of EEE8, wherein the step of converting at least the first block and the second block into the converted block includes at least the step of concatenating the first block and the second block into the converted block.

[0105] EEE13. The step of receiving block length information is: A step of receiving a representation of the time-domain window function for each block of the frame, wherein the window functions of the first block and the second block partially overlap. The EEE8 method, including the method described above.

[0106] EEE14. The step of converting at least the first block and the second block into the converted block is: The steps include: inversely transforming the quantized transformation coefficients into windowed time-domain representations of the first and second blocks; The steps include: adding the windowed time-domain representations of the first block and the second block together; The steps include: converting the time-domain representation of the first and second blocks, which are added together, into a transformed block having the first number of quantized transformation coefficients; The EEE13 method, including...

[0107] EEE15. A method for obtaining at least one training block for training a computer-implemented neural network system to predict at least one conversion coefficient of an adaptive block length media signal, A step of obtaining a set of transformation blocks, each containing a number of transformation coefficients representing the frequency content of a media signal, wherein the number of transformation coefficients in each block is either a first number or a second number, and the first number is greater than the second number. The first step is to determine that the first block contains the conversion coefficient of the second number, The steps include: converting the first block into a converted block having a first number of conversion coefficients, Steps include obtaining the target prediction block from the converted block, The steps include quantizing the converted blocks and Steps to obtain training blocks from quantized transformed blocks and A method that includes this.

[0108] EEE16. A computer-implemented neural network system for predicting conversion coefficients representing the frequency content of an adaptive block-length media signal, Receiving a frame containing a set of quantized transformation coefficients representing a partial time segment of a media signal, The process involves receiving block length information indicating the number of quantized transformation coefficients for each block in the above frame, wherein the number of quantized transformation coefficients is one of a first number or a second number, and the first number is greater than the second number. It is determined that at least the first block has the conversion coefficient of the second number above, At least the first block is transformed into a transformed block having the first number of quantized transformation coefficients. An adaptive block preprocessing unit configured to perform the following: A main neural network, the main neural network is trained to predict at least one output variable given at least one conditioned variable based on conditioned information, the conditioned information includes a representation of the transformed block and a representation of block length information for the first block, An output stage configured to provide at least one predicted transformation coefficient from the above at least one output variable, A neural network system equipped with [a specific feature / ability].

[0109] A neural network decoder equipped with a computer-implemented neural network system conforming to EEE17 and EEE16 standards.

[0110] EEE18. Further equipped with an inverse conversion unit, The above inverse conversion unit is, The above includes receiving at least one predicted conversion coefficient and block length information, Converting at least one of the above predicted conversion coefficients into a time-domain signal A neural network decoder for EEE17 configured to perform the following actions.

[0111] EEE19. The above neural network system is To provide a set of target prediction blocks, The adaptive block preprocessing unit is provided with a set of training blocks comprising at least one training block having the first number of conversion coefficients and at least one training block having the second number of conversion coefficients, wherein the set of training blocks is a degraded representation of the set of target prediction blocks. Obtain a set of prediction blocks from the above set of training blocks from the above output stage, To calculate the scale of the set of prediction blocks relative to the set of target prediction blocks mentioned above, Modifying the weights of the above neural network system to reduce the scale and An EEE16 neural network system that is trained by performing the following.

[0112] EEE20. The above measures are one of negative likelihood, mean squared error, or absolute error for a neural network system in EEE19.

Claims

1. A method for generating at least one representation of a conversion coefficient representing the frequency content of an adaptive block-length media signal using a computer-implemented neural network system, A step of receiving a frame comprising one or more blocks, wherein each block of the frame comprises a set of quantized transformation coefficients representing a partial time segment of the media signal; A step of receiving block length information indicating the number of quantized conversion coefficients for each block of the frame, wherein the number of quantized conversion coefficients is one of a first number or a second number, and the first number is greater than the second number. A step of converting at least a first block of the frame into a converted block having a first number of quantized conversion coefficients based on the block length information, The steps include: conditioning a neural network trained to predict at least one output variable given conditioning information including at least a representation of the converted block and a representation of block length information for the first block; A step of providing the at least one output variable to an output stage configured to provide at least one predicted representation of a conversion coefficient from the at least one output variable; A method that includes this.

2. The method according to claim 1, further comprising the step of determining that at least a first block of the frame has a second number of quantized transformation coefficients.

3. The method according to claim 1 or 2, wherein the quantized transformation coefficients representing frequency content are modified discrete cosine transform (MDCT) coefficients.

4. A computer-implemented neural network system for generating at least one representation of conversion coefficients representing the frequency content of an adaptive block-length media signal, - Receiving a frame containing one or more blocks, wherein each block of the frame contains a set of quantized transformation coefficients representing a partial time segment of a media signal. - Receiving block length information indicating the number of quantized transformation coefficients for each block in the frame, wherein the number of quantized transformation coefficients is one of a first number or a second number, and the first number is greater than the second number, - Based on the block length information, at least the first block is converted into a converted block having the first number of quantized conversion coefficients. An adaptive block preprocessing unit configured to perform the following: A neural network which, given conditional information including at least a representation of the transformed block and a representation of block length information for the first block, is trained to predict at least one output variable, An output stage configured to provide at least one predicted representation of a transformation coefficient from the at least one output variable. A neural network system equipped with [a specific feature / ability].

Images