Neural Network-Based Key Generation Using Neural Network-Based Audio Signal Transformation for Key Guidance

Abstract

A method is disclosed, the method comprising: receiving input audio and target audio having target audio characteristics; generating key parameters using a first neural network trained to generate key parameters representing the target audio characteristics based on one or more of the target audio and the input audio; and configuring a second neural network trained to be configured by the key parameters with the key parameters so that the second neural network performs signal transformation of the input audio to generate output audio having output audio characteristics corresponding to and matching the target audio characteristics.

Description

Technical Field

[0001] The present disclosure relates to signal transformation for performing machine learning (ML) key guidance. Background Art

[0002] ML models of neural networks can model and learn fixed signal transformation functions. When there are multiple different signal transformations or in the case of continuous time-varying transformations, such static ML models tend to learn transformations such as suboptimal random averages. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a high-level block diagram of an example system configured with a trained key generator ML model configured to generate key parameters during inference and a trained audio synthesis ML model to perform dynamic key-guided signal transformation during inference based on the key parameters.

[0004] Figure 2A is a flow chart of an example first training process for jointly training a key generator ML model and an audio synthesis ML model to perform signal transformation.

[0005] Figure 2B is a flow chart of an example key generator ML model training operation that extends the first training process.

[0006] Figure 3A is a flow chart of an example first stage of a second training process for training a key generator ML model.

[0007] Figure 3B is a flow chart of an example second stage of a second training process for training an audio synthesis ML model based on the trained key generator ML model from the first stage.

[0008] Figure 4 is a block diagram of an example advanced communication system in which a key generator ML model and an audio synthesis ML model, once trained using a first training process, can be deployed to perform inference-phase key-directed signal transformation.

[0009] Figure 5 is a flow diagram of an example inference phase transmitter process performed in a transmitter of a communication system.

[0010] Figure 6 is a flow diagram of an example inference phase receiver process performed in a receiver of a communication system.

[0011] Figure 7 is a flow diagram of an example inference phase receiver process performed in a receiver including a key generator ML model and an audio synthesis model once trained using a second training process.

[0012] Figure 8 is a flow chart of an example method for performing key-directed signal transformation using a previously trained key generator ML model and an audio synthesis ML model to perform signal transformation.

[0013] Fig. 9 is a block diagram of an example computer device configured to implement the embodiments presented herein. DETAILED DESCRIPTION

[0014] Example Embodiments

[0015] Embodiments presented herein relate to a machine learning (ML) method or framework that jointly optimizes the generation of a relatively small amount of metadata (e.g., key parameters) and the synthesis of audio having desired audio characteristics based on the metadata. More specifically, a key generator ML model and an audio synthesis ML model are jointly trained for a specific / expected signal / audio transformation. During inference, the trained key generator ML model generates key parameters, and the trained audio synthesis ML model transforms input audio into desired output audio based on the key parameters. In an embodiment where the key parameters are transmitted from a remote transmission end to a local audio synthesis end, the key generator ML model has access to the input audio and the target audio, which guides the generation of the key parameters. In another embodiment where the key parameters are not transmitted from the remote end to the local audio synthesis end, the key generator ML model operates locally and has access only to the input audio.

[0016] A non-limiting example of signal transformation includes resolution enhancement of input audio in the form of pulse-code modulation (PCM) based audio. In this example, given a low-resolution / bandwidth representation and a high-resolution / bandwidth representation of an audio signal, the key generator ML model generates a size-restricted set of metadata (e.g., key parameters) for guiding the audio synthesis ML model. Given a size-restricted set of metadata (e.g., key parameters) and a low-resolution / bandwidth representation of an audio signal, the audio synthesis ML model synthesizes a high-resolution / bandwidth representation of the audio signal.

[0017] In one embodiment, two ML models (i.e., a key generator ML model and an audio synthesis ML model) may be trained jointly (i.e., concurrently and based on a total cost combining the individual costs associated with each ML model) and their corresponding training instances (i.e., inferences) may be deployed in different environments. For example, the key generator ML model inference may be deployed in an HD Radio transmitter, while the audio synthesis ML model inference may be deployed in an HD Radio receiver. In another embodiment, the ML models may be trained separately and sequentially, in which case their inference ML models may both be deployed in an HD Radio receiver. In either deployment arrangement, the HD Radio transmitter may transmit a compressed audio signal at a reduced rate, and the HD Radio receiver may synthesize a higher resolution / bandwidth audio signal from the compressed audio signal based on the key parameters. This represents a form of audio super-resolution.

[0018] refer to Figure 1 , which is a high-level block diagram of an example system 100 that is configured with a previously trained ML model (also referred to as a "neural network model" or simply a "neural network") to perform dynamic key-directed / key-based signal transformations. System 100 performs "inference phase" processing because that processing is performed by the ML model after it has been trained. System 100 is presented as a construction that is useful for describing concepts employed in the different embodiments presented below. Therefore, not all components and signals presented in system 100 are applicable to all of the different embodiments, as will be apparent from the subsequent description.

[0019] The system 100 includes a trained key generator ML model 102 (also referred to simply as "key generator 102") and a trained audio synthesis ML model 104 (also referred to as "audio synthesizer 104"), which can be deployed in a transmitter (TX) / receiver (RX) (TX / RX) system. In an example, the key generator 102 receives key generation data that may include at least an input signal and / or a target or desired signal. Based on the key generation data, the key generator 102 generates a set of transformation parameters KP, also referred to as "key parameters" or "key frame parameters" KP. The key generator 102 can generate the key parameters KP frame by frame or over a group of frames, as described below. The key parameters KP parameterize or represent desired / target signal characteristics of the target signal, such as spectral / frequency-based characteristics or temporal / time-based characteristics of the target signal. In one embodiment of the TX / RX system, the key generator 102 generates the key parameters KP at the transmitter TX and then transmits the key parameters KP to the receiver RX together with the input signal. At the receiver RX, the audio synthesizer 104 receives the input signal and the key parameters KP transmitted by the transmitter TX. The audio synthesizer 104 performs an expected signal transformation of the input signal based on the key parameters KP to generate an output signal having output signal characteristics similar to or matching the expected / target signal characteristics of the target signal. In another embodiment of the TX / RX system, the key generator 102 and the audio synthesizer 104 are both located at the receiver RX.

[0020] The key generator 102 and the audio synthesizer 104 include respective trained neural networks. Each neural network may be a convolutional neural network (CNN) including a series of neural network layers having convolutional filters with weights or coefficients configured based on a conventional stochastic gradient-based optimization algorithm. In another example, each neural network may be based on a recurrent neural network (RNN) model.

[0021] As mentioned above, the key generator 102 is trained to generate key parameters KP. The audio synthesizer 104 is trained to be uniquely configured by the key parameters KP to perform a dynamic key-guided signal transformation of an input signal to produce an output signal such that one or more output signal characteristics match or follow one or more desired / target signal characteristics. For example, the key parameters KP configure the audio synthesizer ML model to perform a signal transformation such that a spectral or temporal characteristic of the output signal matches a corresponding desired / target spectral or temporal characteristic of the target signal.

[0022] In an example where the input signal and the target signal include a corresponding sequence of signal frames (e.g., a corresponding sequence of audio frames), the key generator 102 generates key parameters KP frame by frame to produce a sequence of frame by frame key parameters, and the audio synthesizer 104 is configured by the key parameters to perform a signal transformation of the input signal to an output signal frame by frame. That is, due to the frame-specific key parameters used to guide the transformation of a given input frame, the audio synthesizer 104 generates a uniquely transformed output frame for / corresponding to each given input frame. Therefore, since the expected / target signal characteristics change dynamically from frame to frame and the generator key parameters representing the expected / target signal characteristics change accordingly from frame to frame, the key-guided signal transformation will change accordingly from frame to frame so that the output frame has signal characteristics that track the signal characteristics of the target frame. In this way, the key generator 102 and the audio synthesizer 104 jointly implement / perform a dynamic, key-guided signal transformation on the input signal to produce an output signal that matches the characteristics of the target signal over time.

[0023] In various embodiments, the input signal may represent a preprocessed input signal representing the input signal, and the target signal may represent a preprocessed target signal representing the target signal, such that the key generator 102 generates a key parameter KP based on the preprocessed input and the target signal, and the audio synthesizer 104 performs a signal transformation on the preprocessed input signal. In another embodiment, the key parameter KP may represent an encoded key parameter, such that the encoded key parameter configures the audio synthesizer 104 to perform a signal transformation on the input signal or the preprocessed input signal. Moreover, the input signal may represent an encoded input signal or an encoded preprocessed input signal, such that the key generator 102 and the audio synthesizer 104 each operate on the encoded input signal or the encoded preprocessed input signal. All of these and further variations are possible in various embodiments, some of which are described below.

[0024] By way of example, various aspects of the system 100, ML model inference processing, and training of ML models are now described in the context of an input signal and a target signal being respective audio signals (i.e., "input audio" and "target audio"). It should be understood that the embodiments presented herein are equally applicable to other contexts, such as contexts where the input signal and the target signal include respective radio frequency (RF) signals, images, videos, etc. In the audio context, the target signal may be, for example, a speech or audio signal sampled at 32kHz and buffered as, for example, 32ms frames corresponding to 1024 samples per frame. Similarly, the input signal may be a speech or audio signal, for example:

[0025] a. Either sample at the same sampling rate as the target signal (e.g., 32kHz), or sample at a different sampling rate (e.g., 16kHz, 44.1kHz, or 48kHz).

[0026] b. Buffer with either the same frame duration as the target signal (e.g., 32 ms) or a different duration (e.g., 16 ms, 20 ms, or 40 ms).

[0027] c. A bandlimited version of the target signal. For example, the target signal is a full-band audio signal including frequency components up to the Nyquist frequency, e.g., 16 kHz, while the input signal is bandlimited, having audio frequency components less than the target signal, e.g., up to 4 kHz, 8 kHz, or 12 kHz.

[0028] d. A distorted version of the target signal. For example, the input signal contains unwanted noise or a temporal / spectral distortion of the target signal.

[0029] e. There is no perceptible or understandable correlation with the target signal. For example, the input signal includes speech / dialogue and the target signal includes music; or the input signal includes music from instrument-1 and the target signal includes music from another instrument, and so on.

[0030] As mentioned above, each of the input signal and the target signal may be preprocessed to produce a preprocessed input signal and a preprocessed target signal on which the key generator 102 and the audio synthesizer 104 operate. Example preprocessing operations that may be performed on the input signal and the target signal include one or more of: resampling (e.g., downsampling or upsampling); direct current (DC) filtering to remove low frequencies, such as below 50 Hz; pre-emphasis filtering to compensate for spectral tilt in the input signal; and / or adjusting gain so that the input signal is normalized prior to subsequent signal transformation thereof.

[0031] Combination of the above Figure 1 The described inference phase process relies on each of the key generator 102 and the audio synthesizer 104 being pre-trained. Various training procedures may be employed to train the key generator 102 and the audio synthesizer 104. Figure 2A and 2B A first training process using a first key cost implementation is described. Figure 2A1 is a flow chart of a first training process. The first training process jointly trains the key generator 102 and the audio synthesizer 104 using various training signals. This means that the key generator 102 and the audio synthesizer are trained concurrently using a total cost that combines a first individual cost associated with the key generator / key parameters and a second individual cost associated with the audio synthesizer / signal transformation, as will be described more fully below.

[0032] The training signals include, for example, a training input signal (e.g., training input audio), a training target signal (e.g., training target audio), and training key parameters KPT (generated by key generator 102 during training and used to train audio synthesizer 104) having signal characteristics / characteristics generally similar to the input signal, the target signal, and the key parameters KP used for inference phase processing in system 100; however, the training signals are not the same signals as the inference phase signals. The training signals also include predetermined key constraints KC. Non-limiting examples of key constraints include the total number of bits allocated for transmission of key parameters in inference phase processing (e.g., the length of a vector representing the key parameters), and mutual orthogonality of key parameters (e.g., vectors). The first training process operates on a frame-by-frame basis, i.e., the training process operates on each frame of the input signal and the corresponding concurrent frame of the target signal.

[0033] At 202, the training process preprocesses an input signal frame to produce a preprocessed input signal frame. Example input signal preprocessing operations include: resampling; DC filtering to remove low frequencies, such as below 50 Hz; pre-emphasis filtering to compensate for spectral tilt in the input signal; and / or adjusting the gain so that the input signal is normalized before subsequent signal transformation. Similarly, at 204, the training process preprocesses a corresponding target signal frame to produce a preprocessed target signal frame. Target signal preprocessing may perform all or a subset of the operations performed by preprocessing of the input signal frame.

[0034] At 206, the (initial untrained) key generator 102 generates a set of key parameters KPT corresponding to the input signal frame based on the key constraints and one or more of the input signal frame and the target signal frame. Moreover, the key generator 102 uses / and / or calculates at least one key constraint cost KCC (i.e., a first cost) associated with the key parameters KPT and used to train the key generator, i.e., it will be minimized through training.

[0035] At 210, the (initial untrained) audio synthesizer 104 receives a preprocessed input signal frame and a key parameter KPT for the input signal frame. The key parameter KPT configures the audio synthesizer 104 to perform a signal transformation on the preprocessed input signal frame to produce an output signal frame. In addition, a cost optimizer CO for implementing cost back propagation (CBP) receives a preprocessed target signal frame, an output signal frame, and a key constraint cost KCC. The cost optimizer CO calculates an audio synthesizer output cost / error of the audio synthesizer 104, i.e., a second output cost associated with the signal transformation. The cost optimizer CO calculates a final cost based on the audio synthesizer output cost and the key generator key constraint cost KCC, for example, as a weighted combination of the two costs. The final cost is a back propagation of the cost gradients driving the final cost for (i) the learnable / trainable model parameters employed by the key generator 102, and (ii) the learnable / trainable model parameters employed by the audio synthesizer 104 ( Figure 2A The cost gradient for any particular model parameter further drives the update of that model parameter (e.g., the weight of the model) to minimize the associated cost during the joint training process.

[0036] In a first example, the cost optimizer CO can estimate the mean square error (MSE) or absolute error between the preprocessed target signal and the model output signal as the signal transformation cost. In a second example, assuming that the target signal and the model output signal can be in the time domain, the spectrum domain, or the key parameter domain, the cost optimizer CO can calculate a weighted combination of multiple errors estimated in the time domain, the spectrum domain, or the key parameter domain as the signal transformation cost. Based on the training examples and parameters described in this article, any known or later developed back-propagation technique can be used to train the two models. Figure 2B An example of calculating the key constraint cost KCC is described.

[0037] Operations 202-210 are repeated for successive input and corresponding target signal frames to train the key generator 102 to generate key parameters KPT that configure the audio synthesizer 104 to perform a signal transformation on the input signal so that the output signal characteristics of the output signal match the target signal characteristics for which the signal transformation is intended. The first training process jointly trains the key generator 102 and the audio synthesizer 104 simultaneously, frame by frame, and using back-propagation to minimize the total cost of the two ML models. Once the key generator 102 and the audio synthesizer 104 have been jointly trained on many frames of the input signal and the target signal, the trained key generator and the trained audio synthesizer can be deployed for inference phase processing of the (inference phase) input signal based on the (inference phase) key parameters. The following is combined with Figure 4-7 A further example of inference phase processing is described.

[0038] refer to Figure 2B , which is a flow chart of the training operation 210 extending the first training process, specifically with respect to calculating the key constraint cost KCC. For the purpose of training, the key generator 102 includes a key cost calculator 302 to calculate the key constraint cost KCC based on the key parameters KPT, the key constraints KC and the pre-processed target signal. In an embodiment assuming that the key parameters are configured as vectors, the key constraint cost implementation can be a measure of orthogonality between the key parameter vectors. In particular, assuming that each key parameter vector K m Include all N training data points (e.g., audio files) in a batch b The set of specific key parameters on data points, then

[0039]

[0040] Where K m is the mth key parameter vector, and N k is the number of key parameters to be used to adjust the audio synthesizer 104.

[0041] The normalized key parameter vector is defined as

[0042]

[0043] All N k The normalized parameter vectors, when collected together, form the normalized key parameter data matrix X, that is,

[0044]

[0045] The correlation matrix R is defined as

[0046]

[0047] The superscript T represents the matrix transpose, that is

[0048]

[0049] If the key vectors are orthogonal to each other, then the correlation matrix R will be the identity matrix. Therefore, the key constraint cost is expected to measure the deviation of R from the ideal identity matrix. The key constraint cost can then be expressed as the ratio of the Frobenius norm of the undesired off-diagonal elements of the correlation matrix R to the desired diagonal elements of the correlation matrix R, that is,

[0050]

[0051] Note that, by construction, for all n.

[0052] As mentioned above, the key constraint cost KCC is combined with the audio synthesizer cost to produce the final cost, which drives the back-propagation of the cost gradient.

[0053] Now combine Figure 3A and 3B The second training process is described. At a high level, the second training process includes first and second independent sequential training phases that do not share their respective costs. The first phase trains the key generator 102. The second phase then trains the audio synthesizer 104 using the trained key generator 102 from the first phase. Figure 3A 308 is a flowchart for the first stage of training the key generator 102. The input signal preprocessing operation 202 generates a preprocessed input signal as described above. The target signal preprocessing operation 204 generates a preprocessed target signal as described above. The key estimation operation 306 uses one or more predetermined signal processing algorithms (not ML processes) to estimate the target key parameters used as target / reference key parameters for training the key generator 102, that is, the key estimation algorithm derives the target key parameters. As described below, examples of target key parameters include line spectrum frequency (LSF) keys, harmonic keys, and time envelope keys. At 308, the key generator 102 is trained on the preprocessed input signal without access to the target signal, so that the key generator learns to generate a key parameter KPT that approximates the target key parameter. To this end, the cost optimizer CO calculates the error between the target key parameter and the key parameter KPT, and uses the error to drive the cost back propagation (CBP) to update the learnable parameters of the key generator 102 used to generate the key parameter KPT, thereby minimizing the error. In the example, the cost optimizer CO calculates the MSE or absolute error between the target key parameter and the key parameter KPT.

[0054] In order to estimate the target key parameters, the key estimation operation 306 (also referred to as the key estimator 306) can perform various different analysis operations on the input signal and / or the target signal to produce different sets of corresponding target key parameters. In one example, the key estimation operation 306 performs a linear prediction (LP) analysis on at least one of the target signal, the input signal, or an intermediate signal generated based on the target and input signals. The LP analysis produces LP coefficients (LPC) and LSFs, which generally compactly represent the wider spectral envelope of the underlying signal (i.e., the target signal, the input signal, or the intermediate signal). The LSFs compactly represent the LPCs, where they exhibit good quantization and frame-to-frame interpolation properties.

[0055] The LSF of the target signal (i.e., representing a reference or ground truth) is used as a good representation for the audio synthesizer 104 to learn or imitate the spectral envelope of the target signal (i.e., the target spectral envelope) and to apply a spectral transform to the spectral envelope of the input signal (i.e., the input spectral envelope) to produce a transformed signal (i.e., the output signal) having the target spectral envelope. Thus, in this case, the target key parameters represent or form the basis of a "spectral envelope key" including the spectral envelope key parameters. The spectral envelope key configures the audio synthesizer 104 to transform the input signal into an output signal such that the spectral envelope of the output signal (i.e., the output spectral envelope) matches or follows the target spectral envelope.

[0056] In another example, the key estimation operation 306 performs frequency harmonic analysis on at least one of the target signal, the input signal, or an intermediate signal generated based on the target signal and the input signal. The harmonic analysis generates a representation of a subset of dominant tone harmonics as a target key parameter, for example, it is present in the target signal as a target harmonic and is missing in or from the input signal. The key estimation operation 306 estimates the dominant tone harmonics using, for example, a search for spectral peaks or a sinusoidal analysis / synthesis algorithm. In this case, the target key parameters represent or form the basis of a "harmonic key" including the harmonic key parameters. The harmonic key configures the audio synthesizer 104 to transform the input signal into an output signal so that the output signal includes spectral features present in the target signal but not in the input signal. In this case, the signal transformation can represent a signal enhancement of the input signal to produce an output signal with a perceptibly improved signal quality, which can, for example, include frequency bandwidth extension (BWE). The above-mentioned LP analysis and harmonic analysis that produce the LSF are each examples of spectral analysis.

[0057] In yet another example, the key estimation operation 306 performs a temporal analysis (i.e., a time domain analysis) on at least one of the target signal or an intermediate signal generated based on the target signal and the input signal. The temporal analysis generates target key parameters as parameters that compactly represent, for example, the temporal evolution (e.g., gain variation) in a given frame, or a broad temporal envelope (generally referred to as a "temporal amplitude" characteristic) of the target signal or the intermediate signal. In the case of band-limitation and distortion, the temporal characteristics of the target signal (i.e., a reference or basic truth) serve as a good prototype for the audio synthesizer 104 to learn or imitate the temporal fine structure of the target signal (i.e., the desired temporal fine structure) and to apply such temporal characteristic transformations to the input signal. In this case, the target key parameters represent or form the basis for a "temporal key" including the temporal key parameters. The temporal key configures the audio synthesizer 104 to transform the input signal into an output signal so that the output signal has a desired temporal envelope.

[0058] Figure 3Bis a flow chart of the second stage of the second training process. The second stage follows the first stage and uses the trained key generator 102 to train the audio synthesizer 104. In the second stage, the input signal preprocessing 202 provides the preprocessed input signal to the trained key generator 102 and the initially untrained audio synthesizer 104. The target signal preprocessing 204 provides the preprocessed target signal to the cost optimizer CO. At 320, the key generator 102 operates in an inference mode to generate a key parameter KPT that approximates the above-mentioned target key parameter in response to the preprocessed input signal. At 322, the key parameter KPT configures the audio synthesizer 104 to perform a signal transformation on the preprocessed input signal to produce an output signal. The cost optimizer CO calculates the error between the output signal and the preprocessed target signal, and uses the error to drive the cost back propagation to the trainable parameters of the audio synthesizer 104, thereby minimizing the error. In the example, the cost optimizer CO calculates the MSE or absolute error between the preprocessed target signal and the output signal.

[0059] refer to Figure 4 , which is a block diagram of an example high-level communication system 400 in which a key generator 102 and an audio synthesizer 104, once trained according to a first training process, may be deployed to perform inference phase processing, for example. The communication system 400 includes a transmitter (TX) 402 in which the key generator 102 may be deployed and a receiver (RX) 404 in which the audio synthesizer 104 may be deployed. The key generator 102 of the transmitter 402 generates a key parameter KP (in Figure 4 The transmitter 402 combines the key parameters and the input signal into a bit stream and transmits the bit stream to the receiver 404 via a communication channel. The receiver 404 receives the bit stream from the communication channel and recovers the input signal and the key parameters from the bit stream. The audio synthesizer 104 transforms the input signal recovered from the bit stream based on the key parameters recovered from the bit stream to generate an output signal. Figure 5 as well as Figure 6 and 7 The inference phase processing performed in the transmitter 402 and the receiver 404 is described.

[0060] refer to Figure 5 , which is a flow chart of an example inference phase transmitter process 500 executed by transmitter 402 to produce a bitstream for transmission to receiver 404. Transmitter process 500 operates on a full set of signals (e.g., input signal, target signal, and key parameters KP) that have similar statistical characteristics as the corresponding training signals of training process 200.

[0061] At 502, the process preprocesses an input signal frame to produce a preprocessed input signal frame, and provides the preprocessed signal frame to the key generator 102. Similarly, at 504, the process preprocesses a target signal frame to produce a preprocessed target signal frame, and provides the preprocessed target signal frame to the key generator 102. For example, the preprocessing operations 502 and 504 may include similar operations to the corresponding preprocessing operations 202 and 204 described above.

[0062] At 506, the key generator 102 pre-trained according to the first training process generates key parameters KP (i.e., "key frame parameters") corresponding to the input signal frame based on the pre-processed input signal frame and the target signal frame. At 508, the process encodes the input signal frame to produce an encoded / compressed input signal frame (e.g., encoded input signal frame parameters). The encoding operation 508 may encode the input signal frame using any known or later developed waveform-preserving audio compression technique. At 510, a bitstream multiplexing operation multiplexes the encoded input signal frame and the key parameters for the input signal frame into a bitstream (i.e., a multiplexed signal) for transmission by the transmitter 402 over a communication channel.

[0063] refer to Figure 6 , which is a flow diagram of an example inference phase receiver process 600 performed by receiver 404. The operations of receiver process 600 perform their respective functions on a frame-by-frame basis, similar to the operations of transmitter process 500. Receiver process 600 receives a bitstream transmitted by transmitter 402. Receiver process 600 includes a demultiplexer-decoder operation 602 (also referred to simply as a "decoder" operation) to demultiplex the encoded input signal and key parameters from the bitstream, and decode the encoded input signal to recover a local copy / version of the input signal for the current frame (in Figure 6 referred to as the "decoded input signal" in ) and key parameters.

[0064] Next, an optional input signal preprocessing operation 604 preprocesses the input signal from the bitstream demultiplexer-decoder operation 602 to produce a preprocessed version of the input signal representative of the input signal. Based on the key parameters, at 606, the audio synthesizer 104 pretrained according to the first training process performs a desired signal transformation on the preprocessed version of the input signal to produce an output signal (at Figure 6 In an embodiment where the input signal preprocessing operation 604 is omitted, the audio synthesizer 104 performs the desired signal transformation directly on the input signal. The preprocessed version of the input signal and the input signal may each be more generally referred to as a "signal representing the input signal".

[0065] The receiver process 600 may also include an input-output mixing operation 610 to mix the pre-processed input signal with the output signal to produce the desired signal. The input-output mixing operation 610 may include one or more of the following operations performed frame by frame:

[0066] a. Constant-Overlap-Add (COLA) windowing, e.g., 50% skipping and overlap-adding of two consecutive windowed frames.

[0067] b. Mixing the output signal and a windowed / filtered version of the preprocessed input signal to generate a desired signal, the purpose of the mixing being to control the characteristics of the desired signal in the spectral overlap region between the output signal and the preprocessed input signal. Mixing may also include post-processing the output signal based on key parameters to control the overall tonality and noise in the output signal.

[0068] In summary, process 600 includes (i) receiving input audio and key parameters representing target audio characteristics, and (ii) configuring an audio synthesizer 104 that was previously trained to be configured by the key parameters with the key parameters so that the audio synthesizer performs a signal transformation of audio representing the input audio (e.g., either the input audio or a preprocessed version of the input audio) to produce output audio having output audio characteristics that match the target audio characteristics. The key parameters may represent target spectral characteristics as the target audio characteristics, and configuring includes configuring the audio synthesizer 104 with the key parameters so that the audio synthesizer performs a signal transformation of input spectral characteristics of the input audio to output spectral characteristics of the output audio that match the target spectral characteristics.

[0069] refer to Figure 7, which is a flow chart of an example inference phase receiving / decoding process 700 performed by a receiver, the receiver including a key generator 102 and an audio synthesizer 104, each trained according to a second training process, i.e., using a second key cost implementation. Process 700 employs some but not all operations of process 600. Process 700 receives a bitstream including an encoded input signal but not including key parameters. Decoding operation 702 decodes the encoded input signal in the bitstream to recover a local copy of the input signal, i.e., a decoded input signal. Optional input signal preprocessing operation 604 preprocesses the input signal to produce a preprocessed version of the input signal representing the input signal. At 705, the key generator 102 generates key parameters locally based on the input signal. At 606, the trained audio synthesizer 104 performs a desired signal transformation on the preprocessed version of the input signal (or the input signal) based on the key parameters to produce an output signal. Input-output mixing operation 610 mixes the preprocessed input signal with the output signal to produce a desired signal. Thus, the receive / decode process 700 synthesizes the key parameters locally (ie, at the receiver / decoder) during inference, rather than generating the key parameters at the transmitter / encoder and transmitting them in the bitstream as is done in the transmitter process 500 .

[0070] refer to Figure 8 , which is a flow chart of an example method 800 for performing key-directed signal transformation during inference phase processing using a key generator 102 (referred to as a “first neural network”) and an audio synthesizer 104 (referred to as a “second neural network”).

[0071] At 802, one or more of input audio and target audio having target audio characteristics are received. The input audio and the target audio may each include a sequence of audio frames.

[0072] At 804, key parameters are generated using a first neural network trained to generate key parameters that satisfy one or more predetermined key constraints and represent characteristics of the target audio based on one or more of the target audio and the input audio. The first neural network can generate the key parameters frame by frame to produce a sequence of frame by frame key parameters. The key constraints represent or indicate key costs, such as (1) mutual orthogonality of key vectors when the key parameters are generated from the target audio in a first training process, or (2) MSE matching of key parameters (e.g., LSF keys) estimated by the algorithm when the key parameters are generated from the input audio in a second training process.

[0073] At 806, the second neural network trained to be configured with the key parameters is configured with / using the key parameters so that the second neural network performs a signal transformation of the input audio to produce output audio having output audio characteristics that correspond to and match the target audio characteristics. That is, the signal transformation transforms the input audio characteristics into output audio characteristics that match or are similar to the target audio characteristics. The second neural network may be configured frame by frame by the sequence of frame-by-frame key parameters to transform each input audio frame into a corresponding output audio frame to produce the output audio as a sequence of output audio frames (one output audio frame for each input audio frame and each set of frame-by-frame key parameters).

[0074] Prior to operations 802-806, the first and second neural networks are trained using any of a variety of training processes. For example, the first training process jointly trains the first neural network and the second neural network to perform key generation and signal transformation, respectively, to minimize a combined cost derived based on a first cost associated with the key parameters and a second cost associated with the signal transformation. The combined cost is configured to drive back propagation of a cost gradient of the combined cost for each of the first neural network and the second neural network. The first cost measures mutual orthogonality between vectors representing the key parameters, and the second cost measures an error between the training target audio and the training output audio produced by the signal transformation.

[0075] In another example, the second training process includes first and second stages that are sequentially independent. The first stage trains the first neural network to minimize a first cost associated with a key parameter. That is, the first neural network is trained to generate a key parameter to approximate a target key parameter derived from a target signal by an algorithm so that the key parameter minimizes the error between the key parameter and the target key parameter (i.e., the first cost). Then, the second stage uses the trained first neural network to train the second neural network to minimize a second cost associated with a signal transformation (independent of the first cost).

[0076] refer to Fig. 9 , which is a block diagram of a computer device 900 configured to implement the embodiments presented herein. The computer device 900 has a variety of possible configurations and Fig. 9Just an example. Examples of computer device 900 include tablet computers, personal computers, laptop computers, mobile phones such as smart phones, etc. Moreover, computer device 900 can be incorporated into a transmitter or receiver described herein, such as an HD radio transmitter or receiver. Computer device 900 includes one or more network interface units (NIU) / radio transceivers 908, and memory 914, each of which is coupled to a processor 916. One or more NIU / radio transceivers 908 may include wired and / or wireless connection capabilities that allow processor 916 to communicate through a communication network. For example, NIU / radio transceiver 908 may include an Ethernet card that communicates through an Ethernet connection, a wireless RF transceiver that wirelessly communicates with a cellular network in a communication network, a radio transceiver configured to transmit and receive RF signals (e.g., for HD radio) through an air interface (I / F), an optical transceiver, etc., as recognized by a person of ordinary skill in the relevant art. Processor 916 receives sampled or digitized audio and provides digitized audio to one or more audio devices 918, as is known. The audio device 918 may include a microphone, a speaker, an analog-to-digital converter (ADC), and a digital-to-analog converter (DAC).

[0077] The processor 916 may include a collection of microcontrollers and / or microprocessors, for example, each configured to execute corresponding software instructions stored in the memory 914. The processor 916 may host / implement one or more ML models, including one or more of the key generator ML model 102 and the audio synthesis ML model 104. The processor 916 may be implemented in one or more programmable application specific integrated circuits (ASICs), firmware, or a combination thereof. Portions of the memory 914 (and the instructions therein) may be integrated with the processor 916. As used herein, the terms "acoustic," "audio," and "sound" are synonymous and interchangeable.

[0078] The memory 914 may include a read-only memory (ROM), a random access memory (RAM), a disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible (e.g., non-transitory) memory storage device. Thus, in general, the memory 914 may include one or more computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (by the processor 916), it is operable to perform the operations described herein. For example, the memory 914 stores or encodes instructions for the control logic 920 to implement a module configured to perform operations described herein related to one or both of the ML models, training of the ML models, operation of the ML models during inference, input / target signal preprocessing, input / target signal encoding and decoding, cost calculation, cost minimization, back-propagation, bitstream multiplexing and demultiplexing, input-output mixing (post-processing), and the above methods.

[0079] In addition, the memory 914 stores data / information 922 used and generated by the processor 916, including key parameters, input audio, target audio, and output audio, as well as coefficients and weights used by the ML model, etc.

[0080] In summary, in one embodiment, a method is provided, including: receiving input audio and target audio having target audio characteristics; generating key parameters based on one or more of the target audio and the input audio using a first neural network trained to generate key parameters representing the target audio characteristics based on one or more of the target audio and the input audio; and configuring a second neural network trained to be configured by the key parameters with the key parameters so that the second neural network performs a signal transformation of audio representing the input audio to produce output audio having output audio characteristics that correspond and match the target audio characteristics.

[0081] In another embodiment, a system is provided, comprising: a transmitter, including a radio transceiver, the radio transceiver coupled to a processor and configured to: receive input audio and target audio having target audio characteristics; generate key parameters based on one or more of the target audio and the input audio using a first neural network trained to generate key parameters representing the target audio characteristics based on one or more of the target audio and the input audio; encode the input audio into encoded input audio; and transmit the key parameters and the encoded input audio; and a receiver, including a radio transceiver, the radio transceiver coupled to the processor and configured to: receive the key parameters and the encoded input audio; decode the encoded input audio to recover the input audio; and configure a second neural network trained to be configured by the key parameters with the key parameters so that the second neural network performs a signal transformation of audio representing the input audio to produce output audio having output audio characteristics that correspond and match the target audio characteristics.

[0082] In another embodiment, a method is provided, comprising: receiving a bitstream comprising encoded input audio; decoding the encoded input audio to recover the input audio; generating key parameters based on the input audio using a first neural network trained to generate key parameters representing characteristics of the target audio based on one or more of the target audio and the input audio; and configuring a second neural network trained to be configured by the key parameters with the key parameters so that the second neural network performs a signal transformation of audio representing the input audio to produce output audio having output audio characteristics that correspond and match the characteristics of the target audio.

[0083] In yet another embodiment, one or more non-transitory computer-readable media are provided. The media are encoded with instructions that, when executed by a processor, cause the processor to: receive input audio and target audio having target audio characteristics; generate key parameters based on one or more of the target audio and the input audio using a first neural network trained to generate key parameters representing the target audio characteristics based on one or more of the target audio and the input audio; and configure a second neural network trained to be configured by the key parameters with the key parameters so that the second neural network performs a signal transformation of the audio representing the input audio to generate output audio having output audio characteristics corresponding to and matching the target audio characteristics.

[0084] In a further embodiment, one or more non-transitory computer-readable media are provided. The media are encoded with instructions that, when executed by a processor, cause the processor to: receive a bitstream including encoded input audio; decode the encoded input audio to recover the input audio; generate a key parameter based on the input audio using a first neural network trained to generate a key parameter representing characteristics of the target audio based on one or more of the target audio and the input audio; and configure a second neural network trained to be configured by the key parameter with the key parameter so that the second neural network performs a signal transformation of the audio representing the input audio to produce output audio having output audio characteristics corresponding to and matching the characteristics of the target audio.

[0085] In another embodiment, an apparatus is provided, comprising: a receiver including a radio transceiver coupled to a processor, configured to execute instructions stored in a memory to cause the processor to: receive a bitstream including encoded input audio; decode the encoded input audio to recover the input audio; generate key parameters based on the input audio using a first neural network trained to generate key parameters representing characteristics of the target audio based on one or more of the target audio and the input audio; and configure a second neural network trained to be configured by the key parameters with the key parameters so that the second neural network performs a signal transformation of audio representing the input audio to produce output audio having output audio characteristics that correspond and match the characteristics of the target audio.

[0086] While the technology herein is illustrated and described as being implemented in one or more specific examples, it is not intended to be limited to the details shown, since various modifications and structural changes may be made within the scope and range of equivalents of the claims.

[0087] Each of the claims presented below represents a separate embodiment, and embodiments that combine different claims and / or different embodiments are within the scope of the disclosure and will be apparent to those of ordinary skill in the art after reading this disclosure.

Claims

1. A method comprising: receiving input audio and target audio having target audio characteristics; generating a key parameter representative of a characteristic of the target audio based on one or more of the target audio and the input audio using a first neural network deployed at the radio transmitter, the first neural network being trained to generate the key parameter based on one or more of the target audio and the input audio; encoding the input audio into encoded input audio at the radio transmitter; sending key parameters and encoded input audio from a radio transmitter; receiving key parameters and encoded input audio at a radio receiver; decoding the encoded input audio at the radio receiver to recover the input audio; as well as configuring a second neural network deployed at the radio receiver that was trained to be configured with the key parameters with the key parameters so that the second neural network performs a signal transformation of the audio representative of the input audio to produce output audio having output audio characteristics that correspond and match the target audio characteristics; wherein generating the key parameters is performed at the radio transmitter; and Configuring a second neural network is performed at the radio receiver, wherein: The input audio and the target audio include corresponding audio frame sequences; Generating the key parameters includes generating the key parameters frame by frame; and Configuring the second neural network includes configuring the second neural network with the key parameters generated on a frame-by-frame basis so that the second neural network performs a signal transformation on a frame-by-frame basis to generate output audio as a sequence of audio frames.

2. The method of claim 1, wherein: Generating key parameters with the first neural network includes generating key parameters representing target characteristics of the target audio; and Configuring the second neural network includes configuring the second neural network with the key parameters so that the second neural network performs a signal transformation that is a transformation of input characteristics of the input audio to output characteristics of the output audio that matches the target characteristics.

3. The method of claim 1, wherein: The first neural network and the second neural network are jointly trained to minimize a combined cost including a first cost associated with the key parameter and a second cost associated with the signal transformation.

4. The method of claim 3, wherein the combined cost is configured to drive back propagation of cost gradients of the combined cost for each of the first neural network and the second neural network.

5. The method of claim 3, wherein: The first cost measures the mutual orthogonality between vectors representing key parameters; and The second cost represents the error between the training target audio and the training output audio produced by the signal transformation.

6. The method of claim 1, wherein receiving input audio comprises: receiving a bitstream comprising encoded input audio; as well as Decodes the encoded input audio to restore the input audio.

7. The method of claim 1, wherein: The first neural network is trained to generate a key parameter to approximate a target key parameter derived from the target audio by an algorithm such that the key parameter minimizes an error between the key parameter and the target key parameter.

8. The method of claim 1, wherein: Generating the key parameters using the first neural network includes generating the key parameters as spectral envelope key parameters including LP coefficients (LPC) or line spectral frequencies (LSF) representing a target spectral envelope of the target audio; and The configuring includes configuring the second neural network with the spectral envelope key parameters so that the second neural network performs a signal transformation that is a transformation of an input spectral envelope of the input audio to an output spectral envelope of the output audio that matches the target spectral envelope.

9. The method of claim 1, wherein: Generating the key parameters includes generating the key parameters as harmonic key parameters representing target harmonics present in the target audio; and The configuring includes configuring the second neural network with the harmonic key parameters so that the second neural network performs a signal transformation of the audio representing the input audio so that the output audio includes harmonics that match the target harmonics.

10. The method of claim 1, wherein: Generating the key parameters includes generating the key parameters as temporal key parameters representing target temporal characteristics of the target audio; and The configuring includes configuring the second neural network with the time key parameters so that the second neural network performs a signal transformation that is a transformation of input time characteristics of the input audio to output time characteristics of the output audio that matches the target time characteristics.

11. A system comprising: A transmitter comprising a radio transceiver coupled to a processor and configured to: receiving input audio and target audio having target audio characteristics; generating a key parameter representing a characteristic of the target audio based on one or more of the target audio and the input audio using a first neural network, the first neural network being trained to generate the key parameter based on one or more of the target audio and the input audio; encodes input audio into encoded input audio; as well as Transmit key parameters and encoded input audio; as well as A receiver comprising a radio coupled to a processor and configured to: Receive key parameters and encoded input audio; Decoding the encoded input audio to restore the input audio; as well as configuring a second neural network trained to be configured by the key parameters with the key parameters so that the second neural network performs a signal transformation of the audio representing the input audio to produce output audio having output audio characteristics that correspond and match the target audio characteristics, wherein: The input audio and the target audio include corresponding audio frame sequences; The first neural network is configured to generate a key parameter by generating the key parameter frame by frame; and The receiver is further configured to configure the second neural network by configuring it with the key parameters generated on a frame-by-frame basis so that the second neural network performs a signal transformation on a frame-by-frame basis to produce output audio as a sequence of audio frames.

12. The system of claim 11, wherein: The first neural network is configured to generate key parameters to represent target characteristics of the target audio; and The receiver is configured to configure the second neural network by configuring the second neural network with the key parameters so that the second neural network performs a signal transformation that is a transformation of input characteristics of the input audio to output characteristics of the output audio that matches the target characteristics.

13. The system of claim 11, wherein: The first neural network and the second neural network are jointly trained to minimize a combined cost including a first cost associated with the key parameter and a second cost associated with the signal transformation.

14. The system of claim 13, wherein the combined cost is configured to drive back propagation of cost gradients of the combined cost for trainable model parameters of each of the first neural network and the second neural network.

15. The system of claim 13, wherein: The first cost measures the mutual orthogonality between vectors representing key parameters; and The second cost measures the error between the training target audio and the training output audio produced by the signal transformation.

16. The system of claim 11, wherein the transmitter is further configured to: receiving a bitstream comprising encoded input audio; and Decodes the encoded input audio to restore the input audio.

17. The system of claim 11, wherein: The first neural network is trained to generate a key parameter to approximate a target key parameter derived from the target audio by an algorithm such that the key parameter minimizes an error between the key parameter and the target key parameter.

18. The system of claim 11, wherein: The transmitter is further configured to generate the key parameters as spectral envelope key parameters including LP coefficients (LPC) or line spectral frequencies (LSF) representing a target spectral envelope of the target audio; and The receiver is further configured to configure the second neural network with the spectral envelope key parameters so that the second neural network performs a signal transformation that is a transformation of an input spectral envelope of the input audio to an output spectral envelope of the output audio that matches the target spectral envelope.

19. The system of claim 11, wherein: The transmitter is further configured to generate the key parameter as a harmonic key parameter representing a target harmonic present in the target audio; and The receiver is further configured to configure the second neural network with the harmonic key parameters so that the second neural network performs a signal transformation of the audio representing the input audio such that the output audio includes harmonics that match the target harmonics.

20. The system of claim 11, wherein: The transmitter is further configured to generate the key parameter as a temporal key parameter representing a target temporal characteristic of the target audio; and The receiver is further configured to configure the second neural network with the time key parameters so that the second neural network performs a signal transformation that is a transformation of input time characteristics of the input audio to output time characteristics of the output audio that matches the target time characteristics.

Images