Inference time control of transducers for audio generation

Abstract

An audio system for synthesizing audio sounds with desired audio features executes an autoregressive audio generator, which is trained to generate audio by processing inputs using multiple layers employing multi-head attention, and uses directional inference temporal intervention (ITI) to push at least some outputs of at least some of the multi-head attention into a predetermined direction for the desired audio features.

Description

Technical Field

[0001] This disclosure generally relates to ITI training and / or using audio generation models, and more specifically, to adjusting the inference of a pre-trained audio generation model to generate audio with desired audio features. Background Technology

[0002] Introducing audio generation models with the ability to generate realistic sound clips on demand has the potential to revolutionize audio production. For example, the Autoregressive Generative Audio Transformer (AGAT) is a model architecture for generating audio waveforms. AGAT is based on a transformer architecture originally introduced for natural language processing tasks but adapted for a variety of sequence generation tasks, including audio synthesis. The autoregressive transformer processes discrete inputs (e.g., sequences of text tokens representing notes, phonemes, or text), where autoregression means that during inference, the model generates one or a set of tokens at a time, conditionally based on previously generated tokens. In addition to the autoregressive transformer, the AGAT model includes an encoder-decoder architecture, where the encoder converts the audio signal into a discrete token sequence before it is input to the autoregressive transformer, and the decoder converts the discrete audio tokens predicted by the transformer back into the audio signal. The autoregressive nature of the AGAT model allows for the generation of audio samples of variable length.

[0003] AGAT models can be trained using autoregressive likelihood estimation techniques such as teacher-forced or autoregressive sampling. These techniques involve training the model to predict the next audio sample given previous samples, then using the predicted sample as input to generate the next sample, and so on. However, training audio generation models is a difficult and expensive process. Currently, most existing generative audio transformers require custom hardware, large (often non-public) datasets, and significant financial and human resources for training.

[0004] Dynamic control of audio signals can be considered a fundamental task for most audio professionals. Musicians may need to precisely control their instruments, studio engineers manipulate various controls to achieve desired results, and smartphone users position their phone's microphone near the sound source and maintain appropriate input gain when shooting video.

[0005] The goal is to tune the audio generation model for dynamic control of synthesized audio. Furthermore, it is desirable to tune the audio generation model to generate audio with desired audio features without requiring costly retraining. Summary of the Invention

[0006] Some implementations aim to tune the audio generation model to produce audio with desired audio features without requiring retraining the model. Examples of audio features include adding or removing irritation from sounds, adding characteristics present in outdoor audio recordings, adding rhythmic sounds, and so on.

[0007] Alternatively or additionally, some embodiments aim to provide a system and method that allows control over the inference of an autoregressive generative audio transformer (AGAT) to generate audio with desired audio features. Alternatively or additionally, some embodiments aim to provide a system and method that enables dynamic control of the generative audio transformer for a variety of different audio features without requiring retraining the transformer's audio generation model.

[0008] AGAT is a deep neural network with multiple layers employing multi-head attention. Specifically, the generative audio transformer processes input audio lexical units across multiple layers, with its main computational component being multi-head self-attention blocks. Each head in the multi-head self-attention learns a different projection of the input, and the output of each head is concatenated before undergoing the remaining processing in the layers. These individual attention heads are computational blocks of the network that focus on different audio characteristics of the generated audio.

[0009] Some implementations are based on the understanding that desired audio features can be introduced into synthesized audio by pushing the outputs of at least some heads generated at some time instances into a predetermined orientation according to desired audio features. The orientation can be learned offline using data processing techniques such as machine learning and linear regression. Notably, different audio features are associated with different orientations, for example, determined individually and / or independently for at least some heads with multi-head attention. Each orientation of each audio feature can be applied to the head output individually or together with other orientations, allowing the generation of audio with a variety of desired characteristics.

[0010] Some implementations are based on the understanding that a desired directional push can be introduced by adding a bias to the output of the attention head. For example, since self-attention uses matrix operations, a bias vector can be used to offset the output of a layer or neuron in the self-attention block. This bias vector can be added element-wise to the output of the layer or neuron before applying the activation function. The bias vector for a specific audio feature can be learned to have a fixed, predetermined direction, but a fixed or variable length.

[0011] Temporal inference intervention (ITI) is a technique used in machine learning models to intervene in or modify the model's output during the inference phase based on certain criteria or conditions. ITI can be applied while the model is generating its predictions but before the final output is presented to the user or system. Some implementations are based on the recognition that ITI can be used to inject directional biases (e.g., bias vectors) into the output of the attention head, making some implementations of ITI directional ITI.

[0012] Some implementations are based on the understanding that, for some practical applications, it is undesirable to impose a directional bias on the output of all heads across all layers of the transformer at each time instance, i.e., to process each audio lexical. Due to the principle of the audio generation model employing AGAT, different heads have different sensitivities to different audio features. Furthermore, also because AGAT employs contextual generation of audio samples (i.e., previously generated samples or audio lexicals become the context of subsequently generated samples), if heads sensitive to audio features are always affected by the directional bias, the cumulative effect of directional ITI may have undesirable consequences.

[0013] To address this issue, some implementations employ spatiotemporal ITI, which uses spatiotemporal selection of heads to be injected with bias. The spatial dimension of spatiotemporal ITI prevents the injection of bias into all heads across all layers of the trained AGAT model and selects a subset of heads for the desired audio features. Notably, different subsets can be selected for different audio features. The temporal dimension of spatiotemporal ITI prevents the injection of bias into all heads at once. In this way, the negative impact of excessive bias interfering with the pre-trained audio generation model is reduced.

[0014] Some implementations are based on the understanding that heads with specific audio features can be selected in advance using data analysis of the head's output or dynamically during the inference phase of the audio system. For example, some implementations train a classifier in a supervised manner to classify the outputs of different heads as sensitive or insensitive to specific audio features. For example, some implementations generate positive and negative examples of audio features using an autoregressive audio transformer to examine the head's sensitivity to the audio features, and / or train a classifier to estimate the probability that the head's output has specific audio features. During the execution of the audio system, the classification of the head's output can be used to estimate the demand and extent of targeted ITI.

[0015] Different implementations use classification in a soft and / or hard manner. For example, some implementations do not apply a directional ITI when the probability that the head's output already has the desired audio features is higher than a threshold, i.e., more than 50%. Some implementations always apply a directional ITI, but the degree of application (e.g., the length of the bias vector) depends on the probability. For example, the length of the bias vector for an output with a 30% probability of having the desired audio features is greater than the length of the bias vector for an output with a 60% probability. Some implementations use a combination of these methods, applying a directional ITI to other outputs by not applying it to outputs that already have the desired audio features with a high probability, and by varying the length of the bias vector based on the probability of the classification.

[0016] Therefore, one embodiment discloses an audio system for synthesizing audio sounds with desired audio features, the audio system comprising: a processor coupled to a memory storing instructions for an executable module implementing the audio system, the module comprising: an input interface configured to collect input for synthesizing audio and data indicating desired audio features in the synthesized audio; an autoregressive generative audio transformer trained to generate audio by processing the input using multiple layers employing multi-head attention; an audio feature tuner configured to use directed inference temporal intervention (ITI) to push at least some outputs of at least some heads of the multi-head attention into a predetermined direction for the desired audio features; and an output interface configured to output the audio generated by the autoregressive generative audio transformer.

[0017] Another embodiment discloses a method for synthesizing audio sound having desired audio features, wherein the method uses a processor coupled with stored instructions for implementing the steps of the method, the method comprising the steps of: collecting input for synthesizing audio and data indicating desired audio features in the synthesized audio; executing an autoregressive audio generator trained to generate audio by processing the input using multiple layers employing multi-head attention; using directed inference temporal intervention (ITI) to push at least some outputs of at least some heads of the multi-head attention into a predetermined direction for the desired audio features; and outputting the audio generated by the autoregressive audio generator.

[0018] Many aspects of this disclosure can be better understood with reference to the following accompanying drawings. Components in the drawings are not necessarily drawn to scale; the emphasis is on clearly illustrating the principles of this disclosure. Furthermore, in the drawings, the same reference numerals throughout multiple views denote corresponding parts. While multiple embodiments are described in conjunction with these drawings, this disclosure is not limited to the embodiments disclosed herein. Rather, it is intended to cover all alternatives, modifications, and equivalents. Attached Figure Description

[0019] [ Figure 1A ]

[0020] Figure 1A A block diagram of an audio system for synthesizing audio sounds with desired audio characteristics, according to some embodiments, is shown.

[0021] [ Figure 1B ]

[0022] Figure 1B A schematic diagram of an autoregressive generative audio transformer (AGAT) according to some embodiments is shown, which is trained to generate audio by processing input using multiple layers employing multi-head attention.

[0023] [ Figure 1C ]

[0024] Figure 1C A block diagram is shown of a method for selecting and intervening in a specific attention head, which is sensitive to specific features, according to some embodiments.

[0025] [ Figure 1D ]

[0026] Figure 1D A block diagram of a method for soft classification according to some embodiments is shown.

[0027] [ Figure 1E ]

[0028] Figure 1E A block diagram of a method for hard classification according to some embodiments is shown.

[0029] [ Figure 1F ]

[0030] Figure 1F A general description of the implementation and audio converter according to some implementation methods is provided.

[0031] [ Figure 1G ]

[0032] Figure 1G The diagram illustrates the mapping between different elements according to some implementation methods.

[0033] [ Figure 2 ]

[0034] Figure 2 The audio processing method in the implementation is shown.

[0035] [ Figure 3 ]

[0036] Figure 3 A schematic diagram of an audio converter according to some embodiments is shown, which details the process of learning a classifier probe.

[0037] [ Figure 4 ]

[0038] Figure 4 A schematic diagram of a spatiotemporal ITI according to some implementations is shown.

[0039] [ Figure 5 ]

[0040] Figure 5 A schematic diagram illustrating the direction of intervention for a single attention head, according to some implementation methods, is shown.

[0041] [ Figure 6 ]

[0042] Figure 6 A schematic diagram illustrating the sparse intervention process used by some implementation methods is shown.

[0043] [ Figure 7 ]

[0044] Figure 7 A schematic diagram illustrating the probe monitoring weights used in some implementations is shown.

[0045] [ Figure 8A ]

[0046] Figure 8A A block diagram is shown of a method for calculating probe accuracy weights according to different implementations.

[0047] [ Figure 8B ]

[0048] Figure 8B A block diagram is shown of a method for calculating probe accuracy weights according to different implementations.

[0049] [ Figure 9A ]

[0050] Figure 9A This is a schematic diagram illustrating the probe dispersion weights used in some implementations.

[0051] [ Figure 9B ]

[0052] Figure 9B This is a schematic diagram illustrating the probe dispersion weights used in some implementations.

[0053] [ Figure 10 ]

[0054] Figure 10 An example of a text-based control user interface for enhancing the generation of an audio transducer using inference time intervention techniques, according to some implementations, is shown.

[0055] [ Figure 11 ]

[0056] Figure 11 A block diagram is shown illustrating an example of anomaly detection using the proposed inference time control according to some implementations.

[0057] [ Figure 12 ]

[0058] Figure 12 A computing system is shown that is suitable for implementing the various operating environments, architectures, processes, scenarios, and sequences discussed below with reference to the other figures. Detailed Implementation

[0059] Figure 1A A block diagram of an audio system 100 for synthesizing audio sounds with desired audio characteristics, according to some embodiments, is shown. The audio system 100 includes a processor coupled to a memory that stores instructions for executable modules implementing the audio system.

[0060] The audio system 100 includes the following modules: an input interface 110 configured to collect input for synthesizing audio and data indicating desired audio features in the synthesized audio; an autoregressive generative audio transformer 120 trained to generate audio by processing input using multiple layers employing multi-head attention; an audio feature tuner 130 configured to use directed inference temporal intervention (ITI) to push at least some outputs of at least some heads of the multi-head attention into a predetermined direction for the desired audio features; and an output interface 140 configured to output the audio generated by the autoregressive generative audio transformer.

[0061] System 100 allows adjustment of the audio generation model 120 to generate audio with desired audio features without retraining the audio generation model. Examples of audio features include adding a tingling sensation to the sound, adding characteristics present in outdoor audio recordings, adding rhythmic sounds, etc. Doing so in this way enables dynamic control of the generated audio transformer for a variety of different audio features without requiring retraining the audio generation model of the transformer.

[0062] Some implementations are based on the understanding that desired audio features can be introduced into synthesized audio by pushing the outputs of at least some heads generated at some time instances into a predetermined orientation according to the desired audio features. The orientation can be learned offline using data processing techniques such as machine learning and linear regression. It is noteworthy that different audio features are associated with different orientations. Each orientation of each audio feature can be applied to the head output alone or in combination with other orientations, allowing the generation of audio with a variety of desired characteristics.

[0063] Figure 1B A schematic diagram of an autoregressive generative audio transformer 120 (AGAT) according to some embodiments is shown. This AGAT is trained to generate audio by processing input through multiple layers employing multi-head attention. AGAT is a deep neural network having one or more layers (e.g., layers 151, 152, 153, and 154) employing multi-head attention. Specifically, the generative audio transformer processes input audio lexical units across multiple layers, and its main computational component is... Figure 1B The multi-head self-attention module is shown in circle 160. In multi-head self-attention, each head learns a different projection of the input, and the output of each head is concatenated before undergoing the rest of the processing within that layer. These individual attention heads are computational blocks of the network that focus on different audio characteristics of the generated audio.

[0064] Some implementations are based on the understanding that a desired directional push can be introduced by adding a directional bias 165 to the output of the attention head. For example, since self-attention uses matrix operations, a bias vector can be used to offset the output of the attention head in the self-attention block. This bias vector 165 can be added element-wise to the output of the attention head before the projection matrix that combines all the attention heads in the layer and the remaining nonlinear processing for that layer.

[0065] For different executions of transformer 120, a bias vector for a specific audio feature can be learned, giving it a fixed, predetermined direction. For example, for a first audio feature, direction 165 can be learned offline using, for example, machine learning and / or data-driven techniques. For different (e.g., second) audio features, direction 167 can be learned offline.

[0066] In various implementations, a direction is predetermined for each of at least some heads based on the desired audio characteristics, and thus the direction for each head can be different. For example, for the same desired audio characteristics, the predetermined direction 167 for head 157 can be different from the predetermined direction 169 for head 167.

[0067] It is worth noting that the implementation allows selection of one or more of a plurality of audio features, such as a first audio feature and a second audio feature. Pushes toward directions 165 and 167 can be applied to the outputs of different heads 155 and 156, or to the output of the same head 157. In these implementations, each of the audio features is associated with a set of directions predetermined for a set of heads, wherein the plurality of audio features includes a first audio feature associated with a first set of directions predetermined for a first subset of heads and a second feature associated with a second set of directions predetermined for a second subset of heads, wherein the first subset of heads is different from the second subset of heads, and wherein the first and second subsets of heads include a first head and a second head having different directions predetermined for the first and second audio features.

[0068] Temporal Inference Intervention (ITI) is a technique used in machine learning models to intervene in or modify the model's output based on some criteria or conditions during the inference phase. ITI can be applied while the model is generating its predictions but before the final output is presented to the user or system. Some implementations are based on the recognition that ITI can be used to inject directional biases (e.g., bias vectors) into the output of the attention head, making some implementations of ITI directional ITI.

[0069] Some implementations are based on the understanding that, for some practical applications, it is undesirable to impose a directional bias on the output of all heads across all layers of the transformer at each time instance, i.e., to process each audio word. Due to the principle of the audio generation model employing AGAT, different heads have different sensitivities to different audio features. Furthermore, also because AGAT employs contextual generation of audio samples (i.e., previously generated samples or audio words become the context of subsequently generated samples), if heads sensitive to audio features are always affected by the directional bias, the cumulative effect of directional ITI may lead to undesirable results.

[0070] To address this issue, some implementations employ spatiotemporal ITI, which uses spatiotemporal selection of heads to be injected with bias. The spatial dimension of spatiotemporal ITI prevents the injection of bias into all heads across all layers of the trained AGAT model and selects a subset of heads for the desired audio features. Notably, different subsets can be selected for different audio features. The temporal dimension of spatiotemporal ITI prevents the injection of bias into all heads at once. In this way, the negative impact of excessive bias interfering with the pre-trained audio generation model is reduced.

[0071] For example, in some implementations, the audio feature tuner 130 selects a subset of heads that produce outputs sensitive to desired audio features, and sparsely applies the directional ITI to the selected heads by skipping the application of the directional ITI to the selected heads at some time instances. For example, as Figure 1B As shown, a directional bias 165 is applied to the output of head 156, but not to the output of head 156 itself. Head selection in time and space can be accomplished deterministically, randomly, or in combination thereof. For example, in one implementation, the audio feature tuner uses deterministic selection and applies a directional bias ITI to the selected head unless the directional bias ITI has already been applied to a predetermined number of previous outputs of the selected head. In this way, in some implementations, the output of head 156 is modified using the bias before generating the audio sound, while in other implementations, the output remains unchanged.

[0072] Some implementations are based on the understanding that heads with specific audio features can be selected in advance, either through data analysis of the head's output or dynamically during the inference phase of the audio system. For example, some implementations train a classifier in a supervised manner to classify the outputs of different heads as sensitive or insensitive to specific audio features. For example, some implementations generate an audio transformer by running positive and negative examples of audio features to examine the head's sensitivity to those features and / or at least train a classifier for sensitive heads to classify the probability that the head's output has specific audio features. During the execution of the audio system, the classification of the head's output can be used to estimate the demand and extent of targeted ITI.

[0073] Figure 1C A schematic diagram is shown of a method for dynamically (i.e., during the execution of a generative audio transformer) selecting multi-head attention sensitive to specific audio features, according to some embodiments. In some embodiments, classifier 171 is trained for each of the audio features of interest and for each of the heads 170. For example, if the audio feature tuner is trained for 10 different audio features and the autoregressive generative audio transformer has 100 different heads across all layers, then the audio system 100 stores 1000 classifiers 171. In other words, there is a one-to-one mapping between heads and classifiers. Such classifiers are also referred to herein as classifier probes.

[0074] Classifier 171 classifies the output 170 of the corresponding head to estimate the probability 175 that the output already possesses the desired features. This probability can be used to determine 172 the sensitivity of the output 170 to specific audio features in order to select 173 a subset of heads that are sensitive to the audio features. Alternatively or additionally, this probability can be used to determine 174 the degree of directional push, such as the magnitude or length of the bias vector.

[0075] Different implementations use classification in a soft and / or hard manner. For example, some implementations do not apply a directional ITI when the probability that the head's output already has the desired audio features is higher than a threshold, i.e., more than 50%. Some implementations always apply a directional ITI, but the degree of application (e.g., the length of the bias vector) depends on the probability. For example, the length of the bias vector for an output with a 30% probability of having the desired audio features is greater than the length of the bias vector for an output with a 60% probability. Some implementations use a combination of these methods, applying a directional ITI to other outputs by not applying it to outputs that already have the desired audio features with a high probability, and by varying the length of the bias vector based on the probability of the classification.

[0076] Figure 1D A block diagram of a method for soft classification according to some embodiments is shown. In these embodiments, an audio feature tuner is configured to classify 181 the outputs of a multi-head attention head using a classifier to determine the probability 175 that the corresponding output has a desired audio feature, and to add a bias vector to each output of each head 182, wherein the direction of the bias vector is predetermined for the desired audio feature, and wherein the length of the bias vector is a function of the probability 175.

[0077] Figure 1E A block diagram of a method for hard classification according to some embodiments is shown. In these embodiments, an audio feature tuner adds a bias vector 182 to each output of each head with a probability that the desired audio feature is less than a predetermined threshold 183.

[0078] In this way, different implementations can introduce a self-monitoring process into inference-time intervention operations, thereby applying intervention when the learned classifier probes indicate the need for intervention based on the state of the generative network. This self-monitoring technique enables real-time evaluation of whether the currently generated samples contain the target intervention factor, allowing for the generation of non-divergent audio samples without costly retraining or fine-tuning processes. Given that most existing generative audio transformers require custom hardware, large (often non-public) datasets, and significant financial and effort investment in training, the ability to create custom controls for these models without retraining is crucial. Furthermore, these custom controls can be created from a small number of audio examples (e.g., approximately 10), regardless of whether they possess the desired audio features. Given that the text descriptions used in text-to-audio models can already be lengthy, adding additional words about the custom controls may not be sufficient. Moreover, adding text does not allow for “intensity” control over a given feature of the generative model; therefore, the intervention technique presented in this paper is needed to obtain this fine-grained control.

[0079] Figure 1FGeneral descriptions of the execution of an audio transformer according to some implementations are provided (191 and 192). In some implementations, the audio transformer generates an audio signal 190 based on a large pre-trained model that generates the signal from a text description. Training such a generative model is difficult due to cost, hardware, and data limitations; therefore, the model weights are frozen, as indicated by the snowflake label 195. The audio transformer operates in an autoregressive manner similar to a large language model (LLM), but instead of generating text tokens at each time step, it generates audio tokens representing frames of audio (i.e., hundreds of samples representing fractions of a second). The two time steps t 191 and t+1 192 are as follows... Figure 1F As shown. In order to impose fine-grained control on the audio converter without retraining, the implementation intervenes in different intermediate outputs of the audio generation model.

[0080] An intervention adds a directional bias of 196 to the output of a certain operation in the neural network to change the overall behavior of the network. To ensure that the applied intervention does not impair the overall quality of the generated music, some implementations monitor the output at the same location where intervention 193 is applied, and then, based on the monitored output, apply intervention 194 only if necessary. The decision on where to apply the intervention and where to monitor in the network is based on probes 197, which are simple linear classifiers trained using a dataset of a small number of audio samples, some of which contain desired features (e.g., samples of music with and without specific instruments). Applying an intervention throughout the network could lead to unwanted forgetting, and much of the knowledge learned during transformer training could be lost. The probes identify only those regions in the transformer that are most sensitive to a given feature based on this small set of audio examples. Then, intervention is performed only in those network regions identified by the probes, and the success of the intervention is monitored.

[0081] Figure 1G Mapping between different elements is illustrated according to several implementations. In these implementations, the head 186 of the multi-head attention is uniquely mapped to a direction 187 for each of the different audio features 185. Alternatively, the head 186 may be mapped to an offline, pre-determined classifier 188 to define the probability that the direction 187 and / or the output of the head 186 have the desired audio feature 185. For example, as described below, the direction 11 of the classifier 11 can point from a negative example of the output of the head 01 to a positive example.

[0082] Figure 2A block diagram illustrating the operational principle of a text-to-audio transformer employed in some implementations is shown. The text-to-audio transformer 230 functions similarly to an LLM, but instead of generating words autoregressively, it uses multi-head cross-attention 220 to generate discrete audio words 240 from a text cue 210. These audio words 240 are then converted into an audio waveform 260 using an audio decoder 250. The audio decoder typically consists of a series of transposed convolution operations that expand discrete embedding vectors into an audio signal. In some implementations, 50 audio words 240 are generated for one second of audio signal 260. Unlike an LLM where both input and output are text, in a text-to-audio model, the text cue input 210 is integrated into the transformer that generates audio words through a cross-attention process 220, which weights words in the text cue according to their importance to the currently generated audio words 240.

[0083] Figure 3 A schematic diagram of an audio transformer according to some embodiments is shown, detailing the process of learning a classifier probe. The binary dataset comprises a collection of audio examples having 310 and not having 315 specific audio features. The embodiment then extracts each audio file from the binary dataset and passes it through a convolutional encoder 320 to convert it into a sequence 330 of discrete input audio words. These input audio words are then fed as input to a transformer 350, which consists of multiple layers, the main computational component of which is a multi-head self-attention block. Each head in the multi-head self-attention learns a different projection of the input, and the output of each head is concatenated before undergoing the remaining processing in the layers to produce output audio words 340.

[0084] Each attention head (such as heads 361, 362, and 363) is a network computation block for implementing interventions and monitoring their success. To this end, some implementations train a simple linear classifier probe on the output of each attention head in each layer. For example, to train a classifier for head 363, specifically, the implementation collects the attention head output for the last element of the input audio lexical sequence for each audio file in the binary data set and learns a linear classifier 365 to distinguish between real and fake audio files in the binary data set 370. Then, based on the extent to which the probe for each attention head can classify audio features from the binary data set, the implementation only intervenes on those attention heads whose probes have sufficient classification accuracy. This allows the trained transformer to maintain its overall performance but allows for precise interventions that push the output in a given direction.

[0085] Figure 4A schematic diagram of a spatiotemporal ITI according to some implementations is shown. Implementations add a bias term 440 to the output 430 of each or at least some attention heads, where the bias term is a vector containing a product 450 of direction 410 (intervention direction) and length (intervention weight) 420. In different implementations, the intervention weight 420 is calculated as a product of up to five different component weights, such as sparse intervention weight 421, probe dispersion weight 422, probe accuracy weight 423, probe monitoring weight 424, and total intensity component weight 425.

[0086] In some implementations, the length 420 of the bias vector for the directional ITI is determined based on the offline component 427, the online component 428, and a hybrid component 429, which can be predetermined offline or estimated online. The offline component is based on a classifier trained for desired audio features to drive the output of a particular head. In fact, in some implementations, the audio feature tuner is configured to classify the output of a multi-head attention head using a classifier trained in a supervised manner for the corresponding head to determine the probability that the corresponding output has the desired audio features, thus establishing a one-to-one mapping between the head, the classifier, and the desired audio features. Weights 420 are used to determine the length of the bias vector to be added to each output of each head, where the width of the bias vector is a function of probability. However, some weights can be determined online based on the performance of the classifier, i.e., based on the probability that the classifier classifies the head's output as carrying the desired audio features. Examples of weights that influence the performance of the classifier and thus form the offline component 427 include probe dispersion weights 422 and probe accuracy weights 423.

[0087] However, some other weights are determined online, for example, based on user input or any other desired characteristics of the audio system performance. For instance, the total intensity 425 controls the total amount of intervention applied to the audio generation model. In practice, the user will control this value using a knob or slider. The knob or slider determines the total intensity parameter, which serves as a parameter for a function to be applied to all attention heads to tune the intensity of the applied intervention. In some implementations, the same function is applied to all attention heads, and the total intensity parameter can be a scalar value multiplied by the intervention weights of all heads, or it can be an exponent that increases the intervention weights of all heads to the power specified by the total intensity parameter. Probe monitoring weight 424 monitors online all attention heads identified by their corresponding probes as sensitive to the features we are trying to control. If the probes show that the feature is already present in the generated audio, the implementation does not apply intervention, thereby reducing the risk of intervention impairing the quality of the pre-trained transformer. This is in Figure 7 As shown in the image.

[0088] The probe dispersion weight 422 assumes that some attention heads have tightly clustered data for a given feature, while others are more dispersed across the probe training samples. Some implementations assign head-specific weights based on the dispersion of the probe training data, assigning weights related to... Figure 9A and Figure 9B This will be described in more detail. The probe accuracy weight 423 controls the application of higher weights to the attention head that the learned probes can accurately classify given audio features. Some implementations consider both hard and soft probe accuracy weighting strategies, such as... Figure 8A and Figure 8B As shown.

[0089] The sparse intervention weights 421 account for the rate at which interventions are skipped and can be predetermined offline or varied online during generation. Since the generative music transformer model uses an autoregressive generation process (i.e., the output of previously generated samples becomes the input context for future generated samples), if ITI intervenes on each generated sample, the output context for subsequent generated samples will be overwhelmed by the intervened samples, potentially harming the quality of the pre-trained model. This is in... Figure 6 As shown in the image.

[0090] Figure 5 A schematic diagram of the intervention direction of a single attention head according to some implementations is shown. The binary training dataset is represented as a circle for positive examples 510 and x for negative examples 520. The intervention direction 530 is then an orthogonal (i.e., perpendicular) direction to the boundary 560 of the learned linear classifier that separates the positive and negative probe training data. During directional ITI, the implementation uses a bias vector 570 with direction 530 and weights 550 to move the output 540 of the sample attention head toward the positive training data.

[0091] Figure 6 A schematic diagram illustrating a sparse intervention process used by some implementations is shown. By reducing the frequency of intervention across time steps, the implementation allows the generation process to be better aligned with the underlying rhythm and temporal structure of the generated audio. Figure 6 In the implementation of the method, the intervention weight is each s Each time step is non-zero 610 once, where s These are user-selected hyperparameters. In this way, the audio feature tuner applies the directional ITI to the selected head 620 unless the directional ITI has already been applied to a predetermined number of previous outputs to the selected head. In this example, the hyperparameters... s The value is 3, therefore the directional ITI is applied to head 620.

[0092] Figure 7A schematic diagram illustrating probe monitoring weights used by some implementations is shown. If the overall probe accuracy calculated with respect to the currently generated audio is already higher than a pre-specified threshold of 710, the probe monitoring weight is set to zero. Otherwise, the probe monitoring weight is proportional to the distance from the threshold; that is, if the total probe accuracy is much lower than the threshold, the probe monitoring weight is larger. In some implementations, the total probe accuracy is calculated as the median accuracy of the top K attention heads identified as having a specific audio feature. Additionally or alternatively, some implementations also calculate the threshold dynamically; that is, if learning to classify the probe training dataset is generally more difficult, these implementations lower the threshold compared to audio features that are easier to classify by the individual attention head probes.

[0093] Figure 8A and Figure 8B A block diagram is shown illustrating methods for calculating probe accuracy weights according to different implementations. The aim of each implementation is to select an attention head that performs better in classifying a particular audio feature as having a higher intervention weight, and vice versa. Different implementations use soft 810 weighting and hard 820 weighting methods. The soft weighting method normalizes the classification accuracy 830 of each attention head 833 to the range of zero to one, such that the most accurate attention head in the entire model has a weight of 1, and the least accurate attention head has a weight of 0. Optionally, some implementations use a modulation intensity 835 greater than 1. Then, we can more than double the weight of each normalized probe accuracy to further increase the weight of accurate heads and decrease the weight of less accurate heads.

[0094] In contrast, the hard-weighted method 820 assigns predetermined weights (e.g., weight 1) 845 to a predetermined number of heads that have the most accurate classification.

[0095] Figure 9A and Figure 9B This is a schematic diagram illustrating the probe dispersion weights used by some implementation methods. The weights are calculated as the standard deviation of the binary probe training dataset (positive and negative samples) along the direction of intervention. Figure 9A In the example, most of the dispersion in the data is along the direction of the classifier boundary 920, while the dispersion 930 along the intervention direction 910 is quite low. Therefore, the probe dispersion weight with a smaller attention head is given. Figure 9B In the example, the data is more dispersed by 950 degrees along the intervention direction, meaning that a larger probe dispersion weight will be given to that particular attention head. Probe dispersion weights help make the intervention process more adaptable across different heads and layers of networks with variable behavior.

[0096] Figure 10An example of a text-based control interface for enhancing a generative audio transducer using inference-time intervention techniques, according to some implementations, is shown. First, it is noteworthy that the interface allows multiple interventions to be applied simultaneously. The example provides intervention knobs for three audio features: (1) Reality 1010, i.e., whether the audio sounds like a sound recorded by a microphone or synthesized by an algorithm; (2) Add Rhythmic Sounds 1020, i.e., adding sounds such as drumbeats or thumping; and (3) Itching 1030, high-frequency sounds such as fingernails scratching a blackboard. When the knobs are turned, Figure 4 The “total strength” 425 of the intervention weight 425 described in the text changes. This allows the user to change the output of the generation process online.

[0097] Figure 11 A block diagram illustrating an example of anomaly detection using the proposed inference time control according to some implementations is shown. In this example, directional ITI helps build a more accurate model for detecting the sounds of broken machines in a factory. Specifically, collecting the sounds of anomalous or malfunctioning machines is challenging because it requires intentionally damaging expensive machinery. However, these implementations use a generative model 1110 to generate synthetic anomalous machine sounds 1120 using text and inference time intervention controls, using knowledge of the physical processes of the machines to generate synthetic anomalous sounds to build a probe dataset, and generating a large amount of data by varying the total intervention intensity, but ensuring the data sounds realistic by including the monitoring proposed herein. The implementations then train a classifier 1140 to distinguish between normal machine sounds 1130 and machine sounds combined with realistic synthetic anomalous items 1120 generated by model 1110.

[0098] Example Implementation: Self-Monitoring Inference Time Intervention

[0099] Autoregressive Transformer Model

[0100] Architectures such as MusicGen form the basis of existing music generation models. These models are characterized by an autoregressive generation of a series of audio frames using a transformer model. These models include... A set of multi-head self-attention layers (typically employing residual connections, normalization layers, and fully connected feedforward layers, which will not be described here). At the current time step of generating the sequence. , No. Each self-attention layer is based on the input vector calculate A self-attention head for: (1) in, , , and The query, key, and value projection matrix representing a specific header all belong to... ,and This represents the attention operator. Output vector. It is done by projecting each head back The result is obtained by multidimensional space and summation: (2) in, It is a projection matrix.

[0101] Inference Time Intervention (ITI)

[0102] By intervening (i.e., modifying) the computation of the output of the multi-head self-attention layer in Equation (2), the model's output can be controlled to some extent. This intervention is achieved by modifying the computation of the multi-head self-attention layer before projection. This was accomplished by adding an item: (3) in, To indicate A head-specific vector of ITI "direction" in a head-specific space. yes Head-specific normalization (also known as probe dispersion weighting). It is the system-wide ITI strength, and It is a specific intensity weighting of the head.

[0103] The head-specific ITI orientation is obtained through a classifier probe. Some implementations are based on the recognition that the lateral detection dataset can be used to train a classifier that reflects our ITI objective (e.g., if the objective is to add drums, then a drum classifier is trained). Exemplary implementations run the dataset through our (frozen) generative network and for all... Collection of heads For each and The implementation method then uses parameters Different logistic regression classifier probes are trained, and their predictions are obtained as follows: Once training is complete, the implementation method will be directed towards ITI. Set as final and will Set to target all data obtained on combined probe training and test data of Standard deviation. This implementation also notes the accuracy of the final classifier on the probe test data. In the case of MusicGen, the implementation method applies a generative model for probing in an unconditional generative mode because the probing dataset generally lacks text queries of its audio samples.

[0104] In the implementation, accuracy is detected by the classifier. The aspect found the front Size, their Set to 1, and set the other heads to 1. Set to 0 to set the header-specific weight. For example, the most efficient method can be found through hyperparameter grid search. and .

[0105] Sparse intervention

[0106] For example, for tasks involving continuous audio, implementations have observed that ITIs often lead to overly abrupt and musically unnatural changes (see Section 3.3). As a mitigation, implementations propose reducing the ITI frequency across time steps, potentially allowing the generation process to better align with the underlying rhythmic structure of the generated music. This corresponds to adjusting the ITI weights in Equation (3). Replace with time-varying weights This time-varying weight only applies to Non-zero, where, This is an example of intervention start time. Indicates the number of steps between each ITI (e.g., ITI is executed every 5 time steps. of The value can be set using another criterion, such as based on the previous... The original standard for size.

[0107] soft weighting

[0108] One limitation addressed by the implementation method is the number of headers to be executed for the ITI. (Right now, The opposite The quantity of probes is used as a hyperparameter for tuning. The implementation also proposes a method based on the accuracy of the collected probes. We propose a soft-weighted method without hyperparameters and show that it is sufficient to perform efficient ITI. In fact, we propose setting the weights to... (4) and It applies to all and Minimum and maximum accuracy of the records. It is a power factor, allowing the relative weights of the heads to be modulated using accurate and inaccurate classifier probes (we use...) Through construction, ensure It falls between 0 and 1.

[0109] Automatic intervention modulation through self-monitoring

[0110] The implementation method is expected to be most effective for systems capable of modulating ITI intensity because it has time invariance. The system does not use the state of the inference model during the generation process (and therefore cannot react to it). For this reason, the implementation uses a classifier probe to drive this modulation. The implementation first classifies the previous state by the accuracy of the probe classifier. Size The set is defined as For each new generation time step, the implementation collects data for... The set of output predictions from all classifier probes in the classifier. Then, intervention was carried out on it, that is, (5) The statistical data reflect the probe's confidence in the intervention's success. By comparing them with the baseline accuracy obtained from the probe's training data, we can design a modulation scheme to update... These implementation methods calculate the median. And the amount of change in the median due to the previous generation steps involving intervention. .

[0111] Then, these implementations are based on the same head The accuracy of probes for detecting training data set The threshold is defined using the median and standard deviation. (6) Set the threshold to .

[0112] Then, define the update algorithm as follows: For ITI The first generation time step is set according to equation (4). and set , For ITI For each subsequent generation time step, we have 3 cases: if and Then we set (7) (b) If but Then reset to the initial value. (8) (c) If Then we set .

[0113] Figure 12 A computing device 1201 is illustrated, representing any system or collection of systems in which the various processes, programs, services, and scenarios disclosed herein can be implemented. Examples of the computing device 1201 include, but are not limited to, desktop and laptop computers, tablet computers, mobile computers, audio devices, and wearable devices. Examples may also include server computers, web servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, container, and any variations or combinations thereof.

[0114] The computing device 1201 can be implemented as a single device, system, or apparatus, or it can be implemented in a distributed manner as multiple devices, systems, or apparatuses. The computing device 1201 includes, but is not limited to, a processing system 1202, a storage system 1203, software 1205, a communication interface system 1207, and a user interface system 1209. The processing system 1202 is operatively connected to the storage system 1203, the communication interface system 1207, and the user interface system 1209.

[0115] Processing system 1202 loads and executes software 1205 from storage system 1203. Software 1205 includes and implements the audio generation process 1206 representing the above-described embodiments. When executed by processing system 1202, software 1205 instructs processing system 1202 to operate as described herein, at least for the various processes, operating scenarios, and sequences discussed in the above implementations. Computing device 1201 may optionally include additional means, features, or functions not discussed for the sake of brevity.

[0116] Still refer to Figure 12 The processing system 1202 may include a microprocessor and other circuitry that retrieves and executes software 1205 from the storage system 1203. The processing system 1202 may be implemented within a single processing device, or it may be distributed across multiple processing devices or subsystems that collaboratively execute program instructions. Examples of the processing system 1202 include general-purpose central processing units, graphics processing units, digital signal processors, dedicated processors, and logic devices, as well as any other type of processing device, combinations thereof, or variations thereof.

[0117] Storage system 1203 may include any computer-readable storage medium that can be read by processing system 1202 and is capable of storing software 1205. Storage system 1203 may include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual and non-virtual memory, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other suitable storage medium. In any case, computer-readable storage media are not propagated signals.

[0118] In addition to computer-readable storage media, in some implementations, storage system 1203 may also include a computer-readable communication medium through which at least a portion of software 1205 can be transmitted internally or externally. Storage system 1203 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems located in the same or distributed location relative to each other. Storage system 1203 may include additional elements, such as a controller, capable of communicating with processing system 1202 or possibly other systems.

[0119] Software 1205 (including audio generation process 1206) can be implemented in program instructions and, among other functions, when executed by processing system 1202, can instruct processing system 1202 to operate according to various operating scenarios, sequences, and processes described herein. For example, software 1205 may include program instructions for implementing the inference and training processes described herein.

[0120] Specifically, program instructions may include various components or modules that cooperate or otherwise interact to perform the various processes and operational scenarios described herein. These components or modules may be embodied in compiled or interpreted instructions, or in other variations or combinations thereof. These components or modules may execute synchronously or asynchronously, serially or in parallel, in a single-threaded environment or in multiple threads, or according to any other suitable execution paradigm, variation, or combination thereof. Software 1205 may include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. Software 1205 may also include firmware or some other form of machine-readable processing instructions executable by processing system 1202.

[0121] Generally, when software 1205 is loaded into and executed by processing system 1202, it can transform a suitable device, system, or apparatus (represented by computing device 1201) from a general-purpose computing system into a dedicated computing system tailored to perform inference and / or training in an optimized manner. In fact, the coded software 1205 on storage system 1203 can transform the physical structure of storage system 1203. The specific transformation of the physical structure can depend on various factors in different implementations of this specification. Examples of such factors may include, but are not limited to, the technology used to implement storage system 1203, whether the computer storage medium is characterized as primary or secondary storage, and other factors.

[0122] For example, if the computer-readable storage medium is implemented as a semiconductor-based memory, then when program instructions are encoded therein, the software 1205 can transform the physical state of the semiconductor memory, for example, by transforming the state of the transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. Similar transformations may also occur with magnetic or optical media. Other transformations of the physical medium are possible without departing from the scope of this specification; the examples provided above are merely to facilitate this discussion.

[0123] The communication interface system 1207 may include communication connections and devices that allow communication with other computing systems (not shown) via a communication network (not shown). Examples of connections and devices that allow inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices may communicate via a communication medium to exchange communication with other computing systems or networks of systems (such as metal, glass, air, or any other suitable communication medium). The aforementioned media, connections, and devices are well-known and do not require detailed discussion here.

[0124] Communication between computing device 1201 and other computing systems (not shown) may be conducted through one or more communication networks and in accordance with various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variations thereof. The aforementioned communication networks and protocols are well-known and do not require detailed discussion here.

[0125] Those skilled in the art will understand that aspects of the invention can be embodied as systems, methods, or computer program products. Therefore, aspects of the invention can take the form of entirely hardware implementations, entirely software implementations (including firmware, resident software, microcode, etc.), or implementations combining software and hardware aspects, which are generally referred to herein as “circuit,” “module,” or “system.” Furthermore, aspects of the invention can take the form of computer program products embodied in one or more computer-readable media, on which computer-readable program code is embodied.

[0126] In fact, the included description and figures depict specific embodiments to teach those skilled in the art how to make and use the best mode. For the purpose of teaching the inventive principles, some conventional aspects have been simplified or omitted. Those skilled in the art will understand variations of these embodiments that fall within the scope of this disclosure. They will also understand that the above features can be combined in various ways to form multiple embodiments. Therefore, the invention is not limited to the specific embodiments described above, but is limited to the claims and their equivalents.

Claims

1. An audio system for synthesizing audio sounds having desired audio characteristics, the audio system comprising: A processor, connected to a memory storing instructions for an executable module implementing the audio system, the module comprising: An input interface configured to collect inputs for synthesizing audio and data indicating desired audio features in the synthesized audio; An autoregressive audio generator is trained to generate audio by processing the input using one or more layers employing multi-head attention. An audio feature tuner configured to use directional inference temporal intervention (ITI) to push at least some outputs of at least some heads of the multi-head attention into a predetermined direction for each of the at least some heads based on the desired audio features; and An output interface configured to output the audio generated by the autoregressive audio generator.

2. The audio system according to claim 1, wherein, The directional ITI adds a bias vector pointing in a predetermined direction to the output of the head before applying the activation function of the neurons in the head to the output of the head.

3. The audio system according to claim 1, wherein, The input interface is configured to accept a plurality of audio features, each of which is associated with a set of directions predetermined for a set of heads. The plurality of audio features includes a first audio feature associated with a first set of directions predetermined for a first subset of heads and a second audio feature associated with a second set of directions predetermined for a second subset of heads. The first subset of heads is different from the second subset of heads. The first subset of heads and the second subset of heads include a first head and a second head having different directions predetermined for the first audio feature and the second audio feature.

4. The audio system according to claim 3, wherein the audio system further comprises: A user interface configured to enable selection of one or more of the plurality of audio features, wherein, when selecting a first feature or a second feature, the audio feature tuner is configured to push the output of a first subset of the head into a corresponding direction in a first set of directions, or push the output of a second subset of the head into a corresponding direction in a second set of directions.

5. The audio system according to claim 3, wherein the audio system further comprises: A user interface configured to enable selection of one or more of the plurality of audio features, wherein, when selecting a first feature and a second feature, the audio feature tuner is configured to push the output of a first subset of the head into a corresponding direction in a first set of directions, and push the output of a second subset of the head into a corresponding direction in a second set of directions.

6. The audio system according to claim 1, wherein, The audio feature tuner selects a subset of heads that produce outputs sensitive to the desired audio features, and sparsely applies the directional ITI to the selected heads by skipping the application of the directional ITI to the selected heads at some time instances.

7. The audio system according to claim 6, wherein, The audio feature tuner applies the directional ITI to the selected head unless the directional ITI has already been applied to multiple previous outputs of the selected head.

8. The audio system according to claim 6, wherein, A subset of heads that produce outputs sensitive to the desired audio characteristics is predetermined offline.

9. The audio system according to claim 6, wherein, During the execution of the autoregressive audio generator, a subset of the heads that produce outputs sensitive to the desired audio features are determined online based on the output of the multi-head attention head.

10. The audio system according to claim 9, wherein, The audio feature tuner is configured as follows: The output of the head is classified using a classifier trained in a supervised manner on the corresponding head, such that a one-to-one mapping exists between the head and the classifier; and Based on the classification, a subset of the heads that are sensitive to the audio features is selected.

11. The audio system according to claim 1, wherein, The audio feature tuner is configured as follows: The output of the head receiving the multi-head attention is classified using a classifier trained in a supervised manner on the corresponding head to determine the probability that the corresponding output has the desired audio features. and A bias vector is added to each output of each head, wherein the direction of the bias vector is predetermined for the desired audio features, and wherein the length of the bias vector is a function of the probability.

12. The audio system according to claim 11, wherein, The length of the bias vector of the orientation ITI used for the head's output includes: an offline component determined offline based on weights determined according to the probability that the classifier classifies the head's output as carrying desired audio features, and an online component determined during the execution of the audio system.

13. The audio system according to claim 12, wherein, The weights of the online components include probe monitoring weights, and the weights of the offline components include probe accuracy weights and probe dispersion weights.

14. The audio system according to claim 1, wherein, The audio feature tuner is configured as follows: At least some of the outputs of the multi-head attention are classified using a classifier trained in a supervised manner on the corresponding head to determine the probability that the corresponding output has the desired audio features. and A bias vector is added to each output of each head if the probability of having the desired audio feature is less than a predetermined threshold, wherein the direction of the bias vector is predetermined for the desired audio feature, and wherein the length of the bias vector is a function of the probability.

15. The audio system according to claim 1, wherein, The desired audio characteristics indicate a sound that indicates abnormal machine operation.

16. The audio system according to claim 1, wherein, The input used for synthesizing audio includes text, thus the audio system is a text-to-audio generation system.

17. The audio system according to claim 1, wherein, The input used for synthesizing audio includes a continuous audio signal, thus the audio system is an audio continuous generation system.

18. A method for synthesizing audio sounds having desired audio features, wherein, The method uses a processor linked to stored instructions that implement the steps of the method, and the method includes the following steps: Collect input data for synthesizing audio and data indicating the desired audio features in the synthesized audio; An autoregressive audio generator is executed, which is trained to generate audio by processing the input using multiple layers employing multi-head attention. Using directed inference temporal intervention (ITI), at least some outputs from at least some of the multi-head attention are pushed into a predetermined direction targeting the desired audio features; and The audio generated by the autoregressive audio transformer is output.

19. The method according to claim 18, wherein, The directional ITI adds a bias vector pointing in a predetermined direction to the output of the head before applying the activation function of the neurons in the head to the output of the head.

20. The method of claim 18, further comprising the following steps: Accept multiple audio features, each of which is associated with a set of directions predetermined for a set of heads, wherein the multiple audio features include a first audio feature associated with a first set of directions predetermined for a first subset of heads and a second audio feature associated with a second set of directions predetermined for a second subset of heads, wherein the first subset of heads is different from the second subset of heads, wherein the first subset of heads and the second subset of heads include a first head and a second head having different directions predetermined for the first audio feature and the second audio feature; Push the output of the first subset of the head into the corresponding direction in the first set of directions; and The output of the second subset of the head is pushed into the corresponding direction in the second set of directions.

Images