Relaxed instance frequency normalization for neural network based audio processing

Abstract

Techniques and apparatus for training a neural network to classify audio into one of a plurality of categories and using such trained neural network. An exemplary method generally includes receiving a dataset including a plurality of audio samples. A relaxed feature normalized dataset is generated by normalizing each audio sample of the plurality of audio samples. A neural network is trained to classify audio into one of a plurality of categories based on the relaxed feature normalized dataset, and the trained neural network is deployed.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 252,100, entitled "Relaxed Instance Frequency Normalization for Neural-Network-Based Audio Processing", filed on October 4, 2021, which claims the benefit of and priority to U.S. Provisional Patent Application No. 17 / 937,765, entitled "Relaxed Instance Frequency Normalization for Neural-Network-Based Audio Processing", filed on October 3, 2022, each of which is assigned to the assignee of the present application and is incorporated by reference in its entirety herein. PRIOR ART

[0002]

[0002] Introduction Aspects of the present disclosure relate to using artificial neural networks to process audio content.

[0003]

[0003] Artificial neural networks can be used for various tasks such as content classification. For example, in visual content, artificial neural networks can be used to extract information from various domains in the visual content, such as spatial domain, color domain, and time domain, to perform various tasks on the visual content. In general, visual content can be represented as at least a two-dimensional space with a horizontal axis and a vertical axis. Additional axes can be added to represent other dimensions, such as the depth (Z) axis in three-dimensional imaging or the time axis in video content. Artificial neural networks used in image-based processing tasks can include, for example, two-dimensional convolutional neural networks used to process individual images or individual frames in video content. In these two-dimensional convolutional neural networks, two-dimensional portions of the image (e.g., portions having horizontal and vertical dimensions) can be used in training the two-dimensional convolutional neural network and in performing inference on the input image.

[0004]

[0004] Artificial neural networks may also be used for various tasks related to audio data. For example, artificial neural networks may be used in audio scene (or environment) classification tasks to identify the surrounding environment in which the audio data was captured. Artificial neural networks may be used in speaker identification tasks in which speakers recorded in audio data are identified. In yet another example, artificial neural networks may be used in keyword extraction tasks to identify specific words in an utterance that are relevant to an action to be performed in an application, the subject of a search query, etc. Audio data may generally be represented in a two-dimensional space, with a first dimension corresponding to the frequency dimension and a second dimension corresponding to the time dimension, allowing two-dimensional convolutional neural networks to be used in various tasks related to audio data.

[0005]

[0005] A convolutional neural network for image data may convolve data along the horizontal and vertical spatial dimensions in the image data. A convolutional neural network for audio data may involve processing data along the time and frequency dimensions instead of the horizontal and vertical spatial dimensions of the image data, both of which may convey information that can be used in classifying the data. However, statistically relevant information in the audio data may be concentrated in the neural network. This concentration of statistical information in the audio data may therefore increase the difficulty of extracting usable information from the audio data that can be used to perform various classifications or other machine learning-based tasks on the audio data.

[0006]

[0006] Therefore, what is needed are improved techniques for processing audio content using artificial neural networks. Summary of the Invention

[0007]

[0007] Some aspects of the present disclosure provide a method for training a neural network to classify audio into one of a plurality of categories. An exemplary method generally includes receiving a dataset including a plurality of audio samples. A relaxed feature normalization dataset is generated by normalizing each audio sample of the plurality of audio samples. A neural network is trained to classify the audio into one of a plurality of categories based on the relaxed feature normalization dataset, and the trained neural network is deployed.

[0008]

[0008] Certain aspects of the present disclosure provide a method for classifying audio using a neural network. An exemplary method generally includes receiving an audio input. A relaxed feature-normalized version of the audio input is generated. A classification of the received audio input is generated using a neural network trained to classify audio into one of a plurality of categories and the relaxed feature-normalized version of the audio input. One or more actions are taken based on the classification of the received audio input.

[0009]

[0009] Another aspect provides a processing system configured to perform the aforementioned methods as well as methods described herein; a non-transitory computer readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned methods as well as methods described herein; a computer program product embodied on the computer readable storage medium including code for performing the aforementioned methods as well as methods further described herein; and the processing system including means for performing the aforementioned methods as well as methods further described herein.

[0010]

[0010] The following description and the related drawings set forth in detail certain illustrative features of the one or more aspects. [Brief description of the drawings]

[0011]

[0011] The drawings illustrate some features of one or more aspects and therefore should not be considered as limiting the scope of the disclosure. [Figure 1]

[0012] FIG. 1 illustrates mutual information estimates for various representations of two-dimensional data. [Diagram 2]

[0013] FIG. 1 illustrates example operations for training a neural network to classify audio into one of a number of categories using relaxed frequency normalization of a training dataset of audio data, according to aspects of the disclosure. [Diagram 3]

[0014] FIG. 1 illustrates example operations for classifying audio into one of a plurality of categories based on relaxed frequency normalization of the audio using a neural network, according to aspects of the disclosure. [Figure 4]

[0015] FIG. 1 illustrates example pseudocode for an algorithm for processing audio using relaxed frequency normalization, according to aspects of the present disclosure. [Figure 5A]

[0016] 13A-13C are example scatter plots of audio classification performed based on frequency-related statistics and relaxed frequency normalization, according to aspects of the present disclosure. [Figure 5B] 13A-13C are example scatter plots of audio classification performed based on frequency-related statistics and relaxed frequency normalization, according to aspects of the present disclosure. [Figure 6]

[0017] FIG. 1 illustrates the relationship between mutual information and training and testing accuracy for neural networks, according to aspects of the present disclosure. [Figure 7]

[0018] 1A-1C illustrate example accuracy of neural networks for various audio classification tasks where relaxed frequency normalization is used in training and inference operations, in accordance with aspects of the present disclosure. [Figure 8]

[0019] FIG. 1 illustrates an example neural network pipeline in which a relaxed frequency normalization module is used to normalize data at multiple layers in a neural network, according to aspects of the present disclosure. [Figure 9]

[0020] FIG. 1 illustrates an example implementation of a processing system in which a neural network is trained to perform audio classification based on a relaxed frequency-normalized training dataset, according to aspects of the disclosure. [Figure 10]

[0021] FIG. 1 illustrates an example implementation of a processing system in which audio classification may be performed using a neural network and relaxed frequency normalization, according to aspects of the disclosure.

[0012]

[0022] For ease of understanding, wherever possible, like reference numerals have been used to designate like elements common to the figures, and it is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further detail. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013]

[0023] Certain aspects of the present disclosure provide techniques for classifying audio data using neural networks.

[0014]

[0024] Audio data may be represented as two-dimensional data including a frequency dimension and a time dimension. Because audio data may be represented as two-dimensional data, two-dimensional convolutional networks similar to those used in image processing tasks may be used to process the audio data. However, audio data has different characteristics than image data, and thus techniques used to preprocess image data may not result in a usable model for the audio data. For example, while image data has height and width (spatial dimensions), audio data generally has frequency and time dimensions, and convolutions on the frequency and time dimensions in the audio data may not allow a model to generate usable inferences for the audio data in the same way that convolutions on the height and width dimensions in the image data allow the generation of usable inferences for the image data.

[0015]

[0025] In general, a neural network may be trained using data from a single domain to generate inferences specific to that domain. For example, a neural network may be trained using video data captured from an autonomous driving environment to generate inferences usable in that domain (e.g., object detection, subject pose prediction, etc.). However, a neural network trained using a domain-specific training dataset may generate inferences usable only for data within that domain, but may not be able to generate inferences usable for data outside of that domain. In another example, for audio data, a neural network may be trained using data captured by a particular capture device or in a particular environment, which may allow inferences usable for inputs generated by those particular capture devices or in those particular environments to be generated, but may not allow inferences usable for inputs generated by other capture devices or in other environments to be generated.

[0016]

[0026] In order to train a neural network to generate inferences usable on data from various domains, domain generalization can be used to remove or reduce instance-specific domain mismatches in the data used to train the neural network, thus generalizing the neural network to understand data from unseen domains. In general, for image data, domain generalization can be achieved through various normalization techniques across spatial aspects of the image data. However, while convolution across spatial aspects of the image data may enable some degree of domain generalization in a neural network used for image data, such convolution may not result in domain generalization in a neural network used for audio data. For example, audio data may be convolved across frequency and time dimensions. However, both frequency and time dimensions in audio data may contain domain-related information, and domain-specific task-independent information may not be distributed across different dimensions in audio data. Thus, although an audio signal may be represented by a two-dimensional image (e.g., a log-Mel spectrogram, or a Mel Frequency Cepstral Coefficients (MFCC) representation), training a machine learning model to perform tasks on audio data based on these two-dimensional images using convolutions across different spatial dimensions in these images may not result in a neural network that can understand data from unseen domains.

[0017]

[0027] Aspects of the present disclosure provide techniques that enable audio classification tasks using neural networks that generalize audio data using relaxed frequency normalization techniques. As used herein, relaxed frequency normalization generally normalizes audio data in the frequency domain with additional normalization applied to mitigate loss of discriminatory information in other forms of normalization while enabling domain generalization for the audio data. By using relaxed frequency normalization of the audio data used to train the neural network and performing inference on the audio input, the neural network can leverage domain-related information carried in the frequency domain to enable domain generalization for audio tasks. Thus, a neural network can be trained on a set of audio data that includes samples from multiple domains, and the resulting neural network can be domain generalized and enable classification of the audio data into known and previously unseen domains (e.g., using domain-invariant features generated for the audio data). This can enable the neural network to more accurately classify the audio data, which can enable the audio data to be classified without having to fall back to other models or other classification techniques and without performing unrelated operations based on inaccurate classification of the audio data. Thus, fewer computational resources may be utilized to classify the audio data and perform the resulting tasks based on the classification of the audio data, thereby reducing the amount of power consumed by the computing device to perform these tasks and accelerate the processing of the audio data relative to the amount of power and time used when the neural network is not trained using the relaxed frequency normalization technique.

[0018] Exemplary characteristics of audio data in 2D convolutional neural networks

[0028] FIG. 1 shows mutual information estimates 100 for various representations of two-dimensional data. As shown, the estimates 100 include mutual information estimates for audio data represented as a log-Mel spectrogram in graph 110, audio data represented as a Mel Frequency Cepstral Coefficients (MFCC) representation in graph 120, and raw image data in graph 130. These are estimates of mutual information between statistics on the dimensionality of intermediate hidden features and domain labels I(s,d) (top) or class labels I(s,y) (bottom). Graph 130 uses raw RGB images for the PACS dataset (which includes photographs, art paintings, cartoons, and sketch images), and the average over five seeds, with error bars representing standard deviation.

[0019]

[0029] In general, mutual information refers to a measure of interdependence between two variables. To estimate the mutual information (and to identify which dimensions contain domain-related information), statistics on the dimensions of the hidden features for each dimension can be calculated. To calculate these statistics, a hidden feature x in a data channel may be indexed with index i, where index i specifies an index in each of a plurality of axes. In one example, these axes may include a batch axis (denoted as N), a channel axis (denoted as C), a frequency axis (denoted as F, which is similar to the height dimension of the image data), and a time axis (denoted as T, which is similar to the width dimension of the image data). Thus, index i is expressed as: i=(i N ,i C ,i F ,i T ) (where i N represents the index on the batch axis, and i C represents the index on the channel axis, i F represents the index on the frequency axis, and i T In another example, the hidden feature

[0020]

number

[0021] For s, dimensional statistics can be calculated, including statistics along each of multiple dimensions. (F) =concat(μ (F) ,σ (F) ), the concatenation of the mean and standard deviation along the F-axis (e.g., the frequency dimension in two-dimensional audio represented by the frequency and time dimensions).

[0022]

[0030] Statistical measures such as the mean and standard deviation may be calculated for each instance of data in the batch of data. The mean with respect to frequency may be calculated, for example, according to the following formula:

[0023]

number

[0024]

[0031] The standard deviation with respect to frequency may be calculated according to the following formula:

[0025]

number

[0026] where ∈ represents a small constant,

[0027]

number

[0028] where k represents the samples in the dataset and m represents

[0029]

number

[0030] is the size of i F and k Frepresents sub-indexes of the three-dimensional indexes i and k along the F axis. Statistics for the N, C, and T axes (e.g., the batch, channel, and time dimensions, respectively) may be calculated similarly. Note that in image data, the axes on which statistics are calculated for hidden features x may include a height dimension H instead of the frequency dimension F, and a width dimension W instead of the time dimension T. As shown in FIG. 1, for graph 130, the channel statistics yield higher values ​​than those for width or height. However, frequency statistics may also be highly correlated to class label y in the audio domain.

[0031]

[0032] A residual neural network trained on a multi-domain image dataset may be used to generate the mutual information estimates shown in graph 130 for image data. For the mutual information estimates shown in graphs 110 and 120, a training dataset for multi-device acoustic scene classification (ASC) may be used, in which multiple acoustic scenes (e.g., shopping malls, airports, mass rail transit stations, etc.) are recorded by multiple devices.

[0032]

[0033] Graphs 110, 120, and 130 show mutual information estimates for each of multiple stages in the neural network. As shown in graph 130, for raw image data, the mutual information estimates for hidden features in the channel dimension may be higher than the mutual information estimates for the height and width dimensions at each stage of the neural network. Thus, for image data, the channel dimension may represent domain information in an image-based 2D convolutional neural network, and normalization along the channel domain may be used for domain generalization. However, for audio data, as shown in graphs 110 and 120, the mutual information estimates may be highest in the frequency dimension and lowest in the time dimension. Thus, for audio data, it may be seen that domain-specific information dominates in the frequency dimension rather than the channel dimension.

[0033]

[0034] Since domain information is predominant in the frequency dimension, domain generalization for audio data can be achieved by normalizing the data along the frequency domain, thereby suppressing irrelevant domain-specific information and preserving task-relevant information in the frequency dimension.

[0034] Exemplary Relaxed Instance Frequency Normalization in Audio Data

[0035] To generalize neural networks used in audio classification tasks, aspects of this disclosure provide techniques for normalizing audio data in the frequency domain.

[0035]

[0036] Various types of normalization can be used to suppress irrelevant domain-specific information and allow task-relevant information to be preserved in the data used in training neural networks and performing inference using these neural networks. Batch normalization can be one technique used in normalizing data. In batch normalization, statistical measures (e.g., mean and standard deviation) can be calculated for each of multiple channels across each sample in the training data set and across spatial dimensions. In instance normalization, statistical measures can be calculated for individual channels and for each sample across spatial dimensions. Yet another normalization technique used to normalize training data and inputs from which inferences are to be generated can include group normalization, in which channels are divided into groups and each group is normalized using the statistical measures generated for that group.

[0036]

[0037] More generally, the 4-dimensional index i=(i N ,i C ,i F ,i T ), the elements of the normalized features are given by

[0037]

number

[0038] In this formula, N, C, F and T correspond to the batch, channel, frequency (height) and time (width) axes, respectively. As explained above, the average μ i and standard deviation σ i Measurements are set

[0039]

number

[0040] , the size of which can be calculated over

[0041]

number

[0042] Therefore, the channel normalization can be performed for the same channel index i C may be defined as feature normalization where feature elements with share a mean and variance, such that batch normalization is defined by:

[0043]

number

[0044] The above formula is performed over a data set that contains features with the same channel index. In the above formula, C (k C) represents the sub-index of i(k) along the C-axis. Meanwhile, feature normalization (FN) may be defined as an approach in which feature elements sharing the same frequency feature index f in a four-dimensional space are normalized together. Batch frequency normalization (BFN), instance frequency normalization (IFN), and group frequency normalization (GFN) may be defined. BFN may be defined by the following equation:

[0045]

number

[0046] where i F (k F ) is the sub-index of i(k) along the frequency (F) axis. IFN may be defined according to the following equation:

[0047]

number

[0048]

[0038] IFN generally allows reduction or elimination of instance-specific domain data in audio features. However, it should be noted that frequency statistics generally have a high correlation with domain information and class discrimination information. Because frequency statistics are highly correlated with class discrimination information, using IFN to normalize audio data may cause loss of discrimination information that may be useful in classifying audio inputs. Therefore, to enable domain generalization for audio data while normalizing the audio data in the frequency domain, additional normalization can be used to mitigate or alleviate the loss of discrimination information in IFN. To mitigate IFN, global statistics for instances across N dimensions can be generated and used in layer normalization techniques. These global statistics for instances across N dimensions can be expressed as the formula

[0049]

number

[0050] Using IFN and LN, the relaxed instance frequency normalization (RFN) may be expressed by the following equation:

[0051]

number

[0052] where x is an input to an RFN module in a neural network, with λ∈[0,1]. In general, the λ term represents the degree of relaxation, where a larger value of λ results in a larger amount of relaxation of the instance frequency normalized value of input x, and a smaller value of λ results in a smaller amount of relaxation of the instance frequency normalized value of input x. In general, RFNs do not need to use affine transformations, which can reduce the number of regularization parameters in the network. Furthermore, RFNs can discard unnecessary (or redundant) domain information while reducing the loss of discriminative information. In RFNs, the resulting average statistical measure can be expressed by the following equation:

[0053]

number

[0054] where λ represents the degree of relaxation and μ n,f represents the average over the batch and frequency dimensions, and μ n represents the average over the batch dimension, and σ n represents the standard deviation over the batch dimension. In some aspects, λ=0 results in RFN(x)=IFN(x), and λ=1 results in RFN(x)=LN(x).

[0055] Meanwhile, the standard deviation statistical measure in the resulting RFN can be expressed by the following formula:

[0056]

number

[0057] Here, μ n,f and σ n,f teeth,

[0058]

number

[0059] is calculated over μ n and σ n teeth,

[0060]

number

[0061] This is a statistic over . When λ=1,

[0062]

number

[0063]

number

[0064] corresponds to the mean and standard deviation calculated for IFN.

[0065] 2 illustrates example operations 200 for training a neural network to classify audio into one of a plurality of categories using relaxed frequency normalization of a training dataset of audio data according to aspects of the present disclosure. Operations 200 may be performed, for example, by a computing device (such as processing system 900 shown in FIG. 9 ) on which the neural network may be trained and which may deploy the trained neural network (e.g., to a server, to a user equipment (UE), etc.) for use in performing inference on received audio data.

[0066]

[0041] As shown, the operation 200 may begin at block 210, where a dataset including a plurality of audio samples is received. The dataset of audio samples may include audio samples of various lengths and associated with a plurality of categories. For example, to train a neural network for audio scene classification, the dataset of audio samples may include samples from a plurality of different audio environments. Each sample in the dataset may be labeled or tagged with an indication of the environment in which the sample was collected. In another example, to train a neural network for keyword detection, the dataset of audio samples may include a plurality of audio utterances, where each utterance may be associated with a keyword associated with the utterance. In yet another example, to train a neural network for speaker detection, the dataset of audio samples may include a plurality of audio utterances, where each utterance may be associated with a speaker who made the utterance.

[0067]

[0042] At block 220, a relaxed feature normalization dataset is generated. In general, the relaxed feature normalization dataset is generated by normalizing each audio sample of the plurality of audio samples. As described above, to normalize the audio samples in the received dataset, the audio samples in the received dataset may be normalized using instance frequency normalization (IFN) and relaxed based on another type of normalization and normalization factor λ. In some aspects, layer normalization (LN) may be used to relax the IFN applied to the audio samples in the dataset. In some examples, other types of normalization, such as batch normalization, may be used to relax the IFN applied to the audio samples in the dataset. The selection of the normalization type used to relax the IFN applied to the audio samples in the dataset may be a design choice selected based on the intended classification task for the audio data.

[0068]

[0043] In some aspects, to generate a relaxed feature normalized data set, one or more statistical measures may be calculated for one or more hidden features in a plurality of feature dimensions in the plurality of audio samples. These hidden features may generally correspond to features in a latent space describing the audio samples, from which various correlations may be made to classify the audio samples into one of a plurality of categories. The plurality of feature dimensions may include a batch dimension (denoted by N), a channel dimension (denoted by C), a frequency dimension (denoted by F, corresponding to the height dimension of the image dataset), and a time dimension (denoted by T, corresponding to the width dimension of the image dataset). In one example, calculating these statistical measures may include calculating an average with respect to frequency across the plurality of audio samples and a standard deviation with respect to frequency across the plurality of audio samples. Based on the calculated one or more statistical measures, an instance frequency normalized version of each sample of the plurality of audio samples may be generated.

[0069]

[0044] In some aspects, the normalized audio samples of the plurality of audio samples may be further normalized to generate a relaxed feature normalized data set. The normalized audio samples may be further normalized based on layer normalization or other normalization techniques based on a statistical measure generated over the received data set. In layer normalization, for example, the statistical measure generated over the received data set may be a mean and standard deviation calculated over a batch dimension within samples in the received data set.

[0070]

[0045] In some aspects, further normalizing each audio sample of the plurality of audio samples may be further based on a relaxation factor. The relaxation factor may be associated with a degree of relaxation to be applied to each instance frequency-normalized audio sample of the plurality of audio samples. In some aspects, the relaxation factor may range from 0 to 1, where a relaxation factor of 0 corresponds to no relaxation of image frequency normalization applied to the audio sample, and a relaxation factor of 1 corresponds to full relaxation of instance frequency normalization applied to the audio sample in favor of a different form of normalization (e.g., layer normalization, batch normalization, etc.).

[0071]

[0046] At block 230, a neural network is trained to classify the data into one of a plurality of categories based on the relaxed feature-normalized data set.

[0072]

[0047] At block 240, the trained neural network is deployed. The trained neural network may be deployed, for example, on one or more computing devices that can receive data from mobile devices (e.g., user equipment in communication with these computing devices) and perform inference on audio utterances received from the mobile devices. In another example, the neural network may be deployed on the user equipment (UE) or other mobile device itself for use in performing inference locally (e.g., without incurring the network processing overhead associated with transmitting audio recordings to a remote device for further processing).

[0073] 3 illustrates an example operation 300 for classifying an audio input into one of a plurality of categories using a neural network and relaxed frequency normalization of audio. The operation 300 may be performed, for example, by a computing device on which a neural network may be deployed for use in performing inferences on received audio data. The computing device may include a server, a cloud computing resource, one or more computing devices integrated into or otherwise co-located with a network entity (e.g., gNodeB, eNodeB, etc.) that serves user equipment in a wireless network, a user equipment (UE) or other device on which an inference may be generated locally, etc.

[0074] As shown, the operations 300 may begin at block 310, where an audio input is received for classification. Generally, the audio input may be generated using one or more audio recording devices located on a source device and provided (e.g., over a network connection or on a local communication bus between the audio recording devices and a processor on which the inference generation operations may be performed).

[0075]

[0050] At block 320, a relaxed feature-normalized version of the audio input may be generated. To normalize the audio input, the audio input in the received dataset may be normalized using instance frequency normalization (IFN) and relaxed based on another normalization and normalization factor λ. In some aspects, layer normalization (LN) may be used to relax the IFN applied to the audio input. In some examples, other normalizations such as batch normalization may be used to relax the IFN applied to the input.

[0076]

[0051] Similar to the generation of the relaxed feature-normalized version of the audio sample described above with respect to FIG. 2, to generate the relaxed feature-normalized version of the audio input, one or more statistical measures may be calculated from one or more hidden features in each of a plurality of feature dimensions in the received audio input. The plurality of feature dimensions may include a batch dimension (denoted by N), a channel dimension (denoted by C), a frequency dimension (denoted by F, corresponding to the height dimension of the image data), and a time dimension (denoted by T, corresponding to the width dimension of the image data). In one example, calculating these statistical measures may include calculating an average with respect to frequency across the received audio input and a standard deviation with respect to frequency across the received audio input. An instance frequency-normalized version of the received audio input may be generated based on the calculated one or more statistical measures.

[0077]

[0052] In some aspects, the normalized audio input may be further normalized to generate a relaxed feature-normalized version of the audio input. The normalized audio input may be further normalized based on layer normalization or other normalization techniques based on statistical measures generated over the received audio input. In layer normalization, for example, the statistical measures generated over the received audio input may be the mean and standard deviation calculated over a batch dimension within samples in the received audio input.

[0078]

[0053] In some aspects, further normalizing the received audio input may be further based on a relaxation factor. The relaxation factor may be associated with a degree of relaxation to be applied to the instance frequency normalized audio input. In some aspects, the relaxation factor may range from 0 to 1, where a relaxation factor of 0 corresponds to no relaxation of image frequency normalization applied to the audio input and a relaxation factor of 1 corresponds to full relaxation of instance frequency normalization applied to the audio input in favor of a different form of normalization (e.g., layer normalization, batch normalization, etc.).

[0079]

[0054] In block 330, the received audio input is classified using a relaxed frequency normalized version of the audio input and a neural network trained to classify the audio into one of a plurality of categories. In some aspects, the neural network may include multiple layers. Any number of layers in the neural network may include a relaxed frequency normalization module that applies a relaxed frequency normalization (e.g., instance frequency normalization relaxed by layer normalization or another normalization technique such as batch normalization) to the input of the layer (e.g., the raw audio input provided to an initial layer of the neural network and / or a representation of the raw audio input generated by a layer in the neural network).

[0080]

[0055] In block 340, one or more actions are taken based on the classification of the received audio input. In one example, if the neural network is used for an audio scene classification task, the one or more actions may include identifying a noise cancellation mode to be applied to the audio output. These noise cancellation modes may, for example, allow the environmental noise to be partially, but not completely, suppressed in some situations, while in other situations the environmental noise may be completely or nearly completely filtered. In another example, if the neural network is used for a keyword identification task, the one or more actions may include triggering one or more operations in a software application based on an identified keyword in the received audio input. In yet another example, if the neural network is used for a speaker identification task, the one or more actions may include executing a speaker-specific action on the computing system.

[0081] FIG. 4 shows exemplary pseudocode 400 for an algorithm that performs relaxed frequency normalization on an audio input.

[0082] In this example, the audio input may be normalized using instance frequency-wise normalization (IFN) 410 instead of IN for audio domain generalization. Because frequency statistics are generally highly correlated to class discrimination information, the use of additional features that are not frequency-based normalized may be used to mitigate possible loss of useful information when normalizing audio data. To perform IFN 410, mean and variance measures may be calculated for activations x over a batch of inputs in a multi-dimensional space, such as an audio input having data in a batch axis N, a channel axis C, a frequency axis F, and a time axis T.

[0083]

[0058] The global statistics with respect to the instances, represented as S i = {k | kN = iN}, may be applied to the input using layer normalization (LN) 420. Using I N 410 and LN 420, the output may be obtained using relaxed instance frequency-wise normalization (RFN) 430. In the pseudocode 400, x is provided as an input, and λ ∈ [0,1] represents the degree of relaxation to be applied to the input. In this example, the resulting mean and standard deviation of RFN(x) may be expressed as follows:

[0084]

number

[0085] Here, respectively,

[0086]

number

[0087] is S i ={k|k N =i N ,k F =i F}, and u i and σ i is S i ={k|k N =i N} are statistical values ​​over the

[0088] It should be understood that other types of normalization may also or alternatively be combined with the IFN 410 to perform RFN 430 of the input. For example, other normalization techniques such as batch normalization, group normalization, or identity connection may be used instead of the LN 420 shown in FIG.

[0089]

[0060] Figures 5A and 5B show exemplary scatter plots of audio classification performed based on frequency statistics and relaxed frequency normalization. As shown, in the scatter plot 510 of Figure 5A, it can be seen that audio samples can be classified into multiple categories (e.g., corresponding to different device IDs) based on statistical measures (e.g., mean and standard deviation) on frequency. However, using channel statistics, it can be seen in the scatter plot 515 of Figure 5A that data from each of these categories can be mixed with each other such that data associated with one of the multiple categories is not separated from data associated with another of the multiple categories. Thus, as described above with respect to Figure 1, it can be seen that the frequency statistics provide mutual information that can be used in performing various classifications of audio inputs.

[0090]

[0061] With relaxed frequency normalization as shown in graph 520 of FIG. 5B, it can be seen that features from different devices are well mixed. Meanwhile, with relaxed instance normalization as shown in graph 530 of FIG. 5B, it can be seen that features from different devices are also mixed. Thus, the relaxation techniques described herein can be used to suppress or minimize domain-specific information, thereby allowing this irrelevant information to be ignored when performing an audio classification task. Thus, information related to the audio classification task can be used during training and inference operations without being affected by irrelevant information, thereby improving the accuracy of the audio classification task described above.

[0091]

[0062] Figure 6 shows the relationship between mutual information and training and test accuracy for a neural network for audio classification according to an embodiment of the present disclosure. The mutual information measurements shown in graph 610 show the mutual information measured between different channels in each of the layers in the neural network. As shown in graph 610, the mutual information in the dataset may decrease with each layer of the neural network when no relaxation is applied (e.g., λ=0). However, it can be seen that by varying the level of relaxation (e.g., λ>0), the scale of the mutual information decreasing through different layers of the neural network may be reduced. For acoustic scene classification, representative 2D audio features log-Mel spectrograms and MFCCs are the input. Then, the MI is estimated after each stage, which is a sequence of convolutional blocks whose activations have the same width.

[0092]

[0063] Thus, reducing the scale of the decrease in mutual information in the data set can result in an increase in the training and testing accuracy of the neural network for both domains included in the training data set and domains not included in the training data set. In general, as shown in graph 620, the gap between training accuracy and testing accuracy can be largest in neural networks that are not trained using relaxed frequency normalization techniques, and this gap between training accuracy and testing accuracy can be seen with increasing amounts of relaxation applied to frequency normalization for the input data set. That is, relaxed frequency normalization allows for smaller generalization loss than other normalization techniques, as described herein.

[0093]

[0064] Graph 630 shows the test accuracy for data from domains included in the training data set and unseen domains not included in the training data set. In graph 630, it can be seen that the classification accuracy of neural networks trained using normalization techniques other than relaxed frequency normalization is low, resulting in a noticeable gap between the classification accuracy of data in the domains included in the training data set and data in domains not included in the training data set (e.g., data from a recording device different from the recording device used to record the data in the training data set). In contrast, neural networks trained using relaxed frequency normalization can show a noticeable increase in inference accuracy, especially for data from unseen domains.

[0094]

[0065] In general, the neural network described herein trained using relaxed frequency normalization of the input data set may enable robust and accurate classification of audio inputs in various implementations. For example, in acoustic scene classification where relaxed frequency normalization is applied to the input and at the end of each stage (or layer) of the neural network, the neural network trained using relaxed frequency normalization may provide a noticeable increase (e.g., a 5 percent increase) in accuracy of classifying seen and previously unseen audio inputs into appropriate categories relative to other normalizations applied to the audio data. In keyword spotting tasks where the neural network is trained on a data set of voice commands and multiple speakers, the relaxed frequency normalization technique described herein may enable an increase in keyword identification accuracy for unseen speakers. These increases in keyword identification accuracy may be more noticeable when the neural network is trained using a small number of speakers from which the training data set was generated. In speaker verification tasks where speakers are identified from various genres of speech (e.g., interviews, live conversations, entertainment, drama, etc.), speaker verification using neural networks trained using the relaxed frequency normalization techniques described herein may exhibit reduced error rates relative to other techniques in which the inputs are normalized when training the neural network.

[0095]

[0066] FIG. 7 shows exemplary accuracies of neural networks for various audio classification tasks where relaxed frequency normalization is used in training and inference operations.

[0096]

[0067] Graph 710 shows the accuracy of neural networks for acoustic scene classification using various amounts of relaxation for the frequency normalization technique described herein. As shown, for various configurations of convolutional neural networks with various numbers of layers, the inference accuracy may increase up to a certain relaxation level and then decrease. That is, the best inference accuracy may be achieved for a value λ between 0 (no relaxation and frequency normalized) and 1 (normalized using some technique other than frequency normalization). Similar performance may be seen in graph 720 showing the accuracy of neural networks for keyword spotting tasks.

[0097]

[0068] However, for speaker verification tasks, graph 730 shows error rates for various levels of relaxation relative to complete relaxation (e.g., λ=1). As shown, the reduction in error rate for complete relaxation can be 0 to 1 percent for values ​​of λ from 0 to 1. However, when no relaxation is applied, i.e., when frequency normalization is applied to the training data set for the neural network and the input data for the neural network to infer the identity of the speaker, it can be seen that the reduction in error rate is more than 2 percent relative to complete relaxation. Therefore, for speaker verification tasks, frequency normalization and no relaxation can be used to achieve the highest inference accuracy.

[0098] FIG. 8 shows an example where there is a relaxed frequency normalization module at each of multiple stages of a pipeline 800 in a neural network. Relaxed frequency normalization may be used anywhere in the neural network, and may be used at multiple places in the neural network. In the pipeline 800, inputs may be processed in an initial RFN module 810. The output of the initial RFN module 810 may be provided as an input to a neural network layer 1 820, and the output of the neural network layer 1 820 may be processed through a layer 1 RFN module 830. In the next layer in the neural network, layer 2, the output of the RFN module for the previous layer (e.g., layer 1 RFN module 830) may be provided as an input to the next layer of the neural network (e.g., neural network layer 2 840). In layer 2, the output of the neural network layer 2 840 may be processed through a corresponding layer 2 RFN module 850. More generally, n th In the neural network layer, n-1 th n-1 associated with a neural network layer th The output of the RFN module is th may be provided as input to a neural network layer 860, th The output of the neural network layer 860 is n thThe neural network layers may be processed through corresponding RFN modules 870 for the neural network layers. However, while FIG. 8 shows a pipeline including an input layer and each layer in the neural network with a corresponding RFN module, it should be understood that an RFN module need not be associated with each layer in the neural network. For example, an RFN module may only be present in the input layer in the neural network, may be present in a subset of layers in the neural network, etc. For non-frequency normalized data, applying a relaxed frequency normalization module at the input stage of the neural network may have the greatest impact in increasing the accuracy of the neural network. Additional (albeit smaller) increases in performance may be seen when the relaxed frequency normalization module is attached to subsequent layers in the neural network.

[0099]

[0070] Relaxed frequency normalization is generally an explicit normalization of data. Note that the use of relaxed frequency normalization does not have to be a standalone technique for normalizing audio data when training a neural network to perform an audio classification task and when performing an audio classification task based on a normalized input. For example, relaxed frequency normalization may be combined with cross-gradient training, in which gradient descent is used when training a neural network using a multi-domain data set and / or a canonical signed digit (CSD) multiplier to achieve further improvement in inference accuracy.

[0100] Exemplary Processing System for Audio Classification Using Neural Networks and Relaxed Frequency Normalization

[0071] FIG. 9 shows an example processing system 900 for training a neural network to classify audio into one of a number of categories using a neural network and relaxed frequency normalization, e.g., as described herein with respect to FIG. 2.

[0101] The processing system 900 includes a central processing unit (CPU) 902, which may be a multi-core CPU in some examples. Instructions to be executed in the CPU 902 may be loaded, for example, from a program memory associated with the CPU 902 or from a memory partition (e.g., of memory 924).

[0102]

[0073] The processing system 900 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908, and connection components 912.

[0103]

[0074] An NPU, such as NPU 908, is generally a dedicated circuit configured to perform control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANN), deep neural networks (DNN), random forests (RF), etc. An NPU may alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graph processing unit.

[0104]

[0075] NPUs, such as NPU 908, are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-chip (SoC), while in other examples, they may be part of a dedicated neural network accelerator.

[0105]

[0076] An NPU may be optimized for training or inference, or in some cases may be configured to balance performance between both. In an NPU capable of performing both training and inference, the two tasks may still generally be performed independently.

[0106]

[0077] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating over the dataset, and then adjusting model parameters such as weights and biases to improve model performance. In general, optimization based on erroneous predictions involves backpropagating through layers of the model and determining gradients to reduce prediction errors.

[0107]

[0078] NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs may be configured to input new data and rapidly process the data through already trained models to generate model outputs (e.g., inferences).

[0108]

[0079] In one implementation, the NPU 908 is part of one or more of the CPU 902, GPU 904, and / or DSP 906.

[0109] In some examples, the connectivity component 912 may include subcomponents for, for example, third generation (3G) connectivity, fourth generation (5G) connectivity (e.g., 5G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The connectivity component 912 may further be connected to one or more antennas (not shown).

[0110]

[0081] In some examples, one or more of the processors of processing system 900 may be based on the ARM or RISC-V instruction set.

[0111] Processing system 900 also includes memory 924, which represents one or more static and / or dynamic memories, such as dynamic random access memories, flash-based static memories, etc. In this example, memory 924 includes computer-executable components that may be executed by one or more of the above-mentioned processors of processing system 900.

[0112]

[0083] In particular, in this example, memory 924 includes a training dataset receiving component 924A, a relaxed feature normalized dataset generating component 924B, a neural network training component 924C, a neural network expanding component 924D, and a network / parameters component 924E. The illustrated components, and other components not illustrated, may be configured to perform various aspects of the methods described herein.

[0113] In general, the processing system 900 and / or its components may be configured to perform the methods described herein.

[0114] FIG. 10 shows an example of a processing system 1000 for audio input classification using a neural network and relaxed frequency normalization, for example as described herein with respect to FIG.

[0115] Processing system 1000 includes a central processing unit (CPU) 1002 and may include additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 1004, a digital signal processor (DSP) 1006, a neural processing unit (NPU) 1008, a multimedia processing unit 1010, and wireless connectivity components 1012. CPU 1002, GPU 1004, DSP 1006, and NPU 1008 may be similar to CPU 902, GPU 904, DSP 906, and NPU 908 described above with respect to FIG.

[0116] In some examples, the wireless connectivity component 1012 may include sub-components for, for example, third generation (3G) connectivity, fourth generation (5G) connectivity (e.g., 5G LTE), fifth generation connectivity (e.g., 5G or NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 1012 is further connected to one or more antennas 1014.

[0117]

[0088] The processing system 1000 may also include one or more sensor processing units 1016 associated with any type of sensor, one or more image signal processors (ISPs) 1018 associated with any type of image sensor, and / or a navigation processor 1020, which may include satellite-based positioning system components (e.g., GPS or GLONASS), as well as inertial positioning system components.

[0118]

[0089] The processing system 1000 may also include one or more input and / or output devices 1022, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0119]

[0090] In some examples, one or more of the processors of the processing system 1000 may be based on the ARM or RISC-V instruction set.

[0120] Processing system 1000 also includes memory 1024, which represents one or more static and / or dynamic memories, such as dynamic random access memories, flash-based static memories, etc. In this example, memory 1024 includes computer-executable components that may be executed by one or more of the above-mentioned processors of processing system 1000.

[0121] In particular, in this example, the memory 1024 includes an audio input receiving component 1024A, a relaxed frequency normalized version generating component 1024B, an audio input classifying component 1024C, an action taking component 1024D, and a network / parameters component 1024E. The illustrated components, and other components not illustrated, may be configured to perform various aspects of the methods described herein.

[0122] In general, the processing system 1000 and / or its components may be configured to perform the methods described herein.

[0123]

[0094] Notably, in other aspects, aspects of the processing system 1000 may be omitted, such as when the processing system 1000 is a server computer, etc. For example, the multimedia processing unit 1010, the wireless connectivity component 1012, the sensor processing unit 1016, the ISP 1018, and / or the navigation processor 1020 may be omitted in other aspects. Furthermore, aspects of the processing system 1000, such as training models and using the models to generate inferences, such as user authentication predictions, may be distributed.

[0124] Illustrative clauses

[0095] Clause 1: A method, comprising: receiving a dataset including a plurality of audio samples; generating a relaxed frequency normalized dataset by normalizing each audio sample of the plurality of audio samples; training a neural network to classify audio into one of a plurality of categories based on the relaxed frequency normalized dataset; and deploying the trained neural network.

[0125]

[0096] Clause 2: The method of clause 1, wherein generating a relaxed frequency normalized dataset includes computing one or more statistical measures for one or more hidden features in each of a plurality of feature dimensions within a plurality of audio samples, and generating an instance frequency normalized version of each audio sample of the plurality of audio samples based on the computed one or more statistical measures.

[0126]

[0097] Clause 3: The method of clause 2, wherein the multiple feature dimensions include a batch dimension, a channel dimension, a frequency dimension, and a time dimension.

[0127]

[0098] Clause 4: The method of clause 2 or 3, wherein the one or more statistical measures include an average with respect to frequency across multiple audio samples and a standard deviation with respect to frequency across multiple audio samples.

[0128]

[0099] Clause 5: A method according to any of clauses 2 to 4, wherein generating the relaxed frequency normalized dataset further includes further normalizing each audio sample of the plurality of audio samples based on layer normalization using one or more statistical measures generated across the received dataset.

[0129]

[0100] Clause 6: The method described in clause 5, wherein further normalizing each audio sample of the plurality of audio samples is further based on a relaxation coefficient associated with a degree of relaxation to be applied to each instance frequency-normalized audio sample of the plurality of audio samples.

[0130]

[0101] Clause 7: The method described in Clause 5 or 6, wherein further normalizing each audio sample of the plurality of audio samples includes, for each audio sample, generating a layer normalized representation of the audio sample, and generating a relaxed frequency normalized representation of the audio sample based on a layer normalization relaxation factor, the layer normalized representation of the audio sample, a difference between 1 and the layer normalization relaxation factor, and an instance frequency normalized version of the audio sample.

[0131]

[0102] Clause 8: A method as described in any of clauses 2 to 7, wherein generating the relaxed frequency normalized dataset further includes: for each audio sample, generating a batch normalized representation of the audio sample, and generating a relaxed frequency normalized representation of the audio sample based on a batch normalization relaxation coefficient, the batch normalized representation of the audio sample, a difference between 1 and the batch normalization relaxation coefficient, and an instance frequency normalized version of the audio sample.

[0132]

[0103] Clause 9: A method, comprising: receiving an audio input; generating a relaxed frequency-normalized version of the audio input; generating a classification of the received audio input using the relaxed frequency-normalized version of the audio input and a neural network trained to classify audio into one of a plurality of categories; and taking one or more actions based on the classification of the received audio input.

[0133]

[0104] Clause 10: The method of clause 9, wherein generating a relaxed frequency-normalized version of the audio input includes computing one or more statistical measures for one or more hidden features in each of a plurality of feature dimensions within the received audio input, and generating an instance frequency-normalized version of the received audio input based on the computed one or more statistical measures.

[0134]

[0105] Clause 11: The method of clause 10, wherein the multiple feature dimensions include a batch dimension, a channel dimension, a frequency dimension, and a time dimension.

[0135]

[0106] Clause 12: The method of clause 10 or 11, wherein the one or more statistical measures include an average with respect to frequency across the received audio input and a standard deviation with respect to frequency across the received audio input.

[0136]

[0107] Clause 13: A method described in any of clauses 10 to 12, wherein generating a relaxed frequency-normalized version of the received audio input further includes further normalizing the received audio input based on layer normalization using one or more statistical measures generated across each of multiple portions of the received audio input.

[0137]

[0108] Clause 14: The method of clause 13, wherein further normalizing the received audio input is further based on a relaxation factor associated with a degree of relaxation to apply to the normalized received audio input.

[0138]

[0109] Clause 15: A method as described in clause 13 or 14, wherein further normalizing the received audio input includes generating a layer normalized representation of the received audio input, and generating a relaxed frequency normalized representation of the received audio input based on a layer normalization relaxation factor, the layer normalized representation of the received audio input, a difference between 1 and the layer normalization relaxation factor, and an instance frequency normalized version of the received audio input.

[0139]

[0110] Clause 16: A method according to any of clauses 10 to 15, wherein generating a relaxed frequency normalized version of the received audio input includes generating a batch normalized representation of the received audio input, and generating the relaxed frequency normalized representation of the received audio input based on a batch normalization relaxation factor, the batch normalized representation of the received audio input, a difference between 1 and the batch normalization relaxation factor, and an instance frequency normalized version of the received audio input.

[0140]

[0111] Clause 17: A method according to any one of clauses 9 to 16, wherein the neural network includes a plurality of layers and one or more relaxed frequency normalization modules.

[0141]

[0112] Clause 18: The method of any of clauses 9 to 17, wherein the neural network includes a model for classifying an audio input into one of a plurality of audio scene classifications.

[0142]

[0113] Clause 19: A method as described in any of clauses 9 to 18, wherein the neural network includes a model for classifying audio input into one of a plurality of categories associated with defined keywords for a task in the computing system.

[0143]

[0114] Clause 20: The method of any of clauses 9 to 19, wherein the neural network includes a model for identifying a speaker recorded in the audio input.

[0144]

[0115] Clause 21: The method of any one of clauses 9 to 20, wherein the neural network includes a two-dimensional convolutional neural network.

[0145]

[0116] Clause 22: A processing system comprising a memory containing computer-executable instructions and one or more processors configured to execute the computer-executable instructions and cause the processing system to perform a method according to any of clauses 1 to 21.

[0146]

[0117] Clause 23: A processing system comprising means for carrying out the method according to any one of clauses 1 to 21.

[0147]

[0118] Clause 24: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method described in any of clauses 1 to 21.

[0148]

[0119] Clause 25: A computer program product embodied on a computer-readable storage medium comprising code for performing the method according to any of clauses 1 to 21.

[0149] Additional Considerations

[0120] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The embodiments discussed herein are not intended to limit the scope, applicability, or aspects described in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may also be applied to other aspects. For example, changes may be made in the function and configuration of the elements discussed without departing from the scope of the disclosure. Various embodiments may omit, substitute, or add various procedures or components as necessary. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some embodiments may be combined in some other embodiments. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects described herein. Furthermore, the scope of the disclosure is intended to encompass apparatuses or methods that are practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0150]

[0121] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0151]

[0122] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. By way of example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination having multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0152]

[0123] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching a table, database, or another data structure), ascertaining, and the like. "Determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" may also include resolving, selecting, choosing, establishing, and the like.

[0153]

[0124] The methods disclosed herein include one or more steps or actions for achieving the method. The steps and / or actions of the methods may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software components, including but not limited to circuits, application specific integrated circuits (ASICs), or processors, and / or various hardware and / or software modules. In general, when operations are shown in a figure, the operations may have corresponding equivalent means-plus-function components that are similarly numbered.

[0154]

[0125] The following claims are not intended to be limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. Within the claims, reference to an element by the singular is intended to mean "one and only one" unless expressly stated as such, but rather "one or more." Unless expressly stated otherwise, the term "several" refers to one or more. No element of a claim is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means of" or, in the case of a method claim, unless the element is recited using the phrase "step of." All structural and functional equivalents to the elements of the various embodiments described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly recited in the claims.

Claims

1. 1. A processor-implemented method comprising: receiving an audio input; generating a relaxed frequency normalized version of the audio input; generating a classification of the received audio input using the relaxed frequency-normalized version of the audio input and a neural network trained to classify audio into one of a plurality of categories; taking one or more actions based on the classification of the received audio input; A method comprising:

2. generating the relaxed frequency normalized version of the audio input, computing one or more statistical measures for one or more hidden features in each of a plurality of feature dimensions within the received audio input; generating an instance frequency-normalized version of the received audio input based on the calculated one or more statistical measures; The method of claim 1 , comprising:

3. The method of claim 2 , wherein the plurality of feature dimensions includes a batch dimension, a channel dimension, a frequency dimension, and a time dimension.

4. the one or more statistical measures: an average over frequency across the received audio input; a standard deviation with respect to frequency across the received audio input; and The method of claim 2 , comprising:

5. 3. The method of claim 2, wherein generating the relaxed frequency-normalized version of the received audio input further comprises normalizing the received audio input based on layer normalization using one or more statistical measures generated across each of a plurality of portions of the received audio input.

6. normalizing the received audio input; further based on a mitigation factor associated with a degree of mitigation to apply to the normalized received audio input; or generating a layer-normalized representation of the received audio input; generating a relaxed frequency-normalized representation of the received audio input based on a layer normalization relaxation factor, the layer-normalized representation of the received audio input, a difference between 1 and the layer normalization relaxation factor, and the instance frequency-normalized version of the received audio input; Including, The method of claim 5.

7. generating the relaxed frequency-normalized version of the received audio input; generating a batch normalized representation of the received audio input; generating a relaxed frequency-normalized representation of the received audio input based on a batch normalization relaxation factor, the batch normalized representation of the received audio input, a difference between 1 and the batch normalization relaxation factor, and the instance frequency-normalized version of the received audio input; The method of claim 2 , comprising:

8. The method of claim 1 , wherein the neural network includes multiple layers and one or more relaxed frequency normalization modules.

9. The neural network a model for classifying the audio input into one of a plurality of audio scene classifications; and / or a model for classifying the audio input into one of the plurality of categories associated with defined keywords for a task in a computing system; The method of claim 1 , comprising:

10. The method of claim 1 , wherein the neural network includes a model for identifying a speaker recorded in the audio input.

11. The method of claim 1 , wherein the neural network comprises a two-dimensional convolutional neural network.

12. 1. A processor-implemented method comprising: receiving a data set comprising a plurality of audio samples; generating a relaxed frequency-normalized data set by normalizing each audio sample of the plurality of audio samples; training a neural network to classify audio into one of a plurality of categories based on the relaxed frequency-normalized data set; deploying the trained neural network; A method comprising:

13. 1. A processing system comprising: a memory comprising computer-executable instructions; Executing the computer-executable instructions, the processing system receiving a data set comprising a plurality of audio samples; generating a relaxed frequency-normalized data set by normalizing each audio sample of the plurality of audio samples; training a neural network to classify audio into one of a plurality of categories based on the relaxed frequency-normalized data set; deploying the trained neural network; one or more processors configured to cause A processing system comprising:

14. 1. A processing system comprising: a memory comprising computer-executable instructions; Executing the computer-executable instructions, the processing system receiving an audio input; generating a relaxed frequency normalized version of the audio input; generating a classification of the received audio input using a neural network trained to classify audio into one of a plurality of categories and the relaxed frequency-normalized version of the audio input; taking one or more actions based on the classification of the received audio input; one or more processors configured to cause A processing system comprising:

15. A method executed by a processor as described in claim 12, further comprising steps of a method as described in any one of claims 2 to 11, or a processing system as described in claim 13 or 14, further configured to execute steps of a method as described in any one of claims 2 to 11.

Images