Audio signal comparison with external normalization

Abstract

An audio processing system that compares a query audio sample to a database of a plurality of reference audio samples using external normalization is disclosed. The system includes at least one processor and a memory storing instructions that, when executed by the processor, cause the system to determine an externally normalized bias term based on querying a time-frequency pattern of audio samples. The system also compares the query audio sample to each reference audio sample to generate a similarity score for each comparison. The system combines a bias term with each similarity score to produce a normalized similarity score. The normalized similarity score is then compared to a threshold to generate a comparison result, which is then output.

Description

Technical Field

[0001] This disclosure relates generally to audio signal processing, and more specifically to systems and methods for comparing audio signals. Background Technology

[0002] Previous methods for comparing audio samples with databases of reference audio samples typically relied on conventional audio processing techniques. These methods often involved using various signal processing algorithms to analyze the audio samples to extract relevant features such as spectral content, temporal patterns, time-frequency markers, and amplitude variations. However, when attempting to quantify a match between two audio signals but not an exact copy, these methods, without normalization, may fail to adequately account for variations in total energy levels, frequency balance, and signal length between the query and reference audio samples.

[0003] In some cases, audio processing systems have attempted to address this issue by applying normalization techniques that adjust the overall characteristics of the query audio sample itself. While these techniques can partially mitigate the effects of variations in loudness, overall frequency balance, and signal length, they may not provide a comprehensive solution that takes into account the specific time-frequency patterns of the query audio sample.

[0004] Other approaches focus on using statistical methods such as dynamic time warp or hidden Markov models to compare audio samples. These methods aim to align a query audio sample with a reference audio sample by considering the temporal relationships between different segments of the audio signal. However, these methods may not provide accurate results when comparing audio samples with significant changes in loudness or energy levels or with slight frequency shifts between spectral components.

[0005] Therefore, there is still a need in the art for an audio processing system for comprehensive comparison methods to accurately compare query audio samples with a database of multiple reference audio samples. Summary of the Invention

[0006] Some embodiments aim to provide a system and method for comparing audio signals or audio samples. Additionally or alternatively, some embodiments aim to provide a system and method for comparing audio samples with each other and / or comparing audio samples with a plurality of other audio samples.

[0007] Some implementations are based on the recognition that audio samples need to be normalized for fair comparison. For example, audio samples can be modified to have the same length and loudness. This normalization modifies the audio sample itself and is referred to herein as internal normalization. However, this internal normalization is insufficient for a fair comparison of different audio samples. One reason for this deficiency is that the similarity of two audio samples depends on how the frequency content of the audio samples changes over time, which can also be described as comparing the time-frequency patterns between two audio samples. It should be recognized that some sounds can have time-frequency patterns that match well with a variety of other sounds. An extreme example of this is white noise, which has energy at all frequencies and is constant across time, and can match well with any sound that has a constant frequency characteristic across time. Alternatively, a single, brief click sound surrounded by silence can match well with any other sound with a short duration, regardless of the frequency content of the click. This problem persists even after normalizing the sounds for volume and length before comparison.

[0008] Therefore, in addition to internal normalization of the audio samples, audio comparisons also need to be normalized. This normalization is referred to herein as external normalization. External normalization does not modify the audio samples themselves, but rather modifies the comparison results to make them more fair to different types of audio samples. External normalization allows multiple audio queries to be compared to determine whether a given query matches any reference sample without requiring a query-specific threshold. Using external normalization, a single threshold can be used to detect matches across multiple types of audio queries. In some implementations, this external normalization determines a bias term based on the time-frequency pattern of the query audio sample and adds this bias term to the similarity score generated by comparing the query audio sample with other reference audio samples. Thus, instead of normalizing audio samples through internal normalization, some implementations normalize the similarity score through external normalization.

[0009] In some aspects, the technology described herein relates to an audio processing system that uses external normalization to compare a query audio sample with a database of multiple reference audio samples. The audio processing system includes: at least one processor; and a memory storing instructions thereon, which, when executed by the processor, cause the audio processing system to: determine a bias term for external normalization based on a time-frequency pattern of the query audio sample; compare the query audio sample with each reference audio sample to generate a similarity score for each comparison; combine the bias term with each similarity score to generate a normalized similarity score; compare the normalized similarity score with a threshold to generate a comparison result; and output the comparison result.

[0010] In some aspects, the technique described herein relates to an audio processing method that compares a query audio sample with a database of multiple reference audio samples using external normalization, wherein the method uses a processor coupled to stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, including: determining a bias term for external normalization based on the time-frequency pattern of the query audio sample; comparing the query audio sample with each reference audio sample to generate a similarity score for each comparison; combining the bias term with each similarity score to generate a normalized similarity score; comparing the normalized similarity score with a threshold to generate a comparison result; and outputting the comparison result.

[0011] In some aspects, the techniques described herein relate to a non-transitory computer-readable storage medium on which a program is implemented, executable by a processor, to perform a method for comparing a query audio sample with a database of multiple reference audio samples using external normalization. The method includes: determining a bias term for the external normalization based on a time-frequency pattern of the query audio sample; comparing the query audio sample with each reference audio sample to generate a similarity score for each comparison; combining the bias term with each similarity score to generate a normalized similarity score; comparing the normalized similarity score with a threshold to generate a comparison result; and outputting the comparison result.

[0012] The present disclosure is further described in a detailed description below with reference to the numerous accompanying drawings, which are non-limiting examples of exemplary embodiments of the present disclosure, and in several views of the drawings, the same reference numerals denote similar parts. The drawings shown are not necessarily drawn to scale, but generally focus on illustrating the principles of the embodiments of the present disclosure. Attached Figure Description

[0013] [ Figure 1 ]

[0014] Figure 1 A schematic diagram illustrating the operation of an audio processing system that uses external normalization to compare a query audio sample with a database of multiple reference audio samples, according to some embodiments.

[0015] [ Figure 2 ]

[0016] Figure 2 A schematic diagram illustrating the differences between external and internal normalization used in different implementations is shown.

[0017] [ Figure 3 ]

[0018] Figure 3A flowchart is shown of a method performed by an audio processing system to compare a query audio sample with a database of multiple reference audio samples using external normalization.

[0019] [ Figure 4 ]

[0020] Figure 4 A block diagram of a method for calculating a normalized similarity score according to some embodiments is shown.

[0021] [ Figure 5 ]

[0022] Figure 5 A block diagram of a method for determining bias terms according to some embodiments is shown.

[0023] [ Figure 6 ]

[0024] Figure 6 A block diagram of a method for determining bias terms according to some embodiments is shown.

[0025] [ Figure 7A ]

[0026] Figure 7A A block diagram of a method for determining a similarity metric for audio samples according to one embodiment is shown.

[0027] [ Figure 7B ]

[0028] Figure 7B A schematic diagram of a coarse-resolution Mel spectrogram used in some implementations is shown.

[0029] [ Figure 8 ]

[0030] Figure 8 A block diagram of a method for determining a similarity metric for audio samples according to another embodiment is shown.

[0031] [ Figure 9 ]

[0032] Figure 9 A block diagram is shown of a method for finding training data duplicates according to some implementations.

[0033] [ Figure 10 ]

[0034] Figure 10 A block diagram is shown illustrating a method for finding copies of audio samples to improve the training of audio deep learning models using large datasets, according to some implementations.

[0035] [ Figure 11]

[0036] Figure 11 A schematic diagram is shown illustrating a method for employing a protective mechanism to prevent training data replication according to some implementations.

[0037] [ Figure 12 ]

[0038] Figure 12 A schematic diagram is shown of a method for performing anomaly detection of query audio samples based on the results of an audio comparison with an external normalization employed in some implementations.

[0039] [ Figure 13 ]

[0040] Figure 13 This is a detailed block diagram of an audio processing system according to some embodiments of the present disclosure. Detailed Implementation

[0041] While the accompanying drawings illustrate the embodiments currently disclosed, other embodiments are contemplated, as pointed out in the discussion. This disclosure presents illustrative embodiments by way of representation and not limitation. Those skilled in the art can devise many other modifications and embodiments falling within the scope and spirit of the principles of the embodiments currently disclosed.

[0042] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatus and methods are shown only in block diagram form to avoid obscuring this disclosure. Various changes to the function and arrangement of the elements will be contemplated without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.

[0043] As used in this specification and claims, when combined with a list of one or more components or other items, the terms "for example," "like," and "such as," as well as the verbs "comprising," "having," "including," and their other verb forms, are each interpreted as open-ended, meaning that the list is not considered to exclude other additional components or items. The term "based on" means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0044] Specific details are set forth in the following description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, the same reference numerals and names in the various figures indicate the same elements.

[0045] Figure 1 A schematic diagram illustrates the operation of an audio processing system 102 that compares a query audio sample 110 with a database 170 of multiple reference audio samples using external normalization 120, according to some embodiments. To detect duplication of audio samples in the reference dataset, some embodiments construct the comparison as a copy detection problem suitable for comparing multiple audio samples with each other. For each pair of query q 110 and reference sample r from the reference sample database 170, a similarity score 120 is calculated, and this process is repeated for each sample in the reference sample database 170. Then, by finding the maximum similarity score across all samples in the reference sample database 170, we find the optimal matching similarity 130 for query q. The optimal matching similarity score 130 is above a certain threshold. Any query q in 140 is considered a copy in 150. Otherwise, audio query 110 is considered unique in 160.

[0046] Some implementations are based on the recognition that audio samples need to be normalized for fair comparison. For example, audio samples can be modified to have the same length, the same loudness, the same average frequency characteristics, etc. This normalization modifies the audio sample itself and is referred to herein as internal normalization. However, this internal normalization is insufficient for a fair comparison of different audio samples. One reason for this deficiency is that the similarity of two audio samples depends on how the frequency content of the audio samples changes over time, which can also be described as comparing the time-frequency patterns between two audio samples. It should be recognized that some sounds can have time-frequency patterns that match well with a variety of other sounds. An extreme example of this is white noise, which has energy at all frequencies and is constant across time, and can match well with any sound that has a constant frequency characteristic across time. Alternatively, a single, brief click sound surrounded by silence can match well with any other sound with a short duration, regardless of the frequency content of the click. This problem persists even after normalizing the sound in terms of volume, equalization, and / or length before comparison.

[0047] Therefore, additional normalization is needed for audio comparisons beyond the internal normalization of the audio samples. This normalization, referred to herein as external normalization, does not modify the audio samples themselves but rather modifies the comparison results to make them more fair to different types of audio samples. In some implementations, this external normalization determines a bias term based on the time-frequency pattern of the query audio sample and adds this bias term to the similarity score generated by comparing the query audio sample with other reference audio samples. Thus, alternatively, while audio samples are normalized via internal normalization, some implementations normalize similarity scores via external normalization.

[0048] Figure 2 A schematic diagram illustrating the differences between external and internal normalization used in different implementations is shown. Internal normalization 250 typically occurs before comparison 210 and includes modification 230 of the query audio sample 110. For example, some implementations assume that the audio samples to be compared have similar lengths in time, such that padding or removing the end of a file to make them have the same length can be considered "smaller normalization". If one file is much longer, some implementations may split the longer file into multiple sub-files of the same length as the other file, or alternatively use a sliding window method.

[0049] Another type of internal normalization involves applying gain to audio signals so that they are at the same loudness, where loudness can be measured using a non-perceptual metric such as the root mean square (RMS) level or a perceptually stimulated loudness metric such as loudness units at full scale. Some implementations also use equalization filters to internally normalize the audio signal in terms of its overall frequency content. Alternative implementations do not do this because this type of internal normalization is generally not conducive to comparing sound with highly varying content.

[0050] Conversely, external normalization 260 occurs after comparison 220 and modifies the result of comparison 240. Instead of modifying the audio file for fair comparison, external normalization adds a bias term to the similarity score determined based on the time-frequency pattern of the query audio sample.

[0051] Figure 3 A flowchart is shown of a method 300 performed by an audio processing system 102 for comparing a query audio sample with a database of multiple reference audio samples using external normalization. The method 300 is performed using at least one processor operatively coupled to a computer-readable storage device, on which instructions are stored that, when executed by the processor, cause the audio processing system to perform the steps of method 300.

[0052] At step 310, method 300 determines an externally normalized bias term based on the time-frequency pattern of the query audio sample. At step 320, method 300 compares the query audio sample with each reference audio sample to generate a similarity score for each comparison. At step 330, method 300 adds the bias term to each similarity score to generate a normalized similarity score. At step 340, method 300 compares the normalized similarity score with a threshold to generate a comparison result. At step 350, method 300 outputs the comparison result. Doing so in this way takes into account the details of the audio file to adjust the comparison result.

[0053] It is worth noting that this method can be used to compare multiple audio samples with each other when some audio samples are considered as queries and others as references. For each pair of query q and reference r, a similarity score is calculated to obtain a pairwise similarity matrix. Some implementations follow a two-stage semi-manual approach consisting of a retrieval phase and a verification phase. In the retrieval phase, implementations retrieve the best matching similarity score above a certain threshold. The query q.

[0054] The similarity score between query q and reference r It uses similarity scores based on descriptors extracted from q and r. The similarity score between them is calculated, including but not limited to cosine similarity:

[0055] Some implementations use low-dimensional log-Mel spectrograms as sample descriptors. This is a natural choice because most audio generation models employing latent diffusion extract latent representations from Mel spectrograms, and using low-dimensional Mel spectrograms helps smooth fine-grained details in the audio signal, which can make it difficult to detect approximate repetitions in the dataset. Some implementations use contrastive language audio pretraining (CLAP) descriptors, which are embedding vectors from a pre-trained deep neural network that aligns audio content with its semantic description. The choice of CLAP model varies between implementations. For example, some implementations replace the CLAP model with other deep neural network audio models that provide embedding vectors for the audio signal.

[0056] Additionally, for each query q, some implementations use a bias term to discount its similarity to each reference r based on the average similarity between q and its K nearest neighbors in the background set of other samples, thus obtaining a normalized similarity score:

[0057] in, It is the k-th nearest neighbor of q in the background set (based on the similarity of their descriptors), and It is a scalar. Therefore, during the retrieval phase, we retrieve the following query set:

[0058] Then, during the verification phase, they are checked together with their best match.

[0059] Figure 4 A block diagram of a method for calculating a normalized similarity score according to some embodiments is shown. The method compares a query audio sample q 110 and a reference audio sample r 460. In some embodiments, the comparison can be mathematically described by equation (2) above.

[0060] The similarity metric 430 takes two audio signals 110 and 460 as input and outputs a single number between 0 and 1, where 1 indicates that the two audio signals are identical and 0 means that they have no similar features. Cosine similarity from equation (1) above is commonly used as the similarity metric 430. However, using only the raw output of the similarity metric to find potential matches is insufficient to find approximate matches in a real-world set of audio signals. By normalizing the score, this method prevents some sounds (e.g., noise or clicks) from always being returned when searching for approximate matches in a data-driven manner.

[0061] This method normalizes the similarity score by subtracting the output of the similarity metric 430 from the bias term 425. The externally normalized bias term 410 is determined based on the time-frequency pattern of the query audio sample. For example, in some implementations, the method extracts the time-frequency pattern of the query audio sample and processes the extracted time-frequency pattern with a predetermined analysis function to produce the bias term. Some examples of analysis functions used by some implementations are based on low-level characteristics of the audio signal, such as spectral flatness measures or zero-crossing rates, which can be used as a measure of how noisy the audio sample is. Therefore, sounds with higher zero-crossing rates or spectral flatness can have higher bias terms 425 (as in noise-like sounds that typically have energy at most frequencies) to be more similar to a wide variety of sounds in terms of their time-frequency patterns. In different implementations, the method utilizes a learning function trained with machine learning to process the extracted time-frequency pattern to produce the bias term.

[0062] In some implementations, the bias term computed by similarity normalization 410 uses a background or training dataset with sounds that are different from the sound currently being searched for to potentially match. The bias term is computed as the average similarity from a subset of the background dataset most similar to the query sound q, as described in Equation (2), where the subset consists of the K nearest neighbors in the background dataset. It is useful that the background dataset is different in terms of the time-frequency patterns of the sounds included. If the background dataset overrepresents sounds with certain time-frequency patterns, the similarity normalization may overpenalize sounds with time-frequency patterns similar to those overrepresented in the background dataset. When combining the similarity metric computed by comparing the query audio sample q and the reference audio sample r with the bias from the similarity normalization, it is beneficial to have scaling weights 420 to weigh the two components of the similarity score. These scaling weights are non-negative numbers specified by the user and allow for a trade-off between the importance of the original similarity metric and the bias from the similarity normalization. In cases where the background dataset lacks diversity (which leads to lower quality bias from similarity normalization), a smaller scaling weight is chosen to compensate for this deficiency. However, if the background dataset is extensive and diverse, a larger scaling weight value can be used. Some implementations use a scaling weight of 0.5.

[0063] Figure 5 A block diagram of a method for determining a bias term 530 according to some embodiments is shown. Embodiments compare a query audio sample 110 with a set 540 of training audio samples to generate a set of training similarity measures 510, and determine the bias term 530 based on the average of the K training similarity measures closest to the training audio sample 530. In some implementations, to determine the bias term, the audio processing system updates the bias term by scaling the average of the training similarity measures 530 with a scalar. Some embodiments use a user-defined scalar, such as scaling weight 420. In alternative embodiments, the scalar is a function of the diversity of time-frequency patterns in the training audio samples. In some embodiments, this diversity is measured by comparing all sounds in the background dataset to each other according to the time-frequency similarity measures 430, and then using these similarities as an “expansion” to determine the scalar. The expansion measure is one or a combination of variance, standard deviation, median, and other statistical parameters.

[0064] Figure 6 A block diagram of a method for determining a bias term according to some embodiments is shown. To determine the bias term, the embodiment extracts 610 time-frequency patterns of query audio samples 110 and processes the extracted time-frequency patterns using a learning function trained with machine learning 630 and / or a predetermined analysis function 620 to generate a bias term.

[0065] Examples of extracted time-frequency patterns include the start times of different sound events in an audio signal, different harmonic modes, etc. These time-frequency patterns are typically represented using a time-frequency representation such as a spectrogram, and the frequency axis may also include perceptual frequency groupings, such as in a Mel spectrogram. Features extracted from powerful deep learning models can also be used as a representation of time-frequency patterns in audio signals. Examples of the predetermined analysis function 620 include the spectral flatness or zero-crossing rate of the audio signal as a proxy for the degree to which the audio signal resembles noise. Examples of the learning function include support vector machines and neural networks trained with machine learning or the simple K nearest neighbor average from a background set. For example, in some implementations, the learning function is trained using supervised machine learning with a bias term determined based on the average similarity metric 520 of the training audio samples.

[0066] Figure 7A A block diagram of a method for determining a similarity metric for audio samples according to one embodiment is shown. To compare a query audio sample 710 with a reference audio sample 715, this embodiment is configured to calculate Mel spectrograms 720 and 725 for the query audio sample 710 and the reference audio sample 715, respectively, and determine a similarity metric between the query audio sample and the reference audio sample based on the cosine similarity 760 of the calculated Mel spectrograms.

[0067] In some implementations, this method is configured to normalize the Mel spectrogram using an internal normalization 770. For example, the internal normalization 770 converts Mel spectrograms 720 and 725 to decibel scales 730 and 735, respectively, and normalizes them 740 and 745 such that the largest interval in the Mel spectrogram has a value of 0 dB, and all intervals below -40 dB are clipped to a value of -40 dB. The implementation then flattens the two-dimensional (including time and frequency dimensions) Mel spectrograms (750 and 755) into one-dimensional vectors and calculates the cosine similarity 760 between the two vectors, as shown in Equation (1).

[0068] Figure 7BExamples of Mel spectrograms calculated at a typical resolution 723 are shown, as well as a coarse resolution employed in some embodiments 733. Some embodiments calculate Mel spectrograms at a coarse resolution 733, which has fewer than 20 Mel bands and intervals greater than 20 ms between consecutive time windows. Typically, Mel spectrograms are calculated using tens or hundreds of frequency intervals and short time steps (e.g., 20 ms) between time frames of spectrogram 723. Conversely, some embodiments use a coarse Mel spectrogram 733, for example, with only 16 Mel intervals and a time step of 96 ms. This approach effectively smooths out many low-level details of the signal, which, if present, can reduce the ability to find matches with very small time or frequency offsets. For example, a slightly off-key instrument playing the same musical passage may differ in low-level frequency details from an in tune version, and we would not be able to find these matches using a Mel spectrogram at typical resolution 723. However, by using a coarse-resolution Mel spectrogram 733, these low-level details are smoothed out, which is more effective in finding approximate matches of time-frequency patterns in a sound signal, such as similar frequency content at approximately the same time or the start and offset of sound events.

[0069] Figure 8 A block diagram of a method for determining a similarity measure of audio samples according to another embodiment is shown. To compare a query audio sample with a reference audio sample 810, this embodiment uses a neural network to compute 820 embeddings of the query audio sample and the reference audio sample, and determines 830 a similarity measure between the query audio sample and the reference audio sample based on the cosine similarity of the computed embeddings.

[0070] For example, this implementation can use a contrastive language audio pre-training (CLAP) model. This is a deep neural network that learns the common embedding space between a sound signal and its text description. The CLAP model takes the audio signal as input and returns a 512-dimensional vector. The implementation can compute a similarity metric by using the CLAP embeddings computed for the query audio sample q and the reference audio sample r as shown in equation (1).

[0071] In some implementations, the database of multiple reference audio samples includes query audio samples that compare the reference audio samples to each other. One implementation uses comparisons to refine the database of multiple reference audio samples when a duplicate indicated by the comparison results is detected. The refined database of multiple reference audio samples can be used to train audio deep learning models more efficiently, thereby improving computer operation.

[0072] Figure 9A block diagram is shown of a method for finding copies of training data according to some embodiments. The embodiments first acquire a training dataset 910 containing N audio signals, and use a similarity score calculation technique with external normalization, which uses each of the N audio files as a query sample to obtain... A similarity matrix 920. The implementation then sets the diagonal elements 930 of this self-similarity matrix to zero. This self-similarity matrix contains similarity scores comparing a sound to itself, as these do not contain useful information since any sound should be considered an approximate match to itself. The implementation then binarizes matrix 930 940 by setting all elements of the similarity matrix below a threshold (typically around 0.5) to zero, and setting all elements greater than or equal to the threshold to 1, such that... A matrix is ​​binary, meaning it contains only the values ​​zero and one.

[0073] The implementation finds 950 connected components in the binary similarity matrix, which are treated as a graph adjacency matrix containing N nodes. A value of 1 at the (i, j) index of the similarity matrix indicates that two nodes are connected in the graph, and a value of zero means they are not connected. Various methods can be used to find 950 connected components in the graph, where each connected component can be considered as a cluster of repeating sounds. For example, the processing of the similarity matrix for the case of N=4 is also shown for each processing step. In this example, there is a set of connected components in rows / columns 3 and 4, indicating that the sound files with indices 3 and 4 in the training set are duplicates.

[0074] Figure 10 A block diagram is shown illustrating a method, according to some embodiments, for finding copies of audio samples to improve the training of an audio deep learning model using, for example, a large dataset 1010 scraped from the Internet. Training 1040 large audio deep learning models can sometimes take weeks or months, is quite costly in terms of electricity, and renders the hardware unusable for other training. Reducing the size of the training dataset by removing copies 1020 to make it more efficient 1030 can potentially accelerate the training process. Furthermore, it has been shown that removing copies from the training dataset can potentially improve model performance and reduce privacy risks when training large deep learning models.

[0075] The introduction of audio generation models capable of generating realistic sound clips on demand from text descriptions has the potential to revolutionize how audio is utilized. However, evidence from generative models for text and images has shown extensive evidence that these models often replicate their training data rather than generating novel text / images. How these results extend to audio generation models, however, remains unknown. In this study, we propose a technique for detecting when an audio generation model replicates its training data. Our method uses score normalization to mitigate the problem that some sounds may inherently be more similar to all other sounds. Detecting such replication can be useful for subsequently preventing future replication of training data to ensure its privacy. Furthermore, our detection method is also useful for finding approximate copies in large training datasets, where knowledge of these copies can enable more efficient training algorithms, potentially saving significant computational costs.

[0076] The ability to generate novel digital data such as text, audio, and music using trained machine learning models has recently broken into public consciousness and is often referred to as “generative AI.” These models are typically controlled by natural language and are easy to use. However, it is recognized that generative models do not always create novel data and sometimes output exact or near-exact copies of the data included in the training set. This can lead to security issues, as the training data may be private and copying may violate copyright laws. Research attempting to answer the technical question of how generative models can remember and / or copy their training data in the case of images has begun to emerge. However, audio generative models (such as those based on latent diffusion models) are less mature, and methods for robustly detecting copied training data in audio models are an underdeveloped area.

[0077] Recent research on text-to-music generation has revealed evidence of the lack of novelty in training data memory used for transformer-based models and in samples generated from diffusion-based models. However, these studies fail to consider the fact that some sounds (e.g., constant white noise) may be inherently more similar to all other sounds, and therefore many of the matches they find are sounds lacking diversity (i.e., possessing static characteristics). Furthermore, methods such as audio fingerprinting excel at finding exact matches in short audio clips but may fail to make approximate matches; for example, they cannot identify training data copies in the audio generation model that may have artifacts or other variations but remain perceptually very similar to the training data samples.

[0078] Figure 11A schematic diagram is shown illustrating a method for employing a protection mechanism 1150 to prevent training data duplication 1140 according to some embodiments. These embodiments can be used to identify situations where the text-to-audio generation model 1120 duplicates its training data 1160 to pre-filter 1150 any text queries 1110 that might lead to training data duplication 1130.

[0079] Different implementations detect duplicates during the training and / or execution of the audio generation model. For example, one implementation is configured to train the audio generation model using a database of multiple reference audio samples; and to detect duplicates of the reference samples in the database of multiple reference audio samples and the audio samples generated by the audio generation model. Alternatively, another implementation is configured to execute the audio generation model trained using a database of multiple reference audio samples to generate audio samples; and to send the generated audio samples, unless the audio samples generated using external normalization are duplicates of one or more of the reference audio samples.

[0080] Figure 12 A schematic diagram is shown of a method for performing anomaly detection on a query audio sample based on the result of an audio comparison with an external normalization employed in some embodiments. The method compares the query audio sample 1210 with audio samples in a database representing normal (i.e., non-abnormal) audio samples. If a copy 1230 of sample 1210 with at least one element of database 1220 is found, the audio sample 1210 is considered normal, and the desired operation 1260 is performed. Otherwise, sample 1210 is considered anomalous 1250.

[0081] Figure 13 This is a detailed block diagram 1300 of an audio processing system 102 according to some embodiments of the present disclosure. In some exemplary embodiments, the audio processing system includes one or more sensors 1302, such as acoustic sensors, which collect data from the environment 1306 including query audio samples 110.

[0082] The audio processing system 102 includes a hardware processor 1308 that communicates with computer storage memory (such as memory 1310). Memory 1310 includes stored data, including algorithms, instructions, and other data that can be implemented by the hardware processor 1308. It is conceivable that the hardware processor 1308 may include two or more hardware processors depending on the requirements of a particular application. The two or more hardware processors may be internal or external. The audio processing system 102 may be combined with other components, including output interfaces and transceivers, as well as other devices.

[0083] In some alternative implementations, hardware processor 1308 may be connected to network 1312, which communicates with one or more data sources 1314, computer devices 1316, mobile phone devices 1318, and storage devices 1320. As a non-limiting example, network 1312 may include one or more local area networks (LANs) and / or wide area networks (WANs). Network 1312 may also include enterprise-wide computer networks, intranets, and the Internet. Audio signal processing system 1300 may include one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources may include a single device or multiple devices cooperating in a distributed environment within network 1312.

[0084] In some alternative implementations, hardware processor 1308 may be connected to network-enabled server 1322 connected to client device 1324. Hardware processor 1308 may be connected to external storage device 1326 and transmitter 1328. Furthermore, the output of each target speaker may be output according to the intended use 1330 of a particular user. For example, the intended use 1330 of a particular user may correspond to displaying speech as text (such as voice commands) on one or more display devices (such as monitors or screens), or inputting the text of each target speaker into a computer-related device for further analysis, etc.

[0085] Data source 1314 may include data resources for training recovery operator 104 for a speech enhancement task. For example, in one implementation, training data may include acoustic signals from multiple speakers speaking simultaneously, as well as background noise. Training data may also include acoustic signals from a single speaker speaking alone, acoustic signals from one or more speakers speaking in a noisy environment, and acoustic signals from the noisy environment itself.

[0086] Data source 1314 may also include data resources for training recovery operator 104 for a speech recognition task. The data provided by data source 1314 may include labeled and unlabeled data, such as transcribed and untranscribed data. For example, in one embodiment, the data includes one or more sounds and may also include corresponding transcribed information or tags that can be used to initialize the speech recognition task.

[0087] Furthermore, unlabeled data in data source 1314 may be provided by one or more feedback loops. For example, usage data from spoken search queries performed on a search engine may be provided as untranscribed data. Other examples of data sources, by way of example and not limitation, may include a variety of spoken audio or image sources, including streaming sound or video, web queries, mobile device camera or audio information, webcam feeds, smart glasses and smartwatch feeds, customer care systems, security camera feeds, web documents, directories, user feeds, SMS logs, instant messaging logs, spoken text transcriptions, user interactions in gaming systems (such as voice commands or captured images (e.g., depth camera images)), tweets, chat or video call logs, or social network media. The specific data source 1314 used may be determined based on the application, including whether the data is of a specific category (e.g., data only related to a specific type of sound, including, for example, machine systems, entertainment systems) or is inherently general (non-category-specific).

[0088] The audio processing system 102 may also include third-party devices, which may include any type of computing device, such as an automatic speech recognition (ASR) system on a computing device. For example, the third-party device may include a computer device or mobile device 1318. Mobile device 1318 may include a personal data assistant (PDA), smartphone, smartwatch, smart glasses (or other wearable smart devices), augmented reality headset, virtual reality headset, laptop computer, tablet computer, remote control, entertainment system, vehicle computer system, embedded system controller, appliance, home computer system, security system, consumer electronics device, or other similar electronic device. Mobile device 1318 may also include a microphone or line input for receiving audio information, a camera for receiving video or image information, or communication components (e.g., Wi-Fi functionality) for receiving such information from another source (such as the Internet or data source 1314). In one example implementation, mobile device 1318 may be able to receive input data, such as audio and image information. For example, input data may include a query from a speaker to the microphone of mobile device 1318 when multiple speakers in a room are speaking. The input data may be processed by the ASR in mobile device 1318 using audio processing system 102 to determine the content of the query. The audio processing system 102 can enhance input data by reducing noise in the speaker's environment, separating the speaker from other speakers, or amplifying the audio signal of the query and enabling the ASR to output an accurate response to the query.

[0089] In some example implementations, storage device 1320 may store information including data, computer instructions (e.g., software program instructions, routines, or services), and / or data related to the neural network model of audio processing system 102. For example, storage device 1320 may store data from one or more data sources 1314, one or more deep neural network models, information for generating and training deep neural network models, and computer-usable information output by one or more deep neural network models.

[0090] Furthermore, implementations of the disclosed subject matter can be carried out, at least partially, manually or automatically. They can be performed, or at least assisted by, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, for manual or automatic implementation. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks can be stored in a machine-readable medium. The processor can then perform the necessary tasks.

[0091] The embodiments described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors, whether provided in a single computer or distributed across multiple computers. Such a processor can be implemented as an integrated circuit, with one or more processors within an integrated circuit assembly. However, the processor can be implemented using circuitry of any suitable format.

[0092] Furthermore, the various methods or processes outlined herein can be encoded as software executable on one or more processors employing any of a variety of operating systems or platforms. Additionally, such software can be written using a variety of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine. Typically, in various implementations, the functionality of program modules can be combined or distributed as needed.

[0093] Furthermore, embodiments of this disclosure can be specifically implemented as a method, examples of which have been provided. Actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments can be constructed that perform actions in a different order than those shown, which may include performing some actions simultaneously, even if they are shown as sequential actions in the exemplary embodiments. Therefore, the appended claims are intended to cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

[0094] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adjustments and modifications can be made within the spirit and scope of this disclosure. Therefore, aspects of the appended claims cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. An audio processing system, wherein the audio processing system uses external normalization to compare a query audio sample with a database of multiple reference audio samples, the audio processing system comprising: processor; and a memory storing instructions that, when executed by the processor, cause the audio processing system to: The externally normalized bias term is determined based on the time-frequency pattern of the query audio sample; The query audio sample is compared with each of the reference audio samples to generate a similarity score for each comparison; The bias term is combined with the similarity score for each comparison to produce a normalized similarity score; The normalized similarity score is compared with a threshold to produce the comparison result; as well as Output the comparison results.

2. The audio processing system according to claim 1, wherein, To determine the bias term, the processor is configured to: The query audio samples are compared with a set of training audio samples to generate a set of training similarity metrics; and The bias term is determined based on the average of the training similarity measures of the K closest training audio samples.

3. The audio processing system according to claim 2, wherein, To determine the bias term, the processor is configured to: The bias term is generated by scaling the average of the training similarity metric with a scalar.

4. The audio processing system according to claim 3, wherein, The scalar is a function of the diversity of time-frequency patterns in the training audio samples.

5. The audio processing system according to claim 1, wherein, To determine the bias term, the processor is configured to: Extract the time-frequency pattern of the query audio sample; and The extracted time-frequency patterns are processed using a predetermined analysis function to generate the bias term.

6. The audio processing system according to claim 1, wherein, To determine the bias term, the processor is configured to: Extract the time-frequency pattern of the queried audio sample; as well as The extracted time-frequency patterns are processed using a learning function trained with machine learning to generate the bias term.

7. The audio processing system according to claim 6, wherein, The learning function is trained using supervised machine learning, with a bias term determined based on an average similarity metric of the training audio samples.

8. The audio processing system according to claim 1, wherein, In order to compare the query audio sample with the reference audio sample, the processor is configured to: Calculate the Mel spectrograms of the query audio sample and the reference audio sample; as well as The similarity score between the query audio sample and the reference audio sample is determined based on the cosine similarity of the calculated Mel spectrogram.

9. The audio processing system according to claim 8, wherein, The processor is configured to normalize the Mel spectrogram using internal normalization.

10. The audio processing system according to claim 8, wherein, The processor is configured to calculate the Mel spectrogram at a coarse resolution, the coarse resolution having fewer than 20 Mel bands and an interval greater than 20 ms between consecutive time windows.

11. The audio processing system according to claim 1, wherein, In order to compare the query audio sample with the reference audio sample, the processor is configured to: A neural network is used to compute the embeddings of the query audio sample and the reference audio sample; as well as The similarity score between the query audio sample and the reference audio sample is determined based on the calculated cosine similarity of the embedding.

12. The audio processing system according to claim 1, wherein, The database of the plurality of reference audio samples includes the query audio sample, such that the reference audio samples are compared with each other, wherein the processor is further configured to: The database of the plurality of reference audio samples is streamlined when a copy indicated by the comparison result is detected.

13. The audio processing system according to claim 12, wherein, The processor is also configured to: The audio deep learning model is trained using a streamlined database of multiple reference audio samples.

14. The audio processing system according to claim 1, wherein, The processor is also configured to: The audio generation model is trained using the database of the multiple reference audio samples to generate audio samples; and The generated audio samples are compared with the reference audio samples using the external normalization to detect whether at least some of the generated audio samples are copies of audio samples contained in a database of the plurality of reference audio samples.

15. The audio processing system according to claim 1, wherein, The processor is also configured to: Execute an audio generation model trained using a database of the multiple reference audio samples to generate audio samples; as well as Unless the audio sample generated using the external normalization is a copy of one or more of the reference audio samples, the generated audio sample is sent.

16. The audio processing system according to claim 1, wherein, The processor is also configured to: Anomaly detection of the queried audio sample is performed based on the comparison results.

17. An audio processing method, wherein the audio processing method uses external normalization to compare a query audio sample with a database of multiple reference audio samples, wherein, The method uses a processor coupled with stored instructions that implement the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising the following steps: The externally normalized bias term is determined based on the time-frequency pattern of the query audio sample; The query audio sample is compared with each of the reference audio samples to generate a similarity score for each comparison; The bias term is combined with each of the similarity scores to produce a normalized similarity score; The normalized similarity score is compared with a threshold to produce a comparison result; and Output the comparison results.

18. The audio processing method according to claim 17, further comprising the following steps: The query audio samples are compared with a set of training audio samples to generate a set of training similarity metrics; as well as The bias term is determined based on the average of the training similarity measures of the K closest training audio samples.

19. The audio processing method according to claim 17, further comprising the following steps: Calculate the Mel spectrograms of the query audio sample and the reference audio sample, wherein the Mel spectrograms are calculated at a coarse resolution, the coarse resolution having fewer than 20 Mel bands and an interval greater than 20 ms between consecutive time windows; and The similarity score between the query audio sample and the reference audio sample is determined based on the cosine similarity of the calculated Mel spectrogram.

20. A non-transitory computer-readable storage medium having a program implemented thereon, the program being executable by a processor to perform a method for comparing a query audio sample with a database of multiple reference audio samples using external normalization, the method comprising: The externally normalized bias term is determined based on the time-frequency pattern of the query audio sample; The query audio sample is compared with each of the reference audio samples to generate a similarity score for each comparison; The bias term is combined with each of the similarity scores to produce a normalized similarity score; The normalized similarity score is compared with a threshold to produce a comparison result; as well as Output the comparison results.

Images