Latent spatial representation of audio signals for audio content-based capture

Abstract

Methods and systems are provided for extracting features indicative of variations in pitch, timbre, decay, reverberation, and other psychoacoustic attributes from digital audio signals and training an artificial neural network model for generating, from the extracted features, a contextual latent space representation of the digital audio signals. Methods and systems are also provided for training an artificial neural network model for generating consistent latent space representations of digital audio signals, where the generated latent space representations are comparable for the purpose of determining psychoacoustic similarities between digital audio signals. Methods and systems are also provided for extracting features from digital audio signals and training an artificial neural network model for generating, from the extracted features, a latent space representation of the digital audio signals that is responsible for selecting salient attributes of the signals that are indicative of psychoacoustic differences between the signals.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method and system for learning and using a latent space representation of digital audio signals, and more particularly to such a computer-implemented method and system in the context of audio content-based capture.

Background Art

[0002] Psychoacoustics encompasses the study of the space of sound and the mechanisms of human sound perception. Unlike using visual information, it is usually more difficult for humans to verbally describe specific attributes of sound using objective terms. For example, there are no widely accepted objective terms for explaining differences in timbre.

[0003] Different people may describe the same timbre differently. For example, one person may describe timbre by the instrument producing the sound, another person may describe timbre by the sound quality and pitch (e.g., bright, metallic, shrill, too loud and unpleasant, annoying, noisy, etc.), and still another person may describe timbre by the emotion of the sound (e.g., excited, angry, happy, sad, etc.). Other elements of sound that are not easily explained, especially in music, include rhythm, melody, dynamics, and texture.

[0004] Despite this difficulty, many existing audio content retrieval computing systems are keyword-based. That is, audio content is tagged (e.g., indexed) with keywords that describe the audio content. Users of such computing systems then use the keywords to search for or browse the desired audio content.

[0005] Keyword tagging / indexing works well when audio content is tagged / indexed by objective attributes such as artist name, song title, music genre, chromatic pitch, beats per minute, or other objective attributes. However, keyword-based searching or browsing of audio content does not work well when it is difficult for the user to verbally describe the audio content they are looking for, or when the attributes of the desired audio content that would make it stand out to the user in a psychoacoustic sense are subjective or multi-factory. For example, a user may be looking for a vocal sample that sounds like a particular singer singing a particular melody in a particular way, but is not necessarily that exact part, melody, or way of singing it. Similarly, a user may be looking for a drum loop that sounds similar to a particular rhythmic pattern, but is not necessarily identical.

[0006] Recognizing similar sounds has long been important. Powerful computer implementation techniques exist for detecting similar sounds. The features of digital speech signals used for computer-based sound similarity recognition often include manually selected features such as the spectral centroid, spectral bandwidth, or spectral flatness of the digital speech signal. Manual methods exist for feature selection for sound similarity detection, and these methods offer the advantage of complete knowledge and control over how the digital speech signal is represented, and allow for fine-tuning of the configuration of selected features according to the requirements of the particular implementation at hand. Unfortunately, these methods are often rendered ineffective by omitting useful distinguishing features, failing to recognize useful distinguishing features, or using largely redundant features.

[0007] This invention addresses this and other needs.

[0008] The methods described in this section are methods that may be pursued, but they are not necessarily methods that have been previously conceived or pursued. Therefore, unless otherwise indicated, none of the methods described in this section should be considered prior art simply because they are included in this section.

[0009] Some embodiments of the present invention are shown in the drawings of the accompanying drawings, not as limitations but as examples, where similar reference numerals refer to similar elements. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of a system for learning the latent spatial representation of a digital audio signal, according to several embodiments of the present invention. [Figure 2] This is a schematic diagram of an artificial neural network in a system for learning the latent spatial representation of a digital audio signal, according to some embodiments of the present invention. [Figure 3] This is a schematic diagram of the architecture of an artificial neural network in a system for learning the latent spatial representation of a digital audio signal, according to some embodiments of the present invention. [Figure 4] This is a flowchart of the process performed by a system for learning the latent spatial representation of a digital audio signal, according to some embodiments of the present invention. [Figure 5] This is a mockup of an exemplary graphical user interface for a similar-sound application in an audio content-based retrieval system, according to several embodiments of the present invention. [Figure 6] Figure 5 shows a mockup of a state change in an exemplary graphical user interface in response to end-user input, according to several embodiments of the present invention. [Figure 7] This is a schematic diagram of an exemplary computing device in which several embodiments of the present invention may be implemented. [Figure 8]Figure 7 is a schematic diagram of an exemplary software system for controlling the operation of an exemplary computing device. [Modes for carrying out the invention]

[0011] The figures illustrate several embodiments of the present invention for the purpose of providing clear examples, but some embodiments of the present invention may omit, add, rearrange, or modify any of the elements shown in the figures.

[0012] In the following description, many specific details are given to provide a complete understanding of some embodiments of the present invention for illustrative purposes. However, it is clear that some embodiments of the present invention can be carried out without these specific details. In other examples, well-known structures and devices are shown in block diagram form to avoid unnecessarily complicating some embodiments of the present invention. 1.0 Summary 2.0 Implementation of an Exemplary Latent Space Representation Learning System 2.1 Loop and One-Shot 2.2 Sound Content Categories 2.3 Read / Decode Logic 2.4 Conversion Logic 2.4.1 Preprocessing 2.4.2 Duration Normalization 2.4.3 Time-Frequency Representation 2.4.4 Loop Signal Extension 2.5 Learning Logic 2.5.1 Exemplary Artificial Neural Networks 2.5.2 Exemplary Artificial Neural Network Architectures 3.0 Exemplary Latent Space Representation Learning Process 4.0 Exemplary Application of Sound Embedding 4.1 Similar Sound Search 4.2 Sound as Search Input 4.3 Detection of nearly overlapping sounds 5.0 Implementation Mechanism - Hardware Overview 6.0 Implementation Mechanism - Overview of Software 7.0 Other Aspects of the Disclosure

[0013] 1.0 Summary According to some embodiments of the present invention, a method and system for learning an artificial neural network model for generating a contextual latent space representation of a digital audio signal are provided, which extract features indicating variations in pitch, timbre, attenuation, reverberation, and other psychoacoustic attributes from the digital audio signal and generate the contextual latent space representation of the digital audio signal from the extracted features. Learning such a model for generating such a contextual latent space representation complicates existing learning systems and methods.

[0014] According to some embodiments of the present invention, a method and system for learning an artificial neural network model for generating a consistent latent space representation of a digital audio signal are provided. The generated latent space representation is comparable for the purpose of determining the psychoacoustic similarity between digital audio signals. Learning such a model for generating such a consistently comparable learned latent space representation complicates existing learning methods and systems.

[0015] According to some embodiments of the present invention, a method and system for learning an artificial neural network model for generating a latent space representation of a digital audio signal that is responsible for selecting prominent attributes of the signal that represent psychoacoustic differences between signals from the extracted features are provided. Such selection complicates existing techniques.

[0016] According to some embodiments of the present invention, machine learning techniques are used to learn a latent space representation of a digital audio signal in the form of a set of vectors or numerical codes, hereinafter referred to as "audio embeddings" herein. Audio embeddings can be learned without a teacher, thereby reducing or eliminating the need for manually generated or computer-generated labeling of training examples using a target variable (label). Further, since audio embeddings can encode attributes of audio signals that may not even be recognized by manual techniques performed by humans or are difficult for humans to fully understand, the learned audio embeddings can be of higher quality compared to more manual techniques (i.e., can represent better the differences in sounds). Higher quality audio embeddings mean a better representation of the audio signal, which translates to more consistent and similar measurements between signals, and ultimately leads to high-quality and more robust audio similarity search and recommendation.

[0017] 2.0 Latent Space Representation Learning System FIG. 1 is a schematic diagram of a system for learning an artificial neural network model for generating a latent space representation of a digital audio signal according to some embodiments of the present invention. System 100 reads a digital audio signal source (e.g., a data file containing digital audio data) and is programmed or configured to train an artificial neural network based on the digital audio signal to learn a model that can generate a set of numerical codes (audio embeddings) that provide a latent space representation of the digital audio signal.

[0018] The set of numerical codes generated by a learning model of a digital audio signal that provides a latent space representation of the digital audio signal may hereinafter be referred to as the "audio embedding" of the digital audio signal. The audio embedding can be represented in a computer storage medium by a vector containing the numerical codes. Each numerical code is typically a real number represented in the computer storage medium as a floating-point value.

[0019] For latent spatial representation, embedding sounds of digital audio signals that are closer to each other according to a distance criterion (e.g., cosine similarity, Euclidean distance) will reflect digital audio signals that sound more similar to each other as perceived by the human ear during playback, more so than embedding sounds of digital audio signals that are further apart according to the same distance criterion. Among other applications, sound embedding can be used to identify similar-sounding digital audio signals and to recommend digital audio signals in a corpus that sound similar to a reference digital audio signal.

[0020] Since the learning system 100 can be loaded with instructions in new ordered combinations, as otherwise disclosed herein, in order to implement the method of the present disclosure, the learning system 100 can be implemented using one or more physical computing devices, none of which are intended to be general-purpose computers. Figure 7, described below, is a schematic diagram of an example of a computing device that may be used in the learning computing system to implement the method of the present disclosure.

[0021] The components of system 100 may be implemented at least partially by computer hardware in one or more computing devices, such as one or more hardware microprocessors that execute stored program instructions stored in one or more computer memories to perform the methods described herein. Alternatively, one or more virtual machine instances in a shared computing facility, such as a cloud computing center, may be used.

[0022] The methods described herein are intended to demonstrate operations performed using programming on a dedicated or general-purpose computer. System 100 shows only one of many possible arrangements of components configured to perform the method. Other arrangements may contain fewer or different components, and the division of labor among the components may vary depending on the arrangement.

[0023] One or more components of system 100 may be implemented on the same computing device. Therefore, the isolation of components in system 100 merely illustrates one possible computing architecture. Components of system 100 may be distributed across multiple computing devices in a distributed computing configuration. For example, a distributed computing configuration may be used for reasons of computing power or scalability, according to the requirements of the particular implementation at hand.

[0024] 2.1 Loop and One-Shot System 100 may include read / decode logic 104. Logic 104 is programmed or configured to read a digital audio signal source 102, such as a file or other data container containing digital audio signal data (e.g., a computer database record). In some embodiments of the present invention, the digital audio signal contained in source 102 is a recording of human music or other auditory performance in the form of a loop or a one-shot. The loop or one-shot may also be generated by a computer or machine.

[0025] A loop is a repeating section of audio material created using one of several different techniques, including but not limited to programming using microphones, turntables, digital samplers, looper pedals, synthesizers, sequencers, drum machines, tape machines, delay units, and computer music software. Loops often contain a rhythmic pattern or sequence or progression of notes or chords that corresponds to several musical measures (1, 2, 4, 8 measures, etc.). Loops can repeat indefinitely, but can maintain an audible sense of musical continuity.

[0026] A one-shot can also be created using any one of several different techniques, such as those mentioned in the paragraph above. However, a one-shot is not like a loop in that the section of sound material does not repeat in the one-shot. Instead, a one-shot corresponds to a single audible event in which a sound pattern that usually decays exponentially has a clear beginning at the start of a subsequent event. For example, a one-shot may encompass a single note or chord sound, or a percussion strike (e.g., a strike of a drum kit piece).

[0027] The term "one-shot" refers to a single note, chord, strike, or other sound, but it is not limited to that and can include multiple notes, chords, strikes, or other sounds in sequence or progression. Furthermore, some one-shots may include the aforementioned single audible event pattern with a clear audible start at the beginning, followed by a sound pattern that decays exponentially, while other one-shots may not follow this pattern, such as inverse samples or certain sound effects. In any case, one-shots are, on average, shorter in playback length than loops.

[0028] The digital audio signal contained in Source 102 may include loops or one-shots, but the digital audio signal contained in Source 102 may also include sounds generated or recorded by any computer or machine within the human audible range of approximately 20 to 20,000 Hz.

[0029] In some examples, a digital audio signal source 102 belongs to a corpus of digital audio signal sources. A corpus may contain hundreds, thousands, millions, or more digital audio signal sources. For example, a corpus could be a library of sounds generated or recorded by users, computers, or machines, such as musical sample libraries provided by cloud-based music production and collaboration platforms, such as sounds available from SPLICE.COM in Santa Monica, California and New York City, New York.

[0030] 2.2 Sound Content Categories In some embodiments of the present invention, an artificial neural network is trained to learn a model for generating sound embeddings for digital audio signals of a given sound content category. There may be multiple different sound content categories and multiple corresponding models trained using the artificial neural network. Different models may be used to generate sound embeddings for digital audio signals of different sound content categories.

[0031] Different sound content categories may reflect general differences in the underlying statistical distributions of digital audio signals within those categories. Thus, sound content categories may correspond to classes or types of statistical distributions. By training an artificial neural network to learn a model for generating sound embeddings based on a set of training examples, all belonging to the same sound content category, the artificial neural network can learn the discriminative features of digital audio signals within sound content categories better than other training techniques, and generate higher-quality sound embeddings for digital audio signals within sound content categories. Ultimately, these higher-quality sound embeddings can be used to make better recommendations for similar sounds within sound content categories.

[0032] A set of sound content categories may be predefined, and digital audio signal sources (e.g., 102) may be assigned to or tagged with one or more sound content categories to which they belong. For example, digital audio signal sources may be indexed in a database by labels / tags of one or more sound content categories, or otherwise associated with labels / tags of one or more sound content categories.

[0033] Sound content category labels / tags can be assigned to digital audio signal sources based on the underlying statistical distribution of the digital audio signal. For example, a one-shot digital audio signal typically shows a clear start at the beginning of the signal, usually has a short duration, and has a release pattern that usually decays exponentially. Or, an inverse one-shot is the reverse of this pattern. On the other hand, a loop digital audio signal typically exhibits a periodic or quasi-periodic pattern over the duration of the signal.

[0034] According to some embodiments of the present invention, a corpus of digital audio signal sources is divided into at least two distinct sound content categories, namely loops and one one-shots. Loops and one-shots can be assigned to different sound content categories based on their typically fundamentally different statistical distributions over the time of the signal.

[0035] Loop sound content categories and one-shot sound content categories may be further subdivided based on instruments, instrument types, genres, moods, or other sonic attributes suitable for the requirements of the particular instrument at hand, in order to form a hierarchy of sound content categories. For example, the hierarchy of sound content categories may include top-level sound content categories, namely "loops" and "one-shots." Then, each of those top-level sound content categories may include a second-level category, such as "drums" and "instruments." Instruments may include vocals and instruments other than drums. In this example, four sound content categories are provided: "drum loops," "dump one-shots," "instrument loops," and "instrument one-shots."

[0036] Loops, one-shots, drums, and instruments (e.g., vocals or instruments other than drums) are non-limiting examples of attributes that reflect different statistical distributions of digital audio signals over time, which can be used to segment a corpus of digital audio signal sources into different sound content categories. However, other segmentations are possible, and the invention is not limited to any particular segmentation. For example, another possible hierarchical segmentation of a corpus of digital audio signal sources into sound content categories may be: One shot o drum o musical instruments ·loop o drum o musical instruments • Reverse sample o drum o musical instruments • Sound effects

[0037] Sound content categories may be selected heuristically or empirically according to the requirements of the particular implementation at hand, including based on the expected or discovered statistical distribution of different classes or types of digital audio signals in the corpus over time, or they may be learned or computed according to computer-implemented unsupervised clustering algorithms (e.g., exclusive, overlapping, hierarchical, or probabilistic clustering algorithms).

[0038] For example, digital audio signal sources can be grouped into different clusters (clustered) based on the similarity between one or more attributes extracted or detected from the source's digital audio signal. Such audio attributes that can lead to clustering of audio signals may include, for example, one or more of the statistical distribution of signal amplitude over time, zero crossing rate, spectral centroid, spectral density of the signal, spectral bandwidth of the signal, or spectral flatness of the signal. During clustering, digital audio signals that are more similar to one or more of these audio attributes are more likely to be clustered together within the same cluster, while digital audio signals that are less similar to one or more of these audio attributes are less likely to be clustered together within the same cluster.

[0039] As described above, a corpus of digital audio signal sources can be grouped into sound content categories, and separate artificial neural network models can be trained for each sound content category, using sets of digital audio signal sources assigned to or belonging to those sound content categories as training sets for learning. While a digital audio signal source in a corpus can belong to only one sound content category, it should be noted that a digital audio signal source may belong to multiple sound content categories, for example, if a duplicate clustering algorithm is used to identify sound content categories.

[0040] Therefore, the digital audio signal source 102 can be considered as one digital audio signal source belonging to a training set of multiple (e.g., hundreds, thousands, millions, or more) digital audio signal sources for training an artificial neural network to learn a model for generating sound embeddings for a particular sound content category. A training set containing digital audio signal sources, all belonging to a particular sound content category, may be referred to herein as a “source” training set. A particular sound content category on which the artificial neural network is trained based on the source training set may be referred to herein as a “source” sound content category. And a model of a source sound content category learned by the artificial neural network as a result of training using the source training set may be referred to herein as a “target” model. Different sound content categories may have different source training sets, and different target models may be learned for different sound content categories.

[0041] 2.3 Read / Decode Logic As described above, system 100 may include read / decode logic 104. The read / decode logic 104 is programmed or configured to read a digital audio signal source 102 and generate a "raw" digital audio signal sample based on the digital audio signal source 102. The read / decode logic 104 is communicatively coupled to a conversion logic 106 and can send the raw digital audio sample to the conversion logic 106 for further processing.

[0042] The digital audio signal of Source 102 may be uncompressed, compressed in a lossless encoding format, or compressed in a lossy encoding format. Exceptional possible formats for the digital audio signal of Source 102, indicated by their known file extensions, may include, but are not limited to, .AAC, .AIFF, .AU, .DVF, .M4A, .M4P, .MP3, .OGG, .RAW, .WAV, and .WMA.

[0043] System 100 includes read / decode logic 104 that reads source 102, decodes / decompresses the digital audio signal contained therein as necessary, and generates a raw digital audio signal sample based on the digital audio signal.

[0044] The raw digital audio signal samples generated by the read / decode logic 104 can be obtained in the time domain of the digital audio signal. For example, pulse code modulation (PCM) samples are an example of possible raw digital audio signal samples. A PCM signal is a sequence of raw digital audio signal samples containing data that provides information for reconstructing the original audible audio signal. A PCM sample may represent the amplitude of the audible audio signal at a particular point in time. PCM samples of a PCM signal may be spaced uniformly in time. Amplitude may be the only information explicitly contained in a PCM sample. For example, a PCM sample may contain an integer or floating-point number encoded as a binary number using several bits representing the bit depth of the PCM sample.

[0045] If the digital audio signal itself already consists of raw digital audio signal samples, decoding of the digital audio signal source 102 may not be required by the read / decode logic 104. For example, the digital audio signal may not require decoding and may contain raw digital audio signal samples in WAV format. On the other hand, the .MP3 format is a compressed, lossy format in which decoding of the digital audio signal may be required to generate raw digital audio signal samples. Therefore, the read / decode logic 104 can be programmed or configured to include various digital audio signal source 102 readers and decoders for various different audio formats. The read / decode logic 104 can then select an appropriate reader and / or decoder to generate raw digital audio signal samples based on the specific format of the digital audio signal of source 102.

[0046] If the raw digital audio signal is generated by the read / decode logic 104 from the digital audio signal of source 102, or if the digital audio signal is already a raw digital audio signal, then the raw digital audio signal can be provided to the conversion logic 106 (e.g., sent or transmitted).

[0047] 2.4 Conversion Logic According to some embodiments of the present invention, the conversion logic 106 converts the raw digital audio signal received from the read / decode logic 104 into one or more time-frequency representations of the raw digital audio signal. The conversion logic 106 then provides the one or more generated time-frequency representations to the learning logic 108 for further processing. According to some embodiments, the time-frequency representations are normalized and then input into an artificial neural network to train a target model. According to some embodiments of the present invention, the conversion performed by the conversion logic 106 varies depending on the source sound content category.

[0048] 2.4.1 Preprocessing In some embodiments of the present invention, the conversion logic 106 downsamples the raw digital audio signal. This downsampling may be performed to reduce the dimensionality of the time-frequency representation input to the artificial neural network during training of the target model. For example, the digital audio signal may be downsampled by half or nearly half from 44,100 Hz or 48,000 Hz to 22,050 Hz. By reducing the input dimensionality of the time-frequency representation, computing resources and the time required to train the target model can be reduced without significantly impacting the accuracy effectiveness of the target model.

[0049] In addition to downsampling, raw digital audio signals can be converted from stereo samples to mono audio samples (for example, by computing the center channel) to accelerate the training of artificial neural networks without significantly impacting the effectiveness of the trained target model. Alternatively, only the left channel or only the right channel may be used.

[0050] Other preprocessing of the raw digital audio signal may be performed by the conversion logic 106. For example, a pre-emphasis filter may be applied to balance the frequency spectrum between low-amplitude high frequencies and high-amplitude low frequencies, thereby avoiding numerical problems during the Fourier transform or improving the signal-to-noise ratio (SNR) of the signal.

[0051] Other preprocessing may be performed on the raw digital audio signal, such as compression or equalization, or other preprocessing appropriate to the requirements of the particular implementation at hand.

[0052] 2.4.2 Duration Normalization According to some embodiments of the present invention, the conversion logic 106 performs duration normalization on a raw digital audio signal or a pre-processed raw digital audio signal received from the read / decode logic 104. The raw digital audio signal or the pre-processed raw digital audio signal is hereafter referred to as the “target” signal. The target signal can be normalized to a predetermined duration. The predetermined duration may be based on the source sound content category. Different sound content categories may have different predetermined durations. However, different sound content categories may have the same predetermined duration.

[0053] Normalizing each such target signal with a source training set where each target signal has the same predetermined duration in the source sound content category makes it easier to extract a time-frequency representation of the same dimensions from the target signal for input to the artificial neural network during target model training. In other words, each time-frequency representation generated for each target signal for the source training set may have the same dimensions as the result of normalizing the target signals in the source training set to the same predetermined duration in the source sound content category. As used herein, a reference to “same” duration means that two durations are the same if they have the same length of time with a given time resolution (e.g., milliseconds). For example, two signals, both with a length of 500 milliseconds, have the same duration even if they have different lengths of time with nanosecond time resolution.

[0054] If the target signal has a duration longer than a predetermined duration, the duration normalization of the target signal to the predetermined duration by the conversion logic 106 can be achieved by truncating samples from the end of the target signal or by trimming samples from the beginning of the target signal. If the duration of the target signal is shorter than the predetermined duration, then dummy digital audio signal samples (e.g., zero-value samples) can be added to the target signal until the duration of the target signal becomes equal to the predetermined duration.

[0055] As a result of performing duration normalization on the target signal by the conversion logic 106, the target signal is normalized as a "duration-normalized" signal having a predetermined duration for the source sound content category. As an example of how the predetermined duration can vary between sound content categories, we consider the loop sound content category and the one-shot sound content category.

[0056] If the source sound content category is a one-shot sound content category, then the initial portion of the target signal where the clear start of the one-shot exists may be selected as the duration-normalized signal. This might correspond to, for example, the first second or so of the target signal in the case of a drum one-shot. Therefore, a given duration of the drum one-shot target signal could be, for example, slightly less than one second. In the case of an instrument or vocal one-shot target signal, the duration-normalized signal may be slightly longer than that of the drum one-shot target signal (for example, within the first few seconds), reflecting the typically relatively longer duration of the start before release in the instrument or vocal one-shot target signal. As a result, a given duration of the instrument or vocal one-shot target signal could be between approximately one and three seconds, for example. More generally, the portion of the one-shot target signal interpreted as the duration-normalized signal may be selected based on various attributes of the target signal, including, but not limited to, the portion of the target signal that contains the start of the one-shot sound.

[0057] On the other hand, if the source sound content category is a loop sound content category, then the given duration may be long enough to capture at least one cycle of the repeating periodic or quasi-periodic pattern of each loop signal in the source training set. For example, for a musical loop spanning two, four, or eight measures, the given duration may be longer than three seconds and shorter than five seconds. For example, if the source sound content category is a loop sound content category, the given duration of the source sound content category may be selected based on the average or average duration of one cycle of the repeating periodic or quasi-periodic pattern of the loop signals in the source training set.

[0058] 2.4.3 Time-Frequency Representation According to some embodiments of the present invention, the conversion logic 106 is programmed or configured to receive a duration-normalized signal and generate a time-frequency representation of the duration-normalized signal by calculating a time-frequency transform of the duration-normalized signal. The time-frequency representation is a representation of the spectrum of the frequency of the duration-normalized signal as the frequency of the duration-normalized signal changes over time. According to some embodiments of the present invention, the time-frequency transform is a complex-valued transform consisting of a complex number of the form z = a + ix, where "a" represents the real part of the number and "x" represents the imaginary part of the number.

[0059] According to some embodiments of the present invention, the time-frequency representation is an amplitude spectrogram calculated based on a complex spectrogram of a duration-normalized signal by calculating the absolute value of a complex spectrogram in which the y-axis represents frequency and the x-axis represents time frame. The value of each time-frequency slot represents a specific energy level of the duration-normalized signal. The magnitude matrix can be represented by the conversion logic 106 in the computer storage medium as a matrix of real numbers that can be represented by the conversion logic 106 as floating-point values ​​in the computer storage medium.

[0060] The conversion logic 106 may be programmed or configured to generate a time-frequency representation of a duration-normalized signal by using a time-frequency transform. Some examples of time-frequency transforms that may be used include, but are not limited to, the Short-Time Fast Fourier Transform (STFT), constant Q transform, wavelets, and filter banks.

[0061] The conversion logic 106 may optionally apply nonlinear interpolation to the time-frequency representation of the duration-normalized signal. Applying nonlinear interpolation to the time-frequency representation is intended to give greater weight and improved resolution to frequency bands with better resolution for the human auditory system, thereby improving its ability to learn the discriminative attributes of those frequencies. Under ideal laboratory conditions, the audible range can be as wide as 12 Hz to 28 kHz, but the human auditory system generally has an audible range of 20 Hz to 20 kHz. Therefore, from a practical standpoint of human listening, learning the discriminative attributes of speech is relevant only to frequencies within the human audible range.

[0062] Where applicable, the nonlinear interpolation applied by the transformation logic 106 may use one of many different interpolations based on the context of human auditory scales, sometimes called psychoacoustic scales. Examples of such psychoacoustic scales include the Mel scale, the Burke scale, and / or the equivalent rectangular bandwidth (ERB) scale. Thus, the transformation logic 106 may use one or more of these psychoacoustic scales to apply nonlinear interpolation to the time-frequency representation.

[0063] According to some embodiments of the present invention, the time-frequency transform applied to the duration-normalized signal is a complex short-time Fourier transform (STFT), and the transform logic 106 calculates the complex STFT for the duration-normalized signal according to a selected window size (e.g., 1024 samples) and a selected hop size (e.g., 128 samples, 256 samples, etc.), either or both of which may vary depending on the source sound content category. For example, the time frames of the complex STFT transform may overlap with each other according to the selected window size and the selected hop size. The selected hop size represents how many signal samples advance the starting point of the current time frame of the duration-normalized signal to set the starting point of the next time frame of the duration-normalized signal and undergo the complex STFT transform. For example, if a time frame has 1024 audio signal samples according to the selected window size and the selected hop size is 256 signal samples, then the starting point of the current time frame of the duration-normalized signal proceeds through 256 audio signal samples, resulting in an overlap of 768 audio signal samples (75% overlap) in the duration-normalized signal between the previous current time frame and the new current time frame.

[0064] In some embodiments of the present invention, the hop size of the complex STFT transform is selected based on the source sound content category. In particular, a smaller hop size results in more redundancy being captured between time frames. If the duration-normalized signal of a category has relatively fast transient events, or if the given duration of the duration-normalized signal of a category is relatively short for a source sound content category, such as a drum one-shot, a relatively smaller hop size may be used so that the target model can better distinguish transitions in the spectral time pattern of the duration-normalized signal of the category.

[0065] The disadvantage of using a smaller hop size is that there are more timeframes to process, and less time context is captured by a given number of consecutive timeframes. For source sound content categories that do not have fast transient events or that contain duration-normalized signals with longer predetermined durations, such as one-shot source sound content categories of instruments or vocals, a relatively large hop size may be used to reduce the number of timeframes processed by the transformation logic 108 and to capture a larger amount of time context per given number of consecutive timeframes, without making a significant sacrifice to the discriminative ability of the resulting target model.

[0066] According to some embodiments of the present invention, the magnitude STFT is calculated from the complex STFT of the duration-normalized signal. The magnitude spectrogram is then generated from the magnitude STFT. Nonlinear interpolation is then applied to the magnitude spectrogram. According to some embodiments of the present invention, the power spectrum STFT of the complex STFT is calculated by the transformation logic 106 instead of the magnitude STFT, and then nonlinear interpolation is applied to the power spectrum STFT.

[0067] According to some embodiments of the present invention, a window function, such as a Hamming window function or another suitable window function, is applied to each time frame of the duration-normalized signal in anticipation of a complex STFT transform on the time frame.

[0068] In some embodiments of the present invention, the conversion logic 108 generates a time-frequency representation (e.g., magnitude spectrum) N multiplied by M, where N is the number of different frequency bins and M is the number of time frames, which depends on the window size, hop size, and a given duration of the duration-normalized signal. In some embodiments of the present invention, N is 128 values ​​for each time frame of the duration-normalized signal. However, a specific number of values ​​N is not required.

[0069] As described above, each duration-normalized signal in a source sound content category can have the same predetermined duration. This duration may be referred to herein as the “target duration” of the source content category. For example, each duration-normalized signal in a “drum one-shot” sound content category may have a target duration of 0.78 seconds, each duration-normalized signal in an “instrument one-shot” sound content category may have a target duration of 1.48 seconds, and each duration-normalized signal in a “loop” sound content category may have a target duration of 3.75 seconds. However, a specific target duration is not required. And, as described above, different sound content categories may have different target durations.

[0070] As described above, the time dimension M of the time-frequency representation may vary depending on the target duration of the source sound category, the window size selected for the source sound content category, and the selected hop size. For example, a duration-normalized signal for a drum one-shot category with a target duration of 0.743 seconds and a selected hop size of 128 time-domain samples may have produced a time-frequency representation with a total of 16,384 values ​​multiplied by 128 for each time-frequency representation. A duration-normalized signal for an instrument one-shot category with a target duration of 1.486 seconds and a selected hop size of 256 time-domain samples may also have produced a time-frequency representation with a total of 128 values ​​multiplied by 128 for each. Even if the target duration of the instrument one-shot category is twice that of the drum one-shot category, the dimensions of the time-frequency representations generated for these two categories may be the same in M ​​dimension. This is because the instrument one-shot category is also twice the selected hop size of the drum one-shot category, assuming the same selected size for both categories. As another example, a duration-normalized signal for a drum loop category with a target duration of 3.75 seconds and a selected hop size of 256 may have produced a time-frequency representation with 320 dimensions, which is the sum of the dimensions of 40,960 values ​​per time-frequency representation multiplied by 128.

[0071] Before the time-frequency representation of the duration-normalized signal is passed to the learning logic 108 for training the target model, normalization may be performed on the values ​​of the time-frequency representation. For example, the values ​​of the time-frequency representation can be minimum-maximum normalized to values ​​containing 0 and 1, or between -1 and 1, to help the learning algorithm of the artificial neural network (e.g., gradient descent) converge. Other normalizations are possible, such as mean normalization, Z-score normalization, or scaling to unit length.

[0072] 2.4.4 Loop Signal Extension According to some embodiments of the present invention, multiple time-frequency representations are generated by the transformation logic 106 from a single duration-normalized signal of a loop source sound content category. This extension of the duration-normalized signal of a loop source sound content category to multiple time-frequency representations may be performed during training to learn a target model. In particular, this extension may be performed by the transformation logic 106 to provide a better temporal context when training a target model based on a training set of loop digital audio signal sources. The repeating patterns of the duration-normalized loop signal may have different durations and different periodic or quasi-periodic repeating patterns. Furthermore, the repeating patterns may not be aligned across the duration-normalized loop signals of the source training set. For example, some duration-normalized loop signals for the source training set may start at the beginning of the loop signal's repeating pattern, other duration-normalized loop signals in the source training set may start in the middle of the loop signal's repeating pattern, and yet another duration-normalized signal in the source training set may start at the end of the loop signal's repeating pattern.

[0073] To explain these differences within the duration-normalized loop signals of a source training set, multiple time-frequency representations can be generated from multiple randomly selected "slices" of each duration-normalized signal. Each such selected slice may have the same predetermined duration as the source sound content category, and therefore the same duration as the duration-normalized loop signal. For each selected slice, the time-frequency representation of the slice can be generated using the techniques described above to generate the time-frequency representation of the duration-normalized signal, except that the slice signal is used instead of the duration-normalized signal. In this way, multiple time-frequency representations can be generated for multiple slices of the duration-normalized signal.

[0074] When randomly selecting a slice of a loop duration-normalized signal, the start time of the slice may be randomly selected if the start time is within the duration of the loop signal. The slice can then be interpreted as a time-shifted version of the duration-normalized signal, starting at the randomly selected start time and then wrapping around the end of the duration-normalized signal to the start of the slice, so that the selected slice signal has the same duration as the duration-normalized signal. Alternatively, the selected slice signal may wrap around the beginning of the duration-normalized signal to the start of the slice signal. In this way, multiple slice signals of the duration-normalized signal can be obtained from different randomly selected start times during the duration-normalized signal, and each such slice signal has a predetermined duration of the source sound content category.

[0075] In some embodiments of the present invention, instead of randomly selecting slices from a duration-normalized signal, slices are randomly selected from the time-frequency representation of the entire duration-normalized signal. In these embodiments, a time frame of a magnitude spectrogram may be randomly selected to select slices of a time-frequency representation, such as a magnitude spectrogram. This slice may then be a time-shifted version of the magnitude spectrogram of the entire duration-normalized signal, starting at the randomly selected time frame, wrapping around the end or beginning of the magnitude spectrogram, and ending back at the randomly selected time frame. In this way, multiple randomly selected slices of the time-frequency representation of the entire duration-normalized signal can be obtained from different randomly selected starting time frames of the time-frequency representation.

[0076] In both cases, the duration-normalized loop signal can be extended to multiple time-frequency representations. In some embodiments, the multiple time-frequency representations are separate so that two of the time-frequency representations do not start at the same point in the duration-normalized signal.

[0077] By extending the duration-normalized loop signals of a source training set to multiple time-frequency representations, the diversity of training examples input to the artificial neural network can be increased without needing to acquire more digital audio signal sources to include in the source training set. For example, a source training set of 100,000 loop digital audio signal sources can thus be extended to millions of different training examples input to the artificial neural network during training.

[0078] For example, if batch gradient descent is used during training an artificial neural network based on a selected batch size N (e.g., 32), then N time-frequency representations can be generated from each duration-normalized loop signal for the source training set and passed through the artificial neural network in a forward pass before backpropagation. In doing so, the error for which the gradient is calculated is the average of the errors of all N time-frequency representations in the batch.

[0079] As a result of the processing performed by the conversion logic 106, one or more time-frequency representations of the duration-normalized signal may be provided to the learning logic 108 by the conversion logic 106.

[0080] It should be noted that the operations performed by the read / decode logic 104 and the conversion logic 106 described above may be rearranged in some examples. For example, a digital audio signal or raw audio signal from source 102 can first be duration-normalized to a duration-normalized signal by the conversion logic 106, and then the read / decode logic 104 can preprocess the duration-normalized signal to a preprocessed version of the duration-normalized signal, based on which conversion logic 106 generates one or more time-frequency representations.

[0081] 2.5 Learning Logic The read / decode logic 104 and the conversion logic 106 can be applied to each digital audio signal source in the source training set to generate a set of time-frequency representations. This set of time-frequency representations generated for the source training set can be used by the learning logic 108 to train an artificial neural network to learn a target model.

[0082] It should be noted that the learning logic 108 does not need to generate the entire set of time-frequency representations from the source training set before beginning to train the artificial neural network. Instead, the artificial neural network may be trained in a parallel pipeline manner, where training of the artificial neural network begins based on some of the time-frequency representations in the set, and the entire set is generated.

[0083] According to some embodiments, when training an artificial neural network to learn a target model given training examples, the artificial neural network is programmed or configured to attempt to capture a time-frequency representation, convert it to a reduced latent space representation, and convert the reduced latent space representation back to a time-frequency representation. During training, this may be repeated until the artificial neural network learns to generate a reduced latent space representation that can reproduce the time-frequency representation within an error range according to a loss function.

[0084] 2.5.1 Exemplary Artificial Neural Networks Figure 2 is a schematic diagram of an artificial neural network 216, which may be a component of the learning logic 110 for learning the embedding of an input time-frequency representation 214 of sound, according to several embodiments. As shown in the figure, the artificial neural network 216 comprises three components: an encoder 218, a latent space representation 220, and a decoder 222.

[0085] According to some embodiments, the artificial neural network 216 is programmed or configured to attempt to learn via backpropagation to generate a latent space representation 220 (by encoder 218) and to jointly attempt to learn via backpropagation to generate a time-frequency representation 224 output from the latent space representation 220 (by decoder 222), where the output time-frequency representation 224 is a close (irreversible) representation of the input time-frequency representation 214. As part of learning the target model, the discrepancy between the estimated / reconstructed time-frequency representation 224 and the input spectrogram 214 (quantified by a chosen loss function such as mean squared error) may be iteratively used to adjust weights across the network 216 to reduce the error between the input spectrogram 214 and the estimated / reconstructed time-frequency representation at the output of the network 224.

[0086] The latent spatial representation 220 represents the input time-frequency representation 214 as a lower-dimensional embedding containing fewer numerical values, but still retains significant identifying information about the input time-frequency representation 214. For example, if the input time-frequency representation 214 has a dimension of, say, 128 x 128 values, and the latent spatial representation 220 has a dimension of 256 numerical values, the dimensionality reduction could be 64:1. A specific dimensionality reduction ratio is not required, and 64:1 is just one example.

[0087] In some embodiments of the present invention, the artificial neural network 216 may be trained specifically for a particular source content category. That is, once trained, the target model resulting from training the artificial neural network 216 based on a source training set for a source content category may be used to generate sound embeddings only for audio signals of the source content category, and may not be used to generate meaningful sound embeddings for audio signals that do not belong to or are not assigned to a source content category.

[0088] The estimated / reconstructed time-frequency representation 224 may be irreversible with respect to the input time-frequency representation 214. That is, the estimated / reconstructed time-frequency representation 224 does not need to be the same (reversible) representation as the input time-frequency representation 224.

[0089] An advantage of the artificial neural network 216 is that it does not require pre-labeling of training examples in the source training set. Instead, the artificial neural network 216 can learn lower-dimensional representations of the training examples in the source training set in the form of latent space representations (e.g., 220), and these lower-dimensional representations hold meaningful discriminatory information about the underlying statistical distribution of the source sound content categories.

[0090] The artificial neural network 216 may have several hyperparameters. One hyperparameter may specify the dimension of the hidden layer responsible for learning the sound embedding, also known as the bottleneck layer 220. Other hyperparameters may include, among other possible hyperparameters, the number of hidden (e.g.) layers in the encoder 218 and decoder 222, and the loss function used during training (e.g., mean squared error).

[0091] 2.5.2 Exemplary Artificial Neural Network Architectures Figure 3 is a schematic diagram of possible architectures of an artificial neural network 216 according to several embodiments of the present invention. The architecture includes an input layer 326, an encoder intermediate layer 332, a bottleneck layer 330, a decoder intermediate layer 334, and an output layer 328. The input layer 326 may contain normalized values ​​of the time-frequency representation 214 for compression to the bottleneck layer 330 by the encoder intermediate layer 332. Similarly, the output layer 328 may contain normalized values ​​of the estimated / reconstructed time-frequency representation 224 generated from the bottleneck layer 330 by the decoder intermediate layer 334. The bottleneck layer 330 may be considered the output layer of the encoder 218 and the input layer of the decoder 222.

[0092] During training via backpropagation, the time-frequency representation 214 of the input layer 326 passes through the intermediate layer 332 of the encoder 218 to generate a latent spatial representation 220 in the bottleneck layer 330. The encoder 218 may be a fully connected artificial neural network. Similarly, the decoder 222 may be a fully connected artificial neural network.

[0093] The latent space representation 220 of the bottleneck layer 330 passes through the decoder 222's intermediate layer 334 to generate an estimated / reconstructed time-frequency representation 224 in the output layer 328. Thus, the dimensions of the input layer 326 and the output layer 328 may be the same. The dimension of the bottleneck layer 330 is less than the dimensions of the input layer 326 and the output layer 328. The number of encoder intermediate layers 332 may be one or more. If there are multiple, the intermediate layers closer to the bottleneck layer 330 may, at will, have fewer units than the next closest intermediate layer 332 in order to force the encoder 218 to learn to generate a summarized, irreversible, and dimensionality-reduced latent space representation 220 of the time-frequency representation 214 in the bottleneck layer 330.

[0094] The number of decoder intermediate layers 334 may also be one or more, and may have the same or different number of layers as the encoder intermediate layers 332. If there are multiple decoder intermediate layers 334, then the intermediate layers of decoder 222 closer to output layer 328 may have more than the next closest intermediate layer 334 to force decoder 222 to learn to generate the estimated / reconstructed time-frequency representation 224 of output layer 328 from the latent space representation 220 of bottleneck layer 330, which is close to the time-frequency representation 214. All intermediate layers 332 and 334 of encoder 218 and decoder 222, respectively, may optionally have a higher dimension than bottleneck layer 330. Input layer 326 and output layer 328 may optionally have a higher dimension than intermediate layers 332 and 334.

[0095] In some embodiments of the present invention, the encoder intermediate layer 332 includes a convolutional layer, an activation layer, and a pooling layer, and the decoder intermediate layer 334 includes a convolutional layer, an activation layer, and an upsampling layer. By using convolutional layers in the encoder intermediate layer 332 and the decoder intermediate layer 334, the artificial neural network 216 can be trained without having to modify (and consequently lose spatial information) the time-frequency representation 214, which may be required by feedforward neural network implementations using high-density layers. For example, flattening the two-dimensional time-frequency representation into a one-dimensional vector can be avoided by using convolutional layers. Furthermore, the use of convolutional layers allows the learning process to capture the important two-dimensional spectral time patterns presented in the magnitude STFT.

[0096] In some embodiments of the present invention, the encoder intermediate layer 332 or decoder intermediate layer 334 includes one or more recurrent layers in addition to, or instead of, one or more convolutional layers. For example, the encoder intermediate layer 332 or decoder intermediate layer 334 may include one or more long-short-term memory (LSTM) layers or one or more gated recurrent unit (GRU) layers in addition to, or instead of one or more convolutional layers.

[0097] 3.0 Exemplary Latent Space Representation Learning Process Figure 4 is a flowchart of an exemplary process 436 performed by system 100 to learn a latent spatial representation of a digital audio signal, according to several embodiments. Figure 4 is intended to disclose an algorithm or function description that can be used as a basis for writing a computer program to implement a function that causes one computing system of one or more computing devices to operate in a novel way disclosed herein. Furthermore, Figure 4 is provided for a person skilled in the art to whom this disclosure is intended to be used to communicate such an algorithm with the same level of detail that is typically used among those skilled in the art to communicate plans, designs, and algorithms for other computer programs of similar complexity.

[0098] Process 436 may begin in step 438. In step 438, system 100 obtains a source training set. A source training set may be obtained by tagging or indexing digital audio signal sources, including digital audio signals, in a specific source sound content category. Alternatively, a source training set may be obtained by applying a clustering algorithm to a corpus of digital audio signals to identify a subset of similar digital audio signals that must together belong to a specific source sound content category. In either case, the digital audio signals in the source training set may all be similar to one another in that they all have the same general statistical distribution in the time domain. The statistics considered here may vary, but may include, for example, the time length of the digital audio signals, or general similarities in one or more audible sound attributes of the digital audio signals over time, such as spectral centroid, spectral flatness, or spectral bandwidth. For example, the digital audio signals of a source training set may all have one-shot patterns, all have drum one-shot patterns, all have drum one-shot patterns, all have vocal one-shot patterns, all have instrument one-shot patterns, all have loop patterns, all have vocal loop patterns, all have instrument loop patterns, and all have inverse one-shot patterns.

[0099] In step 440, the system 100, including the read / decode logic 104 and the conversion logic 106, generates one or more time-frequency representations from each digital audio signal in the source training set. This may include, as described above, for each digital audio signal, generating a target signal from the digital audio signal, generating a duration-normalized signal from the target signal, and generating one or more time-frequency representations from the duration-normalized signals.

[0100] In step 442, the system 100, including the learning logic 108, trains an artificial neural network via backpropagation to learn to generate a latent spatial representation of the time-frequency representation generated in the source training set. The artificial neural network may be trained using batch gradient descent, stochastic gradient descent, or minibatch gradient descent. In some embodiments, the artificial neural network is trained using minibatch gradient descent.

[0101] In step 444, once the artificial neural network is trained based on the source training set, the target model is stored in the model database 110 by the system 100. The target model may include the learned parameters of the artificial neural network as a result of training 442. For example, the target model may include the learned parameters of the encoder (e.g., 218) of the artificial neural network.

[0102] In step 446, the target model is assigned to a source content category or used to generate an audio embedding for a digital audio signal belonging to a source content category. To do this for a specific reference digital audio signal, system 100, including read / decode logic 104 and transform logic 106, generates a time-frequency representation of the reference digital audio signal. To do this, for example, read / decode logic 104 may be used to generate a target signal from the reference digital audio signal, and transform logic 106 may be used to generate a duration-normalized signal from the target signal. Transform logic 106 may be used to generate a time-frequency representation of the duration-normalized signal. If the reference digital audio signal is a loop signal, a single time-frequency representation of the reference digital audio signal may be generated, unlike during training where multiple time-frequency representations can be generated from the signal loop signal. Instead, read / decode logic 104 may be used to generate a duration-normalized signal from the reference digital audio signal, and transform logic 106 may be used to generate a target signal from the duration-normalized signal. Transform logic 106 may be used to generate a time-frequency representation of the target signal.

[0103] In any case, the time-frequency representation of the reference digital audio signal can be normalized during training using training examples, as is done by the conversion logic 106.

[0104] The generated normalized time-frequency representation of a reference digital audio signal can be input to an artificial neural network configured with the parameters of a target model in a forward pass to generate a latent space representation (i.e., a sound embedding) as an output. For example, the encoder 218 of the artificial neural network 216 may be configured with the learned parameters of a target model, and the normalized time-frequency representation can be processed from the input layer 326 in a forward pass that goes only through the encoder 218 (i.e., not through the decoder 222) to generate a sound embedding that captures the latent space representation for a particular reference digital audio signal in the bottleneck layer 330.

[0105] This can be repeated for other reference digital audio signals that are assigned to or belong to a source sound content category in order to generate sound embeddings for those signals. Sound embeddings have the useful property that the similarity between two digital audio signals in the source sound content category from which the sound embeddings are generated can be calculated by measuring the distance between the sound embeddings according to an appropriate distance criterion. The closer the distance, the more similar they are in a psychoacoustic sense. The farther the distance, the less similar they are in a psychoacoustic sense. For example, sound embeddings may be vectors of numbers, and those sounds can be similarly calculated by an appropriate distance criterion, such as the cosine distance between two vectors, where the closer the cosine distance, the more similar the audio signals sound.

[0106] Process 436 can be repeated for other sound content categories to train the target model and generate sound embeddings for the audio signals of these other sound content categories.

[0107] 4.0 Exemplary Application The sound embeddings produced using the techniques disclosed herein can be used in many useful applications. No specific application is required. Two specific exemplary applications are described here.

[0108] 4.1 Similar Sound Search According to some embodiments of the present invention, one or more digital audio signals of a particular sound content category that sound similar to a reference digital audio signal of that particular sound content category are identified and presented to the user in a graphical user interface. The similar-sounding digital audio signals may be identified by a server computing system ("server system") of one or more computing devices, and the graphical user interface may be presented to the user's personal computing device ("client device"). The server system and the client may be connected by one or more data communication networks, such as the Internet.

[0109] While the distinction between client and server provides a useful framework for describing similar-sounding embodiments, it is not a necessary distinction, and similar-sounding embodiments can be implemented in the context of other computing architectures. For example, functions described below as being performed by client and server systems may be embodied on a single computing device, some functions described as being performed by a server system may be performed on a client device in some implementations, or some functions described as being performed by a client device may be performed on a server system in some implementations.

[0110] Figure 5 is a mockup of possible computer graphical user interfaces for a similar sound application according to several embodiments of the present invention. In particular, the mockup is of Table 548 which may be presented as a graphical user interface on an end-user computing device. The end-user computing device may be a personal computing device such as a desktop computer or laptop computer, or a portable electronic device such as a mobile phone. The end-user computing device may also be a dedicated music mixing computing device, or other dedicated computing device including or electrically coupled to a video display. No specific type of end-user computing device is required. In this regard, it should be noted that the mockup in Figure 5 is provided for the purpose of illustrating possible implementations and is not intended to cover all possible graphical user interface implementations, which may vary according to the requirements of the particular implementation at present, including the type of end-user computing device on which the graphical user interface is presented (e.g., desktop computer vs. mobile computer).

[0111] Table 548 contains a list of one or more digital audio signal sources in a corpus of digital audio signal sources. The list may be presented in Table 548 in response to a variety of different end-user inputs. For example, the list may be presented as a search result in response to a keyword-based search query entered by an end-user via a graphical user interface to search the corpus for digital audio signal sources associated with, or tagged or indexed by, keywords. However, the list may also be presented for other reasons, such as a list of results from browsing the corpus, a list of popular sounds, a list of favorite sounds, or a list of recently used, accessed, or played sounds, and no specific reason is required.

[0112] Table 548 contains a row for each presented digital audio signal source. In this example, four rows 550, 552, 554, and 546 are presented, but there may be more or fewer in other situations. Each row corresponds to a digital audio signal and shows the signal's playback duration, the signal's waveform, the filename of the signal source, and provides user interface controls (e.g., selectable buttons or icons) to indicate corpus sounds that sound similar to the selected digital audio signal. Rows may contain more or less information. For example, a row may contain one or more sound content categories for the digital audio signal and provide other user interface controls for operations such as playing the audio signal, adding the audio signal source to favorites, or downloading the audio signal source.

[0113] Figure 6 is a mockup of the changes in the state of the graphical user interface in several embodiments, in response to an end user selecting or activating the “Show Similar” user interface control 558 in the state of the graphical user interface shown in Figure 5. In response to the end user's selection, several additional rows 658, 660, 662, 664, and 666 appear in table 548. The additional rows represent digital audio signal sources in the corpus that contain digital audio signals that sound similar to the selected reference audio signal source represented by row 552. In this example, all similar audio signals, including the reference audio signal, sound like a middle C being struck on an electric piano. However, not all audio signals sound identical.

[0114] To determine which audio signal in a corpus sounds most similar to a reference audio signal, sound embeddings generated for the signal by a target model of the sound content category to which the signal belongs may be used. For example, if the calculated cosine distance between sound embeddings for a signal is used as the similarity / dissimilarity criterion for a search algorithm, an approximate nearest neighbor search algorithm or other suitable search algorithm may be used to identify the top N (e.g., top 5) signals in the corpus that sound similar to the reference audio signal. In some embodiments, cosine distance is used as the similarity / dissimilarity criterion for a search algorithm, while in other embodiments, other distance measures are used. For example, Euclidean distance or other suitable distance criteria may be used.

[0115] Therefore, as a result of sound embedding being generated for digital audio signals in the corpus, end users can discover digital audio signal sources (e.g., files) containing digital audio signals that sound similar to a selected reference audio signal in the corpus without having to perform keyword searches for similar-sounding audio signals, and without having to listen to (play back) similar-sounding audio signals to determine whether they sound similar.

[0116] 4.2 Sound as Search Input According to some embodiments, instead of selecting an existing digital speech signal source within a corpus, the end user provides a digital speech recording captured using their personal computing device as input to a similar-sound lookup. In other words, the reference digital speech signal is a signal generated based on a microphone recording of a sound produced or played by the end user. The microphone may be a component of the end user's personal computing device or may be coupled to the personal computing device. The personal computing device may be coupled to a server system by a data communication network (e.g., the Internet). The server system may implement a similar-sound lookup and have access to a corpus of digital speech signal sources and embeddings of sounds generated and associated with them using the techniques disclosed herein.

[0117] For example, in some embodiments of the present invention, the end-user personal computing device is a mobile phone or other portable computing device configured with a microphone. The device is configured with an application (mobile application). Using the application, the end-user can select a “Start Recording” or similar option via the application’s graphical user interface controls. Selecting this option activates the microphone, and the device, in cooperation with the application, begins recording the sound captured by the microphone as a digital audio signal. The end-user can then produce a desired sound captured by the microphone. For example, the end-user may sing a melody of interest, play an instrument, tap a drum pattern of interest, or beatbox a drum or rhythm pattern of interest with their voice. When the end-user has finished producing the desired sound or performance, they can select a “Stop Recording” or similar option via the application’s graphical user interface controls.

[0118] The application may then provide the end user with an option to search for sounds similar to the recorded sound via a graphical user interface control. If this option is selected, the device cooperating with the application may submit the recorded digital audio signal to a server system via a network. The server system may generate sound embeddings for the digital audio signal using the techniques disclosed herein. For example, a magnitude spectrogram of a selected slice of the digital audio signal may be generated using the techniques disclosed herein, and the generated spectrogram may be input through a forward pass of an artificial neural network configured with a target model to generate sound embeddings for the recorded digital audio signal.

[0119] Once sound embeddings are generated, similar sounds within a corpus of digital speech signal sources can be identified using a search algorithm (e.g., approximate nearest neighbor search), and the top N most similar-sounding digital speech signal sources in the corpus can be returned to the end user's device for presentation to the end user by a mobile application. For example, table rows such as rows 658, 660, 662, 664, and 666 may be presented to the end user in the application's graphical user interface for the most similar-sounding digital speech signal sources.

[0120] In some embodiments of the present invention, the sound content category of a recorded digital audio signal is selected by an end user (e.g., via the graphical user interface control of an application). In some embodiments of the present invention, the sound content category of a recorded digital audio signal is automatically detected by an application or a server system. The server system can select an appropriate target model to use for generating sound embeddings for the recorded digital audio signal based on the selected or detected sound content category. The server system can also select a corpus of digital audio signal sources to search for similar sounds based on the selected or detected sound content category, such that all digital audio signal sources considered for search are assigned to the selected or detected sound content category, or belong to the selected or detected sound content category and have equivalent sound embeddings generated by the same target model used to generate sound embeddings for the recorded digital audio signal.

[0121] 4.3 Detection of nearly overlapping sounds According to some embodiments of the present invention, sound embedding generated for digital audio signals is used to detect nearly overlapping sounds. For example, existing digital audio signals in a corpus (library) of digital audio signals may contain original copyrighted works such as, for example, original vocal samples, original drum rhythms, or original instrumental progressions (e.g., original melodic riffs). The digital audio signals may be slightly modified by another person using one or more processes such as compression, equalization, time stretching, or pitch shifting, and this person may add them to the corpus and present them as their own original copyrighted works, generating new digital audio signals. A corpus curator or operator (e.g., a cloud-based digital music library provider) would appreciate techniques to assist in identifying such nearly overlapping digital audio signals for the purpose of preventing copyright disputes or for other purposes where identifying nearly overlapping sounds is useful.

[0122] Using the techniques disclosed herein for generating sound embeddings for digital speech signals, sound embeddings can be generated for digital speech signals submitted or uploaded for addition to a corpus or library of digital speech signals. Sound embeddings generated for submitted or uploaded digital speech signals can be compared by distance to sound embeddings generated for existing digital speech signals in the corpus or library. If the distance (e.g., cosine similarity, Euclidean distance, etc.) between (a) the sound embedding for the submitted or uploaded digital speech signal and (b) the sound embedding for the existing digital speech signal is below a threshold (i.e., close in distance), then the submitted or uploaded digital speech signal and the existing digital speech signal can then be flagged (indicated) (for example, in a computer database or a computer graphical user interface) as nearly overlapping with each other.

[0123] In some embodiments of the present invention, the distance threshold may vary depending on the sound content category of the digital audio signal. For example, the distance threshold may vary depending on the variance of the statistical distribution of the signal in the sound content category. When the variance of the statistical distribution in a given sound content category is small, a small distance threshold may be used for that sound content category, and when the degree of statistical variance is large, a larger distance threshold may be used. For example, since the statistical distribution of signals in a category may have many commonalities (e.g., starting location, uniform duration and decay pattern, low timbre diversity, etc.), signals in the drum one-shot category may have relatively small variance. As a result, sound embeddings generated for signals in the drum one-shot category may, on average, have smaller distances between them, and consequently, sound embeddings generated for signals in, for example, the instrument one-shot category or other category may have greater variance in the statistical distribution of the signal in that category. Consequently, in the case of nearly overlapping detections, a smaller distance threshold may be used for categories with smaller variance to improve the accuracy of the nearly overlapping detections (e.g., to reduce false positives). Larger distance thresholds can also be used for larger variance categories to improve the accuracy of nearly overlapping detections (e.g., to reduce missed detections).

[0124] 5.0 Implementation Mechanism - Hardware Overview Referring now to Figure 7, a block diagram shows an exemplary computing device 700 in which several embodiments of the present invention may be implemented.

[0125] The computing device 700 and its components, including connections, relationships, and functions, are provided for illustrative purposes only and are not intended to limit the implementation of the present invention. Other computing devices suitable for implementing some embodiments of the present invention may have different components, including components with different connections, relationships, and functions.

[0126] The computing device 700 includes a bus 702 or other communication mechanism for addressing the main memory 706 and for transferring data between and within the various components of the device 700.

[0127] The computing device 700 also includes one or more hardware processors 704 coupled to a bus 702 for processing information. The hardware processors 704 may be general-purpose microprocessors such as a central processing unit (CPU), a system-on-a-chip (SoC), a graphics processing unit (GPU), or other types of microprocessors.

[0128] Main memory 706, such as random access memory (RAM) or other dynamic storage devices, is coupled to bus 702 to store information and software instructions executed by processor(s) 704. Main memory 706 may also be used to store temporary variables or other intermediate information during the execution of software instructions by processor(s) 704.

[0129] When a software instruction is stored in a storage medium accessible to the processor(s) 704, it transforms the computing device 700 into a dedicated computing device customized to perform the operation specified in the software instruction. The terms “software,” “software instruction,” “computer program,” “computer-executable instruction,” and “processor-executable instruction” are broadly interpreted to cover any machine-readable information for instructing a computing device to perform a particular action, whether or not it is human-readable, and include, but are not limited to, application software, desktop applications, scripts, binaries, operating systems, device drivers, boot loaders, shells, utilities, system software, JAVASCRIPT®, web pages, web applications, plugins, embedded software, microcode, compilers, debuggers, interpreters, virtual machines, linkers, and text editors.

[0130] The computing device 700 includes a read-only memory (ROM) 708 or other static storage device coupled to the bus 702 for storing static information and software instructions for the processor(s) 704.

[0131] One or more mass storage devices 710 are coupled to the bus 702 for the permanent storage of information and software instructions on fixed or removable media such as magnetic, optical, solid-state, magneto-optical, flash memory, or any other available mass storage technology. The mass storage devices may be shared over a network, or they may be dedicated mass storage devices.

[0132] Typically, at least one of the mass storage devices 710 (for example, the device's main solid-state drive or hard disk) stores the main body of programs and data that instruct the operation of the computing device, including the operating system, user application programs, drivers, and all kinds of other data files, as well as other support files.

[0133] The computing device 700 may be coupled via bus 702 to a display 712, such as a liquid crystal display (LCD) or other electronic visual display, to display information to the computer user. In some configurations, a touch-sensing surface incorporating touch detection technology (e.g., resistive, capacitive, etc.) may be integrated with the display 712 to form a touch-sensing display for communicating touch gesture (e.g., finger or stylus) input to the processor(s) 704.

[0134] The input device 714, which includes alphanumeric keys and other keys, may be coupled to the bus 702 to communicate information and command selections to the processor 704. In addition to, or instead of, the alphanumeric keys and other keys, the input device 714 may include one or more physical buttons or switches, such as a power (on / off) button, a "home" button, and volume control buttons.

[0135] Another type of user input device may be a cursor control 716, such as a mouse, trackball, or cursor directional keys, for transmitting directional information and command selection to the processor 704 and for controlling the movement of the cursor on the display 712. This input device typically has two degrees of freedom on two axes, namely a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position on a plane.

[0136] In some configurations, such as the one shown in Figure 7, one or more of the display 712, input device 714, and cursor control 716 are external components (e.g., peripherals) of the computing device 700, while in other configurations, some or all of the display 712, input device 714, and cursor control 716 are integrated as part of the form factor of the computing device 700.

[0137] The functions of the disclosed systems, methods, and modules may be executed by a computing device 700 in response to a processor 704 executing one or more programs of software instructions contained in main memory 706. Such software instructions may be read into main memory 706 from another storage medium, such as a storage device 710. Upon execution of the software instructions contained in main memory 706, the processor 704 performs the functions of several embodiments of the present invention.

[0138] While the functions and operations of some embodiments of the present invention can be fully implemented by software instructions, hardwired or programmable circuits of the computing device 700 (e.g., ASICs, FPGAs, etc.) may be used in place of or in combination with software instructions to perform the functions in some embodiments of the present invention, according to the requirements of the particular implementation present.

[0139] As used herein, the term “storage medium” refers to any non-temporary medium that stores data or software instructions that cause a computing device to operate in a particular way. Such storage media may include non-volatile media or volatile media. Non-volatile media include, for example, non-volatile random access memory (NVRAM), flash memory, optical disks, magnetic disks, or solid-state drives such as storage device 710. Volatile media include dynamic memory such as main memory 706. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, other optical data storage media, physical media with perforation patterns, RAM, PROM, and EPROM, FLASH®-EPROM, NVRAM, flash memory, or any other memory chips or cartridges.

[0140] A storage medium is distinct from a transmission medium, but can be used in conjunction with one. The transmission medium is involved in the transfer of information between storage mediums. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires with bus 702. Transmission media can also take the form of sound waves or light waves, such as those generated between radio and infrared data communications.

[0141] Various forms of media may be involved in transporting one or more sequences of one or more software instructions to the processor(s) 704 for execution. For example, the software instructions may first be transported on a magnetic disk or solid-state drive of a remote computer.

[0142] A remote computer can load software instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem, local to computing device 700, can receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector receives the data carried by the infrared signal, and appropriate circuitry can place the data on bus 702. Bus 702 carries the data to main memory 706, and processor(s) 704 retrieves the data from main memory 706 and executes the software instructions. Software instructions received by main memory 706 may optionally be stored in storage(s) 710 either before or after execution by processor(s) 704.

[0143] The computing device 700 may also include one or more communication interfaces 718 coupled to the bus 702. The communication interfaces 718 provide bidirectional data communication coupled to a wired or wireless network link 720 connected to a local network 722 (e.g., an Ethernet® network, a wireless local area network, a cellular network, a Bluetooth® wireless network, etc.). The communication interfaces 718 transmit and receive electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information. For example, the communication interface 718 may be a wired network interface card, a wireless network interface card with an integrated wireless antenna, or a modem (e.g., ISDN®, DSL, or cable modem).

[0144] A network link(s) 720 typically provides data communication to other data devices over one or more networks. For example, a network link 720 may provide connection to data devices operated by a host computer 724 or an Internet service provider (ISP) 726 via a local network 722. The ISP 726 then provides data communication services through a global packet data communication network now commonly referred to as the “Internet” 728. The local network(s) 722 and the Internet 728 use electrical, electromagnetic, or optical signals to carry digital data streams. Signals carrying digital data to and from computing devices 700, and signals over network links(s) 720 and communication interfaces(s) 718, are exemplary forms of transmission media.

[0145] The computing device 700 can send messages and receive data, including program code, through a network(s), network links(s) 720, and communication interfaces(s) 718. In the internet example, server 730 may send requested code for an application program through the internet 728, ISP 726, local network(s) 722, and communication interfaces(s) 718. The received code can be executed by processor 704 as it was received, and / or stored in memory device 710 or other non-volatile storage for later execution.

[0146] The received code may be executed by the processor 704 as it was received, and / or stored in the memory device 710 or other non-volatile storage for later execution.

[0147] 6.0 Implementation Mechanism - Software Overview Figure 8 is a block diagram of an exemplary software system 800 that may be employed to control the operation of an exemplary computing device 800. The software system 800 and its components, including connections, relationships, and functions, are intended to be illustrative only and are not intended to limit the implementation of the present invention. Other software systems suitable for implementing some embodiments of the present invention may have different components, including components with different connections, relationships, and functions.

[0148] The software system 800 is provided to instruct the operation of the computing device 700. The software system 800, which may be stored in system memory (RAM) 706 and fixed storage device (e.g., hard disk or flash memory) 710, includes a kernel or operating system (OS) 810.

[0149] OS810 manages low-level aspects of computer operation, including process execution, memory allocation, file input / output (I / O), and device I / O management. One or more application programs, represented as 802A, 802B, 802C, ..., 802N, can be "loaded" (e.g., transferred from fixed storage 710 to memory 706) for execution by system 800. Applications or other software intended for use on device 800 can also be stored as a set of downloadable computer executable instructions, for example, for download and installation from an internet location (e.g., a web server, app store, or other online service).

[0150] The software system 800 includes a graphical user interface (GUI) 815 for receiving user commands and data in a graphical manner (e.g., "point-and-click" or "touch gesture"). These inputs can then be acted upon by the system 800 in accordance with instructions from the operating system 810 or application(s) 802. The GUI 815 also serves to display the results of operations from the OS 810 and application(s) 802, where the user can provide additional input or terminate the session (e.g., log off).

[0151] OS810 can run directly on the bare hardware 820 (e.g., processor(s) 704) of device 700. Alternatively, a hypervisor or virtual machine monitor (VMM) 830 may be interposed between the bare hardware 820 and OS810. In this configuration, VMM830 acts as a software "cushion" or virtualization layer between OS810 of device 700 and the bare hardware 820.

[0152] VMM830 instantiates and runs one or more virtual machine instances ("guest machines"). Each guest machine includes a "guest" operating system, such as OS810, and one or more applications, such as application802, designed to run on the guest operating system. VMM830 provides a virtual operating platform for the guest operating system and manages the execution of the guest operating system.

[0153] In some examples, VMM830 can allow a guest operating system to run as if it were running directly on the bare hardware 820 of device 700. In these examples, the same version of the guest operating system configured to run directly on the bare hardware 820 can also run on VMM830 without modification or reconfiguration. In other words, in some examples, VMM830 can provide the guest operating system with full hardware and CPU virtualization.

[0154] In other examples, the guest operating system may be specifically designed or configured to run on VMM830 for efficiency. In these examples, the guest operating system is "aware" that it is running on the virtual machine monitor. In other words, VMM830 can provide paravirtualization to the guest operating system in some examples.

[0155] The basic computer hardware and software described above are presented for the purpose of illustrating the fundamental computer components that may be employed to implement some embodiments of the present invention. However, the present invention is not necessarily limited to any particular computing environment or computing device configuration. Rather, some embodiments of the present invention may be implemented in any type of system architecture or processing environment that a person skilled in the art would understand, in view of this disclosure, to be able to support the features and functions presented herein.

[0156] 7.0 Other aspects of this disclosure Unless the context explicitly indicates otherwise, the term “or” is used in an inclusive (not exclusive) sense in the foregoing specification and the attached claims, for example, when used to connect a list of elements, the term “or” means one, some, or all of the elements in the list.

[0157] Unless the context explicitly indicates otherwise, the terms “comprising,” “including,” “having,” “based on,” and “encompassing” are used in an open-ended manner within the above-mentioned specification and the attached claims and do not exclude any additional elements, functions, actions, or operations.

[0158] Unless the context explicitly indicates otherwise, conjunctions such as the phrase "at least one of X, Y, and Z" are understood to convey that an article, term, etc., may be X, Y, Z, or a combination thereof. Therefore, such conjunctions are not intended to imply that a particular embodiment requires the presence of at least one X, at least one Y, and at least one Z, respectively.

[0159] Unless the context explicitly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms as used in the above detailed description and the attached claims.

[0160] Unless the context explicitly indicates otherwise, in the above detailed description and the appended claims, the terms, first, second, etc., are used herein to describe various elements in some examples, but these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, a first computing device may be referred to as a second computing device, and similarly, a second computing device may be referred to as a first computing device. Both the first computing device and the second computing device are computing devices, but they are not the same computing device.

[0161] In the above-described specification, several embodiments of the present invention have been described with reference to numerous specific details, which may differ from embodiment to embodiment. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. The sole and exclusive indicator of the scope of the present invention, and what the applicant intends to be the scope of the present invention, is the literal and equivalent scope of the set of claims issued from this application in the particular form in which such claims are issued, including any subsequent modifications.

Claims

1. It is a method, Reading a specific digital audio signal source from a set of digital audio signal sources associated with a specific sound content category, To generate a time-frequency representation based on the aforementioned specific digital audio signal, The method further includes learning a set of numerical codes representing a latent spatial representation of the time-frequency representation using an artificial neural network, wherein the set of numerical codes has dimensions less than the dimensions of the time-frequency representation, and the method further includes To train the artificial neural network, the process of reading, generating, and learning is repeated for each of the multiple other digital audio signal sources in the set of digital audio signal sources of the specific sound content category. From the aforementioned trained artificial neural network, obtain the learned set of model parameters, The computer storage medium stores the set of learned model parameters for the specific sound content category, To generate a first time-frequency representation based on a first digital audio signal associated with the aforementioned specific sound content category, To compute a first set of numerical codes representing the latent spatial representation of the first time-frequency representation, the learned set of model parameters is used, To generate a second time-frequency representation based on a second digital audio signal associated with the aforementioned specific sound content category, To compute a second set of numerical codes representing the latent spatial representation of the second time-frequency representation, the learned set of model parameters is used, Calculating the distance between the first set of numerical codes and the second set of numerical codes, Comparing the aforementioned distance with a distance threshold, Based on the fact that the distance is less than the distance threshold, the computer graphical user interface is made to display a message indicating that the first digital audio signal and the second digital audio signal contain sounds that substantially overlap. A method comprising selecting the distance threshold based on the aforementioned specific sound content category.

2. The method according to claim 1, further comprising causing a computer graphical user interface to display a message indicating that the first digital audio signal and the second digital audio signal contain similar sounds, based on the distance.

3. The further includes receiving the first digital voice signal from a computing device via a data communication network, The method according to claim 1 or 2, wherein the first digital audio signal is captured by the computing device using a microphone coupled to the computing device for recording human audible performance, or a microphone coupled to the computing device for recording human audible performance.

4. The method according to claim 1, further comprising calculating the distance between the first set of numerical codes and the second set of numerical codes based on the cosine similarity between the first set of numerical codes and the second set of numerical codes.

5. The method according to any one of claims 1 to 4, wherein the specific sound content category is selected from a group consisting of loops and one-shots.

6. The method according to any one of claims 1 to 5, wherein the specific sound content category is selected from a group consisting of drum loops, drum one-shots, instrument loops, and instrument one-shots.

7. The method according to any one of claims 1 to 6, wherein the artificial neural network comprises an input layer, one or more encoder intermediate layers, a bottleneck layer, one or more decoder intermediate layers, and an output layer, and the method further comprises obtaining from the bottleneck layer a set of numerical codes representing the latent spatial representation of the time-frequency representation.

8. The method according to claim 7, wherein the one or more encoder intermediate layers comprise one or more convolutional layers, and the one or more decoder intermediate layers comprise one or more convolutional layers.

9. The time-frequency representation is generated based on the aforementioned specific digital audio signal. Acquiring a pre-processed signal based on the aforementioned specific digital audio signal, A duration-normalized signal is generated based on the aforementioned pre-processed signal, The selection of a time-shifted slice signal of the duration-normalized signal, wherein the time-shifted slice signal has a specific duration. The method according to any one of claims 1 to 8, wherein the time-frequency representation is generated based on the time-shifted slice signal.

10. A method, Reading a specific digital audio signal source from a set of digital audio signal sources associated with a specific sound content category, To generate a time-frequency representation based on the aforementioned specific digital audio signal, The method further includes learning a set of numerical codes representing a latent spatial representation of the time-frequency representation using an artificial neural network, wherein the set of numerical codes has dimensions less than the dimensions of the time-frequency representation, and the method further includes To train the artificial neural network, the process of reading, generating, and learning is repeated for each of the multiple other digital audio signal sources in the set of digital audio signal sources of the specific sound content category. From the aforementioned trained artificial neural network, obtain the learned set of model parameters, The computer storage medium stores the set of learned model parameters for the specific sound content category, To generate a first time-frequency representation based on a first digital audio signal associated with the aforementioned specific sound content category, To compute a first set of numerical codes representing the latent spatial representation of the first time-frequency representation, the learned set of model parameters is used, To generate a second time-frequency representation based on a second digital audio signal associated with the aforementioned specific sound content category, To compute a second set of numerical codes representing the latent spatial representation of the second time-frequency representation, the learned set of model parameters is used, This includes calculating the distance between a first set of numerical codes and a second set of numerical codes. The time-frequency representation is generated based on the aforementioned specific digital audio signal. Acquiring a pre-processed signal based on the aforementioned specific digital audio signal, A duration-normalized signal is generated based on the aforementioned pre-processed signal, The selection of a time-shifted slice signal of the duration-normalized signal, wherein the time-shifted slice signal has a specific duration. Based on generating the time-frequency representation based on the time-shifted slice signal, the method is A method further comprising selecting the specific duration based on the specific sound content category.

11. The time-frequency representation is generated based on the aforementioned specific digital audio signal. The process involves generating a duration-normalized signal based on the aforementioned specific digital audio signal, To generate a preprocessed signal based on the aforementioned duration-normalized signal, The selection of a time-shifted slice signal of the pre-processed signal, wherein the time-shifted slice signal has a specific duration. The method according to any one of claims 1 to 8, wherein the time-frequency representation is generated based on the time-shifted slice signal.

12. The process involves generating multiple time-frequency representations based on the aforementioned specific digital audio signal, The method according to any one of claims 1 to 11, further comprising using the artificial neural network to learn a plurality of sets of numerical codes representing latent spatial representations of the plurality of time-frequency representations.

13. A computing system, One or more processors, Storage media and The system comprises instructions stored in the storage medium, and when the instructions are executed by the computing system, the computing system will... To generate multiple time-frequency representations based on multiple digital audio signals associated with a specific sound content category, Training an artificial neural network to learn multiple sets of numerical codes representing multiple latent spatial representations of the aforementioned multiple time-frequency representations, From the aforementioned trained artificial neural network, obtain the learned set of model parameters, The computer storage medium stores the set of learned model parameters for the specific sound content category, To generate a first time-frequency representation based on a first digital audio signal associated with the aforementioned specific sound content category, To compute a first set of numerical codes representing the latent spatial representation of the first time-frequency representation, the learned set of model parameters is used, To generate a second time-frequency representation based on a second digital audio signal associated with the aforementioned specific sound content category, To compute a second set of numerical codes representing the latent spatial representation of the second time-frequency representation, the learned set of model parameters is used, Calculating the distance between the first set of numerical codes and the second set of numerical codes, Comparing the aforementioned distance with a distance threshold, Based on the fact that the distance is less than the distance threshold, the computer graphical user interface is made to display a message indicating that the first digital audio signal and the second digital audio signal contain sounds that substantially overlap. A computing system that performs the task of selecting the distance threshold based on the aforementioned specific sound content category.

14. When executed by a computing device, the computing device will To generate multiple time-frequency representations based on multiple digital audio signals associated with a specific sound content category, Training an artificial neural network to learn multiple sets of numerical codes representing multiple latent spatial representations of the aforementioned multiple time-frequency representations, From the aforementioned trained artificial neural network, obtain the learned set of model parameters, The computer storage medium stores the set of learned model parameters for the specific sound content category, To generate a first time-frequency representation based on a first digital audio signal associated with the aforementioned specific sound content category, To compute a first set of numerical codes representing the latent spatial representation of the first time-frequency representation, the learned set of model parameters is used, To generate a second time-frequency representation based on a second digital audio signal associated with the aforementioned specific sound content category, To compute a second set of numerical codes representing the latent spatial representation of the second time-frequency representation, the learned set of model parameters is used, Calculating the distance between the first set of numerical codes and the second set of numerical codes, Comparing the aforementioned distance with a distance threshold, Based on the fact that the distance is less than the distance threshold, the computer graphical user interface is made to display a message indicating that the first digital audio signal and the second digital audio signal contain sounds that substantially overlap. A computer program that performs the task of selecting the distance threshold based on the aforementioned specific sound content category.

Images