Latent spatial representation of audio signals for audio content-based capture

Abstract

This invention provides a method and system for training an artificial neural network model to extract features indicating variations in psychoacoustic attributes from digital audio signals and generate contextual latent spatial representations. [Solution] The learning system consists of a read / decode logic that reads a specific digital audio signal source from a set of digital audio signal sources associated with a specific sound content category, a transform logic that generates a time-frequency representation based on the specific digital audio signal, and a learning logic that uses an artificial neural network to learn a set of numerical codes that provide a latent spatial representation of the time-frequency representation. The read, generate, and learn process is repeated to train the artificial neural network. The model database retrieves a set of model parameters learned from the trained artificial neural network and stores them in a computer storage medium.

Description

Technical Field

[0001] The present invention relates to a computer implementation method and system for learning and using a latent space representation of digital audio signals, and more particularly, to such a computer implementation method and system in connection with audio content-based capture.

Background Art

[0002] Psychoacoustics encompasses the study of the space of sound and the mechanisms of human sound perception. Unlike using visual information, it is usually more difficult for humans to verbally describe specific attributes of sound using objective terms. For example, there is no widely accepted objective term to explain the difference in timbre.

[0003] Various people may describe the same timbre differently. For example, one person may describe timbre by the musical instrument producing the sound, while another person may describe timbre by the sound quality and pitch (e.g., bright, metallic, shrill, too loud and unpleasant, annoying, noisy, etc.). Furthermore, another person may describe timbre by the emotion of the sound (e.g., excited, angry, happy, sad, etc.). Particularly in music, other elements of sound that are not easily described include rhythm, melody, dynamics, and texture.

[0004] Despite this difficulty, many existing audio content retrieval computing systems are keyword-based. That is, audio content is tagged (e.g., indexed) with keywords that describe the audio content. Such a computer ​​​​​The user of the recording system then uses keywords to select the desired audio content. Search or browse.

[0005] Keyword tagging / indexing is used for audio content, such as artist name and song title. , musical genre, chromatic pitch, beats per minute, or other objective attributes It works well when tagged / indexed by objective attributes such as gender. However, searching or browsing audio content based on keywords is what users are looking for. When it is difficult to express the audio content in words, or in a psychoacoustic sense, For the user, the attributes of the desired audio content that make the desired audio content stand out are subjective. It doesn't work well when there are multiple factors. For example, if a user has a specific singer, The way it's expressed makes it sound like a specific melody is being sung, but it's not necessarily exactly that part, or Sometimes you might be looking for a vocal sample that isn't a melody or a similar expression. Similarly, users may perceive something as sounding similar to a particular rhythmic pattern, but not necessarily identical. Sometimes you're looking for a drum loop that isn't just one.

[0006] Recognizing similar sounds has long been important. A powerful compilation for detecting similar sounds Computer implementation technology exists. It is used for computer-based sound similarity recognition. The characteristics of digital audio signals are often the spectral centroid and spectrum of the digital audio signal. Includes manually selected features such as bandwidth or spectral flatness. There are manual methods for feature selection, and these methods involve the digital audio signal being selected This provides the advantage of complete knowledge and control over how it is expressed, and the specific implementation at hand. This allows for fine-tuning of the configuration of the selected features according to the criteria. Unfortunately, these methods are In many cases, useful distinguishing features are omitted, or useful distinguishing features are not recognized, or Most of these are rendered ineffective by using redundant features.

[0007] This invention addresses this and other needs.

[0008] The methods described in this section are methods that may be pursued, but they do not necessarily match the previously intended approach. This is not a method that has been pursued. Therefore, unless otherwise indicated, the methods described in this section None of the methods described qualify as prior art simply because they are included in this section. It should not be considered to exist.

[0009] Some embodiments of the present invention are examples, not as limitations, and similar reference numbers are similar. The elements are shown in the attached diagram. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of a system for learning the latent spatial representation of a digital audio signal, according to several embodiments of the present invention. [Figure 2] This is a schematic diagram of an artificial neural network in a system for learning the latent spatial representation of a digital audio signal, according to some embodiments of the present invention. [Figure 3] This is a schematic diagram of the architecture of an artificial neural network in a system for learning the latent spatial representation of a digital audio signal, according to some embodiments of the present invention. [Figure 4] This is a flowchart of the process performed by a system for learning the latent spatial representation of a digital audio signal, according to some embodiments of the present invention. [Figure 5]A mock-up of an exemplary graphical user interface for a similar sound application in an audio content-based extraction system according to some embodiments of the present invention. [Figure 6] A mock-up of a state change in an exemplary graphical user interface of FIG. 5 in response to an end-user input according to some embodiments of the present invention. [Figure 7] A schematic diagram of an exemplary computing device in which some embodiments of the present invention may be implemented. [Figure 8] A schematic diagram of an exemplary software system for controlling the operation of the exemplary computing device of FIG. 7.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The figures illustrate some embodiments of the present invention for the purpose of showing clear examples, but some embodiments of the present invention may omit, add, rearrange, or modify any of the elements shown in the figures. Some embodiments may omit, add, rearrange, or modify any of the elements shown in the figures. Some embodiments may omit, add, rearrange, or modify any of the elements shown in the figures.

[0012] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of some embodiments of the present invention. However, it will be apparent that some embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring some embodiments of the present invention. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of some embodiments of the present invention. However, it will be apparent that some embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring some embodiments of the present invention. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring some embodiments of the present invention. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring some embodiments of the present invention. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring some embodiments of the present invention. 1.0 Overview 2.0 Implementation of an Exemplary Latent Space Representation Learning System 2.1 Loop and One-Shot 2.2 Categories of Sound Content 2.3 Reading / Decoding Logic 2.4 Conversion Logic 2.4.1 Preprocessing 2.4.2 Duration Normalization 2.4.3 Time-Frequency Representation 2.4.4 Loop Signal Extension 2.5 Learning Logic 2.5.1 Exemplary Artificial Neural Networks 2.5.2 Exemplary Artificial Neural Network Architectures 3.0 Exemplary Latent Space Representation Learning Process 4.0 Exemplary Application of Sound Embedding 4.1 Similar Sound Search 4.2 Sound as Search Input 4.3 Detection of nearly overlapping sounds 5.0 Implementation Mechanism - Hardware Overview 6.0 Implementation Mechanism - Software Overview 7.0 Other aspects of disclosure

[0013] 1.0 Summary According to some embodiments of the present invention, pitch, timbre, decay, reverberation, and other psychoacoustics Features indicating changes in target attributes are extracted from the digital audio signal, and from the extracted features, digital Artificial neural network model for generating contextual latent space representations of audio signals A method and system for learning such contextual latent space representations are provided. Training such a model to generate requires combining existing learning systems and methods. They're doing it sloppily.

[0014] According to some embodiments of the present invention, a consistent latent spatial representation of a digital audio signal is produced. A method and system for training an artificial neural network model to achieve this goal have been proposed. The generated latent spatial representation determines the psychoacoustic similarity between digital audio signals. For the purpose of being comparable, and such consistently comparable learned latent space representations Training such models for generation complicates existing learning methods and systems. It's changing.

[0015] According to some embodiments of the present invention, features are extracted from a digital audio signal, and the extracted features are Based on these characteristics, it is responsible for selecting the prominent attributes of the signals that represent the psychoacoustic differences between them. An artificial neural network model for generating latent spatial representations of digital audio signals. Methods and systems for learning are provided, and such selections complicate existing methods. It is.

[0016] According to some embodiments of the present invention, a vector referred to herein as “sound embedding” To learn the latent spatial representation of a digital audio signal in the form of a set of numbers or numerical codes, Machine learning techniques are used. Sound embedding can be learned unsupervised, thereby achieving the goal. Variables (labels) are used to manually generate or computer-generated training examples. Reduce or eliminate the need for generated labels. Furthermore, sound embedding allows humans to The manual methods implemented may not even be recognized, or may be difficult for humans to fully understand. Since the attributes of the audio signal can be encoded, the learned sound embedding is less manual. Compared to other methods, it can be of higher quality (i.e., it can better express the differences in sound). High-quality sound embedding means a better representation of the audio signal, which in turn leads to greater consistency between signals. This replaces similar measurements with higher quality and more robust voice similarity searches. This leads to recommendations.

[0017] 2.0 Latent Space Representation Learning System Figure 1 shows the generation of a latent spatial representation of a digital audio signal according to several embodiments of the present invention. This is a schematic diagram of a system for training an artificial neural network model. System 100 is a digital audio signal source (for example, data containing digital audio data). It reads a file and trains an artificial neural network based on the digital audio signal. Programmed or configured to refine and provide a latent spatial representation of a digital audio signal. We will train a model that can generate sets of numerical codes (sound embeddings).

[0018] A learning model for digital audio signals provides a latent spatial representation of digital audio signals. The resulting set of numerical codes is hereafter referred to as "sound embedding" in digital audio signals. Sometimes, sound embedding is done by vectors containing numerical codes in computer storage media. Each numerical code can be represented as a floating-point value on a computer storage medium. It is a real number.

[0019] For latent space representation, distance criteria (e.g., cosine similarity, Euclidean distance, etc.) Therefore, the sound embedding of digital audio signals that are closer to each other is further determined according to the distance criterion. Rather than embedding sound from a distant digital audio signal, it is perceived by the human ear during playback. It reflects digital audio signals that sound more similar to other applications. Furthermore, sound embedding identifies similar-sounding digital audio signals and uses a reference digital audio signal. To recommend digital audio signals within a corpus of digital audio signals that sound similar to [specific number]. It can be used for this purpose.

[0020] Learning system 100 is otherwise disclosed herein for implementing the method of this disclosure. Thus, since instructions can be loaded in new ordered combinations, the learning system 1 00 is one or more of which are not intended to be general-purpose computers. This can be implemented using physical computing devices. Figure 7, described below, illustrates this. Computers that can be used in learning computing systems to implement disclosure methods This is a schematic diagram of an example of a ting device.

[0021] The components of system 100 are one for performing the method described herein. Or one that executes a stored program instruction stored in multiple computer memory or multiple hardware microprocessors, one or more computing It can be implemented in the device, at least partially, by computer hardware. This is one of the shared computing facilities, such as a cloud computing center. Multiple virtual machine instances may be used.

[0022] The methods described herein are professionally designed for use in a dedicated computer or a general-purpose computer. This is intended to demonstrate the operations performed using grammar. System 100 is This shows only one of many possible arrangements of components configured to execute the law. Other arrangements may contain fewer or different components. The division of labor among the members may vary depending on their assignments.

[0023] One or more components of system 100 are the same computing device It can be implemented in the system. Therefore, the separation of the components of system 100 is one possibility. This is merely an example of a computing architecture. System 100 A component is a set of multiple computing devices in a distributed computing configuration. It can be distributed across multiple platforms. For example, a distributed computing configuration can be a given implementation of a particular setup. Depending on the requirements, it may be used for reasons of computing power or scalability.

[0024] 2.1 Loop and One-Shot System 100 may include read / decode logic 104. Logic 104 is , files or other data containers containing digital audio signal data (for example, computer Professionals can read digital audio signal sources 102, such as database records. Grams or constitutes. In some embodiments of the present invention, source 102 includes digital The audio signal is a human-made music or other auditory performance in the form of a loop or one shot. It is a monthly record. Loops or one-shots are also produced by a computer or machine. It is possible.

[0025] Loops are produced using microphones, turntables, digital samplers, looper pedals, and synths. Sequencers, PLCs, drum machines, tape machines, delay units, computers This includes, but is not limited to, programming using music software, etc. Repeated sections of audio material created using one of several different techniques A loop is often a series of musical measures (1 measure, 2 measures, 4 measures, 8 measures). Rhythmic patterns corresponding to measures, etc., or sequences of notes or chords. Or it encompasses progression. Loops can repeat indefinitely, but they do not create an audible sense of musical continuity. It can be retained.

[0026] One shot is also any of several different techniques, such as those mentioned in the paragraph above. It can be created using one of the following. However, a one-shot is a section of sound material It's not like a loop in that it's not repeated in a single shot. Instead, one shot A note is usually a sound pattern that decays exponentially, and it has a clear beginning at the start of an event that follows. It corresponds to a single audible event having a single note or chord. For example, a one-shot is a single audible event having a single note or chord. This may include sound, or the impact of a percussion instrument (e.g., the impact of a drum kit piece).

[0027] The term "one shot" means a single note, chord, strike, or other sound. However, a single shot is not limited to that; it can also consist of multiple notes, chords, strikes, or sequences. Or it may include other sounds in the progression. Furthermore, some one-shots are exponential. The aforementioned single sound has a clear audible start, followed by a pattern of sound that decays functionally. While it may include audible event patterns, other one-shots may, for example, be reverse samples. There may be cases where this pattern is not followed, such as certain sound effects. In any case, one shot On average, the playback length of a loop is shorter than that of a typical loop.

[0028] The digital audio signal contained in Source 102 may include loops or one-shots, The digital audio signal contained in Source 102 is within the human audible range of approximately 20 to 20,000 Hz. This may include sounds generated or recorded on any computer or machine located within the system. ru.

[0029] In some examples, the digital audio signal source 102 is the code of the digital audio signal source. It belongs to a corpus. A corpus is a collection of hundreds, thousands, millions, or more digital audio recordings. It may include signal sources. For example, a corpus may include, for example, Santa Monica, California. Sounds and other materials available from SPLICE.COM and New York City, New York Examples of what is offered by loud-based music production and collaboration platforms. For example, musical sample libraries, etc., depending on the user, computer, or machine This could be a library of generated or recorded sounds.

[0030] 2.2 Sound Content Categories In some embodiments of the present invention, an artificial neural network processes a given sound. Learn a model for generating sound embeddings for digital audio signals in the content category. It is trained to learn. Multiple different sound content categories, and artificial neural There may be multiple corresponding models trained using a different network. Dell is developing sound embedding for digital audio signals across different sound content categories. It can be used to generate.

[0031] Different sound content categories are different digital sound content categories. This can sometimes reflect general differences in the statistical distributions underlying audio signals. Wound content categories may correspond to classes or types of statistical distributions. Based on a set of training examples, all of which belong to the same sound content category, Train an artificial neural network to learn a model for generating embeddings. By doing so, artificial neural networks have been compared to other training techniques. This allows for better learning of the identification features of digital audio signals in the sound content category. This allows for higher quality sound for digital audio signals in the sound content category. Generate an embed. Ultimately, better recommendations for similar sounds in the sound content category. To achieve this, high-quality sound embedding can be used.

[0032] A set of sound content categories may be predefined, including digital audio signal sources. (For example, 102) is one or more to which they are assigned or to which they belong. It can be tagged with several sound content categories. For example, digital audio signal source. This applies to labels / tags for one or more sound content categories within the database. Therefore, it can be indexed, or otherwise one or more sound contents It can be associated with category labels / tags.

[0033] Sound content category labels / tags are based on the statistical data of the digital audio signal. Based on the distribution, it can be assigned to a digital audio signal source. For example, one-shot data Digital audio signals typically show a clear start at the beginning of the signal and usually have a short duration. Typically, it has a release pattern that decays exponentially. Or, an inverse one-shot The reverse of this pattern. On the other hand, loop digital audio signals typically have a period over the duration of the signal. It exhibits a regular or quasi-periodic pattern.

[0034] According to some embodiments of the present invention, the corpus of digital audio signal sources is less Both are two different sound content categories: loops and single shots. It is divided into. Loops and one-shots are typically fundamentally different over the time of the signal. Based on different statistical distributions, they can be assigned to different sound content categories.

[0035] The Loop Sound Content category and the One-Shot Sound Content category are To form a hierarchy of sound content categories, we have categorized by instrument, instrument type, genre, Further subdivided based on mood, or other sound attributes suitable for the requirements of the specific instrument at hand. It is possible. For example, the hierarchy of sound content categories is such that the top level is sound content Categories may include "loops" and "one shots." Then, those Each of the top-level sound content categories has "drums" at the second level of the hierarchy. The category may include "instruments". Instruments include vocals and other instruments besides drums. This may include: drum loops, dump one-shots, and instrument loops. Four sound content categories are offered: "Roop," "Instrument One-Shot," and "Sound."

[0036] Loops, one-shots, drums, and instruments (e.g., vocals or instruments other than drums) ) divides a corpus of digital audio signal sources into different sound content categories. This reflects the different statistical distributions of digital audio signals over time that can be used to obtain signals. This is a non-limiting example of the attributes. However, other divisions are also possible, and the present invention does not specify any particular It is not limited to the division of the sound content of a corpus of digital audio signal sources. Another possible hierarchical division into the "N's" category could be: One shot o drum o musical instruments ·loop o drum o musical instruments • Reverse sample o drum o musical instruments • Sound effects

[0037] The sound content category includes digital audio signals within the corpus that are expected or discovered. This includes being based on the current specific statistical distribution over time of different classes or types. The sound content category may be selected heuristically or empirically according to the implementation requirements. A computer-based unsupervised clustering algorithm (e.g., exclusive clustering algorithm) is a computer-based unsupervised clustering algorithm. (Gorhythm, overlap algorithm, hierarchical algorithm, or probabilistic clustering algorithm) It can be learned or calculated accordingly.

[0038] For example, a digital audio signal source is extracted or detected from the digital audio signal of the source. Based on the similarity between one or more attributes, the groups are divided into different clusters (classes). Audio signals can be clustered. Such sound attributes can be clustered, for example, over time. Statistical distribution of signal amplitude, zero crossover rate, spectral centroid, spectral density of the signal, signal This may include one or more of the spectral bandwidth or spectral flatness of the signal. When clustering, the digital sounds that are more similar to one or more of these sound attributes are selected. Audio signals are likely to be clustered together within the same cluster, and these Digital audio signals with low similarity to one or more of the sound attributes within the same cluster It is unlikely that they will be clustered together.

[0039] As mentioned above, the corpus of digital audio signal sources is in the sound content category. The grouped, separate artificial neural network models are trained for learning. As a set, it is assigned to the sound content category or sound content Using a set of digital audio signal sources belonging to the "sound content" category, It can be learned by category. The digital audio signal sources in the corpus are single sound codes. It can belong only to the content category, but for example, the sound content category When a duplicate clustering algorithm is used for identification, the digital audio signal source Please note that this may belong to multiple sound content categories.

[0040] Therefore, the digital audio signal source 102 is a specific sound content category An artificial neural network to train a model for generating sound embeddings for Multiple (e.g., hundreds, thousands, millions, or more) digital sounds for training the quat It can be considered one digital audio signal source belonging to a training set of voice signal sources. All include digital audio signal sources belonging to specific sound content categories. The training set may be referred to as the "source" training set in this specification. An artificial neural network is trained based on a specific source training set. The sound content category, as used herein, is the "source" sound content category. They are sometimes called upon. And the results of training using the source training set Source sound content categories learned by artificial neural networks The model of the Rei is sometimes referred to as the “Target” model in this specification. The content category may have different source training sets, and different The target model can learn about different sound content categories.

[0041] 2.3 Read / Decode Logic As described above, system 100 may include read / decode logic 104. The read / decode logic 104 reads the digital audio signal source 102 and digitizes Based on the audio signal source 102, we will generate a "raw" digital audio signal sample. The program or configuration is as follows: Read / decode logic 104 is the conversion logic. 106 is connected to communicate and modifies the raw digital audio sample for further processing. This can be sent to the exchange logic 106.

[0042] The digital audio signal from Source 102 is uncompressed, and compressed in a lossless encoding format. Alternatively, they can be compressed using lossy encoding formats. These are known file extensions. An exemplary possible format for the digital audio signal shown in Source 102 is .AA C, .AIFF, .AU, .DVF, .M4A, .M4P, .MP3, .OGG, .RA This may include, but is not limited to, W, .WAV, and .WMA files.

[0043] System 100 reads source 102 and, if necessary, the digital data contained therein Decode / decompress the audio signal and process the raw digital audio signal based on the digital audio signal. Read / decode logic 1, programmed or configured to generate sample numbers. Includes 04.

[0044] Raw digital audio signal generated by read / decode logic 104 The pull can be obtained in the time domain of a digital audio signal. For example, pulse code modulation. The (PCM) sample is an example of a possible raw digital audio signal sample. The CM signal is raw data containing information to reconstruct the original audible audio signal. This is a sequence of digital audio signal samples. PCM samples are audible at a specific point in time. It can represent the amplitude of an audio signal. PCM samples of a PCM signal can be spaced uniformly in time. Amplitude may be the only information explicitly included in a PCM sample. For example, PCM samples are binary, using several bits that represent the bit depth of the PCM sample. It may include integers or floating-point numbers encoded as numbers.

[0045] If the digital audio signal itself already consists of raw digital audio signal samples, Decoding of the digital audio signal source 102 is required by the read / decode logic 104. Decoding may not be necessary in all cases. For example, digital audio signals do not require decoding. (WAV) It may contain raw digital audio signal samples in the format. On the other hand, the .MP3 format is Decoding of the digital audio signal is necessary to generate a raw digital audio signal sample. It is a compressed, lossy format that may be required in some cases. Therefore, read / decode Code Logic 104 is a versatile digital audio signaling system for various different audio formats. -102 can be programmed or configured to include a reader and decoder. The retrieval / decode logic 104 then determines the digital audio signal of source 102. A suitable leader for generating raw digital audio signal samples based on the format. You can select a decoder and / or a decoder.

[0046] The raw digital audio signal is read from the digital audio signal of source 102 / decoded Generated by code logic 104, or if the digital audio signal is already in its raw state If it is a digital audio signal, the raw digital audio signal is then sent to the conversion logic 106. It can be provided (for example, sent or transmitted).

[0047] 2.4 Conversion Logic According to some embodiments of the present invention, the conversion logic 106 reads / decodes The raw digital audio signal received from JIC 104 is processed into one of the raw digital audio signals. Convert to one or more time-frequency representations. The conversion logic 106 then converts to one or Multiple generated time-frequency representations are provided to the learning logic 108 for further processing. According to some embodiments, the time-frequency representation is normalized, and then the target model Dell is input into an artificial neural network for learning. Some practical applications of this invention According to the implementation, the transformation performed by the transformation logic 106 is the source sound content It varies depending on the pants category.

[0048] 2.4.1 Preprocessing In some embodiments of the present invention, the conversion logic 106 converts the raw digital audio signal Downsampling is performed. This downsampling is done artificially during the training of the target model. This can be done to reduce the dimensionality of the time-frequency representation input to a neural network. For example, digital audio signals range from 44,100 Hz or 48,000 Hz to 22 Hz. It can be downsampled to half or nearly half to 0.50 Hz. Time-frequency representation Reducing the input dimension significantly impacts the accuracy and effectiveness of the target model. Without wasting time on computing resources and training the target model. It can be shortened.

[0049] In addition to downsampling, it has a significant impact on the effectiveness of the trained target model. Without using raw digital audio, in order to accelerate the training of artificial neural networks, The audio signal is derived from stereo samples (for example, by calculating the center channel). It can be converted to a mono audio sample. Alternatively, only the left channel or only the right channel. It may be used.

[0050] Other preprocessing of the raw digital audio signal is performed by the conversion logic 106. For example, a pre-emphasis filter can handle high frequencies with small amplitudes and low frequencies with large amplitudes. It can be applied to balance the frequency spectrum between waves, and the numerical values ​​during the Fourier transform. Either avoid the problem or improve the signal-to-noise ratio (SNR) of the signal.

[0051] Raw data, such as compression, equalization, or other preprocessing appropriate to the requirements of the specific implementation at hand. Other preprocessing steps may be performed on the digital audio signal.

[0052] 2.4.2 Duration Normalization According to some embodiments of the present invention, the conversion logic 106 converts raw digital audio Preprocessed raw digital data received from the signal or read / decode logic 104 Duration normalization is performed on the audio signal. Raw digital audio signal or preprocessed The raw, unprocessed digital audio signal is referred to below as the "target" signal. The target signal is... It can be normalized to a fixed duration. The predetermined duration is the source sound content This may be based on category. Different sound content categories have different predetermined durations. It may take time. However, different sound content categories may have the same predetermined duration. It is possible to have time.

[0053] Each such target signal has the same predetermined duration for the source sound content category. When each target signal is normalized in the source training set, during target model training... A time-frequency table of the same dimension from the target signal for input to an artificial neural network. It becomes easier to extract the actual data. In other words, the goal for the source training set. Each time-frequency representation generated for each signal is used to source the target signal of the training set. The result has the same dimensions as the result normalized to the same predetermined duration in the sound content category. To obtain. When used herein, a reference to the “same” duration means a given time resolution (e.g., For example, if two things have the same duration (in milliseconds), it means they have the same duration. For example, two signals, both with a length of 500 milliseconds, even if they have nanosecond time resolution. Therefore, even if the length of time differs, they have the same duration.

[0054] If the target signal has a duration longer than a predetermined duration, the conversion logic 106 Duration normalization of the target signal to a predetermined duration is performed by cutting the sample from the end of the target signal. By discarding samples, or by trimming samples from the beginning of the target signal... This can be achieved. If the duration of the target signal is shorter than the predetermined duration, then The dummy digital audio signal continues until the duration of the target signal equals the predetermined duration. Samples (e.g., zero-value samples) can be added to the target signal.

[0055] As a result of performing duration normalization on the target signal by the conversion logic 106, the target The signal is "duration normalized" to have a predetermined duration for the source sound content category. The signal is normalized as a given duration. As an example of what could be changed, loop sound content category and one-shot sound Consider the WOOD content category.

[0056] The source sound content category is the one-shot sound content category. In that case, the first part of the target signal where a clear start of one shot exists, duration It can be selected as a normalized signal. This is, for example, in the case of a drum one-shot, the target signal This can correspond to the first 1 second or so of the drum one-shot target signal. The duration of a single note can be, for example, slightly less than one second. In the case of a shot target signal, the duration-normalized signal is slightly less than the drum one-shot target signal. It can be quite long (for example, within the first few seconds), and one shot of an instrument or vocal. This reflects the typically relatively long duration of the start before release in the target signal. The predetermined duration of a one-shot target signal for an instrument or vocal is approximately 1 second and 3 seconds, etc. It can be in between. More generally, a one-shot target that is interpreted as a duration-normalized signal. The signal portion includes, but is not limited to, the target signal portion, which includes the start of a one-shot sound. It can be selected based on various attributes of the target signal.

[0057] On the other hand, the source sound content category is the loop sound content category. In this case, the predetermined duration is then repeated for each loop signal of the source training set. Long enough to capture at least one cycle of a periodic or quasi-periodic pattern. There are cases where this is not the case. For example, in the case of a musical loop that spans 2, 4, or 8 measures, The duration can be longer than 3 seconds and shorter than 5 seconds. For example, the source sound container If the contents category is the loop sound content category, the source sound content The predetermined duration of the category is the repeating period of the loop signal of the source training set. Selected based on the average or average duration of one cycle of a specific or quasi-periodic pattern. It is possible.

[0058] 2.4.3 Time-Frequency Representation According to some embodiments of the present invention, the conversion logic 106 converts the duration normalized signal By receiving and calculating the time-frequency transformation of the duration-normalized signal, duration normalization is performed. The program or configuration is used to generate a time-frequency representation of a signal. As the frequency of the duration-normalized signal changes over time, This is a representation of the spectrum. According to some embodiments of the present invention, the time-frequency conversion is z This is a complex number transformation consisting of complex numbers of the form a+ix, where "a" represents the real part of the number and "x" represents the real part of the number. The symbol " represents the imaginary part of a number.

[0059] According to some embodiments of the present invention, the time-frequency representation is such that the y-axis represents frequency and the x-axis represents frequency. Duration normalization is performed by calculating the absolute value of the complex spectrogram that represents the time frame. This is the amplitude spectrogram calculated based on the complex spectrogram of the signal. The value of the frequency slot represents a specific energy level of the duration-normalized signal. The dude matrix is ​​converted to a floating-point value in the computer storage medium by the conversion logic 106. As a matrix of real numbers that can be represented by, the conversion logic 106 in the computer storage medium It can be expressed.

[0060] The conversion logic 106 converts the duration normalized signal by using time-frequency conversion. It can be programmed or configured to generate a time-frequency representation. The time frequencies that can be used... Some examples of numerical transformations include the Short-Time Fast Fourier Transform (STFT), the Constant Q Transform, and the Wave Transform. This includes, but is not limited to, lets, filter banks, etc.

[0061] The conversion logic 106 optionally provides a nonlinear complement to the time-frequency representation of the duration-normalized signal. It can be applied to the interval. Applying nonlinear interpolation to the time-frequency representation allows the human auditory system to... By giving greater weight and improved resolution to frequency bands with high resolution, This is intended to improve the ability to learn the discriminative attributes of those frequencies. Ideally Under laboratory conditions, the audible range can be as wide as 12 Hz to 28 kHz, but for humans The auditory system generally has an audible range of 20 Hz to 20 kHz. Therefore, the human tendency From a practical auditory standpoint, learning the discriminative attributes of speech is possible within the human audible frequency range. They are related.

[0062] If applicable, the nonlinear interpolation applied by the transformation logic 106 is psychoacoustic. Many different supplements based on the context of the human auditory scale, sometimes called the "scale." One of the intermediate scales can be used. Examples of such psychoacoustic scales are the Mel scale and the Berkske scale. Includes a scale and / or equivalent rectangular bandwidth (ERB) scale. Therefore, the conversion logic CK 106 applies nonlinear interpolation to these psychoacoustic scales in time-frequency representation. You may use one or more of these.

[0063] According to some embodiments of the present invention, a time-frequency variable applied to a duration-normalized signal is used. The conversion is a complex short-time Fourier transform (STFT), and the conversion logic 106 is either The choice may vary depending on the source sound content category, or both. The selected window size (e.g., 1024 samples) and the selected hop size ( For example, the duration-normalized signal is applied according to 128 samples, 256 samples, etc. The complex STFT is calculated. For example, the time frame for the complex STFT transformation is the selected window. There may be overlaps between the selected hop size and the selected hop size. The sample size determines how many signal samples are included in the current timeframe of the duration-normalized signal. By advancing the starting point, the starting point of the next time frame of the duration-normalized signal is set, and the complex STFT transform is received. It indicates how much time can be used. For example, the time frame can be 1024 sounds according to the selected window size. If you have a voice signal sample and the selected hop size is 256 signal samples, then Then, the starting point of the current time frame of the duration-normalized signal is reached after 256 audio signal samples. 768 voice messages in the duration normalized signal between the previous current time frame and the new current time frame This results in a 75% overlap in sample numbers.

[0064] In some embodiments of the present invention, the hop size of the complex STFT transform is the source sound. Selected based on content category. In particular, the smaller the hop size, the more likely it is to be time-framed. The redundancy incorporated in between increases. The duration-normalized signal of the category is relatively fast transient. A predetermined duration of a duration-normalized signal having an event or category is, for example, For relatively short source sound content categories such as Ram One Shot, the target The model better captures the transitions in the spectral time patterns of the duration-normalized signals of categories. To make them more distinguishable, relatively smaller hop sizes may be used.

[0065] The disadvantage of using a smaller hop size is that it requires more time frames to process, and given a number The advantage is that the amount of temporal context captured by a continuous time frame is reduced. It does not have fast transient events, or for example, a one-shot source of an instrument or vocals. Duration-normalized signals with longer predetermined durations, such as sound content categories. For the included source sound content category, identify the target model to be generated. Without paying a significant price to its capabilities, the number of timeframes processed by the conversion logic 108 can be increased. In order to reduce it, and to take a larger amount of time context for each given number of consecutive time frames To achieve this, a relatively large hop size may be used.

[0066] According to some embodiments of the present invention, the magnitude STFT is a duration-normalized signal. It is calculated from the complex STFT of the magnitude. Then the magnitude spectrum is calculated from the magnitude Generated from the STFT. Nonlinear interpolation is then applied to the magnitude spectrogram. Applicable. According to some embodiments of the present invention, the power spectrum S of the complex STFT The TFT is calculated by the conversion logic 106 instead of the magnitude STFT, and then Then, nonlinear interpolation is applied to the power spectrum STFT.

[0067] According to some embodiments of the present invention, a Hamming window function or other suitable window function such as The window function anticipates the complex STFT transform on the time frame and applies to each time frame of the duration-normalized signal. It applies.

[0068] In some embodiments of the present invention, the conversion logic 108 is a time-frequency table of N multiplied by M. Generate a current (e.g., magnitude spectrum), where N is the number of different frequency bins. Yes, M is the number of timeframes, which is the window size of the duration-normalized signal, hop It depends on the size and a predetermined duration. In some embodiments of the present invention, N is the duration These are 128 values ​​for each time frame of the time-normalized signal. However, a specific number N of values ​​is not required. stomach.

[0069] As mentioned above, each duration-normalized signal in the source sound content category is the same It may have a fixed duration. This duration is defined herein as the source content This is sometimes called the "target duration" in a category. For example, "drum one shot" Each duration-normalized signal in the sound content category has a target duration of 0.78 seconds. There is a possibility that the duration normalization of each "instrument one-shot" sound content category The signal may have a target duration of 1.48 seconds, creating a "looping" sound content. Each duration-normalized signal in the two categories may have a target duration of 3.75 seconds. However, a specific target duration is not required. And, as mentioned above, different sounds Content categories may have different target durations.

[0070] As mentioned above, the time dimension M of the time-frequency representation is the target duration of the source sound category. In between, the window size and selected for the source sound content category This may vary depending on the hop size used. For example, if the target duration is 0.743 seconds, The selected hop size is the duration of the drum one-shot category for 128-hour domain samples. The time-normalized signals are each multiplied by 128, for a total of 16,384 values ​​for each time-frequency representation. It is possible that a time-frequency representation with 128 dimensions was generated. The target duration is 1. In 486 seconds, the selected hop size is 256 hours domain samples for instrument one-shot categorization. Gori's duration-normalized signals also have a time dimension of 128 x 128, each with a value of 128 x 128. A frequency representation may have been generated. The target duration for the instrument one-shot category is drum. Even if the target duration is twice that of the one-shot category, it will still generate in these two categories. The dimensions of the time-frequency representation can be the same in M ​​dimensions. This is because one-shot instruments Also, assuming the same selected size for both categories, Drum One Shot This is because it is twice the selected hop size for the category. Duration of the drum loop category is 3.75 seconds, with a selected hop size of 256. The internormalized signals each have 40,960 values ​​for each time-frequency representation, with a total dimension of 128 It is possible that a time-frequency representation with 320 dimensions was generated.

[0071] The time-frequency representation of the duration-normalized signal is used for training the target model. Before being passed to 108, normalization may be performed on the time-frequency representation values. For example, the time-frequency representation value is used in the learning algorithm of an artificial neural network (for example) For example, to help the gradient descent converge, include 0 and 1, or between -1 and 1. The values ​​can be normalized to minimum or maximum values. For example, mean normalization, Z-score normalization, or Other normalization methods are possible, such as scaling to a unit length.

[0072] 2.4.4 Loop Signal Extension According to some embodiments of the present invention, multiple time-frequency representations are converted using the conversion logic 106 Generated from a single duration-normalized signal of the loop source sound content category This is done. Multiple time frequencies of duration normalized signals in the loop source sound content category. This extension to wavenumber representation can be done during training to learn the target model. In particular, this extension is based on a training set of loop digital audio signal sources. When training the target model, a transformation log is used to provide a better temporal context. This can be performed by Jick 106. The repeating pattern of the duration-normalized loop signal is: In cases where there are different durations and different periodic or quasi-periodic repeating patterns Yes. Furthermore, the repeating pattern is a duration-normalized loop of the source training set. In some cases, alignment may not occur across the signal. For example, in a source training set... Some duration-normalized loop signals are at the beginning of the loop signal's repeating pattern. It may start, and other duration-normalized loop signals in the source training set will The loop signal may start in the middle of a repeating pattern, and the source training set Furthermore, other duration-normalized signals start at the end of the repeating pattern of the loop signal. There is.

[0073] This explains the differences in the duration-normalized loop signals of the source training set. To achieve this, multiple time-frequency representations are randomly selected from multiple options for each duration-normalized signal. It can be generated from "slices". Each such selected slice is a source sound. The same predetermined duration as the content category, and therefore the same duration as the duration-normalized loop signal. It may have a duration. For each selected slice, the time-frequency representation of the slice is the duration. Except for using a slice signal instead of a normalized signal, the time frequency of the duration normalized signal The wavenumber representation can be generated using the techniques described above. In this way, multiple Time-frequency representations can be generated for multiple slices of a duration-normalized signal.

[0074] When a slice of a loop duration normalized signal is randomly selected, the start time is the same as the loop duration normalized signal. If within the duration range, the start time of the slice may be randomly selected. Next, it starts at a randomly selected start time, and then the selected slice signal holds Slide the end of the duration-normalized signal so that it has the same duration as the duration-normalized signal. Interpret as a time-shifted version of a duration-normalized signal that wraps around the start of the signal. It is possible. Alternatively, the selected slice signal is the beginning of the duration-normalized signal. It can wrap around to the start of the signal. In this way, multiple duration-normalized signals The slice signal is obtained by randomly selecting different start times during the duration-normalized signal. It can be obtained from, and each such slice signal is in the source sound content category It has a predetermined duration.

[0075] In some embodiments of the present invention, slices are randomly selected from a duration-normalized signal. Instead, slices are randomly selected from the time-frequency representation of the entire duration-normalized signal. In these embodiments, a time-frequency representation such as a magnitude spectrogram is used. To select a slice, the time frame of the magnitude spectrogram is randomly selected. This slice then starts at a randomly selected time frame, and the magnitude The end or beginning of the pectrogram is wrapped around a randomly selected time frame. The time shift of the entire magnitude spectrogram of the duration-normalized signal that returns to the end. It may be a version. In this way, the time-frequency representation of the entire duration-normalized signal. Multiple randomly selected slices were randomly chosen to have different time-frequency representations. It can be obtained from the start time slot.

[0076] In both cases, the duration-normalized loop signal can be extended to multiple time-frequency representations. In some embodiments, multiple time-frequency representations are such that two of the time-frequency representations have They are separate so that the time-normalized signals do not start at the same point.

[0077] Extending the duration-normalized loop signal of the source training set to multiple time-frequency representations. By doing so, more digital audio signals can be included in the source training set. Training examples to be input into an artificial neural network without needing to obtain the source The diversity can be expanded. For example, a source of 100,000 loop digital audio signals. The training set is thus used to train the artificial neural network during training. It can be extended to millions of different training examples input into the system.

[0078] For example, an artificial neural network based on a selected batch size N (e.g., 32) If batch gradient descent is used during work training, then a number of time frequencies N Numerical representations are generated from each duration-normalized loop signal for the source training set. Before backpropagation, the data can pass through the artificial neural network in a forward path. By doing so, the error in the calculation of the gradient is the N-time frequency representation within the batch. This represents the average of the errors.

[0079] As a result of the processing performed by the transformation logic 106, one of the duration-normalized signals Alternatively, multiple time-frequency representations are provided to the learning logic 108 by the conversion logic 106. It is possible.

[0080] The above is performed by the read / decode logic 104 and the conversion logic 106. Note that the operations can be rearranged in some examples. For example, Source 102 The digital audio signal or raw audio signal is first converted to the conversion logic 106. Therefore, the duration normalized signal can be used for duration normalization, and then read / decoded. The code logic 104 determines which conversion logic 106 converts one or more time-frequency representations. Based on what is generated, the duration-normalized signal is a preprocessed version of the duration-normalized signal. It can be pre-treated.

[0081] 2.5 Learning Logic The read / decode logic 104 and the conversion logic 106 set the time-frequency representation. To generate the source, apply to each digital audio signal source in the source training set. This set of time-frequency representations generated for the source training set is - Learning logic to train an artificial neural network to learn a get model It can be used by CK 108.

[0082] Before learning logic 108 begins training the artificial neural network, source training It should be noted that it is not necessary to generate the entire set of time-frequency representations from the initial set. Instead, artificial neural networks use some of the time-frequency representations within a set. Based on this, training of the artificial neural network begins, and the entire set is generated in parallel. It can be trained using the Iplene method.

[0083] According to some embodiments, a target model is trained with given training examples. When training an artificial neural network in this way, the artificial neural network will Attempts are made to capture the time-frequency representation, and then it is converted into a reduced latent space representation. The program attempts to convert the given latent space representation back to a time-frequency representation. It is composed of artificial neural networks. During training, this means that the artificial neural network is composed of artificial neural The linear network can reproduce the time-frequency representation within the error range according to the loss function. This process can be repeated until the system learns to generate a latent spatial representation.

[0084] 2.5.1 Exemplary Artificial Neural Networks Figure 2 shows the embedding of the input time-frequency representation 214 of sound in several embodiments. A learning logic for learning 110 components of an artificial neural network This is a schematic diagram of Work 216. As shown in the diagram, the artificial neural network 216 consists of three parts. The components are the encoder 218, the latent space representation 220, and the decoder 222 It includes.

[0085] According to some embodiments, the artificial neural network 216 is (encoder 2 (by 18) Learn via backpropagation to generate latent space representation 220 We will try to learn and the time-frequency representation 224 output from the latent space representation 220 (deco (By -da222) learning through backpropagation to generate The program or configuration is set to attempt the same thing, and in the latent space representation 220, when output The inter-frequency representation 224 is a closely related (irreversible) representation of the input time-frequency representation 214. As part of training the target model, the estimated / reconstructed time-frequency representation 224 And the discrepancy between that and the input spectrogram 214 (for example, the selection of mean squared error, etc.) (quantified by the loss function) adjusts the weights across the entire network 216, In the output of twerk 224, the input spectralgram 214 and the estimated / reconstructed It can be used repeatedly to reduce errors between the given time-frequency representation and the given time-frequency representation.

[0086] The latent space representation 220 is input as a lower-dimensional embedding containing fewer numerical values. This represents a time-frequency representation 214, but there is no significant recognition of the input time-frequency representation 214. It still retains other information. For example, if the input time-frequency representation 214 is, for example, 12 The latent space representation 220 has a dimension of 8 x 128, and the latent space representation 220 has a dimension of 256. In this case, the dimensionality reduction can be 64:1. A specific dimensionality reduction ratio is not required, and 64:1 is one option. This is merely an example.

[0087] In some embodiments of the present invention, the artificial neural network 216 is particularly specific. It can be trained for the source content category. That is, once trained, Artificial neural network based on source training set for source content category The target model resulting from training network 216 is source content category It can be used to generate sound embeddings solely for the audio signal, source code Not belonging to the Contents category, or assigned to the Source Contents category It cannot be used to generate meaningful sound embeddings for non-existent audio signals. There is a possibility.

[0088] The estimated / reconstructed time-frequency representation 224 is given by the input time-frequency representation 214. It may be irreversible. That is, the estimated / reconstructed time-frequency representation 224 is the input It is not necessary that the time-frequency representation 224 is identical (reversible).

[0089] The advantage of the artificial neural network 216 is that it can be trained with a source training set. The advantage is that it is not necessary to pre-label the examples. Instead, artificial neural networks The network 216 is source-trained in the form of a latent space representation (e.g., 220). It can learn lower-dimensional representations of training examples within the set, and lower-dimensional representations Currently, meaningful identification of the fundamental statistical distribution of source sound content categories. Retain information.

[0090] The artificial neural network 216 may have several hyperparameters. One of the per parameters is learning the sound embedding, also known as the bottleneck layer 220. You can specify the dimensions of the intermediate layer responsible for doing what. Other hyperparameters are other possible Among the hyperparameters, the intermediate ones in the encoder 218 and decoder 222 (for example) This includes the number of hidden layers and the loss function used during training (e.g., mean squared error). obtain.

[0091] 2.5.2 Exemplary Artificial Neural Network Architectures Figure 3 shows the capabilities of an artificial neural network 216 according to several embodiments of the present invention. This is a schematic diagram of a possible architecture. The architecture consists of an input layer 326 and an encoder middle layer. It includes an intermediate layer 332, a bottleneck layer 330, a decoder intermediate layer 334, and an output layer 328. The input layer 326 is used for compression to the bottleneck layer 330 by the encoder intermediate layer 332. Therefore, it can include the normalized value of the time-frequency representation 214. Similarly, the output layer 328, The estimated / reconstructed data generated from the bottleneck layer 330 by the decoder intermediate layer 334 The normalized value of the time-frequency representation 224 may be included. The bottleneck layer 330 is encoding This may be considered the output layer of decoder 218 and the input layer of decoder 222.

[0092] During training via backpropagation, the time-frequency representation 21 of the input layer 326 4 is the encoder 218 to generate the latent space representation 220 in the bottleneck layer 330. It passes through the hidden layer 332. Encoder 218 is a fully connected artificial neural network. It may be a workpiece. Similarly, decoder 222 is a fully connected artificial neural network. It can be twerking.

[0093] The latent spatial representation 220 of the bottleneck layer 330 is estimated / reconstructed in the output layer 328 when The signal passes through the intermediate layer 334 of the decoder 222 to generate the inter-frequency representation 224. Therefore, the dimensions of the input layer 326 and the output layer 328 may be the same. Bottleneck layer 330 The dimension of the encoder intermediate layer 332 is less than the dimension of the input layer 326 and the output layer 328. The number may be one or more. If there are multiple, then the bottleneck layer 330 The intermediate layer closer to the encoder 218 has a bottleneck layer 330 with a time-frequency representation 214. We force the model to learn to generate a latent spatial representation 220 of the summary, irreversible, and dimensionality reduction. To that end, it may optionally have fewer units than the next nearest intermediate layer 332.

[0094] The number of decoder intermediate layers 334 may also be one or more, and encoder intermediate layers It may have the same number of layers as layer 332 or a different number of layers. Next, the intermediate layer of the decoder 222, which is closer to the output layer 328, is supplied to the decoder 222. The resulting estimated / reconstructed time-frequency representation 224 is close to the time-frequency representation 214. To force learning to generate the estimated / reconstructed time-frequency representation 224 of the output layer 328 from the latent space representation 220 of the neck layer 330 The next closest intermediate layer 334 may have more. All intermediate layers 332 and 334 of the encoder 218 and decoder 222 respectively may optionally have a higher dimension than the bottleneck layer 330. The input layer 326 and output layer 328 may optionally have a higher dimension than the intermediate layers 332 and 334. In some embodiments of the present invention, the encoder intermediate layer 332 includes a convolutional layer, an activation layer, and a pooling layer, and the decoder intermediate layer 334 includes a convolutional layer, an activation layer, and an upsampling layer. By using convolutional layers in the encoder intermediate layer 332 and the decoder intermediate layer 33

[0095] 4, the artificial neural network 216 can be trained without the need to modify (and consequently lose spatial information) the time-frequency representation 214 that might be required by a feed-forward neural network implementation using dense layers. For example, flattening a two-dimensional time-frequency representation into a one-dimensional vector can be avoided by using convolutional layers. Also, using convolutional layers allows the learning process to capture the important two-dimensional spectral time patterns presented in the magnitude STFT. In some embodiments of the present invention, the encoder intermediate layer 332 or the decoder intermediate layer 33 4 includes one or more recurrent layers in addition to or instead of one or more convolutional layers. For example, the encoder intermediate layer 332 or the decoder intermediate layer 334 can capture the important two-dimensional spectral time patterns presented in the magnitude STFT without the need to modify (and consequently lose spatial information) the time-frequency representation 214 that might be required by a feed-forward neural network implementation using dense layers. For example, flattening a two-dimensional time-frequency representation into a one-dimensional vector can be avoided by using convolutional layers. Also, using convolutional layers allows the learning process to capture the important two-dimensional spectral time patterns presented in the magnitude STFT. In some embodiments of the present invention, the encoder intermediate layer 332 or the decoder intermediate layer 33 4 includes one or more recurrent layers in addition to or instead of one or more convolutional layers. For example, the encoder intermediate layer 332 or the decoder

[0096] In some embodiments of the present invention, the encoder intermediate layer 332 or the decoder intermediate layer 33 4 includes one or more recurrent layers in addition to or instead of one or more convolutional layers. For example, the encoder intermediate layer 332 or the decoder intermediate layer 334 includes one or more recurrent layers in addition to or instead of one or more convolutional layers. For example, the encoder intermediate layer 332 or the decoder The intermediate layer 334 may include one or more long short-term memory (LSTM) layers or one or more gated recurrent unit (GRU) layers, in addition to or instead of one or more convolutional layers.

[0097] 3.0 Exemplary Latent Space Representation Learning Process FIG. 4 is a flowchart of an exemplary process 436 executed by the system 100 for learning the latent space representation of a digital audio signal according to some embodiments. FIG. 4 is intended to disclose an algorithm or functional description that can be used as a basis for writing a computer program to implement the functions that operate one of one or more computing devices in a new way disclosed herein. Further, FIG. 4 provides the same level of detail as is typically used among those skilled in the art to convey the plan, design, and algorithms for other computer programs of similar levels of complexity, and to convey such algorithms.

[0098] The process 436 may begin at step 438. At step 438, the system 100 obtains a source training set. The source training set may be obtained by tagging or indexing a digital audio signal source including digital audio signals with a particular source sound content category. Alternatively, the source training set identifies a subset of similar digital audio signals that must belong together to a particular source sound content category by applying a clustering algorithm to the digital audio signals. It may also be obtained by applying it to the corpus. In either case, source training The digital audio signals of a setting set all have the same general statistical distribution in the time domain. In terms of possessing something, they may all be similar to one another. The statistics considered here will not change. It is possible that, for example, the time length of a digital audio signal, or for example, the spectral centroid, One of the temporal characteristics of a digital audio signal, such as spectral flatness or spectral bandwidth. This may include general similarities in the attributes of multiple audible sounds. For example, source training set All digital audio signals may have one-shot patterns, and all drum one-shots It can have a set pattern, all drum one-shot patterns, all vocal patterns It can have one-shot patterns, all instruments can have one-shot patterns, all loops It may have patterns, all may have vocal loop patterns, all may have instrument loop patterns It may have all of the following characteristics, and all of them may have the reverse one-shot pattern.

[0099] Step 440 includes read / decode logic 104 and conversion logic 106. System 100 selects one or more digital audio signals from each source training set. This generates a time-frequency representation for each digital audio signal. The process involves generating a target signal from an audio signal and generating a duration-normalized signal from the target signal. This includes generating one or more time-frequency representations from a duration-normalized signal. It is visible.

[0100] In step 442, system 100, which includes learning logic 108, is source training To learn to generate a latent spatial representation of the time-frequency representation generated in a set, Train an artificial neural network via backpropagation. The artificial neural network can be trained with batch gradient descent, stochastic gradient descent, or mini-batch gradient descent. In some embodiments, the artificial neural network is trained with mini-batch gradient descent.

[0101] In step 444, when the artificial neural network is trained based on the source training set, the target model is stored in the model database 110 by the system 100. The target model may include the learned parameters of the artificial neural network as a result of training 442. For example, the target model may include the learned parameters of an encoder (e.g., 218) of the artificial neural network.

[0102] In step 446, the target model is assigned to a source content category or used to generate an audio embedding for a digital audio signal belonging to the source content category. To do this for a particular reference digital audio signal, the system 100 including the read / decode logic 104 and the conversion logic 106 generates a time-frequency representation of the reference digital audio signal. To do this, for example, the read / decode logic 104 can be used to generate a target signal from the reference digital audio signal, and the conversion logic 106 can be used to generate a duration-normalized signal from the target signal. The conversion logic 106 can be used to generate a time-frequency representation of the duration-normalized signal. If the reference digital audio signal is a loop signal, multiple time-frequency representations Unlike during training, when the signal can be generated from a signal loop signal, the reference digital audio signal A single time-frequency representation may be generated. Alternatively, the read / decode logic 104 It can be used to generate a duration-normalized signal from a reference digital audio signal, and converts the logic The 106 can be used to generate a target signal from a duration-normalized signal. The Jick 106 can be used to generate a time-frequency representation of a target signal.

[0103] In any case, the time-frequency representation of the reference digital audio signal is used during training. The example can be normalized as performed by the transformation logic 106.

[0104] The normalized time-frequency representation of the reference digital audio signal is output to the reference signal as follows: To generate the latent spatial representation (i.e., sound embedding), the target model is used in the forward pass. It can be input into an artificial neural network composed of the parameters of the `L`. For example, artificial The encoder 218 of the neural network 216 uses the learned pattern of the target model. The normalized time-frequency representation, which can be composed of lamellae, is specific to the bottleneck layer 330. To generate an audio embedding that captures the latent spatial representation for a reference digital audio signal, The input layer is routed in a forward path that passes only through encoder 218 (i.e., bypasses decoder 222). It can be processed from 326.

[0105] This generates sound embedding for those signals, source sound code Assigned to the Contents category, or belonging to the Source Sound Contents category. This can be repeated for other reference digital audio signals. Sound embedding is The similarity between two digital audio signals in the generated source sound content category is appropriate. It can be calculated by measuring the distance between sound embeddings according to a precise distance standard. They possess similar characteristics. The closer they are, the more similar they become in a psychoacoustic sense. The greater the distance, the less similar they become in a psychoacoustic sense. For example, sound embedding The sounds can be vectors of numbers, and for example, the closer the cosine distance, the closer the sounds are. Appropriate distance criteria, such as the cosine distance between two vectors, that make voice signals sound more similar. Therefore, it can be calculated similarly.

[0106] Process 436 is repeated for other sound content categories, targeting The model is trained to fill in the sound for audio signals in these other sound content categories. It can generate a mix.

[0107] 4.0 Exemplary Application Sound embedding generated using the technology disclosed herein is useful in many applications. It can be used with applications. No specific application is required. Here We will describe two specific exemplary applications.

[0108] 4.1 Similar Sound Search According to some embodiments of the present invention, a standard digital for a specific sound content category One or more specific sound content categories that sound similar to a talus voice signal. The digital audio signal is identified and presented to the user via a graphical user interface. It is shown that digital audio signals that sound similar are one or more computing Identified by the device's server computing system ("Server System") Furthermore, a graphical user interface allows the user's personal computing It may be presented to the client device. Server system and client The network is connected by one or more data communication networks, such as the Internet. It can be connected.

[0109] The distinction between client and server is a useful framework for describing similar sound embodiments. It provides a distinction, but it is not a necessary distinction, and the similar sounds are in other computer forms. It can also be implemented in a client-server system. The functions described below, which are performed by the M, are performed by a single computing device Some of the functions that can be embodied in S and are described as being executed by the server system are, Some implementations run on the client device and become larger, or on the client device. Therefore, some of the functions described as being executed are actually executed on the server system in some implementations. It is possible.

[0110] Figure 5 shows a possible combination of similar-sound applications according to several embodiments of the present invention. This is a mockup of a computer graphical user interface. In particular, the mockup The application provides a graphical user interface for end users' computing devices. Table 548 may be presented on the face. End user computing A computer is a desktop computer or a laptop computer. This may be a personal computing device or a portable electronic device such as a mobile phone. The end user's computing device also has a dedicated music mixing compilation. A routing device, or including a video display, or a video display It may also be other dedicated computing equipment electrically coupled to it. Certain types End-user computing devices are not required. In this respect, Figure 5 shows The quad is provided for the purpose of illustrating possible implementations and graphical user interfaces. - The type of end-user computing device on which the face is presented (for example, In accordance with the requirements of the current specific implementation, including desktop computers versus mobile computers. This network covers all possible graphical user interface implementations, which may vary. Please note that this is not intended to be a generalization.

[0111] Table 548 shows one or more digital audio sources from a corpus of digital audio signal sources. Includes a list of voice signal sources. The list responds to various different end-user inputs. This may be presented in Bull 548. For example, the list may be associated with a keyword, or Searching for digital audio signal sources tagged or indexed by keywords To search for paths, users can access a graphical user interface to end users. Therefore, in response to the entered keyword-based search query, it can be presented as a search result. However, the list is based on other reasons (a list of popular sounds resulting from browsing a corpus). Lists, lists of favorite sounds, lists of recently used, accessed, or played sounds It may be presented for reasons such as strikes, and no specific reason is required.

[0112] Table 548 contains a row for each digital audio signal source presented. In this example, Lines 550, 552, 554, and 546 are presented, but there are more in other situations. It will be even less. Each line corresponds to a digital audio signal, and the playback time of the signal, the signal The waveform and signal source file name are shown, and the selected digital audio signal sounds similar to it. User interface controls for indicating corpus sounds (e.g., selectable) Provides buttons or icons. Each row contains more or less information. It will be. For example, a row is one or more sound content cables of a digital audio signal. This includes categories such as playing audio signals and registering audio signal sources as favorites. Other users may be affected by operations such as downloading audio signal sources. It will provide control over the center face.

[0113] Figure 6 shows several embodiments of the graphical interface shown in Figure 5, which allows the end user to interact with the graphical interface shown in Figure 5. In the state of the user interface, the "type" of user interface control 558 The graphical user interface responds to selecting or activating "Show Similar" This is a mockup of the changes in the state of the system. In response to the end user's choices, several additional options are available. Rows 658, 660, 662, 664, and 666 are displayed in Table 548. The row in parentheses is a digital sound that sounds similar to the selected reference audio signal source represented by row 552. Represents a digital audio signal source within a corpus that contains voice signals. In this example, the reference audio signal is All of the similar audio signals, including this one, sound like a middle C being played on an electric piano. However, not all audio signals sound the same.

[0114] In order to determine which speech signal in the corpus sounds most similar to a reference speech signal, The signal is generated by the target model of the sound content category to which it belongs. A calculated cosine embedding can be used for a signal. For example, a calculated cosine embedding between sound embeddings for a signal. When distance is used as the criterion for similarity / dissimilarity in a search algorithm, approximate nearest If a search algorithm or other suitable search algorithm sounds similar to the reference audio signal It can be used to identify the top N (e.g., top 5) signals within a corpus. In some embodiments, cosine distance is used as the criterion for similarity / dissimilarity of search algorithms. It is used as such, but in other embodiments other distance measures are used. For example, Euclidean Distance or other appropriate distance criteria will be used.

[0115] Therefore, as a result of sound embedding being generated for digital audio signals in the corpus, End users do not need to perform keyword searches to find similar-sounding audio signals. and similar-sounding audio signals, in order to determine whether they sound similar. Without the need to listen (playback), it sounds similar to a selected reference speech signal within the corpus. To discover a digital audio signal source (e.g., a file) that contains a digital audio signal. It is possible.

[0116] 4.2 Sound as Search Input According to some embodiments, the end user can access existing digital voice communications within the corpus. Instead of selecting a source, use your personal computer as input for similar-sound search. It provides digital audio recordings captured using a recording device. In other words, Quasi-digital audio signals are based on microphone recordings of sounds generated or performed by the end user. The signal is generated based on the end user's personal computer. It is a component of a lighting device or a personal computing device. It may be attached to a chair. Personal computing devices are data communication networks. It can be connected to the server system via a network (e.g., the internet). M implements similar sound search, a corpus of digital audio signal sources, and disclosed herein. The technology used to generate and associated with the embedded sounds is accessible. It is possible.

[0117] For example, in some embodiments of the present invention, an end-user personal computer The microphone device is a mobile phone or other portable computer consisting of a microphone. It is a mobile device. The device is composed of applications (mobile applications). It is done. Using the application, the end user can access the application's graphics. The user interface controls allow you to start recording or select similar options. You can select an option. Selecting this option will activate the microphone and Devices that work in conjunction with the application digitally capture sound through the microphone. The recording begins as an audio signal. The end user is then captured by the microphone. It can generate the desired sound. For example, an end user can sing a melody of interest. Play an instrument, tap a drum pattern you're interested in, or play a drum you're interested in. Alternatively, rhythmic patterns can be beatboxed using voice. The end user can generate the desired sound. Alternatively, when performance ends, the end user will see the application's graphical interface. "Stop recording" or similar option via user interface controls You can choose this option.

[0118] The application then controls the graphical user interface. This can provide end users with the option to search for sounds similar to the recorded sound. Selecting this option means that the device interacting with the application will be network-based. Digital audio signals recorded on the server system can be submitted via the server system. Using the techniques disclosed herein, sound embedding for digital audio signals is generated. Possible. For example, the magnitude spectrum of a selected slice of a digital audio signal. The RAM can be generated using the techniques disclosed herein, and the generated spectrogram This generates sound embedding for recorded digital audio signals, target model It can be input through the forward path of an artificial neural network composed of Dell.

[0119] Once the sound embedding is generated, similar sounds within the corpus of digital audio signal sources are searched. It can be identified using an algorithm (e.g., approximate nearest neighbor search) within the corpus. The top N most similar-sounding digital audio signal sources are used in mobile applications. The end user's device can be returned for presentation to the end user. For example, table rows like rows 658, 660, 662, 664, and 666 are the most similar. For the application's graphical user, the digital audio signal source that can be heard is... It can be presented to the end user through an interface.

[0120] In some embodiments of the present invention, the sound content of the recorded digital audio signal The category is (for example, the graphical user interface control of an application) Selected by the end user (via). In some embodiments of the present invention, recorded The sound content category of the digital audio signal is the application or server The system automatically detects the selected or detected sounds. Sound embedding for digital audio signals recorded based on content category The server system can select an appropriate target model to use for achieving this. Furthermore, all digital audio signal sources considered for the search were selected or detected. Sounds that are assigned to a sound content category, or selected or detected. Belongs to the content category and generates sound embedding for recorded digital audio signals. The same target model used to generate equivalents produced by the same target model Based on selected or detected sound content categories, to have sound embedding. You can select a corpus of digital audio signal sources to search for similar sounds. .

[0121] 4.3 Detection of nearly overlapping sounds According to some embodiments of the present invention, the sound generated for a digital audio signal Coding is used to detect sounds that are nearly identical. For example, coding in digital audio signals. Existing digital audio signals within the path (library) are, for example, the original vocal sample. Original drum rhythms, or original instrument progressions (for example, original melodies) It may include original copyrighted works such as those by Deriff. Digital audio signals can be used by other people. This allows for one or more processes such as compression, equalization, time stretching, and pitch shifting. It was slightly modified using Rothes, and this person added it to the corpus and made it their own original It is possible to generate new digital audio signals to be provided as original copyrighted works. Operators or operators (e.g., cloud-based digital music library providers) This is useful for preventing copyright disputes or for identifying nearly identical sounds. For this purpose, we evaluate techniques that assist in identifying such largely overlapping digital audio signals. They will.

[0122] The techniques disclosed herein for generating sound embeddings for digital audio signals Using this method, sound embedding is used for adding digital audio signals to a corpus or library. Can be generated for digital audio signals submitted or uploaded to [platform name]. The sound embedding generated for a projected digital audio signal is based on distance. , embedding sounds generated for existing digital audio signals in a corpus or library (a) Sound embedding for submitted or uploaded digital audio signals (b) the distance between the embedding and the sound embedding for an existing digital audio signal (for example, When cosine similarity (e.g., Euclidean distance) is below a threshold (i.e., they are close in distance) The submitted or uploaded digital audio signals and existing digital audio signals are then For example, as things that largely overlap with each other (in computer databases, or in computer databases) (In the graphical user interface) you can flag (indicate) .

[0123] In some embodiments of the present invention, the distance threshold is the sound content of the digital audio signal. It may vary depending on the category. For example, the distance threshold may vary depending on the sound content category. This may vary depending on the variance of the statistical distribution of the signal in the given sound content. When the variance of the statistical distribution in the category is small, a small distance threshold is given for a given sound code. When used in content categories, if the degree of statistical variance is greater, then greater Thresholds may be used. For example, the statistical distribution of a category of signals is common to many (e.g., open (It may have a starting point, a uniform duration and decay pattern, and limited tonal diversity.) Therefore, signals in the drum one-shot category may have relatively small variance. As a result, the sound embedding generated for the drum one-shot category signal is flat On average, they can have smaller distances between them, and as a result, for example, one shot of an instrument Sound embedding generated for a category or other category of signal is a category signal It has a larger variance in the statistical distribution. As a result, in the case of nearly overlapping detections, nearly overlapping To improve the accuracy of multiple detections (for example, to reduce false positives), use a smaller distance threshold. It can be used for variance categories with smaller values. Larger distance thresholds also largely overlap. To improve detection accuracy (for example, to reduce missed detections), use larger variance categories. It can be used for this purpose.

[0124] 5.0 Implementation Mechanism - Hardware Overview Referring now to Figure 7, an exemplary composite in which several embodiments of the present invention can be implemented The cutting device 700 is shown in the block diagram.

[0125] Computing device 700 and its components, including connections, relationships, and functions. The examples provided are for illustrative purposes only and are not intended to limit the implementation of the present invention. Other computing devices suitable for implementing some embodiments of the present invention are , containing different components, including components with different connections, relationships, and functions It is possible.

[0126] The computing device 700 addresses the main memory 706, and bus 7 for transferring data between and within the various components of device 700 Includes 02 or other communication mechanisms.

[0127] The computing device 700 is also coupled to bus 702 to process information. Includes one or more hardware processors 704. 04 refers to the central processing unit (CPU), system-on-a-chip (SoC), and graphics. General-purpose processing units (GPUs) or other types of microprocessors It could be a microprocessor.

[0128] Main memory such as Random Access Memory (RAM) or other dynamic storage devices 706 It stores information and software instructions executed by the processor(s) 704. It is coupled to bus 702 for this purpose. Main memory 706 is also the processor(s) During the execution of software instructions performed by 704, temporary variables or other intermediate information It can be used for storage.

[0129] Software instructions are stored in a storage medium accessible to the processor(s) 704. When this happens, the computing device 700 executes the operation specified in the software instruction. To create a dedicated computing device customized for that purpose. (The term "software") "Software," "Software Instructions," "Computer Program," "Computer Executable" "Executable instructions" and "processor-executable instructions" are related to whether they are human-readable or not. Regardless, any arbitrary command to instruct a computing device to perform a specific action. It is broadly interpreted as covering machine-readable information of application software. Desktop applications, scripts, binaries, operating systems, Device drivers, boot loaders, shells, utilities, system software, J AVASCRIPT®, web page, web application, plugin Embedded software, microcode, compilers, debuggers, interpreters, virtual This includes, but is not limited to, machine tools, linkers, and text editors.

[0130] Computing device 700 provides static information for processor(s) 704. and to store software instructions, a read-only memory coupled to bus 702 ( Includes ROM)708 or other static storage devices.

[0131] One or more high-capacity storage devices 710 are magnetic, optical, solid-state, magneto-optical, Fixed media such as flash memory or any other available mass storage technology To permanently store information and software instructions on a movable medium, it is connected to bus 702. They are combined. Mass storage devices may be shared over a network, or they may be mass storage devices. The storage device may also be a dedicated high-capacity storage device.

[0132] Typically, at least one of the mass storage devices 710 (for example, the main solids of the device) A hard drive (or storage drive) is the operating system, user applications Not only application programs, drivers, and all kinds of other data files. Professionals for instructing the operation of computing devices, including other support files. Stores the gram and data components.

[0133] The computing device 700 displays information to the computer user. via bus 702, liquid crystal displays (LCDs) or other electronic visual displays Any display 712 can be coupled to it. In some configurations, touch detection technology (for example) Touch sensor surfaces incorporating resistance, capacitiveness, etc., allow for touch gestures (for example) Touch sensors (finger or stylus) communicate input to the processor(s) 704. It can be integrated with display 712 to form a type S display.

[0134] The input device 714, which includes alphanumeric keys and other keys, processes information and command selections with the processor. It can be coupled to bus 702 to communicate with 704. In addition to alphanumeric keys and other keys, Alternatively, instead of alphanumeric keys and other keys, the input device 714 can, for example, turn the power on / off. One or more physical buttons such as the ) button, home button, volume control buttons This may include a switch.

[0135] Another type of user input device transmits directional information and command selections to the processor 704. To reach and control the movement of the cursor on display 712, the mouse , trackball, or cursor control such as arrow keys 716 This input device typically has two axes that allow the device to specify its position on a plane. In other words, it has two degrees of freedom along a first axis (e.g., x) and a second axis (e.g., y).

[0136] In some configurations, such as the one shown in Figure 7, the display 712 and the input device 714 are used. , and one or more of the cursor controls 716, computing device 7 External components of 00 (e.g., peripherals), but in other configurations, the display 712, input device 714, and some or all of cursor control 716 are computer It will be integrated as part of the form factor of the cutting device 700.

[0137] The functions of the disclosed system, method, and module are as follows: , execute one or more programs of software instructions contained in main memory 706. In response to this, it can be executed by the computing device 700. These software instructions are sent from another storage medium, such as storage device 710, to main memory. It can be loaded into 706. When a software instruction contained in main memory 706 is executed... The processor(s) 704 perform functions according to some embodiments of the present invention.

[0138] The functions and operations of some embodiments of the present invention can be fully implemented with software instructions. However, the computing device 700 has hardwired circuitry or is programmable The circuit (e.g., ASIC, FPGA, etc.) is, in some embodiments of the present invention, In accordance with the requirements of a specific implementation, software instructions may be used to perform the function instead of or It can be used in combination with software instructions.

[0139] As used herein, the term “storage medium” refers to a specific computing device. This refers to any non-temporary medium that stores data or software instructions to operate in that manner. Such storage media may include non-volatile media or volatile media. Non-volatile media are, For example, non-volatile random-access memory (NVRAM), flash memory, optical displays Includes solid-state drives such as magnetic disks or storage devices 710. Volatile The storage medium includes dynamic memory such as main memory 706. A common format for storage media is, for example, Floppy disks, flexible disks, hard disks, solid-state drives Live, magnetic tape, or any other magnetic data storage medium, CD-ROM, other optical data Data storage medium, physical medium having a perforated pattern, RAM, PROM, and EPROM, FL ASH (registered trademark) - EPROM, NVRAM, flash memory, any other memory chip Includes top or cartridge.

[0140] A storage medium is separate from a transmission medium, but it can be used in conjunction with a transmission medium. A transmission medium is... It is involved in the transfer of information between storage media. For example, the transmission medium is a wire including bus 702. This includes coaxial cables, copper wires, and optical fibers. The transmission medium also includes radio data communications and These can take the form of sound waves or light waves, such as those generated during infrared data communication.

[0141] Various forms of media require one or more software instructions to be executed. It may be involved in transporting sequences of numbers to processor(s) 704. For example, Software instructions first access the magnetic disk or solid-state drive of the remote computer. It can be transported by drive.

[0142] The remote computer loads software instructions into its dynamic memory and uses the modem. This allows software commands to be sent via telephone lines. The modem, which is local to Device 700, receives data over the telephone line and transmits via infrared. A transmitter can be used to convert data into infrared signals. An infrared detector uses infrared The signal carries data, and the appropriate circuit can place the data on bus 702. 702 transports data to main memory 706, and processor(s) 704 is the main Data is retrieved from memory 706 and software instructions are executed. Main memory 706 Software instructions received by the processor(s) 704 are executed before execution by the processor(s) 704. Either of the following can be arbitrarily selected and stored in the memory device (multiple devices are possible) 710.

[0143] The computing device 700 also has one or more coupled to the bus 702. It may include communication interfaces 718. Communication interface 718 is 722 (for example, Ethernet® network, wireless network) Area networks, mobile phone networks, Bluetooth® wireless networks, etc. Bidirectional data traffic is coupled to a wired or wireless network link 720 connected to ( ). It provides communication. The communication interface 718 is digital data representing various types of information. Transmits and receives electrical, electromagnetic, or optical signals that carry a stream. For example, The communication interface 718 is a wired network interface card, integrated wireless A wireless network interface card or modem with a wire antenna (for example, This could be ISDN®, DSL, or a cable modem.

[0144] Network link(s) 720 typically connects via one or more networks. It provides data communication to other data devices. For example, network link 720 is via local network 722, host computer 724 or the Internet A service provider (ISP) 726 may provide connectivity to data equipment operated by an ISP. ISP726 was then the next global standard, now commonly referred to as "Internet" 728. Provides data communication services through a local packet data communication network. Network(s) 722 and Internet 728 are digital data streams It uses electrical, electromagnetic, or optical signals to carry them. Computing devices Signals and networks that carry digital data between 700 and other devices, and Signals passing over work links (multiple possible) 720 and communication interfaces (multiple possible) 718 This is an exemplary form of a transmission medium.

[0145] Computing device 700 includes network(s), network links. Send messages via (multiple) 720 and communication interface (multiple) 718. It can send and receive data, including program code. An example from the internet is... Server 730 is connected to the Internet (728), ISP (726), and local network (multiple). Application programmers via (possibly) 722 and communication interface (possibly multiple) 718 Gram may send the requested code. The received code is received It can be executed by processor 704 as specified, and / or for subsequent execution, It may be stored in the storage device 710 or other non-volatile storage.

[0146] The received code may be executed by processor 704 exactly as it was received. For and / or subsequent execution, to storage device 710 or other non-volatile storage It can be remembered.

[0147] 6.0 Implementation Mechanism - Software Overview Figure 8 shows the techniques employed to control the operation of an exemplary computing device 800. This is a block diagram of an exemplary software system 800, showing its connections, relationships, and functions. The included software system 800 and its components are intended for illustrative purposes only. This does not intend to limit the implementation of the present invention. Some embodiments of the present invention Other software systems suitable for implementing this have different connections, relationships, and functions. It may have different components, including the component that does the following.

[0148] The software system 800 instructs the operation of the computing device 700. Provided for: System memory (RAM) 706 and fixed storage device (e.g., hard The software system 800 which can be stored on disk or flash memory 710 is , including kernel or operating system (OS) 810.

[0149] OS810 handles process execution, memory allocation, file input / output (I / O), and data It manages low-level aspects of computer operation, including vice I / O management. 802A, 8 One or more application processes represented as 02B, 802C, ... 802N Gram is "loaded" (for example, fixed storage device 710) for execution by system 800. (Can be transferred from memory 706). Applications intended for use on device 800 Other software may also use, for example, internet locations (e.g., web sites). Download and install from the app store or other online service. Stored as a set of downloadable computer executable instructions, such as for the purpose of writing. obtain.

[0150] The software system 800 is graphical (for example, "point and click"). A graphics card for receiving user commands and data via a "touch gesture" method. Includes a physical user interface (GUI) 815. These inputs are then, In accordance with instructions from operating system 810 or application(s) 802 This can be operated by System 800. GUI815 also operates on OS810 and App It functions to display the results of operations from Relation(802), where the user You may then provide additional input or terminate the session (e.g., log off).

[0151] OS810 is bare hardware of device 700 820 (for example, processor (multiple) It can be run directly on (704). Alternatively, it can run on a hypervisor or virtual machine. The machine monitor (VMM) 830 is an intermediary between the bare hardware 820 and the OS 810. This is also acceptable. In this configuration, the VMM830 has OS810 on device 700 and bare hardware It functions as a software "cushion" or virtualization layer between the 820 and the 820.

[0152] VMM830 can host one or more virtual machine instances ("guest machines"). Instantiate and run. Each guest machine will run a "guest" operating system such as OS810. The guest operating system and the application designed to run on the guest operating system. Includes one or more applications such as Relation(802) VM The M830 is a virtual operating platform for guest operating systems. It provides and manages the execution of the guest operating system.

[0153] In some cases, VMM830 makes the guest operating system appear as if it were... This runs as if it were running directly on the bare hardware 820 of device 700. This could make it possible to run directly on bare hardware 820. The same version of the guest operating system configured to do so will also be modified or re-modified. It can run on VMM830 without configuration. In other words, VMM830 can run several In this example, full hardware and CPU virtualization is provided to the guest operating system. It is possible.

[0154] In another example, the guest operating system runs on VMM830 for efficiency. They can be specially designed or configured to do so. In these examples, the guest operating system The system "knows" that it will run on the virtual machine monitor. In other words, VMM830 provides paravirtualization to guest operating systems in some cases. It is possible.

[0155] The basic computer hardware and software described above are some of the practical applications of the present invention. The basic underlying computer components that can be used to implement the implementation form This invention is presented for demonstration purposes. However, the present invention is not applicable to any particular computing The present invention is not necessarily limited to the environment or computing device configuration. Instead, Some embodiments of the present invention will be understood by those skilled in the art in view of the present disclosure, and will be based on the features and mechanisms presented herein. Any type of system architecture that you would understand to be able to support the function This can be implemented in the processing environment.

[0156] 7.0 Other aspects of this disclosure Unless the context clearly indicates otherwise, the term “or” in the above specification and In the attached claims, used in an inclusive (not exclusive) sense. For example, when used to connect a list of elements, the term "or" refers to the elements within the list. It means one, a part, or all.

[0157] Unless the context clearly indicates otherwise, the term "comprising" is used. "including", "having", "based "on)" and "encompassing" refer to the above specification and attached patents. Used in an open-ended manner within the scope of the claim, and does not include any additional elements, functions, actions, or operations. Do not exclude it.

[0158] Unless the context explicitly indicates otherwise, the phrase "at least one of X, Y, and Z" Connective words such as "X, Y, or Z" refer to items, terms, etc., or combinations thereof. It is understood that it is acceptable. Therefore, such conjunctions are The embodiment is such that at least one of X, at least one of Y, and at least one of Z It is not intended to imply that their existence is necessary.

[0159] Unless the context clearly indicates otherwise, the above detailed description and the attached claims apply. As used, the singular forms "a," "an," and "the" also include the plural forms. It is illustrated.

[0160] Unless the context clearly indicates otherwise, the above detailed description and the attached claims apply. The terms, such as "first," "second," etc., are used in some examples to explain various elements. Although used in the details, these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, the first These computing devices are sometimes referred to as second computing devices. Yes, and similarly, the second computing device is the first computing device. It may sometimes be referred to as the first computing device and the second computer. Both computing devices are computing devices, but they are the same computer It is not a cutting device.

[0161] In the above specification, some embodiments of the present invention may differ from one embodiment to another. This has been explained with reference to a number of specific details. Therefore, the specification and drawings are intended to be used for reference. It is to be considered illustrative rather than restrictive. The sole and exclusive indicator of the scope of the present invention, And what the applicant intends to be the scope of the invention includes any subsequent modifications, such as The literal and They are within the same range.

Claims

1. It is a method, A set of digital audio signal sources associated with a specific sound content category. Reading a specific digital audio signal source, To generate a time-frequency representation based on the aforementioned specific digital audio signal, An artificial neural network is used to provide a latent spatial representation of the aforementioned time-frequency representation. Learning a set of numerical codes, wherein the set of numerical codes is determined to be the same as the time cycle. The learning process described above involves a dimension less than the dimension of the wavenumber representation, To train the aforementioned artificial neural network, the specific sound content The set of digital audio signal sources in the category of multiple other digital audio signal sources For each of these, the reading, generating, and learning processes are repeated. doing, From the aforementioned trained artificial neural network, the learned model parameters are set To obtain the The computer storage medium is used to store the learned data for the specific sound content category. To store the set of model parameters The method, including the method described above.

2. Based on a first digital audio signal associated with the aforementioned specific sound content category Next, the first time-frequency representation is generated, A first set of numerical codes that provide a latent spatial representation of the first time-frequency representation is calculated. To do this, the set of learned model parameters is used, Based on a second digital audio signal associated with the aforementioned specific sound content category Next, a second time-frequency representation is generated, A second set of numerical codes that provide a latent spatial representation of the second time-frequency representation is calculated. To do this, the set of learned model parameters is used, The distance between the first set of numerical codes and the second set of numerical codes is calculated. What to do and The method according to claim 1, further comprising:

3. Based on the distance, the first digital audio signal and the second digital audio signal An indication that similar sounds are included should be added to the computer graphical user interface. The method of claim 2, further comprising causing to demonstrate.

4. From the computing device via the data communication network the first digital Further including receiving audio signals, The first digital audio signal is used by the computer to record a human audible performance. The recording device, or the computer that records human audible performance Using a microphone coupled to the input device in an operable manner, the computer Captured by the shooting device, The method according to claim 2.

5. Comparing the aforementioned distance with a distance threshold, Based on the fact that the distance is less than the distance threshold, the first digital audio signal and An indication that the second digital audio signal contains sounds that largely overlap is displayed using computer graphics. To present it in the physical user interface and The method according to claim 2, further comprising:

6. The distance threshold is further selected based on the aforementioned specific sound content category. The method according to claim 5, which includes the method described in claim 5.

7. Cosine similarity between the first set of numerical codes and the second set of numerical codes. Based on the nature, between the first set of numerical codes and the second set of numerical codes The method according to claim 5, further comprising calculating the aforementioned distance.

8. The aforementioned specific sound content category consists of loops and one-shots. The method according to claim 1, selected from the following.

9. The aforementioned specific sound content categories include drum loops, drum one-shots, and instruments. Selected from the group consisting of loops and instrument one-shots, according to claim 1. method.

10. The artificial neural network comprises an input layer and one or more encoder intermediate layers. The method comprises a bottleneck layer, one or more decoder intermediate layers, and an output layer. , the set of numerical codes that provides the latent spatial representation of the time-frequency representation is the bottle The method according to claim 1, further comprising obtaining from the Lunec layer.

11. The one or more encoder intermediate layers comprises one or more convolutional layers, Claim 10, wherein one or more decoder intermediate layers comprise one or more convolutional layers. Methods used.

12. The time-frequency representation is generated based on the aforementioned specific digital audio signal. Acquiring a pre-processed signal based on the aforementioned specific digital audio signal, A duration-normalized signal is generated based on the aforementioned pre-processed signal, The selection of a time-shifted slice signal of the duration-normalized signal, wherein the time The shift slice signal has a specific duration, and the selection of the above, To generate the time-frequency representation based on the time-shifted slice signal. The method according to claim 1, based on the present invention.

13. Selecting a specific duration based on the aforementioned specific sound content category. The method according to claim 12, further comprising:

14. The time-frequency representation is generated based on the aforementioned specific digital audio signal. The process involves generating a duration-normalized signal based on the aforementioned specific digital audio signal, To generate a preprocessed signal based on the aforementioned duration-normalized signal, Selecting a time-shifted slice signal of the pre-processed signal, the time-shifted sliced ​​signal of the pre-processed signal. The selection of the soft slice signal having a specific duration, To generate the time-frequency representation based on the time-shifted slice signal. The method according to claim 1, based on the present invention.

15. To generate multiple latent spatial representations based on the aforementioned specific digital audio signal, Using the aforementioned artificial neural network, the latent space among the multiple latent space representations Learning multiple sets of numerical codes that provide interrepresentations and The method according to claim 1, further comprising:

16. A computing system, One or more hardware processors, Storage media and Instructions stored in the storage medium, which are implemented by the computing system. When this is done, the computing system will be as described in any one of claims 1 to 15. The command and The computing system comprising the above.

17. One or more non-temporary storage media that store one or more sequences of instructions The instruction is executed by one or more computing devices. And, to the one or more computing devices, any one of claims 1 to 15 One or more non-temporary storage media that perform the method described in the section.

Images