User-guided adaptive playlist using joint audio-text embedding

Abstract

A method includes providing, through an audio playback interface, an initial playlist including audio tracks. The method includes receiving a user preference associated with an initial track during a listening session, wherein the user preference indicates a listening mood of a user and includes one or more of user behavior or natural language input. The method includes generating a representation of the user preference in a joint audio-text embedding space by applying a twin-tower model including an audio embedding network and a text embedding network. The two embedded proximity indicates a semantic similarity. The method includes training a machine learning model for generating an updated playlist in response to the listening mood of the user during the listening session. The method includes applying the machine learning model to generate the updated playlist.

Description

Background Art

[0001] Music recommendations may be made based on user preferences, listening history, etc. Music playlist generation and music discovery may be generated in a user interface for music playback at the beginning of each listening session. Summary of the invention

[0002] Music playlist generation and music discovery can be generated at the beginning of each listening session based on various factors, such as user listening history, seed songs, co-viewing data, and music context. However, these playlists usually do not take into account user behavior in a given listening session. Therefore, there is a need to provide instant adaptation of music playlists based on the user's mood in the current listening session. The user's mood can be inferred by analyzing listening / skipping behavior and / or based on natural language input.

[0003] In one aspect, a computer-implemented method is provided. The method includes providing an initial playlist including one or more initial audio tracks through an interactive audio playback interface. The method also includes receiving user preferences associated with initial audio tracks of the initial playlist during a listening session, wherein the user preferences indicate the listening mood of the user during the listening session, and wherein the user preferences include one or more of user behavior on the initial audio track or natural language input associated with the initial audio track. The method further includes generating a representation of the user preference in a joint audio-text embedding space by applying a dual-tower model, the dual-tower model including an audio embedding network for generating audio embeddings of the initial audio track and a text embedding network for generating text embeddings of the natural language input, wherein the proximity of the two embeddings in the joint audio-text embedding space indicates semantic similarity. The method further includes training a machine learning model for generating an updated playlist including one or more updated audio tracks based on the representation of the user preferences, wherein the one or more updated audio tracks are in response to the listening mood of the user during the listening session. The method also includes applying the trained machine learning model to generate an updated playlist. The method further includes replacing the initial playlist with the updated playlist in the interactive audio playback interface.

[0004] In another aspect, a computing device is provided. The computing device includes one or more processors and a data storage device. The data storage device stores computer executable instructions, which, when executed by the one or more processors, cause the computing device to perform functions. The functions include: providing an initial playlist including one or more initial audio tracks through an interactive audio playback interface; receiving user preferences associated with initial audio tracks of the initial playlist during a listening session, wherein the user preferences indicate a listening mood of the user during the listening session, and wherein the user preferences include one or more of user behavior on the initial audio track or natural language input associated with the initial audio track; generating a representation of the user preferences in a joint audio-text embedding space by applying a two-tower model, the two-tower model including an audio embedding network for generating an audio embedding of the initial audio track and a text embedding network for generating a text embedding of the natural language input, wherein the proximity of the two embeddings in the joint audio-text embedding space indicates semantic similarity; training a machine learning model for generating an updated playlist including one or more updated audio tracks based on the representation of the user preferences, wherein the one or more updated audio tracks are responsive to the listening mood of the user during the listening session; applying the trained machine learning model to generate the updated playlist; and replacing the initial playlist with the updated playlist in the interactive audio playback interface.

[0005] In another aspect, a computer program is provided. The computer program includes instructions, which when executed by a computer cause the computer to perform functions. The functions include: providing an initial playlist including one or more initial audio tracks through an interactive audio playback interface; receiving user preferences associated with initial audio tracks of the initial playlist during a listening session, wherein the user preferences indicate a listening mood of the user during the listening session, and wherein the user preferences include one or more of user behavior on the initial audio track or natural language input associated with the initial audio track; generating a representation of the user preferences in a joint audio-text embedding space by applying a two-tower model, the two-tower model including an audio embedding network for generating an audio embedding of the initial audio track and a text embedding network for generating a text embedding of the natural language input, wherein the proximity of the two embeddings in the joint audio-text embedding space indicates semantic similarity; training a machine learning model for generating an updated playlist including one or more updated audio tracks based on the representation of the user preferences, wherein the one or more updated audio tracks are responsive to the listening mood of the user during the listening session; applying the trained machine learning model to generate the updated playlist; and replacing the initial playlist with the updated playlist in the interactive audio playback interface.

[0006] In another aspect, an article of manufacture is provided. The article of manufacture includes one or more computer-readable media having computer-readable instructions stored thereon, which when executed by one or more processors of a computing device cause the computing device to perform functions. The functions include: providing an initial playlist including one or more initial audio tracks through an interactive audio playback interface; receiving user preferences associated with initial audio tracks of the initial playlist during a listening session, wherein the user preferences indicate a listening mood of the user during the listening session, and wherein the user preferences include one or more of user behavior on the initial audio track or natural language input associated with the initial audio track; generating a representation of the user preferences in a joint audio-text embedding space by applying a two-tower model, the two-tower model including an audio embedding network for generating an audio embedding of the initial audio track and a text embedding network for generating a text embedding of the natural language input, wherein the proximity of the two embeddings in the joint audio-text embedding space indicates semantic similarity; training a machine learning model for generating an updated playlist including one or more updated audio tracks based on the representation of the user preferences, wherein the one or more updated audio tracks are responsive to the listening mood of the user during the listening session; applying the trained machine learning model to generate the updated playlist; and replacing the initial playlist with the updated playlist in the interactive audio playback interface.

[0007] In another aspect, a system is provided. The computing device includes: a component for providing an initial playlist including one or more initial audio tracks through an interactive audio playback interface; a component for receiving user preferences associated with initial audio tracks of the initial playlist during a listening session, wherein the user preferences indicate a listening mood of the user during the listening session, and wherein the user preferences include one or more of user behavior to the initial audio track or natural language input associated with the initial audio track; a component for generating a representation of the user preferences in a joint audio-text embedding space by applying a dual-tower model, the dual-tower model including an audio embedding for generating an audio embedding of the initial audio track; An embedding network and a text embedding network for generating a text embedding of the natural language input, wherein the proximity of the two embeddings in the joint audio-text embedding space indicates semantic similarity; a component for training a machine learning model for generating an updated playlist including one or more updated audio tracks based on the representation of the user's preferences, wherein the one or more updated audio tracks are responsive to the listening mood of the user during the listening session; a component for applying the trained machine learning model to generate the updated playlist; and a component for replacing the initial playlist with the updated playlist in the interactive audio playback interface.

[0008] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a diagram illustrating an example audio-text embedding framework according to an example embodiment.

[0010] Figure 2 is a diagram illustrating an example adaptive playlist system according to an example embodiment.

[0011] Figure 3 is a diagram illustrating the training phase and the inference phase of a machine learning model according to an example embodiment.

[0012] Figure 4 A distributed computing architecture according to an example embodiment is depicted.

[0013] Figure 5 is a block diagram of a computing device according to an example embodiment.

[0014] Figure 6 Depicted is a network of computing clusters arranged as a cloud-based server system according to an example embodiment.

[0015] Figure 7 is a flowchart of a method according to an example embodiment. DETAILED DESCRIPTION

[0016] Music tagging and content-based retrieval systems have traditionally been built using predefined ontologies, covering a strict set of musical properties or textual queries. Classifiers are typically trained to label examples using a predefined and fixed list of classes, which are often manually specified as a structured ontology indicating inter-class relationships. While the visual domain benefits from the availability of a large number of captioned images on the web, in the general ambient audio domain, such large-scale audio-caption pairs are less readily available, and related work has relied on small captioned datasets. Crucially, these small captioned datasets do not span the diversity of sound description languages, and they have not been successful in the more difficult zero-shot setting. While general ambient audio consists of background sounds that are unlikely to elicit spontaneous descriptions, music audio is often the central focus. As a result, text associated with music videos is more likely to be related to underlying music concepts (e.g., genre, artist, mood, structure). Therefore, a flexible language interface is described by which music concepts can be linked to related music audio.

[0017] Both discriminative and generative model approaches have been used in both unsupervised and self-supervised pre-training. For example, in discriminative training, existing models have been designed to learn representations that assign higher similarity to audio segments extracted from the same recording than to segments from different recordings. Also, for example, intermediate embeddings of generative models have been shown to provide audio representations for downstream classification. Various forms of weak supervision, such as user preference statistics and visual cues, have also been examined.

[0018] Similar to using contrastive learning to align image features and free-form natural language using large-scale data, tri-modal architectures are available where audio towers are used for image-text models and contrastive learning is used to enforce cross-modal alignment. In the audio domain, contrastive learning has been used to align latent representations of audio and related tokens. The tokens can be obtained from a fixed vocabulary of size 1K in the dataset Freesound, and the input to the text encoder can be multi-hot encoded tokens. Pre-trained non-contextual word embedding (Word2Vec) models can be used to support generalization to new terms beyond the 1K tokens. Contrastive learning for zero-shot audio classification has also been explored using the AudioSet and ESC-50 datasets. However, these models do not support generalization to free-form natural language.

[0019] Some existing methods use text label classes to ground semantics in music through multi-label classification tasks. For example, a large vocabulary of n-grams (e.g., about 100K) can be mined from noisy natural language text associated with music videos. Then, a cross-entropy loss can be employed to train a music audio encoder, where the softmax layer weights are used as text label embeddings that can be aligned with audio features by construction. Various training tasks (e.g., classification, regression, metric learning) have also been explored to align free-form text and music audio, relying on pre-existing sentiment labels to connect the modalities. Moreover, for example, a large number of audio-caption pairs (e.g., about 250K) can be mined from private production music libraries and used to train a multimodal Transformer, where the two modalities are early fused using a triplet loss. However, the choice of early fusion achieved through a cross-attention layer limits the practicality of the resulting embeddings for transfer learning applications.

[0020] Therefore, there is a lack of acoustic models that directly connect music audio to unconstrained natural language music descriptions. Content-based music information retrieval can be greatly enhanced by connecting the rich semantics expressible as free-form text to a wide range of and fine-grained musical properties. As described in this paper, a two-tower parallel encoder approach produces a joint embedding space that provides a natural language interface to arbitrary music audio. Such an architecture facilitates downstream opportunities in cross-modal retrieval, zero-shot labeling, and language understanding. Moreover, late fusion of two modalities with a contrastive loss enables efficient and effective use of batched negative samples to accelerate training, compared to, for example, a triplet loss with a single random negative.

[0021] Also as described herein, less restrictive natural language interfaces can be developed to access the classification information behind the original content signal. A cross-modal supervision model using a large number of text annotations weakly associated with music audio is described. The model relies on large-scale training resources and a flexible neural network architecture that can be configured to simulate the complex, non-monotonic relationship between language and other modalities. As described herein, music recordings (e.g., 44 million music recordings corresponding to approximately 370K hours) and weakly associated free-form text annotations can be used to train the twin tower joint audio-text embedding model. A large number of text label classes can be generated to base semantics in music through multi-label classification tasks. This can be achieved by extracting text annotations from metadata, comments, and playlist data can be collected and mapped to a training set (e.g., a collection of more than 44 million Internet music videos). As with some image-text model training, text data represents music content in a small number of cases. Therefore, in some embodiments, a separately trained text classifier can be used to apply text pre-filtering to identify music descriptions.

[0022] Such a large-scale dataset can be used to train a semantically structured music audio embedding model equipped with a natural language interface. The model adopts a dual-tower parallel encoder architecture, using a contrastive loss objective that induces a shared embedding space between music audio and text. For the audio tower, state-of-the-art ResNet-50 and transformer-based audio modeling architectures can be evaluated, each initialized with a different pre-training strategy. The Bidirectional Encoder Transformer (BERT) neural language model architecture can be used for the text tower, which can be warm-started using publicly available pre-trained checkpoints.

[0023] These evaluations demonstrate that the model achieves state-of-the-art performance in transfer learning on a variety of music information retrieval tasks. The model also achieves a range of capabilities in cross-modal text-to-music retrieval, zero-shot music tagging, and language understanding in the music domain.

[0024] Thus, a shared embedding space for music audio and free-form natural language text is described, where proximity predicts shared semantics both within and across modalities. To achieve this, cross-modal contrastive learning can be used with a simple two-tower architecture. A large-scale training dataset of (audio, text) pairs is mined and used to train the model. This can be combined with a text pre-filtering mechanism to improve the supervision quality of the contrastive target.

[0025] Figure 1 is a diagram illustrating an example audio-text embedding framework 100 according to an example embodiment. Figure 1 A high-level schematic diagram of a machine learning framework is shown. Some embodiments involve generating a representation of user preferences in a joint audio-text embedding space by applying a two-tower model, the two-tower model comprising an audio embedding network for generating audio embeddings of an initial audio track and a text embedding network for generating text embeddings of a natural language input, wherein the proximity of the two embeddings in the joint audio-text embedding space indicates semantic similarity. For example, each adaptive playlist model consists of two separate embedding networks for audio and text input modalities. In some embodiments, these networks can each terminate at a point having the same dimension of Normalize the embedding space. In some embodiments, the network may not share weights. Audio Embedding Network 115 , taking as input a logarithmic Mel spectrogram context window 105 with F Mel channels and T frames. Text Embedding Network 120 Term Word List The above null-padded text word sequence 110 of length n is taken as input.

[0026] Given a collection of music recordings and the associated textual elements for each recording, a cross-modal training dataset of (audio, text) pairs can be generated. For each recording, an F-channel log-mel spectrogram can be determined, and a set of T-frame context windows can be extracted. Each associated textual element can be null-padded or truncated to a fixed length n. Thus, each mini-batch It can be in the form of of In some embodiments, the context window of the spectrogram can be selected by first selecting a random recording and sampling a random spectrogram context window therefrom. to sample each target pair. Next, the associated text elements can be randomly selected Based on this sampling scheme, multiple epochs can be utilized to cover the entire training audio and all associated text.

[0027] In some embodiments, for each mini-batch of music video tracks and the associated set of text annotations at the video level, a mini-batch of (audio, text) pairs may be constructed by extracting a random 10 second window from each track and selecting a random associated text annotation from the desired source. In some embodiments, the remainder of the track may not be used.

[0028] The audio embedding network 115 can generate an audio embedding 125, and the text embedding network 130 can generate a text embedding 130. In some embodiments, multiple text annotations for each example can be connected. Some embodiments involve contrasting training of the audio embedding network and the text embedding network based on an audio-text contrast loss. For example, the training of the audio embedding network 125 and the text embedding network 130 can include training to minimize the audio-text contrast loss 135 (e.g., a batch contrast multi-view encoding loss function). In such an embodiment, the audio-text contrast loss is a cross-modal extension of the information noise contrast estimation (InfoNCE) loss and the normalized temperature scaled cross entropy (NT-Xent) loss. For example, the audio-text contrast loss 135 can be a cross-modal extension of the InfoNCE and NT-Xent losses. For each batch , audio-text contrast loss 135 Takes the following form:

[0029]

[0030] (Equation 1)

[0031] where h is given by Given a critic function, ,and is a trainable temperature hyperparameter. The normalized embedding model output, the inner product can be the cosine similarity. The goal of the critic function h is to produce a large positive value for the target audio-text pair 140 and a small value close to zero for all non-target audio-text pairs 140 constructed within the batch.

[0032] Audio Embedded Network

[0033] One or more audio frameworks may be used to embed audio into the network 115 In some embodiments, the audio embedding network includes one or more of: (i) a modified Resnet-50 architecture where the stride 2 in the first convolutional layer is removed, or (ii) an Audio Spectrogram Transformer (AST). For example, the Resnet-50 architecture can be modified appropriately where the stride 2 in the first convolutional layer can be removed, and the architecture can be applied to log-Mel spectrograms treated as grayscale images (e.g., F = 64 Mel channels, 25 ms Hanning window, 10 ms step size). In order to be able to model long-term musical structure, an audio spectrogram transformer (AST) can be used. 10-second windows in the form of spectrogram patches (randomly selected from each training clip) are used for training.

[0034] In some embodiments, SpecAugment may be applied to each spectrogram during training before feeding each spectrogram 105 to the embedding network. In some embodiments, a final mean pooling operation may be applied across temporal and mel channels, followed by a linear fully connected layer with d = 128 units, the output of which is Normalized. All layers except the last linear transformation layer can be pre-trained via logistic regression on AudioSet (e.g., including all 527 classes). In some embodiments, the last classifier layer can be removed before fine-tuning on the playlist generation task.

[0035] Another architecture that can be used is the Audio Spectrogram Transformer (AST), which is a port of the Visual Transformer (ViT) infrastructure and is commonly used in the audio event classification space. In some embodiments, the AST may include a stack of 12 Transformer blocks (e.g., hidden dimension 768, 12 self-attention heads) that can be applied to a flat linear transformation A sequence of “words” corresponding to a set of time-frequency patches (e.g., stride 10 along both axes) that can be obtained from The SpecAugment can be applied during training. Similar to Transformer-based language models, a trainable positional encoding can be added to the sequence of patch tokens, and a [CLS] token can be added to the front of the sequence as a summary of the contextual patch embedding. In some embodiments, a d = 128 unit and A normalized linear fully connected layer is applied to the final 768-dimensional encoding at the [CLS] token position, and this forms the audio embedding network 115 All layers except the final linear transformation layer can be warm-started for training, for example by using a common AST checkpoint.

[0036] In some embodiments, a large pre-training dataset of over 50M random internet video tracks can be used, where a vocabulary of 10K video-level metadata tags (mostly not music related) can be predicted, and the last classifier layer can be removed before fine-tuning of the joint embedding.

[0037] Text Embedding Network

[0038] In some embodiments, the text embedding network includes a bidirectional encoder transformer (BERT) with a base-uncased architecture. For example, BERT with a base-uncased architecture can be used for the text embedding model 130. In general, BERT contains a stack of 12 transformer blocks (e.g., hidden dimension 768 and 12 self-attention heads). The BERT word fragment tokenizer can be applied to convert the text input string into a word unit sequence n = 512. The output of the text embedding network 120 is defined as [CLS] word unit embedding 130, which is linearly transformed into a shared audio-text embedding space (e.g., dimension d = 128) and then Normalization. The text embedding network 120 can be warm-started using publicly available checkpoints.

[0039] The audio embedding 125 and the text embedding 130 can be jointly embedded in a joint embedding space that semantically drives proximity. For example, for a word with a given meaning, nearby music in the embedding space will be associated with the meaning of the moving word.

[0040] Training Dataset

[0041] A collection of 50 million Internet music videos can be used as a starting point for assembling the large-scale collection of (audio, text) pairs needed to train the playlist generation embedding model. From the soundtrack of each video, a 30-second clip can be extracted starting at the 30-second mark. Subsequently, a pre-existing music audio detector can be applied, and clips with less than half music content can be removed. After this filtering, there are potentially about 44 million 30-second clips, equivalent to nearly 370K hours of audio.

[0042] One or more noisy text data sources may be used for each music video, including, for example: (i) short-form (SF) text including video titles and tags; (ii) long-form (LF) text including video descriptions and comments; and (iii) titles of 171 million playlists (PL) linking to Internet music videos in our dataset. In general, there is no guarantee that these text sources may refer to the musical properties of a track. In particular, commentary data may include a lot of noise and may be subjective or not directly related to the music content. Table 1 below shows examples of features that may be associated with music to provide each type of text annotation.

[0043]

[0044] Table 1

[0045] In some embodiments, since the text is very noisy, the training playlists can be generated using filtered SF and LF text data to obtain a cleaner set of music description annotations (PL is used without filtering). Therefore, the pre-trained BERT model can be fine-tuned on a small selected set of 700 sentences through a binary classification task. The sentences in the selected set can be manually labeled as music descriptive or non-music descriptive. A text classifier can be applied to filter the sentences in the LF annotations. To filter the playlist titles, a perplexity threshold using a language model that has been fine-tuned on a selected set of 7000 high-quality playlist titles can be applied. In some embodiments, a set of rule-based filtering heuristics can be applied independently to clean up the SF annotations.

[0046] Table 2 below shows the size and coverage of each of these text sources before and after filtering. Token counts (in billions) cover all 44M videos. APV represents the average number of text annotations (i.e., individual free-form strings) per video, including videos without annotations. In some embodiments, playlist titles and / or filtered long-form annotations may only be available for a small number of recordings in the dataset (18M and 6.8M out of a total of 44M, respectively).

[0047]

[0048] Table 2

[0049] In some embodiments, AudioSet can be converted to a set of audio-text pairs denoted as ASET. Specifically, all examples of all 527 classes can be included, using each label string attached to the example as the associated text annotation. This may produce a set of about 2 million 10-second training clips, with an average of 1.8 label annotations per clip.

[0050] In general, there may be a size imbalance between the four different data sources due to differences in the corresponding language richness and quality. Therefore, in some embodiments, each mini-batch can be constructed with a prescribed set of ratios, which can be selected without optimization: 2:2:1:1 for SF:LF:PL:ASET. This means that despite the small size, the filtered LF annotations may still include 1 / 3 of each mini-batch.

[0051] For each mini-batch of music video tracks and the associated set of text annotations at the video level, a mini-batch of (audio, text) pairs can be constructed by extracting a random 10 second window from each track (discarding the rest) and selecting a random associated text annotation from the desired source. Such a sampling scheme can be performed by using multiple epochs to cover the training audio and associated text.

[0052] Figure 2 2 is a diagram illustrating an example adaptive playlist system 200 according to an example embodiment. Some embodiments relate to providing an initial playlist including one or more initial tracks through an interactive audio playback interface. The playlist interface 200A can be a user interface for music playback. For example, multiple main menus 202 can include menu options for "Home", "Explore", "Library", "Upgrade" and "Search". The playlist interface 200A can include playback options, such as a rewind button 204, a play / pause button 206, and a forward button 208. The album cover 210 of the current track being played can be displayed. The elapsed time indicator 212 can indicate how much the track has been played. Moreover, for example, a thumbnail 214 of the album cover 210 can be provided, as well as features 216 associated with the album (e.g., singer, song, genre, release year, etc.). The like icon 218 can enable users to indicate that they like the current track, while the dislike icon 220 can enable users to indicate that they do not like the current track. A volume adjustment control 222 can also be provided to adjust the output level of the audio.

[0053] For the current listening session, additional submenus may be provided, such as "Next" 224, "Lyrics" 226, and "Related" 228. Upon selection of "Next" 224, one or more recommended keywords 230 may be provided as selectable icons, such as, for example, "All," "Familiar," "Discover," "Hot," "Deep Cuts," "Like Radiohead," and the like. The user may select the selectable icons to indicate a preference, and the system may adjust the playlist based on the selected keyword. A current playlist 232 listing the tracks selected for the user is displayed. For example, the first track "Track 1" may display an associated "Playing" indicator indicating that the track is currently playing.

[0054] In general, each user can be represented as a bag of fine-grained music interest prototypes in some abstract space, as a highly specialized intersection of genres and moods. Over an extended time span, each user may span a broad sampling of these interest prototypes, but in each listening session, may get the best listening experience for a relatively small number of interest prototypes. Pre-generated playlists may not know a priori which interest prototypes are more suitable for the user in any given listening session. Therefore, user behavior in a given session can be a nearly accurate indicator of the user's interests during that session.

[0055] One way to characterize target archetypes is to use a binary classifier trained on top of a generic content representation that characterizes moods and genres and can therefore act as a proxy for user interests. This binary classifier can be a means of identifying which interest archetypes are active for a user in a given session. For example, an interest space with a natural clustering structure can be generated where the centroids define interest archetypes. The users in each session are bags of such interest centroids. A content embedding space can be used as a proxy for the interest space, and a classifier can be trained to learn which set of these archetypes are activated by operating in the embedding space. In some embodiments, a nearest neighbor model can be used as the ML model 246. However, the geometry of the embedding space may impose additional constraints that may make nearest neighbor models somewhat inefficient. In some embodiments, a family of general models (e.g., a multilayer perceptron (MLP)) can be used that can accommodate more complex regions in the embedding space (i.e., regions with respect to the training examples). distance is not easy to model).

[0056] In each listening session, the user may provide one or more simultaneous inputs, such as: (i) listening / skipping behavior for that session, or (ii) a set of user-provided natural language inputs describing their current interests. Typically, an adaptive playlist model may be constructed using audio and / or co-viewing embeddings. Joint embeddings with text provide additional data that may capture user preferences. As used herein, “joint” means that a single classifier will process both types of embeddings (e.g., the dual-tower structure described herein). At the start of a listening session, an initial candidate playlist or library of relevant tracks may be initialized. For example, this may be accomplished by applying a seed generation program (e.g., currently used by YOUTUBE ™ Music and YouTube ™ Mix generated program) to obtain the initial playlist.

[0057] When a user listens to tracks in a session, existing recommendation systems typically represent the user as an average of the long-term listening behavior (e.g., picking tracks from a user distribution that captures the long-term behavior). However, such a representation is typically unable to adapt to the user's mood in the current session. The user's long-term behavior can be represented as a collection of patterns, and at a given point in time, tracks can be extracted from one of these patterns rather than from the entire user distribution. Therefore, the user's behavior during the current session may indicate one or more patterns from which tracks can be selected for the current session. In addition, when the user's behavior during the current session is represented using a joint audio-text embedding, it may be easier for the model to identify playlists.

[0058] For example, the joint embedding space is relatively compact, so relatively simple classifiers can be built in the joint embedding space. Moreover, the joint embedding space is constructed in a way that simple models can make musical sense, for example. Therefore, recommendations are less dependent on metadata associated with tracks, or preferred genres, artists, etc. Musical features such as rhythm, instruments, genres, beats, melody, rhythm, etc. are represented in the joint embedding space. Therefore, hyperplanes can be constructed that can separate points in the embedding space in subtle ways, rather than roughly separating likes and dislikes.

[0059] For example, a rock band may have been around for a long time, and the band may have dabbled in different genres over the years, or their style may have changed, or a singer, guitarist, or drummer may have left or joined the band. Therefore, a band may have several different types of music, and a user may not be interested in only certain types of music produced by rock bands. Classifiers based on artists, genres, etc. may not be able to distinguish different aspects of such a rock band's musical work. In contrast, more refined similarity-based methods can provide meaningful distinctions. Construct a joint embedding space so that semantically similar music and words are co-embedded near each other.

[0060] In general, users can provide various signals indicating listening mood. The term "listening mood" can refer to the user's music preferences during a session. For example, even if the user may usually listen to jazz or rock music, in a given listening session, the user may be more interested in western flute instruments. Moreover, for example, various factors such as weather, time of day, season, social gathering, vacation, special occasion, road trip, etc. may affect the user's listening mood during any specific listening session. Therefore, the user can choose to listen to the entire track (for example, track 1), and / or listen to most of the track. Such a signal can be marked as a positive example indicating listening mood. For example, if track 1 is Mozart's flute concerto, the adaptive playlist generation system 200B can infer that the user's listening mood includes western flute instruments. In some embodiments, the adaptive playlist generation system 200B can use known user preferences for Mozart and Beethoven (for example, based on the music repository 242) to infer that the user's listening mood includes western flute instruments of Mozart and Beethoven.

[0061] Moreover, for example, the user can choose to skip one or more tracks in the playlist 232. For example, after listening to track I, the user can choose to skip tracks II and III (or listen to a small portion of tracks). Such a signal can be marked as a negative example indicating a listening mood. For example, track I can be a flute concerto by Mozart, and tracks II and III can be concertos for flute and piano. Based on the positive signal associated with track I and the negative signal associated with tracks II and III, the adaptive playlist generation system 200B can infer that the user's listening mood includes a flute instrument, but does not include flute and piano instruments. As another example, track I can be a flute concerto by Mozart, track II can be a concerto for flute and piano, and track III can be a track for flute and string quartet. The user can listen to tracks I and II and skip track II. Therefore, a positive signal can be associated with tracks I and III, and a negative signal can be associated with track II. Based on such signals, the adaptive playlist generation system 200B may infer that the user's listening mood includes a flute instrument, a flute and string combination, but does not include a flute and piano combination.

[0062] The term "initial track" can refer to any track in playlist 232. In some embodiments, the initial track can be the track at the top of playlist 232, and / or the track currently being played. In some embodiments, the initial track can be a skipped track, or a track that the user has listened to for less than a threshold amount (e.g., less than 5% of the track). In addition, as described herein, playlist 232 can be updated by listening to, skipping each track and / or natural language input. Therefore, initial playlist 232 will then be considered as an updated playlist representing the next iteration of the playlist generation process based on user preferences.

[0063] Some embodiments involve training a machine learning model based on a representation of user preferences to generate an updated playlist including one or more updated tracks, where the one or more updated tracks are responsive to the user's listening mood during a listening session. For example, a binary classifier can be repeatedly trained at each step of a listening session (i.e., skipping / listening to tracks, keyword guidance) (e.g., by using the entire session history to that point). The goal of the binary classifier is to give high scores to ideal tracks and low scores to other tracks. In some embodiments, the classifier can be applied to re-prioritize the remaining tracks of a pre-generated playlist, or to mine new playlist candidates from a larger set.

[0064] For example, at time , all tracks in the initial playlist product 232 can be considered unlabeled with respect to the user's interest in the session. It can be assumed that each track in the playlist is either skipped or listened to, as determined by an appropriate heuristic (e.g., at least 50% of the tracks are played back to meet the listening criteria). Listened tracks can be associated with positive labels, while skipped tracks can be associated with negative labels. In addition, natural language tags describing the user's interests can be considered as additional positive examples.

[0065] Some embodiments relate to receiving user preferences associated with an initial track of an initial playlist during a listening session, wherein the user preferences indicate a listening mood of the user during the listening session, and wherein the user preferences include one or more of a user behavior on the initial track or a natural language input associated with the initial track. User preferences (e.g., user input 238) may generally refer to any user input indicating a preference for music in the current playlist. For example, a track may be skipped, which may be a negative example. Moreover, for example, the play time of the track may be less than a threshold amount of time, thereby indicating that the track is actually skipped. This user behavior may also be marked as a negative example. However, when the user listens to the track, this may be marked as a positive example. Moreover, for example, any text entered by the user may be marked as a positive example. Tags may also be associated with user preferences for like buttons 218 or dislike buttons 220. In some embodiments, the user may reorder the playlist 232, and the reordering may be used to determine the weight in the ranking score to be output by the ML model 246.

[0066] Adaptive playlist generation 200B may receive user input 238. An audio-text joint embedding 240 may be generated. For example, given a joint audio-text embedding model, the track audio and natural language guidance may be embedded into a compatible space. Thus, at each point T in the listening session, an audio-text joint embedding of the form A collection of labeled examples, where each is the embedding of the ith user input (e.g., skip / listen for audio input or text for natural language input), is a label that can be set to 1 for all natural language inputs and majority-played tracks and set to 0 for early-skipped tracks, and 1 in the case where the i-th example is an audio embedding, and 0 in the case where the i-th example is a text embedding. For example, user input 238 can be a combination of a skip track audio embedding, a listen track audio embedding, or a text embedding of a keyword entered by the user. In some embodiments, the user behavior for the initial track includes an indication of whether the user listens to or skips the initial track. Such an embodiment involves assigning a negative label to the initial track if the initial track is skipped, or assigning a positive label to the initial track if the initial track is listened to. For example, each embedding can be associated with a negative label, a positive label, and a positive label, respectively. In some embodiments, keywords entered by the user can be assigned a higher relative weight.

[0067] In some embodiments, a query input box 234 may be provided to enable the user to set user preferences in the form of text input. In some embodiments, the text input is a natural language input. However, the user input may also be a voice command issued by the user. In some embodiments, the text input is a transcription of the user's voice input. For example, the user may be talking to the device, and perhaps in the process of a playlist listening experience, the user may say "make it more likemore rock and roll" or "make it higher energy". Such voice commands can be transcribed into text and used as keywords entered by the user. Entering keywords into a text embedding network (e.g., Figure 1 The text embedding network 120 of the embodiment of the present invention generates a text embedding (e.g., text embedding 130) in the joint embedding space. The user can enter a text string in the query input box 234 to indicate a preference. In some embodiments, a microphone 236 can be provided to enable the user to enter voice instructions into the query input box 234. In some embodiments, the user's voice is transcribed and displayed as text in the query input box 234. Some embodiments involve assigning positive labels to text inputs.

[0068] In some embodiments, the ML model 246 may be retrained from scratch for each session. Generally speaking, the original playlist product is a good neutral playlist that has been designed specifically for the user. Therefore, the ML model 246 must implicitly infer the user's mood for the current session relative to the broader historical profile. Thus, after a few examples, large portions of the playlist may be removed from consideration (e.g., the user's broader historical profile may indicate a preference for jazz or classical music, but the mood of the current session is not jazz or classical music).

[0069] In some embodiments, a large number of popular songs (e.g., 150K) of originally random genres can be used as the original playlist (e.g., from the music repository 242). Such a selection can allow exploration of spaces outside the user's usual comfort zone (e.g., the primary value of audio features relative to co-viewing). Typically, not enough connection with the user's background tastes may require a considerable amount of skip / listen activity in the current session to reduce noise in the quality of recommendations. Therefore, in some embodiments, a large number of songs can be used, and the user's listening history can be used as a prior for the model, and then some labeled examples (e.g., skip / listen activity in the current session) can be used to efficiently generate mood-related examples.

[0070] In some embodiments, the embedding space may be 128-dimensional and the ML model 246 may be a linear classifier that is trained after each skip / listen and may perform inference on the remainder of the playlist followed by a sorting operation. These operations typically require very little computation and can be performed on a device (e.g., a smartphone) with little additional latency.

[0071] In some embodiments, the machine learning model can be a linear classifier trained after receiving the user preferences. In such embodiments, the training of the linear classifier involves training the classifier by loss weighting. For example, adaptive playlist generation 200B can involve using To train the classifier by loss weighting 246 Although a classifier is used for illustration purposes, a more general ML model 246 may be used based on the current session. In some embodiments, the loss weighting may include a loss weighting that depends on each In some embodiments, user behavior on an initial track is associated with a relatively smaller loss weight than text input. For example, a user's skip / listen action can be associated with a smaller weight than a natural language input from the user. In some embodiments, earlier user preferences are associated with a relatively smaller loss weight than more recent user preferences. For example, loss weighting can include a per-example loss weight that depends on a historical position. For example, a recent action can be associated with a larger weight than an earlier action. In some embodiments, a time threshold can be used to identify recent actions.

[0072] Some embodiments involve applying a trained machine learning model to generate an updated playlist. In some embodiments, applying the trained machine learning model includes applying the trained machine learning model to one or more of: the remaining initial tracks in the initial playlist, or the music library. For example, after training, the classifier may be 246 applies to the remaining tracks in the current playlist 244, and / or a broader library, such as the music repository 242. In some embodiments, the music library includes a collection of tracks associated with the user's listening history. For example, the music repository 242 may include tracks previously listened to by the user, a personal library associated with the user, a large co-viewing cluster, and the like.

[0073] In some embodiments, application of the trained machine learning model involves sorting the updated playlist based on the relevance of the tracks to the user's listening mood during the listening session. For example, classifier 246 can sort the updated playlist in descending order of scores, where higher scores indicate greater relevance to the user's mood in the current session. For example, the set of N tracks in the joint embedding space can be collectively referred to as The matrix S, where d is the dimension of the joint embedding space. S can be multiplied with a d-dimensional vector w of weights, such as Sw, which can sort the collection of tracks.

[0074] Some embodiments involve replacing an initial playlist with an updated playlist in an interactive audio playback interface. The next track can be presented to the user from the ordered updated playlist, and user preferences (e.g., the user's skip / listen behavior and / or the user's natural language input) can be recognized. Thus, it can be determined Again, as mentioned above, this can be done by using replace To train the classifier to repeat the process.

[0075] As the process iteratively proceeds, the term "initial playlist" as used herein may refer to the first playlist at the start of a session (e.g., a seed playlist), or may refer to the current playlist during a listening session. For example, an initial playlist at time T may be updated with an updated playlist, which may be the initial playlist at time T+1. Also, for example, the term "initial track" may generally refer to a track in the initial playlist at time T, or a track in an updated playlist, which may be the initial playlist at time T+1. In some embodiments, the initial track may be the track currently being played.

[0076] In some embodiments, the machine learning model can be a nearest neighbor retrieval model. Such an embodiment also relates to applying the nearest neighbor retrieval model in the joint audio-text embedding space to generate an updated playlist, which includes one or more tracks close to the representation of the user's preference. For example, the user can provide natural language input "chill folk music (calm folk music)" or "high energy rock music (high energy rock music)" in the query input box 234. In the initial stage of the current session, there may not be enough labeled examples to train the classifier 246. Therefore, the input text can be embedded in the audio-text joint embedding 240, and one or more tracks can be identified based on the nearest neighbor search in the audio-text joint embedding 240. When playing these tracks, additional input can be received from the user, and then this can enable the training of the classifier 246. Therefore, the initial text input can be regarded as a positive example, the track based on the nearest neighbor search can be played, additional positive examples and negative examples can be received, the classifier 246 can be trained based on the additional examples, and the adaptive playlist 250 can be output based on the score generated by the classifier 246. Thus, after generating a threshold number of positive and negative examples, a smooth transition can be made from a nearest neighbor model-based playlist to a playlist based on the classifier 246. As the listening session progresses, the number N of labeled examples can increase, which can improve the quality of the classifier 246 and, after re-prioritization, also improve the inference of the user's listening mood.

[0077] In some embodiments, the user may be in a session for a long time, and the user may provide multiple positive examples and negative examples during the session. At some point, the user may want to change from rock music to jazz music, and a voice command "switch to jazz" may be provided. Therefore, the classifier 246 may be iteratively trained as described to slowly turn from rock music to jazz music. However, considering a large number of labeled examples related to rock music in the current session, the classifier 246 may continue to provide some rock music tracks until a sufficient number of labeled examples related to jazz music are received. Another strategy may be to assign a larger weight to the text input, indicating a change in genre from rock to jazz. Based on the significantly larger weight of the positive example related to the text input, the classifier 246 may be trained to adapt to the new genre more quickly, and fewer tracks of the rock genre are provided.

[0078] Although the above process has been described using an adaptive classifier to prioritize a single user session, long-term listening history can also be used to define additional training examples. For example, a per-sample loss weighting that reflects the time that has passed since the example was collected can be used. For example, a track that was skipped some time ago (e.g., a month ago) may be associated with a lower contribution than a track that was skipped more recently (e.g., 10 minutes ago). Another metric might be a score for bad views (e.g., a given video was viewed for less than k seconds), where the user's previous views of the video were also viewed for less than k seconds.

[0079] Also, for example, the length of the current session can indicate the type of ML model 246 to use. For example, an initial simple linear classifier can be replaced with a more complex classifier based on the length of the session and / or the amount of labeled data received. In some embodiments, a neural network can be used to determine the playlist. In some embodiments, the machine learning model is a neural network.

[0080] For example, the behavioral data of each session can be used as long-term label data, and more complex neural networks (such as recurrent neural networks (RNNs) and long short-term memory (LSTM) networks) can be applied to recognize the nuances of user moods and generate more relevant music. In some embodiments, a single session can be associated with multiple machine learning models that determine adaptive playlists, and the models can evolve over time. Figure 2 As shown, the ML model 246 can represent multiple models. For example, the ML model 246 can represent an initial nearest neighbor retrieval model, followed by a simple classifier model that can be subsequently replaced with a more complex classifier (e.g., a complex linear classifier, a nonlinear classifier, etc.). In some embodiments, when the length of the listening session lasts for more than a certain time threshold, and / or when the number of user preferences during the current session exceeds a threshold number, the ML model 246 can be a more complex neural network (e.g., RNN, LSTM). In general, the selection of the ML model 246 can depend on many factors, including, for example, depending on the number and type of user preferences during the current session, multiple changes in broad music categories (e.g., genres, singers, periods, languages, etc.), the length of the session, ML models used during previous listening sessions, and the like.

[0081] Some embodiments involve identifying a second listening session that is different from the listening session. Such embodiments also involve receiving a second user preference about a second initial playlist during the second listening session. The training of the machine learning model can be based on the second user preference. The machine learning model can be trained to generate a second updated playlist associated with an updated listening mood of the user during the second listening session. For example, the ML model 246 can be reinitialized during a new listening session to identify the user's listening mood and provide an adaptive playlist customized according to the different mood in the new listening session.

[0082] In some embodiments, the ML model 246 may include a first model based on the mood in the current session, and a second background model that is a slow background model of the user's overall preferences. Tracks that are rejected based on the current listening mood may be weakly fed back into the slower, emotionless second background model. Thus, if the user repeatedly skips a previously recommended track, the first model may learn to skip that track. However, the second model slowly offers tracks over time because there may be tracks that the user may skip in the current session with the current mood, but may not skip in a future session with a different mood. In some embodiments, the hybrid ranker scores generated after the initial playlist may be combined with the scores generated within the current session. A simple score fusion is applied between . If the hybrid ranker score reflects a longer-term user history, re-ranking can be performed by combining the two scores, where the combination weight can depend on T (i.e., the amount of evidence accumulated in the current session).

[0083] experiment

[0084] The adaptive playlist model can be evaluated using the Resnet-50 audio encoder (M-Resnet-50) and the AST audio encoder (M-AST). In both cases, the BERT-base-uncased architecture can be used as the text encoder. In some embodiments, the model can be trained for 14 epochs on a set of audio-text pairs mined from 44M music recordings and processed text labels in all the following categories: AudioSet (ASET), Short Form Tags (SF), Long Form Sentences (LF), Playlist Information (PL). An Adam optimizer with weight decay regularization can be used, and a step decay learning rate schedule with a decay factor of 0.9 applied every 40K steps, and an initial value of for M-Resnet-50 is , and the initial value for M-AST is For all models, the temperature parameter can be initialized as M-Resnet-50 can be trained with a batch size of B = 6144 pairs, while M-AST can use B = 5120 pairs (e.g., due to memory limitations). Since M-AST and M-Resnet-50 show roughly similar performance on the considered evaluation tasks, M-Resnet-50 can be used throughout the text ablation study to obtain better training efficiency.

[0085] Evaluate

[0086] The approach can be evaluated by pre-training joint music-text embedding models on a large-scale dataset of (audio, text) pairs, and then evaluating their utility on several types of downstream tasks, which are described in turn below.

[0087] i. Zero-shot music labeling

[0088] Given a music clip and a set of candidate text tags, each prediction score can be defined as the cosine similarity between the audio embedding of the music clip and the text embedding of each tag string. The generalization ability of the proposed method to potentially unseen target labels is achieved by: (i) using a contextual text encoder that provides a flexible prediction space and (ii) anchoring language semantics to audio representations using cross-modal contrastive learning.

[0089] Evaluation can be performed on two music tagging benchmarks: MagnaTagATune (MTAT) and the music-related part of AudioSet. For MagnaTagATune, the well-practiced set of top 50 tags can be used as well as the full set of 188 tags. The standard train / validation / test partition can be used (note that zero-shot experiments do not use train / validation). The class-balanced area under the receiver operating characteristic curve (AUC-ROC) on the test set can be obtained. The audio clips in MagnaTagATune are 29 seconds long, so they can each be split into three non-overlapping 10-second segments and the segment-level embeddings can be averaged to obtain clip-level embeddings. For AudioSet, one can consider a 25-way genre tagging task (Gen-25) or a richer 141-way tagging task (Mu-141) that includes the entire music subtree of the AudioSet ontology. In both cases, the larger target tag vocabulary enables measuring generalization to a more diverse set of semantic concepts.

[0090] In general, AudioSet is included in the contrastive training, and a small fraction of MTAT classes overlap with the AudioSet ontology. Therefore, from a label exposure perspective, AudioSet and, to a lesser extent, MTAT evaluation may not be strictly zero-shot. However, during playlist generation training, explicit, matching AudioSet supervision may be weakened by a large amount of free-form language supervision. Therefore, by comparing the adaptive playlist model and a traditional AudioSet classifier, we can measure the cost of moving to a flexible natural language interface that also supports classes outside the AudioSet ontology.

[0091] ii. Transfer Learning Using Linear Probes

[0092] In addition to the zero-shot experiments presented above, the audio encoder can also be evaluated as a general-purpose feature extractor for downstream tagging tasks. Two benchmarks, MagnaTagATune and AudioSet, are available, and independent per-class logistic regression layers can be trained on top of frozen 128-dimensional audio embeddings using the training datasets. Directly comparing performance is achieved using the same evaluation protocol as in past transfer learning studies using these datasets.

[0093] iii. Retrieving music from text queries

[0094] Given a music search collection and a text query, playlist generation provides the ability to retrieve the music clips that are closest to the query in the embedding space. This evaluation can be relevant to music retrieval applications, where content features can provide more fine-grained and complete similarity information when compared to metadata-based methods. A proprietary collection of 7,000 expert-curated playlists can be considered, which do not overlap with the playlist information used in training. Each expert-curated playlist has a title and description and consists of 10-100 music recordings. Playlist titles are typically phrases, including a mixture of genres, subgenres, moods, activities, artist names, and compositional elements (e.g., "Indie Pop Fitness", "Easy Korean Pop"). The playlist description consists of one or more complete sentences (see the pos / neg entries in the "Playlist" row of Table 3 below for examples). The playlist evaluation can include approximately 100 K unique recordings.

[0095] Two cross-modal retrieval evaluation sets can be constructed from expert-curated playlist data, one using titles as queries and the other using descriptions. For each dataset, recordings belonging to the corresponding playlist can be used as ground-truth retrieval targets and all 100K recordings can be used as candidate pools. AUC-ROC and mean average precision (mAP) can be reported. The same embedding averaging and cosine similarity-based scoring mechanism as in the zero-shot labeling case can be used. However, the playlist information is inherently very different from the tags involved in the music tagging benchmark. Instead of having a small vocabulary of major basic genres and instruments, playlist titles and descriptions have more fine-grained information, similar to queries presented to music search engines.

[0096] iv. Text triple classification

[0097] Compared to the traditional pre-trained BERT model, the text encoder is fine-tuned using in-domain music data and a cross-modal contrastive loss. In general, there is no text-only training objective. In order to measure whether the proposed method deepens the text encoder's understanding of music-related text, the text embedding can be directly evaluated with a triple classification task. Each triplet consists of three text strings of the form (anchor, pos, neg), and is considered correct if pos is closer to anchor than neg in the text embedding space. Two such text triplet evaluation sets can be evaluated. The first one uses the AudioSet ontology: for each of the 141 music-related classes, 5 triplets can be constructed using the label string as anchor text, the long-form description as positive text, and the long-form descriptions of 5 random classes as negative text. Table 3 shows an example of such a triplet.

[0098] For the second set, 400 triplets can be sampled from the expert-curated playlist data in a similar way: one playlist can be sampled, an anchor can be set, and the positive text can be considered as the title and description respectively, and then the negative text is set to the description of another randomly sampled playlist. Table 3 shows examples for both sets. Table 3 below shows examples of such text triplets.

[0099]

[0100] Table 3

[0101] v. Musical notation

[0102] The music tagging results reported in AUC-ROC are shown in Table 4 below. Table 4 shows the zero-shot tagging metrics where M-Resnet-50 and M-AST achieve comparable performance.

[0103]

[0104] Table 4

[0105] In some embodiments, there may be a significant misalignment between the word meanings of the labels in the label evaluation and the word meanings of the labels in the training text. This can lead to degraded performance relative to an explicitly supervised linear probe setting, in which the semantics of the labels expected by the task can be learned. The MTAT gap is significantly larger than that for AudioSet because (i) MTAT labels with non-specific or multiple meanings, such as "weird" and "beats", and (ii) MTAT labels involving simple negations (e.g., "not rock", "no piano") perform particularly poorly. This may be due to the text encoder not adequately modeling the meaning of these negated concepts, which is a well-known problem with BERT (e.g., the text embedding of "not rock" is similar to "rock", and performance suffers).

[0106] Table 5 below shows the results of a text ablation study that aims to understand the benefits of text labels from different sources.

[0107]

[0108] Table 5

[0109] In some embodiments, training with AudioSet alone achieves the highest AUC in AudioSet evaluation, where the text encoder learns the precise label semantics reflected in the test data. On the other hand, including more data sources generally improves performance on all other downstream tasks (MTAT, retrieval / text triplet evaluation in Tables 5 and 6), and the loss on AudioSet AUC appears to be relatively small.

[0110] For the music tagging task considered, training with unfiltered data appears to achieve comparable performance to the filtered version. The model appears to learn similarly useful associations without being overwhelmed by the large amount of noise in the original text data. The text filtering used may be too aggressive, removing annotations that are not clearly related to the music but are semantically important. Since contrastive learning is highly resistant to noise, the gains from limiting the use of more strongly aligned audio-text pairs may be offset by the loss of a large number of additional useful pairs.

[0111] In Table 5, the adaptive playlist model is evaluated on the query retrieval evaluation task (including using text / filter ablations), where queries are constructed using expert-curated playlist titles and descriptions. Despite using a BERT checkpoint pre-trained on a large language resource as a starting point, training the adaptive playlist model using only AudioSet clips and label annotations has very limited ability to embed music in the domain natural language. This limited cross-modal supervision may not generalize to the rich semantics present in playlist titles and descriptions, which are more consistent with the complex queries presented to real-world music search engines. Significant gains are observed after including large-scale short-form tags mined from the Internet, which help the model learn to support more fine-grained musical concepts. Additional gains may exist when including review and playlist data, where full sentences help support more complex queries, including multi-term queries (e.g., “instrumental action movie dubs”), composed queries (e.g., “classical music with Middle Eastern influences”), and even queries with negation (e.g., “hard rock without vocals”). Training appears to be highly robust to annotation noise, with similar performance obtained using unfiltered training text.

[0112] The text query music retrieval evaluation results are shown in Table 6 below. For example, the text triple classification accuracy AudioSet ontology evaluation and playlist title to description evaluation results are shown. The text ablation / unfiltered model uses M-Resnet-50.

[0113]

[0114] Table 6

[0115] Table 6 shows that when applying linear probes to the adaptive playlist model audio embeddings, SOTA transfer learning performance can be achieved on the labeling tasks. This shows that the pre-trained audio encoder of the adaptive playlist model continues to produce high-quality general music audio embeddings, while also enabling new natural language applications. End-to-end trained baselines for three of these tasks are shown. The linear probe results exceed 2 / 3 and are only slightly behind the SOTA AST AudioSet classifier.

[0116] The Adaptive Playlist Model text embeddings can be evaluated against the following baselines: SentenceTransformer, SimCSE, Universal Sentence Embeddings, and Average Token Embeddings of BERT-base-uncased. All baselines are Transformer-based models of similar size to the Adaptive Playlist Model described in this paper. The first three are trained with sentence-level contrastive losses, while BERT is trained with masked language prediction. The Adaptive Playlist Model text encoder can be warm-started using this same BERT baseline, but may then be trained only with cross-modal losses. The resulting text embedding models, now specialized for the music domain, appear to outperform the universal sentence embedding models when long-form text annotations are included. Thus, successful specialization can be achieved without using any text-only fine-tuning losses.

[0117] Train machine learning models to generate inferences / predictions

[0118] Figure 3 A diagram 300 is shown showing a training phase 302 and an inference phase 304 of a trained machine learning model 332, according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input training data set to identify patterns in the training data and provide output inferences and / or predictions about (the patterns in) the training data. The resulting trained machine learning algorithm may be referred to as a trained machine learning model. For example, Figure 3 A training phase 302 is shown, wherein one or more machine learning algorithms 320 are being trained on training data 310 to become a trained machine learning model 332. Then, during an inference phase 304, the trained machine learning model 332 may receive input data 330 and one or more inference / prediction requests 340 (perhaps as part of the input data 330), and responsively provide one or more inferences and / or predictions 350 as output.

[0119] Thus, the trained machine learning model 332 may include one or more models of one or more machine learning algorithms 320. The machine learning algorithm 320 may include, but is not limited to, an artificial neural network (e.g., a convolutional neural network, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system as described herein). The machine learning algorithm 320 may be supervised or unsupervised, and may implement any suitable combination of online learning and offline learning.

[0120] In some examples, the machine learning algorithm 320 and / or the trained machine learning model 332 can be accelerated using an on-device coprocessor, such as a graphics processing unit (GPU), a tensor processing unit (TPU), a digital signal processor (DSP), and / or an application-specific integrated circuit (ASIC). Such an on-device coprocessor can be used to accelerate the machine learning algorithm 320 and / or the trained machine learning model 332. In some examples, the trained machine learning model 332 can be trained, can reside and be executed to provide reasoning on a specific computing device, and / or can otherwise make reasoning for a specific computing device.

[0121] During the training phase 302, the machine learning algorithm 320 may be trained using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques by providing at least the training data 310 as training input. Unsupervised learning involves providing a portion (or all) of the training data 310 to the machine learning algorithm 320, and the machine learning algorithm 320 determining one or more output inferences based on the portion (or all) of the provided training data 310. Supervised learning involves providing a portion of the training data 310 to the machine learning algorithm 320, wherein the machine learning algorithm 320 determines one or more output inferences based on the portion of the provided training data 310, and accepting or correcting the output inferences based on correct results associated with the training data 310. In some examples, supervised learning of the machine learning algorithm 320 may be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning algorithm 320.

[0122] Semi-supervised learning involves obtaining the correct result for a portion (but not all) of the training data 310. During semi-supervised learning, supervised learning is used for a portion of the training data 310 that has the correct result, and unsupervised learning is used for a portion of the training data 310 that does not have the correct result. Reinforcement learning involves the machine learning algorithm 320 receiving a reward signal about a previous reasoning, where the reward signal can be a numerical value. During reinforcement learning, the machine learning algorithm 320 can output an inference and receive a reward signal in response, where the machine learning algorithm 320 is configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time. In some examples, the machine learning algorithm 320 and / or the trained machine learning model 332 can be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0123] In some examples, the machine learning algorithm 320 and / or the trained machine learning model 332 may use transfer learning techniques. For example, the transfer learning technique may involve the trained machine learning model 332 being pre-trained on a data set and additionally trained using the training data 310. More specifically, the machine learning algorithm 320 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model is provided to the computing device CD1, where CD1 intends to execute the trained machine learning model during the inference phase 304. Then, during the training phase 302, the pre-trained machine learning model may be additionally trained using the training data 310, where the training data 310 may be derived from the kernel data and non-kernel data of the computing device CD1. Such further training of the machine learning algorithm 320 and / or the pre-trained machine learning model using the training data 310 of the CD1 data may be performed using supervised learning or unsupervised learning. Once the machine learning algorithm 320 and / or the pre-trained machine learning model has been trained at least on the training data 310, the training phase 302 may be completed. The trained resulting machine learning model may be used as at least one of the trained machine learning models 332.

[0124] In particular, once the training phase 302 has been completed, the trained machine learning model 332 can be provided to the computing device (if not already on the computing device). After the trained machine learning model 332 is provided to the computing device CD1, the inference phase 304 can begin.

[0125] During the inference phase 304, the trained machine learning model 332 may receive input data 330 and generate and output one or more corresponding inferences and / or predictions 350 about the input data 330. Thus, the input data 330 may be used as an input to the trained machine learning model 332 for providing corresponding inferences and / or predictions 350 to kernel components and non-kernel components. For example, the trained machine learning model 332 may generate inferences and / or predictions 350 in response to one or more inference / prediction requests 340. In some examples, the trained machine learning model 332 may be executed by a portion of other software. For example, the trained machine learning model 332 may be executed by an inference or prediction daemon so that it is readily available upon request for providing inferences and / or predictions. The input data 330 may include data from the computing device CD1 executing the trained machine learning model 332 and / or input data from one or more computing devices other than CD1.

[0126] Input data 330 may include training data as described herein, such as user preference data with the described interface, including user data from multiple users, devices, platforms, inputs, etc. Other types of input data are also possible. For example, the training data may include data collected for training a dual-tower joint embedding network.

[0127] Inferences and / or predictions 350 may include task outputs, values, and / or other output data produced by trained machine learning model 332 operating on input data 330 (and training data 310). In some examples, trained machine learning model 332 may use output inferences and / or predictions 350 as input feedback 360. Trained machine learning model 332 may also rely on past inferences as input for generating new inferences.

[0128] After training, the trained version of the neural network can be an example of the trained machine learning model 332. In this approach, an example of one or more inference / prediction requests 340 can be a request to predict an updated playlist related to the user's mood in the current listening session, and a corresponding example of the inference and / or prediction 350 can be the predicted updated playlist. Another example of one or more inference / prediction requests 340 can be a request to predict a joint embedding based on the user's preferences in the current listening session, and a corresponding example of the inference and / or prediction 350 can be the predicted joint embedding.

[0129] In some examples, a computing device CD_SOLO may include a trained version of the neural network, perhaps after training. The computing device CD_SOLO may then receive a request for an updated playlist relevant to the user's mood in the current listening session, and use the trained version of the neural network to predict an updated playlist relevant to the user's mood in the current listening session.

[0130] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide output; for example, a first computing device CD_CLI may generate a request to predict an updated playlist relevant to the user's mood in the current listening session and send the request to a second computing device CD_SRV. CD_SRV may then use a trained version of a neural network to predict an updated playlist relevant to the user's mood in the current listening session and respond to the request from CD_CLI. Upon receiving a response to the request, CD_CLI may then provide the requested output (e.g., using a user interface and / or display, a printed copy, an electronic communication, etc.).

[0131] Sample Data Network

[0132] Figure 4 A distributed computing architecture 400 is depicted according to an example embodiment. The distributed computing architecture 400 includes server devices 408, 410 configured to communicate with programmable devices 404a, 404b, 404c, 404d, 404e via a network 406. The network 406 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, an intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 406 may also correspond to a combination of one or more LANs, WANs, intranets, and / or the public Internet.

[0133] although Figure 4 Only five programmable devices are shown, but the distributed application architecture can serve tens, hundreds, or thousands of programmable devices. In addition, programmable devices 404a, 404b, 404c, 404d, 404e (or any additional programmable devices) can be any kind of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted device (HMD), a network terminal, a mobile computing device, and the like. In some examples, such as shown in programmable devices 404a, 404b, 404c, 404e, the programmable device can be directly connected to the network 406. In other examples, such as shown in programmable device 404d, the programmable device can be indirectly connected to the network 406 via an associated computing device (such as programmable device 404c). In this example, programmable device 404c can act as an associated computing device for transmitting electronic communications between programmable device 404d and network 406. In other examples, such as shown in programmable device 404e, the computing device can be part of a vehicle (such as a car, truck, bus, boat or ship, airplane, etc.) and / or located inside the vehicle. In Figure 4 In other examples not shown, the programmable device can be connected to the network 406 both directly and indirectly.

[0134] The server devices 408, 410 may be configured to perform one or more services as requested by the programmable devices 404a-404e. For example, the server devices 408 and / or 410 may provide content to the programmable devices 404a-404e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data (such as compiled software), images, audio, and / or video. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0135] As another example, server devices 408 and / or 410 may provide programmable devices 404a-404e with access to software for database, search, computing, graphics, audio, video, web / internet utilization, and / or other functions. Many other examples of server devices are possible.

[0136] Computing device architecture

[0137] Figure 5 is a block diagram of an example computing device 500 according to an example embodiment. Specifically, Figure 5 The illustrated computing device 500 may be configured to perform at least one function of and / or related to the neural network 100 and / or the method 500 .

[0138] The computing device 500 may include a user interface module 501, a network communication module 502, one or more processors 503, a data storage device 504, one or more cameras 518, one or more sensors 520 and a power system 522, all of which may be linked together via a system bus, network or other connection mechanism 505.

[0139] The user interface module 501 may be operable to send data to and / or receive data from an external user input / output device. For example, the user interface module 501 may be configured to send data to and / or receive data from a user input device such as a touch screen, a computer mouse, a keyboard, a keypad, a touch pad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 501 may also be configured to provide output to a user display device such as one or more cathode ray tubes (CRTs), liquid crystal displays, light emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, whether known now or developed later. The user interface module 501 may also be configured to generate audible output using devices such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. The user interface module 501 may further be configured with one or more tactile devices that may generate tactile outputs such as vibrations and / or other outputs that may be detected by touch and / or physical contact with the computing device 500. In some examples, user interface module 501 may be used to provide a graphical user interface (GUI) for utilizing computing device 500 , such as, for example, a graphical user interface of a mobile telephone device.

[0140] The network communication module 502 may include one or more devices that provide one or more wireless interfaces 507 and / or one or more wired interfaces 508 that can be configured to communicate via a network. The wireless interface 507 may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, a LTE™ transceiver, and / or other types of wireless transceivers that can be configured to communicate via a wireless network. The wired interface 508 may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a universal serial bus (USB) transceiver, or a similar transceiver that can be configured to communicate via a twisted pair, a coaxial cable, a fiber optic link, or a similar physical connection to a wired network.

[0141] In some examples, the network communication module 502 can be configured to provide reliable, protected and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., ensuring message delivery) can be provided, which may be part of a message header and / or a message trailer (e.g., packet / message ordering information, encapsulation header and / or encapsulation trailer, size / time information, and transmission verification information (such as a cyclic redundancy check (CRC) and / or a parity check value)). One or more encryption protocols and / or algorithms can be used to protect (e.g., encode or encrypt) and / or decrypt / decode communications, such as, but not limited to, the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, the Diffie-Hellman algorithm, a secure socket protocol (such as, a secure socket layer (SSL) or a transport layer security (TLS)), and / or a digital signature algorithm (DSA). Other cryptographic protocols and / or algorithms besides those listed herein may be used to protect (and then decrypt / decode) communications.

[0142] The one or more processors 503 may include one or more general purpose processors and / or one or more special purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 503 may be configured to execute computer readable instructions 506 contained in the data storage device 504 and / or other instructions as described herein.

[0143] The data storage device 504 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors 503. The one or more computer-readable storage media may include volatile and / or non-volatile storage components that may be fully or partially integrated with at least one of the one or more processors 503, such as optical, magnetic, organic or other memory or disk storage devices. In some examples, the data storage device 504 may be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disk storage unit), while in other examples, the data storage device 504 may be implemented using two or more physical devices.

[0144] Data storage 504 may include computer-readable instructions 506 and possibly additional data. In some examples, data storage 504 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, data storage 504 may include storage for trained neural network model 512 (e.g., a model of a trained neural network such as neural network 100). Specifically, in these examples, computer-readable instructions 506 may include instructions that, when executed by one or more processors 503, enable computing device 500 to provide some or all of the functionality of trained neural network model 512.

[0145] In some examples, computing device 500 may include one or more cameras 518. Cameras 518 may include one or more image capture devices, such as still cameras and / or video cameras, that are configured to capture light and record the captured light in one or more images; that is, cameras 518 may generate images of the captured light. The one or more images may be one or more still images and / or one or more images used in video imagery. Cameras 518 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0146] In some examples, computing device 500 may include one or more sensors 520. Sensors 520 may be configured to measure conditions within computing device 500 and / or conditions in the environment of computing device 500 and provide data about these conditions. For example, sensors 520 may include one or more of the following: (i) sensors for obtaining data about computing device 500, such as, but not limited to, a thermometer for measuring the temperature of computing device 500, a battery sensor for measuring the charge of one or more batteries of power system 522, and / or other sensors for measuring conditions of computing device 500; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, where identification sensors may be configured to read identifiers (such as RFID tags, barcodes, QR codes) and / or other devices and / or objects that are configured to be read and provide at least identification information; (iii) sensors for measuring the location and (iv) an environmental sensor for obtaining data indicative of the environment of computing device 500, such as, but not limited to, an infrared sensor, an optical sensor, a light sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) a force sensor for measuring one or more forces (e.g., inertial forces and / or gravity) acting on computing device 500, such as, but not limited to, one or more sensors measuring force, torque, ground force, friction in one or more dimensions, and / or a ZMP sensor that identifies a zero moment point (ZMP) and / or the location of the ZMP. Many other examples of sensors 520 are also possible.

[0147] The power system 522 may include one or more batteries 524 and / or one or more external power interfaces 526 for providing power to the computing device 500. Each of the one or more batteries 524 may act as a storage power source for the computing device 500 when electrically coupled to the computing device 500. The one or more batteries 524 of the power system 522 may be configured to be portable. Some or all of the one or more batteries 524 may be easily removable from the computing device 500. In other examples, some or all of the one or more batteries 524 may be located inside the computing device 500 and may therefore not be easily removable from the computing device 500. Some or all of the one or more batteries 524 may be rechargeable. For example, a rechargeable battery may be charged via a wired connection between the battery and another power supply device (such as by being located outside the computing device 500 and connected to the computing device 500 via one or more external power interfaces). In other examples, some or all of the one or more batteries 524 may be non-rechargeable batteries.

[0148] The one or more external power interfaces 526 of the power system 522 may include one or more wired power interfaces, such as USB cables and / or power cords, that implement a wired power connection to one or more power supply devices located outside the computing device 500. The one or more external power interfaces 526 may include one or more wireless power interfaces, such as Qi wireless chargers, that implement wireless power connections to one or more external power supply devices. Once a power connection to an external power source is established using the one or more external power interfaces 526, the computing device 500 can draw power from the external power source through the established power connection. In some examples, the power system 522 may include associated sensors, such as battery sensors or other types of power sensors associated with one or more batteries.

[0149] Cloud-based servers

[0150] Figure 6 A cloud-based server system is depicted according to an example embodiment. Figure 6, the functionality of the neural network and / or computing devices may be distributed among computing clusters 609a, 609b, 609c. Computing cluster 609a may include one or more computing devices 600a, cluster storage array 610a, and cluster router 611a connected via a local cluster network 612a. Similarly, computing cluster 609b may include one or more computing devices 600b, cluster storage array 610b, and cluster router 611b connected via a local cluster network 612b. Likewise, computing cluster 609c may include one or more computing devices 600c, cluster storage array 610c, and cluster router 611c connected via a local cluster network 612c.

[0151] In some embodiments, computing clusters 609a, 609b, 609c may be a single computing device residing in a single computing center. In other embodiments, computing clusters 609a, 609b, 609c may include multiple computing devices in a single computing center, or even multiple computing devices located in multiple computing centers located in different geographic locations. For example, Figure 6 Each of the computing clusters 609a, 609b, 609c is depicted as residing in a different physical location.

[0152] In some embodiments, the data and services at the computing clusters 609a, 609b, 609c may be encoded as computer-readable information stored in a non-transitory tangible computer-readable medium (or computer-readable storage medium) and accessible by other computing devices. In some embodiments, the computing clusters 609a, 609b, 609c may be stored on a single disk drive or other tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located at one or more different geographic locations.

[0153] In some embodiments, each of computing clusters 609a, 609b, and 609c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster may depend on one or more computing tasks assigned to each computing cluster.

[0154] For example, in computing cluster 609a, computing device 600a can be configured to perform various computing tasks of a conditional, axial self-attention-based neural network and / or computing device. In one embodiment, various functionalities of the neural network and / or computing device can be distributed between one or more of computing devices 600a, 600b, 600c. The computing devices 600b and 600c in the respective computing clusters 609b and 609c can be configured similarly to the computing device 600a in computing cluster 609a. On the other hand, in some embodiments, computing devices 600a, 600b, and 600c can be configured to perform different functions.

[0155] In some embodiments, computing tasks and stored data associated with the neural network and / or computing devices may be distributed across computing devices 600a, 600b, and 600c based at least in part on processing requirements of the neural network and / or computing devices, processing capabilities of computing devices 600a, 600b, 600c, latency of network links between computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that may impact cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the overall system architecture.

[0156] The cluster storage arrays 610a, 610b, 610c of the computing clusters 609a, 609b, 609c may be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, may also be configured to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0157] Similar to the way that the functionality of the conditional, axial self-attention-based neural network and / or computing devices can be distributed across the computing devices 600a, 600b, 600c of the computing clusters 609a, 609b, 609c, the various active and / or backup portions of these components can be distributed across the cluster storage arrays 610a, 610b, 610c. For example, some cluster storage arrays can be configured to store a first layer of the neural network and / or a portion of the data of the computing device, while other cluster storage arrays can store a second layer of the neural network and / or other portions of the data of the computing device. Also, for example, some cluster storage arrays can be configured to store data for an encoder of a neural network, while other cluster storage arrays can store data for a decoder of a convolutional neural network. Additionally, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0158] The cluster routers 611a, 611b, 611c in the computing clusters 609a, 609b, 609c may include networking equipment configured to provide internal and external communications for the computing clusters. For example, the cluster router 611a in the computing cluster 609a may include one or more Internet switching and routing devices configured to (i) provide local area network communications between the computing device 600a and the cluster storage array 610a via the local cluster network 612a, and (ii) provide wide area network communications between the computing cluster 609a and the computing clusters 609b and 609c via the wide area network link 613a to the network 406. The cluster routers 611b and 611c may include similar networking equipment as the cluster router 611a, and the cluster routers 611b and 611c may perform networking functions for the computing clusters 609b and 609b similar to the networking functions performed by the cluster router 611a for the computing cluster 609a.

[0159] In some embodiments, the configuration of the cluster routers 611a, 611b, 611c can be based at least in part on the data communication requirements of the computing devices and the cluster storage arrays, the data communication capabilities of the network equipment in the cluster routers 611a, 611b, 611c, the latency and throughput of the local cluster network 612a, 612b, 612c, the latency, throughput and cost of the wide area network links 613a, 613b, 613c, and / or other factors that may affect the cost, speed, fault tolerance, resilience, efficiency and / or other design criteria of the adjusted system architecture.

[0160] Example methods of operation

[0161] Figure 7 is a flow chart of a method 700 according to an example embodiment. The method 700 may be performed by a computing device (such as computing device 500).

[0162] Method 700 may begin at block 710, where the method involves providing an initial playlist including one or more initial audio tracks via an interactive audio playback interface.

[0163] At block 720, the method involves receiving user preferences associated with an initial audio track of an initial playlist during a listening session, wherein the user preferences indicate a listening mood of a user during the listening session, and wherein the user preferences include one or more of user behavior on the initial audio track or natural language input associated with the initial audio track.

[0164] At block 730, the method involves generating a representation of the user preferences in a joint audio-text embedding space by applying a two-tower model, the two-tower model comprising an audio embedding network for generating an audio embedding of an initial audio track and a text embedding network for generating a text embedding of a natural language input, wherein proximity of two embeddings in the joint audio-text embedding space indicates semantic similarity.

[0165] At block 740, the method involves training a machine learning model based on the representation of the user preferences to generate an updated playlist including one or more updated audio tracks, wherein the one or more updated audio tracks are responsive to a listening mood of the user during the listening session.

[0166] At block 750, the method involves applying the trained machine learning model to generate an updated playlist.

[0167] At block 760, the method involves replacing the initial playlist with the updated playlist in the interactive audio playback interface.

[0168] In some embodiments, the user behavior on the initial track includes an indication of whether the user listened to or skipped the initial track. Such embodiments involve assigning a negative label to the initial track if the initial track is skipped, or assigning a positive label to the initial track if the initial track is listened to.

[0169] Some embodiments relate to assigning positive labels to text input.

[0170] In some embodiments, the natural language input includes text entered by a user.

[0171] In some embodiments, the natural language input is a transcription of the user's speech input.

[0172] In some embodiments, the machine learning model is a linear classifier trained after receiving the user preferences. In such embodiments, the training of the linear classifier involves training the classifier using loss weighting. In some embodiments, user behavior for the initial track is associated with a relatively smaller loss weight than the text input. In some embodiments, earlier user preferences are associated with a relatively smaller loss weight than more recent user preferences.

[0173] In some embodiments, applying the trained machine learning model includes applying the trained machine learning model to one or more of: the remaining initial tracks in the initial playlist, or the music library. In some embodiments, the music library includes a collection of tracks associated with the user's listening history.

[0174] In some embodiments, application of the trained machine learning model involves sorting the updated playlist based on the relevance of the tracks to the user's listening mood during the listening session.

[0175] Some embodiments involve identifying a second listening session that is different from the listening session. Such embodiments also involve receiving second user preferences regarding a second initial playlist during the second listening session. Training of the machine learning model can be based on the second user preferences. The machine learning model can be trained to generate a second updated playlist associated with an updated listening mood of the user during the second listening session.

[0176] In some embodiments, the machine learning model may be a nearest neighbor retrieval model. Such embodiments also involve applying the nearest neighbor retrieval model in the joint audio-text embedding space to generate an updated playlist that includes one or more tracks that are close to the representation of the user's preferences.

[0177] In some embodiments, the machine learning model is a neural network.

[0178] Some embodiments relate to contrastively training an audio embedding network and a text embedding network based on an audio-text contrastive loss. In such embodiments, the audio-text contrastive loss is a cross-modal extension of the information noise contrastive estimation (InfoNCE) loss and the normalized temperature scaled cross entropy (NT-Xent) loss.

[0179] In some embodiments, the audio embedding network comprises a modified Resnet-50 architecture in which the stride 2 in the first convolutional layer is removed.

[0180] In some embodiments, the text embedding network includes a Bidirectional Encoder Transformer (BERT) with a base-uncased architecture.

[0181] The present disclosure is not limited in terms of the specific embodiments described in this application, which are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from its spirit and scope. In addition to those methods and devices described herein, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0182] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the drawings, similar symbols generally identify similar components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized and other changes may be made without departing from the spirit and scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure as generally described herein and shown in the accompanying drawings may be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0183] With respect to any or all ladder diagrams, scenarios, and flow charts discussed in the figures and herein, each block and / or communication may represent information processing and / or information transmission according to an example embodiment. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, the functions described as blocks, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functions involved. In addition, more or fewer blocks and / or functions may be used with any ladder diagram, scenario, and flow chart discussed herein. Figure 1 and these ladder diagrams, scenarios, and flow charts can be combined with each other in part or in whole.

[0184] The block representing the processing of information may correspond to a circuit that can be configured to perform a specific logical function of the method or technology described herein. Alternatively or in addition, the block representing the processing of information may correspond to a module, a fragment, or a portion of a program code (including associated data). The program code may include one or more instructions that may be executed by a processor to implement a specific logical function or action in a method or technology. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a disk or hard drive or other storage medium.

[0185] Computer readable media may also include non-transitory computer readable media, such as non-transitory computer readable media that store data in a short period of time, such as register memory, processor cache, and random access memory (RAM). Computer readable media may also include non-transitory computer readable media that store program code and / or data in a longer period of time, such as auxiliary or persistent long-term storage devices, for example, such as read-only memory (ROM), optical disks or magnetic disks, compact disk read-only memory (CD-ROM). Computer readable media may also be any other volatile or non-volatile storage system. Computer readable media may be considered, for example, as a computer readable storage medium or a tangible storage device.

[0186] In addition, the blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be between software modules and / or hardware modules in different physical devices.

[0187] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for purposes of explanation and are not intended to be limiting, with the true scope being associated with the claims.

Claims

1. A computer-implemented method comprising: providing an initial playlist including one or more initial audio tracks through an interactive audio playback interface; receiving, during a listening session, user preferences associated with an initial audio track of the initial playlist, wherein the user preferences indicate a listening mood of a user during the listening session, and wherein the user preferences include one or more of user behavior with respect to the initial audio track or natural language input associated with the initial audio track; generating a representation of the user preferences in a joint audio-text embedding space by applying a two-tower model, the two-tower model comprising an audio embedding network for generating an audio embedding of the initial audio track and a text embedding network for generating a text embedding of the natural language input, wherein proximity of two embeddings in the joint audio-text embedding space indicates semantic similarity; training a machine learning model for generating an updated playlist including one or more updated audio tracks based on the representation of the user preferences, wherein the one or more updated audio tracks are responsive to the listening mood of the user during the listening session; applying the trained machine learning model to generate the updated playlist; as well as The initial playlist is replaced with the updated playlist in the interactive audio playback interface.

2. The computer-implemented method of claim 1 , wherein the user behavior with respect to the initial audio track comprises an indication of whether the user listened to or skipped the initial audio track, and the method further comprises: The initial track is assigned a negative label if it is skipped, or a positive label if it is listened to.

3. The computer-implemented method of claim 1 , further comprising: Assigns a positive label to the text input.

4. The computer-implemented method of claim 1, wherein the natural language input comprises text entered by the user.

5. The computer-implemented method of claim 1, wherein the natural language input is a transcription of the user's speech input.

6. The computer-implemented method of claim 1, wherein the machine learning model is a linear classifier trained after receiving the user preferences.

7. The computer-implemented method of claim 6, wherein the training of the linear classifier comprises training the classifier with a loss weight.

8. The computer-implemented method of claim 7, wherein the user behavior on the initial audio track is associated with a relatively smaller penalty weight than the text input.

9. The computer-implemented method of claim 7, wherein earlier user preferences are associated with relatively smaller penalty weights than more recent user preferences.

10. The computer-implemented method of claim 1, wherein the applying the trained machine learning model comprises applying the trained machine learning model to one or more of: remaining initial tracks in the initial playlist, or a music library.

11. The computer-implemented method of claim 10, wherein the music library comprises a collection of audio tracks associated with a listening history of the user.

12. The computer-implemented method of claim 1, wherein the applying the trained machine learning model comprises sorting the updated playlist based on relevance of audio tracks to the listening mood of the user during the listening session.

13. The computer-implemented method of claim 1 , further comprising: identifying a second listening session different from the listening session; as well as receiving second user preferences regarding a second initial playlist during the second listening session, and Wherein the training of the machine learning model is based on the second user preferences, and wherein the machine learning model is trained to generate a second updated playlist associated with an updated listening mood of the user during the second listening session.

14. The computer-implemented method of claim 1, wherein the machine learning model is a nearest neighbor retrieval model, and the method further comprises: The nearest neighbor retrieval model is applied in the joint audio-text embedding space to generate the updated playlist, the updated playlist including one or more audio tracks that are close to the representation of the user preference.

15. The computer-implemented method of claim 1, wherein the machine learning model is a neural network.

16. The computer-implemented method of claim 1, further comprising: The audio embedding network and the text embedding network are contrastively trained based on audio-text contrast loss.

17. The computer-implemented method of claim 16, wherein the audio-text contrastive loss is a cross-modal extension of an Information Noise Contrastive Estimation (InfoNCE) loss and a Normalized Temperature Scaled Cross Entropy (NT-Xent) loss.

18. The computer-implemented method of claim 1, wherein the audio embedding network comprises one or more of: (i) a modified Resnet-50 architecture in which the stride 2 in the first convolutional layer is removed, or (ii) an Audio Spectrogram Transformer (AST).

19. The computer-implemented method of claim 1, wherein the text embedding network comprises a Bidirectional Encoder Transformer (BERT) with a base-uncased architecture.

20. A computing device comprising: one or more processors; as well as A data storage device having computer executable instructions stored thereon, the computer executable instructions, when executed by the one or more processors, causing the computing device to perform functions, the functions comprising: providing an initial playlist including one or more initial audio tracks through an interactive audio playback interface; receiving, during a listening session, user preferences associated with an initial audio track of the initial playlist, wherein the user preferences indicate a listening mood of a user during the listening session, and wherein the user preferences include one or more of user behavior with respect to the initial audio track or natural language input associated with the initial audio track; generating a representation of the user preferences in a joint audio-text embedding space by applying a two-tower model, the two-tower model comprising an audio embedding network for generating an audio embedding of the initial audio track and a text embedding network for generating a text embedding of the natural language input, wherein proximity of two embeddings in the joint audio-text embedding space indicates semantic similarity; training a machine learning model for generating an updated playlist including one or more updated audio tracks based on the representation of the user preferences, wherein the one or more updated audio tracks are responsive to the listening mood of the user during the listening session; applying the trained machine learning model to generate the updated playlist; and The initial playlist is replaced with the updated playlist in the interactive audio playback interface.

21. An article of manufacture comprising one or more non-transitory computer-readable media having computer-readable instructions stored thereon, the computer-readable instructions, when executed by one or more processors of a computing device, causing the computing device to perform functions comprising: providing an initial playlist including one or more initial audio tracks through an interactive audio playback interface; receiving, during a listening session, user preferences associated with an initial audio track of the initial playlist, wherein the user preferences indicate a listening mood of a user during the listening session, and wherein the user preferences include one or more of user behavior with respect to the initial audio track or natural language input associated with the initial audio track; generating a representation of the user preferences in a joint audio-text embedding space by applying a two-tower model, the two-tower model comprising an audio embedding network for generating an audio embedding of the initial audio track and a text embedding network for generating a text embedding of the natural language input, wherein proximity of two embeddings in the joint audio-text embedding space indicates semantic similarity; training a machine learning model for generating an updated playlist including one or more updated audio tracks based on the representation of the user preferences, wherein the one or more updated audio tracks are responsive to the listening mood of the user during the listening session; applying the trained machine learning model to generate the updated playlist; as well as The initial playlist is replaced with the updated playlist in the interactive audio playback interface.

Images