Generating a media file using a generative machine learning model

Abstract

A computer-implemented method of generating a media file comprises: receiving a user prompt 20; receiving tailoring information, from a tailoring information database, for tailoring the media file; generating, using an encoder 14 of the generative machine learning model, an embedding based on the user prompt; generating, by a generator 16 of the generative machine learning model, a sequence of media tokens based on the embedding; decoding, by a decoder 18 of the generative machine learning model, the sequence of media tokens to obtain a media file; and outputting the media file 12, wherein at least one of generating an embedding based on the user prompt and generating the sequence of media tokens based on the embedding is also based on the tailoring information. Tailoring information may comprise, situational information 22, e.g., location or environmental information, and contextual information 24, e.g., preference information provided by user prompts or feedback. The media file generated may be a music track. Inputs may be words, images, video, or audio. Determined user engagement with the media file may then be used to further train and fine tune the model.

Description

Field

[001] The present application generally relates to methods for generating media files using generative machine learning models. Background

[002] Generative machine learning models can generate media files, e.g. music tracks, based on user prompts from a different modality, e.g. text prompts. However, such efforts rely on things such as accuracy of the user’s prompts, and often may require multiple iterations to arrive at a media file that is relevant to the user.

[003] The aim of the subject-matter matter of the present disclosure it to improve on the prior art. Summary

[004] According to an aspect of the present disclosure, there is provided a computer-implemented method of generating a media file using a generative machine learning model, the computer-implemented method comprising: receiving a user prompt; receiving tailoring information, from a tailoring information database, for tailoring the media file; generating, using an encoder of the generative machine learning model, an embedding based on the user prompt; generating, by a generator of the generative machine learning model, a sequence of media tokens based on the embedding; decoding, by a decoder of the generative machine learning model, the sequence of media tokens to obtain a media file; and outputting the media file, wherein at least one of generating an embedding based on the user prompt and generating the sequence of media tokens based on the embedding is also based on the tailoring information.

[005] In other words, either generating the embedding based on the user prompt or generating the sequence of media tokens based on the embedding is also based on the tailoring information. In this way, the media file will be better tailored to the user, thus reducing a number of user prompt iterations because the user is happy with the media file.

[006] In an embodiment, the method comprises transmitting the media file across a network that the processor and storage are linked to.

[007] In an embodiment, generating the sequence of media tokens based on the embedding and the tailoring information comprises: generating a user-specific token using the tailoring information; and generating the sequence of media tokens based on the embedding and the user-specific token.

[008] In an embodiment, generating the sequence of media tokens based on the embedding and the embedding token comprises: merging the embedding and the user-specific token; and inputting the merged embedding and user-specific token to a generator of the generative machine learning model.

[009] In an embodiment, generating the sequence of media tokens based on the embedding and the user-specific token comprises: inputting the embedding to a generator of the generative machine learning model; and inputting separately an embedding token to the generator.

[010] In an embodiment, when a media modality of the user prompt is text, generating the embedding based on the user prompt and the tailoring information comprises: obtaining at least one keyword based on the tailoring information; and adding the at least one keyword to the user prompt.

[011] In an embodiment, obtaining at least one keyword based on the tailoring information comprises: inputting the user prompt to a language model; determining a probability that each keyword out of a plurality of keywords is relevant to the user prompt; and selecting the at least one keyword with a highest probability out of the plurality of keywords.

[012] In an embodiment, the computer-implemented method further comprises: collecting the tailoring information in the tailoring information database from a plurality of users of the generative machine learning model.

[013] In an embodiment, collecting the tailoring information from the plurality of users of the generative machine learning model comprises: identifying a plurality of connections, on a social media platform, to a user inputting the user prompt; and storing tailoring information associated with the plurality of connections in the tailoring information database.

[014] In an embodiment, collecting the tailoring information from the plurality of users of the generative machine learning model comprises: comparing a similarity of each new tailoring information to tailoring information stored in the tailoring information database; storing the new tailoring information if the similarity is greater than or equal to a similarity threshold; and disregarding the new tailoring information if the similarity is less than the similarity threshold. Advantageously, trends within the plurality of users, e.g. within a community of users, can be identified.

[015] In an embodiment, collecting the tailoring information from the plurality of users of the generative machine learning model comprises: comparing a similarity of each new tailoring information to tailoring information stored in the tailoring information database; storing the new tailoring information if the similarity is less than or equal to a similarity threshold; and disregarding the new tailoring information if the similarity is greater than the similarity threshold. Advantageously, duplicate tailoring information is not stored. As a result, the storage can be smaller in size.

[016] In an embodiment, when the tailoring information stored in the tailoring information database is text, comparing the similarity of each new tailoring information to tailoring information stored in the tailoring information database comprises: generating, using a pretrained model an encoding of each new tailoring information and of each item of tailoring information in the tailoring information database; calculating a distance between each new tailoring information and each item of tailoring information in the tailoring information database; and comparing the distance to a distance threshold.

[017] In an embodiment, collecting the tailoring information from the plurality of users of the generative machine learning model comprises: comparing a location associated with each new tailoring information to a geographical area; storing the new tailoring information if the location of the new tailoring information is outside the geographical area; and disregarding the new tailoring information if its location is outside the geographical area. Advantageously, the generated media files will be more similar to media trending in an area that is local to the user.

[018] In an embodiment, the tailoring information comprises at least one of contextual awareness information and situational awareness information.

[019] In an embodiment, the contextual awareness information comprises previous user prompts, user information from other applications, and direct feedback on previously generated media files.

[020] In an embodiment, direct feedback comprises a user rating.

[021] In an embodiment, direct feedback comprises a user correction of a token, e.g. an output token.

[022] In an embodiment, the situational awareness information comprises sensor information describing sensor data from a plurality of sensors of different modalities associated with a device used by a user inputting the user prompt.

[023] In an embodiment, the generative machine learning model is a transformer model transforming a first media modality to a second media modality, wherein the second media modality is music.

[024] In an embodiment, the first media modality is one or more of: text, images, video, and audio.

[025] According to an aspect of the present disclosure, there is provided a computer-implemented method of training a generative machine learning model to generate a media file based on a user prompt, the computer-implemented method comprising: providing a pretrained generative machine learning model; receiving a user prompt from a user; generating, using the pre-trained generative machine learning model, a media file based on the user prompt and tailoring information; determining a reward based on the user’s engagement with the generated media file; and modifying the pre-trained generative machine learning model based on the reward.

[026] Optionally, the modified pre-trained generative machine learning model may be transmitted over a network that the processor and storage are connected to.

[027] In an embodiment, determining a reward based on the user’s engagement with the generated media file comprises: determining a positive reward in response to the user performing at least one of saving the generated media file and sharing the generated media file; and determining a negative reward in response to the user performing at least one of discarding the generated media file and inputting an edited user prompt.

[028] According to an aspect of the present disclosure, there is provided a transitory, or non-transitory, computer-readable media having instructions stored thereon that when executed by at least one processor cause the at least one processor to perform the computer-implemented method of any preceding aspect or embodiment. Brief description of drawings

[029] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[030] Figure 1 shows a flow chart summarising operation of a generative machine learning model used for generating a media file based on a user prompt, according to at least one embodiment;

[031] Figure 2 shows a more detailed version of the flow chart from Figure 1,;

[032] Figure 3 shows a flow chart summarising operation of collecting situational-awareness information for a situational-awareness database of the generative machine learning model of Figure 2;

[033] Figure 4 shows flow chart summarising collecting contextual-awareness information for a contextual-awareness database of the generative machine learning model of Figure 2;

[034] Figure 5 shows a schematic of a user feedback mechanism for use with the generative machine learning model from Figure 2;;

[035] Figure 6 shows a flow chart summarising collecting contextual-awareness information from a plurality of users for the contextual-awareness database;

[036] Figures 7A and 7B show flow charts summarising comparing similarities in user tokens used by the generative machine learning model of Figure 2;

[037] Figure 8 shows a graph comparing similarities calculated using the system of Figures 7A and 7B;

[038] Figure 9 shows a flow chart summarising modifying a text input when using the generative machine learning model of Figure 2;

[039] Figure 10 shows a schematic representation of obtaining keywords from text inputs using a language model, according to at least one embodiment;

[040] Figure 11 shows a flow chart summarising modifying a token when using the generative machine learning mode of Figure 2;

[041] Figure 12 shows a flow chart summarising modifying the token using a specific method of merging;

[042] Figure 13 shows a flow chart summarising modifying the token using a specific method of separate addition;

[043] Figure 14 shows a detailed flow chart of the part of the flow chart of Figure 13;

[044] Figure 15 shows a flow chart summarising how to use reinforcement learning to refine the pre-trained generative machine learning model of Figure 2;

[045] Figure 16 shows a flow chart summarising a computer-implemented method of generating a media file using a generative machine learning model, according to at least one embodiment;

[046] Figure 17 shows a flow chart summarising a computer-implemented method of training a generative machine learning model to generate a media file based on a user prompt, according to at least one embodiment; and Figure 18 shows an electronic device used for implementing the generative machine learning model of Figure 2. Detailed description of drawings

[047] With reference to Figure 1, generally, a generative machine learning model 10 for generating a media file 12 includes an encoder 14, a generator 16, and a decoder 18.

[048] The encoder 14 may encode an input 20 to an embedding. The generator 16 may generate an embedding of a media file from the embedding. The embedding of the media file may be a sequence of tokens. The decoder 18 may decode the sequence of media tokens to form a media file 12. The media file may be a sequence of media chunks, for example a sequence of audio samples when the media file is a music file. The sequence of audio samples combine to form a music track. The input may be a first media modality and the media file may be a second media modality. The generative machine learning model 10 may be a transformer model in the sense that the first and second media modalities are different. For example, the first media modality may be text 20_1, images 20_2, video 20_3, or audio 20_4 (e.g. humming). In some embodiments, the first media modality may be a combination of two or more of the foregoing modalities. Multiple modalities can be joined in the encoder. The second modality may be any of the foregoing media modalities but for the purposes of the embodiments below, the second modality may be assumed to be audio, e.g. music.

[049] With this in mind, the encoder 14 may be at least one of a text encoder 14_1, an image encoder 14_2, a video encoder 14_3, and an audio encoder 14_4.

[050] With reference to Figure 2, a computer-implemented method of generating a media file 12 using the generative machine learning model 10 is provided. The method comprises receiving the input 20 in the form of a user prompt. The method also includes receiving tailoring information, from a tailoring information database, for tailoring the media file 12. The tailoring information database may include at least one of a situational-awareness database 22 and a contextual-awareness database 24. The situational-awareness database 22 may be a database storing situational-awareness information. The contextual-awareness database 24 may be a database storing contextual-awareness information. In this way, the tailoring information comprises at least one of contextual-awareness information and situational-awareness information.

[051] With reference to Figure 3, situational-awareness information may include information about a current situtation of a user using a variety of available sensors 26. The sensors may be different sensor modalities. The sensors 26 may include a gyorscope 26_1, an accelerometer 26_2, a global positioning system (GSP) 26_3, a barometer 26_4, a microphone 26_5, a camera 26_6, and a clock. In this way the situational-awareness information may respectively include angular information, positional information, a global position, atmospheric pressure, audio, images, and time of day, relating to a current situation of the user. The method may comprise monitoring the sensor data using one or more sensor monitoring modules 28. The sensor monitoring modules may store the sensor data in the situational-awareness database 22. The sensor data may be stored as sensor information, e.g. may be in the form of text or tokens in the an embedding form.

[052] With further reference to Figure 4, contextual-awareness information may include previous user prompts 34_1, user information from other applications 34_2, and direct feedback 34_3 on previously generated media files. When the input is not text, the input may be considered keywords 34_4 associated with the input. The contextual awareness information may be stored as text, for example, or as tokens, e.g. in an embedding form.

[053] For example, a user pompt in the form of a text input may be “Rock energetic music”. The generative machine learning model 10 may generate a media file 12 and play it to the user. The user may input their preferences using direct feedback 34_3, e.g. they like the media file or do not like the media file. The direct feedback may be stored in the contextual-awreness database 24. Similarly, the user prompt “Rock energetic music” may be directly stored in the contextual-awareness database 24. In some embodiments, the user prompt “Rock energetic music” may also be stored in the contextual awareness database 24 together.

[054] In another example, the input may be an image. An image to keyword generator model 36 may be used to generate keywords 34_4 associated with the input image. The image may be input to the generative machine learning model 10 to produce a media file 12, e.g. an audio track. The user may provide direct feebdack 34_3. The keywords 34_4 may be stored in the contextual-awareness database 24 together with or separately to the direct feedback 34_3.

[055] With reference to Figure 5, the direct feedback may comprise a user rating. The user rating may be input by the user using a slider 30. The slider may include a slide input where a control knob is slid along the slider to adjust the input for a particular attribute. The attribute may be one of a plurality of attributes including tempo 32_1, loudness 32_2, danceability 32_3, and valence 32_4. The attributes may be changed by a user to increase or decrease a number of attributes. The model itself can provide the attributes. User feedback may also be user corrected tokens.

[056] With further reference to Figure 4, the user information from other applications 34_2 may include information such as music genres most often played in a user’s favourite music player. For example, if a user most commonly listens to country music in another music application, country music is stored in the contextual-awareness database 24.

[057] With reference to Figure 6, the method may comprise collecting the tailoring information in the tailoring information database from a plurality of users 38 of the generative machine learning model 10 (Figure 1).

[058] This may comprise identifiring a plurality connections to a user and storing tailoring information associated with the plurality of connection in the tailoring information database. The plurality of connections may be connectins on a social media platform. The tailoring information collected from the plurality of users 38 may be the same as collected for the user of the model. For example, user prompts, direct feedback, etc. may be collected for the plurality of users. The collected tailoring information may be stored in the contextural-awareness database 24.

[059] With reference to Figure 7A and Figure 7B, rather than store all of the collected tailoring information in the contextual-awareness database 24, the method may comprise: comparing a similarity of each new tailoring information to tailoring information stored in the tailoring information database, storing the new tailoring information if the similarity is greater than or equal to a similarity threshold, and disregarding the new tailoring information if the similarity is less than the similarity threshold. In this way, it is possible to identify trending themes and store only contextual-awareness information that is trending in the community of the plurality of users. The method may also comprise discarding old / unused tailoring information.

[060] Conversely, the method may comprise: comparing a similarity of each new tailoring information to tailoring information stored in the tailoring information database, storing the new tailoring information if the similarity is less than or equal to a similarity threshold, and disregarding the new tailoring information if the similarity is greater than the similarity threshold. In this way, a filter is applied to new contextual-awareness information so that duplicate instances are not stored in the contextual-awareness database 24. Advantageously, this means that less data is stored in the contextual-awareness database 24.

[061] With reference to Figures 7A and 7B When the information stored in the contextual-awareness database 24 is in a text modality, comparing the similarity of each new tailoring information to tailoring information stored in the tailoring information database comprises: generating, using a pre-trained model an encoding of each new tailoring information and of each item of tailoring information in the tailoring information database. This may be done as items of new tailoring information are added to the contextual-awareness database 24. The encoding may be a token as described above. The method may also comprise calculating a distance between each new tailoring information item and each item of tailoring information in the tailoring information database. The distance may be calculated using cosine similarity 40. The distance may be compared to the similarity threshold. The pre-trained model may be a contrastive langauge audio pre-training, CLAP, model.

[062] With reference to Figure 8, positions of a first encoding (or token) 42_1, a second encoding (or token) 42_2, and a third encoding (or token) 42_3, are shown on a graph. As an example, the first encoding 42_1 may relate to a first user prompt including keywords “calm jazz music”, “relaxing violin”, and “classical music”. The second encoding 42_2 may relate to a second user prompt including keywords “placid background music”, and “peaceful Cello”. The third encoding 42_3 may relate to a third user prompt including keywords “heavy guitar”, “fast pace rock”, and “hardcore solo drum”. The first encoding 42_1 and the second encoding 42_2 are similar and so are closely positioned on the graphc. The third encoding 42_3 is dissimilar to the first and second encodings 42_1 and 42_2 and so is located further away on the graph. The similarity threshold may be a boundary line on the graph.

[063] Another way in which new tailoring information can be filtered is by comparing a location associated with each new tailoring information to a geographical area, storing the new tailoring information if the location of the new tailoring information is outside the geographical area, and disregarding the new tailoring information if its location is outside the geographical area. In this way, the media file will be generated taking account of trending media in a local or regional area. This may change as the user moves, e.g. goes on holiday. For example, if a certain genre of music is trending with users where the user is usually based, and another genre of music is trending when the user is on holiday, the new trends will affect the genre of a music track generated by the model.

[064] With further reference to Figures 2 and 9, the method also comprises generating, using the encoder 14 of the generative machine learning model 10, an embedding based on the user prompt 20 and also the tailoring information. When the media modality of the user prompt is text, this may comprise obtaining at least one keyword 44 based on the tailoring information and adding the at least one keyword to the user prompt.

[065] With reference to Figure 10, obtaining the at least one keyword 44 based on the tailoring information may include inputting the user prompt 20 to a language model 46. The language model 46 may compare a plurality of sample keywords 44_1 to the user prompt 20. The language model 46 may determine a probability 48 that each key word of the plurality of sample keywords 44_1 is relevant to the user prompt 20. The method may then comprise selecting the at least one (sample) keyword 44_1 with a highest probability out of the plurality of (sample) keywords 44_1 to be added to the user prompt. The selected sample keyword is thus the at least one obtained keyword 44.

[066] For example, the user prompt 20 may be “Fast tempo and sad song”. The sample keywords 44_1 may be “soft female vocal”, “piano melody”, “sad”, and “ballad”. The probability 48 for “soft female vocal” may be 0.8, 0.9 for “piano melody”, 0.1 for “sad” and 0.1 for “ballad”. Therefore, “piano melody” is selected as the obtained keyword 44 because it has a higher probability than the other sample keywords. As a result, “piano melody” is added to the user prompt 20 “Fast tempo and sad song” to form a modified user pomptto be input to the encoder.

[067] With further reference to Figure 9, the keywords may be obtained from the contextual-awareness information from the contextual-awareness database. For example, the contextual-awareness information may be previous user prompts 34_1. The contextual-awareness information may also be modified previous user prompts with situational-awareness information and / or contextual-awareness information, and optionally user feedback on a resulting media file.

[068] Examples of previous user prompts may include “A ballad song that contains sustained strings, mellow piano melody and soft female vocal singing over it. It sounds sad and soulful, like something you would hear at Sunday services”, “A ballad song that consists of passionate, muddy female vocal singing over arpeggiated piano melody”, “Someone is playing a guitarlike instrument in a tremolo fashion along with someone playing a melody on a harp along with a bass playing the root note. A male voice is singing, sounding sad and sensitive. A backing voice that seems to be female is singing along, providing harmonies. This song may be playing in a dance performance”, and “Slow ballad featuring a female voice singing the main melody. This is accompanied by a repetitive piano melody. The bass plays the root notes of the chords in staccato style in eighth note patterns. Synth horns play in the background at different pitches. The song has an ambient and ethereal feel. The mood of the song is haunting. This song can be played in a sad scene in a movie”. The keywords may be extracted by the language model and for the above user prompt examples may be “soft female vocal”, “piano melody”, “sad”, and “ballad”. Therefore, the sample keywords described above may be the keywords obtained from the previous user prompts.

[069] The obtained keyword(s) 44 are input 50 to the current user prompt 20 to obtain a modified user prompt 52.

[070] With refernce to Figure 2 and 11, the method also comprises generating, by the generator 16 of the generative machine learning model 10, a sequence of media tokens based on the embedding and the tailoring information. The sequence of media tokens may be generated by generating a user-specific token using the tailoring information, and generating the sequence of media tokens based on the embedding from the encoder, and the userspecific token. To obtain the user-specific token, the method may comprise obtaining contextual-awareness information from the contextual-awareness database 24. A preference encoding module 54 encodes the contextual awareness information to obtain the use-specific token 56. The preference encoding module may be an encoder trained to encode tailoring information from one modality into a token. For example, the preference encoding module 54 may encode text stored in the contextual-awareness database 24 into the user-specific token 56.

[071] Generating the sequence of media tokens based on the embedding and the userspecific token can be achieved in various ways, as described below with reference to Figures 12 and 13.

[072] With reference to Figure 12, generating the sequence of media tokens comprises merging, e.g. using a meging module 58, the embedding from the encoder 14 with the user specific token 56. Then, the merged embedding and user-specific token 56 are input to the generator 16.

[073] Merging may be achieved in various ways. For example, the embedding and the userspecific token 56 may be concatenated, e.g. [“embedding”; “user-specific token”]. In another example, the embedding and the user-specific token 56 may be averaged, e.g. [embedding + user-specific token] / 2. In another example, attention may be user, e.g. Query=embedding; Key=Value=user-specific token.

[074] With reference to Figure 13, generating the sequence of media tokens comprises inputting the embedding, from the encoder 14, to the generator 16, and inputting separately, the user-specific token 56 to the generator 16.

[075] With reference to Figure 14, , each multi-head attention layer’s input is added with the output of that multi-head attention block’s output and normalised by the normalisation operation. In this architecutre, user specific tokens and comdition embeddings are fused with the existing imput.

[076] With further reference to Figure 2, the method also comprises decoding, by the decoder 18, the sequence of media tokens to obtain the media file 12. The media file 12 is then output. This output may include sending the media file across a network. The output may include presenting the media file to the user. For example, presenting may include outputting the media file, e.g. a track, through a speaker system so the user can hear the generated track.

[077] With reference to Figure 15, there is also provided a computer-implemented method of training a generative machine learning model to generate a media file based on a user prompt. The method comprises providing a pre-trained generative machine learning model 10, and receiving a user prompt 20 from a user. The method also comprises generating, using the model 10, the media file 12 based on the user prompt and tailoring information. Then, the method comprises determining a reward based on the user’s engagement with the generated media file 12.

[078] Determining the reward may include determining a positive reward 70 or a negative reward 72. The positive reward may include things like a user performing at least one of saving the generated media file 12 and sharing the generated media file 12. The negative reward includes things like a user performing at least one of discarding the generated media file and inputting an edited user prompt 20 because they were not satisfied with the media file that was generated.

[079] Then, the pre-trained generative machine learning model is modified based on the reward.

[080] This may be described in more detail as firstly, letting the model (Mo) 10 conditioned by a reinforcement learning model model R' with policy ( k). The Mo10 is pre-trained and RL model fine-tunes the model to user preferences. The RL model is pre-trained with some global data Y. Then, a user fine-tuning step is defined by: o The model 10 generating music m for user input X via function f(Mo, n, R, X) -> m o User uses or discards music to produce rewards r ■ Reward is calculated via reward predictor model, which could take input user features such as discard speed, impression, etc o RL step fine-tunes Mo to produce M1 based on RL model and policy n o The policy n is updated based on reward received over M1 o Finally, these steps are repeated iteratively.

[081] With reference to Figure 16, the computer-implemented methods of generating a media file using a generative machine learning model described herein may be summarised as comprising: receiving 1002 a user prompt; receiving 1004 tailoring information, from a tailoring information database, for tailoring the media file; 1006 generating, using an encoder of the generative machine learning model, an embedding based on the user prompt; generating 1008, by a generator of the generative machine learning model, a sequence of media tokens based on the embedding; decoding 1010, by a decoder of the generative machine learning model, the sequence of media tokens to obtain a media file; and outputting 1012 the media file, wherein at least one of generating an embedding based on the user prompt and generating the sequence of media tokens based on the embedding is also based on the tailoring information.

[082] With reference to Figure 17, the computer-implemented methods of training a generative machine learning model to generate a media file based on a user prompt described herein may be summarised as comprising: providing 2002 a pre-trained generative machine learning model; receiving 2004 a user prompt from a user; generating 2006, using the pretrained generative machine learning model, a media file based on the user prompt and tailoring information; determining 2008 a reward based on the user’s engagement with the generated media file; and modifying 2010 the pre-trained generative machine learning model based on the reward.

[083] With reference to Figure 18, a device 3000 may be used to implement the methods described herein. The device 3000 may be a mobile device or may be another type of device. The mobile device may include a camera 3002, a microphone 3004, a display 3006, a speaker 3007, a processor 3008, and storage 3010. The camera 3002 may capture images and the microphone 3004 may capture audio inputs. The display 3006 may display items such as text, images, videos, etc. The display 3006 may also include a user input function such as a touchscreen where a user may input text, for example, using an electronic keyboard. The speaker 3007 may output the media file in sound form when the media file is an audio track.

[084] The storage 3010 includes instructions stored thereon that when executed by the processor 3008 cause the processor 3008 to perform the computer-implemented methods described herein.

[085] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

1. A computer-implemented method of generating a media file using a generativemachine learning model, the computer-implemented method comprising:receiving a user prompt;receiving tailoring information, from a tailoring information database, for tailoring the media file;generating, using an encoder of the generative machine learning model, an embedding based on the user prompt;generating, by a generator of the generative machine learning model, a sequence of media tokens based on the embedding;decoding, by a decoder of the generative machine learning model, the sequence of media tokens to obtain a media file; andoutputting the media file,wherein at least one of generating an embedding based on the user prompt and generating the sequence of media tokens based on the embedding is also based on the tailoring information.

2. The computer-implemented method of Claim 1, wherein generating the sequence of media tokens based on the embedding and the tailoring information comprises: generating a user-specific token using the tailoring information; and generating the sequence of media tokens based on the embedding and the user-specific token.

3. The computer-implemented method of Claim 2, wherein generating the sequence of media tokens based on the embedding and the embedding token comprises: merging the embedding and the user-specific token; and inputting the merged embedding and user-specific token to a generator of the generative machine learning model.

4. The computer-implemented method of Claim 2, wherein generating the sequence of media tokens based on the embedding and the user-specific token comprises: inputting the embedding to a generator of the generative machine learning model; andinputting separately an embedding token to the generator.

5. The computer-implemented method of any preceding claim, wherein when a media modality of the user prompt is text, generating the embedding based on the user prompt and the tailoring information comprises:obtaining at least one keyword based on the tailoring information; and adding the at least one keyword to the user prompt.

6. The computer-implemented method of Claim 5, wherein obtaining at least one keyword based on the tailoring information comprises:inputting the user prompt to a language model;determining a probability that each keyword out of a plurality of keywords is relevant to the user prompt; andselecting the at least one keyword with a highest probability out of the plurality of keywords.

7. The computer-implemented method of any preceding claim, further comprising:collecting the tailoring information in the tailoring information database from a plurality of users of the generative machine learning model.

8. The computer-implemented method of Claim 7, wherein collecting the tailoring information from the plurality of users of the generative machine learning model comprises: identifying a plurality of connections, on a social media platform, to a user inputting the user prompt; andstoring tailoring information associated with the plurality of connections in the tailoring information database.

9. The computer-implemented method of Claim 7 or Claim 8, wherein collecting the tailoring information from the plurality of users of the generative machine learning model comprises:comparing a similarity of each new tailoring information to tailoring information stored in the tailoring information database;storing the new tailoring information if the similarity is greater than or equal to a similarity threshold; anddisregarding the new tailoring information if the similarity is less than the similarity threshold.

10. The computer-implemented method of Claim 7 or Claim 8, wherein collecting the tailoring information from the plurality of users of the generative machine learning model comprises:comparing a similarity of each new tailoring information to tailoring information stored in the tailoring information database;storing the new tailoring information if the similarity is less than or equal to a similarity threshold; anddisregarding the new tailoring information if the similarity is greater than the similarity threshold.

11. The computer-implemented method of Claim 9 or Claim 10, wherein when the tailoring information stored in the tailoring information database is text, comparing the similarity of each new tailoring information to tailoring information stored in the tailoring information database comprises:generating, using a pre-trained model an encoding of each new tailoring information and of each item of tailoring information in the tailoring information database;calculating a distance between each new tailoring information and each item of tailoring information in the tailoring information database; andcomparing the distance to a distance threshold.

12. The computer-implemented method of any of Claims 7 to 10, wherein collecting the tailoring information from the plurality of users of the generative machine learning model comprises:comparing a location associated with each new tailoring information to a geographical area;storing the new tailoring information if the location of the new tailoring information is outside the geographical area; anddisregarding the new tailoring information if its location is outside the geographical area.

13. The computer-implemented method of any preceding claim, wherein the tailoring information comprises at least one of contextual awareness information and situational awareness information.

14. The computer-implemented method of Claim 12, wherein the contextual awareness information comprises previous user prompts, user information from other applications, and direct feedback on previously generated media files.

15. The computer-implemented method of Claim 13, direct feedback comprises a user rating.

16. The computer-implemented method of any of Claims 13 to 15, wherein the situational awareness information comprises sensor information describing sensor data from a plurality of sensors of different modalities associated with a device used by a user inputting the user prompt.

17. The computer-implemented method of any preceding claim, wherein the generative machine learning model is a transformer model transforming a first media modality to a second media modality, wherein the second media modality is music.

18. The computer-implemented method of Claim 16, wherein the first media modality is one or more of: text, images, video, and audio.

19. A computer-implemented method of training a generative machine learning model to generate a media file based on a user prompt, the computer-implemented method comprising:providing a pre-trained generative machine learning model;receiving a user prompt from a user;generating, using the pre-trained generative machine learning model, a media file based on the user prompt and tailoring information;determining a reward based on the user’s engagement with the generated media file; andmodifying the pre-trained generative machine learning model based on the reward.

20. The computer-implemented method of Claim 16, wherein determining a reward based on the user’s engagement with the generated media file comprises:determining a positive reward in response to the user performing at least one of saving the generated media file and sharing the generated media file; anddetermining a negative reward in response to the user performing at least one of discarding the generated media file and inputting an edited user prompt.

21. A transitory, or non-transitory, computer-readable media having instructions5 stored thereon that when executed by at least one processor cause the at least one processor to perform the computer-implemented method of any preceding claim.15

Images