Method, system, computer-readable storage medium and program for automatic and fast generation of music audio content for video

Abstract

This disclosure describes techniques for automatically and rapidly generating music for videos. The techniques include receiving a video from a user. The video may include a plurality of frame segments. Information may be extracted from the video including information indicative of a rate of motion in the video, information indicative of motion saliency in the video, information indicative of scene transitions in the video, and timing information associated with the video. A plurality of musical note sets may be generated matching the plurality of frame segments based at least in part on the extracted information. A plurality of vectors corresponding to the plurality of musical note sets may be generated. A plurality of musical audio corresponding to the plurality of frame segments may be generated based at least in part on the plurality of vectors.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Application No. 17 / 815,402, filed on July 27, 2022, entitled "Automatic and Rapid Generation of Music Audio Content for Video," the entire contents of which are incorporated herein by reference in their entirety. [Background technology]

[0002] Increasingly, communication is conducted using Internet-based tools. Internet-based tools can be any software or platform. Existing social media platforms allow users to communicate with each other by sharing information such as images and videos through static applications or web pages. As communication devices such as mobile phones become more and more sophisticated, people continue to seek new ways of entertainment, social networking and communication. [Brief description of the drawings]

[0003] The following detailed description can be better understood when read in conjunction with the accompanying drawings, in which: For purposes of illustration, there are shown in the accompanying drawings exemplary embodiments of various aspects of the disclosure; however, the invention is not limited to the specific methods and instrumentalities disclosed.

[0004] [Figure 1] FIG. 1 illustrates an exemplary system for distributing content in accordance with the present disclosure.

[0005] [Diagram 2] FIG. 2 is an exemplary diagram illustrating information extracted from a video in accordance with the present disclosure.

[0006] [Diagram 3] FIG. 13 is an exemplary diagram showing musical notes generated based on information extracted from a video in accordance with the present disclosure.

[0007] [Figure 4] FIG. 13 is an exemplary diagram illustrating vectors corresponding to generated musical notes in accordance with the present disclosure.

[0008] [Diagram 5] FIG. 2 is an exemplary diagram illustrating music audio generated based on vectors corresponding to musical notes in accordance with the present disclosure.

[0009] [Figure 6] FIG. 2 illustrates an exemplary process for automatically generating music audio for a video in accordance with the present disclosure.

[0010] [Figure 7] FIG. 2 illustrates another exemplary process for automatically generating music audio for a video in accordance with the present disclosure.

[0011] [Figure 8] FIG. 2 illustrates another exemplary process for automatically generating music audio for a video in accordance with the present disclosure.

[0012] [Figure 9] FIG. 2 illustrates another exemplary process for automatically generating music audio for a video in accordance with the present disclosure.

[0013] [Figure 10] FIG. 2 illustrates another exemplary process for automatically generating music audio for a video in accordance with the present disclosure.

[0014] [Figure 11] FIG. 1 illustrates an example computing device that can be used to perform any of the methods disclosed herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Users of a content creation platform may struggle to select music to be featured within their content. For example, a user of a content creation platform may be creating a video to be shared on the content creation platform. However, the user may find it difficult to select an appropriate song to play within the video. For example, the user may need to keep re-shooting clips of the video to match the beat and / or style of the selected song. This process may be frustrating, inefficient, and time-consuming. Additionally, it may be difficult for the user to find a song that fits the length of the video. For example, the user may be creating a 10-minute video and may struggle to find a song for the video that lasts at least 10 minutes. For business-oriented (i.e., creating content to promote a business) content creators, it may be difficult to find high-quality music that does not violate territory-specific music license agreements. Alternatively, content creators may attempt to create their own music to accompany their videos. However, creators who do not have professional equipment may be discouraged from undertaking music creation or may be unable to produce high-quality songs.

[0016] Therefore, improvements in content creation technologies, especially for music generation, are desirable. Described herein are technologies that enable efficient smart music generation and professional music editing. After a content creator creates a video, music or audio is automatically generated that matches the transitions, intensity, and movements in the video. Such music or audio may be royalty-free and of any length.

[0017] The music generation techniques described herein may be utilized by a system for distributing content. Figure 1 shows an exemplary system 100 for distributing content. The system 100 may include a server 102 and a number of client devices 104a-104n. The server 102 and the number of client devices 104a-104n may communicate with each other via one or more networks 132.

[0018] The server 102 may be located in a data center, such as a single building, or may be distributed across different geographic locations (e.g., several buildings). The server 102 may provide services via one or more networks 132. The network 132 may include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar devices. The network 132 may include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, etc. The network 132 may include wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0019] The server 102 may include multiple computing nodes hosting various services. In one embodiment, the nodes host a content service 112. The content service 112 may include a content streaming service, such as an Internet Protocol video streaming service. The content service 112 may be configured to deliver content 123 via various transmission technologies. The content service 112 is configured to provide content 123, such as video, audio, text data, combinations thereof, etc. The content 123 may include content streams (e.g., video streams, audio streams, information streams), content files (e.g., video files, audio files, text files), and / or other data. The content 123 may be stored in a database 122. For example, the content service 112 may include a video sharing service, a video hosting platform, a content distribution platform, a collaborative gaming platform, etc.

[0020] In one embodiment, the content 123 delivered or provided by the content service 112 includes video. The video may have a duration up to a predetermined time limit, such as one minute, five minutes, or other predetermined minutes. By way of example and not by way of limitation, a video may include at least one and no more than four 15-second segments joined together. Short video durations can provide viewers with entertainment in rapid succession, allowing users to watch a large amount of video within a short time frame. Such rapid succession entertainment can be popular on social media platforms.

[0021] The videos may include pre-recorded audio overlays, such as music or sounds from a television show or movie. When the short videos include pre-recorded audio overlays, the short videos may feature one or more people lip-syncing, dancing, or otherwise moving their bodies along with the pre-recorded audio. For example, the short videos may feature a "dance challenge" that an individual completes to a hit song, or the short videos may feature two people participating in a lip-sync or two-person dance. As another example, the short videos may feature individuals completing a challenge that requires them to move their bodies to correspond to the pre-recorded audio overlay, e.g., to the beat or rhythm of a pre-recorded song that is featured by the pre-recorded audio overlay. Other videos may not include pre-recorded audio overrides. For example, these videos may feature individuals playing sports, playing pranks, or giving advice, such as beauty and fashion advice, cooking tips, home decor tips, etc.

[0022] In one embodiment, the content 123 may be output to different client devices 104 over the network 132. The content 123 may be streamed to the client devices 104. The content stream may be a stream of video received from the content service 112. Multiple client devices 104 may be configured to access the content 123 from the content service 112. In one embodiment, the client devices 104 may include a content application 106. The content application 106 outputs (e.g., displays, renders, presents) the content 123 to a user associated with the client device 104. The content may include video, audio, comments, text data, etc.

[0023] The client devices 104 may include any type of computing device, such as a mobile device, a tablet device, a laptop computer, a desktop computer, a smart television or other smart device (e.g., a smart watch, a smart speaker, smart glasses, a smart helmet), a gaming device, a set-top box, a digital streaming device, a robot, etc. The client devices 104 may be associated with one or more users. A single user may access the server 102 using one or more of the client devices 104. The client devices 104 may travel to different locations and access the server 102 using different networks.

[0024] The content service 112 may be configured to receive input from a user. The user may be registered as a user of the content service 112 or may be a user of a content application 106 running on a client device 104. The user input may include a video created by the user, a user comment associated with the video, or a "like" associated with the video. The user input may include a connection request and user input data, such as text data, digital image data, or user content. The connection request may include a request from a client device 104a-d to connect to the content service 112. The user input data may include information, such as a video and / or user comments, that a user connected to the content service 112 desires to share with other connected users of the content service 112.

[0025] The content service 112 may be able to receive different types of input from users using different types of client devices 104. For example, a user using the content application 106 on a first user device, such as a mobile phone or tablet, may be able to use the content application 106 to create and upload videos. A user using the content application 106 on a different mobile phone or tablet may be able to view, comment on, and "like" videos or comments written by other users. In another example, a user using the content application 106 on a smart TV, laptop, desktop, or gaming device may not be able to use the content application 106 to create and upload videos or comment on videos. Instead, a user using the content application 106 on a smart TV, laptop, desktop, or gaming device may only be able to use the content application 106 to view videos, view comments left by other users, and "like" videos.

[0026] In one embodiment, a user may use a content application 106 on a client device 104 to create a video, e.g., a short video, and upload the video to the server 102. The client device 104 may access an interface 108 of the content application 106. The interface 108 may include input elements. For example, the input elements may be configured to allow the user to create the video. To create the short video, the user may give the content application 106 permission to access an image capture device, such as a camera or a microphone, of the client device 104. Using the content application 106, the user may select the duration of the video or set the speed of the video, e.g., "slow motion" or "speed up."

[0027] A user may edit a video using the content application 106. A user may add effects, such as one or more text, filters, sounds, or beauty effects, to a video. To add a pre-recorded audio overlay to a video, a user may select a song or sound clip from a sound library of the content application 106. The sound library may include different songs, sound effects, or audio clips from movies, albums, and television shows. In addition to or instead of adding a pre-recorded audio overlay to a video, a user may add narration to a video using the content application 106. The narration may be sound recorded by the user using a microphone on the client device 104. A user may add text overlays to a short video and may specify when they want the text overlay to appear in the video using the content application 106. A user may assign a caption, a location tag, and one or more hashtags to a video to indicate the subject of the video. The content application 106 may prompt the user to select a frame of the video to use as a "cover image" for the video.

[0028] After a user creates a video, the user may use the content application 106 to upload the video to the server 102 and / or store the video locally on the user device 104. When a user uploads a video to the server 102, the user may select whether the video is viewable by all other users of the content application 106 or by only a subset of users of the content application 106. The content service 112 may store the uploaded video and any metadata associated with the video in one or more databases 122.

[0029] In one embodiment, a user may provide input on a video using a content application 106 on a client device 104. The client device 104 may access an interface 108 of the content application 106 that allows the user to provide input associated with the video. The interface 108 may include an input element. For example, the input element may be configured to receive input from a user, such as a comment or a "like" associated with a particular video. If the input is a comment, the content application 106 may allow the user to set an emoji to be associated with their input. The content application 106 may determine time information about the input, such as when the user wrote the comment. The content application 106 may send the input and associated metadata to the server 102. For example, the content application 106 may send the comment, an identifier of the user who wrote the comment, and time information about the comment to the server 102. The content service 112 may store the input and associated metadata in a database 122.

[0030] The content service 112 may be configured to output the uploaded videos and user inputs to other users. A user may be registered as a user of the content service 112 and view videos created by other users. The user may be a user of a content application 106 running on a client device 104. The content application 106 may output (display, render, present) the videos and user comments to a user associated with the client device 104. The client device 104 may access an interface 108 of the content application 106. The interface 108 may include an output element. The output element may be configured to display information about different videos so that a user can select and watch the videos. For example, the output element may be configured to display multiple cover images, subtitles, or hashtags associated with the videos. The output element may also be configured to arrange the videos according to a category associated with each video.

[0031] In one embodiment, user comments associated with a video may be output to other users viewing the same video. For example, all users accessing the video may see the comments associated with the video. The content service 112 may output the comments associated with the video simultaneously. The comments may be output by the content service 112 in real-time or near real-time. The content application 106 may display the video and comments on the client device 104 in a variety of ways. For example, the comments may be displayed in an overlay on top of the content or next to the content. As another example, a user who wants to see other users' comments associated with the video may need to select a button to view the comments. The comments may be animated when displayed. For example, the comments may scroll across the video or across an overlay.

[0032] As described above, a user may create a video and upload it to the server 102 using a content application 106 on a client device 104. In one embodiment, a video created by a user via a content application 106 on a client device 104 may be a video that does not include pre-recorded audio overlays, such as pre-recorded song clips or audio from a television program or movie. Alternatively, music may be automatically generated for a video after a user has already created the video. For example, music audio may be automatically generated for a video locally on the client device 104 after a user has already created the video and before the user uploads it to the server 102. Additionally or alternatively, music audio may be automatically generated for a video by the content service 112 after a user has already uploaded the video to the server 102. Music audio may be generated using, for example, an extraction model 117, a note generation model 118, a vector generation model 119, an audio generation model 120, and / or a refinement model 121.

[0033] In one embodiment, at least one of the content service 112 or the client device 104 includes an extraction model 117. The extraction model 117 may be utilized, at least in part, to retrieve (e.g., determine, extract, etc.) information from a user-created video. For example, the extraction model 117 may be utilized to extract information associated with video motion speed, video motion saliency (i.e., how clearly an object in the video is visible), scene transitions, and / or timing of the video. For example, if a video depicts a man walking, the extraction model 117 may be utilized to extract information associated with the man's walking speed, whether / for how long he stops, etc. Information extracted from the user-created video by the extraction model 117 may be stored in database 124, for example, as extracted data 125.

[0034] FIG. 2 illustrates an exemplary diagram 200 illustrating information extracted from a video 202. The video 202 may include multiple segments 204a-d. The segments 204a-d may include one or more video frames from the video 202. The extraction model 117 may extract information from the entire video 202 (e.g., from all segments 204a-d) associated with video motion speed in the video 202. The extraction model 117 may additionally extract information from the video 202 associated with video motion saliency in the video 202. The extraction model 117 may additionally extract information from the video 202 associated with scene transitions in the video 202. The extraction model 117 may additionally extract information from the video 202 associated with timing in the video 202. For example, the extraction model 117 may determine, for each item of information extracted from the video 202, at which time in the video 202 the information was extracted. For example, if the video 202 has a time interval of 60 seconds, the extraction model 117 may extract information associated with video motion saliency, video motion speed, and / or scene transitions at a particular time, such as 30 seconds. Timing information associated with this item of information may indicate that it was extracted from the video 202 at the 30 second mark. The extracted information associated with video motion speed, video motion saliency, scene transitions, and / or timing may collectively be referred to as extracted information 204. The extracted information 204 may be stored, for example, in database 124 as extracted data 125.

[0035] Returning to FIG. 1 , in one embodiment, at least one of the content service 112 or the client device 104 includes a musical note generation model 118. The musical note generation model 118 may be utilized to automatically generate musical notes that correspond, at least in part, to the extracted data 125. For example, the musical note generation model 118 may utilize information extracted from the video by the extraction model 117 to automatically generate a set of musical notes for the video for each segment of the video. For example, the musical note generation model 118 may utilize information associated with the video's motion rate, motion saliency (i.e., visibility of objects in the video), scene transitions, and / or timing of the video to automatically generate musical notes for the video.

[0036] In an embodiment, to generate musical notes for a video, the musical note generation model 118 may retrieve extracted data 125 from the database 124. The extracted data 125 may be fed into a trained model, such as a compound word transformation model. The model may be pre-trained and configured to correlate video motion speed with musical note density, correlate video motion saliency with musical note intensity, correlate video scene transitions with musical composition, and correlate video timing with musical beats. For example, the model may receive the extracted data 125 as an input. The model may output musical notes (e.g., musical notation) associated with a particular note density, note intensity, musical composition, and / or musical beat. The generated notes may be stored in the database 126, for example, as generated notes data 127.

[0037] In an embodiment, the model is a compound word conversion model trained on a large number of midi files (e.g., 3000 midi files). Adding parameters in the training process may allow control of the music generation. The compound word conversion model may utilize different feed-forward heads to model different types of tokens. The different types of tokens may include, for example, note types and metric types. Using an expansion compression technique, the compound word conversion can convert the music into a compound word sequence by grouping adjacent tokens, greatly reducing the length of the token sequence.

[0038] FIG. 3 is an exemplary diagram 300 illustrating musical notes generated based on information extracted from a video. As described above with respect to FIG. 2, the video 202 may include one or more segments 204a-d of video frames. The extraction model 117 may extract information associated with video motion speed, video motion saliency, scene transitions, and / or timing within each of the segments 204a-d from the video 202. The extracted information associated with video motion speed, video motion saliency, scene transitions, and / or timing may be collectively referred to as extracted information 204. The musical note generation model 118 may generate musical notes having a note density correlating to each item of extracted information associated with video motion speed in the video 202. The musical notes generated by the musical note generation model 118 may additionally have a note intensity correlating to each item of extracted information associated with video motion saliency in the video 202. Additionally, the notes generated by the note generation model 118 may have a note configuration that correlates to each item of extracted information associated with a video scene transition in the video 202. Finally, the notes generated by the note generation model 118 may have a musical beat that correlates to each item of extracted information associated with timing in the video 202. Notes with corresponding note density, note intensity, musical configuration, and / or musical beat may be collectively referred to as generated notes.

[0039] In an embodiment, the generated notes may be divided into, for example, a plurality of note sets 304a-d. Each of the note sets 304a-d may correspond to one of the segments 204a-d of the video 202. For example, the note set 304a may correspond to the segment 204a of the video 202, the note set 304b may correspond to the segment 204b of the video 202, the note set 304c may correspond to the segment 204c of the video 202, and the note segment 304d may correspond to the segment 204d of the video 202. The plurality of note sets 304a-d may be stored in the database 126 as note data 127.

[0040] Returning to FIG. 1 , in one embodiment, at least one of the content service 112 or the client device 104 includes a vector generation model 119. The vector generation model 119 may be utilized, at least in part, to automatically generate vectors corresponding to the musical note data 127. For example, the vector generation model 119 may utilize the musical note data 127 generated by the musical note generation model 118 to automatically generate vectors for each of the plurality of musical note sets 304a-d. Each vector may indicate at least one musical characteristic of the musical audio associated with the corresponding musical note set. For example, the at least one musical characteristic may include a musical style (e.g., musical genre and / or mood), a bar structure, or an instrument. The musical genre may indicate whether the musical audio associated with the corresponding musical note set is closest to pop, rock, hip hop, country, etc. The musical mood may indicate whether the musical audio associated with the corresponding musical note set has a particular energy (e.g., lively, sad, slow, bright, etc.).

[0041] In an embodiment, to generate the vectors, the vector generation model 119 may retrieve musical note data 127 from the database 126. The vector generation model 119 may be a trained model configured to determine musical characteristics based on musical notes. For example, the vector generation model 119 may receive the musical note data 127 as input. The vector generation model 119 may output a vector (e.g., a one-dimensional data row or a multi-dimensional vector) indicating musical characteristics of the input musical note data 127. The vector may have a format consumable by, for example, the audio generation model 120. The generated vectors may be stored in the database 130, for example, as vector data 131.

[0042] In an embodiment, a user may be able to specify one or more musical preferences. Such preferences may be utilized by the vector generation model 119 while generating one or more vectors. For example, a user may indicate that he / she would like upbeat music to accompany a video. As such, the user may specify this preference, and the vector generation model 119 may utilize the user-defined preference in addition to, or as an alternative to, the plurality of musical note sets 304a-d. The preferences may indicate, for example, the genre, mood, style, and / or one or more instruments that the user would like the generated music to reflect. The preferences may harmonize or clash with the plurality of musical note sets 304a-d. For example, a set of musical notes may indicate a slow or sad energy. Nevertheless, the user may indicate that he / she would like music content for the video to have an upbeat or happy energy. Thus, the vectors may reflect the preferences selected by the user, rather than the preferences indicated by the musical notes.

[0043] FIG. 4 illustrates an exemplary diagram 400 showing vectors generated based on musical notes. As described above with respect to FIG. 3, a plurality of sets of notes 304a-d may be generated. Each of the sets of notes 304a-d may correspond to one of the segments 204a-d of the video 202. The vector generation model 119 may generate a plurality of vectors 402a-d. Each of the plurality of vectors 402a-d may correlate to a particular set of notes 304a-d. For example, vector 402a may correlate to set of notes 304a, vector 402b may correlate to set of notes 304b, vector 402c may correlate to set of notes 304c, and vector 402d may correlate to set of notes 304d.

[0044] The vectors 402a-d generated by the vector generation model 119 may indicate at least one musical feature of the musical audio associated with the corresponding set of notes. For example, the vector 402a may indicate at least one musical feature of the musical audio associated with the set of notes 304a. The musical audio associated with the set of notes 304a may be a musical audio signal (e.g., an actual audio signal, not just notes) that may be generated based on the set of notes 304a. Similarly, the vector 402b may indicate at least one musical feature of the musical audio associated with the set of notes 304b, the vector 402c may indicate at least one musical feature of the musical audio associated with the set of notes 304c, and the vector 402d may indicate at least one musical feature of the musical audio associated with the set of notes 304d. The vectors 402a-d may be stored in the database 130 as vector data 131.

[0045] The vectors 402a-d may be consumable by the audio generation model 120. For example, the audio generation model 120 may not be able to read the notes from the note sets 304a-d. Thus, the vector generation model 119 may be utilized to translate or convert the note sets 304a-d into a format readable or consumable by the audio generation model 120.

[0046] As described above, each vector 402a-d may indicate at least one musical characteristic of the musical audio associated with the corresponding note set 304a-d. For example, the at least one musical characteristic may include a musical style (e.g., musical genre and / or mood), a bar structure, or an instrument. A musical genre may indicate whether the musical audio associated with the corresponding note set is closest to pop, rock, hip hop, country, etc. A musical mood may indicate whether the musical audio associated with the corresponding note set has a particular energy (e.g., lively, sad, slow, bright, etc.).

[0047] Returning to FIG. 1 , in one embodiment, at least one of the content service 112 or the client device 104 includes an audio generation model 120. The audio generation model 120 may be utilized to generate a plurality of music audios based at least in part on the vector data 131. The audio generation model 120 may receive the vector data 131 as an input and output a plurality of musics. Each music from the plurality of musics may correspond to a particular vector from the plurality of vectors generated by the vector generation model 119. As described above, each vector from the plurality of vectors corresponds to a particular set of notes from the plurality of sets of notes generated by the note generation model 118. As described above, each set of notes from the plurality of sets of notes corresponds to a particular video segment. Thus, each music from the plurality of musics generated by the audio generation model 120 may correspond to a particular video segment. Each music audio generated by the audio generation audio may represent or reflect the musical characteristics of that particular music audio as indicated by the corresponding vector. The music audio may be stored in a database 132, for example, as music audio data 133.

[0048] In an embodiment, to generate the musical audios, the audio generation model 120 may utilize one or more templates. The one or more templates may be from a plurality of templates pre-stored in a database, for example, template 129 in database 128. Each of the plurality of templates may include an audio file. The audio file may be associated with at least a main track (e.g., chords) and / or a base music associated with a particular musical style or type. Based on the vector data 131, the audio generation model 120 may determine one or more templates associated with the appropriate musical style or type. For example, the audio generation model 120 may retrieve the one or more templates from database 128.

[0049] In an embodiment, the audio generation model 120 may modify or add features to the template(s) after the template(s) have already been retrieved. For example, the audio generation model 120 may modify or add features to the template(s) based on a corresponding set of notes previously generated by the note generation model 118. For example, the audio generation model 120 may generate a main melody, ornamental melodies, and / or instrumental tracks based on the corresponding set of notes. The audio generation model 120 may add the main melody, ornamental melodies, and / or instrumental tracks to the audio file (e.g., main audio track) of the template(s) retrieved from the database.

[0050] By utilizing one or more templates, the music audio generation process is quick and does not require excessive computing power. For example, by utilizing one or more templates to provide a base track for the music audio, the need to generate a new audio track from scratch each time music audio content for a video is generated can be prevented. As a result, a user does not have to wait a long time for the music audio content to be generated. In comparison, if a new audio track had to be generated each time music audio content for a video was needed, this would require a large amount of computing resources and would also require a user to wait a longer time for the music audio content to be generated.

[0051] In an embodiment, the audio generation model 120 may generate a final music audio content for the video based at least in part on synthesizing the plurality of music audios after generating the plurality of music audios. The final music audio content for the video may match the motion, intensity, and transitions in the video.

[0052] FIG. 5 illustrates an example diagram 500 showing music generated at least in part based on the vectors. As described above with respect to FIG. 4, a plurality of vectors 402a-d may be generated. Each of the vectors 402a-d may correspond to one of the musical note sets 304a-d. Each of the musical note sets 304a-d may correspond to one of the segments 204a-d of the video 202. Thus, each of the vectors 402a-d may correspond to one of the segments 204a-d of the video. The audio generation model 120 may generate a plurality of musical audios 502a-d based at least in part on the vectors 402a-d. Each of the plurality of musical audios 502a-d may be correlated to the vectors 402a-d. For example, music audio 502a may be correlated to vector 402a, music audio 502b may be correlated to vector 402b, music audio 502c may be correlated to vector 402c, and music audio 502d may be correlated to vector 402d. If each of vectors 402a-d corresponds to one of segments 204a-d of the video, music audio 502a may be correlated to segment 204a, music audio 502b may be correlated to segment 204b, music audio 502c may be correlated to segment 204c, and music audio 502d may be correlated to segment 204d. Music audio 502a-d may be synthesized to generate a final music audio content for the video 202 that matches the motion, intensity, and transitions in the video 202. The video audio matching between the video 202 and the music audio content may be fine-tuned, for example, by applying video warping while generating the music audio content and / or while adding the music audio content to the video 202.

[0053] Returning to FIG. 1 , in one embodiment, at least one of the content service 112 or the client device 104 includes a refinement model 121. The refinement model 121 may be utilized to refine or modify, at least in part, the final music audio content generated for the video. For example, the refinement model 121 may determine whether or not a user likes the generated music audio content, for example, based on user input. For example, the refinement model 121 may receive an indication that the user likes the generated music audio content. If the refinement model 121 receives an indication that the user likes the generated music audio content, the generated music audio content may remain unchanged. Conversely, if the refinement model 121 receives an indication that the user does not like the generated music audio content, the refinement model 121 may present or cause a plurality of options to be displayed. For example, the refinement model 121 may present or cause a plurality of options to be displayed on the interface 108a-d of the content application 106. Each of the plurality of options may represent a potential modification or change that may be made to the generated music audio content. The user may select (e.g., click) one or more of the plurality of options. The refinement model 121 may receive an indication of the selected option or options. The refinement model 121 may update or modify the music audio content based on the option or options selected by the user. This process may be repeated until the user indicates that he or she likes the music audio content.

[0054] 6 illustrates an example process 600 performed by a content service (e.g., content service 112) and / or a client device (e.g., client device 104). The content service and / or client device can perform this process automatically to efficiently generate music audio content for a video. Although illustrated as a series of operations in FIG. 6, one skilled in the art will recognize that various embodiments may add, remove, rearrange, or modify the described operations.

[0055] As described above, a user of a content service may create a video for distribution to other users of the content service. At 602, a video may be received from the user. The video may include a number of frame segments. The video may have already been created by the user. The video may not include background music. The user may wish to automatically and efficiently generate background music to correspond to the video.

[0056] To automatically generate music corresponding to the video frames, information may be extracted from the video frames. At 604, information may be extracted from the video. The extracted information may include information indicative of motion speed in the video, information indicative of motion saliency in the video, information indicative of scene transitions in the video, and timing information associated with the video. For example, if a video depicts a man walking, information may be extracted from the video related to the man's walking speed, whether / for how long he stops, etc. The extracted information related to video motion speed, video motion saliency, scene transitions, and / or timing may collectively be referred to as extracted information.

[0057] The note generation model may be utilized, at least in part, to automatically generate a set of notes for each of the segments of the video frames. For example, the note generation model may utilize the extracted information to automatically generate a set of notes for each of the segments of the video frames. At 606, the model (e.g., the note generation model) may be used to generate a set of notes matching the frame segments based at least in part on the extracted information. The model may be pre-trained and may have been learned to correlate video motion speed with note density, correlate video motion saliency with note intensity, correlate video scene transitions with music composition, and correlate video timing with music beats. As described above, in an embodiment, the model may be a compound word conversion model trained on a large number of midi files (e.g., 3000 midi files).

[0058] The vector generation model may be utilized, at least in part, to automatically generate vectors corresponding to a plurality of sets of notes. For example, the vector generation model may utilize a plurality of sets of notes to automatically generate a vector for each of the plurality of sets of notes. Each vector may indicate at least one musical feature of musical audio associated with a corresponding set of notes.

[0059] At 608, a plurality of vectors corresponding to the plurality of sets of notes may be generated. Each of the plurality of vectors may indicate at least one musical feature of one of the plurality of musical audios. For example, the at least one musical feature may include a musical style (e.g., musical genre and / or mood), a bar structure, or an instrument. A musical genre may indicate whether the musical audio associated with the corresponding set of notes is closest to pop, rock, hip hop, country, etc. A musical mood may indicate whether the musical audio associated with the corresponding set of notes has a particular energy (e.g., lively, sad, slow, bright, etc.). The vectors may have a format consumable by an audio generation model, for example.

[0060] An audio generation model may be utilized to generate a plurality of music audios based at least in part on the plurality of vectors. The audio generation model may receive the vectors as input and output a plurality of musics. At 610, a plurality of music audios corresponding to the plurality of frame segments may be generated based at least in part on the plurality of vectors. Each music from the plurality of musics may correspond to a particular vector from the plurality of vectors. As described above, each vector from the plurality of vectors corresponds to a particular set of notes from a plurality of sets of notes generated by the note generation model. As described above, each set of notes from the plurality of sets of notes corresponds to a particular video segment. Thus, each music from the plurality of musics generated by the audio generation model may correspond to a particular video segment. Each music audio generated by the audio generation audio may represent or reflect musical characteristics of the particular music audio as indicated by the corresponding vector.

[0061] 7 illustrates an example process 700 that may be performed by a content service (e.g., content service 112) and / or a client device (e.g., client device 104). The content service and / or client device may perform this process 700 to efficiently generate music for a video. Although illustrated in FIG. 7 as a series of operations, one skilled in the art will recognize that various embodiments may add, remove, rearrange, or modify the operations described.

[0062] The vector generation model may be utilized, at least in part, to automatically generate vectors corresponding to a plurality of sets of notes. For example, the vector generation model may utilize a plurality of sets of notes to automatically generate a vector for each of the plurality of sets of notes. Each vector may indicate at least one musical feature of musical audio associated with a corresponding set of notes.

[0063] At 702, a plurality of vectors corresponding to the plurality of sets of notes may be generated. Each of the plurality of vectors may indicate at least one musical feature of one of the plurality of musical audios. For example, the at least one musical feature may include a musical style (e.g., musical genre and / or mood), a bar structure, or an instrument. A musical genre may indicate whether the musical audio associated with the corresponding set of notes is closest to pop, rock, hip hop, country, etc. A musical mood may indicate whether the musical audio associated with the corresponding set of notes has a particular energy (e.g., lively, sad, slow, bright, etc.). The vectors may have a format consumable by an audio generation model, for example.

[0064] To generate the plurality of musical audios, the audio generation model may utilize one or more templates. The one or more templates may be from a plurality of templates stored in a database. Each of the plurality of templates may include an audio file. The audio file may be associated with at least a main track (e.g., chords) and / or base music associated with a particular musical style or type. At 704, at least one template may be obtained based on each of the plurality of vectors. For example, based on musical features indicated by the plurality of vectors, the audio generation model may determine one or more templates associated with an appropriate musical style or type. Such templates may be retrieved from a database. At 706, musical audio corresponding to each of the plurality of frame segments may be generated based at least in part on the at least one audio file.

[0065] In an embodiment, after the one or more templates have been retrieved and the music audio corresponding to each of the plurality of frame segments has been generated, the audio generation model may make modifications or additions to the audio file. At 708, the at least one audio file may be modified by adding a melody based on one of the plurality of note sets. For example, the audio generation model may generate a main melody, an ornamental melody, and / or an instrument track based on the corresponding note set. The audio generation model may add the main melody, the ornamental melody, and / or the instrument track to the audio file of the template (e.g., the main audio track).

[0066] 8 illustrates an example process 800 performed by a content service (e.g., content service 112) and / or a client device (e.g., client device 104). The content service and / or client device may perform this process 800 to efficiently generate music for a video. Although illustrated in FIG. 8 as a series of operations, one skilled in the art will recognize that various embodiments may add, remove, rearrange, or modify the operations described.

[0067] As described above, a user of a content service may create a video for distribution to other users of the content service. At 802, a video may be received from the user. The video may include a number of frame segments. The video may have already been created by the user. The video may not include background music. The user may wish to automatically and efficiently generate background music to correspond to the video.

[0068] To automatically generate music corresponding to the video frames, information may be extracted from the video frames. At 804, information may be extracted from the video. The extracted information may include information indicative of motion speed in the video, information indicative of motion saliency in the video, information indicative of scene transitions in the video, and timing information associated with the video. For example, if a video depicts a man walking, information may be extracted from the video related to the man's walking speed, whether / for how long he stops, etc. The extracted information related to video motion speed, video motion saliency, scene transitions, and / or timing may collectively be referred to as extracted information.

[0069] The note generation model may be utilized, at least in part, to automatically generate a set of notes for each of the segments of the video frames. For example, the note generation model may utilize the extracted information to automatically generate a set of notes for each of the segments of the video frames. At 806, the model (e.g., the note generation model) may be used to generate a plurality of sets of notes matching the plurality of frame segments based at least in part on the extracted information. The model may be pre-trained and may learn to correlate video motion speed with note density, correlate video motion saliency with note intensity, correlate video scene transitions with music composition, and correlate video timing with music beat. The model may be pre-trained and may have been learned to correlate video motion speed with note density, correlate video motion saliency with note intensity, correlate video scene transitions with music composition, and correlate video timing with music beat. As described above, in an embodiment, the model may be a compound word conversion model trained on a large number of MIDI files (e.g., 3000 MIDI files).

[0070] The vector generation model may be utilized, at least in part, to automatically generate vectors corresponding to a plurality of sets of notes. For example, the vector generation model may utilize a plurality of sets of notes to automatically generate a vector for each of the plurality of sets of notes. Each vector may indicate at least one musical feature of musical audio associated with a corresponding set of notes.

[0071] At 808, a plurality of vectors may be generated corresponding to the plurality of sets of notes. Each of the plurality of vectors may indicate at least one musical feature of one of the plurality of musical audios. For example, the at least one musical feature may include a musical style (e.g., musical genre and / or mood), a bar structure, or an instrument. A musical genre may indicate whether the musical audio associated with the corresponding set of notes is closest to pop, rock, hip hop, country, etc. A musical mood may indicate whether the musical audio associated with the corresponding set of notes has a particular energy (e.g., lively, sad, slow, bright, etc.). The vectors may have a format consumable by an audio generation model, for example.

[0072] An audio generation model may be utilized to generate a plurality of music audios based at least in part on the plurality of vectors. The audio generation model may receive the vectors as input and output a plurality of musics. At 810, a plurality of music audios corresponding to the plurality of frame segments may be generated based at least in part on the plurality of vectors. Each music from the plurality of musics may correspond to a particular vector from the plurality of vectors. As described above, each vector from the plurality of vectors corresponds to a particular set of notes from a plurality of sets of notes generated by the note generation model. As described above, each set of notes from the plurality of sets of notes corresponds to a particular video segment. Thus, each music from the plurality of musics generated by the audio generation model may correspond to a particular video segment. Each music audio generated by the audio generation audio may represent or reflect musical characteristics of the particular music audio as indicated by the corresponding vector. At 812, music audio content may be generated for the video based at least in part on synthesizing the plurality of music audios. The music audio content may match motion, intensity, and transitions in the video.

[0073] 9 illustrates an example process 900 performed by a content service (e.g., content service 112) and / or a client device (e.g., client device 104). The content service and / or client device may perform this process 900 to efficiently generate music for a video. Although illustrated in FIG. 9 as a series of operations, one skilled in the art will recognize that various embodiments may add, remove, rearrange, or modify the described operations.

[0074] As described above, music audio content may be generated for the video. At 902, music audio content may be generated for the video based at least in part on combining the multiple music audios. The music audio content may match motion, intensity, and transitions in the video.

[0075] At 904, it may be determined whether the user likes the generated musical audio content based on, for example, user input. At 904, it may be determined whether the user likes the musical audio content based on, for example, user input. For example, an indication may be received indicating that the user likes the generated musical audio content. If an indication is received indicating that the user likes the generated musical audio content, the generated musical audio content may remain unchanged. Conversely, an indication may be received indicating that the user does not like the generated musical audio content. At 906, in response to determining that the user does not like the musical audio content, a number of options may be presented. For example, the number of options may be presented or displayed on an interface of a content application.

[0076] At 908, the musical audio content may be updated based on one or more options selected by the user from the plurality of options. Each of the plurality of options may represent a potential modification or change that may be made to the generated musical audio content. The user may select (e.g., click) one or more of the plurality of options. An indication of the selected option or options may be received. The musical audio content may be updated, modified, or refined based on the option or options selected by the user. This process may be repeated until the user indicates that he or she likes the musical audio content.

[0077] 10 illustrates an example process 1000 performed by a content service (e.g., content service 112) and / or a client device (e.g., client device 104). The content service and / or client device may perform the process 1000 to efficiently generate music for a video. Although illustrated in FIG. 10 as a series of operations, one skilled in the art will recognize that various embodiments may add, remove, rearrange, or modify the operations described.

[0078] As described above, a user of a content service may create a video for distribution to other users of the content service. At 1002, a video may be received from the user. The video may include a number of frame segments. The video may have already been created by the user. The video may not include background music. The user may wish to automatically and efficiently generate background music to correspond to the video.

[0079] An audio generation model may be utilized to generate a plurality of music audios based at least in part on the plurality of vectors. The audio generation model may receive the vectors as input and output a plurality of musics. At 1004, a plurality of music audios corresponding to the plurality of frame segments may be generated based at least in part on the plurality of vectors. Each music from the plurality of musics may correspond to a particular vector from the plurality of vectors. As described above, each vector from the plurality of vectors corresponds to a particular set of notes from a plurality of sets of notes generated by the note generation model. As described above, each set of notes from the plurality of sets of notes corresponds to a particular video segment. Thus, each music from the plurality of musics may correspond to a particular video segment. Each music may represent or reflect musical characteristics of the particular music audio as indicated by the corresponding vector.

[0080] At 1006, music audio content may be generated for the video based at least in part on combining the plurality of music audios. The music audio content may match motion, intensity, and transitions in the video. The video audio matching between the video and the music audio content may be fine-tuned, for example, by applying video warping while generating the music audio content and / or while adding the music audio content to the video. At 1008, the video audio matching between the video and the music audio content may be fine-tuned by applying video warping while generating the music audio content.

[0081] Figure 11 illustrates a computing device that may be used in various aspects, such as the services, networks, modules, and / or devices illustrated in Figure 1. With respect to the exemplary architecture of Figure 1, any one or all of the models, servers, content services, and client devices may each be realized by one or more instances of computing device 1100 of Figure 11. The computer architecture illustrated in Figure 11 illustrates a conventional server computer, workstation, desktop computer, laptop computer, tablet, network appliance, PDA, electronic reader, digital mobile phone, or other computing node that may be used to perform any aspect of the computer described herein, such as implementing the methods described herein.

[0082] Computing device 1100 may include a board or "motherboard," which is a printed circuit board that can be connected to multiple components or devices via a system bus or other electrical communication paths. One or more central processing units (CPUs) 1104 may operate in conjunction with a chipset 1106. CPU 1104 may be a standard programmable processor that performs arithmetic and logical operations necessary for the operation of computing device 1100.

[0083] The CPU 1104 may transition from one discrete physical state to the next and perform the necessary operations by manipulating switching elements that distinguish between these states. Switching elements may typically include electronic circuits, such as flip-flops, that maintain one of two binary states, and electronic circuits, such as logic gates, that provide an output state based on a logical combination of the states of one or more other switching elements. These basic switching elements may be combined to create more complex logic circuits, including registers, adders and subtractors, arithmetic logic units, floating point units, etc.

[0084] CPU 1104 may be augmented or replaced by other processing units, such as GPUs, which may include processing units specialized, but not necessarily limited to, highly parallel computations such as graphics and other visualization-related processing.

[0085] The chipset 1106 may provide an interface between the CPU 1104 and the remaining components and devices on the board. The chipset 1106 may provide an interface to a random access memory (RAM) 1108 used as a main memory in the computing device 1100. The chipset 1106 may also provide an interface to a computer-readable storage medium, such as a read-only memory (ROM) 1120 or a non-volatile RAM (NVRAM) (not shown), for storing basic routines that may facilitate starting the computing device 1100 and transmitting information between various components and devices. According to aspects described herein, the ROM 1120 or the NVRAM may store other software components necessary for the operation of the computing device 1100.

[0086] The computing device 1100 may operate in a networked environment using logical connections to remote computing nodes and computer systems via a local area network (LAN). The chipset 1106 may include functionality for providing network connectivity via a network interface controller (NIC) 1122, such as a Gigabit Ethernet adapter. The NIC 1122 may be capable of connecting the computing device 1100 to other computing nodes via the network 1116. It should be understood that multiple NICs 1122 may be present within the computing device 1100 to connect the computing device to other types of networks and remote computer systems.

[0087] The computing device 1100 may be connected to a mass storage device 1128 that provides non-volatile storage for the computer. The mass storage device 1128 may store system programs, application programs, other program modules, and data, as described in more detail herein. The mass storage device 1128 may be connected to the computing device 1100 through a storage controller 1124 that is connected to the chipset 1106. The mass storage device 1128 may be comprised of one or more physical storage units. The mass storage device 1128 may include a management component 1110. The storage controller 1124 may interface with the physical storage units through a Serial Attached SCSI (SAS) interface, a Serial Advanced Technology Attachment (SATA) interface, a Fibre Channel (FC) interface, or any other type of interface for physically connecting and transmitting data between the computer and the physical storage units.

[0088] Computing device 1100 may store data on mass storage device 1128 by transforming the physical state of the physical storage units to reflect the stored information. The particular transformation of the physical state may depend on a variety of factors and different implementations of the present specification. Examples of such factors include, but are not limited to, the technology for implementing the physical storage devices and whether mass storage device 1128 is characterized as a primary storage device, a secondary storage device, etc.

[0089] For example, the computing device 1100 may store information in the mass storage device 1128 by issuing instructions via the storage controller 1124 to change the magnetic properties of a particular location in a magnetic disk drive unit, the reflective or refractive properties of a particular location in an optical storage unit, or the electrical properties of a particular capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of physical media are possible without departing from the scope and spirit of this specification, and the foregoing examples are provided merely for ease of explanation thereof. The computing device 1100 may further read information from the mass storage device 1128 by detecting a physical state or characteristic of one or more particular locations in the physical storage unit.

[0090] In addition to the mass storage device 1128 discussed above, computing device 1100 may also access other computer-readable storage media for storing and retrieving information such as program modules, data structures, or other data. Those skilled in the art will appreciate that computer-readable storage media can be any available media that provides non-transitory storage of data and that can be accessed by computing device 1100.

[0091] By way of example and not limitation, computer readable storage media may include volatile and non-volatile, transient and non-transient computer readable storage media, as well as removable and non-removable media implemented in any manner or technology, including, but not limited to, RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory or other solid state memory technology, compact disc ROM (CD-ROM), digital versatile disc (DVD), high definition DVD ("HD-DVD"), BLU-RAY or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage, other magnetic storage devices, or any other medium that can be used to store the desired information in a non-transitory manner.

[0092] A mass storage device, such as mass storage device 1128 shown in FIG. 11, may store an operating system for controlling the operation of computing device 1100. The operating system may include a version of the LINUX® operating system. The operating system may include a version of Microsoft's WINDOWS SERVER® operating system. According to another aspect, the operating system may include a version of the UNIX® operating system. Also, various mobile phone operating systems, such as IOS® and ANDROID®, may be utilized. It should be understood that other operating systems may also be utilized. Mass storage device 1128 may store other system or applications and data used by computing device 1100.

[0093] The mass storage device 1128 or other computer-readable storage medium may also be encoded with computer-executable instructions that, when loaded into computing device 1100, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing aspects described herein. As described above, these computer-executable instructions transform computing device 1100 by defining how CPU 1104 transitions between states. Computing device 1100 may have access to a computer-readable storage medium that stores computer-executable instructions that, when executed by computing device 1100, can perform the methods described herein.

[0094] A computing device, such as the computing device 1100 shown in Figure 11, may further include an input / output controller 1132 for receiving and processing input from multiple input devices, such as a keyboard, a mouse, a touchpad, a touch screen, an electronic stylus pen, or other type of input device. Similarly, the input / output controller 1132 may provide output to a display, such as a computer monitor, a flat panel display, a digital projector, a printer, a plotter, or other type of output device. It should be understood that the computing device 1100 may not include all of the components shown in Figure 11, may include other components not explicitly shown in Figure 11, or may utilize an entirely different architecture than the architecture shown in Figure 11.

[0095] As described herein, a computing device may be a physical computing device, such as computing device 1100 of Figure 11. A computing node may also include a virtual machine host process and one or more virtual machine instances. Computer-executable instructions may be executed indirectly by the physical hardware of the computing device by interpreting and / or executing instructions that are stored and executed within the context of a virtual machine.

[0096] It should be understood that the methods and systems are not limited to particular methods, components, or implementations, and it should also be understood that the terminology used herein is used for the purpose of describing particular embodiments only, and is not intended to be limiting.

[0097] As used in the specification and the appended claims, the singular forms "a," "one," and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value, and / or to "about" another particular value. When such a range is expressed, another embodiment includes the range from the one particular value and / or to the other particular value. Similarly, when values ​​are expressed as approximations, by use of the antecedent "about," it is to be understood that the particular value forms another embodiment. It is further to be understood that the endpoints of each of the ranges are significant both relative to the other endpoint, and independently of the other endpoint.

[0098] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and the specification includes cases where said event or circumstance occurs and cases where said event or circumstance does not occur.

[0099] Throughout the description and claims of this specification, the word "comprise" and variations of that word, such as "including" and "including," mean "including, but not limited to," and are not intended to exclude, for example, other elements, integers, or steps. "Exemplary" denotes "an example of," and is not intended to convey an indication of a preferred or desirable embodiment. "Such as" is used for purposes of interpretation rather than a limiting sense.

[0100] Components are described that can be used to perform the described methods and systems. In describing combinations, subsets, interactions, groups, etc. of these components, specific references to each of the various individual and collective combinations and permutations of these components may not be explicitly described, and each of which is specifically contemplated and described herein for all methods and systems. This applies to all aspects of this application, including but not limited to the operations in the described methods. Thus, if there are various additional operations that can be performed, it should be understood that each of these additional operations can be performed in any particular embodiment or combination of embodiments of the described methods.

[0101] The present method and system may be more readily understood by reference to the following detailed description of the preferred embodiments and examples contained therein, and the accompanying drawings and their descriptions.

[0102] As will be appreciated by those skilled in the art, the method and system may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the method and system may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. More specifically, the method and system may take the form of web-implemented computer software. Any suitable computer-readable storage medium may be utilized, including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.

[0103] Embodiments of the methods and systems are described below with reference to block diagrams and flowcharts of methods, systems, devices, and computer program products. It should be understood that each block of the block diagrams and flowcharts, and combinations of blocks in the block diagrams and flowcharts, respectively, may be realized by computer program instructions. These computer program instructions may be loaded into a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the computer or other programmable data processing apparatus, generate means for implementing the functions specified in one or more blocks of the flowcharts.

[0104] These computer program instructions may be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including computer readable instructions for implementing the functions defined in one or more blocks of the flowcharts. The computer program instructions may be loaded into a computer or other programmable data processing apparatus to cause the computer or other programmable data processing apparatus to perform a series of operational steps to produce a computer implemented process, such that the instructions that execute on the computer or other programmable data processing apparatus provide steps for implementing the functions defined in one or more blocks of the flowcharts.

[0105] The various features and processes described above may be used independently of each other or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Furthermore, in some implementations, some method or process blocks may be omitted. The methods and processes described herein are not limited to any particular order, and the blocks or states associated therewith may be performed in other orders as appropriate. For example, the described blocks or states may be performed in an order other than the order specifically described, or multiple blocks or states may be combined within a single block or state. The example blocks or states may be performed sequentially, in parallel, or in some other manner. Blocks or states may be added to or deleted from the described example embodiments. The example systems and components described herein may be configured differently than described. For example, elements may be added, deleted, or rearranged as compared to the described example embodiments.

[0106] It should also be understood that while various items are shown as being stored in memory or on a storage device during use, these items, or portions thereof, may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or systems may run in memory on another device and communicate with the illustrated computing system via computer-to-computer communications. Furthermore, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least partially in firmware and / or hardware, including, but not limited to, one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and the like. Some or all of the modules, systems, and data structures may be stored (e.g., as software instructions or structured data) on a computer-readable medium, such as a hard disk, memory, network, or portable media product, for reading by an appropriate device or via an appropriate connection. The systems, modules, and data structures may be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagated signal) over a variety of computer-readable transmission media, including wireless-based media and wire / cable-based media, which may take a variety of forms (e.g., as part of a single or multiple analog signal, or as a number of discrete digital packets or frames). In other embodiments, such computer program products may take other forms. Thus, the invention may be practiced with other computer system configurations.

[0107] While the methods and systems have been described in connection with preferred embodiments and specific examples, it is not intended that the scope be limited to the specific embodiments, as the embodiments herein are intended in all respects to be illustrative and not restrictive.

[0108] Unless expressly stated otherwise, the methods described herein do not require that their operations be performed in a particular order. Thus, unless a method claim actually recites the order in which its operations are to be followed, or unless the claims or specification specifically recite that the operations are limited to a particular order, no order is intended to be inferred in any manner. This applies to any possible non-expressive basis for interpretation, including logical issues regarding the arrangement of steps or operational flow, the simple meaning derived from grammatical constructions and punctuation, or the number or type of embodiments described in the specification.

[0109] As will be apparent to those skilled in the art, various modifications and variations can be made without departing from the scope or spirit of the present disclosure. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice described herein. It is intended that the specification and illustrative drawings be considered as exemplary only, with its true scope and spirit being indicated by the following claims.

Claims

1. 1. A method for automatically and efficiently generating music audio content for video, comprising: receiving a video from a user, the video comprising a plurality of frame segments; extracting information from the video, the information including information indicative of a rate of motion within the video, information indicative of a saliency of motion within the video, information indicative of scene transitions within the video, and timing information associated with the video; generating a plurality of sets of musical notes matching the plurality of frame segments based at least in part on the extracted information using a pre-trained model that learns to correlate video motion speed with musical note density, to correlate video motion saliency with musical note intensity, to correlate video scene transitions with musical composition, and to correlate video timing with musical beat; generating a plurality of vectors corresponding to the plurality of sets of musical notes, each of the plurality of vectors indicative of at least one musical feature of one of the plurality of musical audios; generating the plurality of music audios corresponding to the plurality of frame segments based at least in part on the plurality of vectors; The method includes:

2. obtaining at least one template comprising at least one audio file based on each of the plurality of vectors; generating musical audio corresponding to each of the plurality of frame segments based at least in part on the at least one audio file; The method of claim 1 further comprising:

3. Obtaining at least one template based on each of the plurality of vectors includes: retrieving at least one template from at least one database containing a plurality of templates, each of the plurality of templates including an audio file having a particular musical characteristic; The method of claim 2 further comprising:

4. Generating musical audio corresponding to each of the plurality of frame segments based at least in part on the at least one audio file includes: modifying the at least one audio file by adding a melody based on one of the plurality of musical note sets. The method of claim 2 further comprising:

5. the at least one musical feature of one of the plurality of musical audios includes at least one of a musical style, a bar structure, or an instrument; The method of claim 1.

6. generating music audio content for the video based at least in part on synthesizing the plurality of music audios, the music audio content matching motion, intensity, and transitions in the video. The method of claim 1.

7. Determining whether the user likes the music audio content based on user input. The method of claim 6 further comprising:

8. presenting a plurality of options in response to the user determining that the music audio content is not to the user's liking; updating the music audio content based on one or more options selected by the user from the plurality of options; The method of claim 7 further comprising:

9. applying video warping during generation of the music audio content to fine-tune video audio matching between the video and the music audio content; The method of claim 6 further comprising:

10. 1. A system for automatically and efficiently generating music audio content for a video, comprising: at least one processor; and at least one memory communicatively coupled to the at least one processor and storing computer readable instructions, the system comprising: The computer readable instructions, when executed by the at least one processor, cause the at least one processor to: receiving a video from a user, the video comprising a plurality of frame segments; extracting information from the video, the information including information indicative of a rate of motion within the video, information indicative of a saliency of motion within the video, information indicative of scene transitions within the video, and timing information associated with the video; generating a plurality of sets of musical notes matching the plurality of frame segments based at least in part on the extracted information using a pre-trained model that learns to correlate video motion speed with musical note density, to correlate video motion saliency with musical note intensity, to correlate video scene transitions with musical composition, and to correlate video timing with musical beats; generating a plurality of vectors corresponding to the plurality of sets of musical notes, each of the plurality of vectors indicative of at least one musical feature of one of the plurality of musical audios; generating the plurality of music audios corresponding to the plurality of frame segments based at least in part on the plurality of vectors; Execute an operation including system.

11. The operation comprises: obtaining at least one template comprising at least one audio file based on each of the plurality of vectors; generating musical audio corresponding to each of the plurality of frame segments based at least in part on the at least one audio file; The system of claim 10 further comprising:

12. Obtaining at least one template based on each of the plurality of vectors includes: retrieving at least one template from at least one database containing a plurality of templates, each of the plurality of templates including an audio file having a particular musical characteristic; The system of claim 11 further comprising:

13. Generating musical audio corresponding to each of the plurality of frame segments based at least in part on the at least one audio file includes: modifying the at least one audio file by adding a melody based on one of the plurality of musical note sets. The system of claim 11 further comprising:

14. the at least one musical feature of one of the plurality of musical audios includes at least one of a musical style, a bar structure, or an instrument; The system of claim 10.

15. The operation comprises: generating music audio content for the video based at least in part on synthesizing the plurality of music audios, the music audio content matching motion, intensity, and transitions in the video. The system of claim 10.

16. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, The computer readable instructions, when executed by a processor, cause the processor to: receiving a video from a user, the video comprising a plurality of frame segments; extracting information from the video, the information including information indicative of a rate of motion within the video, information indicative of a saliency of motion within the video, information indicative of scene transitions within the video, and timing information associated with the video; generating a plurality of sets of musical notes matching the plurality of frame segments based at least in part on the extracted information using a pre-trained model that learns to correlate video motion speed with musical note density, to correlate video motion saliency with musical note intensity, to correlate video scene transitions with musical composition, and to correlate video timing with musical beats; generating a plurality of vectors corresponding to the plurality of sets of musical notes, each of the plurality of vectors indicative of at least one musical feature of one of the plurality of musical audios; generating the plurality of music audios corresponding to the plurality of frame segments based at least in part on the plurality of vectors; Execute an operation that includes A non-transitory computer-readable storage medium.

17. The operation comprises: obtaining at least one template comprising at least one audio file based on each of the plurality of vectors; generating musical audio corresponding to each of the plurality of frame segments based at least in part on the at least one audio file; 20. The non-transitory computer-readable storage medium of claim 16, further comprising:

18. Obtaining at least one template based on each of the plurality of vectors includes: retrieving at least one template from at least one database containing a plurality of templates, each of the plurality of templates including an audio file having a particular musical characteristic; 20. The non-transitory computer readable storage medium of claim 17, further comprising:

19. Generating musical audio corresponding to each of the plurality of frame segments based at least in part on the at least one audio file includes: modifying the at least one audio file by adding a melody based on one of the plurality of musical note sets.

20. The non-transitory computer readable storage medium of claim 17, further comprising:

20. The operation comprises: generating music audio content for the video based at least in part on synthesizing the plurality of music audios, the music audio content matching motion, intensity, and transitions in the video.

20. The non-transitory computer-readable storage medium of claim 16.

Images