Information processing device, learning device, and image generation method

Abstract

Efficiently generates visuals that match the image of the music. [Solution] The information processing device 1 comprises an acquisition unit 11 that acquires music data DM representing a song, an estimation unit 12 that uses a trained model MJ that has learned the relationship between the features of a song and the features of an image to estimate the features of an image corresponding to the song represented by the music data DM at predetermined intervals, and a generation unit 13 that generates video data DMV representing video synchronized with the song represented by the music data DM based on the estimation results of the estimation unit 12, wherein the predetermined period is an integer multiple of the beat duration of the song represented by the music data DM.

Claims

1. An acquisition unit that acquires music data indicating a song including performance and singing, A separation unit separates vocal data representing the singing voice included in the song indicated by the aforementioned song data from the aforementioned song data, A speech recognition unit generates lyric data as text data describing the lyrics of the song indicated by the song data, based on the vocal data. A music analysis unit that analyzes the aforementioned music data to generate musical score data representing the musical score of the music indicated by the music data, and generates period data representing a predetermined period that is an integer multiple of the beat duration of the music indicated by the music data, based on the musical score data and the lyrics data, A music score feature extraction unit generates music score feature data by extracting the feature quantities of the music score for each predetermined period based on the period data and the music score data. A lyric feature extraction unit generates lyric feature data by extracting emotional feature quantities expressed in the lyrics, as shown in the lyric data, for each predetermined period, based on the aforementioned period data and the lyric data. An input / output unit that uses a trained model that has been machine-learned to determine the relationship between the features of the musical score and the emotional features associated with each predetermined period, and the features of the image, to estimate the features of the image corresponding to the musical piece shown in the musical piece data for each predetermined period, based on the musical score feature data and the lyric feature data. A generation unit generates video data that shows video synchronized with the music indicated by the music data, based on the image feature quantities estimated by the input / output unit at predetermined intervals, An information processing device equipped with the following features.

2. The information processing apparatus according to claim 1, wherein the predetermined period is a period that is longer than or equal to the period corresponding to the measure of the music indicated by the music data.

3. The information processing apparatus according to claim 2, wherein the predetermined period is an integer multiple of the period corresponding to the measure of the music indicated by the music data.

4. The information processing apparatus according to claim 1 or 2, wherein the music analysis unit generates the period data so as to change the length of the predetermined period.

5. The information processing apparatus according to claim 4, wherein the music analysis unit generates the period data such that the length of the predetermined period changes according to the rest period of the music indicated by the music data.

6. The information processing apparatus according to claim 4, wherein the music analysis unit generates period data such that the length of the predetermined period changes based on the sentences, phrases, or words of the lyrics indicated by the lyrics data.

7. The information processing apparatus according to claim 1 or 2, wherein the lyric feature extraction unit extracts a feature relating to at least one of the emotions of joy, anger, sadness, and happiness that can be expressed in the lyrics shown in the lyric data, as the emotion feature.

8. The aforementioned trained model is a machine learning model that has learned the relationship between the features of music, the features of images, and the features of attributes. The information processing apparatus according to claim 1 or 2, wherein the input / output unit uses the trained model to estimate the feature quantities of an image corresponding to the music indicated by the music data at predetermined intervals, based on the musical score feature quantity data and the lyric feature quantity data, as well as attribute data indicating the attributes of the music indicated by the music data.

9. The information processing apparatus according to claim 8, wherein the acquisition unit acquires the attribute data in addition to the music data.

10. The information processing apparatus according to claim 1 or 2, wherein the generation unit adjusts the timing of video switching indicated by the video data according to the rest period of the music indicated by the music data.

11. An acquisition unit that acquires a plurality of music video data showing a song including performance and singing and video synchronized with the song, A separation unit separates music data representing the song, vocal data representing the singing voice, and video data representing the video from each of the aforementioned multiple music video data. A speech recognition unit generates lyric data as text data describing the lyrics of the song indicated by the song data, based on the vocal data. A music analysis unit that analyzes the aforementioned music data to generate musical score data representing the musical score of the music indicated by the music data, and generates period data representing a predetermined period that is an integer multiple of the beat duration of the music indicated by the music data, based on the musical score data and the lyrics data, A music score feature extraction unit generates music score feature data by extracting the features of the music score for each predetermined period based on the period data and the music score data. A lyric feature extraction unit generates lyric feature data by extracting emotional feature quantities expressed in the lyrics shown in the lyric data for each predetermined period based on the period data and the lyric data. Includes an image feature extraction unit that generates image feature data by extracting image features from the video data at predetermined intervals based on the aforementioned period data and the aforementioned video data, A training unit generates a trained model by machine learning the relationship between the musical score features, lyrics features, and image features associated for each predetermined period, based on the aforementioned musical score feature data, lyrics feature data, and image feature data. A learning device equipped with the following features.

12. One or more processors acquire music data indicating a song including performance and singing, The one or more processors separate vocal data representing the singing voice included in the song indicated by the song data from the song data, The one or more processors generate lyric data as text data describing the lyrics of the song indicated by the song data, based on the vocal data. The one or more processors analyze the music data to generate musical score data representing the musical score of the music indicated by the music data, and generate period data representing a predetermined period that is an integer multiple of the beat duration of the music indicated by the music data, The one or more processors generate musical score feature data by extracting the musical score feature quantities for each predetermined period based on the period data and the musical score data. The one or more processors generate lyric feature data by extracting emotional feature quantities expressed in the lyrics, as shown in the lyric data, for each predetermined period, based on the period data and the lyric data. The one or more processors use a trained model that has learned the relationship between the musical score features and emotional features associated with each predetermined period and the image features to estimate the image features corresponding to the music shown in the music data for each predetermined period, based on the musical score feature data and the lyric feature data. The one or more processors generate video data that shows video synchronized with the music indicated by the music data, based on the estimated image features. A method for generating video, including the above.

Images