Information processing device, learning device, and image generation method

Abstract

Efficiently generates visuals that match the image of the music. [Solution] The information processing device 1 comprises an acquisition unit 11 that acquires music data DM representing a song, an estimation unit 12 that uses a trained model MJ that has learned the relationship between the features of a song and the features of an image to estimate the features of an image corresponding to the song represented by the music data DM at predetermined intervals, and a generation unit 13 that generates video data DMV representing video synchronized with the song represented by the music data DM based on the estimation results of the estimation unit 12, wherein the predetermined period is an integer multiple of the beat duration of the song represented by the music data DM.

Claims

1. An acquisition unit that acquires song data indicating a song, An estimation unit that uses a pre-trained model that has learned the relationship between the features of a song and the features of an image, to estimate the features of an image corresponding to the song shown in the song data at predetermined intervals, based on the features of the song shown in the song data. The system includes a generation unit that generates video data showing video synchronized with the music indicated by the music data, based on the estimation results of the estimation unit, The predetermined period is an integer multiple of the beat duration of the music indicated by the music data. Information processing device.

2. The predetermined period is a period that is longer than or equal to the period corresponding to the measure of the music indicated in the music data. The information processing apparatus according to claim 1.

3. The predetermined period is an integer multiple of the period corresponding to the measure of the music indicated in the music data. The information processing apparatus according to claim 2.

4. The estimation unit changes the length of the predetermined period. The information processing apparatus according to claim 1 or 2.

5. The estimation unit changes the length of the predetermined period according to the rest period of the music indicated by the music data. The information processing apparatus according to claim 4.

6. The songs shown in the aforementioned music data include vocals. The estimation unit changes the length of the predetermined period based on the sentences, phrases, or words of the song lyrics indicated by the song data. The information processing apparatus according to claim 4.

7. The estimation unit, A musical score feature extraction unit 12d extracts the feature quantities of the musical score of the song shown in the aforementioned music data at predetermined intervals, Includes a lyric feature extraction unit that extracts the characteristic features of the song lyrics shown in the aforementioned song data at predetermined intervals, The aforementioned trained model is a machine learning model that analyzes the relationship between the features of musical scores, lyrics, and images. The information processing apparatus according to claim 1 or 2.

8. The estimation unit includes a music analysis unit that generates musical score data representing the musical score of the music indicated by the music data by analyzing the music data. The music analysis unit generates period data indicating the predetermined period based on the musical score data. The information processing apparatus according to claim 1 or 2.

9. The aforementioned trained model is a machine learning model that has learned the relationship between the features of music, the features of images, and the features of attributes. The estimation unit uses the trained model to estimate the feature quantities of the image corresponding to the music indicated by the music data at predetermined intervals, based on the music data and attribute data indicating the attributes of the music indicated by the music data. The information processing apparatus according to claim 1 or 2.

10. The acquisition unit is capable of acquiring the attribute data in addition to the music data. The information processing apparatus according to claim 9.

11. The generation unit adjusts the timing of the video switching indicated by the video data according to the rest periods of the music indicated by the music data. The information processing apparatus according to claim 1 or 2.

12. An acquisition unit that acquires multiple music video data showing a song and video synchronized with that song, The system includes a learning unit that generates a trained model by machine learning the relationship between song features and image features over a predetermined period of time based on the aforementioned multiple music video data, The predetermined period is an integer multiple of the beat duration of the song shown in each of the multiple music video data. Learning device.

13. Obtaining song data that identifies a song, Using a pre-trained model that has been machine-learned to determine the relationship between song features and image features, the features of the image corresponding to the song represented by the song data are estimated at predetermined intervals based on the song data. This includes generating video data that shows video synchronized with the music indicated by the music data, based on the estimated features, The predetermined period is an integer multiple of the beat duration of the music indicated by the music data. Video generation method.

Images