Information processing device, information processing method, and program

Abstract

Provided is an information processing device comprising: a first encoder that extracts a first feature quantity from text data; a second encoder that extracts a second feature quantity from musical composition data; and a similarity determination unit that identifies a musical composition corresponding to the text data on the basis of the similarity between the first feature quantity and the second feature quantity.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] Various technologies for assisting song search and song selection have been proposed. For example, Patent Literature 1 discloses a technology for categorizing songs using a learning model that has learned the relationship between lyric data and song category data. Patent Literature 2 discloses a technology for presenting songs that are close to keywords selected by a user using a correlation value between a song and keywords associated with the song.

[0003] JP 2010-165160 A JP 2004-326840 A

[0004] On the other hand, there is a demand for more advanced music search and selection, especially among DJs. Specifically, when a DJ creates a playlist, he or she needs to consider various factors, such as the purpose and time of the event where the music will be played, and the attributes of the main audience. For example, with simple filtering based on indicators such as genre and tempo using technologies such as those described in Patent Literature 1 and Patent Literature 2, or with music recommendation using similar music detection technology, it is difficult to automatically generate a playlist by selecting music that takes into account the above-mentioned factors.

[0005] Therefore, an object of the present invention is to provide an information processing device, an information processing method, and a program that enable music selection that better reflects the user's intentions.

[0006] [1] An information processing device comprising: a first encoder that extracts first features from text data; a second encoder that extracts second features from music data; and a similarity determination unit that identifies a song corresponding to the text data based on the similarity between the first features and the second features. [2] The information processing device according to [1], wherein the first encoder and the second encoder are trained by contrastive learning using a loss function based on the similarity between the first features and the second features for the same song. [3] The information processing device according to [1] or [2], wherein the text data is acquired using an interactive user interface. [4] The information processing device according to any one of [1] to [3], further comprising a playlist generation unit that generates a playlist using the identified songs. [5] An information processing method including: a step of extracting first features from text data by a first encoder; a step of extracting second features from music data by a second encoder; and a step of identifying a song corresponding to the text data based on the similarity between the first features and the second features. [6] A program for causing a computer to function as a first encoder that extracts a first feature from text data, a second encoder that extracts a second feature from music data, and a similarity determination unit that identifies a music piece corresponding to the text data based on the similarity between the first feature and the second feature.

[0007] FIG. 1 is a diagram showing the overall configuration of a system according to an embodiment of the present invention. FIG. 2 is a block diagram showing the functional configuration of a smartphone in the example of FIG. 1. FIG. 3 is a diagram showing an example of the processing of an encoder learning unit in FIG. 2. FIG. 4 is a diagram showing an example of contrastive learning in FIG. 3. FIG. 5 is a diagram showing an example of the processing of a similarity determination unit in FIG. 2. FIG. 6 is a diagram conceptually showing an example of a user experience in a system according to an embodiment of the present invention. FIG. 7 is a diagram showing an example of an interactive user interface in FIG.

[0008] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0009] 1 is a diagram showing the overall configuration of a system according to an embodiment of the present invention. In a system 10 according to this embodiment, a smartphone 120, a PC (Personal Computer) 130, speakers 141 and 142, a microphone 150, and headphones 160 are connected to a DJ controller 110, which is an example of DJ equipment, either wired or wirelessly. For example, a USB (Universal Serial Bus) cable or an audio cable is used for the wired connection, and for example, Bluetooth (registered trademark) is used for the wireless connection.

[0010] The peripheral devices shown in the figure are illustrative and not limiting. For example, if the performance is not being streamed, the PC 130 may not be connected. Alternatively, regardless of whether a performance is being streamed or not, the PC 130 may be connected instead of the smartphone 120, or the smartphone 120 may be replaced by another terminal device such as a tablet. Furthermore, the microphone 150 and headphones 160 may be integrated into a headset.

[0011] In the system 10 described above, the smartphone 120 stores, processes, and plays back music audio data in response to user input acquired through controls on the DJ controller 110 (described below). The smartphone 120 includes a display that displays information to the user. The smartphone 120 may also acquire user input directly through an input device such as a touch panel. The played back audio data is output, for example, from speakers 141 and 142, or distributed from the PC 130 via the Internet or the like. The microphone 150 is used for audio input during a performance, and the headphones 160 are used for monitoring.

[0012] FIG. 2 is a block diagram showing the functional configuration of the smartphone in the example of FIG. 1 . In the system 10, the smartphone 120 functions as an information processing device that displays information visualizing the characteristics of music. In other embodiments, a PC 130 connected to the DJ controller 110, or a server connected to the smartphone 120 or the PC 130 via a network, may have similar functions instead of the smartphone 120. In still another example, DJ equipment itself, such as a mixer or an all-in-one DJ system (a digital audio player with communication and mixing functions), may have similar functions. The functions of the information processing device described below are realized by a processor, such as a CPU, included in these devices operating in accordance with a program stored in memory.

[0013] In the illustrated example, the smartphone 120 includes, as functional components, a learning data acquisition unit 121, an encoder learning unit 122, a text data acquisition unit 123, a music data acquisition unit 124, a similarity determination unit 125, and a playlist generation unit 126. Details of these functions are described below. While functional components 120A, 120B, and 120C for each processing step are illustrated, these functional components 120A, 120B, and 120C may be implemented in a single device such as the smartphone 120, or some or all of them may be implemented in separate devices, specifically, for example, a server connected via a network. The remaining functional components of the smartphone 120 are similar to those of a typical smartphone used in connection with DJ equipment, and therefore will not be described in detail.

[0014] Functional configuration 120A is a part in which encoder learning unit 122 learns text encoder 122A (first encoder) and music encoder 122B (second encoder) using data acquired by learning data acquisition unit 121. Learning data acquisition unit 121 acquires text data TXT related to music and music data AUD. The text data TXT and music data AUD acquired by learning data acquisition unit 121 are associated with each other.

[0015] The text data TXT acquired here represents information about the music piece, such as genre, tempo, atmosphere, effects, artist, era, instrumentation, emotion, and reviews, and may be acquired using, for example, an interactive user interface. In this case, a large-scale language model may be used to extract the text data TXT used for training from text entered by the user. The text data TXT may be acquired in real time when processing such as that described below is performed, or data acquired and stored in advance may be read. Meanwhile, the music piece data AUD may be the audio signal of the music piece, or data such as a spectrogram obtained by analyzing the audio signal of the music piece. This data may be stored, for example, in the storage of the smartphone 120, the storage of another device connected to the smartphone 120 via USB or a network, or a removable recording medium attached to the smartphone 120.

[0016] The encoder learning unit 122 learns the text encoder 122A and the music encoder 122B based on the text data TXT and the music data AUD acquired by the learning data acquisition unit 121. More specifically, as shown in Fig. 3, contrastive learning is performed based on the similarity between the text feature vector Vt obtained by inputting the text data TXT to the text encoder 122A, and the music feature vector Va obtained by inputting the music data AUD to the music encoder 122B. For example, as shown in Fig. 4, the contrastive learning involves the use of the text feature vector Vt １ , Vt ２ , Vt ３ , ..., Vt Ｎ and the song feature vector Va １ , Va ２ , Va ３ , ..., Va Ｎ The similarity Va of the combination of feature vectors for the same song among the combinations of １ Vt １ , Va ２ Vt ２ , Va ３ Vt ３ , ..., VaＮ Vt Ｎ We use a loss function that maximizes the similarity of the feature vectors for different songs, and minimizes the similarity of the combinations of feature vectors for different songs.

[0017] Functional configuration 120B is a part that uses trained text encoder 122A and music encoder 122B to identify music data corresponding to newly input text data. In functional configuration 120B, text data acquisition unit 123 acquires text data TXT related to the newly input music. The text data TXT may be acquired using an interactive user interface and a large-scale language model, similar to the text data acquired by training data acquisition unit 121. The text data TXT acquired by text data acquisition unit 123 is related to music but is not associated with a specific music. Such text data TXT may express, for example, genre, tempo, atmosphere, effects, artist, era, instrumentation, emotion, reviews, and the like, which a DJ needs when creating a playlist.

[0018] The song data acquisition unit 124 acquires song data AUD. The acquired song data AUD may be, for example, data similar to that acquired by the training data acquisition unit 121, but is not associated with the text data TXT acquired by the text data acquisition unit 123. The song data acquisition unit 124 may acquire the same data as the song data AUD acquired by the training data acquisition unit 121, but the text data acquisition unit 123 does not acquire the text data TXT associated with that song data AUD. The song data acquisition unit 124 may select songs from which to acquire song data AUD from among all available songs, based on tag information extracted from a dialogue with a user by the same large-scale language model used to acquire the text data TXT by the text data acquisition unit 123, for example.

[0019] As shown in FIG. 5, the similarity determination unit 125 determines whether the text data TXT acquired by the text data acquisition unit 123 is a ＸThe text feature vector Vt is the first feature extracted by the text encoder 122A (trained model) from Ｘ and the music feature vector Va, which is a second feature quantity extracted by the music encoder 122B (trained model) from the music data AUD acquired by the music data acquisition unit 124, Ｘ More specifically, the similarity determination unit 125 identifies one or more songs corresponding to the text feature vector Vt Ｘ The degree of association between the music feature vector Va and the music feature vector Va is measured based on the vector distance, and the music associated with the music feature vector Va having the highest degree of association is stored as text data TXT Ｘ Identify the song as corresponding to

[0020] Functional configuration 120C is a part in which the playlist generation unit 126 generates a playlist from the songs identified by the similarity determination unit 125. In functional configuration 120C, the playlist generation unit 126 generates a playlist including one or more songs identified by the similarity determination unit 125. For example, the similarity determination unit 125 may identify multiple songs for one piece of text data acquired by the text data acquisition unit 123, and the playlist generation unit 126 may generate a playlist including these songs. Alternatively, the similarity determination unit 125 may identify one or more songs for each of multiple pieces of text data acquired sequentially by the text data acquisition unit 123 through a dialogue with a user using a large-scale language model, for example, and the playlist generation unit 126 may generate a playlist in which these songs are arranged in the order in which the text data was acquired.

[0021] FIG. 6 conceptually illustrates an example of a user experience in a system according to an embodiment of the present invention. Once trained models for the text encoder 122A and the music encoder 122B are prepared, a user, such as a DJ, can generate a playlist based on text data. More specifically, as illustrated, a user U can specify the use and purpose of the playlist in the form of a dialogue in an interactive user interface (interactive UI) (step S101). The interactive UI may request additional information from the user U if any information is missing (step S102). An example of such an interactive UI is shown in FIG. 7.

[0022] Based on the information obtained through the above-described dialogue, a parameter list for the song can be estimated using a large-scale language model (step S103). The parameter list includes tag information such as artist, genre, and tempo, as well as descriptive text not classified as tag information. By filtering the song database using the tag information, songs whose song data will be acquired by the song data acquisition unit 124 can be selected (step S104). Meanwhile, the descriptive text becomes text data acquired by the text data acquisition unit 123 (step S105).

[0023] The similarity determination unit 125 (text / music multimodal model) selects songs that match the description based on the song data and text data (step S106). The playlist generation unit 126 generates a playlist based on the selected songs (step S107). The playlist generated in this manner reflects the intentions expressed by the user U through the dialogue. The generated playlist may be displayed to the user U (step S108), allowing the user U to refer to it and further modify the playlist through dialogue.

[0024] In the embodiment of the present invention described above, a first encoder (text encoder 122A) that extracts features from text data and a second encoder (music encoder 122B) that extracts second features from music data are trained, and a song corresponding to the text data is identified based on the similarity of the features extracted by these encoders. By learning the relationship between the text data and the features of the songs in advance, music selection that better reflects the intention expressed by the user through the text data becomes possible. The encoders can be trained, for example, by contrastive learning using a loss function based on the similarity of the features for the same songs. Furthermore, by acquiring text data using an interactive user interface and a large-scale language model, the user's intention can be acquired in a more natural manner. The identified songs can be used, for example, to generate playlists for DJs.

[0025] 10...System, 110...DJ controller, 120...Smartphone, 120A, 120B, 120C...Functional configuration, 121...Learning data acquisition unit, 122...Encoder learning unit, 122A...Text encoder, 122B...Music encoder, 123...Text data acquisition unit, 124...Music data acquisition unit, 125...Similarity determination unit, 126...Playlist generation unit, 141...Speaker, 142...Speaker, 150...Microphone, 160...Headphones.

Claims

1. An information processing device comprising: a first encoder that extracts a first feature from text data; a second encoder that extracts a second feature from music data; and a similarity determination unit that identifies a music piece corresponding to the text data based on the similarity between the first feature and the second feature.

2. The information processing device according to claim 1, wherein the first encoder and the second encoder are trained by contrastive learning using a loss function based on the similarity between the first feature and the second feature for the same song.

3. The information processing device according to claim 1 or 2, wherein the text data is acquired using an interactive user interface.

4. The information processing device according to any one of claims 1 to 3, further comprising a playlist generating unit that generates a playlist using the identified music pieces.

5. An information processing method comprising: a step of a first encoder extracting a first feature from text data; a step of a second encoder extracting a second feature from music data; and a step of identifying a music piece corresponding to the text data based on the similarity between the first feature and the second feature.

6. A program for causing a computer to function as a first encoder that extracts a first feature from text data, a second encoder that extracts a second feature from music data, and a similarity determination unit that identifies a music piece corresponding to the text data based on the similarity between the first feature and the second feature.

Images