Information processing system, information processing method, and terminal device

Abstract

An information processing system according to the present disclosure comprises: an acquisition unit that acquires first lyrics to be searched for; a feature amount calculation unit that calculates a first feature amount on the basis of the first lyrics; a similarity degree calculation unit that calculates the similarity degree between the first feature amount and each of a plurality of second feature amounts calculated in advance on the basis of each of a plurality of second lyrics; and an output unit that outputs information indicating the plurality of second lyrics according to the similarity degree. The plurality of second feature amounts are calculated by using a learning model trained so as to maximize the similarity degree between arbitrarily-defined lyrics and each of a plurality of lyrics generated on the basis of the arbitrarily-defined lyrics and a prescribed instruction.

Description

Information processing system, information processing method, and terminal device

[0001] The present disclosure relates to an information processing system, an information processing method, and a terminal device.

[0002] A similar lyrics search system has been proposed that searches for similar songs based on their lyrics.

[0003] Haven Kim, 3 others, "A COMPUTATIONAL EVALUATION FRAMEWORK FOR SINGABLE LYRIC TRANSLATION", arXiv:2308.13715v1, August 23, 2023 Nils Reimers, 1 other, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", arXiv:1908.10084v1, August 27, 2019

[0004] However, conventional similar lyric search systems lack a variety of similarity axes, making it difficult to consider comprehensive similarity.

[0005] Therefore, an object of the present disclosure is to provide an information processing system, an information processing method, and a terminal device that are capable of searching for similar songs based on lyrics while taking into consideration overall similarity.

[0006] The information processing system according to the present disclosure includes an acquisition unit that acquires first lyrics to be searched for; a feature calculation unit that calculates a first feature based on the first lyrics; a similarity calculation unit that calculates a similarity between the first feature and each of a plurality of second feature values ​​calculated in advance based on a plurality of second lyrics; and an output unit that outputs information indicating the plurality of second lyrics in accordance with the similarity, wherein the plurality of second feature values ​​are calculated using a learning model that is trained to maximize a similarity between any lyrics and each of a plurality of lyrics generated based on the any lyrics and predetermined instructions.

[0007] FIG. 1 is a schematic diagram illustrating an overview of similar lyrics search. FIG. 2 is a schematic diagram illustrating an overview of another example of similar lyrics search. FIG. 3 is a schematic diagram illustrating an example of the configuration of a similar lyrics search system applicable to an embodiment. FIG. 4 is a block diagram illustrating the hardware configuration of an example of a server applicable to an embodiment. FIG. 5 is a block diagram illustrating the hardware configuration of an example of a user terminal applicable to an embodiment. FIG. 6 is a functional block diagram illustrating an example of the function of a server according to an embodiment. FIG. 7 is a functional block diagram illustrating an example of the function of a user terminal applicable to an embodiment. FIG. 8 is a schematic diagram illustrating lyric similarity index modeling applicable to an embodiment. FIG. 9 is a schematic diagram illustrating a similar lyrics search model according to an embodiment. FIG. 10 is a flowchart illustrating an example of similar lyrics search processing according to an embodiment.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0009] Hereinafter, embodiments of the present disclosure will be described in the following order: 1. Overview of the present disclosure 1-1. Overview of similar lyric search 1-2. Regarding existing technology 2. Embodiments of the present disclosure 2-1. Configuration applicable to the embodiment 2-2. Functional configuration according to the embodiment 2-3. Processing according to the embodiment 2-3-1. Lyric similarity index modeling 2-3-2. Similar lyric search model 2-3-2-1. Building a learning model for similar lyric search 2-3-2-2. Example of similar lyric search processing 2-3-2-3. Regarding evaluation of similar lyric search processing

[0010] (1. Overview of the Present Disclosure) The present disclosure will be briefly described. The present disclosure relates to a technology for searching existing songs for lyrics similar to lyrics entered by a user, and providing the user with the searched lyrics and songs corresponding to the lyrics.

[0011] More specifically, a similar lyrics search system according to the present disclosure acquires first lyrics to be searched and calculates a first feature based on the acquired first lyrics. The similar lyrics search system calculates the similarity between the calculated first feature and each of a plurality of second feature values ​​calculated in advance based on a plurality of second lyrics. The similar lyrics search system outputs information indicating the plurality of second lyrics according to the similarity. Here, the similar lyrics search system according to the present disclosure calculates the plurality of second feature values ​​using a learning model trained to maximize the similarity between any lyrics and each of a plurality of lyrics generated based on the any lyrics and predetermined instructions.

[0012] According to the similar lyrics search system of the present disclosure, in order to determine the similarity between a first lyric to be searched and a plurality of second lyrics, a plurality of second feature quantities corresponding to each of the plurality of second lyrics are calculated using a learning model based on one arbitrary lyric and a plurality of lyrics generated based on predetermined instructions. As a result, it is possible to train the learning model for calculating the second feature quantities at a lower cost.

[0013] Furthermore, the predetermined prompt for generating lyrics to be used to calculate the second features may include an instruction to generate at least some lyrics that are similar in meaning, style, and rhyme to a given lyric. Therefore, the similar lyrics search system according to the present disclosure can generate the lyrics based on a wider variety of similarity axes, thereby enabling the similarity between the first lyrics to be searched and the second lyrics to be calculated based on a wider variety of similarity axes.

[0014] (1-1. Overview of Similar Lyrics Search) Fig. 1 is a schematic diagram for explaining an overview of similar lyrics search. In Fig. 1, an example of a music screen 50 is shown on the left side, and an example of a search result screen 51 is shown on the right side. The music screen 50 and the search result screen 51 are displayed on the screen of, for example, a terminal device (user terminal) used by a user.

[0015] On the music screen 50, an area 500 is an area where music information about a music piece selected to be playable is displayed. The user can play the music piece displayed in the area 500 in response to a predetermined operation on the user terminal, for example.

[0016] On the music screen 50, an area 501 displays commands that can be executed for the music displayed in area 500. In the example of FIG. 1 , area 501 includes the commands "Liked," "Add to Playlist," "Add to Queue," "Search Songs with Similar Lyrics," "(View Artists), "Share," and "Go to Song Radio." By specifying the command "Search Songs with Similar Lyrics" displayed in area 501 (indicated by a frame 502 in the figure) through a user operation, the user can search for music that contains lyrics similar to those contained in the music displayed in area 500.

[0017] When the command "Search Songs with Similar Lyrics" is specified in response to a user operation, the user terminal transmits song information indicating the songs displayed in area 500 to the server device via a communication network such as the Internet.

[0018] The server device searches existing songs for songs that contain lyrics similar to lyrics included in the song information transmitted from the user terminal. The server device transmits a list of song information indicating songs obtained as a result of the search to the user terminal. At this time, the server device may sort the song information included in the list according to the similarity of the lyrics included in the song indicated by the song information to the lyrics of the search target song.

[0019] The user terminal displays the list of search results transmitted from the server device in the list display area 510 of the search result screen 51. For example, the user terminal may display song information in the list display area 510 in descending order of lyric similarity. The user terminal can play a song specified by a user operation from among the songs displayed in the list.

[0020] 2 is a schematic diagram for explaining an overview of another example of similar lyrics search. The search screen 52 shown in Fig. 2 shows an example of a UI (User Interface) for searching existing songs for songs that contain lyrics similar to lyrics defined by the user.

[0021] On the search screen 52, the input area 520 is an area for inputting user-defined lyrics, for example, as text data. User-defined lyrics may be, for example, lyrics written by the user. However, the user-defined lyrics may be any text data, such as a diary or poem written by the user. Existing lyrics may also be input as user-defined lyrics. Furthermore, the user-defined lyrics may be part of the lyrics of a song. By operating the registration button 521, the text data input in the input area 520 is transmitted to the server device.

[0022] On the search screen 52, a list display area 522 displays a list of song information as search results transmitted from the server device in response to input to the input area 520. Each piece of song information displayed in the list display area 522 can be played sequentially or randomly by operating a play button 523.

[0023] (1-2. Existing Technology) When users search for their favorite songs, it is believed that there are a certain number of cases in which lyrics are important. For this reason, several technologies have been proposed for searching for songs based on similarities in lyrics.

[0024] On the other hand, there are various possible similarity axes for the similarity of lyrics, such as (A) similarity in the style of the music containing the target lyrics, (B) similarity in the meaning of the lyrics at the sentence level, and (C) similarity in the pronunciation of the lyrics. Note that the style of the music may indicate primarily the auditory tendencies of the music, such as the genre or mood of the music. Furthermore, the similarity in the pronunciation of the lyrics may be the "rhyme" that is brought about by the pronunciation of the lyrics.

[0025] However, existing technology for similarity index modeling of lyrics has not been designed to take into account human cognition, even though it has been able to determine similarity individually for these various similarity axes.

[0026] Therefore, this disclosure proposes a similar lyrics search system that can perform song searches based on lyrics in accordance with human cognition by determining the similarity of lyrics by comprehensively considering these various similarity axes.

[0027] (2. Embodiments of the Present Disclosure) Next, embodiments of the present disclosure will be described.

[0028] (2-1. Configuration Applicable to the Embodiment) First, a configuration applicable to the embodiment will be described. Fig. 3 is a schematic diagram showing the configuration of an example of a similar lyrics search system applicable to the embodiment.

[0029] In FIG. 3, an information processing system 1 as a similar lyrics search system according to the embodiment includes a server 30 and a DB (database) 31, and is configured by connecting a user terminal 10 to the server 30 via a communication network 2 such as the Internet.

[0030] DB31 stores lyric features calculated in advance for existing songs in association with song information. Server 30 calculates features based on the lyrics information of the lyrics to be searched, transmitted from user terminal 10, and calculates similarities between the calculated features and each feature stored in DB31. Server 30 returns each calculated similarity and song information corresponding to each similarity to user terminal 10 that transmitted the lyrics information to be searched. User terminal 10 is equipped with, for example, a web browser function, and presents each piece of song information returned from server 30 to the user via the web browser function.

[0031] 3, the server 30 and DB 31 are configured on a cloud network 3 connected to a communication network 2. However, the server 30 may be configured as a single computer device, or may be configured in a distributed manner by a plurality of computer devices that are connected to each other so as to be able to communicate with each other.

[0032] The user terminal 10 may be a general personal computer, a smartphone, a tablet computer, etc. However, the user terminal 10 is not limited to this and may be a game console or a head mounted display (HMD) that has a function of communicating with the communication network 2, an audio playback function, and a web browser function.

[0033] 4 is a block diagram showing an example of the hardware configuration of the server 30 applicable to the embodiment. In the example of FIG. 4, for the sake of explanation, the server 30 is shown as being configured by a single computer.

[0034] In FIG. 4, the server 30 includes a CPU (Central Processing Unit) 3000, a ROM (Read Only Memory) 3001, a RAM (Random Access Memory) 3002, a storage device 3004, a data interface (I / F) 3005, and a communication I / F 3006, and these components are connected to each other via a bus 3010 so that they can communicate with each other.

[0035] The storage device 3004 is a non-volatile storage medium such as a flash memory, a hard disk drive, etc. The CPU 3000 controls the overall operation of the server 30 in accordance with the programs stored in the storage device 3004 and the ROM 3001, and uses the RAM 3002 as a work memory.

[0036] The data I / F 3005 is an interface for communicating with an external device. The server 30 may transmit and receive data to and from the DB 31 via the data I / F 3005. The communication I / F 3006 controls communication with the communication network 2.

[0037] The server 30 may be connected to an input device that accepts user operations, and a display and display control unit that presents information to the user.

[0038] 5 is a block diagram showing an example of the hardware configuration of a user terminal 10 applicable to the embodiment. In the example of FIG. 5, for the sake of explanation, the user terminal 10 is shown as a general personal computer.

[0039] In Figure 5, the user terminal 10 includes a CPU 1000, a ROM 1001, a RAM 1002, a storage device 1004, a data I / F 1005, a communication I / F 1006, and a display control unit 1007, and each of these units is connected to each other via a bus 1030 so that they can communicate with each other.

[0040] The storage device 1004 is a non-volatile storage medium such as a flash memory or a hard disk drive. The CPU 1000 controls the overall operation of the user terminal 10 in accordance with the programs stored in the storage device 1004 and the ROM 1001, using the RAM 1002 as a work memory.

[0041] The data I / F 1005 is an interface for communicating with external devices. In the example of Fig. 5, an input device 1020 and an audio I / F 1021 are connected to the data I / F 1005. The input device 1020 is a device for accepting user operations, and may be a pointing device such as a mouse, a keyboard, or the like.

[0042] The audio I / F 1021 has a function of converting a digital audio signal supplied from the data I / F 1005 into an analog audio signal. The analog audio signal output from the audio I / F 1021 may be amplified by an amplifier and may drive, for example, left and right speakers 1022L and 1022R. This allows the digital audio signal generated by, for example, the CPU 1000 in accordance with a program to be output as sound.

[0043] The display control unit 1007 generates a display signal that can be displayed on the display 1010, based on display information generated by the CPU 1000 in accordance with a program. The display 1010 has a display device, such as a display panel using an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode), and a drive circuit that drives the display device in accordance with the display signal. The display 1010 displays a screen in accordance with the display signal supplied from the display control unit 1007. This allows the display 1010 to display a screen in accordance with the program. Note that the display 1010 may be a touch panel formed integrally with the input device 1020.

[0044] (2-2. Functional Configuration According to the Embodiment) Next, the functional configuration according to the embodiment will be described.

[0045] FIG. 6 is a functional block diagram illustrating an example of the functions of the server 30 according to the embodiment.

[0046] 6, the server 30 includes a control unit 300, a communication unit 301, a UI control unit 302, a lyrics information acquisition unit 303, a feature calculation unit 304, and a similarity calculation unit 305. The server 30 is also connected to a feature DB 310 corresponding to the DB 31 described above.

[0047] The control unit 300, communication unit 301, UI control unit 302, lyric information acquisition unit 303, feature amount calculation unit 304, and similarity calculation unit 305 may be configured by an information processing program according to the embodiment running on the CPU 3000. However, some or all of the control unit 300, communication unit 301, UI control unit 302, lyric information acquisition unit 303, feature amount calculation unit 304, and similarity calculation unit 305 may be configured by hardware circuits that operate in cooperation with each other.

[0048] 6 , a control unit 300 controls the overall operation of the server 30. A communication unit 301 controls communication over the communication network 2. The server 30 transmits and receives data to and from, for example, the user terminal 10 under the control of the communication unit 301. A UI control unit 302 generates a UI to be provided to the user terminal 10 and controls the generated UI. The UI control unit 302 generates control information for displaying, for example, the music screen 50 and search result screen 51 shown in FIG. 1 or the search screen 52 shown in FIG. 2 on the user terminal 10.

[0049] For the sake of explanation, the information processing system 1 according to this embodiment will be described below as using the music screen 50 and search result screen 51, or the search screen 52 as UIs to perform similar lyric searches in response to user operations.

[0050] The lyrics information acquisition unit 303 acquires lyrics to be searched.

[0051] 1 is displayed on the user terminal 10, the lyric information acquisition unit 303 acquires the lyrics of the song displayed in the area 500 of the song screen 50. For example, the lyric information acquisition unit 303 may acquire song information indicating the song via the communication network 2. If lyrics are included in the attribute information added to or included in the acquired song information, the lyric information acquisition unit 303 may acquire the lyrics. Alternatively, the lyric information acquisition unit 303 may acquire the lyrics of the song from an external server or the like via the communication network 2. The lyric information acquisition unit 303 may also acquire the lyrics of the song by extracting vocal audio from the audio data of the song and analyzing the extracted vocal audio.

[0052] When the search screen 52 shown in Figure 2 is displayed on the user terminal 10, the lyrics information acquisition unit 303 regards the text data entered in the input area 520 of the search screen 52 as lyrics and acquires it via the communication network 2.

[0053] The lyric information acquisition unit 303 may also collect and acquire lyrics of existing songs. For example, the lyric information acquisition unit 303 may collect existing songs and their lyrics via the communication network 2. For example, existing songs may also be prepared in advance. The lyric information acquisition unit 303 may collect and acquire lyrics from existing songs that have been prepared in advance.

[0054] The feature calculation unit 304 calculates feature amounts of the lyrics acquired by the lyric information acquisition unit 303. For example, the feature calculation unit 304 calculates feature amounts for the lyrics acquired by the lyric information acquisition unit 303 using a pre-trained learning model. A learning method for the learning model will be described later. The feature amounts are expressed, for example, as multidimensional vectors. The feature calculation unit 304 may store the calculated feature amounts in the feature DB 310 in association with music information.

[0055] The similarity calculation unit 305 calculates the similarity between the feature calculated based on the lyrics acquired from the user terminal 10 by the lyrics information acquisition unit 303 and each of the feature stored in the feature DB 310. The method used by the similarity calculation unit 305 to calculate the similarity is not particularly limited, and may be, for example, a cosine similarity method.

[0056] The similarity calculation unit 305 passes to the UI control unit 302 each similarity calculated for each feature stored in the feature DB 310 and calculated based on the lyrics acquired from the user terminal 10 by the lyric information acquisition unit 303. The UI control unit 302 associates each similarity passed from the similarity calculation unit 305 with music information. The UI control unit 302 generates display control information for displaying a list of pairs of associated similarities and music information. At this time, the UI control unit 302 preferably sorts each pair according to their similarity to generate the list. The UI control unit 302 arranges pairs in the list in descending order of similarity, for example, from most similar to least similar.

[0057] FIG. 7 is a functional block diagram illustrating an example of functions of the user terminal 10 applicable to the embodiment.

[0058] 7, the user terminal 10 includes a control unit 100, a communication unit 101, and a UI unit 102. The control unit 100, communication unit 101, and UI unit 102 may be realized by, for example, an operating system (OS) and an application program pre-installed in the user terminal 10. For example, the control unit 100 and communication unit 101 may be configured by an OS pre-installed in the user terminal 10. Furthermore, the UI unit 102 may be configured by a web browser application (hereinafter abbreviated as a browser) that is generally installed in the user terminal 10.

[0059] Alternatively, the control unit 100, the communication unit 101, and the UI unit 102 may be partially or entirely configured by hardware circuits that operate in cooperation with each other. The UI unit 102 may also be configured by a program according to the embodiment.

[0060] The control unit 100 controls the overall operation of the user terminal 10. The communication unit 101 controls communication with the communication network 2. The user terminal 10 transmits and receives data to and from the server 30, for example, under the control of the communication unit 101.

[0061] The UI unit 102 generates display information for displaying, for example, the music screen 50 and the search result screen 51 shown in Fig. 1 or the search screen 52 shown in Fig. 2 on the display 1010, based on a control signal transmitted from the server 30, for example. The UI unit 102 also generates control information in response to a user operation performed on the input device 1020 in response to, for example, the music screen 50 and the search result screen 51, or the search screen 52. The UI unit 102 may transmit the control information generated in response to the user operation to the server 30.

[0062] In the server 30, the CPU 3000 executes the information processing program according to the embodiment, thereby configuring the control unit 300, communication unit 301, UI control unit 302, lyric information acquisition unit 303, feature calculation unit 304, and similarity calculation unit 305, for example, as modules in the main memory area of ​​the RAM 3002.

[0063] The information processing program can be obtained from outside via the communication network 2, for example, by communication via the communication I / F 3006, or can be obtained from a storage medium connected to the data I / F 3005 and installed on the server 30.

[0064] (2-3. Processing According to the Embodiment) Next, the processing according to the embodiment will be described in more detail.

[0065] (2-3-1. Lyric Similarity Index Modeling) Lyric similarity index modeling applicable to the embodiment will be described. As described above, the similarity of lyrics is considered to include various similarity axes as follows: (A) Similarity in style of music that includes the target lyrics (B) Similarity in meaning at the sentence level of lyrics (C) Similarity in pronunciation of lyrics

[0066] Fig. 8 is a schematic diagram illustrating lyric similarity index modeling applicable to the embodiment. Fig. 8 illustrates a case where the similarity between songs 43a and 43b is evaluated based on the similarity axes (A) to (C) described above. Song 43a includes audio data 44a and lyrics 45a. Similarly, song 43b includes audio data 44b and lyrics 45b.

[0067] (A) Similarity in style of songs containing target lyrics. This similarity indicating the similarity (A) may be calculated by applying an audio tagging CNN (Convolutional Neural Network) 40 to the audio data 44a and 44b of each song 43a and 43b. The audio tagging CNN 40 is a CNN configured and trained to tag audio data according to genre and mood. As such a CNN, musiCNN, a set of pre-trained convolutional neural networks for tagging music audio, can be used. In the following description, the audio tagging CNN 40 is assumed to be musiCNN.

[0068] The calculation of the similarity in style of music containing target lyrics will be explained using the following equation (1).

[0069]

[0070] Equation (1) calculates the similarity in terms of music style. Note that in the following, letters with a tilde (~) immediately above them to indicate a tilde matrix are represented by adding a tilde (~), such as "x~." The symbols "X" and "X~," as well as "T" and "T~," each represent a multidimensional vector. "R" represents the set of all real numbers.

[0071] Style is often reflected in genre and affects perceptual similarity. That is, lyrics of the same genre tend to have the same style characteristics. To quantify this, an example is known in which audio tracks (audio data) T and T~ corresponding to lyrics X and X~, respectively, are used. The style similarity (Sim) is calculated by calculating the cosine similarity between audio tracks T and T~, which is the penultimate feature derived from musiCNN. sty ) and st (T) and st (T~)∈R 200 This value shows a reasonable correlation with human perception, e.g., a Pearson correlation of 0.48 (p=0.03).

[0072] (B) Similarity of meaning at the sentence level of lyrics The similarity indicating the similarity of (B) may be calculated by applying a natural language processing model 41 to the lyrics 45a and 45b of each of the songs 43a and 43b. The natural language processing model 41 is a model related to technology for processing human written and spoken words by a computer. As the natural language processing model 41, sentence-BERT, which is a type of natural language processing model called BERT (Bidirectional Encoder Representations from Transformers), can be applied. The following description will be given assuming that the natural language processing model 41 is sentence-BERT.

[0073] The calculation of the similarity of the meaning of the target lyrics will be explained using the following equation (2).

[0074]

[0075] Semantics (meaning) plays an important role in perceptual similarity in any kind of text. To illustrate this in the context of lyric similarity, a pre-trained sentence-BERT model, all-MiniLM-L6-v2, is employed. For a pair of lyrics X and X~, the respective embeddings obtained from this model are se (X) and se The cosine similarity of (X~) is used to calculate the semantic similarity (Sim sem ) This index correlates well with human perception, with a Pearson correlation of e.g. 0.65 (p=0.00).

[0076] (C) Similarity of Lyric Pronunciation This similarity (C) may be calculated by performing phoneme analysis 42 on the lyrics 45a and 45b of each song 43a and 43b. The sounds of lyrics spoken are recognized as a factor that significantly influences perceptual lyric similarity. To objectively measure the sound similarity between a pair of lyrics, there is a known example of using a mathematical formula originally introduced to evaluate the "ease of singing" of translated lyrics. This example shows that differences in sounds in lyrics are influenced primarily by two factors: the degree of repetition and the choice of phonemes.

[0077] The calculation of the pronunciation similarity of the target lyrics will be explained using the following equation (3).

[0078]

[0079] To assess the degree of repetition, in this example, lyric X is broken down into phonemes, organized into bigrams, and the ratio pho(X) of the unique bigram count to the total bigram count is calculated. This ratio pho(X) is inversely proportional to the degree of repetition; a smaller value indicates more repetition, and a larger value indicates less repetition. The difference in the degree of repetition between two lyric sets, X and X~, is calculated by taking the difference between the ratio pho(X) and the ratio pho(X~).

[0080] Furthermore, to identify differences in the use of phonemes between lyrics X and X~, the ratio pho is calculated for the exclusive OR (X(EX-OR)X~) of the concatenated lyrics X and X~, with a small value indicating a high degree of phoneme overlap. Note that in formula (3), the exclusive OR is indicated by a "+" (plus) in a "○" (circle). The difference in sounds between the lyrics set of two lyrics X and X~, Diff, sou is determined by the sum of the lyric pho value (pho(X(EX-OR)X~)) combined with the repetition degree (|pho(X)-pho(X~)|). The Pearson correlation for this method is 0.74 (p=0.00), which is a stronger correlation with human perception than methods (A) and (B) above.

[0081] (2-3-2. Similar Lyrics Search Model) Next, the similar lyrics search model according to the embodiment will be described in more detail. As described above, when searching for similar lyrics, it is possible to conduct searches based on a variety of similarity axes, such as (A) similarity in the genre or mood of the music containing the target lyrics, (B) similarity in the meaning of the lyrics at the sentence level, and (C) similarity in the pronunciation of the lyrics. However, with existing technologies, searches are only performed based on a single similarity axis, such as only the vocabulary contained in the lyrics, only the topic expressed by the lyrics, only the meaning of the sentences that make up the lyrics, or only the mood of the music containing the lyrics, and comprehensive similarity is not taken into consideration.

[0082] In the embodiment, a similar lyrics search model is proposed that enables similar lyrics search taking into account these various similarity axes.

[0083] (2-3-2-1. Building a learning model for similar lyrics search) Generally, a learning model for searching for similar lyrics is built by learning from a large number of lyrics. In the similar lyrics search model according to this embodiment, a large language model (LLM) is used to generate multiple lyrics based on one arbitrary lyric. The similar lyrics search model according to this embodiment trains a learning model for searching for similar lyrics using the multiple lyrics that have been generated and the lyrics used to generate the multiple lyrics.

[0084] In the following description, it is assumed that the information processing system 1 according to the embodiment trains a learning model for searching for similar lyrics. In the following description, unless otherwise specified, the "learning model for searching for similar lyrics" will be referred to simply as the "learning model" as appropriate.

[0085] 9 is a schematic diagram illustrating a similar lyrics search model according to the embodiment. In the information processing system 1 according to the embodiment, the server 30, for example, acquires one lyric 60 using the lyric information acquisition unit 303 shown in FIG. 6 . The lyric 60 is not particularly limited and may be any lyric. The server 30 generates new lyrics 62 based on the lyrics 60 using an LLM 61 using the lyric information acquisition unit 303. Hereinafter, the lyrics 60 will be referred to as seed lyrics 60, and the lyrics 62 will be referred to as generated lyrics 62.

[0086] When the server 30 generates the generated lyrics 62 using the LLM 61 based on the seed lyrics 60, the server 30 specifies a prompt 63 to the LLM 61 to indicate elements to be included in the generated lyrics 62. In an embodiment, the prompt 63 may include the following content:

[0087] (a) Generate lyrics that are similar in terms of style, meaning, and rhyme; (b) Avoid word-for-word paraphrases; (c) Add creativity; and (d) Avoid inappropriate comments.

[0088] By instructing (a) "Generate lyrics similar in terms of style, meaning, and rhyme" in the prompt 63, the style, meaning, and rhyme elements of the seed lyrics 60 can be included in the generated lyrics 62. In other words, by instructing (a) "Generate lyrics similar in terms of style, meaning, and rhyme," it is possible to construct training data that enables searches that take into account comprehensive similarity based on various similarity axes. This enables searches that are in line with human cognition.

[0089] By instructing the prompt 63 to (b) "avoid word-for-word paraphrasing" and (c) "add creativity," it is possible to generate generated lyrics 62 with greater diversity. That is, in the similar lyrics retrieval model, a learning model is trained using pairs of seed lyrics 60 and generated lyrics 62. In this case, if generated lyrics 62 are generated using word-for-word paraphrasing of the seed lyrics 60, the problem becomes too simple and does not reflect reality very well, making it difficult to build training data with useful value. By instructing the LLM 61 to avoid word-for-word paraphrasing of the seed lyrics 60 and to generate lyrics with an added creativity, it is possible to generate generated lyrics 62 with greater diversity and higher value.

[0090] The instruction (d) "Do not include inappropriate comments" in the prompt 63 can suppress the output of comments that are not included in the generated lyrics 62, such as "Lyrics have been generated" or "What do you think of these lyrics?" after the generated lyrics 62 have been generated. In other words, if the generated lyrics 62 contain such inappropriate comments, it becomes difficult to execute an automatic process such as repeatedly generating generated lyrics 62. By including this instruction (d) "Do not include inappropriate comments" in the prompt 63, it is possible to ensure that only lyrics are output.

[0091] For example, the server 30 uses the feature calculation unit 304 to calculate features for the seed lyrics 60 and the generated lyrics 62 using a natural language processing model 64. BERT may be applied as the natural language processing model 64. For example, the information processing system 1 calculates features for the seed lyrics 60 using a natural language processing model 64a, and calculates features for the generated lyrics 62 using a natural language processing model 64b. The features calculated here are, for example, multidimensional vectors.

[0092] The feature calculation unit 304 may use one natural language processing model 64 in common as the natural language processing models 64a and 64b.

[0093] The feature calculation unit 304 performs pooling processing 65a, for example, using average pooling, on the output of the natural language processing model 64a to calculate a feature U based on one vector from the features based on multidimensional vectors. Similarly, the feature calculation unit 304 performs pooling processing 65b, for example, using average pooling, on the output of the natural language processing model 64b to calculate a feature V based on one vector from the features based on multidimensional vectors.

[0094] In the server 30, the similarity calculation unit 305 calculates the similarity (U, V) between the feature U output by the pooling process 65a and the feature V output by the pooling process 65b, for example, using the cosine similarity 66. The similarity calculation unit 305 trains the natural language processing model 64 so that the calculated similarity (U, V) becomes larger, that is, so that the similarity between the feature U and the feature V becomes higher. As a result, the parameters of both the natural language processing models 64a and 64b are updated.

[0095] At this time, the similarity calculation unit 305 may, in accordance with the calculated similarity (U, V), control the learning of the natural language processing model 64. For example, the similarity calculation unit 305 compares the similarity (U, V) with a threshold, and trains the natural language processing model 64 so that the feature quantity (U, V) becomes larger when the similarity (U, V) is equal to or larger than the threshold, and so that the feature quantity (U, V) becomes smaller when the similarity (U, V) is less than the threshold.

[0096] 9 , the feature calculation unit 304 may use the LLM 61 to generate a large number of generated lyrics 62 from one seed lyric 60. For example, the feature calculation unit 304 may use the LLM 61 to generate tens of thousands to a million different generated lyrics 62 based on one seed lyric 60, and calculate a feature V for each generated lyric 62 using a natural language processing model 64b and pooling process 65b. The similarity calculation unit 305 calculates the similarity (U, V) of the feature U based on the seed lyric 60 for each feature V generated based on each generated lyric 62, and trains the natural language processing model 64 using the calculated similarities (U, V).

[0097] According to the learning method of the embodiment, the similarity calculation unit 305 can train the learning model in the similar lyrics retrieval model without using human instruction (annotation) for lyrics.

[0098] The server 30 calculates the feature value V using the natural language processing model 64 trained as described above based on the seed lyrics 60 and the generated lyrics 62, based on the lyrics of each existing song collected by the lyrics information acquisition unit 303. The server 30 stores the calculated feature value V for the lyrics of each song in the feature value DB 310 in association with song information indicating the song.

[0099] The song information stored in the feature DB 310 may be song data (e.g., audio data) itself indicated by the song information, or may be information indicating the location of the song data (e.g., location on a storage medium, location on the communication network 2, etc.). The song information may also include attribute information of the song indicated by the song information.

[0100] (2-3-2-2. Example of similar lyrics search processing) Based on the lyrics transmitted from the user terminal 10, the server 30 uses, for example, the natural language processing model 64 trained as described above to search for songs containing lyrics similar to the lyrics of existing songs.

[0101] 10 is a flowchart illustrating an example of the similar lyrics search process according to the embodiment. Note that the similar lyrics search process is performed using the music screen 50 and search result screen 51 described with reference to FIG. 1, or the search screen 52 described with reference to FIG. 2.

[0102] 10, in step S10, the server 30 causes the lyrics information acquisition unit 303 to acquire lyrics to be searched in response to a user operation.

[0103] 1 is displayed on the user terminal 10, the lyrics information acquisition unit 303 acquires the lyrics of the song displayed in the area 500 of the song screen 50. At this time, the lyrics information acquisition unit 303 may acquire all of the lyrics of the song as the lyrics to be searched, or may acquire a range of the lyrics of the song specified by the user (for example, the chorus part) as the lyrics to be searched.

[0104] 2 is displayed on the user terminal 10, the lyrics information acquisition unit 303 acquires text data entered in the input area 520 of the search screen 52, regarding it as lyrics. At this time, the content of the text data entered in the input area 520 is not particularly limited.

[0105] In the next step S11, the server 30 calculates a feature amount U' of the lyrics acquired in step S10 using the feature amount calculation unit 304. If the lyrics are those of the song displayed in area 500 of the music screen 50 and the song information and feature amounts of the song are already stored in the feature amount DB 310, the feature amount calculation unit 304 may acquire the feature amount of the song stored in the feature amount DB 310 as the feature amount U'.

[0106] In the next step S12, the similarity calculation unit 305 acquires the calculated lyric feature V' from the feature DB 310. At this time, the similarity calculation unit 305 may acquire the feature V' by narrowing down the feature stored in the feature DB 310 according to conditions (such as genre) specified by a user operation on the user terminal 10, for example.

[0107] In the next step S13 , the similarity calculation unit 305 calculates the similarity (U′, V′) between the feature U′ of the lyrics to be searched and the feature V′ acquired from the feature DB 310 .

[0108] In the next step S14, the similarity calculation unit 305 determines whether or not similarities (U', V') have been calculated for all feature quantities V' stored in the feature quantity DB 310. If the similarity calculation unit 305 determines that a feature quantity V' for which a similarity (U', V') has not been calculated exists among the feature quantities V' stored in the feature quantity DB 310 (step S14, "No"), the similarity calculation unit 305 returns the process to step S12, acquires the next feature quantity V' for which a similarity (U', V') has not been calculated among the feature quantities V' stored in the feature quantity DB 310, and calculates the similarity (U', V').

[0109] On the other hand, if the similarity calculation unit 305 determines in step S14 that the similarity (U', V') has been calculated for all features V' stored in the feature DB 310 (step S14, "Yes"), it transitions the processing to step S15.

[0110] In step S15, the similarity calculation unit 305 obtains lyrics from the feature DB 310 and sorts the obtained lyrics in order of similarity. In the next step S16, the similarity calculation unit 305 creates a list in which music information corresponding to each lyric is arranged in the sorting order in step S15. The similarity calculation unit 305 creates the list of music information by arranging the lyrics in order of decreasing similarity (U', V'), for example.

[0111] The similarity calculation unit 305 transmits the created list to the user terminal 10. Based on the list transmitted from the server 30, the user terminal 10 displays, for example, the search result screen 51 shown in Fig. 1 or the search screen 52 shown in Fig. 2. At this time, the similarity calculation unit 305 may create the list for lyrics stored in the feature DB 310 that have a similarity (U', V') with the lyrics to be searched that is equal to or greater than a predetermined value.

[0112] 10, the processes of steps S12 to S14 are shown as being executed serially for each feature V' stored in the feature DB 310, but this is for the sake of explanation and is not limited to this example. That is, in reality, the processes of steps S12 to S14 for each feature V' may be executed in parallel for each feature V'. For example, the similarity calculation unit 305 assigns a natural language processing model 64 to each feature V' by duplication or the like, and executes the processes for each feature V' in parallel.

[0113] Furthermore, the similarity (U', V') may be calculated for each paragraph of lyrics. That is, since one lyric contains multiple paragraphs, a feature V' is calculated for each paragraph. In this case, the similarity (U', V') may be calculated and similar lyrics may be detected at the paragraph level. However, it is also possible to calculate a single feature V' by averaging multiple feature V' calculated for each paragraph, and use this feature V' to calculate the similarity (U', V') and detect similar lyrics at the music level.

[0114] (2-3-2-3. Regarding Evaluation of Similar Lyrics Search Process) The similarity calculation unit 305 may evaluate the similarity (U', V') when sorting the lyrics in order of similarity (U', V') in step S15. The similarity calculation unit 305 may modify the similarity (U', V') depending on the evaluation result. More specifically, the similarity calculation unit 305 can evaluate the similarity (U', V') based on similarity axes such as (A) similarity in style of music containing the target lyrics, (B) similarity in meaning at the sentence level of the lyrics, and (C) similarity in pronunciation of the lyrics, as described with reference to FIG. 8 .

[0115] For example, the similarity calculation unit 305 may calculate the similarity in style of a song containing the lyrics of the target (A) based on audio data of the song containing the lyrics to be searched and audio data of the song corresponding to the feature V' stored in the feature DB 310. For example, if the similarity (U', V') between the lyrics to be searched and the lyrics corresponding to a certain feature V' stored in the feature DB 310 is "0.7" and the similarity in style of the song is "0.8", the similarity calculation unit 305 may correct the similarity (U', V') from "0.7" to "0.8".

[0116] The similarity (U', V') based on this evaluation result can also be applied to the sentence-level semantic similarity of lyrics (B) and the pronunciation similarity of lyrics (C). The similarity calculation unit 305 may combine multiple evaluation results to modify the similarity (U', V').

[0117] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0118] Note that the present technology can also be configured as follows: (1) An information processing system including: an acquisition unit that acquires first lyrics to be searched; a feature calculation unit that calculates a first feature based on the first lyrics; a similarity calculation unit that calculates a similarity between the first feature and each of a plurality of second feature values ​​calculated in advance based on each of a plurality of second lyrics; and an output unit that outputs information indicating the plurality of second lyrics in accordance with the similarity, wherein the plurality of second feature values ​​are calculated using a learning model that is trained to maximize the similarity between any lyrics and each of a plurality of lyrics generated based on the any lyrics and predetermined instructions. (2) The information processing system described in (1), wherein the predetermined instructions include an instruction to generate at least some lyrics similar in meaning, style, and rhyme to the any lyrics. (3) The information processing system described in (2), wherein the predetermined instructions further include an instruction to generate lyrics while avoiding paraphrasing words contained in the any lyrics. (4) The information processing system according to (2) or (3), wherein the predetermined instructions further include an instruction to add creativity to the arbitrary lyrics. (5) The information processing system according to any one of (2) to (4), wherein the predetermined instructions further include an instruction to prohibit the generation of unnecessary comments on the generated lyrics. (6) The information processing system according to any one of (1) to (5), wherein the plurality of second lyrics are generated using a large-scale language model based on the arbitrary lyrics and the predetermined instructions, and the predetermined instructions are prompts for the large-scale language model. (7) The information processing system according to any one of (1) to (6), wherein the plurality of second lyrics are each lyrics of an existing song, and the output unit outputs information of the existing song to which the plurality of second lyrics correspond in accordance with the degree of similarity. (8) The information processing system according to any one of (1) to (7), wherein the similarity calculation unit calculates a similarity between a genre of music corresponding to the first lyrics and a genre of music corresponding to each of the plurality of second lyrics, and further uses the calculated similarity to calculate a similarity between the first feature and the plurality of second feature.(9) The information processing system according to any one of (1) to (8), wherein the similarity calculation unit calculates a similarity between a meaning of the first lyrics and a meaning of each of the plurality of second lyrics, and further uses the similarity to calculate a similarity between the first feature amount and the plurality of second feature amounts. (10) The information processing system according to any one of (1) to (9), wherein the similarity calculation unit calculates a similarity between a phoneme included in the first lyrics and a phoneme included in each of the plurality of second lyrics, and further uses the similarity to calculate a similarity between the first feature amount and the plurality of second feature amounts. (11) An information processing method, executed by a computer, comprising: an acquisition step of acquiring first lyrics to be searched; a feature calculation step of calculating a first feature based on the first lyrics; a similarity calculation step of calculating a similarity between the first feature and each of a plurality of second feature amounts calculated in advance based on each of a plurality of second lyrics; and an output step of outputting information indicating the plurality of second lyrics in accordance with the similarity, wherein the plurality of second feature amounts are calculated using a learning model trained to maximize a similarity between any lyrics and each of a plurality of lyrics generated based on the any lyrics and predetermined instructions. (12) A terminal device comprising: a communication unit that communicates with an information processing device; a transmission unit that transmits first lyrics to be searched, input in response to a user operation, to the information processing device; and an output unit that outputs information indicating a plurality of second lyrics transmitted from the information processing device in response to the first lyrics transmitted by the transmission unit, wherein the output unit outputs the information indicating the plurality of second lyrics transmitted from the information processing device in response to information indicating the similarity between a first feature calculated based on the first lyrics transmitted from the information processing device and each of a plurality of second feature amounts calculated in advance using a learning model trained to maximize the similarity between arbitrary lyrics based on each of the plurality of second lyrics and each of a plurality of lyrics generated based on the arbitrary lyrics and predetermined instructions.

[0119] REFERENCE SIGNS LIST 1 Information processing system 2 Communication network 3 Cloud network 10 User terminal 30 Server 31 DB 40 Audio tag CNN 41, 64, 64a, 64b Natural language processing model 42 Phoneme analysis 43a, 43b Music 44a, 44b Audio data 45a, 45b Lyrics 50 Music screen 51 Search result screen 52 Search screen 60 Seed lyrics 61LLM 62 Generated lyrics 63 Prompt 65a, 65b Pooling process 100, 300 Control unit 101, 301 Communication unit 102 UI unit 302 UI control unit 303 Lyric information acquisition unit 304 Feature calculation unit 305 Similarity calculation unit 310 Feature DB

Claims

1. An information processing system comprising: an acquisition unit that acquires first lyrics to be searched; a feature calculation unit that calculates a first feature based on the first lyrics; a similarity calculation unit that calculates the similarity between the first feature and each of a plurality of second feature values ​​calculated in advance based on each of a plurality of second lyrics; and an output unit that outputs information indicating the plurality of second lyrics in accordance with the similarity, wherein the plurality of second feature values ​​are calculated using a learning model that has been trained to maximize the similarity between any lyrics and each of a plurality of lyrics generated based on the any lyrics and predetermined instructions.

2. The information processing system according to claim 1, wherein the predetermined instructions include instructions to generate at least a portion of lyrics that are similar in meaning, style, and rhyme to the given lyrics.

3. The information processing system according to claim 2, wherein the predetermined instructions further include instructions to generate lyrics that avoid paraphrasing words contained in the arbitrary lyrics.

4. The information processing system according to claim 2, wherein the predetermined instructions further include instructions for adding creativity to the arbitrary lyrics.

5. The information processing system according to claim 2, wherein the predetermined instructions further include an instruction to prohibit the generation of unnecessary comments on the lyrics to be generated.

6. The information processing system of claim 1, wherein the plurality of second lyrics are generated using a large-scale language model based on the arbitrary lyrics and the predetermined instructions, and the predetermined instructions are prompts for the large-scale language model.

7. The information processing system according to claim 1, wherein the plurality of second lyrics are lyrics of existing songs, and the output unit outputs information about the existing songs to which the plurality of second lyrics correspond according to the degree of similarity.

8. The information processing system of claim 1, wherein the similarity calculation unit calculates the similarity between the genre of the music corresponding to the first lyrics and the genre of each of the music corresponding to each of the plurality of second lyrics, and further uses the similarity to calculate the similarity between the first feature and the plurality of second feature.

9. The information processing system of claim 1, wherein the similarity calculation unit calculates the similarity between the meaning of the first lyrics and the meaning of each of the plurality of second lyrics, and further uses the similarity to calculate the similarity between the first feature and the plurality of second feature.

10. The information processing system of claim 1, wherein the similarity calculation unit calculates the similarity between a phoneme contained in the first lyrics and a phoneme contained in each of the plurality of second lyrics, and further uses the similarity to calculate the similarity between the first feature and the plurality of second feature.

11. An information processing method executed by a computer, comprising: an acquisition step of acquiring first lyrics to be searched; a feature calculation step of calculating a first feature based on the first lyrics; a similarity calculation step of calculating a similarity between the first feature and each of a plurality of second feature values ​​calculated in advance based on a plurality of second lyrics; and an output step of outputting information indicating the plurality of second lyrics in accordance with the similarity, wherein the plurality of second feature values ​​are calculated using a learning model trained to maximize the similarity between any lyrics and each of a plurality of lyrics generated based on the any lyrics and predetermined instructions.

12. A terminal device comprising: a communication unit that communicates with an information processing device; a transmission unit that transmits first lyrics to be searched, input in response to user operation, to the information processing device; and an output unit that outputs information indicating a plurality of second lyrics transmitted from the information processing device in response to the first lyrics transmitted by the transmission unit, wherein the output unit outputs the information indicating the plurality of second lyrics transmitted from the information processing device in response to information indicating the similarity between a first feature calculated based on the first lyrics transmitted from the information processing device and each of a plurality of second feature values ​​calculated in advance using a learning model trained to maximize the similarity between arbitrary lyrics based on each of the plurality of second lyrics, and each of the arbitrary lyrics and a plurality of lyrics generated based on predetermined instructions.

Images