Information processing system, information processing method, and terminal device

Abstract

An information processing system according to an embodiment includes: an acquisition unit that acquires first musical composition data, and a presentation unit that presents information indicating similarity of the first musical composition data to second musical composition data as existing musical composition data, in which the presentation unit causes a screen including a display of a first form indicating a similarity for each of a plurality of items for evaluating the similarity to musical composition data to be displayed.

Description

Field

[0001] The present disclosure relates to an information processing system, an information processing method, and a terminal device.Background

[0002] Similar music retrieval is known as one of musical composition data processing techniques. For example, Patent Document 1 discloses a technique of searching for other musical composition data including a partial track that is similar to part of a music track (musical composition data) selected by a user, in temporal change.Citation ListPatent Literature

[0003] Patent Literature 1: WO 2024 / 042962 A1SummaryTechnical Problem

[0004] In the similar music retrieval, there is a demand for a technique further facilitating grasping of similarity of other musical composition data to target musical composition data.

[0005] Therefore, an object of the present disclosure is to provide an information processing system, an information processing method, and a terminal device that further facilitate grasping of similarity of other musical composition data to target musical composition data. Solution to Problem

[0006] For solving the problem described above, a information processing system according to one aspect of the present disclosure has an acquisition unit that acquires first musical composition data; and a presentation unit that presents information indicating similarity of the first musical composition data to second musical composition data as existing musical composition data, wherein the presentation unit causes a screen including a display of a first form indicating a similarity for each of a plurality of items for evaluating the similarity to musical composition data to be displayed.Brief Description of Drawings

[0007] FIG. 1 is a schematic diagram illustrating an exemplary configuration of an information processing system according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram illustrating an example of a UI screen indicating a result of search of second musical composition data by a server, displayed on a terminal device, according to an embodiment. FIG. 3 is a functional block diagram illustrating exemplary functions of a server according to an embodiment. FIG. 4 is a functional block diagram for illustrating functions of an information processing unit in the server according to an embodiment in more detail. FIG. 5 is a functional block diagram illustrating exemplary functions of the terminal device according to an embodiment. FIG. 6 is a block diagram illustrating an exemplary hardware configuration of the server applicable to an embodiment. FIG. 7 is a block diagram illustrating an exemplary hardware configuration of the terminal device applicable to an embodiment. FIG. 8 is a schematic diagram for schematically illustrating functions of the information processing system according to a first embodiment. FIG. 9 is a schematic diagram for illustrating a learning process of a search model according to the first embodiment. FIG. 10 is a flowchart illustrating an exemplary search process according to the first embodiment. FIG. 11 is a flowchart illustrating exemplary display control of the UI screen according to the first embodiment. FIG. 12A is a schematic diagram illustrating an example of a UI screen having first and second forms according to the first embodiment. FIG. 12B is a schematic diagram illustrating an example of a UI screen having the first and second forms according to the first embodiment. FIG. 12C is a schematic diagram illustrating an example of a UI screen having the first and second forms according to the first embodiment. FIG. 13A is a schematic diagram illustrating an example of a UI screen having display of a third form according to the first embodiment. FIG. 13B is a schematic diagram illustrating an example of a UI screen having display of the third form according to the first embodiment. FIG. 13C is a schematic diagram illustrating an example of a UI screen having display of the third form according to the first embodiment. FIG. 13D is a schematic diagram illustrating an example of a UI screen having display of the third form according to the first embodiment. FIG. 14A is a schematic diagram illustrating an example of a UI screen according to a second embodiment. FIG. 14B is a schematic diagram illustrating an example of a UI screen according to the second embodiment. FIG. 15 is a schematic diagram illustrating an example of a UI screen displaying a correspondence relationship of similar portions between first musical composition data and second musical composition data according to the second embodiment. FIG. 16 is a schematic diagram illustrating another example of a UI screen displaying a correspondence relationship of similar portions between the first musical composition data and the second musical composition data according to the second embodiment. FIG. 17 is a schematic diagram illustrating a first example of comparing musical composition data according to the second embodiment. FIG. 18 is a schematic diagram illustrating a second example of comparing musical composition data according to the second embodiment. FIG. 19 is a schematic diagram illustrating a third example of comparing musical composition data according to the second embodiment. FIG. 20 is a flowchart illustrating an example of determination process for a possibility of causing a problem of copyright, according to the second embodiment. Description of Embodiments

[0008] Embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that in the following embodiments, the same portions are denoted by the same reference numerals, and redundant descriptions thereof will be omitted.

[0009] Hereinafter, the embodiments of the present disclosure will be described in the following order. 1. Embodiment of present disclosure 1-1. Overview of embodiment of present disclosure 1-2. Configuration applicable to embodiments 2. First embodiment of present disclosure 2-1. Processing according to first embodiment 2-1-1. Overview of processing according to first embodiment 2-1-2. Specific examples of processing according to first embodiment 2-2. Exemplary UI screen according to first embodiment 2-2-1. UI screen display process according to first embodiment 2-2-2. Specific examples of UI screen according to first embodiment 3. Second embodiment of present disclosure 3-1. Avoidance of copyright infringement by existing technology 3-2. Specific examples of second embodiment 3-2-1. Specific examples of UI screen according to second embodiment 3-2-2. Specific examples of comparison between musical compositions according to second embodiment 3-3. Modifications of second embodiment 4. Third embodiment of present disclosure (1. Embodiment of present disclosure)

[0010] An embodiment of the present disclosure will be described.(1-1. Overview of embodiment of present disclosure)

[0011] First, an embodiment of the present disclosure will be schematically described. In the embodiment of the present disclosure, for example, similarity of first musical composition data of the user to existing second musical composition data is determined, and according to a result of the determination, a screen including of display of a first form that indicates the similarity for each of a plurality of items for evaluating the similarity is presented to the user. In the embodiment of the present disclosure, the similarity of the first musical composition data to the second musical composition data is visualized, and the similarity of the second musical composition data to the first musical composition data can be more intuitively grasped.

[0012] Note that each piece of the musical composition data represents audio data for reproducing music (musical composition). The musical composition data is not limited to this configuration, and may include, in addition to the audio data, metadata (such as song name) for music reproduced by the musical composition data. In the following description, it is assumed that the musical composition data is the audio data unless otherwise specified.

[0013] FIG. 1 is a schematic diagram illustrating an exemplary configuration of an information processing system according to an embodiment of the present disclosure.

[0014] In FIG. 1, an information processing system 1 includes a server 10 and a terminal device 20. The server 10 and the terminal device 20 are communicably connected to each other via a communication network 2 such as the Internet. Note that in FIG. 1, the server 10 is illustrated as being configured in a cloud network 3 connected to the communication network 2. The server 10 is not limited to this configuration, and may be configured by a single computer or may be configured by being distributed to a plurality of computers.

[0015] The terminal device 20 is, for example, a general computer device, and provides a user interface (UI) that receives an operation by a user and presents information to the user by screen display or the like. For example, the terminal device 20 may transmit a command or the like according to the user's operation to the server 10 via the communication network 2. Furthermore, for example, the terminal device 20 may present information transmitted from the server 10, to the user by a display or the like.

[0016] The server 10 is connected to a storage unit 11. The storage unit 11 is a non-volatile storage medium, such as a hard disk drive or flash memory. The storage unit 11 may store a music catalog including a large number of pieces of second musical composition data. The second musical composition data included in the music catalog may be modified in accordance with determination of the similarity described above. Furthermore, the storage unit 11 may store a machine learning model used for analyzing the musical composition data.

[0017] In the configuration as described above, the terminal device 20 transmits the first musical composition data to the server 10, for example, according to the user's operation. The server 10 analyzes the first musical composition data transmitted from the terminal device 20. On the basis of a result of the analysis of the first musical composition data, the server 10 searches the music catalog stored in the storage unit 11 for the second musical composition data including, for example, a portion (similar portion) having a predetermined similarity or more to at least a part of the first musical composition data.

[0018] The server 10 generates a user interface (UI) screen indicating a result of the search for the second musical composition data, and transmits the generated UI screen to the terminal device 20.

[0019] Note that the UI screen may be displayed by a standard browser application program installed on the terminal device 20, on the basis of display control information generated in the server 10. In addition to this configuration, a dedicated application program for causing the UI screen to be displayed may be installed on the terminal device 20, for transmitting only parameters for causing the UI screen to be displayed from the server 10, to the terminal device 20.

[0020] FIG. 2 is a schematic diagram illustrating an example of the UI screen indicating a result of the search for the second musical composition data by the server 10, displayed on the terminal device 20, according to an embodiment.

[0021] In FIG. 2, a waveform display area 510, an upload button 520, and a mode selection area 521 are arranged in an upper area of a UI screen 500. The upload button 520 is a button for transmitting, for example, musical composition data held in the terminal device 20 or accessible from the terminal device 20, to the server 10, as the first musical composition data.

[0022] In the waveform display area 510, a waveform of the first musical composition data transmitted from the terminal device 20 to the server 10 is displayed in time series. In the example of FIG. 2, in the waveform display area 510, information (verse, chorus, intro, solo, pre-chorus) indicating the configuration of the musical composition based on the musical composition data is displayed at an upper end according to a position (time) in the music composition. Furthermore, in the waveform display area 510, information indicating a chord is displayed at a lower end according to a position in the musical composition.

[0023] The mode selection area 521 is an area for selecting a display mode of the screen indicating the result of the search for the second musical composition data, displayed in a lower area of the UI screen 500. In the example of FIG. 2, the mode selection area 521 enables selection of one of three display modes of Similar, Similar & Common, and Similar & Rare. Details of each display mode selectable in the mode selection area 521 will be described later.

[0024] The lower side of the UI screen 500 displays a screen based on the result of the search for the similar portion in the first musical composition data to the second musical composition data. In the embodiment, the result of the search is displayed according to three forms of the first form, a second form, and a third form. Display of the first to third forms is configured using a hierarchical structure so that the first form is a first layer, the second form is a second layer, and the third form is a third layer. Display of the first form being the first layer is arranged in an area 530, display of the second form being the second layer is arranged in an area 540, and display of the third form being the third layer is arranged in an area 550. Note that the position of the area 550 is a position specified by user operation.

[0025] In the first form and the second form, the result of the search is displayed as the similarity for each of the plurality of items. In the embodiment, the plurality of items indicating the result of the search includes at least one of the items of melody, chord, impression, rhythm, lyrics, and overall musical composition. Details of the items will be described later.

[0026] In the display of the first form in the area 530, the number of pieces of second musical composition data including a portion whose similarity to at least a part of the first musical composition data is equal to or larger than a threshold is indicated using the length of a bar 531, for each item. In addition to this configuration, the length of the bar 531 may indicate the overall similarity of the first musical composition data to the second musical composition data. Display of the second form in the area 540 displays a portion having high similarity in the first musical composition data is indicated as a similar portion 541 in time series, for each item.

[0027] In this way, in the embodiment, the similarity to an existing musical composition (second musical composition data) can be visualized by the display of the first and second forms. Furthermore, in the display of the first form, the similarity of the overall first musical composition data is indicated for each item, and in the display of the second form, the similarity of each portion obtained by dividing the first musical composition data in time series is indicated, for each item. Therefore, the user can intuitively grasp how the first musical composition data is similar to the musical composition data of an existing musical composition.

[0028] Furthermore, in the display of the second form, specifying a portion on the time series at which high similarity is indicated in any of the items by user operation or the like displays the area 550 including the display of the third form. In the area 550, a list of the second musical composition data that is the basis of the high similarity in the specified portion is displayed as the display of the third form.

[0029] In the example of FIG. 2, the area 550 displays a waveform of the audio data of a portion of the second musical composition data corresponding to the specified portion of the first musical composition data, the information (melody in the example of FIG. 2) corresponding to the specified item, and an icon indicating the metadata, for each piece of second musical composition data. The area 550 may be configured so that specifying a portion of the waveform enables reproduction of sound related to the waveform. Furthermore, the area 550 may be configured so that specifying a metadata icon enables browsing of the metadata corresponding to the metadata icon. Furthermore, in the example of the drawing, the area 550 is displayed by specifying the similar portion in the melody, and a melody display 560 for the first musical composition data in the specified similar portion may be further displayed.

[0030] In this way, in the embodiment, display of the third form facilitates confirmation of the basis of high similarity to the second musical composition data in the first musical composition data.

[0031] In the embodiment of the present disclosure, use of the UI screen 500 on which the similarity of the musical composition data is visualized for each item facilitates finding the musical composition data of the existing musical composition similar to the first musical composition data. In other words, on the basis of the display of the first form in the area 530 of the UI screen 500, the user can receive comprehensive notification that a similar music to the first musical composition data being a target has been found. Furthermore, on the basis of the display of the second form in the area 540 of the UI screen 500, the user can receive individual notification that a similar music has been found for each position in time series of the first musical composition data being the target. Furthermore, specifying the similar portion 541 in the area 540 by the user enables acquisition of the information about the second musical composition data corresponding to this similar portion 541 and listening to a similar music that is the reason for alert.

[0032] Furthermore, in the embodiment of the present disclosure, use of the UI screen 500 facilitates, for example, finding of the similar portion of the first musical composition data produced by the user to the second musical composition data. Therefore, the user can readily correct the similar portion in the first musical composition data by himself / herself or correct the similar portion by using artificial intelligence (AI). In addition, listening to the second musical composition data similar to the first musical composition data from the display of the third form in the area 550 of the UI screen 500 promises to inspire the user, for usefulness in music production.

[0033] Furthermore, in the embodiment of the present disclosure, use of the UI screen 500 enables the user to search, for example, the existing music held by the user, for the second musical composition data similar to the first musical composition data, facilitating listening to the second musical composition data or purchasing. In addition, the use of the UI screen 500 enables the user to compare the first musical composition data and the second musical composition data similar to the first musical composition data, thereby listening to the musical composition or study thereof, with deep understanding.(1-2. Configuration applicable to embodiments)

[0034] Next, a configuration applicable to an embodiment of the present disclosure will be described.

[0035] FIG. 3 is a functional block diagram illustrating exemplary functions of the server 10 according to an embodiment. In FIG. 3, the server 10 includes an overall control unit 100, a communication unit 101, an information processing unit 102, and a UI generation unit 103.

[0036] The overall control unit 100, the communication unit 101, the information processing unit 102, and the UI generation unit 103 may be configured on the server 10 by operation of an information processing program according to an embodiment on a central processing unit (CPU) of the server 10. The overall control unit 100, the communication unit 101, the information processing unit 102, and the UI generation unit 103 are not limited to this configuration, and some or all of the overall control unit 100, the communication unit 101, the information processing unit 102, and the UI generation unit 103 may be configured by hardware circuits that operate in cooperation with each other.

[0037] The overall control unit 100 controls the overall operations of the server 10. The communication unit 101 controls communication via the communication network 2. The first musical composition data transmitted from the terminal device 20 is received by the communication unit 101. Furthermore, the communication unit 101 may acquire the second musical composition data via the communication network 2. The communication unit 101 stores the acquired second musical composition data in the storage unit 11. The second musical composition data stored in the storage unit 11 is registered in the music catalog.

[0038] The information processing unit 102 analyzes the first musical composition data received by the communication unit 101. In addition, the information processing unit 102 analyzes the second musical composition data stored in the storage unit 11 and registered in the music catalog. Known 12 tone analysis may be applied as an analysis method for the first musical composition data and the second musical composition data. In the 12 tone analysis, a musical structure such as a tempo, a melody configuration, chord progression, and the number of notes is analyzed, for each tone of a chromatic twelve-tone scale, on the basis of the intensity, length, and the like of the tone. Use of the 12 tone analysis enables detection of not only the configuration of the musical composition data (intro, verse, pre-chorus, chorus, etc.) but also the emotion of the musical composition data.

[0039] On the basis of the analyzed first musical composition data and second musical composition data, the information processing unit 102 searches the second musical composition data registered in the music catalog, for a piece of second musical composition data including a portion having the predetermined similarity or more to at least a part of the first musical composition data. At this time, the information processing unit 102 searches for the similarity, for each of the above-described items.

[0040] The UI generation unit 103 generates the UI screen 500 on the basis of a result of the search by the information processing unit 102. The UI generation unit 103 causes the terminal device 20 to display the generated UI screen 500. The UI generation unit 103 appropriately performs selection or the like of information from the result of the search by the information processing unit 102, in response to an instruction transmitted from the terminal device 20 according to user operation, and generates the UI screen 500.

[0041] Hereinafter, a description that the UI generation unit 103 generates the UI screen 500 and causes the terminal device 20 to display the UI screen 500 may be described as that the UI generation unit 103 causes the terminal device 20 to display the UI screen 500.

[0042] FIG. 4 is a functional block diagram for illustrating functions of the information processing unit 102 in the server 10 according to an embodiment in more detail.

[0043] In FIG. 4, the information processing unit 102 includes a musical composition data collection unit 120, analysis units 121a and 121b, a musical composition data acquisition unit 122, and a search unit 123. Note that a music catalog 110 is stored in the storage unit 11.

[0044] The musical composition data collection unit 120 collects musical composition data, for example, via the communication network 2. The musical composition data collection unit 120 stores the collected musical composition data in the storage unit 11, as the second musical composition data, and registers the second musical composition data in the music catalog 110. At this time, the musical composition data collection unit 120 may include metadata, tag information, and the like related to the musical composition data, as the musical composition data, in addition to the audio data. The tag information may include a genre, a main instrument, an atmosphere of music, and the like of a musical composition related to the musical composition data.

[0045] The analysis unit 121a analyzes the second musical composition data registered in the music catalog 110 using a machine learning model having been trained in advance, and registers the second musical composition data after the analysis, in the music catalog 110, as analyzed second musical composition data, in association with the second musical composition data. The analysis unit 121a divides, for example, the second musical composition data into a plurality of portions in a time division manner, and analyzes each of the divided portions. A result of the analysis may represent, for example, a feature amount of each divided portion.

[0046] The analysis unit 121a may divide the second musical composition data in units of time or in units of samples. The analysis unit 121a may extract tempo information from the second musical composition data and divide the second musical composition data in units of bars on the basis of the extracted tempo information.

[0047] The musical composition data acquisition unit 122 acquires musical composition data transmitted from the terminal device 20. The musical composition data acquisition unit 122 passes the acquired musical composition data to the analysis unit 121b, as the first musical composition data. The analysis unit 121b analyzes the first musical composition data transferred from the musical composition data acquisition unit 122. A result of the analysis may represent, for example, a feature amount of each divided portion.

[0048] As a method of analyzing the first musical composition data by the analysis unit 121b, the analysis method by the analysis unit 121a, which is described above, may be similarly applied. For example, the analysis unit 121b divides the first musical composition data into a plurality of portions in a time division manner, and analyzes each of the divided portions. The analysis unit 121b passes the analyzed first musical composition data to the search unit 123.

[0049] Note that FIG. 4 illustrates that the analysis unit 121b that analyzes the first musical composition data and the analysis unit 121a that analyzes the second musical composition data are separated from each other, for the sake of explanation, but the analysis units 121a and 121b can be implemented with a common configuration.

[0050] The search unit 123 searches the music catalog 110 on the basis of the analyzed first musical composition data passed from the analysis unit 121b, and extracts the second musical composition data including a portion having the predetermined similarity or more to at least a part of the first musical composition data.

[0051] For example, the search unit 123 compares the feature amount of each portion of the analyzed first musical composition data and the feature amount of each portion of each piece of the analyzed second musical composition data registered in the music catalog 110. As a result of the comparison, the search unit 123 extracts, for example, a feature amount having a difference equal to or less than a predetermined value from a specific feature amount of the analyzed first musical composition data, among the feature amounts of the respective portions of the analyzed second musical composition data. The search unit 123 may determine that the portion of the analyzed second musical composition data related to the extracted feature amount has the predetermined similarity or more to the portion of the analyzed first musical composition data related to the specific feature amount.

[0052] The search unit 123 passes information indicating a result of the search obtained by searching the music catalog 110 on the basis of the first musical composition data, to the UI generation unit 103, as described above. For example, the search unit 123 may pass, to the UI generation unit 103, information indicating the second musical composition data having the predetermined similarity or more to at least a part of the first musical composition data and information indicating a portion of the first musical composition data having a high similarity to a portion of the second musical composition data, which are searched for on the basis of the first musical composition data, as the result of the search.

[0053] In this way, the search unit 123 functions as a similarity evaluation unit that evaluates the similarity of the first musical composition data to the second musical composition data, for each of the plurality of items.

[0054] The UI generation unit 103 generates, for example, the UI screen 500 described with reference to FIG. 2, on the basis of the information indicating the result of the search passed from the search unit 123. The UI screen 500 generated by the UI generation unit 103 is transmitted to the terminal device 20 and displayed by a display unit of the terminal device 20.

[0055] FIG. 5 is a functional block diagram illustrating exemplary functions of the terminal device 20 according to an embodiment. In FIG. 5, the terminal device 20 includes an overall control unit 200, a communication unit 201, a musical composition data acquisition unit 202, a display unit 203, and an operation reception unit 204.

[0056] The overall control unit 200, the communication unit 201, the musical composition data acquisition unit 202, and the operation reception unit 204 may be configured by, for example, a standard program (operating system (OS), browser application program, and the like) installed on the terminal device 20. Note that the display unit 203 may be configured by hardware.

[0057] The overall control unit 200, the communication unit 201, the musical composition data acquisition unit 202, and the operation reception unit 204 are not limited to this configuration, and may be configured on the terminal device 20 by operation of a terminal device program according to an embodiment on CPU of the terminal device 20. Furthermore, some or all of the overall control unit 200, the communication unit 201, the musical composition data acquisition unit 202, the display unit 203, and the operation reception unit 204 may be configured by hardware circuits that operate in cooperation with each other.

[0058] The overall control unit 200 controls the overall operation of the terminal device 20. The communication unit 201 controls communication via the communication network 2. The terminal device 20 transmits musical composition data acquired by the musical composition data acquisition unit 202, which is described later, as the first musical composition data, to the server 10 via the communication network 2. Furthermore, the communication unit 201 receives the UI screen 500 generated in the server 10 via the communication network 2. Furthermore, the communication unit 201 transmits an instruction or the like according to user operation based on the UI screen 500, to the server 10 via the communication network 2.

[0059] The musical composition data acquisition unit 202 acquires the musical composition data. For example, the musical composition data acquisition unit 202 is a data interface (I / F), and may acquire musical composition data from an external device that is connected via the data I / F. The musical composition data acquisition unit 202 is not limited to this configuration and may include an analog audio signal input unit and an analog to digital (A / D) converter that converts the voice signal into digital audio data. In this configuration, the analog audio signal may be a signal reproduced by an external reproduction device or a signal output by a voice input device such as a microphone.

[0060] The display unit 203 generates a display signal for causing a display device to display, on the basis of a display control signal. For example, the display unit 203 generates the display signal for causing the UI screen 500 to be displayed, on the basis of the display control signal received from the server 10 by the communication unit 201.

[0061] The operation reception unit 204 receives a control signal according to user operation on an input device included in the terminal device 20 or connected to the terminal device 20.

[0062] FIG. 6 is a block diagram illustrating an exemplary hardware configuration of the server 10 applicable to an embodiment. Note that, here, for ease of description, the server 10 is assumed to be configured as single hardware.

[0063] In FIG. 6, the server 10 includes a CPU 1000, a read only memory (ROM) 1001, a random access memory (RAM) 1002, a storage device 1003, a data I / F 1004, and a communication I / F 1005 that are communicably connected to each other via a bus 1010.

[0064] The storage device 1003 is a non-volatile storage medium, such as a flash memory or a hard disk drive. The CPU 1000 operates with the RAM 1002 as a work memory, according to a program stored in the storage device 1003 and the ROM 1001, and controls the overall operation of the server 10. The data I / F 1004 controls data communication with an external device. The communication I / F 1005 controls communication via the communication network 2.

[0065] Furthermore, an input device that receives user input or a display device that displays information may be connected to the server 10. Furthermore, the server 10 may further include a training processing unit for training the machine learning model. The training processing unit may be configured by, for example, a digital signal processor (DSP) .

[0066] In the server 10, upon execution of the information processing program according to an embodiment, the CPU 1000 configures the overall control unit 100, the communication unit 101, the information processing unit 102, and the UI generation unit 103, which are described above, for example, as a module, in a main storage area in the RAM 1002.

[0067] The information processing program is configured to be acquired from the outside via the communication network 2 by communication, for example, via the communication I / F 1005 and installed in the server 10. Furthermore, the information processing program is not limited to this configuration and may be provided by being stored in a removable storage medium such as a compact disk (CD), a digital versatile disk (DVD), or a universal serial bus (USB) memory.

[0068] FIG. 7 is a block diagram illustrating an exemplary hardware configuration of the terminal device 20 applicable to an embodiment.

[0069] In FIG. 7, the terminal device 20 includes a CPU 2000, a ROM 2001, a RAM 2002, a graphic I / F 2003, a storage device 2004, an input device 2005, a data I / F 2006, a communication I / F 2007, and an audio I / F 2008 that are communicably connected to each other by a bus 2010.

[0070] The storage device 2004 is a non-volatile storage medium, such as a flash memory or a hard disk drive. The CPU 2000 operates with the RAM 2002 as a work memory, according to a program stored in the storage device 2004 and the ROM 2001, and controls the overall operation of the terminal device 20. The data I / F 2006 controls data communication with an external device. The communication I / F 2007 controls communication via the communication network 2.

[0071] The input device 2005 receives user operation and outputs the control signal according to the received user operation. The input device 2005 may include a character input device such as a keyboard and a pointing device such as a mouse or a touch screen. The character input device and the pointing device may be configured integrally or in common.

[0072] The graphic I / F 2003 converts the display control signal generated by the CPU2000 into the display signal that is displayable by a display device 2020. For example, a liquid crystal display (LCD) or an organic EL display may be applied to the display device 2020, and is built in or externally connected to the terminal device 20. The display device 2020 causes a screen corresponding to the display signal output from the graphic I / F 2003 to be displayed.

[0073] The audio I / F 2008 includes a digital to analog (D / A) converter, and converts, for example, the digital audio data supplied from the bus 2010 into the analog audio signal. The audio I / F 2008 may further include an amplifier to drive a speaker 2030 by the audio signal obtained by converting the digital audio data, for reproduction of sound. Furthermore, the audio I / F 2008 may further include an A / D converter to convert, for example, the analog audio signal output from the microphone into the digital audio data and output to the bus 2010.

[0074] When the overall control unit 200, the communication unit 201, the musical composition data acquisition unit 202, and the operation reception unit 204, which are described above, are configured by dedicated application programs, the CPU2000 configures, for example, as a module, the overall control unit 200, the communication unit 201, the musical composition data acquisition unit 202, and the operation reception unit 204, in a main storage area in the RAM 2002, upon execution of the terminal device program according to an embodiment.

[0075] The terminal device program is configured to be acquired from the outside (e.g., the server 10), for example, via the communication network 2 by communication via the communication I / F 2007 and installed in the terminal device 20. Furthermore, the terminal device program is not limited to this configuration and may be provided by being stored in a removable storage medium such as a compact disk (CD), a digital versatile disk (DVD), or a universal serial bus (USB) memory.

[0076] Note that, in the above description, the terminal device 20 is described as a general computer device, but is not limited to this example. The terminal device 20 may be a tablet computer or a smartphone.(2. First embodiment of present disclosure)

[0077] Next, a first embodiment of the present disclosure will be described.(2-1. Processing according to first embodiment)

[0078] First, processing according to the first embodiment of the present disclosure will be described.(2-1-1. Overview of processing according to first embodiment)

[0079] FIG. 8 is a schematic diagram for schematically illustrating functions of the information processing system 1 according to the first embodiment. In the information processing system 1, the information processing unit 102 of the server 10 analyzes input data 30 as the first musical composition data and extracts melody data 31 indicating a melody. The information processing unit 102 further searches for pieces of second musical composition data including the similar portion to at least a part of the input data 30, and outputs pieces of output data 40 including pieces of second musical composition data 41a, 41b, 41c, 41d, and 41e obtained by searching. Furthermore, the information processing unit 102 extracts pieces of melody data 42a, 42b, 42c, 42d, and 42e indicating melodies from the pieces of second musical composition data 41a, 41b, 41c, 41d, and 41e and outputs the pieces of melody data.(First function)

[0080] A first function of the information processing system 1 according to the first embodiment is a function of searching for the second musical composition data having the similar portion to the first musical composition data having been input. The first function may be applied to finding the similar portion in each of the pieces of second musical composition data 41a to 41e, from the input data 30 as the first musical composition data. In this case, a target from which the similar portion is to be found may include musical composition data automatically generated by AI. In the first function, the information processing system 1 may search for musical composition data including the similar portion to at least a part of the first musical composition data, from each piece of second musical composition data registered in the music catalog 110.(Second function)

[0081] A second function of the information processing system 1 according to the first embodiment is a function of converting the input first musical composition data into musical score data. The input data 30 is generally a track (referred to as 2mix) obtained by mixing a plurality of tracks including a melody track into two channels, for example, a left (L) channel and a right (R) channel.

[0082] For example, as will be described later, the information processing unit 102 performs an audio source separation process on the input data 30 which is 2mix audio data to separate the tracks of the respective instruments, and extracts a vocal track from each of the separated tracks. The information processing unit 102 extracts the melody data 31 by using the extracted vocal track. The information processing unit 102 performs f0 estimation on the vocal track and extracts the melody data 31.

[0083] Here, the information processing unit 102 performs predetermined processing such as low-pass filtering on the data (referred to as f0 data) obtained by the f0 estimation, and the melody data 31 similar to the melody data obtained by Musical Instruments Digital Interface (MIDI) estimation can be obtained. Note that the melody data obtained by the MIDI estimation can be expressed by, for example, two-dimensional information with a vertical axis as pitch and a horizontal axis as time. The f0 estimation by the information processing unit 102 will be described later.

[0084] The second function may be applied to musical composition generation and musical composition editing based on the input data 30. Furthermore, use of the melody data 31 obtained by the second function, as input data for a model performing the MIDI estimation enables the MIDI estimation with higher accuracy.(2-1-2. Specific examples of processing according to first embodiment)

[0085] Next, the processing according to the first embodiment will be described more specifically.(Training of search model)

[0086] In the information processing system 1 according to the first embodiment, the machine learning model having been trained by using any musical composition data is prepared. The information processing system 1 uses this machine learning model to obtain the feature amount for each portion of the first musical composition data and the second musical composition data. The information processing system 1 searches for the second musical composition data including a portion similar to at least a part of the first musical composition data, on the basis of the feature amount of each portion of the first musical composition data and the second musical composition data. Hereinafter, unless otherwise specified, the machine learning model that obtains the feature amount for each portion of the first and second musical composition data is referred to as a search model.

[0087] FIG. 9 is a schematic diagram for illustrating a learning process of the search model according to the first embodiment. Here, it is assumed that the information processing unit 102 trains the search model. Training of the search model may be performed not only by the information processing unit 102, but also by another configuration on the cloud network 3, another server device connected to the communication network 2, or the like.

[0088] In Step S50, the information processing unit 102 performs the audio source separation process on input data 60 as the 2mix audio data, separates the data of each instrument and the like included in the input data 60, and acquires each piece of audio source data. In the example of the figure, the information processing unit 102 acquires, from the input data 60, audio source data 61a of a guitar, audio source data 61b of a drum, audio source data 61c of a keyboard instrument, and audio source data 61d of a vocal, by the audio source separation process. Among the pieces of audio source data 61a to 61d, the audio source data 61d is data about a main melody, and the other pieces of audio source data 61a to 61c are data about the accompaniments to the main melody. For the main melody, not only the audio source data of the vocal, but also other audio source data of a guitar, a piano, or the like may be used, in some cases.

[0089] Note that, in Step S50, the information processing unit 102 may further analyze the input data 60 by using the 12 tone analysis or the like to extract information relating to the configuration of the musical composition, such as the tempo information, rhythm configuration information, chord information, and music configuration information, from the input data 60. The music configuration information can be estimated, for example, on the basis of a change in the chord information and / or the rhythm configuration information in a time-series direction.

[0090] In the next Step S51, the information processing unit 102 performs an f0 estimation process on the audio source data 61d of the vocal among the pieces of audio source data 61a to 61d acquired in Step S50 to obtain f0 data 62. In the next Step S52, the information processing unit 102 performs time-series normalization on the f0 data 62 by using a process called warping, and generates normalized data 63 obtained by normalizing the f0 data 62. For example, the information processing unit 102 performs a time stretching process or a simple time-series compression process on the f0 data 62 to normalize the tempo of the f0 data 62 to, for example, 120 beats per minute (BPM), and generates the normalized data 63.

[0091] Note that the information processing unit 102 may perform low-pass filtering on the f0 data 62 in Step S51 described above (not illustrated). Performing the low-pass filtering on the f0 data 62 levels peak portions in the f0 data 62 illustrated in the drawing, and makes it possible to obtain a waveform which shows an appropriate pitch and appropriate length information. The low-pass filtering may be performed on the normalized data 63.

[0092] In the next Step S53, the information processing unit 102 trains a search model 72 by using the normalized data 63. For the search model 72, a two-dimensional convolutional neural network (CNN) may be applied. In this case, the search model 72 handles, for example, the waveform of the audio data as an image. At this time, the information processing unit 102 performs data augmentation on the normalized data 63 and generates a plurality of pieces of data based on the normalized data 63.

[0093] For example, the information processing unit 102 may generate augmented data by performing zero padding 64 on the normalized data 63 and compacting the normalized data 63 in the time-series direction at random. The information processing unit 102 enters pieces of data including the compacted data generated by compacting the normalized data 63 in the time-series direction, into a query model 70. The query model 70 generates vectors I 1 , I 2 , I 3 ,..., I N , on the basis of the pieces of data having entered the query model 70.

[0094] Furthermore, for example, the information processing unit 102 may perform f0 filling at random on the normalized data 63, add f0 data 65a and f0 data 65b, and extend the normalized data 63 in the time-series direction, for generation of the augmented data. The information processing unit 102 enters pieces of extension data 63aug generated by extending the normalized data 63 in the time-series direction, into a key model 71 having a structure similar to the query model 70. The key model 71 generates vectors T 1 , T 2 , T 3 ,..., and T N on the basis of the pieces of data having entered the key model 71.

[0095] The data extension method used by the information processing unit 102 is not limited to the compaction and extension of the normalized data 63. For example, the information processing unit 102 may generate the augmented data by performing compression in the time-series direction or pitch shifting on the normalized data 63. Furthermore, the information processing unit 102 may perform low-pass filtering on the normalized data 63 to generate the augmented data. For use of the low-pass filtering, it is preferable to add noise such as Gaussian noise to the data subjected to the low-pass filtering.

[0096] The information processing unit 102 trains the search model 72 so that the vectors I 1 , I 2 , I 3 ,..., and I N may approach the vectors T 1 , T 2 , T 3 ,..., and T N . In the example of the drawing, the inner products of the hatched vectors I 1 and T 1 , I 2 and T 2 , I 3 and T 3 ,..., I N and T N are small, and the distances thereof are small, that is, the combinations of these vectors have a large similarity.

[0097] In this manner, the search model 72 is trained to extract features that are not affected by data extension.

[0098] In the first embodiment, global average pooling is applied to an output layer of the search model 72. The global average pooling is a method to divide a multidimensional vector representing the feature amounts is divided into a plurality of channels and integrally calculate an average value of the divided channels. In the global average pooling, even if there is a certain deviation from an original location, the deviation is accommodated by the addition upon calculating the average value, and therefore, place dependence is reduced. The place mentioned here relates to time and pitch in the audio data, and an expression that does not change even when the time and the pitch shift to some extent can be obtained.

[0099] Therefore, application of the global average pooling to the output layer of the search model 72 enables acquisition of invariance to the shifts in the time-series direction and pitch direction can be obtained, and the similarity closer to that a person feels can be modeled.

[0100] For example, in reproduction of musical composition data, even when a melody is octave apart in pitch or different in key, or even when the melody is shifted as a whole (e.g., position on the melody on the time series), a person can perceive that the reproduced musical compositions are the same or similar to each other. Application of the global average pooling enables determination of such similarity closer to that the person feels.(Similar data search process)

[0101] Next, a search process for the second musical composition data including a portion similar to at least a part of the first musical composition data, according to the first embodiment, will be described more specifically. FIG. 10 is a flowchart illustrating an exemplary search process according to the first embodiment.

[0102] In Step S100, in the server 10, the information processing unit 102 acquires musical composition data by the musical composition data collection unit 120 or the musical composition data acquisition unit 122. The musical composition data collection unit 120 registers the acquired musical composition data in the music catalog 110. Although not illustrated, the musical composition data acquisition unit 122 may also register the acquired musical composition data in the music catalog 110.

[0103] In the next Step S101, the information processing unit 102 analyzes the acquired musical composition data by the analysis unit 121a or 121b. The analysis unit 121a or 121b detects component elements of the musical composition, such as a melody configuration, a rhythm, and a chord configuration, on the basis of the musical composition data by analysis using, for example, the 12 tone analysis, and detects emotion (feeling) caused by the musical composition data.

[0104] In the next Step S102, the information processing unit 102 divides the musical composition data acquired by the musical composition data collection unit 120 or the musical composition data acquisition unit 122 in terms of evaluation unit, for example, by the analysis unit 121a or 121b. The evaluation unit may be time or may be a sampling unit for the audio data. The evaluation unit is not limited thereto, and may be a bar unit based on information analyzed in Step S101. Furthermore, the evaluation unit can be switched, for example, according to user operation.

[0105] The analysis unit 121b inputs each of the analyzed and divided pieces of the musical composition data to the search unit 123 together with a result of the analysis. Meanwhile, the analysis unit 121a registers each of the analyzed and divided pieces of the musical composition data in the music catalog 110 in association with the second musical composition data together with a result of the analysis.

[0106] In the next Step S103, the search unit 123 enters the musical composition data input from the analysis unit 121b into the search model 72 for each evaluation unit. In addition, the search unit 123 enters the musical composition data registered in the music catalog 110 into the search model 72 for each evaluation unit. In the next Step S104, the information processing unit 102 performs calculation using the search model 72 on the musical composition data input from the analysis unit 121b or the musical composition data registered in the music catalog 110, in the search unit 123, and obtains the feature amount for each evaluation unit.

[0107] Note that, in Steps S100 to S104, the processing on the second musical composition data acquired by the musical composition data collection unit 120 and the processing on the first musical composition data acquired by the musical composition data acquisition unit 122 are performed respectively at appropriate timing, and do not need to be performed in parallel. Processing after the processing in Step S104 is different between the processing on the first musical composition data and the processing on the second musical composition data.

[0108] The processing on the second musical composition data, that is, the musical composition data registered in the music catalog 110 proceeds to Step S110 after Step S104. In Step S110, for example, the search unit 123 stores (registers) the feature amount obtained on the basis of the second musical composition data, in association with an original second musical composition data, for example, in the music catalog 110.

[0109] Meanwhile, the processing on the first musical composition data, that is, the musical composition data acquired by the musical composition data acquisition unit 122 proceeds to Step S120 after the processing of Step S104. In Step S120, for example, the search unit 123 searches for musical composition data including a portion (evaluation unit) having a difference equal to or less than a predetermined value from the feature amount, that is, a portion whose similarity to the feature amount is equal to or larger than the threshold, among the pieces of musical composition data registered in the music catalog 110 on the basis of the feature amount obtained for each evaluation unit. The search unit 123 may perform the search, for each of the melody configuration, the rhythm, the chord configuration, the emotion,... obtained by the analysis of the musical composition data in Step S101.

[0110] In the next Step S121, the search unit 123 generates display information on the basis of a result of the search in Step S120. In the next Step S122, in the server 10, the UI generation unit 103 generates the UI screen 500 on the basis of the display information generated in Step S121 and causes the terminal device 20 to display the UI screen 500.(2-2. Exemplary UI screen according to first embodiment)

[0111] Next, an example of the UI screen 500 according to the first embodiment will be described more specifically.(2-2-1. UI screen display process according to first embodiment)

[0112] Next, a UI screen display process according to the first embodiment will be described. In the UI screen 500 according to the first embodiment, as described with reference to FIG. 2, a result of the search for the similar portion in the first musical composition data to the second musical composition data is displayed according to the three forms of the first form, the second form, and the third form. Furthermore, In the first form and the second form, the result of the search is displayed for each of one or more items including at least one of the items of "melody", "chord", "impression", "rhythm", "lyrics", and "overall musical composition".

[0113] Among these items, "melody" is an item indicating a similarity of the main melody in the musical composition. For example, the vocal is often assigned to the melody, but a string instrument such as a guitar, a wind instrument such as a saxophone or a trumpet, or a keyboard instrument such as a piano or an organ may be assigned to the melody in some cases. "Chord" is an item indicating a similarity of a change, on the time-series in chords used for the musical composition (chord progression). The chord is generally repeated in combination of several patterns in time series.

[0114] "Impression" is an item indicating a similarity of impression of the musical composition. The impression includes, for example, a combination of component elements of the musical composition, such as the melody, chord, and rhythm. In an example, if the melody and the chord progression are minor, the tempo is slow, and the reverb is deeply applied to the overall musical composition, the impression is considered as "dark". When the genre, style, atmosphere, and the like of the musical composition are tagged in advance as the metadata in the musical composition data, the tag information can also be acquired as the "impression".

[0115] "Rhythm" is an item indicating a similarity of the rhythm of the musical composition. The rhythm is comprehensively configured by, for example, playing a drum, bass, another percussive instrument, and further another instrument used in the musical composition. "Lyrics" is an item indicating a similarity of lyrics sung on vocals. The lyrics may be extracted from a vocal audio source along with the melody. When lyric information is associated with the musical composition, the lyric information can be acquired as "lyrics".

[0116] Furthermore, the "overall musical composition" may be an item indicating a similarity obtained by integrating similarities in all items or two or more items among the items of the "melody", "chord", "impression", "rhythm", and "lyrics", which are described above.

[0117] As described above, the UI screen 500 according to the first embodiment has the three display modes of "Similar", "Similar & Common", and "Similar & Rare" as the display modes for performing display related to similarity. The display mode "Similar" is a display mode relating to the second musical composition data including a portion whose similarity to the first musical composition data is equal to or larger than a first threshold.

[0118] The display mode "Similar & Common" is a display mode relating to the second musical composition data including a portion whose similarity to the first musical composition data is equal to or larger than the first threshold and which is versatile. Here, the portion which is versatile is a portion of the second musical composition data including a pattern that is often used generally in the musical composition. Specifically, when the first musical composition data is a pop music, the versatile patterns include eight-beat in rhythm, Canon chord progression and 3-chord blues progression in chord progression, and the like.

[0119] The display mode "Similar & Rare" is a display mode relating to the second musical composition data including a portion whose similarity to the first musical composition data is equal to or larger than a second threshold higher than the first threshold and which is not versatile. The display mode "Similar & Rare" may further include a condition that the number of pieces of the second musical composition data is 1 to a predetermined number. In the display mode "Similar & Rare", the second musical composition data including a portion that is not versatile but has a very high similarity to the first musical composition data is indicated, and therefore, it is easy to find, for example, infringement on the first musical composition data.

[0120] FIG. 11 is a flowchart illustrating exemplary display control of the UI screen 500 according to the first embodiment. In Step S200, in the server 10, the UI generation unit 103 sets the display mode to, for example, a default display mode. Here, it is assumed that the display mode "Similar" is the default display mode.

[0121] In the next Step S201, the server 10 generates the display information of the first and second forms according to the display mode by the UI generation unit 103. In the next Step S202, the UI generation unit 103 transmits the generated display information of the first and second forms to the terminal device 20, and causes the terminal device 20 to display display screens of the first and second forms.

[0122] In the next Step S203, in the server 10, the UI generation unit 103 determines whether an instruction for changing the display mode on the terminal device 20 is given. When it is determined that the instruction is given (Step S203, "Yes"), the UI generation unit 103 returns the process to Step S201, changes the current display mode to the received display mode, and generates the display information of the first and second forms according to the changed display mode.

[0123] Meanwhile, in Step S203, when the UI generation unit 103 determines that the instruction for changing the display mode is not given (Step S203, "No"), the process proceeds to Step S204. In Step S204, in the server 10, the UI generation unit 103 determines whether a result of the search is specified on the time series in the screen of the second form on the terminal device 20. When the UI generation unit 103 determines that the result of the search is not specified (Step S204, "No"), the process returns to Step S201.

[0124] Meanwhile, in Step S204, when the UI generation unit 103 determines that the result of the search is specified on the time series (Step S204, "Yes"), the process proceeds to Step S205. In Step S205, in the server 10, the UI generation unit 103 causes the UI screen 500 to display the screen of the third form, and presents the list of the second musical composition data corresponding to the specified result of the search.(2-2-2. Specific example of UI screen according to first embodiment)

[0125] Next, the UI screen 500 according to the first embodiment will be described more specifically. FIGS. 12A, 12B, and 12C are schematic diagrams each illustrating an example of the UI screen 500 having the first and second forms according to the first embodiment. Note that the respective portions of the UI screen 500 have been described with reference to FIG. 2, and thus detailed description thereof will not be repeated here.

[0126] FIG. 12A schematically illustrates an example of a UI screen 500a according to the display mode "Similar". In other words, in the example of FIG. 12A, "Similar" is selected in the mode selection area 521. In the example of FIG. 12A, from the display of each bar 531 in the first form of the area 530, it can be seen that the number of pieces of second musical composition data including a portion having high similarity in impression to at least a part of the first musical composition data is large and the number of pieces of second musical composition data including portions having high similarity in chord and rhythm to at least a part of the first musical composition data is small.

[0127] In addition, from the display of the second form in the area 540, it is possible to know which portion on the time series has a portion with a high similarity to each piece of second musical composition data registered in the music catalog 110, for each item, in the first musical composition data. In the example of FIG. 12A, for the melody, portions having high similarity to the second musical composition data are illustrated as similar portions 541a 1 , 541a 2 , and 541a 3 . Similarly, it can be seen that a similar portion 541b 1 for the chord, similar portions 541c 1 and 541c 2 for the impression, a similar portion 541d 1 for the rhythm, similar portions 541e 1 , 541e 2 , and 541e 3 for the lyrics, and similar portions 541f 1 , 541f 2 , and 541f 3 for the overall musical composition are portions having high similarity to the second musical composition data.

[0128] FIG. 12B schematically illustrates an example of a UI screen 500b according to the display mode "Similar & Rare". In other words, in the example of FIG. 12B, "Similar & Rare" is selected in the mode selection area 521. In the example of FIG. 12B, from the display of each bar 531 in the first form of the area 530, it can be seen that pieces of second musical composition data including the portion having the very high similarity are detected from the pieces of second musical composition data registered in the music catalog 110, for the melody, impression, and overall musical composition, but are not detected for the chord, rhythm, and lyrics.

[0129] In addition, from the display of the second form in the area 540, it is possible to know which portion on the time series represents a portion with the very high similarity to each piece of second musical composition data registered in the music catalog 110, in each of the melody, impression, and overall musical composition for which the second musical composition data including a portion having the very high similarity is detected. In the example of FIG. 12B, it can be seen that a similar portion 541a 21 for the melody, a similar portion 541c 11 for the impression, and similar portions 541f 11 and 541f 21 for the overall musical composition are portions having the very high similarity to the second musical composition data.

[0130] Note that these similar portions 541a 21 , 541c 11 , 541f 11 , and 541f 21 correspond to the similar portions 541a 1 , 541c 1 , 541f 1 , and 541f 2 in FIG. 12A, respectively.

[0131] FIG. 12 C schematically illustrates an example of a UI screen 500c according to the display mode "Similar & Common". In other words, in the example of FIG. 12C, "Similar & Common" is selected in the mode selection area 521. In the display of the similar portions 541 in the second form of the area 540, the similar portions 541 are shown by removing the similar portions 541a 1 , 541c 1 , 541f 1 , and 541f 2 corresponding to the similar portions 541a 21 , 541c 11 , 541f 11 , and 541f 21 for which the similarity is detected in the display mode "Similar & Rare" of FIG. 12B, from the example of the display mode "Similar" of FIG. 12A described above.

[0132] Note that the display of each of the similar portions 541 in the area 540 may be displayed in gradation as illustrated in the example of each drawing, and, for example, the similarity in each similar portion 541 may be indicated by color density. Furthermore, for the similar portion 541, a length for target searching can also be set. At this time, a plurality of minimum units of target searching, such as eight seconds and 16 seconds, may be set switchably.

[0133] In the examples of FIGS. 12A to 12C, display of the first form by the area 530 and display of the second form by the area 540 are simultaneously displayed, but are not limited to this example. For example, in the default state, only display of the first form may be performed by the area 530, and display of the second form may be further performed by the area 540 according to user operation or the like.

[0134] Furthermore, in the above description, three display modes of "Similar", "Similar & Common", and "Similar & Rare" are selectable, but are not limited to this example. For example, the display mode may be selected from two of three modes of "Similar", "Similar & Common", and "Similar & Rare", or may be fixed to one of the three modes.

[0135] FIGS. 13A, 13B, 13C, and 13D are schematic diagrams each illustrating an example of the UI screen 500 having the third form according to the first embodiment. Note that the respective portions of the UI screen 500 have been described with reference to FIGS. 2 and 12A to 12C, and thus detailed description thereof will not be repeated here.

[0136] As described above, in the first embodiment, in the display of the second form in the area 540, when a similar portion 541 of any item is specified by user operation or the like, the area 550 including the display of the third form is displayed. FIG. 13A schematically illustrates a state in which the user operates the input device 2005 (pointing device or the like) of the terminal device 20 to specify the similar portion 541a 2 for the melody in the area 540 with a cursor 600. The UI generation unit 103 causes the UI screen 500 to display the area 550 including the display of the third form, according to the specification of the similar portion 541a 2 on the terminal device 20.

[0137] FIG. 13B schematically illustrates a display example of an area 550a according to the specification of the similar portion 541a 2 relating to the melody. For example, the search unit 123 extracts, from the music catalog 110, the second musical composition data including a portion whose similarity to the similar portion 541a 2 is equal to or larger than a threshold. The UI generation unit 103 generates a list screen 551a including the list of the second musical composition data extracted by the search unit 123. The UI generation unit 103 generates a UI screen 500d including the area 550a in which the list screen 551a is arranged, and causes the terminal device 20 to display the UI screen 500d. Furthermore, the UI generation unit 103 may include the melody display 560 of the first musical composition data in the specified similar portion 541a 2 , in the UI screen 500d.

[0138] In the example of FIG. 13B, the list screen 551a includes an audio data waveform 552a and a melody display 552b of a portion of each piece of second musical composition data, similar to the similar portion 541a 2 , and a metadata icon 552c relating to each piece of second musical composition data. In this example, the melody displays 560 and 552b indicate the melodies according to the pitch and the length, with the horizontal axis as time and the vertical axis as pitch.

[0139] The user can specify a position of, for example, the audio data waveform 552a or the melody display 552b on the UI screen 500d, with the cursor 600 or the like by user operation to listen to the portion of the second musical composition data corresponding to the specified position. Furthermore, the user can specify the metadata icon 552c on the UI screen 500d with the cursor 600 or the like by user operation to browse the metadata corresponding to the specified metadata icon 552c.

[0140] FIG. 13C schematically illustrates a display example of an area 550b according to the specification of the similar portion 541b 1 relating to the chord. As in the above description, for example, the search unit 123 extracts, from the music catalog 110, the second musical composition data including a portion whose similarity to the similar portion 541b 1 is equal to or larger than a threshold. The UI generation unit 103 generates a list screen 551b including the list of the second musical composition data extracted by the search unit 123. The UI generation unit 103 generates a UI screen 500e including the area 550b in which the list screen 551b is arranged, and causes the terminal device 20 to display the UI screen 500e. Furthermore, the UI generation unit 103 may include a chord progression display 561 of the first musical composition data in the specified similar portion 541b 1 , in the UI screen 500e.

[0141] In the example of FIG. 13C, the list screen 551b includes an audio data waveform 553a and a chord progression display 553b of a portion of each piece of second musical composition data, similar to the similar portion 541b 1 , and a metadata icon 553c relating to each piece of second musical composition data. In this example, the chord progression displays 561 and 553b show the chord progression of the portion (Gb →Ab → Fm →Bbm → Ebm).

[0142] The user can specify a position of, for example, the audio data waveform 553a or the chord progression display 553b on the UI screen 500e, with the cursor 600 or the like by user operation to listen to the portion of the second musical composition data corresponding to the specified position. Furthermore, the user can specify the metadata icon 553c on the UI screen 500e with the cursor 600 or the like by user operation to browse the metadata corresponding to the specified metadata icon 553c.

[0143] FIG. 13D schematically illustrates a display example of an area 550c according to the specification of the similar portion 541d 1 relating to the rhythm. As in the above description, for example, the search unit 123 extracts, from the music catalog 110, the second musical composition data including a portion whose similarity to the similar portion 541d 1 is equal to or larger than a threshold. The UI generation unit 103 generates a list screen 551c including the list of the second musical composition data extracted by the search unit 123. The UI generation unit 103 generates a UI screen 500f including the area 550c in which the list screen 551c is arranged, and causes the terminal device 20 to display the UI screen 500f. Furthermore, the UI generation unit 103 may include a rhythm display 562 of the first musical composition data in the specified similar portion 541d 1 , in the UI screen 500f.

[0144] In the example of FIG. 13D, the list screen 551c includes an audio data waveform 554a and a rhythm display 554b of a portion of each piece of second musical composition data, similar to the similar portion 541d 1 , and a metadata icon 554c relating to each piece of second musical composition data. In this example, the rhythm displays 562 and 554b indicate the timing of playing each instrument with the horizontal axis as time and the vertical axis as the type of rhythm instrument (kick, snare, hi-hat, etc.).

[0145] The user can specify a position of, for example, the audio data waveform 554a or the rhythm display 554b on the UI screen 500f, with the cursor 600 or the like by user operation to listen to the portion of the second musical composition data corresponding to the specified position. Furthermore, the user can specify the metadata icon 554c on the UI screen 500f with the cursor 600 or the like by user operation to browse the metadata corresponding to the specified metadata icon 554c.

[0146] Note that, in the above description, the examples of the display of the third form for the melody, the chord, and the rhythm have been described, but the impression, the lyrics, and the overall musical composition can be displayed similarly.(3. Second embodiment of present disclosure)

[0147] Next, a second embodiment of the present disclosure will be described. In the first embodiment described above, information of the musical composition similar to the target musical composition is presented to the user by using the UI. Meanwhile, in the second embodiment of the present disclosure, a configuration is proposed that enables the user to avoid the possibility of copyright infringement on a musical composition similar to the target musical composition.(3-1. Avoidance of copyright infringement by existing technology)

[0148] First, avoidance of copyright infringement by an existing technology will be schematically described. Conventionally, many judgements have passed on music copyright infringement. In the judgements on copyright infringement, a court, for example, determines the presence or absence of similarity between a musical composition of an appellant and a musical composition of an appellee to judge whether a musical composition of the appellee infringes a copyright of the musical composition of the appellant.

[0149] Here, a judgment regarding copyright infringement which is commonly referred to as "Kinenju case" (Tokyo High Court Decision, September 6, 2002) will be schematically described. The Kinenju case is a case where a music composer of the existing musical composition and a music publisher who are the copyright holder of the existing musical composition bring an action for damages against a composer of the musical composition "Kinenju", on the grounds that a musical composition "Kinenju", which was released in 1992 and broadcasted as an ending theme of a TV program, is a copy of the existing musical composition released as a commercial song in 1966. In the Kinenju case, although the appellant's action was dismissed in the first trial, the appellant's action was approved in the appeal court decision, and the appellee is ordered to pay damages.

[0150] Hereinafter, for convenience, the musical composition for which copyright infringement is claimed in the judgment is referred to as "copyright-claimed musical compositional composition", and the original musical composition is referred to as "copyrighted original musical composition".

[0151] In this case, the presence or absence of the similarity is determined by comparing elements of music such as a harmony, rhythm, tempo, and form between the copyrighted original musical composition and the copyright-claimed musical composition with a focus on a tune. As a result, the similarity was denied in the first trial, but the similarity was approved in the appeal court decision.

[0152] In the appeal court decision, the similarity is quantitatively determined by the following analysis methods (1) to (3). (1) Extracting a phrase to be compared and contrasted, that is, a phrase for which similarity is to be determined, from each of the copyright-claimed musical composition and the copyrighted original musical composition. (2) Arranging musical scores of the copyright-claimed musical composition and the copyrighted original musical composition which are each adjusted to have the same length in the phrase to be compared and contrasted. (3) Quantitatively measuring the degree of matching of pitch between the musical scores, on the basis of the adjusted musical scores of the copyright-claimed musical composition and the copyrighted original musical composition.

[0153] In the appeal court decision, in addition to a result of the quantitative analysis described above, the following (A) to (D) are examined to qualitatively determine the similarity. (A) Extracting difference portions in melody from the copyright-claimed musical composition and the copyrighted original musical composition to examine whether the difference portion impairs the essential identity in expression of both musical compositions. (B) Extracting difference portions in harmony from both musical compositions to examine whether the difference portions impair the essential identity in expression of both musical compositions. (C) Extracting difference portions in rhythm and tempo of both musical compositions to examine whether the difference portions impair the essential identity in expression of both musical compositions. (D) Extracting difference portions in the form of overall musical compositions to examine whether the difference portions impair the essential identity in expression of both musical compositions.

[0154] Comprehensive determination of these results of the determination makes it possible to determine the similarity between the copyright-claimed musical composition and the copyrighted original musical composition.

[0155] The determinations described above are performed by a person, and the determination requires specialized knowledge and requires a lot of time. In addition, in the above description, the copyright-claimed musical composition as a determination target for determining the similarity to the copyrighted original musical composition is determined in advance, but it is difficult to specify a musical composition having the similarity to the copyrighted original musical composition, from a large number of any pieces of musical compositions by the existing determination methods.

[0156] In the second embodiment, the information processing system 1 according to the first embodiment described above is used to facilitate determination on the copyright infringement.(3-2. Specific examples of second embodiment)

[0157] Next, the second embodiment of the present disclosure will be described more specifically.

[0158] First, a specific example of the UI screen according to the second embodiment will be described. FIGS. 14A and 14B are schematic diagrams each illustrating an example of a UI screen according to the second embodiment.

[0159] In the second embodiment, a predetermined position of the UI screen 500 may be specified by user operation or the like to perform display according to the second embodiment on the UI screen 500. For example, as illustrated in FIG. 14A, on a UI screen 500g equivalent to that in FIG. 12A described above, the bar 531 of an item "Overall" is specified, with a cursor 600 by user operation. For example, according to the operation of the cursor 600, the search unit 123 extracts the second musical composition data having the similarity of the overall musical composition equal to or larger than a threshold, from the music catalog 110. As illustrated in FIG. 14B, the UI generation unit 103 generates a list screen 551d including the second musical composition data extracted by the search unit 123. The UI generation unit 103 causes the UI screen 500g to display an area 550d including display of a fourth form in which the generated list screen 551d is arranged, the UI screen 500g serving as a UI screen 500h.

[0160] In the display of the fourth form, as illustrated in the area 550d, the second musical composition data similar to the first musical composition data as a whole is displayed as the list screen 551d having a ranking format according to the similarity of each piece of second musical composition data to the first musical composition data. Similarly to the list screen 551a illustrated in FIG. 13B, the list screen 551d includes, for example, an overall audio data waveform 555a and melody display 555b of each piece of second musical composition data, and a metadata icon 555c related to each piece of second musical composition data. In this example, as in FIG. 13B, the melody display 555b indicates the melodies according to the pitch and the length, with the horizontal axis as time and the vertical axis as pitch.

[0161] Among the pieces of second musical composition data displayed on the list screen 551d, the user can specify the second musical composition data for which he / she desires comparison with the first musical composition data, by user operation. For example, the user specifies the second musical composition data having the highest similarity, displayed at the top of the list screen 551d, as the second musical composition data for which he / she desires comparison with the first musical composition data, for example, with the cursor 600 by user operation. In the example of the drawing, the specification of the second musical composition data is indicated using a frame 556.

[0162] The UI generation unit 103 generates a UI screen that enables confirmation of a correspondence relationship of similar portions between the first musical composition data and the second musical composition data which is specified according to user operation. FIG. 15 is a schematic diagram illustrating an example of a UI screen displaying a correspondence relationship of similar portions between the first musical composition data and the second musical composition data according to the second embodiment. On a UI screen 500i illustrated in FIG. 15, an audio data waveform 570 based on the first musical composition data is displayed in the waveform display area 510. Furthermore, in a waveform display area 710, an audio data waveform 700 based on the second musical composition data specified with the cursor 600 in FIG. 14B is displayed.

[0163] For example, in the audio data waveform 570, the UI generation unit 103 displays a similar portion (similarity is equal to or larger than a predetermined value) similar to the second musical composition data in melody, in a range 580 including a melody (verse and pre-chorus). In the example of FIG. 15, in a melody portion of the audio data waveform 570, the position of the verse is illustrated as similar portions 581a and 581b that are similar to those of the second musical composition data. Furthermore, similar portions 721a and 721b corresponding to similar portions 581a and 581b, respectively, that is, similar to the similar portions 581a and 581b are illustrated, on the audio data waveform 700 of the second musical composition data.

[0164] Furthermore, the UI generation unit 103 may display a dissimilar portion of the melody that is not similar (similarity is less than the predetermined value) in melody to the second musical composition data in the range 580. In the example of FIG. 15, in the melody portion of the audio data waveform 570, the position of the pre-chorus is illustrated as a dissimilar portion 582 that is not similar to the second musical composition data. Furthermore, a dissimilar portion 722 corresponding to the dissimilar portion 582 is illustrated in the audio data waveform 700 of the second musical composition data.

[0165] In the example of FIG. 15, in the waveform display areas 510 and 710, the audio data waveforms 570 and 700 are arranged so that the similar portions 581a and 581b and the dissimilar portion 582, and the similar portions 721a and 721b and the dissimilar portion 722 correspond to each other in position in the time-series direction. In other words, in the example of FIG. 15, in the waveform display areas 510 and 710, the audio data waveforms 570 and 700 are arranged to have a relative positional relationship, on the basis of the similar portions 581a and 581b and the dissimilar portion 582, and the similar portions 721a and 721b and the dissimilar portion 722.

[0166] Arrangement of the audio data waveforms in the waveform display areas 510 and 710 is not limited to the example illustrated in FIG. 15. For example, in the waveform display areas 510 and 710, the respective audio data waveforms may be arranged with their absolute positions in the time-series direction matching each other.

[0167] FIG. 16 is a schematic diagram illustrating another example of a UI screen 500 displaying a correspondence relationship of similar portions between the first musical composition data and the second musical composition data according to the second embodiment. On a UI screen 500j illustrated in FIG. 16, in the waveform display areas 510 and 710, the audio data waveforms 570 and 701 are arranged with their absolute positions matching each other. For example, in the waveform display areas 510 and 710, the audio data waveforms 570 and 701 are arranged with their head positions aligned.

[0168] In the example of FIG. 16, as a result of arranging the audio data waveform 570 and the audio data waveform 701 so that the absolute positions in the time-series direction match each other, for example, the similar portion 581a in the audio data waveform 570 and a similar portion 721c in the audio data waveform 701, which are similar to each other between the audio data waveforms 570 and 701, are arranged at positions separated in the time-series direction.

[0169] As illustrated in FIGS. 15 and 16, explicit display of the similar portions between the first musical composition data and the second musical composition data, in each of the waveform display areas 510 and 710 makes it possible to readily grasp the presence / absence of possibility of causing a problem about copyright of the second musical composition data by the first musical composition data. In addition, when there is a possibility that the first musical composition data causes the problem about copyright of the second musical composition data, it is possible to readily take measures. Note that an example of the problem about copyright may be copyright infringement.

[0170] Note that, in the example of FIG. 16, a play button 621 is arranged for each of the similar portions 581a and 721c. Operation of the play button 621 arranged close to the similar portion 581a enables reproduction of sound based on the first musical composition data in the similar portion 581a. Similarly, operation of the play button 621 arranged close to the similar portion 721c enables reproduction of sound based on the second musical composition data in the similar portion 721c. The play button 621 can also be applied to the example of FIG. 15.

[0171] In this way, providing reproducing means at the similar portion, for reproducing sound of musical composition data in a similar portion makes it possible to more readily grasp the presence or absence of the possibility of causing the problem about copyright of the second musical composition data by the first musical composition data, and makes it possible to readily take measures when the possibility of causing the problem about copyright of the second musical composition data by the first musical composition data.(3-2-2. Specific examples of comparison between musical compositions according to second embodiment)

[0172] Next, the comparison of the musical composition data according to the second embodiment will be described more specifically with reference to FIGS. 17 to 19. Examples illustrated in FIGS. 17 to 19 may be displayed so as to be superimposed on the UI screen 500i or 500j or in place of the UI screen 500i or 500j by performing predetermined user operation on the UI screen 500i illustrated in FIG. 15 or the UI screen 500j illustrated in FIG. 16.

[0173] FIG. 17 is a schematic diagram illustrating a first example of comparing musical composition data according to the second embodiment. In the first example, it is possible to visually determine the similarity by comparing the audio data waveforms of the musical composition data.

[0174] For example, the UI generation unit 103 extracts audio data waveforms 80 and 81 of melody portions in the first musical composition data and the second musical composition data. The UI generation unit 103 causes the extracted audio data waveforms 80 and 81 to be displayed in a superimposed manner so as to be matched in position in the time series direction and matched in scale.

[0175] In the example of FIG. 17, the audio data waveforms 80 and 81 have normalized levels and are superimposed with the positions of level "0" shifted. The audio data waveforms 80 and 81 are not limited to this configuration, and may be superimposed with the positions of level "0" matching each other. Furthermore, in the example of FIG. 17, the audio data waveform 81 translucent is superimposed on the audio data waveform 80.

[0176] In this way, the audio data waveforms 80 and 81 at the similar portions are superimposed and displayed with the positions in the time-series direction matching, and the user can visually confirm the similar portions and the dissimilar portions of the audio data waveforms 80 and 81. In the example of FIG. 17, it can be seen that in the audio data waveforms 80 and 81 in the melody portions, a range 82 is a dissimilar portion and the other portions are similar portions.

[0177] In the example of FIG. 17, for example, the UI generation unit 103 may calculate and display a matching rate between the audio data waveform 80 and the audio data waveform 81. In this configuration, for the range 82 that is the dissimilar portion, the matching rate may be calculated and displayed separately. In this way, display of the matching rate of the audio data waveforms 80 and 81 makes it possible to quantitatively indicate the matching rate or the similarity.

[0178] In the first example described above, when the first musical composition data and the second musical composition data are different in sound forming a melody or a configuration of accompaniments to the melody, the comparison of the audio data waveforms may make determination of the similarity difficult. In addition, the audio data waveform may be different depending on data processing (mastering or the like) on the musical composition data, and it may be difficult to determine the similarity.

[0179] FIG. 18 is a schematic diagram illustrating a second example of comparing musical composition data according to the second embodiment. In the second example, visual determination of the similarity is enabled on the basis of musical score information of a melody estimated from the musical composition data by the MIDI estimation or the like.

[0180] In FIG. 18, the vertical axis represents pitch, the horizontal axis represents time, the scale of the vertical axis represents half tone steps, and the length of each block represents the length of the tone. In other words, the example of FIG. 18 is an example in which so-called piano roll display is applied, in which each block represents a note and represents a musical score 90 as a whole. The UI generation unit 103 generates the musical score 90 that includes, for example, notes according to a melody of the first musical composition data and notes according to a melody of the second musical composition data, that has pieces of musical composition data normalized in the time-series direction so that corresponding portions are matched in length, and that is transposed so as to be matched in key, causing the musical score 90 to be displayed.

[0181] In the musical score 90, a black block (note) 92 indicates that a note of the first musical composition data and a note of the second musical composition data overlap each other. Furthermore, blocks 91a, 91b, and 91c with different colors indicate different notes between the first musical composition data and the second musical composition data.

[0182] In this way, collective display of the first musical composition data and the second musical composition data on the musical score 90, and different musical note presentations according to overlapping of notes of the first musical composition data and the second musical composition data enable the user to readily visually confirm the similar portion and the dissimilar portion between the first musical composition data and the second musical composition data. In addition, presentation of the first musical composition data and the second musical composition data on the musical score 90 enables suppression of the influence due to the difference in the musical instrument configuration or due to the difference in audio processing between the first musical composition data and the second musical composition data.

[0183] In the second example described above, the melody is represented by the block indicating the pitch and the length of the tone by using the piano roll display, but the method of presenting the melody is not limited to this example. FIG. 19 is a schematic diagram illustrating a third example of comparing musical composition data according to the second embodiment. In the third example, the melody is subjected to curve approximation on the basis of the musical score information of the melody estimated from the musical composition data by the MIDI estimation or the like.

[0184] In FIG. 19, the vertical axis represents pitch, the horizontal axis represents time, and the scale of the vertical axis represents half tone steps. For example, the UI generation unit 103 performs normalization in the time-series direction so that the lengths of the corresponding portions (e.g., melody portions) of the first musical composition data and the second musical composition data match, and performs adjustment in pitch so as to match the key. Then, the UI generation unit 103 performs curve approximation on the change of the pitch in the time-series direction, for each melody of the first musical composition data and the second musical composition data to generate characteristic lines 95 and 96, and causes the characteristic lines to be displayed on the same chart.

[0185] In this way, the curve approximation on the change of the pitch in the time-series direction for display on the same chart, for each melody of the first musical composition data and the second musical composition data enables the user to readily visually confirm the similar portion and the dissimilar portion between the first musical composition data and the second musical composition data. In addition, the curve approximation on the melody of the first musical composition data and the melody of the second musical composition data enables suppression of the influence due to the difference in the musical instrument configuration or due to the difference in audio processing between the first musical composition data and the second musical composition data.

[0186] FIG. 20 is a flowchart illustrating an example of determination process for a possibility of causing the problem about copyright, according to the second embodiment. In the flowchart of FIG. 20, the UI generation unit 103 determines whether there is a possibility that the first musical composition data based on, for example, a musical composition written and composed by the user may infringe the copyright of the existing second musical composition data.

[0187] In Step S300, the UI generation unit 103 extracts a comparison target portion from the first musical composition data and the second musical composition data. For example, the UI generation unit 103 may extract the comparison target portions specified for the first and second musical composition data according to user operation in the terminal device 20, from the first and second musical composition data, or may extract the comparison target portions from the first and second musical composition data using the method described in the first embodiment.

[0188] In the next Step S301, the UI generation unit 103 adjusts the lengths of the comparison target portions so that the comparison target portions extracted from the first and second musical composition data have the same length. In the next Step S302, the UI generation unit 103 appropriately transposes the comparison target portions to match the keys of the comparison target portions.

[0189] In the next Step S303, the UI generation unit 103 calculates the matching degree of the pitch of each tone and the matching degree of the connection of the tones, for the comparison target portion of each of the first and second musical composition data which have been adjusted in length and key in Step S301 and step S302. The connection of the tones may be, for example, a change in pitch from a certain tone to the next tone.

[0190] In the next Step S304, the UI generation unit 103 specifies the similar portion and the dissimilar portion in the comparison target portion of the first and second musical composition data, on the basis of a result of the calculation in Step S303. For example, the UI generation unit 103 may evaluate the matching degree in the pitch and connection of the tones, in each portion obtained by dividing the comparison target portion in a time division manner, and may define a section having a predetermined matching degree as the similar portion and a section having a matching degree less than the predetermined value, as the dissimilar portion.

[0191] In the next Step S305, the UI generation unit 103 presents the similar portion specified in Step S304 to the user as a portion having a possibility of causing the problem about copyright of the second musical composition data by the first musical composition data. Using the presented information, the user can readily take measures such as change in arrangement of the specified similar portion for the first musical composition data.

[0192] Note that, in Step S300, when the comparison target portion is extracted from the first and second musical composition data by using the method described in the first embodiment, processing in Steps S301 to S304 can be skipped. Furthermore, in this case, the UI generation unit 103 may perform a process according to the flowchart of FIG. 20, on each of the pieces of the second musical composition data stored in the music catalog 110.(3-3. Modifications of second embodiment)

[0193] Next, a modification of the second embodiment will be described.

[0194] In recent years, a technique of generating an accompaniment or the like by using AI to assist composition has been put into practical use. Hereinafter, the AI that assists composition is referred to as composition assist AI. For example, with the composition assist AI, it is possible to accompany a melody created by a creator or a melody generated by the AI, with a harmony, decoration sound, and rhythm, automatically arrange the configuration of a musical composition, and work up the musical composition to some extent. Furthermore, for example, in the composition assist AI, the accompaniment or the configuration can be automatically generated similarly on the basis of the melody or lyrics created by the creator.

[0195] An input melody input by the creator to such composition assist AI may cause the problem of copyright of an existing musical composition. Therefore, applying a process according to the second embodiment, with the data of the input melody input to the composition assist AI, as the first musical composition data makes it possible to confirm the possibility of causing the problem of copyright by the input melody, before musical composition generation by the composition assist AI, and it is possible to avoid the copyright infringement of musical composition generated on the basis of the input melody by the composition assist AI.(4. Third embodiment of present disclosure)

[0196] Next, a third embodiment of the present disclosure will be described. In the third embodiment, the possibility of causing the problem of copyright by the first musical composition is determined using a model trained on the basis of a legal precedent relating to copyright infringement.(First method)

[0197] A first method according to the third embodiment will be described. The first method is a method of determining a possibility of copyright infringement of the first musical composition by using a copyright infringement determination device using a machine learning model trained by supervised learning on the basis of musical composition data used in the legal precedent relating to the copyright infringement of musical composition. In the first method, regardless of how the determination of the similarity is performed, the copyright infringement determination device is trained using only the result of the legal precedent. The copyright infringement determination device may be included, for example, in the UI generation unit 103.(Second method)

[0198] A second method according to the third embodiment will be described. The second method is a method of determining a possibility of copyright infringement of the first musical composition by using a copyright infringement determination device using a machine learning model trained by supervised learning on the basis of the legal precedent relating to the copyright infringement of musical composition and a musical composition data used in the legal precedent.

[0199] In the second method, a multimodal foundation model capable of multimodal processing of text data and audio data may be used. For example, a text data tokenizer and an audio data encoder are used to encode the data of each modal, and a vector is obtained for each modal. The obtained vectors for the respective modals are combined into one vector and input to a machine learning model constituting the copyright infringement determination device. The copyright infringement determination device may be included, for example, in the UI generation unit 103.

[0200] In each of the first method and the second method described above, for example, the first musical composition data and the second musical composition data are input to the copyright infringement determination device which uses the machine learning model trained by each method, and the copyright infringement determination device is instructed to determine the similarity between the input first and second musical composition data. The copyright infringement determination device may output, for example, a result "Because 70% of the comparison target portion is determined to match in pitch, the first musical composition data may have a possibility of copyright infringement concern." in response to the instruction.

[0201] Note that the effects described herein are merely examples and are not limited to the descriptions, and other effects may be provided.

[0202] Note that the present technology may also have the following configurations. (1) An information processing system comprising: an acquisition unit that acquires first musical composition data; and a presentation unit that presents information indicating similarity of the first musical composition data to second musical composition data as existing musical composition data, wherein the presentation unit causes a screen including a display of a first form indicating a similarity for each of a plurality of items for evaluating the similarity to musical composition data to be displayed. (2) The information processing system according to the above (1), wherein the presentation unit causes the screen including display of the first form indicating the similarity to at least one of the plurality of items including a melody, a chord, an impression, rhythm, lyrics, and an overall musical composition to be displayed. (3) The information processing system according to the above (2), further comprising a reception unit that accepts selection of a mode by a user, wherein the presentation unit switches display of the first form according to the selected mode. (4) The information processing system according to the above (3), wherein the mode includes a first mode relating to the second musical composition data including a portion having the similarity equal to or larger than a first threshold, a second mode relating to the second musical composition data including a portion having the similarity equal to or larger than the first threshold and being versatile, and a third mode relating to the second musical composition data including a portion having the similarity equal to or larger than a second threshold larger than the first threshold and being not versatile. (5) The information processing system according to any one of the above (1) to (4), wherein the presentation unit causes the screen including display of a second form in which information about each of the plurality of items is indicated in a time series to be displayed. (6) The information processing system according to the above (5), wherein the presentation unit, in the display of the second form, arranges information indicating a similar portion of the first musical composition data having similarity to the second musical composition data, for each of the plurality of items. (7) The information processing system according to the above (6), wherein the presentation unit causes the screen further including display of a third form indicating information about the second musical composition data corresponding to a specified similar portion to be displayed, among the similar portions arranged in the display of the second form. (8) The information processing system according to any one of the above (5) to (7), wherein the presentation unit further causes the screen to display a waveform of the first musical composition data corresponding to the time series in display of the second form. (9) The information processing system according to any one of the above (1) to (8), further comprising: a storage unit that stores the second musical composition data; and a similarity evaluation unit that evaluates the similarity of the first musical composition data to the second musical composition data, for each of the plurality of items. (10) The information processing system according to the above (9), wherein the similarity evaluation unit obtains a first feature amount of each of portions obtained by dividing the first musical composition data in a time division manner, and a second feature amount of each of portions obtained by dividing the second musical composition data in a time division manner, and compares the first feature amount and the second feature amount with each of portions obtained by dividing the first musical composition data in a time division manner and each of portions obtained by dividing the second musical composition data in a time division manner to evaluate similarity of each of portions obtained dividing the first musical composition data in a time division manner to each of portions obtained by dividing the second musical composition data in a time division manner. (11) The information processing system according to the above (10), wherein the similarity evaluation unit performs f0 estimation on audio source data including a melody, among pieces of audio source data extracted by performing audio source separation on musical composition data, generates a machine learning model trained by unsupervised learning based on a waveform obtained by the f0 estimation and an augmented waveform obtained by extending the waveform, and obtains the first feature amount and the second feature amount by using the machine learning model, based on each of the first musical composition data and the second musical composition data. (12) The information processing system according to the above (11), wherein the similarity evaluation unit performs low-pass filtering and normalization in a time-series direction, on the waveform obtained by the f0 estimation. (13) The information processing system according to the above (11) or (12), wherein the similarity evaluation unit normalization in a time-series direction, on the waveform obtained by the f0 estimation. (14) The information processing system according to any one of the above (11) to (13), wherein the similarity evaluation unit applies global average pooling to an output layer of the machine learning model. (15) The information processing system according to any one of the above (1) to (14), wherein the presentation unit extracts, as a similar portion, from the first musical composition data, a portion having a predetermined similarity or more to at least a part of the second musical composition data, and presents the similar portion and a corresponding portion to the similar portion in the second musical composition data, in association with each other. (16) The information processing system according to the above (15), wherein the presentation unit causes a waveform of the first musical composition data and a waveform of the second musical composition data to be aligned in a time-series direction on one screen to be displayed, causes the similar portion on the waveform of the first musical composition data to be displayed, and causes the corresponding portion on the waveform of the second musical composition data to be displayed. (17) The information processing system according to the above (16), wherein the presentation unit causes a waveform of the similar portion in the first musical composition data and a waveform of the corresponding portion in the second musical composition data to be displayed in a partially overlapped manner so that one of the waveforms is translucently superimposed on the other. (18) The information processing system according to the above (15), wherein the presentation unit causes first musical score information based on the first musical composition data and second musical score information based on the second musical composition data to be displayed on one screen, so to be aligned in a time-series direction, causes the similar portion to be displayed on the first musical score information, and causes the corresponding portion to be displayed on the second musical score information. (19) The information processing system according to the above (18(, wherein the presentation unit causes the first musical score information and the second musical score information to be displayed, with an axis in a pitch direction and a time axis in common. (20) The information processing system according to asny one of the above (15) to (19), wherein the presentation unit causes a screen including a display of a fourth form displaying a list of the second musical composition data in which the similar portion is extracted from the first musical composition data to be displayed, and presents the similar portion and the corresponding portion of the second musical composition data specified from the display of the list, in association with each other, in the display of the fourth form. (21) The information processing system according to any one of the above (1) to (20), further comprising a similarity determination unit that determines the similarity by using a machine learning model trained on the basis of a legal precedent relating to copyright on musical composition data. (22) The information processing system according to the above (21), wherein the similarity determination unit determines the similarity by using a machine learning model trained by supervised learning based on the musical composition data used in the legal precedent. (23) The information processing system according to the above (22), wherein the similarity determination unit determines the similarity by using a machine learning model trained by supervised learning further based on text of the legal precedent. (24) An information processing method comprising: an acquisition step of acquiring first musical composition data; and a presentation step of presenting information indicating similarity of the first musical composition data to second musical composition data as existing musical composition data, wherein the presentation step causes a screen including a display of a first form indicating a similarity for each of a plurality of items for evaluating the similarity to musical composition data to be displayed. (25) A terminal device comprising: a communication unit that communicates with an information processing device; a display unit that displays information indicating similarity of first musical composition data to second musical composition data as existing musical composition data, the information being presented from the information processing device according to the first musical composition data transmitted to the information processing device by the communication unit, wherein the display unit displays a screen including a display of a first form indicating a similarity for each of a plurality of items for evaluating the similarity to musical composition data. (26) The terminal device according to the above (25), further comprising an operation reception unit that receives user operation, wherein the communication unit transmits mode specification received by the operation reception unit according to the user operation, to the information processing device, and the display unit displays a screen including display of the first form switched in the information processing device according to the mode specification. (27) The terminal device according to the above (25) or (26), wherein the display unit displays the screen including display of a second form indicating information of each of the plurality of items presented from the information processing device, in a time series. (28) The terminal device according to the above (27), wherein in the display of the second form, information indicating a similar portion similar to the second musical composition data in the first musical composition data is arranged for each of the plurality of items. (29) The terminal device according to the above (28), wherein the display unit displays the screen further including display of a third form indicating information about the second musical composition data corresponding to a specified similar portion, presented by the information processing device, among the similar portions arranged in the display of the second form. (30) The terminal device according to any one of the above (27) to (29), wherein the display unit further causes a waveform of the first musical composition data transmitted from the information processing device to be displayed, in association with the time series in the display of the second form. (31) The terminal device according to any one of the above (25) to (30), wherein the display unit displays a similar portion of a portion having a predetermined similarity or more to at least a part of the second musical composition data extracted from the first musical composition data, and a corresponding portion corresponding to the similar portion in the second musical composition data, in association with each other. (32) The terminal device according to the above (31), wherein the display unit displays a waveform of the first musical composition data and a waveform of the second musical composition data to be aligned in a time-series direction on one screen, displays the similar portion on the waveform of the first musical composition data, and displays the corresponding portion on the waveform of the second musical composition data. (33) The terminal device according to the above (32), wherein the display unit displays a waveform of the similar portion in the first musical composition data and a waveform of the corresponding portion in the second musical composition data in a partially overlapped manner so that one of the waveforms is translucently superimposed on the other. (34) The terminal device according to the above (31), wherein the display unit displays first musical score information based on the first musical composition data and second musical score information based on the second musical composition data on one screen, so to be aligned in a time-series direction, displays the similar portion on the first musical score information, and displays the corresponding portion on the second musical score information. (35) The terminal device according to the above (34), wherein the display unit displays the first musical score information and the second musical score information with an axis in a pitch direction and a time axis in common. (36) The terminal device according to any one of the above (31) to (35), wherein the display unit displays a screen including a display of a fourth form displaying a list of the second musical composition data in which the similar portion is extracted from the first musical composition data, and displays the similar portion and the corresponding portion of the second musical composition data specified from the display of the list, in association with each other, in the display of the fourth form, according to user operation. Reference Signs List

[0203] 1 INFORMATION PROCESSING SYSTEM 10 SERVER 11 STORAGE UNIT 20 TERMINAL DEVICE 30, 60 INPUT DATA 61a, 61b, 61c, 61d AUDIO SOURCE DATA 62 f0 DATA 63 NORMALIZED DATA 72 SEARCH MODEL 102 INFORMATION PROCESSING UNIT 103 UI GENERATION UNIT 110 MUSIC CATALOG 120 MUSICAL COMPOSITION DATA COLLECTION UNIT 121a, 121b ANALYSIS UNIT 122, 202 MUSICAL COMPOSITION DATA ACQUISITION UNIT 123 SEARCH UNIT 203 DISPLAY UNIT 204 OPERATION RECEPTION UNIT 500, 500a, 500b, 500c, 500d, 500e, 500f, 500g, 500h, 500i, 500j UI SCREEN 510, 710 WAVEFORM DISPLAY AREA 521 MODE SELECTION AREA 531 BAR 541, 541a 1 , 541a 2 , 541a 3 , 541a 21 , 541b 1 , 541c 1 , 541c 2 , 541c 11 , 541d 1 , 541e 1 , 541e 2 , 541e 3 , 541f 1 , 541f 2 , 541f 3 , 541f 11 , 541f 21 SIMILAR PORTION 570, 700 AUDIO DATA WAVEFORM 581a, 581b, 721a, 721b SIMILAR PORTION 582, 722 DISSIMILAR PORTION 600 CURSOR 621 PLAY BUTTON 2005 INPUT DEVICE 2020 DISPLAY DEVICE

Claims

1. An information processing system comprising: an acquisition unit that acquires first musical composition data; and a presentation unit that presents information indicating similarity of the first musical composition data to second musical composition data as existing musical composition data, wherein the presentation unit causes a screen including a display of a first form indicating a similarity for each of a plurality of items for evaluating the similarity to musical composition data to be displayed.

2. The information processing system according to claim 1, wherein the presentation unit causes the screen including display of the first form indicating the similarity to at least one of the plurality of items including a melody, a chord, an impression, rhythm, lyrics, and an overall musical composition to be displayed.

3. The information processing system according to claim 2, further comprising a reception unit that accepts selection of a mode by a user, wherein the presentation unit switches display of the first form according to the selected mode.

4. The information processing system according to claim 3, wherein the mode includes a first mode relating to the second musical composition data including a portion having the similarity equal to or larger than a first threshold, a second mode relating to the second musical composition data including a portion having the similarity equal to or larger than the first threshold and being versatile, and a third mode relating to the second musical composition data including a portion having the similarity equal to or larger than a second threshold larger than the first threshold and being not versatile.

5. The information processing system according to claim 1, wherein the presentation unit causes the screen including display of a second form in which information about each of the plurality of items is indicated in a time series to be displayed.

6. The information processing system according to claim 5, wherein the presentation unit, in the display of the second form, arranges information indicating a similar portion of the first musical composition data having similarity to the second musical composition data, for each of the plurality of items.

7. The information processing system according to claim 6, wherein the presentation unit causes the screen further including display of a third form indicating information about the second musical composition data corresponding to a specified similar portion to be displayed, among the similar portions arranged in the display of the second form.

8. The information processing system according to claim 5, wherein the presentation unit further causes the screen to display a waveform of the first musical composition data corresponding to the time series in display of the second form.

9. The information processing system according to claim 1, further comprising: a storage unit that stores the second musical composition data; and a similarity evaluation unit that evaluates the similarity of the first musical composition data to the second musical composition data, for each of the plurality of items.

10. The information processing system according to claim 9, wherein the similarity evaluation unit obtains a first feature amount of each of portions obtained by dividing the first musical composition data in a time division manner, and a second feature amount of each of portions obtained by dividing the second musical composition data in a time division manner, and compares the first feature amount and the second feature amount with each of portions obtained by dividing the first musical composition data in a time division manner and each of portions obtained by dividing the second musical composition data in a time division manner to evaluate similarity of each of portions obtained dividing the first musical composition data in a time division manner to each of portions obtained by dividing the second musical composition data in a time division manner.

11. The information processing system according to claim 10, wherein the similarity evaluation unit performs f0 estimation on audio source data including a melody, among pieces of audio source data extracted by performing audio source separation on musical composition data, generates a machine learning model trained by unsupervised learning based on a waveform obtained by the f0 estimation and an augmented waveform obtained by extending the waveform, and obtains the first feature amount and the second feature amount by using the machine learning model, based on each of the first musical composition data and the second musical composition data.

12. The information processing system according to claim 11, wherein the similarity evaluation unit performs low-pass filtering and normalization in a time-series direction, on the waveform obtained by the f0 estimation.

13. The information processing system according to claim 11, wherein the similarity evaluation unit applies global average pooling to an output layer of the machine learning model.

14. The information processing system according to claim 1, wherein the presentation unit extracts, as a similar portion, from the first musical composition data, a portion having a predetermined similarity or more to at least a part of the second musical composition data, and presents the similar portion and a corresponding portion to the similar portion in the second musical composition data, in association with each other.

15. The information processing system according to claim 14, wherein the presentation unit causes a waveform of the first musical composition data and a waveform of the second musical composition data to be aligned in a time-series direction on one screen to be displayed, causes the similar portion on the waveform of the first musical composition data to be displayed, and causes the corresponding portion on the waveform of the second musical composition data to be displayed.

16. The information processing system according to claim 15, wherein the presentation unit causes a waveform of the similar portion in the first musical composition data and a waveform of the corresponding portion in the second musical composition data to be displayed in a partially overlapped manner so that one of the waveforms is translucently superimposed on the other.

17. The information processing system according to claim 14, wherein the presentation unit causes first musical score information based on the first musical composition data and second musical score information based on the second musical composition data to be displayed on one screen, so to be aligned in a time-series direction, causes the similar portion to be displayed on the first musical score information, and causes the corresponding portion to be displayed on the second musical score information.

18. The information processing system according to claim 17, wherein the presentation unit causes the first musical score information and the second musical score information to be displayed, with an axis in a pitch direction and a time axis in common.

19. The information processing system according to claim 14, wherein the presentation unit causes a screen including a display of a fourth form displaying a list of the second musical composition data in which the similar portion is extracted from the first musical composition data to be displayed, and presents the similar portion and the corresponding portion of the second musical composition data specified from the display of the list, in association with each other, in the display of the fourth form.

20. The information processing system according to claim 1, further comprising a similarity determination unit that determines the similarity by using a machine learning model trained on the basis of a legal precedent relating to copyright on musical composition data.

21. The information processing system according to claim 20, wherein the similarity determination unit determines the similarity by using a machine learning model trained by supervised learning based on the musical composition data used in the legal precedent.

22. The information processing system according to claim 21, wherein the similarity determination unit determines the similarity by using a machine learning model trained by supervised learning further based on text of the legal precedent.

23. An information processing method comprising: an acquisition step of acquiring first musical composition data; and a presentation step of presenting information indicating similarity of the first musical composition data to second musical composition data as existing musical composition data, wherein the presentation step causes a screen including a display of a first form indicating a similarity for each of a plurality of items for evaluating the similarity to musical composition data to be displayed.

24. A terminal device comprising: a communication unit that communicates with an information processing device; a display unit that displays information indicating similarity of first musical composition data to second musical composition data as existing musical composition data, the information being presented from the information processing device according to the first musical composition data transmitted to the information processing device by the communication unit, wherein the display unit displays a screen including a display of a first form indicating a similarity for each of a plurality of items for evaluating the similarity to musical composition data.

25. The terminal device according to claim 24, further comprising an operation reception unit that receives user operation, wherein the communication unit transmits mode specification received by the operation reception unit according to the user operation, to the information processing device, and the display unit displays a screen including display of the first form switched in the information processing device according to the mode specification.

26. The terminal device according to claim 24, wherein the display unit displays the screen including display of a second form indicating information of each of the plurality of items presented from the information processing device, in a time series.

27. The terminal device according to claim 26, wherein in the display of the second form, information indicating a similar portion similar to the second musical composition data in the first musical composition data is arranged for each of the plurality of items.

28. The terminal device according to claim 27, wherein the display unit displays the screen further including display of a third form indicating information about the second musical composition data corresponding to a specified similar portion, presented by the information processing device, among the similar portions arranged in the display of the second form.

29. The terminal device according to claim 26, wherein the display unit further causes a waveform of the first musical composition data transmitted from the information processing device to be displayed, in association with the time series in the display of the second form.

30. The terminal device according to claim 24, wherein the display unit displays a similar portion of a portion having a predetermined similarity or more to at least a part of the second musical composition data extracted from the first musical composition data, and a corresponding portion corresponding to the similar portion in the second musical composition data, in association with each other.

31. The terminal device according to claim 30, wherein the display unit displays a waveform of the first musical composition data and a waveform of the second musical composition data to be aligned in a time-series direction on one screen, displays the similar portion on the waveform of the first musical composition data, and displays the corresponding portion on the waveform of the second musical composition data.

32. The terminal device according to claim 31, wherein the display unit displays a waveform of the similar portion in the first musical composition data and a waveform of the corresponding portion in the second musical composition data in a partially overlapped manner so that one of the waveforms is translucently superimposed on the other.

33. The terminal device according to claim 30, wherein the display unit displays first musical score information based on the first musical composition data and second musical score information based on the second musical composition data on one screen, so to be aligned in a time-series direction, displays the similar portion on the first musical score information, and displays the corresponding portion on the second musical score information.

34. The terminal device according to claim 33, wherein the display unit displays the first musical score information and the second musical score information with an axis in a pitch direction and a time axis in common.

35. The terminal device according to claim 30, wherein the display unit displays a screen including a display of a fourth form displaying a list of the second musical composition data in which the similar portion is extracted from the first musical composition data, and displays the similar portion and the corresponding portion of the second musical composition data specified from the display of the list, in association with each other, in the display of the fourth form, according to user operation.

Images