Information processing device, information processing method and information processing program

Abstract

According to the present disclosure, the information processing device includes: an acquisition unit that acquires music information; an extraction unit that extracts multiple types of feature quantities from the music information acquired by the acquisition unit; and a generation unit that generates information in which the multiple types of feature quantities extracted by the extraction unit are associated with predetermined identification information as music feature information, and the music feature information is used as learning data in a composition process using machine learning.

Description

[0001] Related Applications Technical Field

[0002] The present disclosure relates to an information processing device, an information processing method, and an information processing program. Background Art

[0003] With the advancement of artificial intelligence (AI), the application of computers in this field has made progress. For example, a technology is known in which machine learning is performed on existing music as learning data to generate a model for music generation, and a computer is caused to compose new music (for example, Patent Document 1). In such a technology, it is possible to imitate the characteristics of existing music or generate a more natural melody by using a Markov model.

[0004] Reference List

[0005] Patent Literature

[0006] Patent document 1: US 9,110,817 Summary of the invention

[0007] Technical issues

[0008] According to the conventional technology, since music information proposed (generated) by AI can be used for composition work, users can compose music based on more various viewpoints.

[0009] The automatic composition function of AI is set for general users, and general users can only receive automatically composed music information by setting images such as light and dark. On the other hand, since the maker of the music often specifically sets the characteristics of the music, such as chord progression and bass progression, in the process of composing music, the maker requires to receive the provision of music information that matches the characteristics of the music rather than images.

[0010] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that can improve the convenience of music creation for users.

[0011] Solution to the problem

[0012] In order to solve the above-mentioned problem, the information processing device according to the present disclosure includes: an acquisition unit, which acquires music information; an extraction unit, which extracts multiple types of feature quantities from the music information acquired by the acquisition unit; and a generation unit, which generates information in which the multiple types of feature quantities extracted by the extraction unit are associated with predetermined identification information as music feature information, and the music feature information is used as learning data in a composition process using machine learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a conceptual diagram showing the flow of information processing according to an embodiment.

[0014] Figure 2 is a diagram showing an example of a data configuration of style information according to an embodiment.

[0015] Figure 3 is a diagram showing an example of a display screen of a user terminal according to an embodiment.

[0016] Figure 4 is a diagram showing an example of a display screen of a user terminal according to an embodiment.

[0017] Figure 5 is a diagram showing an example of a display screen of a user terminal according to an embodiment.

[0018] Figure 6 is a diagram showing an example of a display screen of a user terminal according to an embodiment.

[0019] Figure 7 is a diagram showing an example of a display screen of a user terminal according to an embodiment.

[0020] Figure 8 is a conceptual diagram showing the flow of information processing according to the embodiment.

[0021] Fig. 9 is a diagram showing an example of an information processing system according to an embodiment.

[0022] Fig.10 The configuration of the copyrighted work management device according to the embodiment is explained.

[0023] Fig.11 is a diagram showing an example of a copyright-protected music information storage unit according to an embodiment.

[0024] Fig.12 is a diagram showing an example of a music storage unit according to an embodiment.

[0025] Fig.13 is a diagram illustrating an example of a feature information storage unit according to an embodiment.

[0026] Fig.14 is a diagram illustrating a configuration of overall information according to an embodiment.

[0027] Fig.15 : is a diagram showing an example of time series information constituting music feature information according to an embodiment.

[0028] Fig.16 is a diagram showing a configuration example of an information processing apparatus according to an embodiment.

[0029] Fig.17 is a diagram illustrating an example of a user information storage unit according to an embodiment.

[0030] Fig.18 is a diagram illustrating an example of a style information storage unit according to an embodiment.

[0031] Fig.19 A diagram showing an example of an ownership information storage unit according to an embodiment;

[0032] Fig. 20 : is a diagram showing an example of a production information storage unit according to an embodiment.

[0033] Fig.21 is a diagram showing a configuration example of a user terminal according to an embodiment.

[0034] Fig. 22 is a sequence diagram showing the procedure of information processing according to the embodiment.

[0035] Fig.23 is a sequence diagram showing the procedure of information processing according to the embodiment.

[0036] Fig.24 is a sequence diagram showing the procedure of information processing according to the embodiment.

[0037] Fig.25 A diagram showing an example of a conceptual diagram of a configuration of an information processing system.

[0038] Fig.26 is a diagram illustrating an example of a user interface according to an embodiment.

[0039] Fig. 27 is a diagram illustrating an example of a user interface according to an embodiment.

[0040] Fig.28 is a hardware configuration diagram showing an example of a computer that realizes the functions of an information processing device, a copyrighted work management device, and a user terminal. DETAILED DESCRIPTION

[0041] The following will describe the embodiments of the present disclosure in detail based on the accompanying drawings. It should be noted that the information processing device, information processing method, and information processing program according to the present embodiment are not limited by the embodiment. In addition, in each embodiment described below, the same parts are represented by the same reference numerals, and repeated descriptions will be omitted.

[0042] The present disclosure will be described in the order of the items described below.

[0043] 1. Implementation Method

[0044] 1-1. Example of information processing according to the present embodiment

[0045] 1-2. Configuration of information processing system according to the present embodiment

[0046] 1-3. Configuration of Copyrighted Works Management Device According to the Present Embodiment

[0047] 1-4. Configuration of Information Processing Device According to the Present Embodiment

[0048] 1-5. Configuration of User Terminal According to Embodiment

[0049] 1-6. Information Processing Procedure According to the Present Embodiment

[0050] 1-6-1. Processing of generating style information of copyrighted music information

[0051] 1-6-2. Processing of generating style information of music information created by a producer

[0052] 1-6-3. Processing of updating style information

[0053] 1-7. Conceptual diagram of information processing system configuration

[0054] 1-7-1. About the overall configuration

[0055] 1-7-2. About copyright-protected music management server equipment

[0056] 1-7-3. About server equipment

[0057] 1-7-4. About the Music Maker Application Unit

[0058] 2. Effects of this embodiment

[0059] 3. Other Implementation Methods

[0060] 3-1. Other configuration examples

[0061] 3-2. Others

[0062] 4. Hardware Configuration

[0063] [1. Implementation Method]

[0064] [1-1. Example of information processing according to embodiment]

[0065] First, refer to Figure 1 An example of information processing according to the present disclosure is described. Figure 12 is a conceptual diagram showing the flow of information processing according to the embodiment. The information processing according to the embodiment is performed by the copyrighted work management apparatus 100, the information processing apparatus 200, and the user terminal 300.

[0066] In the present embodiment, a case where the information processing device 200 is an information processing device that provides services related to creation of content (information) as a copyrighted work (also referred to as "service") will be described as an example. It should be noted that, hereinafter, music (music content) is described as an example of content, but the content is not limited to music, and may be various types of content such as video content of a movie or character content such as a book (novel, etc.). In addition, the music referred to herein is not limited to one complete music (whole), and is a concept including a part of a sound source constituting one song (music) and various music information such as short sounds used for sampling.

[0067] The information processing device 200 is connected to the dedicated network N2 (see Fig. 9 ) communicates with the copyrighted work management device 100 that manages copyrighted music information. The private network N2 is, for example, a secure virtual private network (VPN) with ID address authentication or Internet communication. The information processing device 200 communicates with the copyrighted work management device 100 that manages copyrighted music information by using the network N1 such as the Internet (see Fig. 9 ) communicates with the user terminal 300 of the user who uses the service provided by the information processing device 200. It should be noted that the number of the user terminals 300 is not limited to Figure 1 The quantity shown in .

[0068] The copyrighted work management device 100 is a server device that registers and manages copyrighted music information. The copyrighted work management device 100 periodically registers copyrighted music information. The copyrighted work management device 100 extracts multiple types of feature quantities from the registered copyrighted music information, and transmits the extracted feature quantities to the information processing device 200 via the dedicated network N2.

[0069] The user terminal 300 is an information processing terminal such as a personal computer (PC) or a tablet terminal. Various program applications are installed in the user terminal 300. Applications related to music creation are installed in the user terminal 300. For example, the user terminal 300 has an automatic composition function through AI, which is added by a plug-in (extended application) to an application such as DAW that implements a comprehensive music production environment. For example, the plug-in may take the form of Steinberg's Virtual Studio Technology (VST) (registered trademark), AudioUnits, Avid Audio eXtension (AAX), etc. In addition, the user terminal 300 is not limited to DAW, and mobile applications (such as iOS) may be used, for example.

[0070] The user terminal 300 activates and executes the automatic composition function of the DAW and AI, communicates with the information processing device 200 , and receives provision of music information composed by the information processing device 200 .

[0071] The user of the user terminal 300 is any one of a manager who operates and manages the entire system, a composer who creates music, an arranger, a producer such as a studio engineer, and a general user who receives provision of music information via the automatic composition function. In this embodiment, it is assumed that the producer Uc uses the user terminal 300.

[0072] The information processing device 200 is a server device that performs information processing related to the automatic composition function of the AI ​​of the user terminal 300. For example, the information processing device 200 is a so-called cloud server, performs automatic composition by AI according to instruction information of the user terminal 300, and provides the generated music information to the user terminal 300.

[0073] The information processing device 200 performs machine learning to generate a composition model for music generation. For example, the information processing device 200 provides the user terminal 300 with music information automatically composed using a Markov model or the like.

[0074] The information processing device 200 uses style information (music feature information) as learning data for a composition model. Style information is information in which multiple types of feature quantities such as chord progression, melody, bass progression, etc. extracted from music information are associated with predetermined identification information and used for composition processing using machine learning. The information processing device 200 obtains multiple types of feature quantities from copyrighted music information or music information created by a producer, compiles the feature quantities, and assigns a style information ID (predetermined identification information) to each piece of music information to generate multiple pieces of style information and create a database.

[0075] Figure 2 7 is a diagram showing an example of a data configuration of style information according to an embodiment. The style information includes style information ID 710 as identification information of the style information, style palette sequence information 720 (music sequence information), style palette information 730 (music format information), score information 740, and lyrics information 750.

[0076] The score information 740 includes multiple types of feature quantities extracted from music. The score information 740 includes score ID, melody information, chord progression information, bass information, and drum information. The score ID is identification information of the score information. The melody information is a melody in a measure having a prescribed length. The chord progression information is information indicating a chord progression in a measure having a prescribed length. The bass information is information indicating a bass sound progression in a measure having a prescribed length. The drum information is information indicating a drum sound progression (a pattern or rhythm of a drum) in a measure having a prescribed length.

[0077] The lyrics information 750 includes a lyrics ID and lyrics information. The lyrics ID is identification information of the lyrics information. The lyrics information is information indicating the lyrics in a measure having a prescribed length. The lyrics information is, for example, a phrase or a character keyword as a source of lyrics. The information processing device 200 can also perform automatic lyrics writing by using a plurality of lyrics information 750 of the style information 700.

[0078] The style palette information 730 is information in which the score ID of the score information 740 and the lyrics ID of the lyrics information 750 for the same measure are registered in association with the style palette ID which is identification information of the style palette information.

[0079] The style palette sequence information 720 is information indicating the order of the style palette information 730. The style palette sequence information 720 includes a plurality of sets, each set including a style palette ID and a bar index that uniquely indicates the style palette information 730, so as to serve as information for managing the order of the style palette information 730 in the music. Figure 2 In the case of the example shown in , the first to fourth measures of the defined music correspond to the style palette ID 731a, the fifth to eighth measures correspond to the style palette ID 731b, and the xth to yth measures correspond to the style palette ID 731z.

[0080] The information processing device 200 performs machine learning using the style information 700 as learning data and performs composition processing. Therefore, the information processing device 200 does not learn the music information itself, but learns the style information including multiple types of feature quantities such as chord progression, melody, bass progression, etc. extracted from the music information. That is, because the information processing device 200 learns multiple feature quantities extracted in advance from the music information, the load of information processing is small compared to double learning the music information itself, and the music information can be effectively provided to the user.

[0081] Specifically, the process of music creation by the producer will be described. Figures 3 to 7 3 is a diagram showing an example of a display screen of the user terminal 300 according to an embodiment. When the producer activates the automatic composition function on the user terminal 300, Figure 3 A window 370 shown in FIG. 3 is displayed on the user terminal 300 .

[0082] Note that the window 370 includes a composition parameter setting unit 371, a style information display unit 372, a composition control unit 373, and a produced music display editing unit 374. The composition parameter setting unit 371 is an area in which parameters such as note duration and complexity can be set. The style information display unit 372 is an area in which style information to be used for composition can be selected by keyword input or drop-down selection. The composition control unit 373 is an area in which composition instructions can be made by selecting a composition execution instruction button. The produced music display editing unit 374 is an area in which a plurality of piano rolls showing the melody and lyrics are displayed.

[0083] Then, if Figure 4 As shown, when the style palette selection pull-down 372a is selected, the chord progression of each style information included in the information processing device 200 is displayed in a list as a candidate.

[0084] The chord progression candidates may be displayed in any order, such as alphabetical order, order of largest number of times used by the producer, order of largest number of times used by all users, and order of generation of the style information. With regard to the chord progression, all or only a portion of the style information included in the information processing device 200 may be displayed. When there are many chord progression candidates, a display area may be selected with a pager. In addition, when the producer inputs a desired chord progression in the search keyword input field 372b, the information processing device 200 may extract the style information including the chord progression and display a list of the chord progression information for each piece of the extracted style information in the style palette selection drop-down 372a.

[0085] The producer selects a desired chord progression from the chord progressions presented in the style palette selection drop-down 372a, and selects the composition execution instruction button. Thus, the information processing device 200 extracts style information having the selected chord progression, performs machine learning using the extracted style information 700 as learning data, and performs composition processing. Then, the information processing device 200 provides the music information to the user terminal 300. As a result, the producer can receive provision of music information generated according to the chord progression simply by selecting a desired chord progression from the chord progressions presented in the style palette selection drop-down 372a.

[0086] In addition, since the style information 700 includes lyrics information as a feature amount, the producer can receive presentation of style information matching the lyrics simply by inputting desired lyrics. Figure 5 As shown, when the producer enters the desired lyrics in the search keyword input field 372b, the information processing device 200 can extract style information having lyrics or lyrics similar to the lyrics and display a list of lyrics information of each extracted style information in the style palette selection drop-down 372a.

[0087] The producer selects desired lyrics information from the lyrics information presented in the style palette selection drop-down 372a and selects the composition execution instruction button. Thus, the information processing device 200 extracts the style information having the selected lyrics information, performs machine learning using the extracted style information 700 as learning data, performs composition processing, and provides music information to the user terminal 300.

[0088] As a result, the producer can receive provision of music information generated from lyrics simply by selecting desired lyrics from the lyrics presented in the style palette selection drop-down 372a. At this time, the information processing device 200 can automatically generate lyrics based on the generated music, and provide the user terminal 300 with music information in which the melody is associated with the lyrics. In this case, on the screen of the user terminal 300, Figure 6 The melody display piano roll 374a displays the melody and the lyrics corresponding to the melody.

[0089] In addition, if Figure 7 As shown, in the case where the producer inputs lyrics, the user terminal 300 can display a candidate list of chord progressions of the style information presented from the information processing device 200 and support the producer's music creation.

[0090] As described above, the information processing device 200 generates style information having multiple types of feature quantities of music information as a learning data set for the composition model, and causes the composition model to learn the style information. Thus, the information processing device 200 provides the producer with music information composed according to the characteristics of the music. Figure 1 The flow of style information generation processing in information processing according to the present embodiment will be described.

[0091] like Figure 1 As shown, when copyrighted music information is newly registered (step S11), the copyrighted work management device 100 extracts multiple types of feature quantities from each newly registered copyrighted music information (step S12). At this time, for example, the copyrighted work management device 100 extracts the feature quantities using a twelve-tone analysis technique. Then, the information processing device 200 acquires multiple types of feature quantities from the copyrighted work management device 100 (step S13), compiles the feature quantities, and assigns a style information ID to each piece of music information to generate style information corresponding to each piece of music information (step S14).

[0092] Furthermore, when a feature quantity such as a chord progression, melody, bass progression, etc. is input through the operation of the producer Uc of the user terminal 300 to create new music, the information processing device 200 acquires music information including each feature quantity (step S21). Then, the information processing device 200 extracts the feature quantity such as the chord progression, melody, bass progression, etc. from the acquired music information (step S22). Then, the information processing device 200 generates style information corresponding to each piece of music information by compiling the feature quantity and assigning a style information ID to each piece of music information (step S23). The information processing device 200 generates style information corresponding to each piece of music information by executing Figure 1 The process shown in generates a plurality of pieces of style information 700 and creates a database.

[0093] Then, in the information processing according to the present embodiment, the style information can be updated. Figure 8 2 is a conceptual diagram showing the flow of information processing according to this embodiment. Figure 8 As shown, while the producer Uc is composing music, when the producer Uc selects style information (step S31), the information processing device 200 learns the style information to perform composition processing, and outputs the music information to the user terminal 300 (step S32). The user Uc further performs a performance based on the provided music information. When the performance information is received (step S33), the information processing device 200 adds the performance information to the style information used for composition and updates the style information (step S34).

[0094] As described above, in the information processing according to this embodiment, the style information used is updated according to the music creation of the producer Uc. Thus, the information processing device 200 can make the style information closer to the music creation style of the producer Uc, and arrange and provide music information matching the style of the producer Uc.

[0095] The above has described an overview of the overall flow of information processing according to the present embodiment. Fig. 9 In the following drawings, the configuration of the information processing system 1 including the copyrighted work management apparatus 100, the information processing apparatus 200, and the user terminal 300 will be described, and the details of various processes will be described in order.

[0096] [1-2. Configuration of information processing system according to embodiment]

[0097] Fig. 9 1 is a diagram showing an example of an information processing system 1 according to an embodiment. Fig. 9 As shown, the information processing system 1 includes user terminals 300-1 to 300-3, an information processing device 200, and a copyrighted work management device 100. The information processing system 1 is used as a copyrighted work management system or a learning model information management system. Fig. 9 In the example of FIG. 1 , three user terminals 300 - 1 to 300 - 3 are shown, but are referred to as user terminal 300 when not particularly distinguished.

[0098] The information processing apparatus 200 and the user terminal 300 are communicably connected to each other by wire or wirelessly via the network N1. Furthermore, the information processing apparatus 200 and the copyrighted work management apparatus 100 are communicably connected to each other by wire or wirelessly via the dedicated network N2.

[0099] The copyrighted work management device 100 manages copyrighted music information. The copyrighted work management device 100 periodically registers copyrighted music information. The copyrighted work management device 100 extracts multiple types of feature quantities from the registered copyrighted music information and transmits the extracted feature quantities to the information processing device 200.

[0100] The user terminal 300 transmits music information composed by a producer to the information processing device 200, and receives provision of music information composed by the information processing device 200 when the automatic composition function is activated.

[0101] The information processing device 200 generates style information as learning data from copyrighted music information or music information created by a producer, and performs machine learning to generate a composition model. The information processing device 200 provides the user terminal 300 with music information automatically composed using the generated model.

[0102] [1-3. Configuration of Copyrighted Works Management Device According to Embodiment]

[0103] Next, we will refer to Fig.10 Described in Fig. 9 The configuration of the copyrighted work management apparatus 100 is shown in FIG. Fig.10 1 is a diagram showing a configuration example of a copyrighted work management device 100 according to this embodiment. Fig.10 As shown, the copyright protected work management apparatus 100 includes a communication unit 110 , a storage unit 120 , and a control unit 130 .

[0104] The communication unit 110 is implemented by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to the dedicated network N2 by wire or wirelessly, and transmits and receives information to and from the information processing device 200 via the dedicated network N2.

[0105] The storage unit 120 is implemented by, for example, a semiconductor storage element such as a random access memory (RAM) or a flash memory or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various data used for information processing. The storage unit 120 includes a copyrighted music information storage unit 121, a music storage unit 122, and a feature information storage unit 123.

[0106] The copyrighted music information storage unit 121 stores information on copyrighted music that is a copyrighted work produced in the past. Fig.11 2 is a diagram showing an example of the copyright-protected music information storage unit 121 according to the embodiment.

[0107] like Fig.11 As shown, the copyrighted music ID, the copyrighted music meta information and the MP3 file name are stored in the copyrighted music information storage unit 121. For example, the copyrighted music ID is identification information for uniquely specifying copyrighted music such as the Beatles, Michael Jackson, etc. The copyrighted music meta information is, for example, additional information about the copyrighted music, such as the title, artist name, album name, release date, genre, mood and lyrics of the copyrighted music. The MP3 file name is the file name of the MP3 file for specifying the copyrighted music.

[0108] The music storage unit 122 stores music information of music protected by copyright. Fig.12 1 is a diagram showing an example of the music storage unit 122 according to the embodiment. Fig.12 As shown, the music storage unit 122 stores an MP3 file of each copyrighted music. The MP3 file can be specified by the MP3 file name stored in the copyrighted music information storage unit 121.

[0109] The characteristic information storage unit 123 stores various types of characteristic quantities of copyrighted music. Fig.13 is a diagram showing an example of the feature information storage unit 123 according to an embodiment. Fig.13 As shown, copyrighted music feature quantities are stored in association with copyrighted music IDs in the feature information storage unit 123. The copyrighted music feature information includes overall information indicating features of the entire copyrighted music and time series information indicating that the features change in time series.

[0110] Fig.14 is a diagram showing an example of overall information according to an embodiment. Fig.14 As shown, the overall information includes, for example, tempo information, time signature information, and music type information such as genre and mood. The music type information is, for example, information of an automatic channel such as twelve-tone analysis technology.

[0111] Fig.15 2 is a diagram showing an example of time series information constituting copyrighted music feature information according to an embodiment. Fig.15 As shown, the time series information includes, for example, beat information, chord progression information, music structure information, and melody information. The beat information includes, for example, time position information of each beat and time position information of each measure, such as a metronome. The chord progression information is information of the time series progression of accompaniment chords according to the music stream, for example, information indicating a chord progression such as "CF-Am-G7". The music structure information is information such as "Section 1 - Section 2 - Harmony - Section 1 - Harmony" as the structure of the overall music. For example, the melody information includes information of a melody line inferred from the audio information of the music.

[0112] Return to reference Fig.10, will continue to describe. The control unit 130 is implemented by, for example, a central processing unit (CPU), a microprocessing unit (MPU), etc. that executes a program stored inside the user terminal 300 using a random access memory (RAM) or the like as a work area. In addition, the control unit 130 is a controller and can be implemented by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0113] The control unit 130 includes a management unit 131 , an extraction unit 132 , and a transmission unit 133 , and implements or performs the functions or operations of information processing described below.

[0114] The management unit 131 manages various information related to the copyrighted work management device 100. For example, the management unit 131 stores various information in the storage unit 120 and appropriately updates the stored information. Specifically, the management unit 131 stores new copyrighted music in the copyrighted music information storage unit 121 and updates information about the new copyrighted music.

[0115] The extraction unit 132 extracts a plurality of types of feature quantities from the copyrighted music information. The extraction unit 132 acquires the copyrighted music information registered regularly at a preset timing, and extracts chord progression information, beat information, melody information, and drum information as feature quantities from the acquired copyrighted music information. The extraction unit 132 extracts the music feature quantity of the MP3 file of each copyrighted music with respect to each newly registered copyrighted music, and obtains feature information. The extraction unit 132 extracts various feature quantities from the music information using, for example, twelve-tone analysis technology, and registers the extracted feature quantities as copyrighted music feature information in the feature information storage unit 123.

[0116] The extraction unit 132 receives a copyrighted music extraction instruction from an administrator terminal (not shown) used by a system administrator via the information processing device 200 to perform feature amount extraction processing on the copyrighted music information. Alternatively, the extraction unit 132 performs feature amount extraction processing with respect to the copyrighted music information by receiving the copyrighted music extraction instruction from the information processing device 200.

[0117] The transmission unit 133 transmits the plurality of types of feature quantities of the copyrighted music information extracted by the extraction unit 132 together with, for example, information about the copyrighted music information (such as music structure information, copyrighted music meta information, etc.) to the information processing device 200.

[0118] [1-4. Configuration of information processing device according to embodiment]

[0119] Next, we will refer to Fig.16 Described in Fig. 9 The configuration of the information processing device 200 is shown in FIG. Fig.16 2 is a diagram showing a configuration example of an information processing device 200 according to an embodiment. Fig.16 As shown, the information processing device 200 includes a communication unit 210, a storage unit 220, and a control unit 230. It should be noted that the information processing device 200 may include an input unit (e.g., a keyboard, a mouse, etc.) for receiving various operations from an administrator or the like who manages the information processing device 200, and a display unit (e.g., a liquid crystal display, etc.) for displaying various information.

[0120] The communication unit 210 is implemented by, for example, a NIC, etc. The communication unit 210 is connected to the network N1 and the dedicated network N2 by wire or wirelessly, and transmits and receives information to and from the user terminal 300, the copyrighted work management device 100, etc. via the network N1 or the dedicated network N2.

[0121] The storage unit 220 is implemented by, for example, a semiconductor storage element such as a RAM or a flash memory or a storage device such as a hard disk or an optical disk. The storage unit 220 stores various data used for information processing.

[0122] like Fig.16 As shown, the storage unit 220 includes a user information storage unit 221 , a style information storage unit 222 , an ownership information storage unit 223 , a production information storage unit 224 , and an operation history information storage unit 225 .

[0123] The user information storage unit 221 stores various information about users (user information). Fig.17 is a diagram illustrating an example of the user information storage unit 221 according to an embodiment.

[0124] The user information storage unit 221 stores user information including a user ID, user meta information, and authority information. The user information storage unit 221 stores the user meta information or authority information corresponding to each user ID in association with each user ID.

[0125] The user ID represents identification information for uniquely specifying a user. For example, the user ID represents identification information for uniquely specifying a user such as a producer, a general user, a system administrator, etc. User meta information is, for example, additional information of the user, such as the name and address of the user. As authority information, for example, values ​​for identifying authority, such as system administrator authority information, producer authority information, and general user authority information, are stored. Note that the user information storage unit 221 is not limited to the above content, and various types of information can be stored according to the purpose. Various information about the user can be stored in the user meta information. For example, in the case where the user is a natural person, demographic attribute information, psychological attribute information (such as the user's gender and age), etc. can be stored in the user meta information.

[0126] The style information storage unit 222 stores information on composition models. Fig.18 is a diagram showing an example of the style information storage unit 222 according to the embodiment.

[0127] The style information storage unit 222 stores the learning model information, including the model information ID, the creator ID, the model information meta information, the style information 700, the copyrighted work ID, and the sharing availability information. The style information storage unit 222 stores the creator ID, the model information meta information, the style information, the copyrighted work ID, and the sharing availability information corresponding to each model information ID in association with each model information ID.

[0128] The model information ID represents identification information for uniquely specifying the composition model information. The creator ID represents identification information for uniquely specifying the creator of the corresponding composition model information. For example, the creator ID represents identification information for uniquely specifying a user such as a system administrator, a producer, a general user, etc.

[0129] The model information meta-information is, for example, information indicating the characteristics of the copyrighted work to be learned. The learning model information meta-information is information such as the tempo of the music, the style, the atmosphere such as light and dark, the structure of the music such as the first bar, the second bar and the harmony, the chord progression, the scale and the church mode.

[0130] The style information 700 is learning data of a composition model generated by the generation unit 233 (described below) included in the information processing device 200. Figure 2 As shown, the style information is information in which a plurality of types of feature quantities such as chord progression, melody, and bass progression extracted from music information are associated with predetermined identification information.

[0131] The sharing availability information indicates, for example, whether the corresponding learning model can be shared. As the sharing availability information, for example, a value for specifying and identifying whether the corresponding learning model can be shared is stored.

[0132] Note that the style information storage unit 222 is not limited to the above and may store various types of information according to the purpose. For example, the composition model information meta information may store various types of additional information related to the composition model, such as information related to the date and time when the composition model was created.

[0133] The ownership information storage unit 223 stores various information on style information selected by a producer of composed music when composing music. Fig.19 : is a diagram showing an example of the ownership information storage unit 223 according to the embodiment. The ownership information storage unit 223 stores the user ID of the producer who creates music and the style information ID selected by the producer in association with each other.

[0134] The production information storage unit 224 stores various information on produced music. Fig. 20 2 is a diagram showing an example of the production information storage unit 224 according to an embodiment. Fig. 20 As shown, the production information storage unit 224 stores the user ID of the producer who produces the music and the ID of the music score produced by the producer in association with each other.

[0135] The operation history information storage unit 225 stores the operation history information of the producer on the user terminal 300. The operation history information storage unit 225 stores the operation history corresponding to each user ID in association with each user ID. The operation history information represents the operation history of the producer. For example, the operation history information may include various information about the producer's operation, such as the content of the operation performed by the producer, the date and time when the operation was performed, etc. Examples of the operation include the selection of style information presented from the information processing device 200, the selection of a composition execution instruction button, and the reproduction and editing of music information received from the information processing device 200.

[0136] Return to reference Fig.16 , will be described further. The control unit 230 is implemented by, for example, a CPU, an MPU, etc. that executes a program (for example, an information processing program according to the present embodiment) stored in the information processing device 200 using, for example, a RAM, etc. as a work area. In addition, the control unit 130 is a controller and can be implemented by, for example, an integrated circuit (such as an ASIC or an FPGA).

[0137] The control unit 230 includes an acquisition unit 231, an extraction unit 232, a generation unit 233, a reception unit 234, a selection unit 235, a transmission unit 236, a composition unit 237, and an update unit 238, and implements or executes the functions or operations of information processing described below.

[0138] The acquisition unit 231 acquires music information. The acquisition unit 231 communicates with the user terminal 300 via the network N1 to acquire music information composed by a producer. The music information is composed by the producer using an application (i.e., an automatic composition function) related to music composition installed in the user terminal 300, and includes feature quantities related to music, such as chord progression, melody, bass progression, and drum progression. In addition, the acquisition unit 231 communicates with the copyrighted work management device 100 via the dedicated network N2, and receives a plurality of types of feature quantities of the copyrighted music information extracted by the extraction unit 132 of the copyrighted work management device 100 and information about the copyrighted music information. That is, the acquisition unit 231 receives copyrighted music feature information for each newly registered copyrighted music from the copyrighted work management device 100 via the dedicated network N2.

[0139] The extraction unit 232 extracts various types of feature quantities from the music information. The extraction unit 132 extracts chord progression information, beat information, melody information, and drum information as feature quantities from the music information composed by the producer. The extraction unit 232 extracts various feature quantities from the music information using, for example, twelve-tone analysis technology.

[0140] The generation unit 233 generates style information in which a plurality of types of feature amounts extracted by the extraction unit 232 are associated with a style information ID as learning data in the composition process. The generation unit 233 registers the style information ID of the style information 700 regarding the music information composed by the producer and the user ID of the producer in association with each other in the production information storage unit 224. The generation unit 233 can associate the copyrighted music ID with the style ID of the style information 700 regarding the music information registered in the copyrighted work management device 100.

[0141] The receiving unit 234 receives various information transmitted from the user terminal 300. For example, the receiving unit 234 receives information about a producer who uses the automatic composition function in the user terminal 300 and information about style information selected by the producer. In addition, the receiving unit 234 may also receive registration of music to be linked to the style information 700, editing of style information, and the like.

[0142] When the automatic composition function is activated in the user terminal 300 , the selection unit 235 selects all or part of the style information.

[0143] The transmission unit 236 transmits the presentation information of the style information selected by the selection unit 235 to the user terminal 300. Thus, a list of chord progression or lyrics information of each style information is displayed as a candidate in the style palette selection drop-down 372a of the user terminal 300. Then, when instruction information giving an instruction on selecting any presented style information is received from the user terminal 300, the selection unit 235 selects the selected style information from the style information storage unit 222.

[0144] The composition unit 237 composes music information using machine learning based on the style information selected by the selection unit 235, and transmits the composed music information to the user terminal 300. The composition unit 237 can use various existing music generation algorithms to compose music. For example, the composition unit 237 can use a music generation algorithm using a Markov chain, or can use a music generation algorithm using deep learning. In addition, the composition unit 237 can generate multiple pieces of music information for the instruction information sent from the user terminal 300. Thus, the producer can receive multiple suggestions from the composition unit 237, and thus can use more various information to perform composition work.

[0145] In the case where the performance information based on the music information composed by the composing unit 237 is received from the user terminal 300, the updating unit 238 adds the performance information to the selected style information to update the selected style information.

[0146] [1-5. Configuration of User Terminal According to Embodiment]

[0147] Next, we will refer to Fig.21 Described in Fig. 9 The configuration of the user terminal 300 is shown in FIG. Fig.21 is a diagram showing a configuration example of a user terminal 300 according to an embodiment. Fig.21 As shown, the user terminal 300 includes a communication unit 310 , an input unit 320 , an output unit 330 , a storage unit 340 , a control unit 350 , and a display unit 360 .

[0148] The communication unit 310 is implemented by, for example, a NIC, a communication circuit, etc. The communication unit 310 is connected to the network N1 by wire or wirelessly, and transmits information to another device, etc. (such as, the information processing device 200, another terminal device, etc.) via the network N1 and receives information from another device, etc. (such as, the information processing device 200, another terminal device, etc.).

[0149] Various operations are input from the user to the input unit 320. The input unit 320 includes a keyboard and a mouse connected to the user terminal 3000. The input unit 320 receives input from the user. The input unit 320 receives user input using a keyboard or a mouse. The input unit 320 may have a function of detecting voice. In this case, the input unit 320 may include a microphone for detecting voice.

[0150] Various information can be input to the input unit 320 via the display unit 360. In this case, the input unit 320 may have a touch panel capable of realizing functions equivalent to those of a keyboard and a mouse. In this case, the input unit 12 receives various operations from the user via the display screen through the functions of the touch panel realized by various sensors. It should be noted that as a method for detecting user operations through the input unit 320, a capacitive method is mainly used in a tablet terminal, but any method can be used as long as the user's operation can be detected and the function of the touch panel can be realized, such as a resistive film method, a surface acoustic wave method, an infrared method, and an electromagnetic induction method, which are other detection methods. In addition, the user terminal 300 may include an input unit that also receives operations through buttons, etc.

[0151] The output unit 330 outputs various information. The output unit 330 includes a speaker that outputs sound.

[0152] The storage unit 340 is implemented by, for example, a semiconductor storage element such as a RAM or a flash memory or a storage device such as a hard disk or an optical disk. The storage unit 340 stores various information for display information.

[0153] The control unit 350 is implemented, for example, by a CPU, an MPU, etc. that executes a program stored in the user terminal 300 using a RAM, etc. as a work area. In addition, the control unit 350 is a controller and can be implemented by an integrated circuit such as an ASIC or an FPGA. The control unit 350 includes a display control unit 351, a registration unit 352, a transmission / reception unit 353, a selection unit 354, a reproduction unit 355, and a performance reception unit 356.

[0154] The display control unit 351 controls various displays with respect to the display unit 360. The display control unit 351 controls the display of the display unit 360. The display control unit 351 controls the display of the display unit 360 based on information received from the information processing device 200. The display control unit 351 controls the display of the display unit 360 based on information generated by processing of each component of the control unit 350. The display control unit 351 may control the display of the display unit 360 using an application that displays an image.

[0155] The display control unit 351 causes the display unit 360 to display a window 370 (see FIG. 10 ) by using the automatic composition function through DAW and AI. Figures 3 to 7 ) etc. In addition, when receiving presentation information of style information from the information processing device 200, the display control unit 351 selects the style palette drop-down 372a (see Figure 4 ) displays the chord progression and lyrics of the presented style information. In addition, when receiving music information and lyrics information generated by the information processing device 200, the display control unit 351 displays the piano roll 374a (see Figure 7 ) displays the melody and the lyrics corresponding to the melody.

[0156] The registration unit 352 receives registration of various information. For example, the registration unit 352 receives registration of a drum pattern, a chord progression, and a melody set by the user when activating a DAW or the like. For example, by displaying images IM11, IM21, IM31, and IM41 ( Fig.26 and Fig. 27 ) applications to register drum patterns, chord progressions, and melodies.

[0157] The transmission / reception unit 353 communicates with the information processing device 200, and transmits and receives various information. The transmission / reception unit 353 transmits the music information including the drum pattern, chord progression, and melody received by the registration unit 352 to the information processing device 200. In addition, when the automatic composition function is activated, the transmission / reception unit 353 receives the presentation information of the style information transmitted from the information processing device 200. The transmission / reception unit 353 transmits the instruction information giving the instruction about selecting the style information to the information processing device 200. Then, the transmission / reception unit 353 receives the music information and lyrics information generated by the information processing device 200. In addition, the transmission / reception unit 353 transmits the performance information about the performance received by the user terminal 300 to the information processing device 200.

[0158] The selection unit 354 selects any style information presented from the information processing device 200. For example, the user selects the style palette selection drop-down 372a (see Figure 4 ) in the chord progression displayed in the chord progression. Thus, the selection unit 354 sends instruction information for selecting pattern information corresponding to the selected chord progression from the sending / receiving unit 353 to the information processing device 200.

[0159] The reproduction unit 355 reproduces the music information generated by the information processing device 200. Specifically, the reproduction unit 255 sets arbitrary instrument information for each of the melody, chord, and bass included in the music data, and reproduces each piece of data. Note that the reproduction unit 255 can also reproduce the combination of the melody, chord, and bass.

[0160] The performance receiving unit 356 receives the performance of the producer when the producer performs performance and composes using the automatic composition function. For example, the performance receiving unit 356 receives performance information to be performed according to the reproduction of the music information generated by the information processing device 200 by the automatic composition function.

[0161] The display unit 360 displays various information. The display unit 360 is implemented by, for example, a liquid crystal display, an organic electroluminescent (EL) display, etc. The display unit 360 displays various information according to the control of the display control unit 351. The display unit 360 can also display information such as an image provided from the information processing device 200.

[0162] [1-6. Procedure of information processing according to embodiment]

[0163] [1-6-1. Processing of Generating Style Information of Copyrighted Music Information]

[0164] Next, refer to Fig. 22 , describing the procedures of various information processing according to the embodiment. First, as one piece of information processing according to the embodiment, automatic generation processing of style information about copyrighted music information will be described. Fig. 22 is a sequence diagram showing the procedure of information processing according to the embodiment.

[0165] like Fig. 22 As shown in , when copyrighted music information is newly registered (step S101), the copyrighted work management device 100 extracts a plurality of types of feature quantities from each newly registered copyrighted music information (step S102). At this time, for example, the copyrighted work management device 100 extracts the feature quantities using a twelve-tone analysis technique. Then, the copyrighted work management device 100 transmits the copyrighted music feature information including the plurality of types of feature quantities extracted from the copyrighted music information and the information on the copyrighted music information to the information processing device 200 via the dedicated network N2 (step S103).

[0166] The information processing device 200 automatically creates style information 700 based on the received copyrighted music feature information (step S104). For example, the information processing device 200 may generate score information 740 based on the beat information, chord progression information, and melody information of the copyrighted music feature information. For example, the information processing device 200 may generate lyrics information 750 from lyrics information of copyrighted music meta information.

[0167] For example, the information processing device 200 may bundle similar chord progressions of chord information from multiple pieces of score information 740 and lyrics information 750 to generate the style palette information 730. A similar chord progression is, for example, the same chord progression. Alternatively, a similar chord progression may be such that each chord is classified into a tonic (Tonic), a subdominant (S), and a dominant (D), and the sequence of T, S, and D is the same. Note that in the case of C major and A minor, T is C / Em / Am, S is F and Dm, and D is G and Dm7-5. Then, since the chord progressions CDGC and Em-Dm-Bm7-5-Am are both TSDT, they can be considered to be the same chord progression. In addition, with respect to similar chord progressions, similar chord progressions may be classified, for example, based on machine learning or deep learning rather than using music theory.

[0168] In addition, the information processing device 200 may independently register the automatically generated style palette information 730 in the style palette sequence information 720. The information processing device 200 may generate and register the style palette sequence information 720 in which a plurality of style palette information 730 are arranged. When arranging a plurality of style palette information 730, the information processing device 200 may arrange the style palette information 730 with reference to the music structure information.

[0169] Subsequently, the information processing apparatus 200 registers the generated style information in association with the identification information of the copyrighted music information (step S105 ), and stores the style information in the style information storage unit 222 .

[0170] [1-6-2. Processing of generating style information of music information created by a producer]

[0171] Style information can also be generated relative to music information created by the producer. Fig.23 A process of generating style information about music information composed by a producer is described. Fig.23 is a sequence diagram showing the procedure of information processing according to the embodiment.

[0172] like Fig.23As shown, when the producer operates the user terminal 300, the composition function such as DAW is activated (step S111), and the creation of style information is selected (step S112), and the composition start information indicating the start of composition is sent from the user terminal 300 (step S113). Then, when receiving the composition start information, the information processing device 200 sends information required for composition (step S114).

[0173] Subsequently, the user terminal 300 registers the drum pattern (e.g., beats, measure numbers, and beat positions of Hight Hat, Bass Drumus, Snare) by the producer's operation according to the UI instruction (step S115). When step S115 ends, the user terminal 300 registers the chord progression according to the producer's operation (step S116).

[0174] After step S116 ends, the composition function application automatically executes the drum process and the chord process, so the producer inputs the melody to the user terminal 300 multiple times. Through this input operation, the user terminal 300 registers the input melody (step S117). The user can use the composition function to input lyrics information in addition. The user terminal 300 continues the melody registration until the user's melody input ends. When the user's melody registration ends (step S118), the user terminal 300 sends the producer's music information to the information processing device 200 (step S119). The music information includes feature quantities such as drum patterns, chord processes, and melody.

[0175] The information processing device 200 extracts each feature amount included in the music information to generate score information and lyrics information, and generates style palette information from the score information and the lyrics information (step S120 ).

[0176] Then, the information processing device 200 obtains each piece of style palette sequence information of the music information, and generates style information 700 by associating the style information ID with the score information, lyrics information, style palette information, and style palette sequence information (step S121 ).

[0177] The information processing device 200 registers the style information 700 in the style information storage unit 222 (step S122). At the same time, the information processing device 200 registers data in which the style information ID and the user ID of the producer are associated with each other in the ownership information storage unit 223. Then, the information processing device 200 registers data in which the score ID and the user ID of the producer are associated with each other in the production information storage unit 224, and ends the style information generation process. Note that the style palette sequence information can be generated by the producer arranging a plurality of pieces of style palette information using the composition function.

[0178] By executing Fig. 22 and Fig.23 By the process of , a plurality of pieces of style information 700 are accumulated in the information processing device 200. A user such as a producer can select a desired piece of style information from the plurality of pieces of style information and use automatic composition.

[0179] [1-6-3. Processing of updating style information]

[0180] Style information can also be updated by the author. Therefore, reference Fig.24 A process of generating style information about music information composed by a producer is described. Fig.24 is a sequence diagram showing the procedure of information processing according to the embodiment.

[0181] Upon receiving composition start information (step S132) according to activation of the automatic composition function on the user terminal 300 by the producer (step S131), the information processing device 200 selects style information (step S133) and transmits presentation information of the style information to the user terminal 300 (step S134). For example, the information processing device 200 selects all style information, style information used by the producer more than a predetermined number of times, or style information used by all users more than a predetermined number of times from the style information storage unit 222, and transmits presentation information of the selected style information.

[0182] Then, the user terminal 300 displays a list of style information based on the presentation information (step S135). For example, the user terminal 300 displays a list of chord progressions of style information as candidates. Then, in the user terminal 300, when the producer selects style information (step S136), selection information indicating the selected style information is sent to the information processing device 200 (step S137).

[0183] The information processing device 200 extracts the selected style information, performs machine learning using the extracted style information as learning data and performs composition processing (step S138), and provides the music information to the user terminal 300 (step S139). It is to be noted that the information processing device 200 extracts the feature amount of the composed music information through the extraction unit 232, stores the new score information including the feature amount in the storage unit 220, and registers the new score information in the ownership information storage unit 223.

[0184] When reproducing the provided music (step S140), the user terminal 300 receives the performance processing (step S141). In the case where the producer uses a MIDI keyboard or the like to perform, the performance information is MIDI information. When receiving the transmission of the performance information (step S142), the information processing device 200 extracts the feature quantity from the performance information (step S143).

[0185] The information processing device 200 updates the style information by adding the feature quantity extracted from the performance information as the score information to the style information selected by the producer (step S144). Thus, since the producer's actual performance is added to the style information selected by the producer, the music information automatically composed using the style information is close to the producer's style. That is, the style information can be made close to the style of the producer who performed the music. The processing of steps S140 to S142 is repeated until the producer finishes the performance. Then, the score information generated in the repeated processing of steps S140 to S142 is stored in the storage unit 220 until the music is completed.

[0186] Thus, using the information processing system 1, since the producer's actual performance is added to the style information selected by the producer, the music information automatically composed using the style information is close to the producer's style. That is, using the information processing system 1, the style information can be made close to the style of the producer who performed the music.

[0187] Then, when the producer's performance ends (step S145: Yes), the producer operates the user terminal 300, for example, to perform setting processing (step S146) and mixing and mastering processing (step S147).

[0188] [1-7. Conceptual diagram of configuration of information processing system]

[0189] Here, we will refer to Fig.25 Conceptually describe each function, hardware configuration, and data in the information processing system. Fig.25 1 is a diagram showing an example of a conceptual diagram of a configuration of an information processing system. Specifically, Fig.25 : is a schematic diagram showing an overview of functions of a system as an example to which the information processing system 1 is applied.

[0190] [1-7-1. About the overall structure]

[0191] Fig.25 The copyrighted music management server device shown in corresponds to the copyrighted work management device 100 in the information processing system 1. Fig.25 The server device shown in corresponds to the information processing device 200 in the information processing system 1. In addition, Fig.25 The system manager application unit shown in corresponds to an application installed in a terminal used by the system manager. Fig.25 The producer application unit shown in corresponds to an application installed in the user terminal 300 in the information processing system 1. In addition, Fig.25 The general user application unit shown in FIG. 1 corresponds to an application installed in a terminal used by a general user. Fig.25 In the example of FIG. 1 , one system manager application unit, one music maker application unit, and one general user application unit are shown, but a plurality of these units may be included depending on the number of corresponding terminals.

[0192] Fig.25 The control unit of the copyrighted music management server device shown in FIG. 1 corresponds to the control unit 130 of the copyrighted work management apparatus 100. For example, the copyrighted music management unit of the copyrighted music management server device corresponds to the management unit 131 of the copyrighted work management apparatus 100. The copyrighted music feature information analysis function of the copyrighted music management server device corresponds to the extraction unit 132 of the copyrighted work management apparatus 100. The copyrighted music management server database unit of the copyrighted music management server device corresponds to the storage unit 120 of the copyrighted work management apparatus 100.

[0193] Fig.25 The learning processing unit and the control unit of the server device shown in FIG. 2 correspond to the control unit 230 of the information processing device 200. For example, the learning processing unit of the server device corresponds to the composition unit 237 of the information processing device 200. The server database unit of the server device corresponds to the storage unit 220 of the information processing device 200.

[0194] Fig.25 The display operation unit and control unit of the music maker application unit shown in FIG. 3 correspond to the control unit 350 of the user terminal 300. For example, the display operation unit of the music maker application unit corresponds to the display control unit 351 of the user terminal 300. The display operation unit and control unit of the system manager application unit and the general user application unit correspond to the control unit of the terminal device used by each user.

[0195] like Fig.25 As shown, the server device is connected to a system manager application unit, a music producer application unit, and a general user application unit via a network N1 such as the Internet. In addition, the server device is connected to a copyrighted music management server device via a dedicated network N2.

[0196] [1-7-2. Regarding copyright-protected music management server devices]

[0197] First, the configuration related to the copyrighted music management server device will be described.

[0198] The copyrighted music management server device includes a control unit and a copyrighted music management server database unit. The control unit of the copyrighted music management server device includes a copyrighted music management function and a copyrighted music characteristic information analysis function.

[0199] [1-7-3. About server equipment]

[0200] First, the configuration related to the server device will be described.

[0201] The server device includes a control unit, a learning processing unit and a server database unit. The control unit of the server device has a generated music information management function, a style information management function, a user operation history information management function and a copyrighted music analysis function. The learning processing unit of the server device has a machine learning processing function and a deep learning processing function.

[0202] [1-7-4. About the Music Maker Application Unit]

[0203] Next, the configuration related to the music maker application unit will be described.

[0204] The music maker application unit includes a display operation unit and a control unit. The display operation unit of the music maker application unit has a function of displaying the produced music information and a function of displaying and editing the style information. The music maker application unit has a function of sharing the style information and a function of sending the user operation history information.

[0205] The music maker application unit is, for example, music editing software (DAW, etc.), and is capable of displaying, for example, music information through a produced music information display function. For example, when the DAW has an AI-assisted music production function, new music information can be produced using a learning model information display editing function. The system manager application unit and the general user application unit have the same configuration, and the user has different permissions to the system.

[0206] [1-8.UI (User Interface)]

[0207] Here, we will refer to Fig.26 and Fig. 27 The description includes details of the automatic composition function through information display of an application (music creation application). Fig.26 and Fig. 27 is a diagram illustrating an example of a user interface according to an embodiment.

[0208] Fig.26 An example of a user interface when a music composition application is displayed on the screen of the user terminal 300 is shown.

[0209] exist Fig.26In the example shown in FIG. 1 , the user interface IF11 displays music data received by the music creation application. Note that, although the details will be described below, the music data in the music creation application includes three types of data: melody, chord, and bass. Fig.26 The user interface IF11 shown in shows data related to the melody among the three types of data.

[0210] The setting information ST11 displays information about a style palette as an example of setting information in the automatic composition function. The style palette is designation information for designating style information that becomes learning data for machine learning.

[0211] The setting information ST12 displays information about harmony, which is an example of setting information in the automatic composition function. The information about harmony is, for example, information for determining the probability that the component sounds included in the chord appear in the melody in the music data composed by the information processing device 200. For example, when the user sets the information about harmony to "strict", the probability that the component sounds included in the chord appear in the melody in the automatically composed music data increases. On the other hand, when the user sets the information about harmony to "loose", the probability that the component sounds included in the chord appear in the melody in the automatically composed music data decreases. Fig.26 The example of instructs the user to apply information about harmony to "strict".

[0212] The setting information ST13 displays note duration information, which is an example of setting information in the automatic composition function. For example, the note duration information is information for determining the note duration in the music data composed by the information processing device 200. For example, when the user sets the note duration information to "long", the probability that a note to be sounded with a relatively long length (e.g., a whole note, a half note, etc.) appears in the music data for automatic composition increases. On the other hand, when the user sets the note duration information to "short", the probability that a note to be sounded with a relatively short length (e.g., an eighth note, a sixteenth note, etc.) appears in the music data for automatic composition increases.

[0213] The setting information ST14 displays information for determining the type and amount of material music other than the material music included in the designated information (the style palette designated by the user), which is an example of the setting information in the automatic composition function. Such information is, for example, information for determining whether to strictly perform learning based on the music included in the style palette designated by the user in the music data composed by the information processing device 200. For example, when the user sets such information to "never", music other than the music included in the style palette is less likely to be used in the learning of the automatic composition. On the other hand, when the user sets such information to "only", music other than the music included in the style palette is more likely to be used in the learning of the automatic composition.

[0214] The music data MDT1 shows specific music data transmitted from the information processing device 200. Fig.26 In the example of , the music data MDT1 includes information indicating a chord progression such as Cm, information indicating pitches or note values ​​in a measure, transitions of pitches of notes (in other words, melody), and the like. Fig.26 As shown, for example, the music data MDT1 may include four types of different contents. That is, the information processing device 200 may send multiple pieces of music data instead of only one type of automatically composed music data. Thus, the user can select his favorite music data from a plurality of generated music data candidates, or compose a favorite music by combining a plurality of music data.

[0215] It should be noted that Fig.26 The user interface IF11 shown in FIG. 1 displays data related to the melody among three types of data: melody, chords, and bass included in the music data, and other data is displayed on another user interface. Fig. 27 Describe this.

[0216] like Fig. 27 As shown, in addition to the user interface IF11 displaying data related to the melody, the user terminal 300 may also display a user interface IF12 displaying data related to the chord and a user interface IF13 displaying data related to the bass sound on the screen. Fig. 27 Although not shown in the figure, note information different from the music data MDT1 in the user interface IF11 is displayed on the user interface IF12 or the user interface IF13. Specifically, note information related to the chord corresponding to the melody of the music data (for example, the constituent sounds of the Cm chord, etc.) is displayed on the user interface IF12. In addition, note information related to the bass of the melody or chord corresponding to the music data (for example, in the case of the Cm chord, the "C" sound, etc.) is displayed on the user interface IF13.

[0217] The user can select information to be copied from the displayed user interface IF11 , user interface IF12 , and user interface IF13 , and perform work such as editing a part of the bass.

[0218] [2. Effects of this embodiment]

[0219] As described above, the information processing device according to the present embodiment (the information processing device 200 in the embodiment) includes an acquisition unit (the acquisition unit 231 in the embodiment), an extraction unit (the extraction unit 232 in the embodiment), and a generation unit (the generation unit 233 in the embodiment). The acquisition unit acquires music information. The extraction unit extracts a plurality of types of feature quantities from the music information acquired by the acquisition unit. The generation unit generates information in which the plurality of types of feature quantities extracted by the extraction unit are associated with predetermined identification information as music feature information (style information in the embodiment) used as learning data in a composition process using machine learning.

[0220] As described above, the information processing device according to the embodiment can generate style information having multiple types of feature quantities of music information as a learning data set for the composition model. The information processing device according to the embodiment enables the composition model to learn the style information so that music information composed according to the characteristics of the music can be provided to each user including the producer. Therefore, the information processing device according to the embodiment can improve the convenience of music creation for the user.

[0221] In addition, the acquisition unit acquires music information by receiving music information including feature quantities related to music created by a producer using a music creation related application installed in the terminal device from a terminal device (a user terminal 300 in the embodiment). The extraction unit extracts multiple types of feature quantities included in the music information. The generation unit associates the producer's identification information with the music feature information. Thus, the information processing device can generate music feature information about the music information created by the producer.

[0222] In addition, the feature quantity related to the music created by the producer is chord progression information representing the chord progression in a measure having a prescribed length, melody information representing the melody, and a bass signal representing the bass progression. Therefore, regarding the music information created by the producer, since the information processing device can generate the music feature information using the feature quantity related to the music created by the producer, the music feature information can be quickly generated.

[0223] In addition, the feature quantity related to the music composed by the producer is drum progression information indicating a drum progression in a measure having a prescribed length. Therefore, the information processing device can generate music feature information including the drum progression information.

[0224] In addition, the acquisition unit acquires copyrighted music information that is registered regularly at a preset timing. The extraction unit extracts multiple types of feature quantities from the copyrighted music information. The generation unit associates identification information of the copyrighted music information with style information. Therefore, the information processing device 200 can automatically generate music feature information about the copyrighted music information that is registered regularly.

[0225] In addition, the information processing device includes a sending unit (sending unit 236 in this embodiment) that sends presentation information of music feature information according to instruction information received from a terminal device installed with a music creation related application. The information processing device includes a composing unit (composing unit 237 in the embodiment), and when a selection of music feature information is received from the terminal device, the composing unit uses machine learning to compose the music information based on the selected feature information, and sends the composed music information to the terminal device. Thus, the information processing device presents the music feature information corresponding to the instruction information to the terminal device, so that the producer can select the desired music feature information from the music feature information. Then, the information processing device can provide music information composed based on the music feature information desired by the producer.

[0226] In addition, the information processing device further includes an updating unit (updating unit 238 in the embodiment), which, when receiving performance information based on the music information sent by the composition unit from the terminal device, adds the performance information to the selected music feature information and updates the selected music feature information. Thus, the information processing device can make the music feature information closer to the style of the producer who performed the music by adding the producer's performance information to the selected music feature information.

[0227] In addition, the extraction section extracts chord progression information indicating a chord progression in a measure having a specified length, melody information indicating a melody, and bass information indicating a bass progression from the music information as feature quantities. The generation unit generates music score information including chord progression information indicating a chord progression in a measure having a specified length, melody information indicating a melody, and bass information indicating a bass sound progression, and sets the music score information as a component of the music feature information. Thus, the information processing device can generate music feature information including chord progression information, melody information, and bass information. Then, when composing music, the information processing device learns feature quantities such as chord progression information, melody information, and bass information, rather than the music information itself, so that the music information can be effectively provided to the user.

[0228] The extracting unit extracts drum information representing a drum progression in a measure having a specified length from the music information as a feature quantity. The generating unit also adds the drum progression information to the score information. Thus, the information processing device can generate music feature information including chord progression information, melody information, bass information, and drum information.

[0229] The generating unit generates lyrics information representing lyrics in a measure having a prescribed length based on lyrics information added to the music information, and sets the lyrics information as a component of the music feature information. Thus, in the case where the terminal device searches for lyrics, since the information processing device can extract music feature information including lyrics or lyrics similar to lyrics, and present the music feature information to the terminal device, the convenience of the user's music creation can be improved. In addition, the information processing device can automatically generate lyrics.

[0230] The generation unit generates music format information in which identification information of the score information and identification information of the lyrics information for the same measure are registered in association with each other, and sets the music format information as a component of the music feature information. The information processing device can further provide music information desired by the user by learning the music feature information.

[0231] The generating unit adds identification information of the score information having chord progression information similar to the chord progression information of the score information registered in the music format information and registers it in the music format information. Therefore, the information processing device can compose the music information along with the structure of the music.

[0232] The generating unit generates music sequence information indicating the sequence of the music format information and sets the music sequence information as a component of the music feature information. Since the information processing device can also learn the sequence of the music format information, the learning accuracy can be further improved.

[0233] [3. Other embodiments]

[0234] In addition to the above-described embodiment and modifications, the processing according to the above-described embodiment and modifications may be performed in various different forms (modifications).

[0235] [3-1. Other configuration examples]

[0236] Each of the above configurations is exemplary, and the information processing system 1 may be any system configuration as long as the above information processing can be realized. For example, the copyrighted work management device 100 and the information processing device 200 may be integrated.

[0237] [3-2. Others]

[0238] In addition, in each process described in the above embodiments, all or some of the processes described as being automatically performed may be performed manually, or all or some of the processes described as being manually performed may be performed automatically by a known method. In addition, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters indicated in the document and the drawings may be arbitrarily changed. For example, the various information shown in each drawing is not limited to the information shown.

[0239] In addition, each component of each device shown in the drawings is conceptual in function and is not necessarily physically configured as shown in the drawings. That is, the specific forms of distribution and integration of the devices are not limited to those shown, and all or part of them may be functionally or physically distributed and integrated into arbitrary units according to various loads, usage conditions, etc.

[0240] Furthermore, the above-described embodiments and modifications may be appropriately combined within a range not contradicting the processing contents.

[0241] Furthermore, the effects described in this specification are merely examples and not restrictive, and other effects may exist.

[0242] [4. Hardware Configuration]

[0243] For example, by having Fig.28 The computer 1000 of the configuration shown in embodies the information apparatus (eg, the information processing device 200, the copyrighted work management device 100, the user terminal 300, etc.) according to the above-described embodiments and modifications. Fig.28 1 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of an information processing device 200, a copyrighted work management device 100, and a user terminal 300. Hereinafter, the information processing device 200 according to the present embodiment will be described as an example. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0244] The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400 and controls each unit. For example, the CPU 1100 loads the program stored in the ROM 1300 or the HDD 1400 to the RAM 1200 and executes processing corresponding to various programs.

[0245] The ROM 1300 stores a boot program such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is activated, a program depending on the hardware of the computer 1000 , and the like.

[0246] The HDD 1400 is a computer-readable recording medium that non-transiently records a program executed by the CPU 1100 , data used by the program, etc. Specifically, the HDD 1400 is a recording medium that records an information processing program according to the present disclosure as an example of program data 1450 .

[0247] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.

[0248] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard and a mouse via the input / output interface 1600. In addition, the CPU 1100 transmits data to output devices such as a display, a speaker, a printer, etc. via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium), etc. For example, the medium is an optical recording medium such as a digital versatile disk (DVD), a phase change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.

[0249] For example, in the case where the computer 1000 is used as the information processing apparatus 200 according to the embodiment, the CPU 1100 of the computer 1000 executes the information processing program loaded on the RAM 1200 to realize the functions of the control unit 130, etc. In addition, the HDD 1400 stores the information processing program and data according to the present disclosure in the storage unit 120. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, these programs may be acquired from another device via the external network 1550.

[0250] It should be noted that the present technology can also have the following configurations.

[0251] (1) An information processing device comprising:

[0252] An acquisition unit acquires music information;

[0253] an extracting unit that extracts a plurality of types of feature quantities from the music information acquired by the acquiring unit; and

[0254] A generating unit generates information in which a plurality of types of feature amounts extracted by the extracting unit are associated with predetermined identification information as music feature information, the music feature information being used as learning data in a composition process using machine learning.

[0255] (2) The information processing device according to (1), wherein

[0256] The acquisition unit acquires music information by receiving, from the terminal device, music information composed by a producer using a music composition related application installed in the terminal device,

[0257] The extracting unit extracts a plurality of types of feature quantities in the music information, and

[0258] The generating unit associates the identification information of the producer with the music feature information.

[0259] (3) The information processing device according to (2), wherein the music information composed by the producer includes chord progression information indicating a chord progression in a measure having a prescribed length, melody information indicating a melody, and a bass signal indicating a bass progression.

[0260] (4) The information processing apparatus according to (3), wherein the music information composed by the producer includes drum progression information indicating a drum progression in a measure having a prescribed length.

[0261] (5) The information processing device according to (1), wherein

[0262] The acquisition unit acquires copyrighted music information registered regularly at a preset timing,

[0263] The extracting unit extracts a plurality of types of feature quantities from the copyrighted music information, and

[0264] The generating unit associates identification information of copyrighted music information with the music characteristic information.

[0265] (6) The information processing device according to (1), further comprising:

[0266] a sending unit, which sends presentation information of music characteristic information according to instruction information received from a terminal device installed with a music creation related application; and

[0267] The composition unit, when receiving a selection of music feature information from a terminal device, composes the music information using machine learning based on the selected music feature information, and transmits the composed music information to the terminal device.

[0268] (7) The information processing device according to (6), further comprising:

[0269] The updating unit, when receiving the performance information based on the music information transmitted by the composing unit from the terminal device, adds the performance information to the selected music characteristic information and updates the selected music characteristic information.

[0270] (8) The information processing device according to (1), wherein

[0271] The extracting unit extracts, as feature amounts, chord progression information indicating a chord progression, melody information indicating a melody, and a bass signal indicating a bass progression in a measure having a prescribed length from the music information, and

[0272] The generating unit generates musical score information including chord progression information indicating a chord progression, melody information indicating a melody, and bass information indicating a bass progression in a measure of a prescribed length, and sets the musical score information as a component of the music characteristic information.

[0273] (9) The information processing device according to (8), wherein

[0274] The extracting unit extracts drum information indicating the progression of drum sounds in a measure having a prescribed length from the music information as a feature amount, and

[0275] The generation unit also adds drum information to the score information.

[0276] (10) The information processing device according to (8), wherein the generation unit generates lyric information representing lyrics in a measure having a prescribed length from lyric information added to the music information, and sets the lyric information as a component of the music characteristic information.

[0277] (11) An information processing device according to (10), wherein the generation unit generates music format information and sets the music format information as a component of the music feature information, in which identification information of the score information and identification information of the lyrics information for the same measure are registered in association with each other.

[0278] (12) The information processing device according to (11), wherein the generation unit adds identification information of the musical score information having chord progression information similar to chord progression information of the musical score information registered in the music format information and registers it in the music format information.

[0279] (13) The information processing device according to (11), wherein the generation unit generates music sequence information indicating the sequence of the music format information and sets the music sequence information as a component of the music feature information.

[0280] (14) An information processing method executed by a computer, the method comprising:

[0281] Get music information;

[0282] extracting a plurality of types of feature quantities from the acquired music information; and

[0283] Information in which the extracted plurality of types of feature amounts are associated with predetermined identification information is generated as music feature information to be used in a composition process using machine learning.

[0284] (15) An information processing program that enables a computer to:

[0285] Get music information;

[0286] extracting a plurality of types of feature quantities from the acquired music information; and

[0287] Information in which the extracted plurality of types of feature amounts are associated with predetermined identification information is generated as music feature information to be used in a composition process using machine learning.

[0288] Reference Numbers List

[0289] 1 Information Processing System

[0290] 100 Copyrighted Works Management Device

[0291] 110, 210, 310 communication units

[0292] 120, 220, 340 storage units

[0293] 130, 230, 350 control unit

[0294] 121 Copyrighted music information storage unit

[0295] 122 Music storage unit

[0296] 123 Feature information storage unit

[0297] 131 Management Unit

[0298] 132 Extraction Unit

[0299] 133 Transmission Unit

[0300] 200 Information processing equipment

[0301] 221 User information storage unit

[0302] 222 Style information storage unit

[0303] 223 Ownership information storage unit

[0304] 224 Create an information storage unit

[0305] 225 Operation history information storage unit

[0306] 231 Get Unit

[0307] 232 Extraction Unit

[0308] 233 Generate Unit

[0309] 234 Receiving Unit

[0310] 235 Selection Unit

[0311] 236 Transmission Unit

[0312] 237 Composition Unit

[0313] 238 Update Unit

[0314] 300 user terminals

[0315] 320 Input Units

[0316] 330 Output unit

[0317] 351 Display Control Unit

[0318] 352 registered units

[0319] 353 Send / Receive Unit

[0320] 354 Select Unit

[0321] 355 Reproduction Unit

[0322] 356 Performance Receiver Unit

[0323] 360 display unit.

Claims

1. An information processing device, comprising: An acquisition unit acquires music information; an extracting unit for extracting a plurality of types of feature quantities from the music information acquired by the acquiring unit; as well as a generating unit that generates information in which the plurality of types of feature amounts extracted by the extracting unit are associated with predetermined identification information as music feature information, the music feature information being used as learning data in a composition process using machine learning; Upon receiving composition start information according to activation of an automatic composition function on a user terminal by a producer, the information processing device selects style information and transmits presentation information of the style information to the user terminal; the information processing device extracts the style information selected by the producer and uses the extracted style information as learning data to perform machine learning, and performs composition processing to generate music information; the user terminal receives the music information from the information processing device, and performs reproduction to perform performance processing; The information processing device extracts feature amounts from the performance information from the user terminal, and updates the style information by adding the feature amounts extracted from the performance information as score information to the style information selected by the producer.

2. The information processing device according to claim 1, wherein The acquisition unit acquires music information by receiving, from a terminal device, music information composed by a producer using a music composition related application installed in the terminal device, The extracting unit extracts the plurality of types of feature quantities included in the music information, and The generating unit associates the identification information of the producer with the music characteristic information.

3. The information processing device according to claim 2, wherein: The music information composed by the producer includes chord progression information indicating a chord progression in a measure having a prescribed length, melody information indicating a melody, and a bass signal indicating a bass progression.

4. The information processing device according to claim 3, wherein: The music information composed by the producer includes drum progression information indicating a drum progression in a measure having a prescribed length.

5. The information processing device according to claim 1, wherein The acquisition unit acquires copyrighted music information registered regularly at a preset timing, The extracting unit extracts the plurality of types of feature quantities from the copyrighted music information, and The generation unit associates identification information of the copyrighted music information with the music characteristic information.

6. The information processing device according to claim 1, further comprising: A sending unit, which sends presentation information of music characteristic information according to instruction information received from a terminal device installed with a music creation related application; as well as The composition unit, when receiving the selection of the music feature information from the terminal device, composes the music information using machine learning based on the selected music feature information, and transmits the composed music information to the terminal device.

7. The information processing device according to claim 6, further comprising: An updating unit, when receiving performance information based on the music information transmitted by the composing unit from the terminal device, adds the performance information to the selected music characteristic information and updates the selected music characteristic information.

8. The information processing device according to claim 1, wherein The extracting unit extracts, as feature amounts, chord progression information indicating a chord progression, melody information indicating a melody, and a bass signal indicating a bass progression in a measure having a prescribed length from the music information, and The generation unit generates musical score information including chord progression information indicating a chord progression, melody information indicating a melody, and bass information indicating a bass progression in a measure having a prescribed length, and sets the musical score information as a component of the music characteristic information.

9. The information processing device according to claim 8, wherein The extracting unit extracts drum information indicating a drum progression in a measure having a prescribed length from the music information as a feature amount, and The generating unit also adds the drum information to the musical score information.

10. The information processing device according to claim 8, wherein: The generating unit generates lyrics information indicating lyrics in a measure having a prescribed length from lyrics information added to the music information, and sets the lyrics information as a component of the music characteristic information.

11. The information processing device according to claim 10, wherein: The generation unit generates music format information in which identification information of the score information and identification information of the lyrics information for the same measure are registered in association with each other, and sets the music format information as a component of the music characteristic information.

12. The information processing device according to claim 11, wherein: The generation unit adds and registers, into the music format information, identification information of the music score information having chord progression information similar to the chord progression information of the music score information registered in the music format information.

13. The information processing device according to claim 11, wherein: The generating unit generates music sequence information indicating the sequence of the music format information and sets the music sequence information as a component of the music characteristic information.

14. An information processing method executed by a computer, the method comprising: Get music information; Extracting multiple types of feature quantities from the acquired music information; as well as generating information in which the extracted plurality of types of feature amounts are associated with predetermined identification information as music feature information to be used in a composition process using machine learning; When composition start information is received in accordance with activation of an automatic composition function, style information is selected and presentation information of the style information is sent; the selected style information is extracted and machine learning is performed using the extracted style information as learning data, and composition processing is performed to generate music information; the music information is received, and reproduction is performed to perform performance processing to generate performance information; feature quantities are extracted from the performance information, and the style information is updated by adding the feature quantities extracted from the performance information as music score information to the selected style information.

15. A computer-readable recording medium having recorded thereon an information processing program, which, when executed, causes a computer to: Get music information; extracting a plurality of types of feature quantities from the acquired music information; and generating information in which the extracted plurality of types of feature amounts are associated with predetermined identification information as music feature information to be used in a composition process using machine learning; When composition start information is received in accordance with activation of an automatic composition function, style information is selected and presentation information of the style information is sent; the selected style information is extracted and machine learning is performed using the extracted style information as learning data, and composition processing is performed to generate music information; the music information is received, and reproduction is performed to perform performance processing to generate performance information; feature quantities are extracted from the performance information, and the style information is updated by adding the feature quantities extracted from the performance information as music score information to the selected style information.

Images