Information processing device, information processing method, and computer-readable storage medium

Abstract

The information processing device (100) controls a first application (20) and a second application (20), the second application serving as a plug-in for extending the function of the first application. The first application includes a control unit (161) for controlling the operation of the second application in the first application, and the second application includes: a selection unit (166) for selecting setting information for controlling a composition function based on machine learning; and a transmission / reception unit (167) for transmitting the setting information to an external server (200) through a network and receiving music data composed by the external server, the external server performing the composition function based on machine learning.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. In particular, the present disclosure relates to the use of music data composed based on machine learning. Background Art

[0002] With the advancement of artificial intelligence (AI), the use of computers in the field of technology is being promoted.

[0003] For example, a technique is known that performs machine learning using existing music as learning data to generate a learning model for music generation and allows a computer to create new music (e.g., Patent Document 1). In such a technique, it is possible to imitate the characteristics of existing music or generate a more natural melody by using a Markov model.

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: U.S. Patent No. 9,110,817 Summary of the invention

[0007] Problems to be solved by the present invention

[0008] According to conventional technology, since music data proposed (generated) by AI can be used for creation, users can create music based on a wider variety of viewpoints.

[0009] However, using the above-mentioned traditional technology, it is not always possible to improve the convenience of the automatic composition function through AI. For example, currently, many users use digital audio workstations (DAW) to compose, adapt and record. However, when users use the above-mentioned traditional technology in combination with DAW, they work back and forth between different work environments, which may reduce work efficiency. In addition, since the automatic composition function of AI usually has a heavy information processing burden, if the automatic composition function is executed simultaneously with the DAW in the terminal device, it is possible that sufficient functions may not be displayed or the processing on the DAW side may be delayed.

[0010] In this context, the present disclosure proposes an information processing device, an information processing method, and an information processing program that can improve the convenience of an automatic music composition function of AI.

[0011] Solution to the problem

[0012] In order to solve the above problems, according to one form of the present disclosure, an information processing device is an information processing device that controls a first application and a second application, wherein the second application is used as a plug-in to expand the function of the first application, wherein the first application includes a control unit, which controls the operation of the second application in the first application, and the second application includes: a selection unit, which selects setting information for controlling a composition function based on machine learning; and a transmission / reception unit, which transmits the setting information to an external server, which performs a composition function based on machine learning, and receives music data composed by the external server via a network. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a conceptual diagram showing the flow of information processing according to an embodiment;

[0014] Figure 2 is a diagram (1) showing an example of a user interface according to an embodiment;

[0015] Figure 3 is a diagram (2) showing an example of a user interface according to an embodiment;

[0016] Figure 4 is a diagram showing one example of a style palette according to an embodiment;

[0017] Figure 5 is a block diagram showing the flow of information processing according to an embodiment;

[0018] Figure 6 is a diagram illustrating an information processing system according to an embodiment of the present disclosure;

[0019] Figure 7 is a diagram showing a configuration example of a user terminal according to an embodiment;

[0020] Figure 8 is a diagram showing one example of composition setting information according to the embodiment;

[0021] Fig. 9 is a diagram showing one example of composition music information according to the embodiment;

[0022] Fig.10 is a diagram showing one example of history information according to an embodiment;

[0023] Fig.11 is a diagram showing one example of related musical instrument information according to the embodiment;

[0024] Fig.12is a diagram showing a configuration example of a processing server according to an embodiment;

[0025] Fig.13 is a diagram showing one example of user information according to an embodiment;

[0026] Fig.14 is a diagram showing one example of music information according to the embodiment;

[0027] Fig.15 is a diagram showing one example of style palette information according to an embodiment;

[0028] Fig.16 is a diagram showing an example of style palette sequence information according to an embodiment;

[0029] Fig.17 is a diagram showing one example of user composition information according to an embodiment;

[0030] Fig.18 is a diagram showing one example of history information according to an embodiment;

[0031] Fig.19 is a flowchart showing a procedure of information processing according to an embodiment;

[0032] Fig. 20 is a hardware configuration diagram showing one example of a computer that realizes the function of the information processing device. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in each of the following embodiments, the same parts will be denoted by the same reference numerals, thereby omitting repeated descriptions.

[0034] The present disclosure will be described according to the order of items shown below.

[0035] 1. Example

[0036] 1-1. An example of information processing according to the embodiment

[0037] 1-2. Configuration of information processing system according to embodiment

[0038] 1-3. Configuration of Information Processing Device (User Terminal) According to Embodiment

[0039] 1-4. Configuration of the external server (processing server) according to the embodiment

[0040] 1-5. Information processing process according to the embodiment

[0041] 2. Modification

[0042] 2-1. Music data format

[0043] 2-2. Main application

[0044] 2-3. Controlled by DAW

[0045] 2-4. Model of information processing system

[0046] 3. Other embodiments

[0047] 4. Effects of the information processing device according to the present disclosure

[0048] 5. Hardware Configuration

[0049] (1. Example)

[0050] [1-1. One example of information processing according to the embodiment]

[0051] First, refer to Figure 1 An example of information processing according to the present disclosure is described. Figure 1 1 is a conceptual diagram showing the flow of information processing according to the embodiment. The information processing according to the embodiment is performed by a user terminal 10 as an example of an information processing device according to the present disclosure and a processing server 100 as an example of an external server according to the present disclosure. The user terminal 10 and the processing server 100 use Figure 1 The wired or wireless network N (eg, the Internet, etc.) shown communicates with each other. Note that the number of user terminals 10 and processing servers 100 is not limited to Figure 1 Quantity shown.

[0052] Figure 1 The user terminal 10 shown is an example of an information processing device according to the present disclosure. For example, the user terminal 10 is an information processing terminal such as a personal computer (PC), a tablet terminal, or a smart phone. Various program applications (hereinafter referred to as "applications") are included (installed) in the user terminal 10. The user terminal 10 starts and executes various applications and performs various information processing.

[0053] In this embodiment, the user terminal 10 includes an application (so-called DAW) that implements an integrated music production environment. In the following description, the application (DAW) is referred to as a first application or a main application. According to this embodiment, another application for extending the functionality can be included (inserted) into the first application. That is, it is assumed that the first application can use a so-called plug-in, which is another application for extending the functionality. In this case, the first application is used as a main application containing the plug-in.

[0054] In addition, in this embodiment, the user terminal 10 includes an application with an automatic composition function of AI. In the following description, the application (application with an automatic composition function of AI) is referred to as a second application or a plug-in. The second application according to this embodiment is included as a plug-in of the above-mentioned first application. The plug-in can take the form of, for example, Steinberg's Virtual Studio Technology (VST) (registered trademark), audio unit, Avid audio extension (AAX), etc.

[0055] Figure 1 The processing server 100 shown is a server device that performs information processing related to the second application included in the user terminal 10. For example, the processing server 100 is a so-called cloud server, and performs predetermined information processing based on information commanded by the user terminal 10 through the network N. Specifically, the processing server 100 performs a predetermined learning process based on the information transmitted from the user terminal 10, and generates music data based on the data output from the learning model. In other words, the processing server 100 performs an automatic composition function through AI based on the command of the user terminal 10. For example, the processing server 100 provides the user terminal 10 with music data that is automatically composed using a Markov model, etc., as shown in the above-mentioned conventional technical literature, etc.

[0056] As described above, by using the second application as a plug-in, the user terminal 10 can drag and drop the music data provided by the processing server 100 on the second application to the first application, or perform editing on the first application. In addition, although the automatic composition function depends on the processing performance (CPU power, etc.) of the terminal that performs the processing in the conventional technology, as Figure 1 As shown, the user terminal 10 does not perform the learning process itself, but causes the processing server 100 to perform the processing. As a result, the user terminal 10 can perform the automatic composition function with a higher processing load while saving its own resources. Therefore, the user terminal 10 can solve the DAW processing delay (occurrence delay, etc.), which is a conventional problem. Therefore, the user terminal 10 according to the present disclosure can improve the convenience of the automatic composition function of AI. In the following, reference will be made to Figure 1 An outline of information processing according to the present disclosure is described according to procedures.

[0057] like Figure 1 As shown, the user terminal 10 activates the main application (first application) 20 (step S1). In addition, the user terminal 10 activates the plug-in (second application) 22 as an example of a plug-in operating on the main application 20 (step S2).

[0058] The user terminal 10 selects setting information of the music to be automatically composed in the plug-in 22 according to the user's operation. Although the details will be described later, the user terminal 10 selects setting information such as the chord progression of the music to be automatically composed, the subjective image of the music (dark, bright, etc.), and the composition of the music according to the user's operation. Then, the user terminal 10 transmits the selected setting information to the processing server 100 (step S3).

[0059] The processing server 100 performs a predetermined learning process based on the setting information transmitted from the user terminal 10, and performs a composition process based on the learning result (step S4). For this composition process, for example, the process described in the above-mentioned conventional technical document can be used. Then, the processing server 100 generates the composed music data.

[0060] Subsequently, the processing server 100 transmits the generated music data to the user terminal 10 (step S5). The user terminal 10 receives the music data transmitted from the processing server 100 in the plug-in 22. For example, the music data includes information such as chord progression, melody, and bass note progression generated by the processing server 100. Note that the music data may be standard data, such as musical instrument digital interface (MIDI) data, waveform data, or DAW original standard data. The user may edit the received music data on the plug-in 22, or may copy the music data to the main application 20 and use it on the main application 20.

[0061] As described above, the user terminal 10 controls the main application 20 and the plug-in 22, which serves as a plug-in for extending the function of the main application 20. In addition, the main application 20 controls the operation of the plug-in 22 in the main application 20. In addition, the plug-in 22 selects setting information for controlling the composition function based on machine learning, transmits the setting information to the processing server 100 through the network N, and receives music data composed by the processing server 100.

[0062] That is, the user terminal 10 uses the automatic composition function as a plug-in of the DAW. For this reason, the user can receive support for the automatic composition function in the DAW as a normal working environment. In addition, the user can avoid processing delays in the DAW by having the processing server 100 bear the processing load of the automatic composition function. Therefore, the user terminal 10 can improve the convenience of the automatic composition function of AI.

[0063] Next, we will refer to Figures 2 to 4 Describes the details of the automatic composition function of plug-in 22.

[0064] Figure 2 is a diagram (1) showing one example of a user interface according to an embodiment. Figure 2One example of a user interface when the plug-in 22 is displayed on the screen of the user terminal 10 is shown.

[0065] exist Figure 2 In the example shown, the user interface 30 displays the music data received by the plug-in 22. Note that, although the details will be described later, the music data in the plug-in 22 includes three different types of data: melody, chords, and bass notes. Figure 2 The user interface 30 shown displays data associated with a melody.

[0066] The setting information 31 shows information on a style palette, which is an example of setting information in the automatic music composition function. The style palette is designation information for designating material music as learning data for machine learning.

[0067] The setting information 32 displays information about harmony, which is an example of setting information in the automatic composition function. The information about harmony is, for example, information for determining the probability that composition notes included in a chord appear in the melody of the music data composed by the processing server 100. For example, if the user sets the information about harmony to "strict", the composition notes included in the chord are more likely to appear in the melody of the automatically composed music data. On the other hand, if the user sets the information about harmony to "loose", the composition notes included in the chord are less likely to appear in the melody of the automatically composed music data. Figure 2 The example in shows that the user has applied information about a harmony that is closer to "strict".

[0068] The setting information 33 displays note length information, which is an example of setting information in the automatic composition function. The note length information is information for determining the length of notes in the music data composed by the processing server 100, for example. For example, if the user sets the note length information to "long", notes with longer note lengths (e.g., whole notes, half notes, etc.) are more likely to appear in the music data composed automatically. On the other hand, if the user sets the note length information to "short", notes with shorter note lengths (e.g., eighth notes, sixteenth notes, etc.) are more likely to appear in the music data composed automatically.

[0069] The setting information 34 displays information for determining the type and amount of material music other than the material music included in the designated information (the style palette designated by the user), which is an example of setting information in the automatic composition function. Such information is, for example, information for determining whether to perform learning strictly based on the music included in the style palette designated by the user in the music data composed by the processing server 100. For example, if the user sets such information to "never", the tendency to use music other than the music included in the style palette in the automatic composition learning decreases. On the other hand, if the user sets such information to "only", the tendency to use music other than the music included in the style palette in the automatic composition learning increases.

[0070] The music data 35 shows specific music data transmitted from the processing server 100. Figure 2 In the example of , the music data 35 includes information indicating a chord progression such as Cm, information indicating pitches and note lengths in a measure, transitions in note pitches (in other words, melody), and the like. Figure 2 As shown, for example, the music data 35 may include four different types of content. That is, the processing server 100 may transmit multiple pieces of music data instead of only one type of music data for automatic composition. As a result, the user can select his / her favorite music data from among the generated multiple music data candidates, or combine multiple pieces of music data to create his / her favorite music.

[0071] Note that although Figure 2 The user interface 30 shown in FIG. 1 shows data related to the melody among three different types of data: melody, chords and bass notes included in the music data, but other data is displayed on other user interfaces. This will be referred to in Figure 3 Give a description. Figure 3 is a diagram (2) showing one example of a user interface according to an embodiment.

[0072] like Figure 3 As shown, in addition to the user interface 30 displaying data related to the melody, the user terminal 10 may display a user interface 38 displaying data related to the chord and a user interface 39 displaying data related to the bass note on the screen. Figure 3 Although not shown in the figure, the user interface 38 and the user interface 39 display note information different from the music data 35 in the user interface 30. Specifically, the user interface 38 displays note information related to the chord corresponding to the melody of the music data (for example, the composition note of the chord Cm, etc.). In addition, the user interface 39 displays note information related to the bass note of the melody or chord corresponding to the music data (for example, the note "C" in the case of the chord Cm, etc.).

[0073] For example, the user may select information to be copied to the main application 20 from the displayed user interface 30, user interface 38, and user interface 39, or edit a portion of a bass note.

[0074] Next, we will refer to Figure 4 A style palette as an example of the above-mentioned setting information is described. Figure 4 is a diagram illustrating one example of a style palette according to an embodiment.

[0075] Figure 4 The window 40 shown is an example of a user interface displayed in the plug-in 22. For example, the user refers to the characteristic information of the style palette displayed in the window 40, and selects an image that matches the music to be automatically composed. For example, the user selects a style palette 41 having characteristic information "bright", a style palette 42 having characteristic information "dark", and the like. Alternatively, the user selects a style palette 43 named "America" ​​having the genre and type of music as characteristic information, or the user selects a style palette 44 named "verse->bridge->chorus" having the composition of music as characteristic information.

[0076] Note that, as described above, the style palette is information for specifying music used for learning by the processing server 100. That is, each style palette contains information for identifying existing music composed in advance. For example, it is assumed that the composition music list 50 is associated with the style palette 41. The composition music list 50 includes a plurality of existing music. In addition, it is assumed that the composition music list 51 is associated with the style palette 42. The composition music list 51 includes a plurality of existing music different from the music included in the composition music list 50.

[0077] For this reason, the learning model generated by machine learning based on the style palette 41 is different from the learning model generated by machine learning based on the style palette 42. This is because the learning data in the machine learning varies according to the style palette selected by the user. That is, the style palette can also be considered as designated information for designating learning data in automatic composition.

[0078] The music included in the style palette is pre-recorded by, for example, an administrator, provider, or the like of the plug-in 22. For example, the administrator of the plug-in 22 extracts a plurality of pieces of music that are subjectively perceived as "bright" to generate a composition music list 50, and associates the composition music list 50 with the style palette 41. Note that the style palette and the music corresponding to the style palette can be arbitrarily edited by the user of the plug-in 22. For example, the user can select music from a network service such as a song distribution service or a social networking service (SNS), combine the selected music clips, and generate a desired style palette. Specifically, the user can arbitrarily extract the music included in a playlist automatically generated by a predetermined music application or a playlist provided to the user of the music application, and change the composition music of the style palette he / she created or create a new style palette. As a result, the user can flexibly generate a style palette he / she likes.

[0079] Note that when selecting a style palette as setting information, the user can select multiple style palettes. For example, the user can select style palette 41 as setting information for a part of the song (e.g., the first eight bars), and select style palette 42 as setting information for another part of the song (e.g., the middle eight bars). This information including multiple style palettes is referred to as a style palette sequence hereinafter. In other words, a style palette sequence can be considered as combined designated information, in which multiple pieces of designated information (i.e., style palettes) for specifying music are combined. By setting a style palette sequence for music composition, the user can easily create various music data with multiple features in a piece of music.

[0080] Next, we will refer to Figure 5 The relationship between the main application 20 and the plug-in 22 is conceptually shown. Figure 5 is a block diagram showing the flow of information processing according to an embodiment.

[0081] Figure 5 The processing block 60 shown in FIG. 1 shows the processing flow performed by the DAW (host application 20) and the plug-in 22. Generally, a performer records the sound of a musical instrument performance or creates data, such as MIDI. In addition, in the information processing according to this embodiment, the music data corresponding to the melody, chords, and bass notes are generated by the plug-in 22 instead of the performer. For example, the performer or the plug-in 22 generates a melody 61, a chord 62, or a bass note 63.

[0082] Thereafter, the user records the melody 61, chords 62, and bass notes 63 for the music in a recorder associated with the DAW, and creates each track corresponding to the melody, chords, and bass notes. For example, the user sets instrument information indicating the instrument used in the performance for the melody 61 generated by the plug-in 22. Specifically, the user sets the instrument information, for example, recording the melody 61 played on the guitar in the DAW. Then, the user records the sound of the virtual guitar performance in the recorder, and creates a track corresponding to the guitar. Note that since the DAW can create multiple tracks, a track based on the player's performance sound and a track based on the music data created by the plug-in 22 can coexist.

[0083] Thereafter, the user mixes the tracks on the DAW, and creates music data by performing mixing, etc. In addition, the user performs mastering on the DAW, adjusts the sound signal level, etc., and creates a music file 65 that can be played on a playback device, etc.

[0084] As described above, according to the information processing according to this embodiment, the user can use the data automatically composed by the plug-in 22 based on the performance data played by the performer and the created MIDI data, and create music on the DAW. For example, the user can create music on the DAW by mixing the melody automatically composed by the AI ​​with the performance data played by the performer, or by incorporating the chord progression proposed by the AI ​​into the performance data played by the performer.

[0085] In the above, an overview of the overall flow of information processing according to the present disclosure has been described. Figure 6 In the following drawings, the configuration of the information processing system 1 including the user terminal 10 will be described, and the details of various processes will be described in sequence.

[0086] [1-2. Configuration of Information Processing System According to First Embodiment]

[0087] Figure 6 1 is a diagram showing an example of the information processing system 1 according to the first embodiment of the present disclosure. Figure 1 As shown, the information processing system 1 includes a user terminal 10 , a processing server 100 , and a management server 200 .

[0088] The user terminal 10 is one example of an information processing device according to the present disclosure, and controls operations of a main application 20 and a plug-in 22 .

[0089] The processing server 100 is one example of an external server according to the present disclosure, and performs automatic composition processing in cooperation with the plug-in 22 .

[0090] The management server 200 is, for example, a server managed by a business operator or the like that provides the plug-in 22 .

[0091] For example, the management server 200 manages the user rights of the user of the plug-in 22, and manages the information of the style palette available in the plug-in 22. For example, the management server 200 determines whether the user has the right to use the plug-in 22 based on the user ID that uniquely identifies the user. In addition, the management server 200 creates a style palette, edits the music included in the style palette, and transmits the information about the style palette to the user terminal 10 and the processing server 100. Note that the management server 200 can be configured integrally with the processing server 100.

[0092] [1-3. Configuration of information processing device (user terminal) according to embodiment]

[0093] Next, we will refer to Figure 7 The configuration of the user terminal 10 as one example of the information processing device according to the present disclosure is described. Figure 7 1 is a diagram showing a configuration example of a user terminal 10 according to an embodiment of the present disclosure. Figure 7 As shown, the user terminal 10 includes a communication unit 11 , an input unit 12 , a display unit 13 , a storage unit 15 and a control unit 16 .

[0094] The communication unit 11 is implemented by, for example, a network interface card (NIC), etc. The communication unit 11 is connected to a network N (Internet, etc.) by wire or wireless, and transmits and receives information to and from the processing server 100, the management server 200, etc. through the network N.

[0095] The input unit 12 is an input device that accepts various operations from the user. For example, the input unit 12 is implemented by an operation key or the like included in the user terminal 10. The display unit 13 is a display device for displaying various types of information. For example, the display unit 13 is implemented by a liquid crystal display or the like. Note that when the user terminal 10 adopts a touch panel, a part of the input unit 12 and the display unit 13 are integrated.

[0096] The storage unit 15 is realized by, for example, a semiconductor storage element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk, etc. The storage unit 15 stores various data used for information processing.

[0097] like Figure 7 As shown, the storage unit 15 stores various information, such as composition setting information 151, composition music information 152, history information 153, and associated instrument information 154. Hereinafter, each type of information will be described.

[0098] The composition setting information 151 is information used by the plug-in 22 (second application) when automatic composition is performed. Figure 8 An example of the composition setting information 151 according to the embodiment is shown. Figure 8 is a diagram showing one example of the composition setting information 151 according to the embodiment.

[0099] like Figure 8 As shown, the composition setting information 151 includes composition corpus information and performance style information. As the composition corpus information, the music data used as learning data for automatic composition or the place where the music data is saved (for example, the address of a data server) is stored. In addition, the composition corpus information includes information such as the average length and modulation of notes of each piece of music.

[0100] As the performance style information, information such as the performance style of music used as learning data for automatic composition is stored. For example, the performance style includes information such as the overall shuffle ratio, chord and bass note separation, and overall balance.

[0101] The composition music information 152 is information about music used by the plug-in 22 when automatic composition is performed. Fig. 9 One example of the composition music information 152 according to the embodiment is shown. Fig. 9 : is a diagram showing one example of the composition music information 152 according to the embodiment.

[0102] like Fig. 9 As shown, the composition music information 152 includes a music ID, a style palette ID, and a style palette sequence ID. The music ID indicates identification information for uniquely identifying existing music used as learning data. The style palette ID indicates identification information for identifying a style palette including multiple existing music. The style palette sequence ID indicates identification information for identifying a style palette sequence including multiple style palettes.

[0103] The history information 153 indicates the history of the user's operations in the main application 20 and the plug-in 22 and the history of music created by the user. Fig.10 An example of the history information 153 according to the embodiment is shown. Fig.10 is a diagram showing one example of the history information 153 according to the embodiment.

[0104] like Fig.10 As shown, the history information 153 includes composition music information. The composition music information includes music data transmitted from the processing server 100, a plurality of candidate data included in the music data (for example, including four types of segmented music data 35 generated based on certain setting information, etc., such as Figure 2 as shown) and music data edited by the user.

[0105] The related musical instrument information 154 indicates musical instrument information set for the music data transmitted from the processing server 100 and a plurality of pieces of candidate data included in the music data. Fig.11 An example of the related instrument information 154 according to the embodiment is shown. Fig.11 is a diagram showing one example of the related musical instrument information 154 according to the embodiment.

[0106] like Fig.11 As shown, the related instrument information 154 includes related instrument information. The related instrument information indicates information for identifying an instrument group for virtually performing music data or a plurality of candidate data included in the music data transmitted from the processing server 100, the name of the instrument group for the music data, etc. For example, as the related instrument information, different information may be set for each of the melody, chord, and bass note.

[0107] Back to Figure 7 , will be described further. For example, the control unit 16 is implemented by a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), etc. that executes a program stored inside the user terminal 10 (for example, an information processing program according to the present disclosure), wherein a random access memory (RAM) or the like is used as a working area. In addition, the control unit 16 is a controller and can be implemented by, for example, an integrated circuit, for example, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0108] like Figure 7 As shown, the control unit 16 has a main application control unit 161 and a plug-in application control unit 165, and implements or executes the information processing functions and operations described below. The main application control unit 161 includes a plug-in control unit 162, a playback unit 163, and a display control unit 164. The plug-in application control unit 165 includes a selection unit 166, a transmission / reception unit 167, a playback unit 168, and a display control unit 169. Note that the internal configuration of the control unit 16 is not limited to Figure 7 The configuration shown, and may be another configuration as long as it is a configuration for executing information processing described later.

[0109] The main application control unit 161 controls the main application 20 (DAW as the first application).

[0110] The plug-in control unit 162 controls the operations of various plug-ins in the main application 20. For example, the plug-in control unit 162 controls operations such as calling a plug-in in the main application 20, activating a plug-in on the main application 20, and copying data in the plug-in to the main application 20.

[0111] For example, when the plug-in plays a chord, melody, or bass note included in the music data received from the processing server 100, the plug-in control unit 162 individually sets instrument information for specifying the sound quality. For example, the plug-in control unit 162 reads out information of a virtual musical instrument recorded on the DAW, and sets information about the virtual musical instrument to play each of the chord, melody, or bass note included in the music data of the plug-in.

[0112] The playback unit 163 controls playback processing in the main application 20. The playback unit 163 has a synchronized playback function, a playback information transmission function, a sound synthesis playback function, a playback style arrangement function, and the like in the main application 20.

[0113] For example, the playback unit 163 cooperates with the synchronous playback function of the playback unit 168 of the plug-in to play the music data saved by the plug-in. For example, the playback unit 163 can pass time information indicating the position being played by the main application to the plug-in to obtain and play the melody, chords and bass notes of the part corresponding to the playback position.

[0114] In addition, in the case where a performance style or the like is set in the plug-in, the playback unit 163 may process the playback data according to the performance style and play the processed data.

[0115] The display control unit 164 controls the display control processing in the main application 20. For example, the display control unit 164 has a performance information display function for displaying information about each track on the screen (display unit 13), a composition music information paste function for copying information such as music data to a track, and the like.

[0116] In addition, the display control unit 164 controls the plug-in to individually display each window, each of which displays information about the chord, melody, or bass note included in the music data received from the processing server 100. For example, the display control unit 164 displays a user interface corresponding to each of the chord, melody, or bass note on the screen of the DAW, such as Figure 3 shown.

[0117] In addition, according to the user's operation, the display control unit 164 controls the transmission and reception of information between each window displaying information about chords, melody or bass notes and the window displaying information about the main application. As a result, the user can quickly perform processing, such as copying automatically composed music data to an arbitrary track or performing editing on a DAW.

[0118] Note that the display control unit 164 can control not only the information exchange between the main application and the plug-in, but also the information exchange between the displayed plug-in windows. That is, the display control unit 164 can control the transmission and reception of information between windows, each of which displays information about chords, melody, or bass notes, according to the user's operation.

[0119] The plug-in application control unit 165 controls the operation of the plug-in running on the main application 20. For example, the plug-in application control unit 165 activates the plug-in on the main application 20 according to the user's operation.

[0120] The selection unit 166 selects setting information for controlling the composition function based on machine learning. For example, the selection unit 166 selects, as the setting information, designation information for designating material music as learning data for machine learning. Specifically, the designation information corresponds to Figure 4 and the style palettes shown in the other figures.

[0121] For example, according to the user's operation, the selection unit 166 selects the specified information, which is pre-stored in the storage unit 15 and includes characteristic information indicating the characteristics of the specified information and a plurality of material music pieces associated with the characteristic information. Figure 4 The illustrated window 40 or the like refers to the characteristic information of the style palette (“light”, “dark”, etc.) Then, based on the characteristic information, the user selects a style palette that includes desired characteristic information of the music to be composed by the AI.

[0122] In addition, the selection unit 166 can select the combined designation information in which the first designation information corresponding to some bars of the music data composed by the external server and the second designation information corresponding to some other bars thereof are combined. As described above, the combined designation information corresponds to the style palette sequence. In addition, the first designation information corresponds to the style palette, which is the setting information for composing some bars. In addition, the second designation information corresponds to the style palette, which is the setting information for composing some other bars.

[0123] Furthermore, the selection unit 166 may select detailed setting information on the music data to be composed, in addition to the style palette.

[0124] For example, the selection unit 166 may select, as the setting information, the length information of the musical notes included in the music data composed by the processing server 100 based on the style palette. Figure 2 The slider or the like of the setting information 33 in the user interface 30 or the like shown in the figure accepts the user's selection of the note length information.

[0125] In addition, the selection unit 166 may select information for determining the probability that the composition notes included in the chord appear in the melody of the music data composed by the processing server 100 as the setting information based on the style palette. Figure 2 The slider or the like of the setting information 32 in the illustrated user interface 30 or the like accepts selection from the user of information for determining the probability that a composition note included in a chord appears in the melody.

[0126] In addition, the selection unit 166 may select information for determining the type and amount of material music other than the material music included in the style palette in the music data composed by the processing server 100 as the setting information based on the style palette. Figure 2 The slider or the like of the setting information 34 in the illustrated user interface 30 or the like accepts selection from the user of information for determining the type and amount of material music other than the material music included in the style palette.

[0127] In addition, the selection unit 166 can select information other than the style palette as setting information for automatic composition. As an example, the selection unit 166 can select a chord progression in the composition music as setting information based on the user's operation. In this case, the processing server 100 automatically generates music data based on the chord progression selected by the user.

[0128] The transmission / reception unit 167 transmits the setting information selected by the selection unit 166 to the processing server 100 , which performs a composition function based on machine learning through the network N, and receives music data composed by the processing server 100 .

[0129] For example, the transmission / reception unit 167 transmits the style palette selected by the selection unit 166 to the processing server 100. Then, the transmission / reception unit 167 receives the music data generated by the processing server 100 based on the style palette.

[0130] The transmission / reception unit 167 receives, for example, chords in a measure of a specified length, melody in a measure, and bass notes in a measure as music data. Such information may be data such as MIDI, MusicXML, information of a DAW original standard, or waveform data (WAV file, etc.).

[0131] In addition, the transmission / reception unit 167 may transmit the style palette sequence selected by the selection unit 166 to the processing server 100. In this case, the transmission / reception unit 167 receives the music data generated by the processing server 1007 based on the style palette sequence.

[0132] When the transmission / reception unit 167 receives music data composed by the processing server 100 based on the style palette sequence, the transmission / reception unit 167 may store the music data in association with the style palette sequence in the storage unit 15. As a result, the user can refer to what kind of music data was created by what kind of style palette sequence as a history, so that such information can be used for composition.

[0133] In addition, the transmission / reception unit 167 may transmit various setting information other than the style palette and the style palette sequence to the processing server 100. For example, the transmission / reception unit 167 transmits information set by the user, such as note length information, information for determining the probability of composition notes included in the chord appearing in the melody, and information for determining the type and amount of material music other than the material music included in the style palette, to the processing server 100.

[0134] In addition, when the user performs a playback or editing operation on the music data composed by the processing server 100 after receiving the music data, the transmission / reception unit 167 can transmit information about the playback or editing operation to the processing server 100. As a result, the processing server 100 can obtain information such as how the composition data is used or how much of the composition music data is used. In this case, the processing server 100 can adjust the learning method and the music data to be generated based on such information. For example, the processing server 100 can analyze past music data used by more users and preferentially generate music data having such characteristics.

[0135] The playback unit 168 controls the playback processing in the plug-in. For example, the playback unit 168 plays the music data received by the transmission / reception unit 167. Specifically, the playback unit 168 sets arbitrary instrument information for each of the melody, chord, and bass note included in the music data, and plays each piece of data. Note that the playback unit 168 can play the melody, chord, and bass note in combination.

[0136] The display control unit 169 controls display processing in the plug-in. For example, the display control unit 169 displays a window such as a user interface that displays plug-in information on the screen.

[0137] like Figure 2 As shown, for example, the display control unit 169 obtains four types of music data of four measures and displays four candidates side by side in the user interface. The user can select the final candidate by connecting the parts (i.e., performing accompaniment) along the time axis from the four types of music data candidates. For example, the user can delete or connect some notes in the melody, change the length of the notes, or change the pitch of the notes.

[0138] In addition, the display control unit 169 may perform control to retrieve the history of past music data composed by the processing server 100 from the storage unit 15 and display the history of past music data according to the user's operation. As a result, the user can continue to compose music while referring to the data composed in the past by the processing server 100. For example, the user can determine the final candidate by comparing the latest music created by editing with the history of music edited in the past.

[0139] In addition, the display control unit 169 can perform control to retrieve the history of editing operations performed on past music data composed by the processing server 100 from the storage unit 15, and also display the editing operations performed on the past music data. As a result, the user can refer to the editing operations performed in the past, for example, the music data generated by the editing operations, so that composition can be performed efficiently.

[0140] Note that although Figure 2 and Figure 3 An example is shown in which music data is displayed on the user interface in a format similar to a so-called piano roll that displays pitches and note lengths, but the display control unit 169 may display music data in a staff notation or a format specific to a DAW.

[0141] [1-4. Configuration of external server (processing server) according to the embodiment]

[0142] Next, the configuration of the processing server 100 as one example of the external server according to the present disclosure will be described. Fig.12 is a diagram showing a configuration example of the processing server 100 according to the embodiment.

[0143] like Fig.12 As shown, the processing server 100 includes a communication unit 110, a storage unit 120, and a control unit 130. Note that the processing server 100 may have an input unit (e.g., a keyboard, a mouse, etc.) for accepting various operations from an administrator or the like who manages the processing server 100, and a display unit (e.g., a liquid crystal display, etc.) for displaying various information.

[0144] The communication unit 110 is implemented by, for example, a NIC, etc. The communication unit 210 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the user terminal 10, the management server 20, etc. through the network N.

[0145] The storage unit 120 is realized by, for example, a semiconductor storage element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various data used for information processing.

[0146] like Fig.12As shown, the storage unit 120 stores various information, such as user information 121, music information 122, style palette information 123, style palette sequence information 124, user composition information 125, and history information 126. Hereinafter, each type of information will be described.

[0147] The user information 121 indicates information of the user of the plug-in 22 (second application). Fig.13 An example of the user information 121 according to the embodiment is shown. Fig.13 is a diagram showing one example of the user information 121 according to the embodiment.

[0148] like Fig.13 As shown, user information 121 includes user ID, user meta information and permission information. User ID represents identification information for uniquely identifying a user. User meta information is additional information of a user, such as the user's name and address. Permission information is, for example, identification information, such as whether the user of the plug-in is an administrator, a normal user or a special user.

[0149] The music information 122 indicates information on music used for the automatic composition process. Fig.14 An example of the music information 122 according to the embodiment is shown. Fig.14 : is a diagram showing one example of the music information 122 according to the embodiment.

[0150] like Fig.14 As shown, the music information 122 includes music ID, music meta information, melody information, chord progression information, and bass note progression information. The music ID indicates identification information for uniquely identifying music. The music meta information is, for example, information such as music title, composer, date, and music genre. The melody information is, for example, scale information expressing a sound part, etc. The chord progression information is, for example, time series information indicating a chord transition in the music. The bass note progression information is time series information indicating a root note of the chord progression information.

[0151] The style palette information 123 indicates information on a style palette used for automatic composition processing. Fig.15 An example of the style palette information 123 according to an embodiment is shown. Fig.15 is a diagram showing one example of the style palette information 123 according to the embodiment.

[0152] like Fig.15As shown, the style palette information 123 includes a style palette ID, style palette meta information, and music ID. The style palette ID indicates identification information for uniquely identifying a style palette. The style palette meta information includes, for example, information such as the name of the style palette, subjective feature information such as light or dark, fast or slow, and features such as the structure of a song including a verse, a bridge, and a chorus, and a chord progression. Note chord progression information, etc. may be added to the name of the style palette meta information. As a result, the user can intuitively grasp the features of the style palette. In addition, as Fig.15 As shown, a plurality of music IDs are recorded in association with a style palette. For example, the style palette information 123 is recorded by the management server 200 or the like.

[0153] The style palette sequence information 124 indicates information on a style palette sequence used for automatic composition processing. Fig.16 An example of style palette sequence information 124 according to an embodiment is shown. Fig.16 is a diagram showing one example of the style palette sequence information 124 according to the embodiment.

[0154] like Fig.16 As shown, the style palette sequence information 124 includes a style palette sequence ID, style palette sequence meta information, style palette ID, and formulation information. The style palette sequence ID indicates identification information for uniquely identifying a style palette sequence. The style palette sequence meta information is, for example, the name of the style palette sequence and subjective feature information, such as light or dark, fast or slow. In addition, Fig.16 As shown, a plurality of style palette IDs are recorded in association with the style palette sequence. In addition, the blending information is information on the arrangement of the style palettes in the style palette sequence.

[0155] The user composition information 125 indicates information on the composition received from the user terminal 10 . Fig.17 An example of the user composition information 125 according to the embodiment is shown. Fig.17 is a diagram showing one example of the user composition information 125 according to the embodiment.

[0156] like Fig.17 As shown, the user composition information 125 includes a user ID, a music ID, a style palette ID, and a style palette sequence ID. The user ID indicates identification information for uniquely identifying a user. The music ID indicates identification information for identifying music generated for the user identified by the user ID. The style palette ID indicates identification information for identifying a style palette transmitted from the user identified by the user ID. The style palette sequence ID indicates identification information for identifying a style palette sequence transmitted from the user identified by the user ID.

[0157] The history information 126 is various histories related to information processing by the processing server 100 . Fig.18 An example of the history information 126 according to an embodiment is shown. Fig.18 is a diagram showing one example of the history information 126 according to the embodiment.

[0158] like Fig.18 As shown, the history information 126 includes composition history information and operation history information. The composition history information is the history of music generated by the processing server 100. The operation history information is information such as the history of editing operations of the user on the user terminal 10. The operation history information is, for example, information about the user performing arrangement, selecting music data for composition, editing music data, information about the number of times the playback is performed and the number of times the playback is skipped, etc. This information can be used as learning data for the composition unit 134, which will be described later.

[0159] Back to Fig.12 , will continue to describe. The control unit 130 is implemented by, for example, a CPU, an MPU, a GPU, etc. that executes a program stored inside the processing server 100, wherein the RAM, etc. is used as a working area. In addition, the control unit 130 is a controller and can be implemented by, for example, an integrated circuit (e.g., ASIC or FPGA).

[0160] like Fig.12 As shown, the control unit 130 has a receiving unit 131, a management unit 132, an acquisition unit 133, a composition unit 134, and a transmission unit 135, and implements or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to Fig.12 The configuration shown, and may be another configuration as long as it is a configuration for executing information processing described later.

[0161] The acceptance unit 131 accepts various information transmitted from the management server 200. For example, the acceptance unit 131 accepts information about the user of the plug-in, information about the style palette, information about the material music used in the automatic composition, etc. For example, when a user purchases and activates a product (plug-in, DAW, etc.), the acceptance unit 131 performs a process of issuing a user ID to the user and accepting information about the user. In addition, the acceptance unit 131 accepts the recording of music to be linked to the style palette, the editing of the style palette, etc. according to the operation and command from the management server 200.

[0162] The management unit 132 manages various information accepted by the acceptance unit 131. For example, the management unit 132 stores various information in the storage unit 120, and appropriately updates the stored information.

[0163] For example, when the style palette recording process of the management unit 132 is completed, the user can acquire and browse the list of style palette information.

[0164] The acquisition unit 133 acquires the automatic composition request transmitted from the user terminal 10. In addition, the acquisition unit 133 acquires the setting information transmitted together with the request. For example, the acquisition unit 133 acquires the style palette desired by the user as the setting information.

[0165] The composition unit 134 composes music based on the setting information acquired by the acquisition unit 133. The composition unit 134 can compose music by using various existing music generation algorithms. For example, the composition unit 134 can use a music generation algorithm using a Markov chain, or can use a music generation algorithm using deep learning. As described above, the composition unit 134 generates multiple pieces of music data for a single piece of setting information transmitted from the user. As a result, the user can receive multiple suggestions from the composition unit 134, and can therefore continue to compose by using more diverse information.

[0166] When music data is generated through the composition process, the composition unit 134 associates the generated music data with the user ID of the user who transmitted the style palette, and stores it in the storage unit 120 as history information.

[0167] The transmission unit 135 transmits the music data generated by the composition unit 134 to the user terminal 10 .

[0168] [1-5. Information processing process according to the embodiment]

[0169] Next, we will refer to Fig.19 A procedure of information processing according to an embodiment is described. Fig.19 is a flowchart showing a procedure of information processing according to an embodiment.

[0170] like Fig.19 As shown, the user terminal 10 activates the automatic composition function (plug-in) on the main application according to the user's operation (S101).

[0171] Subsequently, the user terminal 10 determines whether the selection of a style palette or the like is accepted from the user (step S102). If the selection of a style palette or the like is not accepted from the user (step S102, No), the user terminal 10 waits until the selection is accepted.

[0172] On the other hand, if the selection of a style palette or the like is accepted from the user (step S102, Yes), the user terminal 10 selects a style palette according to the user's operation (step S103). Note that in step S102, the user terminal 10 may accept various setting information other than the style palette.

[0173] Thereafter, the user terminal 10 determines whether a composition request is accepted from the user (step S104). If a composition request is not accepted from the user (step S104: No), the user terminal 10 waits until the request is accepted.

[0174] On the other hand, if a composition request is accepted from the user (step S104, yes), the user terminal 10 transmits the accepted setting information together with the composition request to the processing server 100 (step S105). Thereafter, the user terminal 10 receives the music data composed (generated) by the processing server 100 (step S106).

[0175] Subsequently, the user terminal 10 determines whether the user has performed editing processing etc. on the user terminal 10 (step S107). If editing processing etc. has not been performed (step S107, No), the user terminal 10 waits until accepting editing processing etc. (step S107).

[0176] On the other hand, if the editing process or the like has been performed (step S107, Yes), the user terminal 10 reflects the editing and transmits information on the editing operation to the processing server 100 (step S108).

[0177] Thereafter, the user terminal 10 determines whether another composition request is accepted from the user (step S109). If a composition request is accepted from the user (step S109, Yes), the user terminal 10 accepts new setting information from the user.

[0178] On the other hand, if the composition request is not accepted from the user (step S109, No), the user terminal 10 determines whether to accept the main application termination request (step S110). If the main application termination request is not accepted (step S110, No), the user terminal 10 continues the editing process of the currently received music data. On the other hand, if the main application termination request is accepted (step S110 Yes), the user terminal 10 terminates the main application and the plug-in and ends the process.

[0179] (2. Modification)

[0180] The above-described information processing system 1 can be implemented in various forms other than the above-described embodiment. Therefore, modifications of the embodiment will be described below.

[0181] [2-1. Music data format]

[0182] In the above-described embodiment, it is assumed that the information types set in the related instrument information 154 of the music data in the plug-in are melody, chord, and bass notes. However, the present invention is not limited thereto. For example, the related instrument information 154 can be applied not only to melody, chord, and bass notes, but also to the performance part of each instrument of a large band, for example.

[0183] [2-2. Main application]

[0184] In the above embodiment, it is assumed that DAW is the main application. However, the present invention is not limited to this. For example, the main application may be a video editing application, etc., instead of a music editing application.

[0185] [2-3. Controlled by DAW]

[0186] In the above-described embodiment, an example is shown in which the user terminal 10 selects setting information about the plug-in and transmits the selected information to the processing server 100. However, the setting information and the like may be selected by the main application. That is, the user terminal 10 may transmit setting information (e.g., chord progression) and the like selected in the main application to the processing server 100 so that the automatic composition process can be performed. In this case, the main application may provide an application programming interface (API) to the plug-in so that the plug-in uses the information of the main application, and allows the information for generating a style palette to be obtained from the main application, and controls the transmission and reception process of the processing server 100.

[0187] For example, the user terminal 10 generates an arbitrary chord progression using the chord generation function of the DAW as the main application. Then, the user terminal 10 can perform automatic composition based on the chord progression generated by the DAW. For example, the user terminal 10 inputs the chord progression generated by the DAW into the plug-in, and transmits the chord progression to the processing server 100 through the plug-in.

[0188] That is, the main application performs control to transmit information about the chord progression generated in the main application to the plug-in. Then, the plug-in selects the information about the chord progression generated in the main application as setting information. In addition, the plug-in transmits the information about the chord progression generated in the main application to the processing server 100, and receives music data composed based on the information about the chord progression.

[0189] Furthermore, the user terminal 10 may automatically select a style palette to be transmitted to the processing server 100 based on the chord progression generated by the DAW. For example, the user terminal 10 may select a style palette having features similar to the chord progression generated by the DAW, and transmit the style palette to the processing server 100. In addition, the user terminal 10 may sequentially select style palettes based on the chord progression generated by the DAW, generate a style palette sequence, and transmit the generated style palette sequence to the processing server 100.

[0190] In addition, the user terminal 10 may perform settings so that the plug-in is allowed to access the information of the basic track on the main application. For example, the user sets the basic track of the DAW so that the track follows the music data of the automatic composition. In this case, for example, the basic track is automatically supplemented based on the music data generated by the processing server 100 and the chord progression generated by the DAW.

[0191] In addition, the user terminal 10 can perform settings so that the plug-in is allowed to access the information of the melody track on the main application. For example, the user sets the melody track of the DAW so that the track automatically follows the composed music data. In this case, when the user selects a certain measure and requests automatic composition, the generated melody is automatically inserted into the track. In addition, when the user sets the DAW to a mode for editing (called accompaniment mode, etc.) by combining multiple pieces of music data, the user can complete the melody by selecting the desired part of the multiple tracks that appear on the screen.

[0192] In addition, the user terminal 10 can perform settings so that the plug-in is allowed to access the melody track and MIDI input information on the main application. In this case, the user can perform composition by making full use of the automatic composition function and MIDI input. For example, the user enters an arbitrary chord progression in four bars and makes the DAW loop. Then, the user inputs with a MIDI keyboard according to the loop performance. By uploading the chord progression and melody information to the processing server 100, the user terminal 10 can automatically create a personal style palette on the processing server 100 side. For example, in the newly added style palette menu on the DAW, the user can give instructions to start or stop creating, saving, naming or deleting a personal style palette. This personal style palette can be publicly obtained through the style palette menu.

[0193] In addition, the user terminal 10 can perform settings so that the plug-in is allowed to access the information of the audio track on the main application. The audio track is, for example, an audio track that records the sound of an instrument, and is, for example, an audio track that includes the chord performance of a piano, the bass note performance of a bass guitar, the melody of a main musical instrument, etc. The plug-in accesses the audio track, analyzes the audio data through signal processing, for example, the melody, chord and bass notes of each audio track, and obtains the NIDI information of the melody, the chord progression information, etc. The plug-in can be analyzed using, for example, 12-tone analysis technology, etc. In this case, the user terminal 10 can transmit the analyzed information to the processing server 100 to automatically infer the best chord progression through machine learning, etc. Then, the processing server 100 defines a style palette sequence based on the chord progression information. As a result, the user can perform auxiliary composition based on the style palette sequence generated by the processing server 100, so that, for example, the entire composition can be adapted or the composition can be partially adapted and replaced.

[0194] In addition, the user terminal 10 may perform settings so that the plug-in is allowed to access the information of the existing master track on the main application. For example, the master track is obtained by performing a mix in the DAW and down-mixing to two-channel stereo. The plug-in accesses the master track, analyzes the audio data through signal processing, obtains chord progression information, etc. The user terminal 10 may transmit the analyzed information to the processing server 100 to automatically infer the optimal chord progression through machine learning, etc. The processing server 100 then defines a style palette sequence based on the chord progression information. As a result, the user can perform auxiliary composition based on the style palette sequence generated by the processing server 100, so that, for example, the entire composition can be adapted or the composition can be partially adapted and replaced.

[0195] As described above, in the case where the main application is provided with various functions, according to the present disclosure, the user terminal 10 can apply these functions to the plug-in and use them for information processing. For example, as described above, the user terminal 10 can generate a style palette sequence based on the chord progression generated by the DAW, and make the style palette sequence publicly available on the network to stimulate composition processing between users.

[0196] [2-4. Model of information processing system]

[0197] In this embodiment, it is assumed that the processing server 100 is installed on a cloud network. However, the present invention is not limited to this example, and the processing server 100 and the management server 200 may be installed on a network such as a local area network (LAN) as long as communication with the user terminal 10 is possible.

[0198] In this embodiment, an example is shown in which the first application and the second application are installed in the user terminal 10. However, the first application and the second application may be applications installed in different devices. For example, the user terminal 10 may have only the function of the first application, and play the sound source by controlling the second application installed on another device such as a tablet terminal or a smartphone, for example.

[0199] (3. Other embodiments)

[0200] The processing according to each of the above-described embodiments may be performed in various different forms than those of the above-described embodiments.

[0201] In addition, in the processing described in each of the above embodiments, all or part of the processing described as being automatically performed may be performed manually, or all or part of the processing described as being manually performed may be automatically performed by a known method. In addition, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters shown in the above description and drawings may be changed arbitrarily. For example, the various information shown in the drawings is not limited to the information shown.

[0202] In addition, each component of each device shown is a functional concept and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of allocation or integration of each device is not limited to the form shown in the figure, and all or part of the devices can be functionally or physically allocated or integrated in any unit according to various loads and usage conditions.

[0203] In addition, the above-described embodiments and modifications can be appropriately combined as long as the processing contents do not contradict each other.

[0204] In addition, the effects described in this specification are merely illustrative and not restrictive. Therefore, other effects can be obtained.

[0205] (4. Effects of the information processing device according to the present disclosure)

[0206] As described above, the information processing device (in the embodiment, the user terminal 10) according to the present disclosure controls a first application (in the embodiment, the main application 20) and a second application (in the embodiment, the plug-in 22), and the second application is used as a plug-in that extends the function of the first application. The first application includes a control unit (in the embodiment, the main application control unit 161) that controls the operation of the second application in the first application. The second application includes: a selection unit (in the embodiment, the selection unit 166), which selects setting information for controlling a composition function based on machine learning; and a transmission / reception unit (in the embodiment, the transmission / reception unit 167), which transmits the setting information to an external server (in the embodiment, the processing server 100), which performs a composition function based on machine learning and receives music data composed by the external server through a network.

[0207] As described above, the information processing device according to the present disclosure processes the second application with the automatic composition function as a plug-in, and causes the external server to perform the actual composition process. As a result, the information processing device can provide the user with an environment with good work efficiency while suppressing the processing load. That is, the information processing device can improve the convenience of the automatic composition function of AI.

[0208] The transmission / reception unit receives the chords in a measure of a specified length, the melody in a measure, and the bass notes in a measure as music data. As a result, the information processing device can refer to and edit the music data individually, which can improve the convenience of the user.

[0209] The control unit separately sets musical instrument information for specifying the sound quality when playing the chord, melody or bass note included in the music data. As a result, the information processing device can provide various playing environments.

[0210] The control unit performs control to separately display each window displaying information about a chord, melody, or bass note included in the music data. As a result, the information processing apparatus can improve the convenience of the user's editing operation.

[0211] According to the user's operation, the control unit controls the transmission and reception of information between each window displaying information about the chord, melody or bass note and the window displaying information about the first application. As a result, the information processing device can exchange information between the first application and the second application through operations such as drag and drop, thereby improving the convenience of the user's editing operation.

[0212] The control unit controls the transmission and reception of information between windows according to the user's operation, and each window displays information about chords, melody or bass notes. As a result, the information processing device can improve the convenience of user editing operations.

[0213] The selection unit selects, as setting information, designation information (in the embodiment, a style palette) for designating material music as learning data for machine learning. The transmission / reception unit transmits the designation information selected by the selection unit to an external server. As a result, the information processing device can perform automatic composition by designating various features desired by the user.

[0214] According to the user's operation, the selection unit selects the specified information, which is pre-stored in the storage unit (in the embodiment, the storage unit 15) and includes characteristic information indicating the characteristics of the specified information and a plurality of material music pieces associated with the characteristic information. As a result, when the user selects the specified information, the information processing device can improve convenience.

[0215] The selection unit selects the combined designation information (in the embodiment, the style palette sequence) in which the first designation information corresponding to some bars of the music data composed by the external server and the second designation information corresponding to some other bars thereof are combined. As a result, the information processing apparatus can automatically generate various music.

[0216] When the transmission / reception unit receives music data composed by an external server based on the combined designation information, the transmission / reception unit stores the combined designation information in association with the music data in the storage unit. As a result, when the user refers to the combined designation information, etc., which is the basis of the music data created in the past, the information processing device can improve convenience.

[0217] The selection unit selects the length information of the musical note included in the music data composed by the external server as the setting information based on the designated information. The transmission / reception unit transmits the designated information and the musical note length information to the external server. As a result, the information processing device can generate music data having the characteristics desired by the user.

[0218] The selection unit selects information for determining the probability that the composition notes included in the chord appear in the melody of the music data composed by the external server as the setting information based on the designated information. The transmission / reception unit transmits the designated information and the information for determining the probability that the composition notes included in the chord appear in the melody to the external server. As a result, the information processing device can generate music data having characteristics desired by the user.

[0219] The selection unit selects information for determining the type and amount of material music other than the material music included in the designated information in the music data composed by the external server as setting information based on the designated information. The transmission / reception unit transmits the designated information and the information for determining the type and amount of material music other than the material music included in the designated information to the external server. As a result, the information processing device can generate music data having characteristics desired by the user.

[0220] The second application further includes a display control unit (in the embodiment, display control unit 169) that performs control according to the user's operation to retrieve the history of past music data composed by the external server from the storage unit and display the history of the past music data. As a result, the information processing device can improve convenience when the user consults the past operation history, etc.

[0221] The display control unit performs control to retrieve the history of editing operations performed on past music data composed by an external server from the storage unit, and also displays the editing operations performed on the past music data. As a result, the information processing device can improve convenience when the user consults the past operation history, etc.

[0222] When the user performs a playback or editing operation on the music data composed by the external server after receiving the music data, the transmission / reception unit transmits information about the playback or editing operation to the external server. As a result, the information processing device can enable the processing server 100 to perform further learning based on the editing performed by the user, etc.

[0223] The selection unit selects a chord progression in the composed music based on the user's operation as setting information. The transmission / reception unit transmits the chord progression selected by the selection unit to the external server. As a result, the information processing device can provide the music data desired by the user without relying on the specified information.

[0224] The control unit performs control to transmit information about the chord progression generated in the first application to the second application. The selection unit selects the information about the chord progression generated in the first application as setting information. The transmission / reception unit transmits the information about the chord progression generated in the first application to the external server, and receives music data composed based on the information about the chord progression. As a result, the information processing device can perform composition processing using the function of the first application such as a DAW.

[0225] (5. Hardware Configuration)

[0226] For example, the information devices such as the user terminal 10, the processing server 100, and the management server 200 according to each of the above-mentioned embodiments are composed of Fig. 20 The configuration shown is implemented by a computer 1000. Hereinafter, the user terminal 10 according to the embodiment will be described as an example. Fig. 20 1 is a hardware configuration diagram showing one example of a computer 1000 that realizes the functions of the user terminal 10. The computer 1000 has a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each part of the computer 1000 is connected by a bus 1050.

[0227] The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400 and controls each part. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0228] The ROM 1300 stores a boot program, for example, a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and a program depending on the hardware of the computer 1000 .

[0229] The HDD 1400 is a computer-readable recording medium that non-temporarily records a program executed by the CPU 1100 and data used by the program. Specifically, the HDD 1400 is a recording medium for recording an information processing program according to the present disclosure, which is one example of the program data 1450 .

[0230] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device, or transmits data generated by the CPU 1100 to another device through the communication interface 1500.

[0231] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse through the input / output interface 1600. In addition, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer through the input / output interface 1600. In addition, the input / output interface 1600 can be used as a media interface for reading a program recorded on a predetermined recording medium, etc. The medium is, for example, an optical recording medium (e.g., a digital versatile disk (DVD) or a phase-change rewritable disk (PD)), a magneto-optical recording medium (e.g., a magneto-optical disk (MO)), a magnetic tape medium, a magnetic recording medium, or a semiconductor memory.

[0232] For example, in the case where the computer 1000 is used as the user terminal 10 according to the embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 16 and the like by executing the information processing program loaded on the RAM 1200. In addition, the HDD 1400 stores the information processing program and data according to the present disclosure in the storage unit 15. Note that although the CPU 1100 reads and executes the program data 1450 from the HDD 1400, as another example, these programs may be acquired from another device through the external network 1550.

[0233] Note that the present technology can also be configured in the following manner.

[0234] (1) An information processing device for controlling a first application and a second application, the second application being a plug-in for extending the function of the first application, wherein:

[0235] The first application includes:

[0236] a control unit that controls the operation of a second application in the first application, and

[0237] The second application comprises:

[0238] a selection unit that selects setting information for controlling a composition function based on machine learning, and

[0239] A transmission / reception unit that transmits setting information to an external server through a network and receives music data composed by the external server, wherein the external server performs a composition function based on machine learning.

[0240] (2) The information processing device according to (1) above, wherein:

[0241] The transmission / reception unit receives chords in a measure of a designated length, melody in a measure, and bass notes in a measure as music data.

[0242] (3) The information processing device according to (2) above, wherein:

[0243] The control unit individually sets instrument information for specifying sound quality when a chord, a melody, or a bass note included in the music data is played.

[0244] (4) The information processing device according to (3) above, wherein:

[0245] The control unit performs control to individually display each window, each window displaying information on a chord, a melody, or a bass note included in the music data.

[0246] (5) The information processing device according to (4) above, wherein:

[0247] The control unit controls transmission and reception of information between each window displaying information on a chord, a melody, or a bass note and a window displaying information on the first application according to a user's operation.

[0248] (6) The information processing device according to (4) or (5) above, wherein:

[0249] The control unit controls transmission and reception of information between windows each displaying information on a chord, a melody, or a bass note, according to a user's operation.

[0250] (7) The information processing device according to any one of (1) to (6) above, wherein:

[0251] The selection unit selects, as setting information, designation information for designating material music as learning data for machine learning, and

[0252] The transmission / reception unit transmits the specified information selected by the selection unit to the external server.

[0253] (8) The information processing device according to (7) above, wherein:

[0254] According to an operation of a user, the selection unit selects specified information, which is pre-stored in a storage unit and includes characteristic information indicating a characteristic of the specified information and a plurality of material music pieces associated with the characteristic information.

[0255] (9) The information processing device according to (7) or (8) above, wherein:

[0256] The selection unit selects combined designation information in which first designation information corresponding to some measures of music data composed by an external server and second designation information corresponding to some other measures thereof are combined.

[0257] (10) The information processing device according to (9) above, wherein:

[0258] When the transmission / reception unit receives music data composed by an external server based on the combined designation information, the transmission / reception unit stores the combined designation information in association with the music data in the storage unit.

[0259] (11) The information processing device according to any one of (7) to (10) above, wherein:

[0260] The selection unit selects, as setting information, length information of musical notes included in music data composed by an external server based on the designation information, and

[0261] The transmission / reception unit transmits the designation information and the note length information to the external server.

[0262] (12) The information processing device according to any one of (7) to (11) above, wherein:

[0263] The selection unit selects, as setting information, information for determining a probability that a composition note included in a chord appears in a melody of music data composed by an external server based on the designation information, and

[0264] The transmission / reception unit transmits, to an external server, designation information and information for determining a probability that a composition note included in a chord appears in a melody.

[0265] (13) The information processing device according to any one of (7) to (12) above, wherein:

[0266] The selection unit selects, as setting information, information for determining the type and amount of material music other than the material music included in the specified information in the music data composed by the external server based on the specified information, and

[0267] The transmission / reception unit transmits the designation information and information for determining the type and amount of material music other than the material music included in the designation information to the external server.

[0268] (14) The information processing device according to any one of (1) to (13) above, wherein:

[0269] The second application further includes a display control unit that performs control according to a user's operation to retrieve a history of past music data composed by an external server from the storage unit and display the history of the past music data.

[0270] (15) The information processing device according to (14) above, wherein:

[0271] The display control unit performs control to retrieve a history of editing operations performed on past music data composed by an external server from a storage unit and also displays the editing operations performed on the past music data.

[0272] (16) The information processing device according to any one of (1) to (15) above, wherein:

[0273] When a user performs a play or edit operation on music data composed by an external server after receiving the music data, the transmission / reception unit transmits information on the play or edit operation to the external server.

[0274] (17) The information processing device according to any one of (1) to (16) above, wherein:

[0275] The selection unit selects a chord progression in the composed music as setting information based on the user's operation, and

[0276] The transmission / reception unit transmits the chord progression selected by the selection unit to an external server.

[0277] (18) The information processing device according to any one of (1) to (17) above, wherein:

[0278] the control unit performs control to transmit information about the chord progression generated in the first application to the second application,

[0279] The selection unit selects information about the chord progression generated in the first application as setting information, and

[0280] The transmission / reception unit transmits information about a chord progression generated in a first application to an external server, and receives music data composed based on the information about the chord progression.

[0281] (19) An information processing method performed by an information processing device, the information processing device controlling a first application and a second application, the second application serving as a plug-in for extending a function of the first application, wherein:

[0282] The first application controls the operation of a second application in the first application, and

[0283] The second application selects setting information for controlling a composition function based on machine learning, and transmits the setting information to an external server through a network and receives music data composed by the external server, the external server performing the composition function based on machine learning.

[0284] (20) An information processing program that causes an information processing device that controls a first application and a second application to function as follows, the second application serving as a plug-in that extends the function of the first application,

[0285] The first application controls the operation of a second application in the first application, and

[0286] The second application selects setting information for controlling a composition function based on machine learning, and transmits the setting information to an external server through a network and receives music data composed by the external server, the external server performing the composition function based on machine learning.

[0287] Reference numerals list

[0288] 1 Information Processing System

[0289] 10 User Terminal

[0290] 11 Communication unit

[0291] 12 Input Units

[0292] 13 Display unit

[0293] 15 Storage Unit

[0294] 151 Composition setting information

[0295] 152 Composition music information

[0296] 153 Historical Information

[0297] 154 Related instrument information

[0298] 16 Control Unit

[0299] 161 Main application control unit

[0300] 162 Plug-in control unit

[0301] 163 playback units

[0302] 164 Display control unit

[0303] 165 Plug-in application control unit

[0304] 166 Select Unit

[0305] 167 Transmit / Receive Unit

[0306] 168 playback units

[0307] 169 Display control unit

[0308] 100 Processing Servers

[0309] 200 Management Server

Claims

1. An information processing device, the information processing device controlling a first application and a second application, the second application being used as a plug-in for extending the function of the first application, wherein: The first application includes: a control unit that controls the operation of the second application in the first application, and The second application comprises: a selection unit that selects setting information for controlling a composition function based on machine learning according to an operation of a user, wherein the setting information includes designation information for designating material music as learning data for machine learning, and a transmission / reception unit that transmits the setting information to an external server through a network and receives music data composed by the external server, the external server performing a predetermined learning process based on the specified information to generate a learned machine learning model and performing a composition function based on the learned machine learning model, wherein the control unit performs control to transmit information about the chord progression generated in the first application to the second application, the selection unit selects information about the chord progression generated in the first application as the setting information, and the transmission / reception unit transmits information about the chord progression generated in the first application to the external server, and receives music data composed based on the information about the chord progression, The final music is created on the first application by mixing the music data automatically composed by the external server with performance data performed by a performer.

2. The information processing device according to claim 1, wherein: The transmission / reception unit receives, as music data, chords in a measure of a specified length, melody in the measure, and bass notes in the measure.

3. The information processing device according to claim 2, wherein: The control unit individually sets instrument information for specifying a sound quality when the chord, the melody, or the bass note included in the music data is played.

4. The information processing device according to claim 3, wherein: The control unit performs control to individually display each window for displaying information on the chord, the melody, or the bass note included in the music data.

5. The information processing device according to claim 4, wherein: The control unit controls transmission and reception of information between each window displaying information about the chord, the melody, or the bass note and a window displaying information about the first application according to a user's operation.

6. The information processing device according to claim 4, wherein: The control unit controls transmission and reception of information between the windows each displaying information on the chord, the melody, or the bass note, according to a user's operation.

7. The information processing device according to claim 1, wherein: The specification information is pre-stored in a storage unit and includes characteristic information indicating a characteristic of the specification information and a plurality of pieces of material music associated with the characteristic information.

8. The information processing device according to claim 1, wherein: The selection unit selects combined designation information in which first designation information corresponding to some measures of the music data composed by the external server and second designation information corresponding to some other measures of the music data are combined.

9. The information processing device according to claim 8, wherein: When the transmission / reception unit receives the music data composed by the external server based on the combined designation information, the transmission / reception unit stores the combined designation information in a storage unit in association with the music data.

10. The information processing device according to claim 1, wherein: The selection unit selects, as the setting information, length information of musical notes included in the music data composed by the external server based on the designation information, and The transmission / reception unit transmits the designation information and the length information of the musical note to the external server.

11. The information processing device according to claim 1, wherein: The selection unit selects, as the setting information, information for determining a probability that a composition note included in a chord appears in a melody of music data composed by the external server based on the designation information, and The transmission / reception unit transmits the designation information and information for determining the probability that a composition note included in a chord appears in a melody to the external server.

12. The information processing device according to claim 1, wherein: the selection unit selects, as the setting information, information for determining the type and amount of material music other than the material music in the designation information included in the music data composed by the external server based on the designation information, and The transmission / reception unit transmits the specification information and information for determining the type and amount of material music other than the material music included in the specification information to the external server.

13. The information processing device according to claim 1, wherein: The second application further includes a display control unit that performs control according to a user's operation to retrieve a history of past music data composed by the external server from a storage unit and display the history of the past music data.

14. The information processing device according to claim 13, wherein: The display control unit performs control to retrieve a history of editing operations performed on the past music data composed by the external server from the storage unit and also displays the editing operations performed on the past music data.

15. The information processing device according to claim 1, wherein: When a user performs a play or edit operation on the music data after receiving the music data composed by the external server, the transmission / reception unit transmits information about the play or edit operation to the external server.

16. The information processing device according to claim 1, wherein: The selection unit selects a chord progression in composed music as the setting information based on an operation of a user, and The transmission / reception unit transmits the chord progression selected by the selection unit to the external server.

17. An information processing method performed by an information processing device, the information processing device controlling a first application and a second application, the second application serving as a plug-in for extending the function of the first application, wherein: The first application controls the operation of the second application in the first application, and the second application selects setting information for controlling a composition function based on machine learning according to a user's operation, wherein the setting information includes designation information for designating material music as learning data for machine learning, transmits the setting information to an external server via a network and receives music data composed by the external server, the external server performs a predetermined learning process based on the designation information to generate a learned machine learning model and performs a composition function based on the learned machine learning model, wherein the first application performs control to transmit information about a chord progression generated in the first application to the second application, wherein the second application selects information about the chord progression generated in the first application as the setting information, and the second application transmits information about the chord progression generated in the first application to the external server, and receives music data composed based on the information about the chord progression, The final music is created on the first application by mixing the music data automatically composed by the external server with performance data performed by a performer.

18. A computer-readable storage medium storing an information processing program that causes an information processing device that controls a first application and a second application that serves as a plug-in that extends the function of the first application to function so that: The first application controls the operation of the second application in the first application, and The second application selects setting information for controlling a composition function based on machine learning according to a user operation, wherein: The setting information includes designation information that designates material music as learning data for machine learning, and the setting information is transmitted to an external server through a network and music data composed by the external server is received, the external server performs a predetermined learning process based on the designation information to generate a learned machine learning model and performs a composition function based on the learned machine learning model, wherein the first application performs control to transmit information about a chord progression generated in the first application to the second application, wherein the second application selects information about the chord progression generated in the first application as the setting information, and the second application transmits information about the chord progression generated in the first application to the external server, and receives music data composed based on the information about the chord progression, The final music is created on the first application by mixing the music data automatically composed by the external server with performance data performed by a performer.

Images