Information processing method

Abstract

An information processing system (100) includes: an information acquisition unit (22) that acquires performance information (X) indicating a first part corresponding to one or more musical instruments in a musical piece; and an information generation unit (23) that generates performance information (Y) indicating a second part corresponding to a musical instrument different from the one or more musical instruments in the musical piece by processing control data (C) including the performance information (X) through a generation model (G) trained by machine learning.

Description

Technical Field

[0001] This disclosure relates to techniques for generating information representing musical pieces. Background Technology

[0002] Techniques for performing various processing on musical pieces composed of multiple parts have been proposed in the past. For example, Patent Document 1 discloses a technique that generates a learning-completed model by utilizing machine learning of features that affect the performance of instruments, for selecting the part to be played by a specified instrument from the multiple parts included in the musical piece data.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-159146 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] When the parts corresponding to specific instruments in a piece of music have already been created, creating the other parts requires musical expertise and a great deal of labor from the composer. In view of the above, one aspect of this disclosure aims to reduce the burden of creating parts in a piece that are not part of the existing parts.

[0008] means for solving problems

[0009] To address the aforementioned issues, one aspect of this disclosure involves an information processing method that obtains first performance information representing a first voice part corresponding to one or more instruments in a musical piece, and processes control data including the first performance information through a generative model trained by machine learning, thereby generating second performance information representing a second voice part corresponding to an instrument different from the one or more instruments in the musical piece.

[0010] One aspect of the information processing system disclosed herein includes: an information acquisition unit that acquires first performance information representing a first voice part corresponding to one or more instruments in a musical piece; and an information generation unit that processes control data including the first performance information through a generative model trained by machine learning, thereby generating second performance information representing a second voice part corresponding to an instrument different from the one or more instruments in the musical piece.

[0011] One aspect of this disclosure relates to a program that enables a computer system to function as: an information acquisition unit that acquires first performance information representing a first voice part corresponding to one or more instruments in a musical piece; and an information generation unit that processes control data including the first performance information through a generative model trained by machine learning, thereby generating second performance information representing a second voice part corresponding to an instrument different from the one or more instruments in the musical piece. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating the structure of the information processing system in the first embodiment.

[0013] Figure 2 This is an explanatory diagram related to the arrangement of an object-oriented musical piece based on an information processing system.

[0014] Figure 3 This is a block diagram illustrating the functional structure of an information processing system.

[0015] Figure 4 This is a diagram illustrating performance information.

[0016] Figure 5 This is an explanatory diagram of the tokens that constitute performance information.

[0017] Figure 6 This is a diagram illustrating the control data.

[0018] Figure 7 This is a diagram illustrating the performance information generated from control data.

[0019] Figure 8 This is a flowchart of the music arrangement process.

[0020] Figure 9 This is a block diagram illustrating the structure of the machine learning system in the second embodiment.

[0021] Figure 10 It is a block diagram illustrating the functional structure of a machine learning system.

[0022] Figure 11 This is a flowchart of the training data generation and processing.

[0023] Figure 12 This is a flowchart of the training process.

[0024] Figure 13 This is an explanatory diagram related to the arrangement of the musical piece in the third embodiment.

[0025] Figure 14 This is a block diagram illustrating the functional structure of the information processing system in the third embodiment.

[0026] Figure 15 This is a block diagram illustrating the functional structure of the information processing system in the fourth embodiment.

[0027] Figure 16 This is an explanatory diagram related to the arrangement of the musical piece in the variation example.

[0028] Figure 17 This is a block diagram of the functional structure of the information processing system in the variant example. Detailed Implementation

[0029] A: First Implementation Method

[0030] Figure 1 This is a block diagram illustrating the structure of the information processing system 100 in the first embodiment. The information processing system 100 is a computer system for arranging existing musical pieces (hereinafter referred to as "object musical pieces"), and is implemented, for example, by an information device such as a smartphone, tablet terminal, or personal computer.

[0031] Figure 2 This is an explanatory diagram relating to the processing of the information processing system 100. The information processing system 100 generates music data Dy from music data Dx. Music data Dx is temporal data representing the multiple first voices constituting the object music. Specifically, music data Dx represents the temporal sequence of notes in each of the multiple first voices. Each of the multiple first voices is an existing performance voice corresponding to a different type of instrument. Furthermore, a "performance voice" is a part (voice) constituting the music, for example, meaning a group of more than one performer (or a group of more than one instrument) playing a common note. For example, performance voices may be distinguished by each instrument, each range, or the role of each melody (e.g., main melody / subordinate melody).

[0032] The music data Dy represents the timing data of an object piece of music in which a new second voice has been added to multiple first voices. The second voice is the performance voice corresponding to an instrument of a different type than the multiple first voices, and the object piece of music is formed by playing it in parallel with the multiple first voices. Therefore, the second voice is musically harmonious with the multiple first voices. As described above, the information processing system 100 arranges an existing object piece of music composed of multiple first voices into an object piece of music with a new second voice added.

[0033] The musical data Dy represents the temporal sequence of the notes in the first and second parts of the musical piece. Specifically, the musical data Dy includes existing musical data Dx representing multiple first parts and new musical data Da representing the second parts. That is, the information processing system 100 generates the musical data Da for the second parts from the existing musical data Dx. The musical data Da represents the temporal sequence of the notes constituting the second parts.

[0034] Music data Dx and music data Dy specify the pitch and duration of each note constituting each of multiple performance parts. The pitch is any one of a discretely defined scale (e.g., pitch number). The duration is specified, for example, by the start and duration (or end) of the note. Music data Dx and music data Dy are, for example, music files conforming to the MIDI (Musical Instrument Digital Interface) standard. Furthermore, music data Dx is an example of "first music data," and music data Da or music data Dy is an example of "second music data."

[0035] like Figure 1 As illustrated, the information processing system 100 includes a control device 11, a storage device 12, a communication device 13, an operation device 14, a sound source device 15, and a sound playback device 16. The information processing system 100 can be implemented as a single device or as multiple devices that are independently configured.

[0036] The control device 11 is composed of one or more processors from various elements of the control information processing system 100. For example, the control device 11 may be composed of one or more processors such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit). The communication device 13 communicates with external devices via a communication network such as the Internet.

[0037] Storage device 12 is one or more memory devices that store programs executed by storage control device 11 and various data used by control device 11. For example, storage device 12 stores music data Dx of an object piece of music. Storage device 12 is constructed from known recording media such as magnetic recording media or semiconductor recording media. Storage device 12 may also be constructed from a combination of various recording media. In addition, portable recording media that can be detached from information processing system 100, or recording media that can be written to or read from by control device 11 via communication network (e.g., cloud storage) may also be used as storage device 12.

[0038] The operating device 14 is an input device that receives instructions from the user. The operating device 14 may be, for example, a user-operated control or a touchscreen that detects user contact. By operating the operating device 14, the user can specify the type of instrument corresponding to the second voice part. For example, the user can select the desired type of instrument from a pre-prepared pool of candidates by operating the operating device 14. Alternatively, the operating device 14, independent of the information processing system 100, can also be connected to the information processing system 100 via wired or wireless means.

[0039] The sound source device 15 generates an audio signal representing the musical waveform specified by the music data Dx or music data Dy. Alternatively, the function of the sound source device 15 can also be implemented by executing a program through the control device 11.

[0040] The sound playback device 16 reproduces sound waves under the control of the control device 11. The sound playback device 16 is an output device such as a speaker or headphones. Specifically, the sound playback device 16 reproduces the musical tones of the target music represented by the audio signal generated by the online sound source device 15. In addition, the sound source device 15 or the sound playback device 16, which are independent of the information processing system 100, can also be connected to the information processing system 100 via wired or wireless means.

[0041] Figure 3 This is a block diagram illustrating the functional structure of the information processing system 100. The control device 11 executes a program stored in the storage device 12 to perform multiple functions (preprocessing unit 21, information acquisition unit 22, information generation unit 23, and postprocessing unit 24) for generating arranged music data Dy from existing music data Dx of an existing music piece.

[0042] The preprocessing unit 21 generates performance information X from the music data Dx. Performance information X is timing data representing multiple first voices of the target music. Performance information X can also be represented as intermediate or alternative data in a different format than the music data Dx. Furthermore, performance information X is an example of "first performance information."

[0043] Figure 4 This is a schematic diagram of performance information X. Performance information X represents the timing of multiple markers T for each of the first voices in the musical piece. Marker T is the unit that constitutes performance information X. Performance information X may be described, for example, by a series of strings. Figure 4 In this context, the performance information X is represented as a string spanning multiple lines. Each token T is separated by whitespace (e.g., a half-width space).

[0044] Figure 5 This is an explanatory diagram of the symbol T. The multiple symbols T that constitute the performance information X include the note symbol Ta, the time signature symbol Tb, and the auxiliary symbol Tc.

[0045] A note marking Ta is a notation T that represents a note in a musical piece. Specifically, note marking Ta is divided into performance marking Ta1 and duration marking Ta2. Performance marking Ta1 specifies the instrument used to play the note and the pitch of the note (e.g., pitch number). For example, performance marking Ta1 marked "acg_47" indicates a note of pitch 47 that should be played by an acoustic guitar (acg: acousticguitar). Similarly, performance marking Ta1 marked "acp_65" indicates a note of pitch 65 that should be played by an acoustic piano (acp: acoustic piano). Performance marking Ta1 can also represent identification information specifying the instrument type.

[0046] The duration marker Ta2 specifies the duration (length of a note). For example, a duration marker Ta2 marked "len_L" (where L is a natural number) represents a duration equivalent to L units of time. A unit of time corresponds to the duration of a beat interval in the musical piece. Specifically, a unit of time is, for example, equivalent to 1 / 12 of a beat in the musical piece. By playing the combination of the Ta1 and Ta2 markers, a note specifying the instrument, pitch, and duration is represented.

[0047] The beat marker Tb is a marker T that represents the beat of the musical piece. Specifically, a designated moment within the musical piece is represented by the beat marker Tb. The beat marker Tb is further divided into measure marker Tb1 (bar), beat marker Tb2 (beat), and position marker Tb3 (pos). The beat marker Tb representing a designated moment in the musical piece is placed in the sequence of marker T in the performance information X at the position corresponding to that moment.

[0048] The measure marker Tb1 signifies the bar line (the first beat of each measure) of the musical piece. The beat marker Tb2 signifies each beat of the musical piece. The position marker Tb3 signifies a specific moment in the musical piece. For example, a position marker Tb3 marked "pos_K" (K=1~11) signifies a moment after K units of time have elapsed since the bar line (measure marker Tb1) or beat (beat marker Tb2) immediately preceding that position marker Tb3.

[0049] The position marker Tb3 is used, for example, to indicate the position of a note. Specifically, the position of a note is indicated by the position marker Tb3 placed immediately before the note marker Ta (performance marker Ta1 and duration marker Ta2) representing that note. For example, a marker like "pos_9 acp_65 len_27" in performance information X means that a note with a pitch of 65 and a duration of 27 begins to be played by the acoustic piano (acp) at a moment (pos_9) after nine times the duration of the preceding beat. Additionally, the position marker Tb3 is omitted when the starting point of the note coincides with a bar line (measure marker Tb1) or a beat point (beat marker Tb2). As shown in the example above, the conditions for the articulation of a note (pitch, duration, and position) are specified through the performance marker Ta1, duration marker Ta2, and position marker Tb3.

[0050] The auxiliary marker Tc is a marker T that represents various information related to the performance of the subject music piece. Specifically, the auxiliary marker Tc is divided into section marker Tc1 (section), key marker Tc2 (key), and tempo marker Tc3 (tempo). The section marker Tc1 is a marker T that indicates the starting point of each structural section of the subject music piece. A structural section is a section on the timeline that distinguishes the subject music piece according to its musical meaning. Examples of structural sections include the intro, verse A, verse B (bridge), chorus, and outro. The section marker Tc1 is placed in the performance information X at the position corresponding to the starting point of the structural section of the subject music piece.

[0051] The key marker Tc2 is the marker T representing the key of the musical piece. The value specified by the key marker Tc2 is the key number used to identify the key. The key marker Tc2 is positioned in the performance information X corresponding to the moment when the key of the musical piece changes. The tempo marker Tc3 is the marker T representing the tempo of the musical piece. The value specified by the tempo marker Tc3 represents the tempo (BPM: Beats Per Minute). The tempo marker Tc3 is positioned in the performance information X corresponding to the moment when the tempo of the musical piece changes.

[0052] Specific examples of the markers T that constitute the performance information X are as described above. Figure 3 The preprocessing unit 21 converts the music data Dx into performance information X according to the rules in the example above. According to the first embodiment, the widely available existing music data Dx can be used to generate music data Dy (performance information Y described later).

[0053] The information acquisition unit 22 acquires performance information X. In the first embodiment, the information acquisition unit 22 generates control data C including performance information X and instrument information Q. Instrument information Q specifies the type of instrument corresponding to the second voice. Specifically, the type of instrument specified by the user through operation of the operating device 14 is specified by instrument information Q.

[0054] Figure 6 This is a schematic diagram of control data C. The information acquisition unit 22 generates control data C by appending instrument information Q to the beginning of the performance information X. The instrument information Q is appended to the performance information X as a tag T, for example. Furthermore, in the following description, as... Figure 6 The example illustrates the case where a drum kit, consisting of multiple percussion instruments, is designated as the second instrument. Furthermore, the position of the instrument information Q attached to the performance information X is arbitrary.

[0055] Figure 3 The information generation unit 23 generates performance information Y based on the control data C. Performance information Y represents the timing data of the second voice of the arranged musical piece. Figure 7 This is a schematic diagram of performance information Y. Performance information Y represents the timing of multiple markers T for the second voice of the object's musical piece. Similar to performance information X, performance information Y is described, for example, by a series of strings. Furthermore, performance information Y is an example of "second performance information".

[0056] The types, meanings, and references of the various markers T that constitute the performance information Y. Figure 5 The symbols T in the previously described performance information X are the same. That is, the multiple symbols T constituting the performance information Y are the same as... Figure 5 The examples are the same, including the note markings Ta (Ta1, Ta2), the beat markings Tb (Tb1, Tb2, Tb3), and the auxiliary markings Tc (Tc1, Tc2, Tc3).

[0057] In the first embodiment, as previously described, it is assumed that a drum kit consisting of multiple percussion instruments is designated as the instrument for the second voice part. Pitch and duration are not considered when playing the drum kit. Therefore, in Figure 7 In the example performance information Y, the performance marker Ta1 does not include a pitch specification, and the duration marker Ta2 is not used.

[0058] Additionally, the playing designation Ta1 on a drum kit specifies the type of percussion instrument that makes up the kit. For example, Ta1 marked "bass" indicates the bass drum, Ta1 marked "hhcls" indicates the hi-hat, and Ta1 marked "snare" indicates the snare drum. Furthermore, when an instrument other than the drum kit is designated as the second part, such as... Figure 5As shown in the example, a note is represented by a combination of playing the marking Ta1 and the duration marking Ta2.

[0059] When generating performance information Y, the information generation unit 23 uses a trained generative model G. The generative model G is a statistical model that has learned the relationship between control data C and performance information Y through pre-trained machine learning. The information generation unit 23 processes the control data C using the trained generative model G to generate performance information Y.

[0060] The generative model G is achieved by a program that causes the control device 11 to perform a calculation to generate performance information Y from control data C, and a combination of multiple variables (such as bias and weight values) applied to the calculation. The values ​​of the multiple variables are preset through machine learning.

[0061] For example, an encoder-decoder model, namely the Transformer, which includes a self-attention mechanism (specifically a multi-head attention mechanism), can be used as the generative model G. Regarding the Transformer, it was presented, for example, in Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, "Attention Is All You Need," 31st Conference on Neural Information Processing Systems (NIPS 2017). Alternatively, MEGA (Moving Average Equipped Gated Attention) can also be used in the generative model G. Regarding MEGA, it is described in Ma, C. Zhou, X. Kong, J. He, L. Gui, G. Neubig, J. May, and L. Zettlemoyer, "Mega: moving average equipped gated attention," arXiv:2209.10655, 2022.

[0062] The performance information X and performance information Y, which are composed of the timing of multiple markers T, are in a form suitable for processing based on the generative model G. As described above, according to the first embodiment, by utilizing the generative model G suitable for processing the timing of multiple markers T, it is possible to generate a valid second voice that musically matches the first voice.

[0063] Figure 3 The post-processing unit 24 generates music data Dy from the performance information Y. Specifically, the post-processing unit 24 generates music data Dy based on reference... Figure 5The previously described rules generate the second part's musical data Da from the performance information Y, and integrate the pre-arranged musical data Dx with the second part's musical data Da to generate musical data Dy. As described above, musical data Dy is generated from the performance information Y, thus enabling the generation of the second part's musical tone using an existing sound source device 15 capable of processing musical data Dy.

[0064] Figure 8 This is a flowchart of the process by which the control device 11 generates music data Dy from music data Dx (hereinafter referred to as "arrangement processing"). For example, the arrangement processing begins when the user operates the operating device 14.

[0065] After the arrangement process begins, the control device 11 (preprocessing unit 21) generates performance information X (Sa1) from the music data Dx. The control device 11 (information acquisition unit 22) generates instrument information Q (Sa2) specifying the instrument type of the second voice. The control device 11 (information acquisition unit 22) acquires the performance information X and generates control data C (Sa3) including the instrument information Q and the performance information X.

[0066] The control device 11 (information generation unit 23) processes the control data C by the generation model G to generate performance information Y (Sa4). The control device 11 (post-processing unit 24) generates the second part's musical data Da (Sa5) based on the performance information Y. The control device 11 (post-processing unit 24) integrates the musical data Dx with the musical data Da to generate the arranged musical data Dy (Sa6) of the target piece.

[0067] As explained above, in the first embodiment, performance information Y for a second voice corresponding to an instrument different from the first voice is generated from the performance information X representing the existing first voice of the target musical piece. Therefore, the burden on users creating a second voice other than the existing first voice for the target musical piece can be reduced. That is, according to the first embodiment, a unique customer experience can be provided to users, allowing them to obtain a new second voice corresponding to the existing first voice.

[0068] In the first embodiment, performance information Y can be generated specifically for the second voice corresponding to the type of instrument specified by the instrument information Q. The instrument information Q specifies the instrument selected by the user, thus enabling the generation of a second voice that matches the user's intention or preference. Furthermore, by changing the type of instrument specified by the instrument information Q, second voices for multiple instruments can be generated using a single generation model G.

[0069] B: Second Implementation Method

[0070] The second embodiment will be described below. Furthermore, for elements in the following examples that function the same as in the first embodiment, the same symbols as in the first embodiment will be used, and detailed descriptions will be omitted as appropriate.

[0071] Figure 9 This is a block diagram illustrating the structure of the machine learning system 200 in the second embodiment. The machine learning system 200 is a computer system that establishes the aforementioned generative model G used by the information processing system 100 through machine learning. The machine learning system 200 includes a control device 31, a storage device 32, and a communication device 33. In addition, the machine learning system 200 can be implemented as a single device or as multiple devices that are independently configured.

[0072] The control device 31 consists of one or more processors that control the various elements of the machine learning system 200. For example, the control device 31 consists of one or more processors such as CPU, GPU, SPU, DSP, FPGA or ASIC.

[0073] The communication device 33 communicates with external devices, for example, via a communication network such as the Internet. For instance, the communication device 33 communicates with the information processing system 100. The generative model G established by the machine learning system 200 is provided to the information processing system 100 through the communication device 33.

[0074] Storage device 32 is one or more memory devices that store programs executed by storage control device 31 and various data used by control device 31. Storage device 32 is constructed from known recording media such as magnetic recording media or semiconductor recording media. Storage device 32 may also be constructed from a combination of various recording media. In addition, portable recording media that can be detached from information processing system 100, or recording media that can be written to or read from by control device 31 via communication network (e.g., cloud storage) may also be used as storage device 32.

[0075] Storage device 32 stores multiple musical data R for machine learning to generate model G. Each musical data R represents the temporal data of a musical piece (hereinafter referred to as a "reference piece") used for machine learning. Each reference piece consists of H playing parts (H being a natural number greater than 2). The musical data R represents the temporal sequence of notes for each of the H playing parts. In addition, the musical data R specifies the type of instrument for each of the H playing parts. The musical data R is, for example, a music file conforming to the MIDI (Musical Instrument Digital Interface) standard. For example, a large amount of karaoke data produced in the past has been used as musical data R. Therefore, it is easy to prepare a large amount of musical data R. In addition, the number of playing parts H of the reference pieces is different for each reference piece.

[0076] Figure 10 This is a block diagram illustrating the functional structure of the machine learning system 200. The control device 31 implements multiple functions (training data generation unit 41, training processing unit 42) for building the generative model G by executing programs stored in the storage device 32.

[0077] The training data generation unit 41 generates multiple training data Z for machine learning of the generative model G. Each training data Z consists of a combination of training control data Ct and training performance information Yt. The control data Ct is data in the same format as the aforementioned control data C. The performance information Yt is data in the same format as the aforementioned performance information Y. The training data generation unit 41 generates multiple training data Z based on multiple pieces of music data R stored in the storage device 32. The training processing unit 42 establishes the generative model G using machine learning with the multiple training data Z.

[0078] Figure 11 This is a flowchart of the process by which the control device 31 (training data generation unit 41) generates multiple training data Z (hereinafter referred to as "training data generation process"). For example, the training data generation process is started based on an instruction from the administrator of the machine learning system 200. If the training data generation process starts, the control device 31 selects any one of the multiple music data R stored in the storage device 32 (hereinafter referred to as "selecting music data R") (Sb1).

[0079] Control device 31 selects any one of the H playing parts of the reference piece represented by selection music data R as the second part (Sb2), and generates music data Da (Sb3) for the second part based on the selected music data R. For example, the part of the selected music data R corresponding to the second part is extracted as music data Da. Control device 31 generates performance information Yt (Sb4) based on the music data Da of the second part. The process of generating performance information Yt based on music data Da is the same as the process of generating performance information X based on music data Dx by preprocessing unit 21. Control device 31 generates instrument information Q (Sb5) indicating the type of instrument specified for the second part in the selected music data R.

[0080] Control device 31 selects multiple first voices (Sb6) from the (H-1) voices excluding the second voice in the H voices of the reference music represented by the selected music data R. For example, (H-1) or fewer first voices are randomly selected from the (H-1) voices. Alternatively, all combinations of selecting a predetermined number of first voices from the (H-1) voices can be used sequentially. Control device 31 generates music data Dx (Sb7) representing the multiple first voices based on the selected music data R. For example, the portions of the selected music data R corresponding to the multiple first voices are extracted as music data Dx. Control device 31 generates performance information X (Sb8) based on the music data Dx. The generation of performance information X is the same as the processing by which the preprocessing unit 21 generates performance information X based on the music data Dx.

[0081] The control device 31 generates training control data Ct (Sb9) including performance information X corresponding to multiple first voices and instrument information Q for instruments specifying second voices. Then, the control device 31 correlates the control data Ct with the performance information Yt, thereby generating training data Z, and saves the training data Z to the storage device 32 (Sb10).

[0082] Control device 31 determines whether the prescribed termination condition (Sb11) has been met. The termination condition is, for example, that the number of the second part selected from the H parts of the reference music reaches a prescribed value.

[0083] If the termination condition is not met (Sb11: No), the control device 31 transfers the processing to step Sb2. That is, the control device 31 selects the unselected performance part from the H performance parts of the reference music represented by the selected music data R as the second voice (Sb2). As described above, the generation of performance information Yt (Sb2-Sb4), instrument information Q (Sb5), control data Ct (Sb6-Sb9), and training data Z (Sb10) is repeated until the termination condition is met (Sb11: Yes). That is, multiple training data Zs are generated from the music data R of the reference music to make the types of instruments in the second voice different.

[0084] If the termination condition is met (Sb11: Yes), the control device 31 determines whether the above-described (Sb2 to Sb10) processing (Sb12) has been performed on all the music data R stored in the storage device 32. If there is unprocessed music data R (Sb12: No), the control device 31 transfers the processing to step Sb1. That is, the control device 31 selects the unprocessed music data R as the new selected music data R (Sb1). As described above, for each of the multiple music data R, multiple training data Z are repeatedly generated (Sb2 to Sb10). On the other hand, if all music data R has been processed (Sb12: Yes), the control device 31 ends the training data generation process.

[0085] However, in the process of generating training data Z, the method of preparing performance information X and performance information Yt separately requires identifying and managing the temporal correspondence between performance information X and performance information Y. In the first embodiment, performance information X and performance information Yt are generated based on existing music data R, thus eliminating the need to identify and manage the temporal correspondence between performance information X and performance information Yt. Therefore, the burden required to generate training data Z can be reduced.

[0086] Figure 12 This is a flowchart of the process by which the control device 31 (training processing unit 42) builds a generative model G using machine learning with multiple training data Z (hereinafter referred to as "training processing"). After the training data generation processing is performed, the training processing begins, for example, upon instruction from the administrator of the machine learning system 200. The training processing is an example of a method for generating the generative model G.

[0087] If training begins, the control device 31 selects any one of the multiple training data Z stored in the storage device 32 (hereinafter referred to as "selecting training data Z") (Sc1). Figure 10 As illustrated in the example, control device 31 processes control data Ct selected from training data Z using an initial or provisional generative model G (hereinafter referred to as "provisional model G0"), thereby generating performance information Y (Sc2). Control device 31 calculates a loss function representing the error between the performance information Y generated by provisional model G0 and the performance information Yt selected from training data Z (Sc3). Control device 31 updates multiple variables of provisional model G0 to reduce (ideally minimize) the loss function (Sc4).

[0088] Control device 31 determines whether the specified termination condition has been met (Sc5). The termination condition is, for example, that the loss function is lower than a specified threshold, or that the change in the loss function is lower than a specified threshold. If the termination condition is not met (Sc5: No), control device 31 selects the unselected training data Z stored in storage device 32 as the new selected training data Z (Sc1). That is, the processing of multiple variables of the provisional model G0 is repeatedly updated (Sc2 to Sc4) until the termination condition is met (Sc5: Yes).

[0089] If the termination condition is met (Sc5: Yes), the control device 31 terminates the training process. The provisional model G0 at the moment the termination condition is met is determined as the trained generative model G.

[0090] As can be understood from the above explanation, the generative model G learns the potential relationship between control data Ct and performance information Yt in multiple training datasets Z. Therefore, based on the above relationship, the trained generative model G outputs statistically reasonable performance information Y for unknown control data C.

[0091] The trained generative model G is sent from communication device 33 to information processing system 100. Control device 11 of information processing system 100 receives generative model G via communication device 13 and stores it in storage device 12. The generative model G provided through the above steps is used for… Figure 8 Example of music arrangement.

[0092] C: Third Implementation Method

[0093] Figure 13 This is an explanatory diagram of the processing of the information processing system 100 according to the third embodiment. Similar to the first embodiment, the control device 11 generates performance information Y (hereinafter referred to as "performance information Y1") for the second voice part based on control data C including performance information X and instrument information Q. In the third embodiment, performance information Y2 for the third voice part is generated based on new performance information X, which includes performance information X for multiple first voice parts and performance information Y1 for the second voice part. The third voice part is a performance voice part corresponding to instruments of different types than the multiple first and second voice parts, and is used to construct the arranged object music piece by playing in parallel with the multiple first and second voice parts. Therefore, the third voice part is musically coordinated with each of the first and second voice parts. As explained above, in the third embodiment, by accumulating and repeatedly generating performance information Y based on performance information X, the performance voice parts constituting the object music piece are sequentially increased.

[0094] Figure 14This is a block diagram illustrating the functional structure of the information processing system 100 in the third embodiment. The structure and operation of the preprocessing unit 21 in the third embodiment are the same as in the first embodiment. In the third embodiment, the user specifies the type of instrument corresponding to the second and third voice parts by operating the operation device 14. That is, the user specifies two different instruments.

[0095] The information acquisition unit 22 first generates control data C, similar to the first embodiment, which includes performance information X representing multiple first voice parts and instrument information Q1 representing the type of instrument corresponding to the second voice part. The information generation unit 23 processes the control data C through the generation model G, thereby generating performance information Y1 for the second voice part.

[0096] After generating performance information Y1, the information acquisition unit 22 generates new performance information X, which includes the existing performance information X and performance information Y1. That is, it generates performance information X for an object piece of music consisting of multiple first and second voices. The information acquisition unit 22 generates control data C, which includes performance information X with an added second voice and instrument information Q2 indicating the type of instrument corresponding to the third voice. The information generation unit 23 processes the control data C through the generation model G, thereby generating performance information Y2 for the third voice. As described above, in the third embodiment, the information generation unit 23 processes the control data C, which includes performance information X and performance information Y1, through the generation model G, thereby generating performance information Y2 representing the third voice. Performance information Y2 is an example of "third performance information".

[0097] The post-processing unit 24 generates music data Da1 for the second part based on the performance information Y1, and generates music data Da2 for the third part based on the performance information Y2. The post-processing unit 24 integrates the music data Dx before arrangement, the music data Da1 for the second part, and the music data Da2 for the third part, thereby generating music data Dy.

[0098] The third embodiment achieves the same effect as the first embodiment. Furthermore, in the third embodiment, the control data C, including existing performance information X and generated performance information Y1, is processed by the generation model G to generate performance information Y2 representing the third voice. Therefore, it is possible to generate performance information Y2 for the third voice that musically matches both the first and second voices. That is, according to the third embodiment, it is possible to compose an object piece of music consisting of a large number of performance voices. According to the third embodiment, it is possible to provide users with a unique customer experience that allows them to obtain multiple new performance voices (the second and third voices) corresponding to the existing first voice.

[0099] Furthermore, the number of times the information generation unit 23 repeatedly generates the performance information Y is arbitrary. In the (n+1)th processing of the information generation unit 23 (where n is a natural number), performance information Yn+1 is generated from the control data C, which includes the performance information Yn generated in the previous nth processing and the performance information X used in the nth processing as new performance information X.

[0100] D: Fourth Implementation Method

[0101] Figure 15 This is a block diagram illustrating the functional structure of the information processing system 100 in the fourth embodiment. In addition to the same elements as in the first embodiment (preprocessing unit 21, information acquisition unit 22, information generation unit 23, postprocessing unit 24), the control device 11 in the fourth embodiment also functions as a music score generation unit 25.

[0102] The score generation unit 25 generates score data E based on the music data Dy. Score data E represents the score of the edited object music, including the first and second voices. For example, data in MusicXML or PDF format is used as score data E. Any known technique may be used in generating score data E. The score generation unit 25 displays the score represented by score data E on a display device (not shown).

[0103] The fourth embodiment achieves the same effect as the first embodiment. Furthermore, in the fourth embodiment, edited score data E of the target music, including the first and second voice parts, is generated, allowing the user to verify the content of the score of the arranged target music.

[0104] Furthermore, the structure of the third embodiment is also applicable to the fourth embodiment. That is, the musical score data E may include a third voice (and other voices) in addition to the first and second voices.

[0105] E: Variation Example

[0106] The following examples provide specific variations of the methods illustrated above. Alternatively, any two or more methods selected from the following examples can be appropriately combined, provided they do not contradict each other.

[0107] (1) In the above methods, control data C including performance information X and instrument information Q is exemplified, but control data C may also include other information. For example, condition information specifying musical conditions related to the second voice may also be included in control data C. Condition information may specify, for example, whether the constituent notes of the second voice are single notes or chords, or whether the entire range or a portion of the target music is generated for the second voice. In addition, the performance difficulty related to the second voice may also be included in the condition information. According to the above methods, various target music that satisfy the conditions specified by the condition information can be generated.

[0108] (2) In the above methods, the interval in the object music that becomes the unit for processing by the information processing system 100 is arbitrary. For example, the above arrangement processing can be performed sequentially or in parallel for each interval (hereinafter referred to as "unit interval") of multiple intervals that divide the object music on the time axis. Music data Dx and music data Dy represent the note sequence in the object music corresponding to a unit interval. Each unit interval is, for example, an interval with a duration equivalent to a specified number of measures (e.g., 4 to 8 measures) in the object music. Alternatively, music data Da covering the second voice part of the object music can be generated by setting the entire interval of the object music as one arrangement processing of the object.

[0109] (3) In the above methods, the music data Dy is generated by integrating the music data Da of the second voice into the music data Dx of the first voice, but the generation of music data Dy can also be omitted. For example, the music data Da of the second voice can be generated as the final result (musical data Dy). That is, the music data Dy is represented as data indicating the timing of the notes that constitute the second voice. The music data Dy can include voices other than the second voice (e.g., the first voice or the third voice), or it can consist only of the second voice.

[0110] In addition, the score represented by the score data E in the fourth embodiment may be a score that includes the first voice and the second voice (and then the third voice and other voices), a score consisting only of the second voice, or a score consisting of the second voice and the third voice (i.e., a new voice).

[0111] (4) Among the above methods, the method of integrating the music data Da corresponding to the performance information Y into the existing music data Dx is exemplified, but the method of generating the arranged music data Dy is not limited to the above example. For example, the post-processing unit 24 may also integrate the performance information X and the performance information Y, and generate the music data Dy from the integrated performance information.

[0112] (5) Among the above methods, the performance information X represents multiple first voices, but the number of first voices represented by performance information X (musical data Dx) can also be one. Performance information X is generally represented as data representing the first voice corresponding to more than one instrument.

[0113] Furthermore, while the above examples illustrate how performance information Y represents a single second voice, the number of second voices represented by performance information Y (musical data Da) can also be two or more. Performance information Y is generally represented as data indicating a second voice corresponding to one or more instruments.

[0114] (6) In the above methods, the control data C including instrument information Q is processed by a single generative model G to generate the performance information Y of the instrument corresponding to the instrument information Q. However, multiple generative models G corresponding to different instruments can be selectively used in the generation of performance information Y.

[0115] For example, in a large amount of training data Z generated through training data generation processing, multiple training data Zs of selected instrumental playing parts as the second part are applied to the training process, thereby establishing a generative model G corresponding to that instrument. Each instrument's generative model G generates performance information Y for the corresponding instrumental playing part.

[0116] The information generation unit 23 processes the control data C through the generation model G corresponding to the user-selected instrument among multiple generation models G corresponding to different instruments, thereby generating the performance information Y of that instrument. In the above method, the structure of inputting instrument information Q into the generation model G is omitted. That is, the control data C may not include instrument information Q.

[0117] Furthermore, based on the aforementioned methods of inputting instrument information Q into the generation model G, performance information Y for multiple instruments can be generated using a single generation model G. That is, it has the advantage of not requiring a separate generation model G for each instrument.

[0118] (7) In the third embodiment, the second and third parts of the object piece are generated by repeatedly generating performance information Y from performance information X, but the method for generating the object piece including the second and third parts is not limited to the examples described above. For example, such as Figure 16 As in the example, the control device 11 can also execute the processing of generating second voice performance information Y1 from performance information X and the processing of generating third voice performance information Y2 from performance information X sequentially or in parallel, and generate the object music by integrating performance information X (musical data Dx), performance information Y1 (musical data Da1) and performance information Y2 (musical data Da2).

[0119] (8) In the above methods, the Transformer is used as the generative model G, but the structure or type of the generative model G is arbitrary. For example, deep neural networks such as recurrent neural networks (RNN) or long short-term memory networks (LSTM) can also be used as generative models G. The generative model G can also be composed of a combination of various statistical models.

[0120] (9) In the above embodiments, the information processing system 100 and the machine learning system 200 are examples of independent systems, but the functions of the machine learning system 200 (training data generation unit 41 and training processing unit 42) can also be carried in the information processing system 100.

[0121] (10) The above methods illustrate how each performance part of the target musical piece corresponds to an instrument, but some or all of the multiple performance parts constituting the target musical piece can also be singing parts corresponding to singing voices. That is, "instrument" in the above methods can be replaced with "singer". In addition, "instrument" in the above methods can also be replaced with "sound source" that includes both the instrument and the singer. That is, each performance part of the target musical piece corresponds to a different type of sound source.

[0122] (11) The information processing system 100 may also be implemented by a server device that communicates with terminal devices such as smartphones, tablets, or personal computers. For example, the control device 11 receives music data Dx and instrument information Q sent from the terminal device via the communication device 13, and generates music data Dy using the music data Dx and instrument information Q. The control device 11 then sends the music data Dy to the terminal device via the communication device 13.

[0123] Furthermore, in a method where the preprocessing unit 21 for generating performance information X from music data Dx is mounted in a terminal device, such as... Figure 17 As in the example, the preprocessing unit 21 is omitted from the information processing system 100. That is, the information acquisition unit 22 receives the performance information X and instrument information Q sent from the terminal device via the communication device 13, and generates control data C including the performance information X and instrument information Q.

[0124] Furthermore, in the case where the post-processing unit 24, which generates music data Da based on performance information Y, is mounted in the terminal device, such as... Figure 17 As in the example, the post-processing unit 24 is omitted from the information processing system 100. That is, the information generation unit 23 sends the performance information Y generated by the generation model G from the communication device 13 to the terminal device. The terminal device generates music data Da (and thus music data Dy) based on the performance information Y received from the information processing system 100.

[0125] (12) As described above, the functions of the information processing system 100 illustrated above are achieved through cooperation between one or more processors constituting the control device 11 and the program stored in the storage device 12. The program disclosed herein can be provided and installed in a computer in the form of a computer-readable recording medium. The recording medium is, for example, a non-transitory recording medium, with optical recording media (optical discs) such as CD-ROMs being preferred examples, but also includes any known form of recording medium such as semiconductor recording media or magnetic recording media. In addition, non-transitory recording media includes any recording medium other than transient propagating signals, and volatile recording media are not excluded. Furthermore, in a structure where a distribution device distributes a program via a communication network, the storage medium storing the program in the distribution device is equivalent to the aforementioned non-transitory recording medium.

[0126] F: Postscript

[0127] From the above examples, for instance, the following structure can be grasped.

[0128] One aspect (Aspect 1) of this disclosure involves an information processing method that obtains first performance information representing a first voice part corresponding to one or more instruments in a musical piece, and processes control data including the first performance information through a generative model trained by machine learning to generate second performance information representing a second voice part corresponding to an instrument different from the one or more instruments in the musical piece. According to the above method, second performance information for a second voice part corresponding to an instrument different from the first voice part is generated based on the first performance information representing the existing first voice part of the musical piece. Therefore, the burden on users to create second voice parts other than the existing first voice part in a musical piece can be reduced.

[0129] "(First / Second) Performance Information" is data in any form representing the voices that make up a musical piece. For example, performance information is data representing the timing of notes corresponding to a specified voice. For example, performance information consists of the timing of multiple markings representing a specified voice in a musical piece. Each marking, for example, specifies the articulation conditions (e.g., pitch, position, and duration) of each note that makes up the musical piece.

[0130] A "generative model" is a statistical model of arbitrary structure that learns the relationship between control data and secondary performance information through machine learning. For example, a generative model is pre-trained using machine learning data that includes training control data and secondary performance information. That is, the trained generative model generates statistically reasonable secondary performance information for unknown control data based on the latent relationship between control data and secondary performance information in multiple training datasets.

[0131] In a specific example of Method 1 (Method 2), the control data includes instrument information specifying the type of instrument corresponding to the second voice. According to the above method, second performance information can be generated for the voice corresponding to the type of instrument specified by the instrument information. Therefore, for example, in the method where the instrument information specifies the type of instrument selected by the user, a second voice that conforms to the user's intention or preference can be generated. Furthermore, by changing the type of instrument specified in the instrument information, second voices for multiple instruments can be generated using a single generation model.

[0132] In a specific example of Method 1 or Method 2 (Method 3), after generating the second performance information, the control data including the first performance information and the second performance information is processed by the generation model, thereby generating third performance information representing a third voice corresponding to an instrument different from the instruments corresponding to the first and second voices. According to the above method, after generating the second performance information, the control data including the first and second performance information is processed by the generation model, thereby generating third performance information representing the third voice. Therefore, it is possible to generate third performance information for a third voice that musically matches both the first and second voices.

[0133] In a specific example (method 4) of any of methods 1 to 3, the first performance information is the timing of the markings representing the first voice, and the second performance information is the timing of the markings representing the second voice. In the above method, the first performance information, which is the timing of the markings representing the first voice, is processed by a generative model, thereby generating the second performance information, which is the timing of the markings representing the second voice. Therefore, by using a generative model suitable for processing the timing of multiple markings, it is possible to generate a valid second voice that musically matches the first voice.

[0134] In a specific example of Method 4 (Method 5), the first performance information is further generated from first musical data representing the timing of the notes constituting the first voice part, and second musical data representing the timing of the notes constituting the second voice part is generated from the second performance information. In the above method, the first performance information is generated based on the first musical data representing the timing of the notes constituting the first voice part. Therefore, existing musical data can be used to generate the second performance information. Furthermore, second musical data representing the timing of the notes constituting the second voice part is generated from the second performance information. Therefore, the musical tones of the second voice part can be generated from an existing sound source that generates musical tones based on the musical data.

[0135] "(First / Second) Musical Data" is, for example, MIDI format data that specifies the pitch and duration of each note that constitutes a musical piece. (First / Second) performance information can also be represented as intermediate or alternative data in a different form than the musical data.

[0136] In a specific example of Method 5 (Method 6), the second musical data represents the timing of the notes constituting the first and second voices. Furthermore, musical score data representing the score of a piece including the first and second voices is generated based on the second musical data. According to the above method, the musical score can be effectively utilized for edited pieces including the first and second voices. Specifically, the musical score data is used, for example, for displaying the score or for automatically playing the piece.

[0137] The information processing system disclosed herein (method 7) comprises: an information acquisition unit that acquires first performance information representing a first voice part corresponding to one or more instruments in a musical piece; and an information generation unit that processes control data including the first performance information through a generative model trained by machine learning, thereby generating second performance information representing a second voice part corresponding to an instrument different from the one or more instruments in the musical piece.

[0138] One aspect (aspect 8) of this disclosure involves a program that enables a computer system to function as: an information acquisition unit that acquires first performance information representing a first voice part corresponding to one or more instruments in a musical piece; and an information generation unit that processes control data including the first performance information through a generative model trained by machine learning, thereby generating second performance information representing a second voice part corresponding to an instrument different from the one or more instruments in the musical piece.

[0139] Explanation of reference numerals in the attached figures

[0140] 100···Information processing system, 200···Machine learning system, 11, 31···Control device, 12, 32···Storage device, 13, 33···Communication device, 14···Operating device, 15···Sound source device, 16···Sound playback device, 21···Preprocessing unit, 22···Information acquisition unit, 23···Information generation unit, 24···Postprocessing unit, 25···Music score generation unit, 41···Training data generation unit, 42···Training processing unit.

Claims

1. An information processing method, wherein a computer system performs the following processing: Retrieve the first performance information of the first voice part corresponding to more than one instrument in a piece of music. The control data, including the first performance information, is processed by a generative model trained by machine learning to generate second performance information representing the second voice part corresponding to an instrument in the music that is different from the one or more instruments.

2. The information processing method according to claim 1, wherein, The control data includes instrument information specifying the type of instrument corresponding to the second voice part.

3. The information processing method according to claim 1 or 2, wherein, After generating the second performance information, the control data including the first performance information and the second performance information is processed by the generation model to generate third performance information for a third instrument corresponding to an instrument different from the instruments corresponding to the first and second voices.

4. The information processing method according to any one of claims 1 to 3, wherein, The first performance information indicates the timing of the markings for the first voice part. The second performance information is the timing of the markings for the second voice part.

5. The information processing method according to claim 4, wherein, The first performance information is further generated from the first piece of music data representing the timing of the notes constituting the first voice part. Second musical data representing the timing of the notes constituting the second voice part is generated from the second performance information.

6. The information processing method according to claim 5, wherein, The second piece of musical data is data representing the timing of the notes that constitute the first and second voice parts. Further, based on the second musical data, musical score data representing the score of a piece of music including the first voice and the second voice is generated.

7. An information processing system, comprising: The information acquisition unit acquires first performance information representing the first voice part corresponding to one or more instruments in a musical piece; and The information generation unit processes the control data, including the first performance information, through a generation model trained by machine learning, thereby generating second performance information representing the second voice part corresponding to an instrument in the music that is different from the one or more instruments.

8. A program that enables a computer system to function as follows: The information acquisition unit acquires first performance information representing the first voice part corresponding to one or more instruments in a musical piece; and The information generation unit processes the control data, including the first performance information, through a generation model trained by machine learning, thereby generating second performance information representing the second voice part corresponding to an instrument in the music that is different from the one or more instruments.

Images