Acoustic processing method, acoustic processing system, electronic musical instrument, and program

Abstract

This acoustic processing system comprises: a first generation unit which generates singing data according to an acoustic signal representing a singing sound; and a second generation unit which generates acoustic data representing a musical instrument sound correlated with musical elements of the singing sound by inputting input data including the singing data to a trained model obtained by machine-learning the relationship between a singing sound for training and a musical instrument sound for training.

Description

[Technical field]

[0001] This disclosure relates to a method for producing musical instrument sounds. output This document relates to technology that [Background technology]

[0002] Various techniques for controlling the sound of singing sounds, musical instruments, etc. have been proposed in the past. For example, Patent Document 1 discloses a configuration in which a performance mode is specified according to a user's operation on a performance operator, and sound effects applied to singing sounds are controlled according to the performance mode. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-52970 Summary of the Invention [Problem to be solved by the invention]

[0004] By the way, we have a system that can create musical instruments that match the singing sounds produced by users. output The instrument sounds that match the vocal sounds are those whose musical elements, such as pitch, volume, timbre, or rhythm, change in conjunction with the vocal sounds. output In order to do so, a user is required to have specialized knowledge about music. In consideration of the above circumstances, one aspect of the present disclosure is to provide a method for generating instrument sounds correlated with musical elements of singing sounds without requiring specialized knowledge about music. output One of our aims is to [Means for solving the problem]

[0005] In order to solve the above problems, an audio processing method according to one aspect of the present disclosure provides a method for processing singing data corresponding to an audio signal representing a singing sound, the method comprising: acquisitionThen, input data including the singing data is input to a trained model that has learned the relationship between the training singing sound and the training musical instrument sound by machine learning, and audio data representing musical instrument sounds correlated with the musical elements of the singing sound is generated. output In addition, a sound processing method according to another aspect of the present disclosure includes: acquisition Input data including the singing data is input to a machine-learned trained model, and audio data representing instrument sounds correlated with musical elements of the singing sound is generated. output do.

[0006] According to one aspect of the present disclosure, there is provided a sound processing system for generating singing data corresponding to an audio signal representing a singing sound. acquisition and a trained model that learns the relationship between a training singing sound and a training musical instrument sound by machine learning, and input data including the singing data is input to the trained model to generate acoustic data representing musical instrument sounds correlated with musical elements of the singing sound. output and a second generation unit for generating the second signal.

[0007] According to one aspect of the present disclosure, there is provided an electronic musical instrument that generates singing data corresponding to an audio signal representing a singing sound. acquisition and a trained model that learns the relationship between a training singing sound and a training musical instrument sound by machine learning, and input data including the singing data is input to the trained model to generate acoustic data representing musical instrument sounds correlated with musical elements of the singing sound. output and a reproduction control unit that causes a sound emitting device to emit the musical instrument sounds represented by the acoustic data and the musical performance sounds of the musical piece.

[0008] A program according to an embodiment of the present disclosure includes: generating singing data corresponding to an audio signal representing a singing sound; acquisition a first generation unit that generates acoustic data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting the input data including the singing data into a trained model that has been trained by machine learning to learn a relationship between a training singing sound and a training musical instrument sound through machine learning, outputThe computer functions as a second generation unit. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example of the configuration of an electronic musical instrument according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of an electronic musical instrument. [Diagram 3] 10 is a flowchart illustrating a specific procedure of a control process. [Figure 4] FIG. 1 is a block diagram illustrating a configuration of a machine learning system. [Diagram 5] FIG. 11 is an explanatory diagram of a learning process. [Figure 6] 11 is a flowchart illustrating a specific procedure of a learning process. [Figure 7] FIG. 1 is a block diagram illustrating a specific configuration of a trained model. [Figure 8] 13 is a flowchart illustrating a specific procedure of a learning process according to another aspect. [Figure 9] FIG. 11 is an explanatory diagram of a first process. [Figure 10] FIG. 11 is a block diagram illustrating a portion of the functional configuration of an electronic musical instrument according to a second embodiment. [Figure 11] FIG. 13 is an explanatory diagram regarding the use of a musical instrument model in the third embodiment. [Figure 12] FIG. 13 is a block diagram illustrating a specific configuration of a trained model in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A: First embodiment 1 is a block diagram illustrating the configuration of an electronic musical instrument 100 according to a first embodiment. The electronic musical instrument 100 is an acoustic processing system that reproduces sounds corresponding to performance by a user U. The electronic musical instrument 100 includes a performance device 10, a control device 11, a storage device 12, an operation device 13, a sound pickup device 14, and a sound emission device 15. The electronic musical instrument 100 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0011] The performance device 10 is an input device that accepts a performance by a user U. For example, the performance device 10 has a keyboard on which a plurality of keys corresponding to different pitches are arranged. The user U can specify the time sequence of pitches corresponding to each key by sequentially operating desired keys on the performance device 10. In the first embodiment, the user U sings a desired piece of music while playing the music using the performance device 10. For example, the user U sings the melody part of the music and plays the accompaniment part of the music in parallel. However, it does not matter whether the part sung by the user U is the same as the part played by the performance device 10.

[0012] The control device 11 is composed of one or more processors that control each element of the electronic musical instrument 100. For example, the control device 11 is composed of one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0013] The storage device 12 is one or more memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. The storage device 12 may also be a portable recording medium that is detachably attached to the electronic musical instrument 100, or a recording medium (e.g., cloud storage) to which the control device 11 can write or read via a communication network such as the Internet.

[0014] The operation device 13 is an input device that accepts instructions from the user U. The operation device 13 is, for example, a plurality of operators operated by the user U, or a touch panel that detects contact by the user U. The user U can specify one of a plurality of types of musical instruments (hereinafter referred to as a "selected musical instrument") by operating the operation device 13. The types of musical instruments selected by the user U are classified into, for example, keyboard instruments (percussion instruments), bowed string instruments, plucked string instruments, brass instruments, woodwind instruments, electronic instruments, and the like. However, the user U may select various musical instruments included in the above-mentioned classifications. For example, the user U may select a desired musical instrument from a plurality of types of musical instruments including a piano classified as a keyboard instrument, a violin or cello classified as a bowed string instrument, a guitar or harp classified as a plucked string instrument, a trumpet, a horn or trombone classified as a brass instrument, an oboe or clarinet classified as a woodwind instrument, and a portable keyboard classified as an electronic instrument.

[0015] The sound collection device 14 is a microphone that collects surrounding sounds. The user U produces singing sounds of a song around the sound collection device 14. The sound collection device 14 collects the singing sounds of the user U and generates an audio signal (hereinafter referred to as a "singing signal") V that represents the waveform of the singing sounds. For convenience, an A / D converter that converts the singing signal V from analog to digital is omitted from the illustration. In the first embodiment, the sound collection device 14 is mounted on the electronic musical instrument 100, but the sound collection device 14, which is separate from the electronic musical instrument 100, may be connected to the electronic musical instrument 100 by wire or wirelessly. The control device 11 of the first embodiment generates a playback signal Z that represents sounds corresponding to the singing sounds of the user U.

[0016] The sound emitting device 15 emits the sound represented by the playback signal Z. For example, a speaker device, headphones, or earphones are used as the sound emitting device 15. For convenience, a D / A converter that converts the playback signal Z from digital to analog is omitted from the illustration. In the first embodiment, the sound emitting device 15 is mounted on the electronic musical instrument 100, but the sound emitting device 15, which is separate from the electronic musical instrument 100, may be connected to the electronic musical instrument 100 by wire or wirelessly.

[0017] 2 is a block diagram illustrating an example of the functional configuration of the electronic musical instrument 100. The control device 11 executes a program stored in the storage device 12 to realize a plurality of functions (instrument selection section 21, audio processing section 22, musical tone generation section 23, and playback control section 24) for generating a playback signal Z. The instrument selection section 21 receives an instruction of a selected instrument by the user U from the operation device 13, and generates instrument data D that specifies the selected instrument. In other words, the instrument data D is data that specifies one of a plurality of types of musical instruments.

[0018] The audio processing unit 22 generates an audio signal A from the singing signal V and the instrument data D. The audio signal A is a signal representing the waveform of an instrument sound corresponding to a selected instrument specified by the instrument data D. The instrument sound represented by the audio signal A correlates with the singing sound represented by the singing signal V. Specifically, an audio signal A is generated that represents the instrument sound of a selected instrument whose pitch changes in conjunction with the pitch of the singing sound. In other words, the pitch of the singing sound and the pitch of the instrument sound are substantially the same. The audio signal A is generated in parallel with the singing by the user U.

[0019] The musical sound generating unit 23 generates a musical sound signal B representing a waveform of a musical sound (hereinafter referred to as a "performance sound") according to a performance by the user U. That is, a musical sound signal B representing a performance sound of a pitch sequentially specified by the user U through an operation on the performance device 10 is generated. Note that the musical instrument of the performance sound represented by the musical sound signal B and the musical instrument specified by the musical instrument data D may be of the same type or different types. Furthermore, the musical sound signal B may be generated by a sound source circuit separate from the control device 11. A musical sound signal B previously stored in the storage device 12 may be used. That is, the musical sound generating unit 23 may be omitted.

[0020] The reproduction control unit 24 causes the sound emitting device 15 to emit sounds corresponding to the singing signal V, the audio signal A, and the musical tone signal B. Specifically, the reproduction control unit 24 generates a reproduction signal Z by combining the singing signal V, the audio signal A, and the musical tone signal B, and supplies the reproduction signal Z to the sound emitting device 15. The reproduction signal Z is generated, for example, by a weighted sum of the singing signal V, the audio signal A, and the musical tone signal B. The weighting value of each signal (V, A, B) is set, for example, according to an instruction from the user U to the operation device 13. As can be understood from the above explanation, the singing sound (singing signal V) of the user U, the musical instrument sound (audio signal A) of the selected instrument correlated with the singing sound, and the performance sound (musical tone signal B) by the user U are emitted in parallel from the sound emitting device 15. As described above, the performance sound is the musical instrument sound of the same kind or different kind of instrument as the instrument specified by the musical instrument data D.

[0021] As illustrated in FIG. 2, the acoustic processing unit 22 of the first embodiment includes a first generating unit 31 and a second generating unit 32. The first generating unit 31 generates singing data X from a singing signal V. The singing data X is data that represents the acoustic characteristics of the singing signal V. The singing data X will be described in detail later, but includes, for example, features such as the fundamental frequency of the singing sound. The singing data X is generated sequentially for each of a plurality of unit periods on the time axis. Each unit period has a predetermined length. Successive unit periods are consecutive on the time axis. Note that each unit period may partially overlap.

[0022] The second generation unit 32 in FIG. 2 generates audio data Y according to the singing data X and the instrument data D. The audio data Y is a time series of samples constituting a portion of the audio signal A within a unit period. That is, audio data Y representing the instrument sound of a selected instrument whose pitch changes in conjunction with the pitch of the singing sound is generated. The second generation unit 32 generates the audio data Y for each unit period in parallel with the progression of the singing sound. That is, the instrument sound correlated with the singing sound is reproduced in parallel with the singing sound. The time series of audio data Y over a plurality of unit periods corresponds to the audio signal A.

[0023] The second generation unit 32 uses the trained model M to generate the acoustic data Y. Specifically, the second generation unit 32 generates the acoustic data Y by inputting the input data C to the trained model M for each unit period. The trained model M is a statistical estimation model that learns the relationship between vocal sounds and musical instrument sounds (the relationship between the input data C and the acoustic data Y) by machine learning. The input data C for each unit period includes vocal data X for the unit period, musical instrument data D, and acoustic data Y output by the trained model M in the immediately preceding unit period.

[0024] The trained model M is composed of, for example, a deep neural network (DNN). For example, any type of neural network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN), is used as the trained model M. In addition, additional elements, such as a long short-term memory (LSTM), may be installed in the trained model M.

[0025] The trained model M is realized by a combination of a program that causes the control device 11 to execute a calculation to generate acoustic data Y from input data C, and a plurality of variables (specifically, weights and biases) that are applied to the calculation. The program and the plurality of variables that realize the trained model M are stored in the storage device 12. The numerical values ​​of each of the plurality of variables that define the trained model M are set in advance by machine learning.

[0026] 3 is a flow chart illustrating a specific procedure of a process (hereinafter referred to as "control process") Sa in which the control device 11 generates a playback signal Z. The control process Sa is started in response to an instruction from the user U via the operation device 13. The user U performs on the performance device 10 and sings on the sound collection device 14 in parallel with the control process Sa. The control device 11 generates a musical sound signal B corresponding to the performance by the user U in parallel with the control process Sa.

[0027] When the control process Sa is started, the instrument selection unit 21 generates instrument data D that specifies the selected instrument designated by the user U (Sa1). The first generation unit 31 generates singing data X by analyzing a portion of the singing signal V supplied from the sound collection device 14 within a unit period (Sa2). The second generation unit 32 inputs input data C to the trained model M (Sa3). The input data C includes the instrument data D, the singing data X, and the audio data Y of the immediately preceding unit period. The second generation unit 32 acquires the audio data Y output by the trained model M for the input data C (Sa4). That is, the second generation unit 32 generates the audio data Y according to the input data C using the trained model M. The playback control unit 24 generates a playback signal Z by synthesizing the audio signal A, the singing signal V, and the musical tone signal B represented by the audio data Y (Sa5). The playback signal Z is supplied to the sound emitting device 15, whereby the singing sound of the user U, the musical instrument sound that accompanies the singing sound, and the performance sound produced by the performance device 10 are reproduced in parallel from the sound emitting device 15.

[0028] The instrument selection unit 21 judges whether or not a change in the selected instrument is instructed by the user U (Sa6). If a change in the selected instrument is instructed (Sa6: YES), the instrument selection unit 21 generates instrument data D that specifies the changed instrument as a new selected instrument (Sa1). The same processes (Sa2-Sa5) are executed for the changed selected instrument. On the other hand, if a change in the selected instrument is not instructed (Sa6: NO), the control device 11 judges whether or not a predetermined end condition is satisfied (Sa7). For example, the end condition is satisfied when an end of the control process Sa is instructed by an operation on the operation device 13. If the end condition is not satisfied (Sa7: NO), the control device 11 shifts the process to step Sa2. That is, the generation of the singing data X (Sa2), the generation of the sound data Y using the trained model M (Sa3, Sa4), and the generation of the playback signal Z (Sa5) are repeated for each unit period. On the other hand, if the end condition is satisfied (Sa7: YES), the control device 11 ends the control process Sa.

[0029] As can be understood from the above description, in the first embodiment, by inputting input data C including singing data X corresponding to a singing signal V of a singing sound into a trained model M, sound data Y representing an instrument sound correlated with the singing sound is generated. Therefore, an instrument sound that matches the singing sound can be generated without the user U needing specialized knowledge about music.

[0030] The trained model M used by the electronic musical instrument 100 to generate the acoustic data Y is generated by a machine learning system 50 in Fig. 4. The machine learning system 50 is a server device capable of communicating with a communication device 17 via a communication network 200 such as the Internet. The communication device 17 is a terminal device such as a smartphone or a tablet terminal, and is connected to the electronic musical instrument 100 by wire or wirelessly. The electronic musical instrument 100 is capable of communicating with the machine learning system 50 via the communication device 17. Note that the electronic musical instrument 100 may be equipped with a function for communicating with the machine learning system 50.

[0031] The machine learning system 50 is realized by a computer system including a control device 51, a storage device 52, and a communication device 53. The machine learning system 50 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0032] The control device 51 is composed of one or more processors that control each element of the machine learning system 50. The control device 51 is composed of one or more types of processors such as a CPU, an SPU, a DSP, an FPGA, or an ASIC. The communication device 53 communicates with the communication device 17 via the communication network 200.

[0033] The storage device 52 is a single or multiple memories that store the programs executed by the control device 51 and various data used by the control device 51. The storage device 52 is configured, for example, of a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the machine learning system 50, or a recording medium (for example, cloud storage) to which the control device 51 can write or read via the communication network 200 may be used as the storage device 52.

[0034] 5 is a block diagram illustrating an example of a functional configuration of a machine learning system 50. A control device 51 executes a program stored in a storage device 52 to function as a plurality of elements (a training data acquisition unit 61, a learning processing unit 62, and a distribution processing unit 63) for establishing a trained model M by machine learning.

[0035] The learning processing unit 62 establishes a learned model M by supervised machine learning (learning process Sb) using multiple pieces of training data T. The training data acquisition unit 61 acquires multiple pieces of training data T. Specifically, the training data acquisition unit 61 acquires multiple pieces of training data T stored in the storage device 52 from the storage device 52. The distribution processing unit 63 distributes the learned model M established by the learning processing unit 62 to the electronic musical instrument 100.

[0036] Each of the multiple training data T is composed of a combination of singing data Xt, musical instrument data Dt, and acoustic data Yt. The singing data Xt is training singing data X. Specifically, the singing data Xt is data representing acoustic features within a unit period of singing sounds (hereinafter referred to as "training singing sounds") recorded in advance for machine learning of the trained model M. The musical instrument data Dt is data specifying any one of multiple types of musical instruments.

[0037] The sound data Yt of each training data T is correlated with the training singing sound represented by the singing data Xt of the training data T and represents a musical instrument sound (hereinafter referred to as a "training musical instrument sound") corresponding to the musical instrument designated by the musical instrument data Dt of the training data T. That is, the sound data Yt of each training data T corresponds to a correct answer value (label) for the singing data Xt and musical instrument data Dt of the training data T. The pitch of the training singing sound changes in conjunction with the pitch of the training singing sound. Specifically, the pitch of the training singing sound and the pitch of the training musical instrument sound are substantially the same.

[0038] The training instrument sounds clearly reflect the characteristics specific to the instrument. For example, in the training instrument sounds of an instrument whose pitch changes continuously, the pitch changes continuously, and in the training instrument sounds of an instrument whose pitch changes discretely, the pitch changes discretely. In addition, in the training instrument sounds of an instrument whose volume decreases monotonically from the time of playing, the volume decreases monotonically from the sounding point, and in the training instrument sounds of an instrument whose volume is steadily maintained, the volume is steadily maintained. As described above, the training instrument sounds reflecting the tendencies specific to each instrument are recorded in advance as the audio data Yt.

[0039] 6 is a flowchart illustrating a specific procedure of the learning process Sb in which the control device 51 establishes the trained model M. Before the execution of the control process Sa that actually uses the trained model M, the learning process Sb is started, for example, in response to an instruction from an administrator to the machine learning system 50. The learning process Sb is also expressed as a method of generating a trained model M by machine learning (trained model generation method).

[0040] When the learning process Sb is started, the training data acquisition unit 61 selects and acquires one of the multiple training data T (hereinafter referred to as "selected training data T") stored in the storage device 52 (Sb1). The learning processing unit 62 inputs input data Ct corresponding to the selected training data T to an initial or provisional trained model M (Sb2), and acquires audio data Y output by the trained model M in response to the input (Sb3). The input data Ct corresponding to the selected training data T includes the singing data Xt and instrument data Dt of the selected training data T, and the audio data Y generated by the trained model M in the immediately preceding process.

[0041] The learning processing unit 62 calculates a loss function that represents an error between the acoustic data Y acquired from the trained model M and the acoustic data Yt of the selected training data T (Sb4). Then, the learning processing unit 62 updates multiple variables of the trained model M so that the loss function is reduced (ideally minimized) as illustrated in Fig. 4 (Sb5). For example, the backpropagation method is used to update the multiple variables according to the loss function.

[0042] The learning processing unit 62 determines whether a predetermined termination condition is satisfied (Sb6). The termination condition is, for example, that the loss function falls below a predetermined threshold, or that the change amount of the loss function falls below a predetermined threshold. If the termination condition is not satisfied (Sb6: NO), the training data acquisition unit 61 selects unselected training data T as new selected training data T (Sb1). That is, the process of updating multiple variables of the trained model M (Sb2-Sb5) is repeated until the termination condition is satisfied (Sb6: YES). If the termination condition is satisfied (Sb6: YES), the learning processing unit 62 terminates the update of multiple variables (Sb2-Sb5). The multiple variables of the trained model M are fixed to the values ​​at the time of the end of the learning process Sb.

[0043] As can be understood from the above description, the trained model M outputs statistically valid acoustic data Y for unknown input data C, based on the latent relationship between input data Ct (training singing sounds) and acoustic data Yt (training musical instrument sounds) corresponding to a plurality of training data T. In other words, the trained model M is a model that has learned the relationship between the training singing sounds and the training musical instrument sounds by machine learning.

[0044] The distribution processing unit 63 distributes the trained model M established by the above procedure to the communication device 17 via the communication device 53 (Sb7). Specifically, the distribution processing unit 63 transmits a plurality of variables of the trained model M from the communication device 53 to the communication device 17. The communication device 17 transfers the trained model M received from the machine learning system 50 via the communication network 200 to the electronic musical instrument 100. The control device 11 of the electronic musical instrument 100 stores the trained model M received by the communication device 17 in the storage device 12. Specifically, a plurality of variables defining the trained model M are stored in the storage device 12. As described above, the acoustic processing unit 22 generates an acoustic signal A using the trained model M defined by the plurality of variables stored in the storage device 12. The trained model M may be stored in a recording medium included in the communication device 17. The acoustic processing unit 22 of the electronic musical instrument 100 generates an acoustic signal A using the trained model M stored in the communication device 17.

[0045] 7 is a block diagram illustrating a specific configuration of the trained model M in the first embodiment. The singing data X input to the trained model M includes a plurality of features Fx (Fx1 to Fx6) related to singing sounds. The plurality of features Fx include pitch Fx1, onset point Fx2, error Fx3, duration Fx4, intonation Fx5, and timbre change Fx6.

[0046] The pitch Fx1 is the fundamental frequency (pitch) of the singing sound within the unit period. The onset Fx2 is the time point on the time axis when the singing sound starts to be produced, and exists for each note or phoneme, for example. Specifically, the onset Fx2 corresponds to the beat point (i.e., the standard or exemplary beat point of the song) that is closest to the time point when each note of the singing sound starts to be produced among multiple beat points of the song. For example, the onset Fx2 is expressed as a time based on a predetermined time point such as the start point of the audio signal A or the start point of the unit period. Note that the onset Fx2 may be expressed by information (flag) indicating whether each unit period corresponds to the time point when the singing sound starts to be produced.

[0047] The error Fx3 means a time error regarding the time when each note of the singing sound starts to be pronounced. For example, the time difference of the time from the standard or model beat of the song corresponds to the error Fx3. The duration Fx4 is the time length during which the pronunciation of each note of the singing sound continues. For example, the duration Fx4 corresponding to one unit period is expressed by the time length during which the singing sound continues within the unit period. The intonation Fx5 is a time change in the volume or pitch of the singing sound. For example, the intonation Fx5 is expressed by a time series of the volume or pitch within the unit period, or a rate of change or fluctuation range of the volume or pitch within the unit period. The timbre change Fx6 is a time change regarding the frequency characteristics of the singing sound. For example, the timbre change Fx6 is expressed by a time series of indices such as the frequency spectrum of the singing sound or MFCC (Mel-Frequency Cepstrum Coefficients).

[0048] The singing data X includes first data P1 and second data P2. The first data P1 includes a pitch Fx1 and an onset point Fx2. The second data P2 includes features Fx (error Fx3, duration Fx4, intonation Fx5, and timbre change Fx6) different from those of the first data P1. The first data P1 is basic information that represents the musical content of the singing sound. On the other hand, the second data P2 is auxiliary or additional information that represents the musical expression of the singing sound (hereinafter referred to as "musical expression"). For example, the onset point Fx2 included in the first data P1 corresponds to a standard rhythm defined for a song, for example, on a musical score, and the error Fx3 included in the second data P2 corresponds to a rhythm fluctuation that the user U has reflected in the singing sound as a musical expression (a rhythm fluctuation added as a musical expression).

[0049] The trained model M of the first embodiment includes a first model M1 and a second model M2. As described above, each of the first model M1 and the second model M2 is configured with a deep neural network such as a recurrent neural network or a convolutional neural network. The first model M1 and the second model M2 may be either the same type or different types.

[0050] The first model M1 is a statistical estimation model that learns the relationship between the first intermediate data Q1 and the third data P3 by machine learning. That is, the first model M1 outputs the third data P3 in response to the input of the first intermediate data Q1. The second generation unit 32 generates the third data P3 by inputting the first intermediate data Q1 to the first model M1.

[0051] Specifically, the first model M1 is realized by a combination of a program that causes the control device 11 to execute a calculation to generate the third data P3 from the first intermediate data Q1, and a plurality of variables (specifically, weights and biases) that are applied to the calculation. The numerical values ​​of the plurality of variables that define the first model M1 are set by the above-mentioned learning process Sb.

[0052] The first intermediate data Q1 is input to the first model M1 for each unit period. The first intermediate data Q1 for each unit period includes the first data P1 in the singing data X for that unit period, the instrument data D, and the sound data Y output by the trained model M (the second model M2) in the immediately preceding unit period. The first intermediate data Q1 for each unit period may include the second data P2 in the singing data X for that unit period.

[0053] The third data P3 includes a pitch Fy1 and an onset point Fy2 of an instrument sound corresponding to an instrument specified by the instrument data D. The pitch Fy1 is the fundamental frequency (pitch) of the singing sound within a unit period. The onset point Fy2 is the point on the time axis at which the sound of the instrument sound starts to be produced. The pitch Fy1 of the instrument sound correlates with the pitch Fx1 of the singing sound, and the onset point Fy2 of the instrument sound correlates with the onset point Fx2 of the singing sound. Specifically, the pitch Fy1 of the instrument sound matches or is close to the pitch Fx1 of the singing sound, and the onset point Fy2 of the instrument sound matches or is close to the onset point Fx2 of the singing sound. However, the pitch Fy1 and onset point Fy2 of the instrument sound reflect characteristics unique to the instrument. For example, the pitch Fy1 changes along a trajectory specific to the instrument, and the sounding point Fy2 is a point in time according to the sounding characteristics specific to the instrument (a point in time that does not necessarily coincide with the sounding point Fx2 of the singing sound).

[0054] As can be understood from the above description, the first model M1 is also expressed as a trained model that has trained the relationship between the pitch Fx1 and the sound point Fx2 of the singing sound (the first data P1) and the pitch Fy1 and the sound point Fy2 of the musical instrument sound (the third data P3). Note that a form in which the first intermediate data Q1 includes the first data P1 and the second data P2 of the singing data X is also assumed.

[0055] The second model M2 is a statistical estimation model that learns the relationship between the second intermediate data Q2 and the acoustic data Y by machine learning. That is, the second model M2 outputs acoustic data Y in response to the input of the second intermediate data Q2. The second generation unit 32 generates the acoustic data Y by inputting the second intermediate data Q2 to the second model M2. The combination of the first intermediate data Q1 and the second intermediate data Q2 corresponds to the input data C in FIG. 2.

[0056] Specifically, the second model M2 is realized by a combination of a program that causes the control device 11 to execute a calculation to generate the acoustic data Y from the second intermediate data Q2, and a plurality of variables (specifically, weights and biases) that are applied to the calculation. The numerical values ​​of the plurality of variables that define the second model M2 are set by the above-mentioned learning process Sb.

[0057] The second intermediate data Q2 includes the second data P2 of the singing data X, the third data P3 generated by the first model M1, the instrument data D, and the audio data Y output by the trained model M (the second model M2) in the immediately preceding unit period. The audio data Y output by the second model M2 represents an instrument sound reflecting the musical expression represented by the second data P2. The instrument sound represented by the audio data Y is given a musical expression specific to the selected instrument specified by the instrument data D. That is, each feature Fx (error Fx3, duration Fx4, intonation Fx5, timbre change Fx6) included in the second data P2 is converted into a musical expression that can be realized by the selected instrument and then reflected in the audio data Y.

[0058] For example, if the selected instrument is a keyboard instrument such as a piano, musical expressions such as crescendo or decrescendo are imparted to the instrument sounds according to the intonation Fx5 of the vocal sounds. Also, if the selected instrument is a keyboard instrument, musical expressions such as legato, staccato, or sustain are imparted to the instrument sounds according to the duration Fx4 of the vocal sounds.

[0059] If the selected instrument is a bowed instrument such as a violin or cello, musical expressions such as vibrato or tremolo are imparted to the instrument sound according to the intonation Fx5 of the singing sound. Also, if the selected instrument is a bowed instrument, musical expressions such as spiccato are imparted to the instrument sound according to the duration Fx4 or timbre change Fx6 of the singing sound.

[0060] If the selected instrument is a plucked instrument such as a guitar or a harp, a musical expression such as choking is imparted to the instrument sound according to the intonation Fx5 of the singing sound. Also, if the selected instrument is a plucked instrument, a musical expression such as slapping is imparted to the instrument sound according to the duration Fx4 and timbre change Fx6 of the singing sound.

[0061] If the selected instrument is a brass instrument such as a trumpet, horn, or trombone, musical expressions such as vibrato or tremolo are imparted to the instrument sound according to the intonation Fx5 of the singing sound. If the selected instrument is a brass instrument, musical expressions such as tonguing are imparted to the instrument sound according to the duration Fx4 of the singing sound.

[0062] If the selected instrument is a woodwind instrument such as an oboe or clarinet, musical expressions such as vibrato or tremolo are imparted to the instrument sound according to the intonation Fx5 of the singing sound. If the selected instrument is a woodwind instrument, musical expressions such as tonguing are imparted to the instrument sound according to the duration Fx4 of the singing sound. Also, if the selected instrument is a woodwind instrument, musical expressions such as subtones or grow tones are imparted to the instrument sound according to the timbre change Fx6 of the singing sound.

[0063] As described above, in the first embodiment, an instrument sound corresponding to a selected instrument among a plurality of instruments specified by the instrument data D is generated. Therefore, it is possible to generate various kinds of instrument sounds that match the singing sound of the user U. In addition, since the singing data X includes a plurality of feature amounts Fx including the pitch Fx1 and the onset point Fx2 of the singing sound, it is possible to generate with high accuracy the acoustic data Y of an instrument sound appropriate for the pitch Fx1 and the onset point Fx2 of the singing sound.

[0064] In the first embodiment, the trained model M includes a first model M1 and a second model M2. As described above, the first model M1 outputs the third data P3 including the pitch Fy1 and the sounding point Fy2 of the musical instrument sound in response to the input of the first intermediate data Q1 including the pitch Fx1 and the sounding point Fx2 of the singing sound. The second model M2 outputs the sound data Y in response to the input of the second intermediate data Q2 including the second data P2 representing the musical expression of the singing sound and the third data P3 of the musical instrument sound. That is, the first model M1 that processes the basic information of the singing sound (the pitch Fx1 and the sounding point Fx2) and the second model M2 that processes the information corresponding to the musical expression of the singing sound (the error Fx3, the duration Fx4, the intonation Fx5, and the timbre change Fx6) are prepared separately. Therefore, the sound data Y that represents the musical instrument sound appropriate for the singing sound can be generated with high accuracy.

[0065] In the first embodiment, the first model M1 and the second model M2 of the trained model M are established collectively by the learning process Sb illustrated in FIG. 6. However, a form in which the first model M1 and the second model M2 are each established by individual machine learning is also assumed. For example, as illustrated in FIG. 8, the learning process Sb may include a first process Sc1 and a second process Sc2. The first process Sc1 is a process for establishing the first model M1 by machine learning. The second process Sc2 is a process for establishing the second model M2 by machine learning.

[0066] As illustrated in FIG. 9, the first process Sc1 uses a plurality of training data R. Each of the plurality of training data R is composed of a combination of input data r1 and output data r2. The input data r1 includes the first data P1 of the singing data Xt and the instrument data Dt. In the first process Sc1, the learning processing unit 62 calculates a loss function that represents an error between the third data P3 generated by the initial or provisional first model M1 from the input data r1 of each training data R and the output data r2 of the training data R, and updates a plurality of variables of the first model M1 so as to reduce the loss function. The above process is repeated for each of the plurality of training data R to establish the first model M1.

[0067] In the second process Sc2, the same process as the learning process Sb in Fig. 6 is executed. However, in the second process Sc2, the learning processing unit 62 updates multiple variables of the second model M2 while fixing multiple variables of the first model M1. As described above, according to the configuration in which the trained model M includes the first model M1 and the second model M2, there is an advantage that machine learning can be performed individually for each of the first model M1 and the second model M2. Note that multiple variables of the first model M1 may be updated in the second process Sc2.

[0068] B: Second embodiment A second embodiment will be described. Note that, for elements having the same functions as those in the first embodiment in each of the following exemplary aspects, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0069] FIG. 10 is a block diagram illustrating a part of the functional configuration of the electronic musical instrument 100 in the second embodiment. The trained model M in the second embodiment includes a plurality of musical instrument models N corresponding to different musical instruments. Each of the musical instrument models N corresponding to each musical instrument is a statistical estimation model that learns the relationship between the singing sound and the musical instrument sound of the musical instrument by machine learning. Specifically, the musical instrument model N of each musical instrument outputs audio data Y representing the musical instrument sound of the musical instrument in response to input data C. Note that the input data C in the second embodiment does not include musical instrument data D. That is, the input data C of each unit period includes the singing data X of the unit period and the audio data Y of the immediately preceding unit period.

[0070] The second generation unit 32 inputs the input data C to any one of the multiple instrument models N, thereby generating audio data Y representing the instrument sound of the instrument corresponding to the instrument model N. Specifically, the second generation unit 32 selects an instrument model N corresponding to a selected instrument designated by the instrument data D from the multiple instrument models N, and generates audio data Y by inputting the input data C to the selected instrument model N. Thus, audio data Y representing the instrument sound of the selected instrument designated by the user U is generated.

[0071] Each musical instrument model N is established by a learning process Sb similar to that of the first embodiment. However, the musical instrument data D is omitted from each training data T. Each musical instrument model N includes a first model M1 and a second model M2. The musical instrument data D is omitted from the first intermediate data Q1 and the second intermediate data Q2.

[0072] The second embodiment also achieves the same effects as the first embodiment. In the second embodiment, the sound data Y is generated by selectively using one of a plurality of musical instrument models N. Therefore, various kinds of musical instrument sounds that match the singing sounds can be generated.

[0073] C: Third embodiment In the third embodiment, as in the second embodiment, one of a plurality of musical instrument models N is selectively used. FIG. 11 is an explanatory diagram regarding the use of each musical instrument model N in the third embodiment. As in the example of FIG. 4, the electronic musical instrument 100 of the third embodiment communicates with a machine learning system 50 via a communication device 17 such as a smartphone or a tablet terminal. The machine learning system 50 holds a plurality of musical instrument models N generated by a learning process Sb. Specifically, a plurality of variables defining each musical instrument model N are stored in a storage device 52.

[0074] The instrument selection unit 21 of the electronic musical instrument 100 generates instrument data D that specifies a selected instrument, and transmits the instrument data D to the communication device 17. The communication device 17 transmits the instrument data D received from the electronic musical instrument 100 to the machine learning system 50. The machine learning system 50 selects an instrument model N corresponding to the selected instrument specified by the instrument data D received from the communication device 17 from among the multiple instrument models N, and transmits the instrument model N to the communication device 17. The communication device 17 receives the instrument model N transmitted from the machine learning system 50 and holds the instrument model N. The acoustic processing unit 22 of the electronic musical instrument 100 generates an acoustic signal A using the instrument model N held in the communication device 17. Note that the instrument model N may be transferred from the communication device 17 to the electronic musical instrument 100. When a specific instrument model N is held in the electronic musical instrument 100 or the communication device 17, further communication with the machine learning system 50 is not necessary.

[0075] The third embodiment also achieves the same effects as the first and second embodiments. Moreover, in the third embodiment, any one of the multiple musical instrument models N generated by the machine learning system 50 is selectively provided to the electronic musical instrument 100. Therefore, there is an advantage that the electronic musical instrument 100 or the communication device 17 does not need to hold all of the multiple musical instrument models N. As can be understood from the example of the third embodiment, it is not necessary for all of the trained models M (multiple musical instrument models N) generated by the machine learning system 50 to be provided to the electronic musical instrument 100 or the communication device 17. In other words, only a part of the trained models M generated by the machine learning system 50 that is used in the electronic musical instrument 100 may be provided to the electronic musical instrument 100.

[0076] D: Fourth embodiment FIG. 12 is a block diagram illustrating a specific configuration of the trained model M in the fourth embodiment. The sound data Y in the fourth embodiment includes a plurality of types of feature amounts Fy (Fy1 to Fy6) related to the musical instrument sound. The plurality of types of feature amounts Fy include a pitch Fy1, a sounding point Fy2, an error Fy3, a duration Fy4, an intonation Fy5, and a timbre change Fy6. The pitch Fy1 and the sounding point Fy2 are the same as those in the first embodiment. The error Fy3 means a time error related to the time when each note of the musical instrument sound starts to be sounded. The duration Fy4 is the time length during which the sounding of each note of the musical instrument sound continues. The intonation Fy5 is a time change in the volume or pitch of the musical instrument sound. The timbre change Fx6 is a time change related to the frequency characteristics of the musical instrument sound.

[0077] The sound data Y of the fourth embodiment includes third data P3 and fourth data P4. The third data P3 is basic information expressing the musical content of the instrument sound, and includes the pitch Fy1 and the onset point Fy2 as in the first embodiment. The fourth data P4 is auxiliary or additional information expressing the musical expression of the instrument sound, and includes features Fy (error Fy3, duration Fy4, intonation Fy5, and timbre change Fy6) different from the first data P1 and the third data P3.

[0078] In the fourth embodiment, similar to the first embodiment, the trained model M includes a first model M1 and a second model M2. Similarly to the first embodiment, the first model M1 is a statistical estimation model that learns the relationship between the first intermediate data Q1 and the third data P3 by machine learning. That is, the first model M1 outputs the third data P3 in response to the input of the first intermediate data Q1.

[0079] The second model M2 of the fourth embodiment is a statistical estimation model that learns the relationship between the second intermediate data Q2 and the fourth data P4 by machine learning. That is, the second model M2 outputs the fourth data P4 in response to the input of the second intermediate data Q2. The second generation unit 32 outputs the fourth data P4 by inputting the second intermediate data Q2 to the second model M2. Acoustic data Y including the third data P3 output by the first model M1 and the fourth data P4 output by the second model M2 is output from the learned model M.

[0080] The second generation unit 32 of the fourth embodiment generates an audio signal A from audio data Y output by the trained model M. That is, the second generation unit 32 generates an audio signal A representing musical instrument sounds of a plurality of types of features Fy in the audio data Y. Any known audio processing may be adopted to generate the audio signal A. Other operations and configurations are similar to those of the first embodiment.

[0081] The fourth embodiment also achieves the same effects as the first embodiment. As can be understood from the explanations of the first and fourth embodiments, the sound data Y is comprehensively expressed as data representing the instrument sound. That is, in addition to the data representing the waveform of the instrument sound (first embodiment), the data representing the feature amount Fy of the instrument sound (fourth embodiment) is also included in the concept of the sound data Y.

[0082] E: Variation Specific modified embodiments that can be added to each of the above-mentioned embodiments are illustrated below. A plurality of embodiments arbitrarily selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0083] (1) In each of the above-described embodiments, the acoustic data Y output by the trained model M is fed back to the input side (input data C), but the feedback of the acoustic data Y may be omitted. That is, a configuration in which the input data C (first intermediate data Q1, second intermediate data Q2) does not include the acoustic data Y is also possible.

[0084] (2) In each of the above-described embodiments, one of a plurality of musical instrument sounds is selectively generated, but a configuration in which sound data Y representing the musical instrument sound of one musical instrument is generated is also envisioned. That is, the musical instrument selection unit 21 and the musical instrument data D in each of the above-described embodiments may be omitted.

[0085] (3) In each of the above-mentioned embodiments, the musical sound signal B corresponding to the performance by the user U is synthesized into the audio signal A, but the function of the playback control unit 24 to synthesize the musical sound signal B into the audio signal A may be omitted. Therefore, the performance device 10 and the musical sound generation unit 23 may also be omitted. Also, in each of the above-mentioned embodiments, the singing signal V representing the singing sound is synthesized into the audio signal A, but the function of the playback control unit 24 to synthesize the singing signal V into the audio signal A may be omitted. As can be understood from the above explanation, it is sufficient for the playback control unit 24 to be an element that causes the sound emitting device 15 to emit the instrument sound represented by the audio signal A, and the synthesis of the musical sound signal B or the singing signal V with the audio signal A may be omitted.

[0086] (4) In each of the above-mentioned embodiments, the instrument selection unit 21 selects an instrument in response to an instruction from the user U, but the method by which the instrument selection unit 21 selects an instrument is not limited to the above examples. For example, the instrument selection unit 21 may randomly select one of a plurality of instruments. Also, the type of instrument selected by the instrument selection unit 21 may be changed sequentially in parallel with the progression of the singing sounds.

[0087] (5) In each of the above-mentioned embodiments, the sound data Y of the instrument sound whose pitch changes in the same way as the singing sound is generated, but the relationship between the singing sound and the instrument sound is not limited to the above examples. For example, sound data Y representing the instrument sound whose pitch has a predetermined relationship with the pitch of the singing sound may be generated. For example, sound data Y representing the instrument sound whose pitch has a predetermined pitch difference (for example, a perfect fifth) with respect to the pitch of the singing sound is generated. In other words, matching of the pitch between the singing sound and the instrument sound is not essential. Each of the above-mentioned embodiments may also be expressed as a form of generating sound data Y representing the instrument sound whose pitch has the same or similar relationship with the pitch of the singing sound. In addition, the sound processing unit 22 may generate sound data Y of the instrument sound whose volume changes in conjunction with the volume of the singing sound, or the sound data Y of the instrument sound whose tone color changes in conjunction with the tone color of the singing sound. In addition, the sound processing unit 22 may generate sound data Y of the instrument sound that is synchronized with the rhythm of the singing sound (the timing of each sound that constitutes the singing sound).

[0088] As can be understood from the above examples, the sound processing unit 22 is comprehensively expressed as an element that generates sound data Y representing an instrument sound correlated with a singing sound. Specifically, the sound processing unit 22 generates sound data Y representing an instrument sound correlated with a musical element of a singing sound (for example, an instrument sound whose musical element changes in conjunction with the musical element of a singing sound). A musical element is a musical factor related to the sound (a singing sound or an instrument sound). For example, pitch, volume, timbre, or rhythm, or a temporal change related to the above elements (for example, intonation, which is a temporal change in pitch or volume) are included in the concept of a musical element.

[0089] (6) In each of the above embodiments, the singing data X includes a plurality of features Fx extracted from the singing signal V, but the information included in the singing data X is not limited to the above examples. For example, the first generator 31 may generate a time series of samples constituting a portion of one unit period of the singing signal V as the singing data X. As can be understood from the above examples, the singing data X is comprehensively expressed as data corresponding to the singing signal V.

[0090] (7) In each of the above-described embodiments, the machine learning system 50 separate from the electronic musical instrument 100 establishes the trained model M, but the electronic musical instrument 100 may be equipped with a function for establishing the trained model M by a learning process Sb that uses multiple pieces of training data T. For example, the training data acquisition unit 61 and the learning processing unit 62 illustrated in FIG. 5 may be realized by the control device 11 of the electronic musical instrument 100.

[0091] (8) In each of the above-described embodiments, a deep neural network is exemplified as the trained model M, but the trained model M is not limited to a deep neural network. For example, a statistical estimation model such as a hidden Markov model (HMM) or a support vector machine (SVM) may be used as the trained model M. In addition, in each of the above-described embodiments, supervised machine learning using a plurality of training data T is exemplified as the learning process Sb, but the trained model M may be established by unsupervised machine learning that does not require training data T.

[0092] (9) In each of the above-mentioned embodiments, the trained model M that has learned the relationship between the singing sound and the musical instrument sound (the relationship between the input data C and the audio data Y) is used, but the configuration and processing for generating the audio data Y according to the input data C are not limited to the above examples. For example, the second generation unit 32 may generate the audio data Y by using a data table (hereinafter referred to as a "reference table") in which the correspondence between the input data C and the audio data Y is registered. The reference table is stored in the storage device 12. The second generation unit 32 searches the reference table for the input data C including the singing data X generated by the first generation unit 31 and the musical instrument data D generated by the musical instrument selection unit 21, and outputs the audio data Y corresponding to the input data C. The above-mentioned configuration also achieves the same effect as each of the above-mentioned embodiments. The configuration for generating the audio data Y by using the trained model M and the configuration for generating the audio data Y by using the reference table are collectively expressed as the configuration for generating the audio data Y by using the input data C including the singing data X.

[0093] (10) The computer system equipped with the sound processing unit 22 exemplified in each of the above-mentioned embodiments is collectively referred to as a sound processing system. The sound processing system that accepts a performance by the user U corresponds to the electronic musical instrument 100 exemplified in each of the above-mentioned embodiments. Note that the sound processing system may or may not include the performance device 10.

[0094] (11) The sound processing system may be realized by a server device that communicates with a terminal device such as a mobile phone or a smartphone. For example, the sound processing system generates sound data Y from a singing signal V and instrument data D received from the terminal device, and transmits the sound data Y (or sound signal A) to the terminal device.

[0095] (12) As described above, the functions exemplified in each of the above embodiments are realized by the cooperation of one or more processors constituting the control device 11 and the programs stored in the storage device 12. The above programs may be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, the recording medium that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0096] F: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0097] An audio processing method according to one aspect (aspect 1) of the present disclosure generates singing data according to an audio signal representing a singing sound, and inputs input data including the singing data into a trained model that has learned the relationship between a training singing sound and a training musical instrument sound by machine learning, thereby generating audio data representing musical instrument sounds correlated with musical elements of the singing sound. According to the above aspect, input data including singing data corresponding to the audio signal of the singing sound is input into the trained model, thereby generating audio data representing musical instrument sounds correlated with the singing sound. Therefore, musical instrument sounds that match the singing sound can be generated without the user needing specialized knowledge about music.

[0098] "Singing data" is any data corresponding to an audio signal representing a singing sound. For example, singing data may be data representing one or more features of a singing sound, or a time series of samples constituting an audio signal representing a waveform of a singing sound. On the other hand, audio data may be, for example, a time series of samples constituting an audio signal representing a waveform of a musical instrument sound, or data representing one or more features of a musical instrument sound.

[0099] An instrumental sound correlated with a singing sound is a sound played by an instrument suitable for being played in parallel with the singing sound. An instrumental sound correlated with a singing sound can also be said to be an instrumental sound that accompanies the singing sound. A typical example of an instrumental sound is an instrumental sound that represents a melody common to or similar to the singing sound. However, the instrumental sound may be an instrumental sound that represents a separate melody that is musically in harmony with the singing sound, or an instrumental sound that represents an accompaniment that supports the singing sound.

[0100] An acoustic processing method according to another aspect of the present disclosure generates singing data according to an acoustic signal representing a singing sound, and input data including the singing data is input to a machine-learned trained model to generate acoustic data representing an instrument sound correlated to a musical element of the singing sound. According to the above aspect, input data including singing data corresponding to the acoustic signal of the singing sound is input to the trained model to generate acoustic data representing an instrument sound correlated to the singing sound. Therefore, an instrument sound corresponding to the singing sound can be generated without requiring a user to have specialized knowledge about music.

[0101] In a specific example (Aspect 2) of Aspect 1, the sound data is generated in parallel with the progression of the singing sound. According to the above aspect, the sound data is generated in parallel with the progression of the singing sound. In other words, the instrument sound correlated with the singing sound can be reproduced in parallel with the singing sound.

[0102] In a specific example (Aspect 3) of Aspect 1 or Aspect 2, the audio data represents the instrument sound whose pitch changes in conjunction with the pitch of the singing sound. Also, in a specific example (Aspect 4) of Aspect 1 or Aspect 2, the audio data represents the instrument sound whose pitch has a predetermined pitch difference with respect to the pitch of the singing sound.

[0103] In a specific example (Aspect 5) of any one of Aspects 1 to 4, the input data includes acoustic data previously generated by the trained model. According to the above aspect, it is possible to generate suitable acoustic data by taking into account the relationship between adjacent acoustic data.

[0104] In a specific example (aspect 6) of any of aspects 1 to 5, the input data includes instrument data specifying one of a plurality of types of instruments, and the audio data represents the instrument sound corresponding to the instrument specified by the instrument data. In the above aspects, an instrument sound corresponding to the type of instrument specified by the instrument data among the plurality of types of instruments is generated, so that a variety of types of instrument sounds that match the singing sound can be generated. Note that the instrument specified by the instrument data is, for example, an instrument type selected by a user, or an instrument type estimated by analyzing an instrument sound produced by an instrument played by a user.

[0105] In a specific example (aspect 7) of aspect 6, an audio signal representing the singing sound, a signal composed of a time series of the audio data, and a signal representing an instrument sound corresponding to an instrument of a type different from the instrument specified by the instrument data are added together. According to the above aspect, it is possible to reproduce a variety of sounds including singing sound, an instrument sound correlating with a musical element of the singing sound, and an instrument sound of a type different from the instrument sound.

[0106] In a specific example (Aspect 8) of any one of Aspects 1 to 7, the singing data includes a plurality of types of feature amounts related to the singing sound, the plurality of types of feature amounts including the pitch and onset point of the singing sound. According to the above aspect, since the singing data includes a plurality of types of feature amounts including the pitch and onset point of the singing sound, it is possible to generate with high accuracy acoustic data of an instrument sound appropriate for the pitch and onset point of the singing sound. Note that the "onset point" of a singing sound is, for example, the timing at which the singing sound starts to be produced. For example, the "onset point" corresponds to the beat point closest to the time when the singing sound starts to be produced among a plurality of beat points according to the tempo of the singing sound.

[0107] In a specific example (aspect 9) of aspect 1, the singing data includes first data including the pitch and onset of the singing sound among a plurality of types of feature quantities related to the singing sound, and second data including a feature quantity of a different type from the feature quantity included in the first data among the plurality of types of feature quantities, and the trained model includes a first model that outputs third data including the pitch and onset of the musical instrument sound in response to input of first intermediate data including the first data, and a second model that outputs the acoustic data in response to input of second intermediate data including the second data and the third data. According to the above aspect, the trained model includes the first model and the second model. Therefore, acoustic data representing an appropriate musical instrument sound for the singing sound can be generated with high accuracy.

[0108] In a specific example (aspect 10) of aspect 1, the singing data includes first data including the pitch and onset of the singing sound among a plurality of types of feature quantities related to the singing sound, and second data including a feature quantity of a different type from the feature quantity included in the first data among the plurality of types of feature quantities, the trained model includes a first model that outputs third data including the pitch and onset of the musical instrument sound in response to input of first intermediate data including the first data, and a second model that outputs fourth data including a feature quantity of the musical instrument sound that is a different type from the feature quantity included in the first data in response to input of second intermediate data including the second data and the third data, and the acoustic data includes the third data and the fourth data. According to the above aspect, the trained model includes the first model and the second model. Therefore, acoustic data representing an appropriate musical instrument sound for the singing sound can be generated with high accuracy.

[0109] In a specific example (Aspect 11) of Aspect 9 or Aspect 10, the first intermediate data includes instrument data that specifies one of a plurality of instruments. In a specific example (Aspect 12) of Aspect 11, the second intermediate data includes the instrument data.

[0110] In a specific example (Aspect 13) of any one of Aspects 9 to 12, the first intermediate data includes previously generated acoustic data. Also, in a specific example (Aspect 14) of any one of Aspects 9 to 13, the second intermediate data includes previously generated acoustic data. According to Aspect 13 or Aspect 14, it is possible to generate suitable acoustic data by taking into account the relationship between adjacent acoustic data.

[0111] In a specific example (aspect 15) of any of aspects 8 to 14, the multiple types of features include one or more of the error in the onset point of the singing sound, the duration of the onset, the intonation of the singing sound, and the timbre change of the singing sound.

[0112] In a specific example (Aspect 16) of Aspect 1, the trained model includes a plurality of instrument models corresponding to different types of instruments, and in generating the audio data, the input data is input to one of the plurality of instrument models to generate the audio data representing the instrument sound of the instrument. According to the above aspect, audio data is generated by selectively using one of the plurality of instrument models, so that various types of instrument sounds that match the singing sound can be generated.

[0113] An audio processing system according to one embodiment (embodiment 17) of the present disclosure includes a first generation unit that generates singing data corresponding to an audio signal representing a singing sound, and a second generation unit that generates audio data representing an instrument sound correlated to a musical element of the singing sound by inputting input data including the singing data into a trained model that has learned the relationship between a training singing sound and a training instrument sound through machine learning.

[0114] An electronic musical instrument according to one embodiment (embodiment 18) of the present disclosure includes a first generating unit that generates singing data according to an audio signal representing a singing sound, a second generating unit that generates audio data representing an instrument sound correlated with a musical element of the singing sound by inputting input data including the singing data into a trained model that has learned the relationship between a training singing sound and a training instrument sound by machine learning, and a playback control unit that causes a sound emitting device to emit a performance sound of a musical piece and an instrument sound represented by the audio data. The "performance sound of a musical piece" is a performance sound represented by performance data prepared in advance, or a performance sound corresponding to a performance action by a user (e.g., a singer of the singing sound or another player). In addition to the performance sound and the instrument sound, the sound emitting device may emit the singing sound.

[0115] A program according to one embodiment (embodiment 19) of the present disclosure causes a computer to function as a first generation unit that generates singing data corresponding to an audio signal representing a singing sound, and a second generation unit that generates audio data representing an instrument sound correlated to a musical element of the singing sound by inputting input data including the singing data into a trained model that has learned the relationship between training singing sounds and training instrument sounds through machine learning. [Explanation of symbols]

[0116] 100...electronic musical instrument, 10...performance device, 11...control device, 12...storage device, 13...operation device, 14...sound collection device, 15...sound emission device, 17...communication device, 21...instrument selection unit, 22...acoustic processing unit, 23...musical sound generation unit, 24...playback control unit, 31...first generation unit, 32...second generation unit, M...trained model, M1...first model, M2...second model, 50...machine learning system, 51...control device, 52...storage device, 53...communication device, 61...training data acquisition unit, 62...learning processing unit, 63...distribution processing unit.

Claims

1. Singing data corresponding to an audio signal representing a singing sound is obtained; By inputting input data including the singing data into a trained model that has learned the relationship between the training singing sound and the training musical instrument sound by machine learning, audio data representing musical instrument sounds having pitches that are in a predetermined pitch difference relationship with the pitch of the singing sound is output. An audio processing method implemented by a computer system.

2. Singing data corresponding to an audio signal representing a singing sound is obtained; inputting input data including the singing data into a trained model that has learned the relationship between the training singing sound and the training musical instrument sound by machine learning, thereby outputting audio data representing musical instrument sounds correlated with musical elements of the singing sound; The input data includes acoustic data previously output by the trained model. An audio processing method implemented by a computer system.

3. Singing data corresponding to an audio signal representing a singing sound is obtained; inputting input data including the singing data into a trained model that has learned the relationship between the training singing sound and the training musical instrument sound by machine learning, thereby outputting audio data representing musical instrument sounds correlated with musical elements of the singing sound; the input data includes instrument data specifying one of a plurality of instruments; The acoustic data represents the musical instrument sound corresponding to the musical instrument specified by the musical instrument data. An audio processing method implemented by a computer system.

4. Singing data corresponding to an audio signal representing a singing sound is obtained; By inputting input data including the singing data into a trained model that has learned the relationship between the training singing sound and the training musical instrument sound by machine learning, audio data representing musical instrument sounds correlated with musical elements of the singing sound is output.

1. An audio processing method implemented by a computer system, comprising: The singing data is First data including a pitch and an onset point of the singing sound among a plurality of types of feature quantities related to the singing sound; second data including a feature amount of a type different from the feature amount included in the first data among the plurality of types of feature amounts; The trained model is a first model that outputs third data including a pitch and a sound point of the musical instrument sound in response to an input of first intermediate data including the first data; a second model that outputs the acoustic data in response to an input of second intermediate data including the second data and the third data. Acoustic processing methods.

5. Singing data corresponding to an audio signal representing a singing sound is obtained; By inputting input data including the singing data into a trained model that has learned the relationship between the training singing sound and the training musical instrument sound by machine learning, audio data representing musical instrument sounds correlated with musical elements of the singing sound is output.

1. An audio processing method implemented by a computer system, comprising: The singing data is First data including a pitch and an onset point of the singing sound among a plurality of types of feature quantities related to the singing sound; second data including a feature amount of a type different from the feature amount included in the first data among the plurality of types of feature amounts; The trained model is a first model that outputs third data including a pitch and a sound point of the musical instrument sound in response to an input of first intermediate data including the first data; a second model that outputs, in response to an input of second intermediate data including the second data and the third data, fourth data including a feature amount of the musical instrument sound that is a different type from the feature amount included in the first data, The acoustic data includes the third data and the fourth data. Acoustic processing methods.

6. Singing data corresponding to an audio signal representing a singing sound is obtained; By inputting input data including the singing data into a trained model that has learned the relationship between the training singing sound and the training musical instrument sound by machine learning, audio data representing musical instrument sounds correlated with musical elements of the singing sound is output.

1. An audio processing method implemented by a computer system, comprising: the trained model includes a plurality of instrument models corresponding to different types of instruments; In the output of the acoustic data, the input data is input to any one of the plurality of musical instrument models, and the acoustic data representing the musical instrument sound of the musical instrument is output. Acoustic processing methods.

7. A first generator that acquires singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds having a pitch that is in a predetermined pitch difference relationship with the pitch of the singing sound by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sound and the training musical instrument sound by machine learning; An audio processing system comprising:

8. A first generator that acquires singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; Equipped with The input data includes acoustic data previously output by the trained model. Sound processing system.

9. A first generator that acquires singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; Equipped with the input data includes instrument data specifying one of a plurality of instruments; The acoustic data represents the musical instrument sound corresponding to the musical instrument specified by the musical instrument data. Sound processing system.

10. A first generator that acquires singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; Equipped with The singing data is First data including a pitch and an onset point of the singing sound among a plurality of types of feature quantities related to the singing sound; second data including a feature amount of a type different from the feature amount included in the first data among the plurality of types of feature amounts; The trained model is a first model that outputs third data including a pitch and a sound point of the musical instrument sound in response to an input of first intermediate data including the first data; a second model that outputs the acoustic data in response to an input of second intermediate data including the second data and the third data. Sound processing system.

11. A first generator that acquires singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; Equipped with The singing data is First data including a pitch and an onset point of the singing sound among a plurality of types of feature quantities related to the singing sound; second data including a feature amount of a type different from the feature amount included in the first data among the plurality of types of feature amounts; The trained model is a first model that outputs third data including a pitch and a sound point of the musical instrument sound in response to an input of first intermediate data including the first data; a second model that outputs, in response to an input of second intermediate data including the second data and the third data, fourth data including a feature amount of the musical instrument sound that is a different type from the feature amount included in the first data, The acoustic data includes the third data and the fourth data. Sound processing system.

12. A first generator that acquires singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; Equipped with the trained model includes a plurality of instrument models corresponding to different types of instruments; In the output of the acoustic data, the input data is input to any one of the plurality of musical instrument models, and the acoustic data representing the musical instrument sound of the musical instrument is output. Sound processing system.

13. A first generator that acquires singing data corresponding to an audio signal representing a singing sound; A second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; a reproduction control unit that causes a sound emitting device to emit the musical instrument sounds represented by the sound data and the musical instrument sounds of the musical piece; An electronic musical instrument comprising:

14. a first generator for acquiring singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds having a pitch that is in a predetermined pitch difference relationship with the pitch of the singing sound by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sound and the training musical instrument sound by machine learning; A program that makes a computer function as a

15. a first generator for acquiring singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; A program for causing a computer to function as The input data includes acoustic data previously output by the trained model. program.

16. a first generator for acquiring singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; A program for causing a computer to function as the input data includes instrument data specifying one of a plurality of instruments; The acoustic data represents the musical instrument sound corresponding to the musical instrument specified by the musical instrument data. program.

17. a first generator for acquiring singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; A program for causing a computer to function as The singing data is First data including a pitch and an onset point of the singing sound among a plurality of types of feature quantities related to the singing sound; second data including a feature amount of a type different from the feature amount included in the first data among the plurality of types of feature amounts; The trained model is a first model that outputs third data including a pitch and a sound point of the musical instrument sound in response to an input of first intermediate data including the first data; a second model that outputs the acoustic data in response to an input of second intermediate data including the second data and the third data. program.

18. a first generator for acquiring singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; A program for causing a computer to function as The singing data is First data including a pitch and an onset point of the singing sound among a plurality of types of feature quantities related to the singing sound; second data including a feature amount of a type different from the feature amount included in the first data among the plurality of types of feature amounts; The trained model is a first model that outputs third data including a pitch and a sound point of the musical instrument sound in response to an input of first intermediate data including the first data; a second model that outputs, in response to an input of second intermediate data including the second data and the third data, fourth data including a feature amount of the musical instrument sound that is a different type from the feature amount included in the first data, The acoustic data includes the third data and the fourth data. program.

19. a first generator for acquiring singing data corresponding to an audio signal representing a singing sound; a second generation unit that outputs audio data representing musical instrument sounds correlated with musical elements of the singing sounds by inputting input data including the singing data into a trained model that has trained a relationship between the training singing sounds and the training musical instrument sounds by machine learning; A program for causing a computer to function as the trained model includes a plurality of instrument models corresponding to different types of instruments; In the output of the acoustic data, the input data is input to any one of the plurality of musical instrument models, and the acoustic data representing the musical instrument sound of the musical instrument is output. program.

Images