Sound generation method and device using machine learning model
The sound generation method and device address the delay issue in AR models by allowing forced instructions and real-time updates of acoustic features, enabling precise sound generation according to user control values.
Patent Information
- Application Number
- JP2023522703
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-05-18
- Filing Date
- 2022-05-18
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2042-05-18
AI Technical Summary
Existing autoregressive (AR) type generative models for sound generation experience delays in tracking user control values, making it difficult to generate sound signals according to user intention due to the delay in updating control values.
A sound generation method and device that accepts control values at multiple time points and allows for forced instructions at desired time points, using a trained model to process these values and update acoustic features in a temporary memory, either generating alternative features or updating existing ones based on the control values.
Enables the generation of sound signals that closely follow user intentions without significant delay by updating acoustic features to match control values in real-time, ensuring accurate sound production.
Smart Images

Figure 0007578192000001 
Figure 0007578192000002 
Figure 0007578192000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a sound generation method and a sound generation device capable of generating sound. [Background technology]
[0002] For example, AI (artificial intelligence) singers are known as sound sources that sing in the style of a specific singer. By learning the characteristics of the singing of a specific singer, the AI singer can generate any sound signal by imitating the singer. Here, it is preferable that the AI singer generates a sound signal that reflects not only the singing characteristics of the learned singer, but also the singing style instructions given by the user. [Non-Patent Document 1] Jesse Engel, Lamtharn Hantrakul, Chenjie Gu and Adam Roberts, "DDSP: Differentiable Digital Signal Processing", arXiv:2001.04643v1 [cs.LG] 14 Jan 2020 Summary of the Invention [Problem to be solved by the invention]
[0003] Non-Patent Document 1 describes a neural generative model that generates a sound signal based on a user's input sound. In this generative model, the user can specify control values such as pitch or volume to the generative model while the sound signal is being generated. When an AR (autoregressive) type generative model is used as the generative model, even if the user specifies pitch or volume to the generative model at a certain point in time, depending on the sound signal being generated at that point in time, a delay occurs before a sound signal according to the volume is generated. When an AR type generative model is used, it is difficult to generate a sound signal according to the user's intention due to a delay in following the control value.
[0004] An object of the present invention is to provide a sound generation method and a sound generation device capable of generating a sound signal according to a user's intention by using an AR type generative model. [Means for solving the problem]
[0005] A sound generation method according to one aspect of the present invention is implemented by a computer, which receives control values indicating sound characteristics at each of a plurality of time points on a time axis, receives a compulsory instruction at a desired time point on the time axis, processes the control value at each time point and a sequence of acoustic features stored in a temporary memory using a trained model to generate acoustic features for that time point, updates the sequence of acoustic features stored in the temporary memory using the generated acoustic features if a compulsory instruction has not been received at that time point, and generates alternative acoustic features for one or more previous time points in accordance with the control value at that time point and updates the sequence of acoustic features stored in the temporary memory using the generated alternative acoustic features if a compulsory instruction has been received at that time point,
[0006] A sound generation device according to another aspect of the present invention includes a control value receiving unit that receives control values indicating sound characteristics at each of a plurality of time points on a time axis; a compulsory instruction receiving unit that receives a compulsory instruction at a desired time point on the time axis; a generation unit that processes the control value at each time point and an acoustic feature sequence stored in a temporary memory using a trained model to generate acoustic features for that time point; and an update unit that updates the acoustic feature sequence stored in the temporary memory using the generated acoustic features if a compulsory instruction has not been received at that time point, and generates alternative acoustic features for one or more most recent time points in accordance with the control value at that time point and updates the acoustic feature sequence stored in the temporary memory using the generated alternative acoustic features if a compulsory instruction has been received at that time point. Effect of the Invention
[0007] According to the present invention, an AR type generative model can be used to generate a sound signal according to the user's intention. [Brief description of the drawings]
[0008] [Figure 1]FIG. 1 is a block diagram showing the configuration of a processing system including a sound generating device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a trained model as a generator of acoustic features. [Diagram 3] FIG. 3 is a block diagram showing the configuration of the sound generating device. [Figure 4] FIG. 4 is a diagram of feature modification characteristics between original acoustic features and alternative acoustic features generated from the original acoustic features. [Diagram 5] FIG. 5 is a block diagram showing the configuration of the training device. [Figure 6] FIG. 6 is a flowchart showing an example of a sound generation process performed by the sound generation device of FIG. [Figure 7] FIG. 7 is a flowchart showing an example of a sound generation process performed by the sound generation device of FIG. [Figure 8] FIG. 8 is a flowchart showing an example of training processing by the training device of FIG. [Figure 9] FIG. 9 is a diagram for explaining generation of a substitute acoustic feature in the first modified example. [Figure 10] FIG. 10 is a diagram for explaining generation of a substitute acoustic feature in the second modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] (1) Processing system configuration A sound generation method and a sound generation device according to an embodiment of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a processing system including a sound generation device according to an embodiment of the present invention. As shown in Fig. 1, the processing system 100 includes a RAM (random access memory) 110, a ROM (read only memory) 120, a CPU (central processing unit) 130, a storage unit 140, an operation unit 150, and a display unit 160.
[0010] The processing system 100 is realized by a computer such as a PC, a tablet terminal, or a smartphone. Alternatively, the processing system 100 may be realized by the cooperative operation of multiple computers connected by a communication path such as Ethernet. The RAM 110, the ROM 120, the CPU 130, the storage unit 140, the operation unit 150, and the display unit 160 are connected to a bus 170. The RAM 110, the ROM 120, and the CPU 130 constitute the sound generating device 10 and the training device 20. In this embodiment, the sound generating device 10 and the training device 20 are configured by the common processing system 100, but may be configured by separate processing systems.
[0011] The RAM 110 is, for example, a volatile memory, and is used as a working area for the CPU 130. The ROM 120 is, for example, a non-volatile memory, and stores a sound generation program and a training program. The CPU 130 performs sound generation processing by executing the sound generation program stored in the ROM 120 on the RAM 110. The CPU 130 also performs training processing by executing the training program stored in the ROM 120 on the RAM 110. The sound generation processing and training processing will be described in detail later.
[0012] The sound generation program or the training program may be stored in the storage unit 140 instead of the ROM 120. Alternatively, the sound generation program or the training program may be provided in a form stored in a computer-readable storage medium and installed in the ROM 120 or the storage unit 140. Alternatively, when the processing system 100 is connected to a network such as the Internet, a sound generation program distributed from a server (including a cloud server) on the network may be installed in the ROM 120 or the storage unit 140.
[0013] The storage unit 140 includes a storage medium such as a hard disk, an optical disk, a magnetic disk, or a memory card. The storage unit 140 stores data such as a generative model m, a trained model M, a plurality of musical score data D1, a plurality of reference musical score data D2, and a plurality of reference data D3. The generative model m is an untrained generative model or a generative model pre-trained using data other than the reference data D3. Each musical score data D1 indicates a time series of a plurality of notes arranged on a time axis (i.e., a musical score) that constitutes one piece of music.
[0014] The trained model M (as data) is composed of algorithm data indicating the algorithm of a generative model that generates a corresponding acoustic feature sequence according to input data including control values indicating sound characteristics, and variables (trained variables) used in generating the acoustic feature sequence by the generative model. The algorithm is of AR (autoregressive) type, and includes a temporary memory that temporarily stores the most recent acoustic feature sequence, and a DNN (deep neural network) that estimates the current acoustic feature sequence from the input data and the most recent acoustic feature sequence. In the following, for simplicity of explanation, the generative model (as a generator) to which the trained variables are applied is also referred to as the "trained model M."
[0015] The trained model M receives as input data a time series of score features generated from the score data D1, and also receives control values indicating sound characteristics at each time point of the time series, processes the input data received at each time point and the acoustic feature sequence temporarily stored in the temporary memory, and generates acoustic features at that time point corresponding to the input data. Note that each of the multiple time points on the time axis corresponds to each of the multiple time frames used in the short-time frame analysis of the waveform, and the time difference between two adjacent time points is longer than the period of the waveform sample in the time domain, and is generally several milliseconds to several hundred milliseconds. Here, the interval between the time frames is assumed to be 5 milliseconds.
[0016] The control values input to the trained model M are features indicating acoustic features related to pitch, timbre, amplitude, etc., specified in real time by the user. The acoustic feature sequence generated by the trained model M is a time series of features indicating any one of acoustic features of the pitch, amplitude, frequency spectrum (amplitude), frequency spectrum envelope, etc., of the sound signal. Alternatively, the acoustic feature sequence may be a time series of the spectrum envelope of inharmonic components contained in the sound signal.
[0017] In this example, two trained models M are stored in the storage unit 140. Hereinafter, when distinguishing between the two trained models M, one trained model M is referred to as a trained model Ma, and the other trained model M is referred to as a trained model Mb. The acoustic feature sequence generated by the trained model Ma is a time series of pitch, and the control value input by the user is the variance and amplitude of the pitch. The acoustic feature sequence generated by the trained model Mb is a time series of frequency spectrum, and the control value input by the user is amplitude.
[0018] The trained model M may generate a sequence of acoustic features other than a pitch sequence or a frequency spectrum sequence (e.g., amplitude or frequency spectrum tilt, etc.), and the control value input by the user may be an acoustic feature other than pitch variance or amplitude.
[0019] The sound generation device 10 accepts control values at each of a plurality of time points (time frames) on the time axis of a piece of music to be played, and accepts a forcing instruction to make an acoustic feature generated using the trained model M conform relatively strongly to the control value at that time point at a particular time point (desired time point) among the plurality of time points. If a forcing instruction is not accepted at that time point, the sound generation device 10 updates the acoustic feature sequence in the temporary memory using the generated acoustic features. On the other hand, if a forcing instruction is accepted at that time point, the sound generation device 10 generates one or more substitute acoustic features conforming to the control value at that time point, and updates the acoustic feature sequence stored in the temporary memory using the generated substitute acoustic features.
[0020] Each reference score data D2 indicates a time series (musical score) of multiple notes arranged on a time axis that constitutes one piece of music. The score feature sequence input to the trained model M is a time series of features that indicate the characteristics of the notes at each time point on the time axis of the music, generated from each reference score data D2. Each reference data D3 is a time series of samples of the performance sound waveform of the time series of notes played (i.e., waveform data). The multiple reference score data D2 correspond to the multiple reference data D3. The reference score data D2 and the corresponding reference data D3 are used to build the trained model M by the training device 20. The trained model M is built by machine learning by learning the input / output relationship between the reference score feature at each time point, the reference control value at that time point, and the reference acoustic feature sequence immediately before that time point and the reference acoustic feature at that time point. The reference score data D2, the reference data D3, and their derived data (e.g., loudness or pitch variance) used in the training stage are called known data (data seen by the Model) and are distinguished from unknown data (data unseen by the Model) that are not used in the training stage. Known control values such as the reference loudness or reference pitch variance for training are derived data generated from the reference data D3, and unknown control values refer to control values such as loudness or pitch variance that are not used in training.
[0021] Specifically, at each time point, a pitch sequence of the waveform is extracted as a reference pitch sequence from each reference data D3, which is waveform data, and a frequency spectrum of the waveform is extracted as a reference frequency spectrum sequence. The reference pitch sequence or the reference frequency spectrum sequence is an example of a reference acoustic feature sequence. Also, at each time point, a pitch variance is extracted from the reference pitch sequence as a reference pitch variance, and an amplitude is extracted from the reference frequency spectrum sequence as a reference amplitude. The reference pitch variance or the reference amplitude is an example of a reference control value.
[0022] The trained model Ma is constructed by the generative model m learning, through machine learning, the reference score feature at each time point on the time axis, the reference pitch variance at that time point, the reference pitch immediately before that time point, and the input-output relationship between the reference pitch at that time point.The trained model Mb is constructed by the generative model m learning, through machine learning, the reference score feature at each time point on the time axis, the reference amplitude at that time point, the reference frequency spectrum immediately before that time point, and the input-output relationship between the reference frequency spectrum at that time point.
[0023] Some or all of the generative model m, the trained model M, the score data D1, the reference score data D2, the reference data D3, etc. may be stored in a computer-readable storage medium instead of being stored in the storage unit 140. Alternatively, when the processing system 100 is connected to a network, some or all of the generative model m, the trained model M, the score data D1, the reference score data D2, the reference data D3, etc. may be stored in a server on the network.
[0024] The operation unit 150 includes a pointing device such as a mouse or a keyboard, and is operated by the user to give a control value instruction or a forced instruction. The display unit 160 includes, for example, a liquid crystal display, and displays a predetermined GUI (Graphical User Interface) or the like for receiving a control value instruction or a forced instruction from the user. The operation unit 150 and the display unit 160 may be configured as a touch panel display.
[0025] (2) Pre-trained model FIG. 2 is a block diagram showing the configuration of a trained model M as a generator of acoustic features. As shown in FIG. 2, each of the trained models Ma, Mb includes a temporary memory 1, an inference unit 2 that performs DNN calculations, and forced processing units 3, 4. The temporary memory 1 may be considered as part of the DNN algorithm. A generation unit 13 of the sound generation device 10, which will be described later, executes a generation process including processing of this trained model M. In this embodiment, each of the trained models Ma, Mb includes a forced processing unit 4, but each forced processing unit 4 may be omitted. In that case, at each point on the time axis, the acoustic features generated by the inference unit 2 are output as output data of the trained model M.
[0026] Although the trained model Ma and the trained model Mb are two independent models, they basically have the same configuration, and therefore, for the sake of simplicity, similar elements are given the same reference numerals. The description of each element of the trained model Mb basically conforms to that of the trained model Ma. First, the configuration of the trained model Ma will be described. The temporary memory 1 operates, for example, as a ring buffer memory, and sequentially stores acoustic feature strings (pitch strings) generated at a predetermined number of most recent time points. Note that some of the predetermined number of acoustic features stored in the temporary memory 1 are replaced with corresponding alternative acoustic features in response to a forced instruction. A first forced instruction regarding pitch is given to the trained model Ma, and a second forced instruction regarding amplitude is given to the trained model Mb, independently of each other.
[0027] The inference unit 2 is provided with an acoustic feature sequence s1 stored in the temporary memory 1. The inference unit 2 is also provided with a score feature sequence s2, a control value sequence (pitch variance sequence and amplitude sequence) s3, and an amplitude sequence s4 as input data from the sound generating device 10. The inference unit 2 processes the input data (score feature, pitch variance and amplitude as control values) at each point on the time axis and the acoustic feature sequence immediately prior to that point to generate an acoustic feature (pitch) at that point. As a result, the inference unit 2 outputs the generated acoustic feature sequence (pitch sequence) s5.
[0028] The forcing processor 3 is given a first forcing instruction from the sound generator 10 at a certain time point (desired time point) among a plurality of time points on the time axis. The forcing processor 3 is also given a pitch variance sequence s3 and an amplitude sequence s4 as control values from the sound generator 10 at each of a plurality of time points on the time axis. If the first forcing instruction is not given at that time point, the forcing processor 3 updates the acoustic feature sequence s1 stored in the temporary memory 1 using the acoustic feature (pitch) generated at that time point by the inference unit 2. In detail, the acoustic feature sequence s1 in the temporary memory 1 is shifted back by one to discard the oldest acoustic feature, and the most recent acoustic feature is set as the generated acoustic feature. That is, the acoustic feature sequence in the temporary memory 1 is updated in a FIFO (First In First Out) manner. Note that the most recent acoustic feature is synonymous with the acoustic feature at that time point (current time).
[0029] On the other hand, if the first forcing instruction is given at that time, the forcing processing unit 3 generates substitute acoustic features (pitch) for the most recent one or more time points (1+α time points) according to the control value (pitch variance) at that time point, and updates the most recent one or more time points of the acoustic feature sequence s1 stored in the temporary memory 1 using the generated substitute acoustic features. In detail, the acoustic feature sequence s1 in the temporary memory 1 is shifted back by one to discard the oldest acoustic feature, and the most recent one or more acoustic features are replaced with the generated one or more substitute acoustic features. The tracking of the output data of the trained model Ma to the control value is improved even if the generated substitute acoustic feature is only for the most recent one time point, but is further improved if the most recent 1+α time point substitute acoustic feature is generated and updated. Note that substitute acoustic features for all time points in the temporary memory 1 may be generated. The update of the acoustic feature sequence in the temporary memory 1 by the substitute acoustic feature only for the most recent one time point is the same operation as the update by the acoustic feature sequence described above, and can be said to be FIFO-like. The update using the substitute acoustic feature at the most recent (1+α) point in time is almost the same as the update using the acoustic feature sequence described above, except for the update for α, so it is called a quasi-FIFO update.
[0030] The forcing processor 4 is given a first forcing instruction from the sound generating device 10 at a certain point in time (desired point in time) on the time axis. In addition, the forcing processor 4 is given an acoustic feature sequence s5 from the inference unit 2 at each point in time on the time axis. If the first forcing instruction is not given at that point in time, the forcing processor 4 outputs the acoustic feature (pitch) generated by the inference unit 2 as the output data of the trained model Ma at that point in time.
[0031] On the other hand, if the first forcible instruction is given at that time, the forcible processing unit 4 generates one alternative acoustic feature according to the control value (pitch variance) at that time, and outputs the generated alternative acoustic feature (pitch) as output data of the trained model Ma at that time. The most recent feature among the one or more alternative acoustic features may be used as the one alternative acoustic feature. In other words, the forcible processing unit 4 does not need to generate an alternative feature. In this way, when the first forcible instruction is not given, the acoustic feature generated by the inference unit 2 is output from the trained model Ma, and when the first forcible instruction is given, the alternative acoustic feature is output from the trained model Ma, and the output acoustic feature sequence (pitch sequence) s5 is given to the trained model Mb.
[0032] Next, the trained model Mb will be described, focusing on the differences from the trained model Ma. In the trained model Mb, the temporary memory 1 sequentially stores acoustic feature sequences (frequency spectrum sequences) s1 at a predetermined number of immediately preceding points in time. In other words, the temporary memory 1 stores acoustic features for a predetermined number (several frames).
[0033] The inference unit 2 is provided with an acoustic feature sequence s1 stored in the temporary memory 1. The inference unit 2 is also provided with a score feature sequence s2 from the sound generation device 10, a control value sequence (amplitude sequence) s4, and a pitch sequence s5 from the trained model Ma as input data. The inference unit 2 processes the input data (score feature sequence, pitch, amplitude as control value) at each point on the time axis and the acoustic feature immediately before that point to generate an acoustic feature (frequency spectrum) at that point. As a result, the generated acoustic feature sequence (frequency spectrum sequence) s5 is output as output data.
[0034] The forcing processor 3 is given a second forcing instruction from the sound generator 10 at a certain point in time (desired point in time) on the time axis. In addition, the forcing processor 3 is given a control value sequence (amplitude sequence) s4 from the sound generator 10 at each point in time on the time axis. If the second forcing instruction is not given at that point in time, the forcing processor 3 updates the acoustic feature sequence s1 stored in the temporary memory 1 in a FIFO manner using the acoustic feature (frequency spectrum) generated by the inference unit 2 at that point in time. On the other hand, if the second forcing instruction is given at that point in time, the forcing processor 3 generates one or more nearest substitute acoustic features (frequency spectrum) according to the control value (amplitude) at that point in time, and updates one or more nearest acoustic features of the acoustic feature sequence s1 stored in the temporary memory 1 in a FIFO or quasi-FIFO manner using the generated substitute acoustic features.
[0035] The forcing processor 4 is given a second forcing instruction from the sound generator 10 at a certain time point (desired time point) on the time axis. The forcing processor 4 is also given an acoustic feature sequence (frequency spectrum sequence) s5 from the inference unit 2 at each time point on the time axis. If the second forcing instruction is not given at that time point, the forcing processor 4 outputs the acoustic feature (frequency spectrum) generated by the inference unit 2 as the output data of the trained model Mb at that time point. On the other hand, if the second forcing instruction is given at that time point, the forcing processor 4 generates (or uses) one of the most recent alternative acoustic features according to the control value (amplitude) at that time point, and outputs the alternative acoustic feature (frequency spectrum) as the output data of the trained model Mb at that time point. The acoustic feature sequence (frequency spectrum sequence) s5 output from the trained model Mb is given to the sound generator 10.
[0036] (3) Sound generation device Fig. 3 is a block diagram showing the configuration of sound generation device 10. As shown in Fig. 3, sound generation device 10 includes, as functional units, a control value receiving unit 11, a forced instruction receiving unit 12, a generating unit 13, an updating unit 14, and a synthesizing unit 15. The functional units of sound generation device 10 are realized by CPU 130 in Fig. 1 executing a sound generation program. At least a part of the functional units of sound generation device 10 may be realized by hardware such as a dedicated electronic circuit.
[0037] A GUI for receiving control value instructions or forced instructions is displayed on the display unit 160. The user operates the GUI using the operation unit 150 to specify the pitch variance and amplitude as control values at multiple points on the time axis of a piece of music, and to provide a forced instruction at a desired point on the time axis. The control value receiving unit 11 receives the pitch variance and amplitude specified through the GUI from the operation unit 150 at each point on the time axis, and provides the pitch variance sequence s3 and the amplitude sequence s4 to the generating unit 13.
[0038] The compulsory instruction receiving unit 12 receives a compulsory instruction given through the GUI from the operation unit 150 at a desired time point on the time axis, and gives the received compulsory instruction to the generation unit 13. The compulsory instruction may be generated automatically, not from the operation unit 150. For example, if the musical score data D1 includes compulsory instruction information indicating the time point at which the compulsory instruction should be given, the generation unit 13 may automatically generate a compulsory instruction at that time point on the time axis, and the compulsory instruction receiving unit 12 may receive the automatically generated compulsory instruction. Alternatively, the generation unit 13 may analyze the musical score data D1 that does not include compulsory instruction information, detect an appropriate time point in the song (such as a transition between piano and forte), and automatically generate a compulsory instruction at the detected time point.
[0039] The user operates the operation unit 150 to specify the score data D1 to be used for sound generation from among multiple score data D1 stored in the storage unit 140 or the like. The generation unit 13 acquires the trained models Ma, Mb stored in the storage unit 140 or the like and the score data D1 specified by the user. In addition, the generation unit 13 generates score features from the acquired score data D1 at each time point.
[0040] The generating unit 13 supplies the score feature sequence s2, and the pitch variance sequence s3 and amplitude sequence s4 from the control value receiving unit 11 as input data to the trained model Ma. At each point on the time axis, the generating unit 13 uses the trained model Ma to process the input data at that point (score feature sequence, pitch variance and amplitude as control values) and the pitch sequence generated just before that point and stored in the temporary memory 1 of the trained model Ma, and generates and outputs the pitch at that point.
[0041] The generating unit 13 also supplies the score feature sequence s2, the pitch sequence output from the trained model Ma, and the amplitude sequence s4 from the control value receiving unit 11 to the trained model Mb as input data. At each point on the time axis, the generating unit 13 uses the trained model Mb to process the input data at that point (score feature sequence, pitch, and amplitude as a control value) and the frequency spectrum sequence generated immediately before that point and stored in the temporary memory 1 of the trained model Mb, and generates and outputs a frequency spectrum at that point.
[0042] If the forcible instruction receiving unit 12 has not received a forcible instruction at that time, the update unit 14 updates the acoustic feature sequence s1 stored in the temporary memory 1 in a FIFO manner using the acoustic feature generated by the inference unit 2 via the forcible processing unit 3 of each of the trained models Ma and Mb. On the other hand, if a forcible instruction has been received at that time, the update unit 14 generates substitute acoustic features at one or more recent times according to the control value at that time via the forcible processing unit 3 of each of the trained models Ma and Mb, and updates the acoustic feature sequence s1 stored in the temporary memory 1 in a FIFO or quasi-FIFO manner using the generated substitute acoustic features.
[0043] Furthermore, if no forced instruction has been received by the forced instruction receiving unit 12 at that time, the update unit 14 outputs the acoustic feature generated by the inference unit 2 as the current acoustic feature of the acoustic feature sequence s5 via the forced processing unit 4 of each of the trained models Ma and Mb. On the other hand, if a forced instruction has been received at that time, the update unit 14 generates (or uses) the most recent alternative acoustic feature according to the control value at that time via the forced processing unit 4 of each of the trained models Ma and Mb, and outputs the alternative acoustic feature as the current acoustic feature of the acoustic feature sequence s5.
[0044] One or more substitute acoustic features at each time point are generated based on, for example, a control value at that time point and an acoustic feature generated at that time point. In this example, a substitute acoustic feature at each time point is generated by modifying the acoustic feature at each time point so that it falls within an allowable range according to a target value and a control value at that time point. The target value T is a typical value when the acoustic feature follows the control value. The allowable range according to the control value is specified by a floor value and a ceil value included in the forced instruction. Specifically, the allowable range according to the control value is specified by a lower limit value Tf (=T-Floor value) that is smaller than the target value T of the control value by the floor value, and an upper limit value Tc (=T+Ceil value) that is larger than the target value T of the control value by the ceil value.
[0045] FIG. 4 is a diagram of the modification characteristics of features between original acoustic features and alternative acoustic features generated from the original acoustic features. The features are the same as the control values. In FIG. 4, the horizontal axis indicates the feature (volume or pitch variance, etc.) v of the acoustic features generated by the inference unit 2 of the trained model M, and the vertical axis indicates the feature F(v) of the acoustic feature after modification (alternative acoustic feature).
[0046] As shown in range R1 of FIG. 4, when a feature v of a certain acoustic feature is smaller than a lower limit Tf, a substitute acoustic feature is generated by modifying the acoustic feature so that the feature F(v) becomes the lower limit Tf. As shown in range R2 of FIG. 4, when the feature v is equal to or larger than the lower limit Tf and equal to or smaller than the upper limit Tc, the unmodified acoustic feature becomes the substitute acoustic feature, so that the feature F(v) is the same as the feature v. As shown in range R3 of FIG. 4, when the feature v is larger than the upper limit Tc, a substitute acoustic feature is generated by modifying the acoustic feature so that the feature F(v) becomes the upper limit Tc. For example, when the feature v is pitch variance and is larger (or smaller) than the upper limit Tc, a substitute acoustic feature is generated by scaling using a coefficient (Tc / v) so that the variance becomes smaller (or larger) while maintaining the average of the generated pitch (acoustic feature). In addition, if the feature v is the volume and is greater (or smaller) than the upper limit Tc, the entire frequency spectrum (acoustic feature) is scaled by a coefficient (Tc / v) to make the volume smaller (or larger), and a substitute feature is generated.
[0047] When generating substitute acoustic features at multiple time points, the same Floor and Ceil values may be applied to each time point. Alternatively, the older the time point of the substitute acoustic feature, the smaller the degree of modification of the feature. Specifically, the Floor and Ceil values in Fig. 4 are set to the current values, and the older the time point, the larger the previous Floor and Ceil values are set to. If replacement with substitute acoustic features is performed at multiple points, the generated acoustic features will track the control value more quickly.
[0048] The synthesis unit 15 functions as, for example, a vocoder, and generates a sound signal that is a time domain waveform process from the frequency domain acoustic feature sequence (frequency spectrum sequence) s5 generated by the forced processing unit 4 of the trained model Mb in the generation unit 13. The generated sound signal is supplied to a sound system including a speaker and the like connected to the synthesis unit 15, and sound based on the sound signal is output. In this example, the sound generation device 10 includes the synthesis unit 15, but the embodiment is not limited to this. The sound generation device 10 does not have to include the synthesis unit 15.
[0049] (4) Training equipment Fig. 5 is a block diagram showing the configuration of training device 20. As shown in Fig. 5, training device 20 includes an extraction unit 21 and a construction unit 22. The functional units of training device 20 are realized by CPU 130 in Fig. 1 executing a training program. At least a part of the functional units of training device 20 may be realized by hardware such as a dedicated electronic circuit.
[0050] The extraction unit 21 extracts a reference pitch sequence and a reference frequency spectral sequence as a reference acoustic feature sequence by analyzing each of the multiple reference data D3 stored in the storage unit 140 or the like. In addition, the extraction unit 21 processes the extracted reference pitch sequence and reference frequency spectral sequence to extract a reference pitch variance sequence, which is a time series of the variance of the reference pitch, and a reference amplitude sequence, which is a time series of the amplitude of a waveform corresponding to the reference frequency spectrum, as reference control value sequences, respectively.
[0051] The construction unit 22 acquires the generative model m to be trained and the reference score data D2 from the storage unit 140, etc. The construction unit 22 also generates a reference score feature sequence from the reference score data D2, and trains the generative model m by using the reference score feature sequence, the reference pitch variance sequence, and the reference amplitude sequence as input data and the reference pitch sequence as a correct value of the output data by a machine learning technique. In the training, the temporary memory 1 (FIG. 2) stores the reference pitch sequence immediately before each time point among the reference pitch sequences generated by the generative model m.
[0052] The construction unit 22 uses the generative model m to process the input data (reference score feature, reference pitch variance and reference volume as control values) at each time point on the time axis and the reference pitch sequence immediately before that time point stored in the temporary memory 1, to generate the pitch at that time point. The construction unit 22 then adjusts the variables of the generative model m so that the error between the generated pitch sequence and the reference pitch sequence (correct answer) becomes small. By repeating this training until the error becomes sufficiently small, a trained model Ma that has learned the input / output relationship between the input data (reference score feature, reference pitch variance and reference amplitude) at each time point on the time axis and the output data (reference pitch) is constructed.
[0053] Similarly, the construction unit 22 trains the generative model m by a machine learning technique, using the reference score feature sequence, the reference pitch sequence, and the reference amplitude sequence as input data and the reference frequency spectral sequence as a correct value of the output data. In the training, the temporary memory 1 stores, among the reference frequency spectral sequences generated by the generative model m, the reference frequency spectral sequence immediately before each time point.
[0054] The construction unit 22 uses the generation model m to process the input data (reference score feature, reference pitch, and reference amplitude as a control value) at each time point on the time axis and the reference frequency spectrum sequence immediately before each time point stored in the temporary memory 1 to generate a frequency spectrum at that time point. Then, the construction unit 22 adjusts the variables of the generation model m so that the error between the generated frequency spectrum sequence and the reference frequency spectrum sequence (correct answer) becomes small. By repeating this training until the error becomes sufficiently small, a trained model Mb that has learned the input / output relationship between the input data (reference score feature, reference pitch, and reference amplitude) at each time point on the time axis and the output data (reference frequency spectrum sequence) is constructed. The construction unit 22 stores the constructed trained models Ma, Mb in the storage unit 140 or the like.
[0055] (5) Sound generation processing Figures 6 and 7 are flow charts showing an example of sound generation processing by the sound generating device 10 of Figure 3. The sound generation processing of Figures 6 and 7 is performed by the CPU 130 of Figure 1 executing a sound generation program stored in the storage unit 140 or the like. First, the CPU 130 determines whether or not the user has selected the musical score data D1 of any song (step S1). If the musical score data D1 has not been selected, the CPU 130 waits until the musical score data D1 is selected.
[0056] When the score data D1 of a certain song is selected, the CPU 130 sets the current time t to the beginning (first time frame) of the song of the score data, and generates score features at the current time t from the score data D1 (step S2). The CPU 130 also accepts the pitch variance and amplitude input by the user at that time as control values at the current time t (step S3). Furthermore, the CPU 130 determines whether or not the first or second forcing instruction from the user has been accepted at the current time t (step S4).
[0057] Moreover, CPU 130 acquires pitch sequences generated at a plurality of time points t immediately before the current time point t from the temporary memory 1 of trained model Ma (step S5). Furthermore, CPU 130 acquires a frequency spectrum sequence generated immediately before the current time point t from the temporary memory 1 of trained model Mb (step S6). Steps S2 to S6 may be executed either first or simultaneously.
[0058] Next, the CPU 130 uses the inference unit 2 of the trained model Ma to process the input data (the score feature generated in step S1, the pitch variance and amplitude accepted in step S3) and the previous pitch acquired in step S5 to generate a pitch at the current time t (step S7). Next, the CPU 130 determines whether or not the first forcing instruction was accepted in step S4 (step S8). If the first forcing instruction was not accepted, the CPU 130 uses the pitch generated in step S7 to update the pitch sequence stored in the temporary memory 1 of the trained model Ma in a FIFO manner (step S9). The CPU 130 also outputs the pitch as output data (step S10) and proceeds to step S14.
[0059] If the first forcing instruction has been accepted, the CPU 130 generates one or more recent alternative acoustic features (alternative pitches) according to the pitch variance accepted in step S3 and the pitch generated in step S7 (step S11). After that, the CPU 130 updates the pitch stored in the temporary memory 1 of the trained model Ma in a FIFO or quasi-FIFO manner using the generated alternative acoustic features at one or more times (step S12). The CPU 130 also outputs the generated alternative acoustic features at the current time as output data (step S13) and proceeds to step S14. Either of steps S12 and S13 may be executed first, or may be executed simultaneously.
[0060] In step S14, the CPU 130 uses the trained model Mb to generate a frequency spectrum at the current time t from the input data (the score feature acquired in step S1, the amplitude accepted in step S3, and the pitch generated in step S7) and the immediately preceding frequency spectrum acquired in step S6 (step S14). Next, the CPU 130 determines whether or not a second forcing instruction has been accepted in step S4 (step S15). If the second forcing instruction has not been accepted, the CPU 130 uses the frequency spectrum generated in step S14 to update the frequency spectrum sequence stored in the temporary memory 1 of the trained model Mb in a FIFO manner (step S16). The CPU 130 also outputs the frequency spectrum as output data (step S17) and proceeds to step S21.
[0061] If the second forcing instruction is accepted, the CPU 130 generates one or more recent substitute acoustic features (substitute frequency spectra) according to the amplitude, based on the amplitude accepted in step S3 and the frequency spectrum generated in step S14 (step S18). After that, the CPU 130 updates the frequency spectrum sequence stored in the temporary memory 1 of the trained model Mb in a FIFO or quasi-FIFO manner using the generated substitute acoustic features at one or more times (step S19). The CPU 130 also outputs the generated substitute acoustic features at the current time as output data (step S20), and proceeds to step S21. Either of steps S19 and S20 may be executed first, or may be executed simultaneously.
[0062] In step S21, the CPU 130 generates a current sound signal from the frequency spectrum output as output data by using any known vocoder technique (step S21). As a result, a sound based on the current sound signal (current time frame) is output from the sound system. After that, the CPU 130 determines whether the performance of the song has ended, that is, whether the current time t of the performance of the musical score data D1 has reached the end of the song (last time frame) (step S22).
[0063] If the current time t is not yet the end of the performance, CPU 130 waits until the next time t (next time frame) (step S23) and returns to step S2. The wait time until the next time t is, for example, 5 milliseconds. Steps S2 to S22 are repeatedly executed by CPU 130 for each time t (time frame) until the performance ends. Here, if it is not necessary to reflect the control value given at each time t in the sound signal in real time, the wait at step S23 can be omitted. For example, if the time change of the control value is predetermined (the control value for each time t is programmed in the musical score data D1), step S23 may be omitted and the process may return to step S2.
[0064] (6) Training process Fig. 8 is a flow chart showing an example of training processing by the training device 20 of Fig. 5. The training processing of Fig. 8 is performed by the CPU 130 of Fig. 1 executing a training program stored in the storage unit 140 or the like. First, the CPU 130 acquires a plurality of reference data D3 (waveform data of a plurality of pieces of music) used for training from the storage unit 140 or the like (step S31). Next, the CPU 130 generates and acquires a reference score feature sequence of a piece of music from the reference score data D2 of the piece of music corresponding to each of the reference data D3 (step S32).
[0065] Next, the CPU 130 extracts a reference pitch sequence and a reference frequency spectrum sequence from each reference data D3 (step S33). After that, the CPU 130 processes the extracted reference pitch sequence and reference frequency spectrum sequence, respectively, to extract a reference pitch variance sequence and a reference amplitude sequence (step S34).
[0066] Next, the CPU 130 acquires one generative model m to be trained, and trains the generative model m using input data (the reference score feature sequence acquired in step S32, the reference pitch variance sequence and the reference amplitude sequence extracted in step S34) and the correct output data (the reference pitch sequence extracted in step S33). As described above, the variables of the generative model m are adjusted so that the error between the pitch sequence generated by the generative model m and the reference pitch sequence is reduced. As a result, the CPU 130 trains the generative model m to machine-learn the input / output relationship between the input data (reference score feature, reference pitch variance, and reference amplitude) at each time point and the correct output data (reference pitch) at that time point (step S35). In this training, the generative model m may generate the pitch at that time point by processing the pitches at the previous multiple time points included in the reference pitch sequence in the inference unit 2, instead of the pitches generated at the previous multiple time points stored in the temporary memory 1.
[0067] Next, CPU 130 judges whether the error has become small enough, that is, whether generative model m has acquired the input-output relationship (step S36). If the error is still large and machine learning is judged to be insufficient, CPU 130 returns to step S35. Steps S35 to S36 are repeated while changing the parameters until generative model m acquires the input-output relationship. The number of times machine learning is repeated varies depending on the quality conditions (type of error to be calculated, threshold value used for judgment, etc.) that one trained model Ma to be constructed must satisfy.
[0068] When it is determined that sufficient machine learning has been performed, the generative model m has acquired the input / output relationship between the input data (including the reference pitch variance and the reference amplitude) at each time point and the correct value (reference pitch) of the output data at that time point through training, and the CPU 130 saves the generative model m that has acquired the input / output relationship as one of the trained models Ma (step S37). In this way, this trained model Ma is trained to estimate the pitch at each time point based on the unknown pitch variance and the pitches at the immediately preceding multiple time points. Here, the unknown pitch variance means the pitch variance that has not been used in the training.
[0069] Moreover, the CPU 130 acquires another generative model m to be trained, and trains the generative model m using input data (the reference score feature sequence acquired in step S32, the reference pitch sequence extracted in step S33, and the reference amplitude sequence extracted in step S34) and the correct output data (reference frequency spectrum sequence). As described above, the variables of the generative model m are adjusted so that the error between the frequency spectrum sequence generated by the generative model m and the reference frequency spectrum sequence is reduced. As a result, the CPU 130 makes the generative model m machine-learn the input / output relationship between the input data (reference score feature, reference pitch, and reference amplitude) at each time point and the correct output data (reference frequency spectrum) at that time point (step S38). In this training, the generative model m may generate a frequency spectrum at that time point by processing the frequency spectra at the previous multiple time points included in the reference frequency spectrum sequence in the inference unit 2, instead of the frequency spectra generated at the previous multiple time points stored in the temporary memory 1.
[0070] Next, CPU 130 judges whether the error has become small enough, that is, whether generative model m has acquired the input-output relationship (step S39). If the error is still large and machine learning is judged to be insufficient, CPU 130 returns to step S38. Steps S38 to S39 are repeated while changing the parameters until generative model m acquires the input-output relationship. The number of times machine learning is repeated varies depending on the quality conditions (type of error to be calculated, threshold value used for judgment, etc.) that the other trained model Mb to be constructed should satisfy.
[0071] When it is determined that sufficient machine learning has been performed, the generative model m has acquired the input / output relationship between the input data (including the reference amplitude) at each time point and the correct value (reference frequency spectrum) of the output data at that time point through training, and the CPU 130 saves the generative model m that has acquired the input / output relationship as the other trained model Mb (step S40), and ends the training process. In this way, this trained model Mb is trained to estimate the frequency spectrum at each time point based on the unknown amplitude and the frequency spectrum at multiple immediately preceding time points. Here, the unknown amplitude means the amplitude not used in the training. Either of steps S35 to S37 and steps S38 to S40 may be executed first, or may be executed in parallel.
[0072] (7) Variations In this embodiment, the CPU 130, as the update unit 14, generates a substitute acoustic feature at each time point by modifying the feature of the acoustic feature at each time point so that it falls within an allowable range according to the target value and control value at that time point, but the generation method is not limited to this. For example, the CPU 130 may generate a substitute acoustic feature at each time point by reflecting an excess amount of the feature of the acoustic feature at each time point from a neutral range (substituting for the allowable range) according to the control value at that time point in the modification of the acoustic feature at a predetermined ratio. This ratio is called a Ratio value.
[0073] FIG. 9 is a diagram for explaining generation of a substitute acoustic feature in the first modified example. The upper limit Tc of the neutral range is (T+Ceil value), and the lower limit Tf is (T-Floor value). In the first modified example, as shown in range R1 of FIG. 9, when a feature v of a certain acoustic feature is smaller than the lower limit Tf, the acoustic feature is modified so that the feature v becomes F(v)=v-(v-Tf)×Ratio value. As shown in range R2 of FIG. 9, when the feature v is equal to or larger than the lower limit Tf and equal to or smaller than the upper limit Tc, the acoustic feature is not modified, and the feature F(v) becomes the same as the feature v. As shown in range R3 of FIG. 9, when the feature v is larger than the upper limit Tc, the acoustic feature is modified so that F(v)=v-(v-Tc)×Ratio value. When generating substitute acoustic features for multiple time points, the Floor value and Ceil value may not be changed depending on the time point, and the Ratio value may be set to a smaller value for older time points.
[0074] In Fig. 9, the feature value F(v) of the acoustic feature after modification when the ratio value is 0, 0.5, and 1 is indicated by a thick dashed line, a thick dotted line, and a thick solid line, respectively. When the ratio value is 0, the feature value F(v) of the acoustic feature after modification is equal to the feature value v indicated by the thin dashed line in Fig. 4, and no constraint is applied. When the ratio value is 1, the feature value F(v) of the acoustic feature after modification is equal to the feature value F(v) of the acoustic feature after modification indicated by the thick solid line in Fig. 4. According to this configuration, when the acoustic feature of the acoustic feature sequence stored in the temporary memory 1 exceeds the neutral range, the excess amount can be reflected in the modification of the alternative acoustic feature at a ratio according to the Ratio value.
[0075] Alternatively, the CPU 130 may generate a substitute acoustic feature at each time point by modifying the acoustic feature at each time point so that the acoustic feature approaches a target value T corresponding to the control value at that time point at a rate corresponding to the Rate value. Fig. 10 is a diagram for explaining generation of a substitute acoustic feature in the second modified example. In the second modified example, as shown in Fig. 10, the acoustic feature is modified so that the feature F(v)=v-(vT)×Rate value is satisfied over the entire range of the feature v. When generating substitute acoustic features at multiple time points, the Rate value may be set to a smaller value for older time points.
[0076] In Fig. 10, the feature value F(v) of the acoustic feature after modification when the Rate value is 0, 0.5, and 1 is shown by a thick dashed line, a thick dotted line, and a thick solid line, respectively. When the Rate value is 0, the feature value F(v) of the acoustic feature after modification is equal to the feature value v shown by the dashed line in Fig. 4, and no constraint is applied. When the Rate value is 1, the feature value F(v) of the acoustic feature after modification is equal to the target value T of the control value, and the strongest constraint is applied.
[0077] (8) Effects of the embodiment As described above, the sound generation method of this embodiment is a method realized by a computer, which accepts control values indicating sound characteristics at each of a plurality of time points on a time axis, accepts a compulsory instruction at a desired time point on the time axis, processes the control value at each time point and the acoustic feature sequence stored in a temporary memory using a trained model to generate acoustic features for that time point, and if a compulsory instruction has not been accepted at that time point, updates the acoustic features of the acoustic feature sequence stored in the temporary memory using the generated acoustic features, and if a compulsory instruction has been accepted at that time point, generates alternative acoustic features in accordance with the control value at that time point, and updates the acoustic features of the acoustic feature sequence stored in the temporary memory using the generated alternative acoustic features.
[0078] According to this method, even if at a certain point in time the acoustic features generated using the trained model deviate from the values corresponding to the control values at that time, by issuing a forced instruction, acoustic features that track the control values relatively closely without a large delay from that point in time are generated, thereby making it possible to generate sound signals according to the user's intentions.
[0079] The trained model may be trained by machine learning to estimate acoustic features at each time point based on acoustic features at multiple immediately preceding time points.
[0080] The substitute acoustic feature at each time point may be generated based on the control value at that time point and the acoustic feature generated at that time point.
[0081] A substitute acoustic feature at each time point may be generated by modifying the acoustic feature at each time point so that it falls within an allowable range according to the control value at that time point.
[0082] The tolerance range according to the control value may be defined by a forcing instruction.
[0083] A substitute acoustic feature at each time point may be generated by subtracting from the acoustic feature at a predetermined ratio an excess amount from the neutral range corresponding to the control value at that time point in the acoustic feature at that time point.
[0084] A substitute acoustic feature at each time point may be generated by modifying the acoustic feature at each time point so that it approaches a target value according to the control value at that time point.
[0085] (9) Other embodiments In the above embodiment, both trained models Ma and Mb are used to generate acoustic features at each time point, but only one of the trained models Ma and Mb may be used to generate acoustic features at each time point. In this case, one of steps S7 to S13 and steps S14 to S20 of the sound generation process is executed, but the other is not executed.
[0086] In the former case, the pitch sequence generated in steps S7 to S13 is supplied to a known sound source, which generates a sound signal based on the pitch sequence. For example, the pitch sequence may be supplied to a speech segment concatenating type singing synthesizer, which generates singing according to the pitch sequence. Alternatively, the pitch sequence may be supplied to a waveform memory sound source or an FM sound source, which generates musical instrument sounds according to the pitch sequence.
[0087] In the latter, steps S14 to S20 receive a pitch sequence generated by a known method other than the trained model Ma and generate a frequency spectrum sequence. For example, a pitch sequence handwritten by a user, a pitch sequence extracted from an instrument sound or a user singing may be received, and a frequency spectrum sequence corresponding to the pitch sequence may be generated using the trained model Mb. In the former, the trained model Mb is not required, and steps S38 to S40 of the training process do not need to be executed. Similarly, in the latter, the trained model Ma is not required, and steps S35 to S37 do not need to be executed.
[0088] In the above embodiment, supervised learning is performed using the reference score data D2, but an encoder that generates a score feature sequence from the reference data D3 may be prepared to perform unsupervised machine learning using the reference data D3 without using the reference score data D2. The encoder process is executed in step S32 with the reference data D3 as input during the training phase, and in step S2 with the instrument sound or the user's singing as input during the utilization phase.
[0089] Although the above embodiment is a sound generation device that generates a sound signal of a musical instrument sound, the sound generation device may generate other sound signals. For example, the sound generation device may generate a speech sound signal from text data with a time stamp. In that case, the trained model M may be, for example, an AR type generative model that receives a text feature sequence generated from text data (instead of the musical score feature) and a control value sequence indicating the volume as input data, and generates a frequency spectrum feature sequence.
[0090] In the above embodiment, the user operates the operation unit 150 to input the control value in real time, but the user may also program the time change of the control value in advance and provide the control value that changes as programmed to the trained model M to generate acoustic features at each point in time.
Claims
1. Control values that indicate characteristics of the sound are received at multiple points on the time axis, Accepting a compulsory instruction at a desired point on the time axis; using the trained model to process the control value at each time point and the acoustic feature sequence stored in the temporary memory to generate an acoustic feature at that time point; If the forcible instruction has not been accepted at that time, updating the acoustic feature sequence stored in the temporary memory with the generated acoustic feature; if the forcible instruction has been accepted at that time point, generating substitute acoustic features for one or more most recent time points in accordance with the control value at that time point, and updating the acoustic feature sequence stored in the temporary memory with the generated substitute acoustic features; A computer-implemented method for generating sound.
2. The sound generation method according to claim 1 , wherein the trained model has been trained by machine learning to estimate acoustic features at each time point based on an unknown control value and acoustic features at multiple immediately preceding time points.
3. The sound generation method according to claim 2 , wherein the acoustic features generated by the trained model include features corresponding to the unknown control value.
4. The sound generation method according to claim 1 , wherein the substitute acoustic feature at each time point is generated based on the control value at that time point and the acoustic feature generated at that time point.
5. 4. The sound generation method according to claim 3, further comprising the steps of: modifying an acoustic feature generated at each point in time, the acoustic feature being the same type as the control value, so that the acoustic feature approaches the control value, thereby generating a substitute acoustic feature at that point in time.
6. The sound generating method according to claim 3 , further comprising the step of generating a substitute acoustic feature at each time point by modifying the acoustic feature so that the acoustic feature at each time point falls within an allowable range corresponding to the control value at that time point.
7. 5. The method of claim 4, wherein the tolerance range according to the control value is defined by the forcing instruction.
8. The sound generation method according to claim 3 , wherein a substitute acoustic feature is generated at each time point by modifying the acoustic feature so that the acoustic feature at each time point approaches a neutral range corresponding to the control value at that time point.
9. 4. The sound generating method according to claim 3, wherein a substitute acoustic feature at each time point is generated by modifying the acoustic feature at each time point so as to approach a target value corresponding to the control value at that time point.
10. 4. The sound generation method according to claim 1, wherein the acoustic feature sequence stored in the temporary memory is updated in a FIFO manner using the generated acoustic feature.
11. 9. The sound generation method according to claim 8, wherein the acoustic feature sequence stored in the temporary memory is updated in a FIFO or quasi-FIFO manner using the generated substitute acoustic features at one or more points in time.
12. The sound generation method according to claim 1 , wherein the control value is a pitch variance and the acoustic feature is a pitch.
13. The sound generation method according to any one of claims 1 to 3, wherein the control value is an amplitude, and the acoustic feature amount is a frequency spectrum.
14. a control value receiving unit that receives control values indicating characteristics of a sound at each of a plurality of time points on a time axis; a forcing instruction receiving unit that receives a forcing instruction at a desired time point on a time axis; a generation unit that processes the control value at each time point and the acoustic feature sequence stored in a temporary memory using a trained model to generate an acoustic feature at that time point; an updating unit that updates the acoustic feature sequence stored in the temporary memory by using the generated acoustic feature if the compulsory instruction has not been accepted at that time, and generates substitute acoustic features for one or more most recent time points in accordance with the control value at that time, and updates the acoustic feature sequence stored in the temporary memory by using the generated substitute acoustic feature if the compulsory instruction has been accepted at that time.
Citation Information
Patent Citations
Multi-singer singing synthesis method and device
CN112466313A
Singing voice synthesis device and singing voice synthesis method
JP2017107228A
Learning device, speech synthesis system and speech synthesis method
JP2018141917A
Electronic music instrument, control method of electronic music instrument and program
JP2019219568A
Information processing method and information processing device
JP2020076843A
