SOUND GENERATION METHOD USING MACHINE LEARNING MODEL, METHOD FOR TRAINING MACHINE LEARNING MODEL, SOUND GENERATION DEVICE, TRAINING DEVICE, SOUND GENERATION PROGRAM, AND TRAINING PROGRAM

Abstract

To provide a sound generating method, a training method, a sound generating device, a training device, a sound generating program, and a training program with which natural voices can be easily acquired.SOLUTION: Input of a first feature quantity column in which musical feature quantities temporally change is received by a receiving unit 12. The first feature quantity column is processed by using a trained model M, and a sound data column corresponding to a second feature quantity column in which the feature quantities change at second fineness is generated by a generating unit 13. The trained model M is a machine learning model that has learned an input-output relationship between an input feature quantity column in which the feature quantities temporally change at first fineness and a reference sound data column corresponding to an output feature quantity column in which the feature quantities temporally change at the second fineness higher than the first fineness.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a sound generation method, a training method, a sound generation device, a training device, a sound generation program, and a training program that are capable of generating sound. [Background technology]

[0002] There is known an application that generates an audio signal based on a time series of volume specified by a user. For example, in the application described in Non-Patent Document 1, a fundamental frequency, a latent variable, and loudness are extracted as features from a sound input by a user. A sound signal is generated by performing spectral modeling synthesis on the extracted features. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Jesse Engel, Lamtharn Hantrakul, Chenjie Gu and Adam Roberts, "DDSP: Differentiable Digital Signal Processing", arXiv:2001.04643v1 [cs.LG] 14 Jan 2020 Summary of the Invention [Problem to be solved by the invention]

[0004] To generate a voice signal that shows a voice that changes naturally like a human singing or playing, using the application described in Non-Patent Document 1, the user needs to specify the time series of the volume in detail. However, it is not easy to specify the time series of the volume in detail.

[0005] An object of the present invention is to provide a sound generation method, a training method, a sound generation device, a training device, a sound generation program, and a training program that can easily obtain a natural voice. [Means for solving the problem]

[0006] The present invention 1. The phase-dependent sound generation method uses a time-dependent musical feature With a given time resolution A first feature sequence is received, and the feature varies over time at a first resolution. With a given time resolution An input feature sequence and a feature that varies over time at a second resolution higher than the first resolution With a given time resolution The first feature sequence is processed using a trained model that has learned an input / output relationship between the output feature sequence and a reference sound data sequence corresponding to the output feature sequence, and the features are changed at a second resolution. With a given time resolution A sound data sequence corresponding to the second feature sequence is generated, and is implemented by a computer. A sound generation method according to a second aspect of the present invention receives an input of a first feature sequence in which musical features vary over time, processes the first feature sequence using a trained model that has acquired the input / output relationship between the input feature sequence in which features vary over time with a first resolution and a reference sound data sequence corresponding to an output feature sequence in which features vary over time with a second resolution higher than the first resolution, and generates a sound data sequence corresponding to a second feature sequence in which features vary over time with a second resolution, wherein the feature at each point in time in the input feature sequence is a representative value of the feature at each point in time in the output feature sequence within a predetermined period including that point in time, and the representative value is a statistical value of the feature at each point in time in the output feature sequence within the predetermined period, and the method is implemented by a computer. A sound generation method according to a third aspect of the present invention receives input of a first feature sequence whose musical features vary over time, processes the first feature sequence using a trained model that has mastered the input / output relationship between an input feature sequence whose features vary over time at a first resolution and a reference sound data sequence corresponding to an output feature sequence whose features vary over time at a second resolution higher than the first resolution, and generates a sound data sequence corresponding to a second feature sequence whose features vary over time at a second resolution, and further presents a reception screen on which the first feature sequence is displayed along a time axis, and the first feature sequence is input using the reception screen and is implemented by a computer.

[0007] The present invention Fourth The training method according to the aspect of the present invention includes: With a given time resolution A reference sound data sequence and an output feature sequence, which is a time series of the feature, are extracted, and from the output feature sequence, a feature that changes over time at a resolution lower than a predetermined resolution is extracted. With a given time resolution An input feature sequence is generated, and a trained model is constructed using machine learning to learn the input / output relationship between the input feature sequence and a reference sound data sequence corresponding to the output feature sequence, and the system is implemented by a computer.

[0008] The present invention Fifth The sound generating device according to the aspect of the present invention is a device for generating a sound having a musical feature that changes over time. With a given time resolution A receiving unit that receives an input of a first feature sequence, and a feature sequence that varies over time with a first resolution. With a given time resolution An input feature sequence and a feature that varies over time at a second resolution higher than the first resolution With a given time resolution The first feature sequence is processed using a trained model that has learned an input / output relationship between the output feature sequence and a reference sound data sequence corresponding to the output feature sequence, and the features are changed at a second resolution. With a given time resolutionand a generating unit that generates a sound data sequence corresponding to the second feature sequence.

[0009] The present invention Sixth The training device according to the aspect of the present invention is a device for training a musical feature quantity that changes over time with a predetermined resolution from reference data that indicates a sound waveform. With a given time resolution An extracting unit extracts a reference sound data sequence and an output feature sequence which is a time series of the feature of the reference sound data sequence, and extracts from the output feature sequence a feature that changes over time at a resolution lower than a predetermined resolution. With a given time resolution The system includes a generation unit that generates an input feature sequence, and a construction unit that constructs a trained model that has learned the input / output relationship between the input feature sequence and a reference sound data sequence corresponding to the output feature sequence through machine learning. A sound generation program according to a seventh aspect of the present invention causes one or more computers to perform a step of accepting an input of a first feature sequence of a predetermined time resolution in which musical features vary over time, and processing the first feature sequence using a trained model that has learned the input / output relationship between an input feature sequence of a predetermined time resolution in which features vary over time with a first resolution, and a reference sound data sequence corresponding to an output feature sequence of a predetermined time resolution in which features vary over time with a second resolution higher than the first resolution, to generate a sound data sequence corresponding to a second feature sequence of a predetermined time resolution in which features vary over time with a second resolution. A training program according to an eighth aspect of the present invention causes one or more computers to perform the steps of extracting, from reference data indicating sound waveforms, a reference sound data sequence of a predetermined time resolution in which musical features change over time with a predetermined resolution and an output feature sequence which is a time series of the features, generating, from the output feature sequence, an input feature sequence of a predetermined time resolution in which features change over time with a resolution lower than the predetermined resolution, and constructing, by machine learning, a trained model which has acquired the input / output relationship between the input feature sequence and the reference sound data sequence corresponding to the output feature sequence. Effect of the Invention

[0010] According to the present invention, natural voice can be easily acquired. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of a processing system including a sound generation device and a training device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing a configuration of a sound generating device. [Diagram 3] FIG. 11 is a diagram for explaining an example of the operation of the sound generating device. [Figure 4] FIG. 11 is a diagram for explaining an example of the operation of the sound generating device. [Diagram 5] 11A to 11C are diagrams illustrating another example of the operation of the sound generating device. [Figure 6] FIG. 2 is a block diagram showing the configuration of a training device. [Figure 7] FIG. 13 is a diagram for explaining an example of the operation of the training device. [Figure 8] 3 is a flowchart showing an example of a sound generation process performed by the sound generation device of FIG. 2. [Figure 9]7 is a flowchart showing an example of a training process by the training device of FIG. 6. [Figure 10] FIG. 11 is a diagram showing an example of a reception screen in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (1) Processing system configuration A sound generation method, a training method, a sound generation device, a training device, a sound generation program, and a training program according to a first embodiment of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a processing system including a sound generation device and a training device according to an embodiment of the present invention. As shown in Fig. 1, the processing system 100 includes a RAM (random access memory) 110, a ROM (read only memory) 120, a CPU (central processing unit) 130, a storage unit 140, an operation unit 150, and a display unit 160.

[0013] The processing system 100 is realized by a computer such as a PC, a tablet terminal, or a smartphone. Alternatively, the processing system 100 may be realized by the cooperative operation of multiple computers connected by a communication path such as Ethernet. The RAM 110, the ROM 120, the CPU 130, the storage unit 140, the operation unit 150, and the display unit 160 are connected to a bus 170. The RAM 110, the ROM 120, and the CPU 130 constitute the sound generating device 10 and the training device 20. In this embodiment, the sound generating device 10 and the training device 20 are configured by the common processing system 100, but may be configured by separate processing systems.

[0014] The RAM 110 is, for example, a volatile memory, and is used as a working area for the CPU 130. The ROM 120 is, for example, a non-volatile memory, and stores a sound generation program and a training program. The CPU 130 performs sound generation processing by executing the sound generation program stored in the ROM 120 on the RAM 110. The CPU 130 also performs training processing by executing the training program stored in the ROM 120 on the RAM 110. The sound generation processing and training processing will be described in detail later.

[0015] The sound generation program or the training program may be stored in the storage unit 140 instead of the ROM 120. Alternatively, the sound generation program or the training program may be provided in a form stored in a computer-readable storage medium and installed in the ROM 120 or the storage unit 140. Alternatively, when the processing system 100 is connected to a network such as the Internet, the sound generation program distributed from a server (including a cloud server) on the network may be installed in the ROM 120 or the storage unit 140.

[0016] The storage unit 140 includes a storage medium such as a hard disk, an optical disk, a magnetic disk, or a memory card. The storage unit 140 stores a trained model M, result data D1, a plurality of reference data D2, a plurality of score data D3, and a plurality of reference score data D4. The plurality of reference data D2 and the plurality of reference score data D4 correspond to each other. The trained model M is a generative model that receives a score feature sequence and a control value (input feature sequence) of the score data, and estimates result data (sound data sequence) according to the score feature sequence and the control value. The trained model M learns the input / output relationship between the score feature sequence and the input feature sequence, and the reference sound data sequence corresponding to the output feature sequence, and is constructed by the training device 20. In this example, the trained model M is an AR (regression) type generative model, but may be a non-AR type generative model.

[0017] The input feature sequence is a time series in which musical features vary over time with a first fineness. The output feature sequence is a time series in which features vary over time with a second fineness higher than the first fineness. The musical features may be, for example, amplitude or its derivative, or pitch or its derivative. The musical features may include, instead of amplitude or the like, a spectral tilt or a spectral center of gravity, or a ratio of high-frequency power to low-frequency power (high-frequency power / low-frequency power). The sound data sequence is, for example, a mel spectrogram.

[0018] Here, the resolution does not mean the number of features within a unit time (time resolution), but means the frequency of feature changes within a unit time or the amount of high frequency components contained. In other words, the input feature sequence is a feature sequence obtained by lowering the resolution of the output feature sequence, such as a feature sequence processed so that most of the output feature sequence is the same as the immediately preceding value, or a feature sequence obtained by applying a certain type of low-pass filter to the output feature sequence. Here, the time resolution is the same for the input feature sequence and the output feature sequence.

[0019] The result data D1 indicates a sound data sequence corresponding to the sound feature sequence generated by the sound generation device 10. The reference data D2 is waveform data used to train the trained model M, i.e., a time series of sound waveform samples. The time series of features (e.g., amplitude) extracted from each waveform data in relation to sound control is called an output feature sequence. The score data D3 and the reference score data D4 each indicate a score including a plurality of notes (sequences of notes) arranged on a time axis. The score feature sequence generated from the score data D3 is used to generate the result data D1 by the sound generation device 10. The reference data D2 and the reference score data D4 are used to construct the trained model M by the training device 20.

[0020] The trained model M, the result data D1, the reference data D2, the score data D3, and the reference score data D4 may be stored in a computer-readable storage medium instead of in the storage unit 140. Alternatively, if the processing system 100 is connected to a network, the trained model M, the result data D1, the reference data D2, the score data D3, or the reference score data D4 may be stored in a server on the network.

[0021] The operation unit 150 includes a pointing device such as a mouse or a keyboard, and is operated by the user to perform predetermined input. The display unit 160 includes, for example, a liquid crystal display, and displays a predetermined GUI (Graphical User Interface) or the results of sound generation processing, etc. The operation unit 150 and the display unit 160 may be configured as a touch panel display.

[0022] (2) Sound generation device Fig. 2 is a block diagram showing the configuration of the sound generation device 10. Figs. 3 and 4 are diagrams for explaining an example of the operation of the sound generation device 10. As shown in Fig. 2, the sound generation device 10 includes a presentation unit 11, a reception unit 12, a generation unit 13, and a processing unit 14. The functions of the presentation unit 11, the reception unit 12, the generation unit 13, and the processing unit 14 are realized by the CPU 130 in Fig. 1 executing a sound generation program. At least a part of the presentation unit 11, the reception unit 12, the generation unit 13, and the processing unit 14 may be realized by hardware such as an electronic circuit.

[0023] As shown in FIG. 3, the presentation unit 11 displays a reception screen 1 on the display unit 160 as a GUI for receiving input from a user. The reception screen 1 is provided with a reference area 2 and an input area 3. In the reference area 2, a reference image 4 showing the positions of multiple notes on the time axis is displayed based on the score data D3 selected by the user. The reference image is, for example, a piano roll. By operating the operation unit 150, the user can select and edit the score data D3 showing a desired score from the multiple score data D3 stored in the storage unit 140 or the like.

[0024] The input area 3 is disposed so as to correspond to the reference area 2. Using the operation unit 150 of FIG. 1, the user roughly inputs each feature amount (amplitude in this example) into the input area 3 while looking at the notes in the reference image 4, so that the feature amount changes over time. This allows the first feature amount string to be input. In the input example of FIG. 3, the amplitude is input so that the amplitude is small in the first to fifth bars of the musical score, large in the sixth to seventh bars, and somewhat large in the eighth to tenth bars. The receiving unit 12 receives the first feature amount string input into the input area 3.

[0025] The trained model M stored in the storage unit 140 or the like includes, for example, a neural network (DNN (deep neural network) L1 in the example of FIG. 4) as shown in FIG. 4. The score data D3 selected by the user and the first feature sequence input to the input area 3 are given to the DNNL1. The generation unit 13 processes the score data D3 and the first feature sequence using the DNNL1 to generate result data D1 including a time series of pitch in the score and a time series of a spectral envelope. The result data D1 indicates a sound data sequence corresponding to the second feature sequence in which the amplitude changes with a second resolution. In addition, in the time series of pitch included in the result data D1, the pitch changes with a high resolution according to the first feature sequence (similar to the amplitude). Note that the result data may be result data D1 indicating a time series of spectrum in the score (for example, a mel spectrogram).

[0026] The amplitude at each time point in the first feature sequence may be a representative value of the amplitude within a predetermined period including the time point in the second feature sequence. The interval between two adjacent time points is, for example, 5 ms, the length of the predetermined period is, for example, 3 s, and each time point is, for example, located at the center of the corresponding predetermined period. The representative value may be a statistical value of the amplitude within the predetermined period in the second feature sequence. For example, the representative value may be the maximum value, average value, median value, mode, variance, or standard deviation of the amplitude.

[0027] However, the representative value is not limited to the statistical value of the amplitude in the second feature sequence within a predetermined period. For example, the representative value may be the ratio between the maximum value of the first harmonic and the maximum value of the second harmonic of the amplitude in the second feature sequence within a predetermined period, or the logarithm of the ratio. Alternatively, the representative value may be the average value of the maximum value of the first harmonic and the maximum value of the second harmonic.

[0028] The generation unit 13 may store the generated result data D1 in the storage unit 140 or the like. The processing unit 14 functions as, for example, a vocoder, and generates an audio signal that is a waveform processing of the time domain from the frequency domain result data D1 generated by the generation unit 13. The generated audio signal is supplied to a sound system including a speaker and the like connected to the processing unit 14, and a sound based on the audio signal is output. In this example, the sound generation device 10 includes the processing unit 14, but the embodiment is not limited thereto. The sound generation device 10 does not need to include the processing unit 14.

[0029] 3, the input area 3 is disposed below the reference area 2 on the reception screen 1, but the embodiment is not limited to this. The input area 3 may be disposed above the reference area 2 on the reception screen 1. Alternatively, the input area 3 may be disposed so as to overlap with the reference area 2 on the reception screen 1.

[0030] 3, the reception screen 1 includes a reference area 2, and a reference image 4 is displayed in the reference area 2, but the embodiment is not limited to this. The reception screen 1 does not need to include the reference area 2. In this case, the user uses the operation unit 150 to draw a desired time series of amplitudes in the input area 3. This allows the user to input a first feature sequence in which the amplitudes change roughly.

[0031] In the example of FIG. 4, the trained model M includes one DNNL1, but the embodiment is not limited thereto. The trained model M may include multiple DNNs. FIG. 5 is a diagram for explaining another operation example of the sound generation device 10. In the example of FIG. 5, the trained model M includes three DNNs: DNNL1, L2, and L3. Music score data D3 selected by the user is provided to each of DNNL1 to L3. In addition, the first feature string input by the user to the input area 3 is provided to DNNL1.

[0032] The generating unit 13 processes the musical score data D3 and the first feature sequence using DNNL1 to generate a first intermediate feature sequence whose amplitude changes over time. The resolution of the time series of amplitude in the first intermediate feature sequence is higher than the resolution (first resolution) of the time series of amplitude in the first feature sequence. The first intermediate feature sequence may be displayed in the input area 3. The user can modify the first intermediate feature sequence displayed in the input area 3 using the operation unit 150.

[0033] Furthermore, the generating unit 13 processes the musical score data D3 and the first intermediate feature sequence using DNNL2 to generate a second intermediate feature sequence whose amplitude changes over time. The resolution of the time series of amplitude in the second intermediate feature sequence is higher than the resolution of the time series of amplitude in the first intermediate feature sequence. The second intermediate feature sequence may be displayed in the input area 3. The user can modify the second intermediate feature sequence displayed in the input area 3 using the operation unit 150.

[0034] Furthermore, the generating unit 13 processes the musical score data D3 and the second intermediate feature sequence using DNNL3 to identify a time series of pitches in the musical score, and generates result data D1 indicating the identified time series of pitches. The resolution (second resolution) of the time series of amplitudes in the second feature sequence indicated by the result data D1 is higher than the resolution of the time series of amplitudes in the second intermediate feature sequence.

[0035] (3) Training equipment Fig. 6 is a block diagram showing the configuration of the training device 20. Fig. 7 is a diagram for explaining an example of the operation of the training device 20. As shown in Fig. 6, the training device 20 includes an extraction unit 21, a generation unit 22, and a construction unit 23. The functions of the extraction unit 21, the generation unit 22, and the construction unit 23 are realized by the CPU 130 in Fig. 1 executing a training program. At least a part of the extraction unit 21, the generation unit 22, and the construction unit 23 may be realized by hardware such as an electronic circuit.

[0036] The extraction unit 21 extracts a reference sound data sequence and an output feature sequence from each of a plurality of reference data D2 stored in the storage unit 140 or the like. The reference sound data sequence includes, for example, a time series of the spectral envelope of the waveform indicated by the corresponding reference data D2 and a time series of the pitch. The output feature sequence is a time series of the features (amplitude) of the waveform corresponding to the reference sound data sequence, and changes over time with a predetermined resolution corresponding to the interval (5 ms). The generation unit 22 generates an input feature sequence from each of the plurality of output feature sequences. In the input feature sequence, the amplitude changes over time with a resolution lower than the resolution of the time series of the amplitude in the output feature sequence.

[0037] Specifically, as shown in FIG. 7, the generating unit 22 extracts a representative value of the amplitude within a predetermined period T including each time point t from the output feature sequence. Note that the interval between two adjacent time points t is, for example, 5 ms, the length of the period T is, for example, 3 s, and each time point t is located, for example, at the center of the period T. In the example of FIG. 8, the representative value of the amplitude for each period T is the maximum value of the amplitude within the period T, but may be other statistical values ​​of the amplitude within the period T. The generating unit 22 generates the input feature sequence by arranging the representative values ​​of the amplitude for the extracted multiple periods T as the amplitudes at multiple time points t in the input feature sequence. The maximum value of the amplitude has the same value for a maximum period of 3 s, and the interval at which the value changes is several tens of times longer than the interval between the time points of 5 ms. In other words, the input feature sequence changes less frequently than the output feature sequence.

[0038] The construction unit 23 prepares a generation model m (untrained or pre-trained) composed of a DNN, and trains the generation model m based on the extracted reference sound data string, the generated input feature string, and the score feature string generated from each reference score data D4 stored in the storage unit 140 or the like. This training constructs a trained model M that has learned the input / output relationship between the input feature string and the score feature string and the reference sound data string. The prepared generation model m may include one DNNL1 as shown in FIG. 4, or may include multiple DNNL1 to L3 as shown in FIG. 5. The construction unit 23 stores the constructed trained model M in the storage unit 140 or the like.

[0039] (4) Sound generation processing Fig. 8 is a flowchart showing an example of sound generation processing by the sound generation device 10 of Fig. 2. The sound generation processing of Fig. 8 is performed by the CPU 130 of Fig. 1 executing a sound generation program stored in the storage unit 140 or the like. First, the CPU 130 determines whether or not the musical score data D3 has been selected by the user (step S1). If the musical score data D3 has not been selected, the CPU 130 waits until the musical score data D3 is selected.

[0040] When the musical score data D3 is selected, the CPU 130 causes the display unit 160 to display the reception screen 1 of Fig. 3 (step S2). In the reference area 2 of the reception screen 1, a reference image 4 based on the musical score data D3 selected in step S1 is displayed. Next, the CPU 130 accepts a first feature sequence in the input area 3 of the reception screen 1 (step S3).

[0041] Next, CPU 130 processes the score feature sequence of score data D3 selected in step S1 and the first feature sequence accepted in step S4 using trained model M to generate result data D1 (step S4). After that, CPU 130 generates an audio signal, which is a waveform in the time domain, from result data D1 generated in step S4 (step S5), and ends the sound generation process.

[0042] (5) Training process Fig. 9 is a flow chart showing an example of a training process by the training device 20 of Fig. 6. The training process of Fig. 9 is performed by the CPU 130 of Fig. 1 executing a training program stored in the storage unit 140 or the like. First, the CPU 130 acquires a plurality of reference data D2 used for training from the storage unit 140 or the like (step S11). Next, the CPU 130 extracts a reference sound data sequence from each of the reference data D2 acquired in step S11 (step S12). In addition, the CPU 130 extracts an output feature sequence (time series of amplitude) from each of the reference data D2 acquired in step S1 (step S13).

[0043] Next, the CPU 130 generates an input feature sequence (a time series of maximum amplitude values) from the output feature sequence extracted in step S3 (step S14). After that, the CPU 130 prepares a generation model m, and trains the generation model m based on the score feature sequence based on the reference score data D4 corresponding to each reference data D2 acquired in step S1, the input feature sequence generated in step S14, and the reference sound data sequence extracted in step S12, thereby having the generation model m machine-learn the input / output relationship between the score feature sequence and the reference input feature sequence, and the reference sound data sequence (step S15).

[0044] Next, CPU 130 determines whether or not sufficient machine learning has been performed for generative model m to learn the input-output relationship (step S16). If the machine learning is insufficient, CPU 130 returns to step S15. Steps S15 to S16 are repeated while changing the parameters until sufficient machine learning is performed. The number of times machine learning is repeated varies depending on the quality conditions that the trained model M to be constructed should satisfy. If sufficient machine learning has been performed, CPU 130 saves it as a trained model M that has learned the input-output relationship between the score feature sequence and the input feature sequence, and the reference sound data sequence through training (step S17), and ends the training process.

[0045] (6) Effects of the embodiment As described above, the sound generation method of this embodiment receives an input of a first feature sequence whose musical features vary over time, processes the first feature sequence using a trained model that has mastered the input / output relationship between an input feature sequence whose features vary over time at a first resolution and a reference sound data sequence corresponding to an output feature sequence whose features vary over time at a second resolution higher than the first resolution, and generates a sound data sequence corresponding to a second feature sequence whose features vary over time at a second resolution, and is realized by a computer.

[0046] According to this method, even if the feature changes in the input first feature sequence are rough, a sound data sequence corresponding to the second feature sequence is generated. In the second feature sequence, the feature changes in detail, and a natural voice is generated from the sound data sequence. Therefore, the user does not need to input a detailed time series of the feature.

[0047] The feature amount at each time point in the input feature amount sequence may be a representative value of the feature amounts within a predetermined period including that time point in the output feature amount sequence.

[0048] The representative value may be a statistical value of the feature amount within a predetermined period in the output feature amount sequence.

[0049] The sound generation method may further present a reception screen 1 on which the first feature sequence is displayed along a time axis, and the first feature sequence may be input using the reception screen 1. In this case, the user can easily input the first feature sequence while visually checking the positions of the features in the first feature sequence on the time axis.

[0050] The training method of this embodiment extracts a reference sound data sequence in which musical features vary over time at a predetermined resolution and an output feature sequence which is a time series of those features from reference data indicating sound waveforms, generates an input feature sequence from the output feature sequence in which features vary over time at a resolution lower than the predetermined resolution, and constructs a trained model which has learned the input / output relationship between the input feature sequence and the reference sound data sequence corresponding to the output feature sequence through machine learning, and is implemented by a computer.

[0051] According to this method, a trained model M is constructed that can generate a sound data sequence corresponding to a second feature sequence in which the features change in detail, even if the changes in the features in the input first feature sequence are rough.

[0052] The input feature sequence may be generated by extracting, as a feature at each time point in the input feature sequence, a representative value of the feature within a predetermined period including that time point in the output feature sequence.

[0053] (7) Other embodiments In the first embodiment, the user inputs the maximum amplitude as the control value to control the generated voice signal, but the embodiment is not limited to this. The control value may be another feature. Below, the sound generating device 10 and the training device 20 according to the second embodiment will be described in terms of commonalities and differences with the sound generating device 10 and the training device 20 according to the first embodiment.

[0054] The sound generating device 10 in this embodiment is similar to the sound generating device 10 in the first embodiment described with reference to FIG. 2, except for the following points. The presentation unit 11 displays a reception screen 1 on the display unit 160 based on the musical score data D3 selected by the user. FIG. 10 is a diagram showing an example of the reception screen 1 in the second embodiment. As shown in FIG. 10, in the reception screen 1 in this embodiment, three input areas 3a, 3b, and 3c are arranged to correspond to the reference area 2, instead of the input area 3 in FIG. 3.

[0055] The user inputs three first feature strings in which the feature (pitch variance in this example) in three parts of the sound corresponding to each note displayed in the reference image 4 changes over time, in the input areas 3a, 3b, and 3c, using the operation unit 150. This allows the first feature strings to be input. As the first feature strings, a time series of pitch variance in the attack part of the sound corresponding to the note is input in the input area 3a, a time series of pitch variance in the sustain part is input in the input area 3b, and a pitch variance in the release part is input in the input area 3c. In the input example of FIG. 10, the pitch variance in the attack part and the release part in the sixth and seventh bars of the musical score is large, and the pitch variance in the sustain part in the eighth and ninth bars is large.

[0056] The generating unit 13 processes the score feature sequence and the first feature sequence based on the score data D3 using the trained model M to generate result data D1. The result data D1 includes a second feature sequence that is a time series of pitches that change at a second resolution. The generating unit 13 may store the generated result data D1 in the storage unit 140 or the like. The generating unit 13 may also generate an audio signal that is a waveform in the time domain based on the frequency domain result data D1, and supply it to a sound system. The generating unit 13 may also display the second feature sequence included in the result data D1 on the display unit 160.

[0057] The training device 20 in this embodiment is similar to the training device 20 in the first embodiment described with reference to Fig. 6, except for the following points. In this embodiment, the pitch time series, which is the output feature sequence to be extracted in step S13 of the training process in Fig. 9, has already been extracted as a part of the reference sound data sequence in the immediately preceding step S12. In step S13, the CPU 130 (extraction unit 21) extracts the amplitude time series in each of the multiple reference data D2 not as an output feature sequence but as an index for separating the sound into three parts.

[0058] In the next step S14, CPU 130 divides the pitch time series (output feature sequence) contained in the reference sound data string into three time series: the attack part of the sound, the release part of the sound, and the body part of the sound between the attack and release parts, based on the amplitude time series, and performs statistical analysis on each of them to obtain a pitch variance time series (input feature sequence) for each part.

[0059] In addition, in steps S15 to S16, the CPU 130 (construction unit 23) repeatedly performs machine learning (training of the generation model m) based on the reference sound data sequence generated from each reference data D2 and the reference score data D4 corresponding to the input feature sequence, thereby constructing a trained model M that has mastered the input / output relationship between the score feature sequence and input feature sequence corresponding to the reference score data, and the reference sound data sequence corresponding to the output feature sequence.

[0060] In the sound generating device 10 according to the present embodiment, the user can roughly input the pitch variance at each time point as the first feature sequence, thereby effectively controlling the pitch change width of the sound generated at that time point, which changes in detail. In addition, by inputting the first feature values ​​for the three parts individually, the pitch change width of the attack part, the body part, and the release part can be individually controlled. Note that the reception screen 1 includes the input areas 3a to 3c, but the embodiment is not limited thereto. The reception screen 1 may not include any one or two of the input areas 3a, 3b, and 3c. Also, in this embodiment, the reception screen 1 may not include the reference area 2. In this embodiment, the sound is controlled by inputting three pitch variance sequences divided into three parts, but it is also possible to input one pitch variance sequence without dividing into three parts to control the entire sound from attack to release. [Explanation of symbols]

[0061] 1...reception screen, 2...reference area, 3, 3a to 3c...input area, 4...reference image, 10...sound generation device, 11...presentation unit, 12...reception unit, 13, 22...generation unit, 14...processing unit, 20...training device, 21...extraction unit, 23...construction unit, 100...processing system, 110...RAM, 120...ROM, 130...CPU, 140...storage unit, 150...operation unit, 160...display unit, 170...bus, D1...result data, D2...reference data, D3...music score data, D4...reference music score data, L1 to L3...DNN, M...trained model, m...generative model

Claims

1. receiving an input of a first feature sequence having a predetermined time resolution in which musical features change over time; a trained model that has learned an input / output relationship between an input feature sequence of the predetermined time resolution, in which the feature sequence varies over time with a first resolution, and a reference sound data sequence corresponding to an output feature sequence of the predetermined time resolution, in which the feature sequence varies over time with a second resolution higher than the first resolution, is used to process the first feature sequence, thereby generating a sound data sequence corresponding to a second feature sequence of the predetermined time resolution, in which the feature sequence varies over time with the second resolution; A computer-implemented method for generating sound.

2. The sound generation method according to claim 1 , wherein the feature at each time point in the input feature sequence is a representative value of the feature within a predetermined period including the time point in the output feature sequence.

3. A method for receiving an input of a first feature sequence in which musical features change over time, a trained model that has learned an input / output relationship between an input feature sequence in which the feature values ​​vary over time at a first resolution and a reference sound data sequence corresponding to an output feature sequence in which the feature values ​​vary over time at a second resolution higher than the first resolution, the trained model processes the first feature sequence to generate a sound data sequence corresponding to a second feature sequence in which the feature values ​​vary over time at the second resolution; the feature quantity at each time point in the input feature quantity sequence is a representative value of the feature quantities within a predetermined period including the time point in the output feature quantity sequence; the representative value is a statistical value of the feature values ​​in the output feature value sequence within the predetermined period; A computer-implemented method for generating sound.

4. A method for inputting a first feature sequence in which musical features change over time, a trained model that has learned an input / output relationship between an input feature sequence in which the feature values ​​vary over time at a first resolution and a reference sound data sequence corresponding to an output feature sequence in which the feature values ​​vary over time at a second resolution higher than the first resolution, the trained model processes the first feature sequence to generate a sound data sequence corresponding to a second feature sequence in which the feature values ​​vary over time at the second resolution; further presenting a reception screen on which the first feature sequence is displayed along a time axis; the first feature sequence is input using the reception screen; A computer-implemented method for generating sound.

5. A reference sound data string having a predetermined time resolution in which musical features change over time with a predetermined precision is extracted from reference data indicating a sound waveform, and an output feature string which is a time series of the features is extracted; generating an input feature sequence of the predetermined time resolution from the output feature sequence, the input feature sequence having the feature values ​​varying over time at a resolution lower than the predetermined resolution; constructing a trained model that has learned an input / output relationship between the input feature sequence and a reference sound data sequence corresponding to the output feature sequence through machine learning; A computer-implemented training method.

6. The training method according to claim 5 , wherein the input feature sequence is generated by extracting, as the feature at each time point in the input feature sequence, a representative value of the feature within a predetermined period including that time point in the output feature sequence.

7. a receiving unit for receiving an input of a first feature sequence having a predetermined time resolution in which musical features change over time; a generation unit that processes the first feature sequence using a trained model that has acquired an input / output relationship between an input feature sequence of the predetermined time resolution, in which the feature sequence varies over time with a first resolution, and a reference sound data sequence corresponding to an output feature sequence of the predetermined time resolution, in which the feature sequence varies over time with a second resolution higher than the first resolution, to generate a sound data sequence corresponding to a second feature sequence of the predetermined time resolution, in which the feature sequence varies over time with the second resolution.

8. an extracting unit that extracts, from reference data indicating a sound waveform, a reference sound data sequence having a predetermined time resolution in which musical features change over time with a predetermined precision, and an output feature sequence that is a time series of the features; a generation unit for generating an input feature sequence of the predetermined time resolution from the output feature sequence, the input feature sequence having a time-varying resolution lower than the predetermined resolution; and a construction unit that constructs a trained model that has learned the input / output relationship between the input feature sequence and a reference sound data sequence corresponding to the output feature sequence through machine learning.

9. On one or more computers, receiving an input of a first feature sequence having a predetermined time resolution in which musical features change over time; A sound generation program that performs a step of processing the first feature sequence using a trained model that has acquired the input / output relationship between an input feature sequence of the predetermined time resolution, in which the feature sequence varies over time with a first resolution, and a reference sound data sequence corresponding to an output feature sequence of the predetermined time resolution, in which the feature sequence varies over time with a second resolution higher than the first resolution, to generate a sound data sequence corresponding to a second feature sequence of the predetermined time resolution, in which the feature sequence varies over time with the second resolution.

10. On one or more computers, A reference sound data string having a predetermined time resolution in which musical features change over time with a predetermined precision is extracted from reference data indicating a sound waveform, and an output feature string which is a time series of the features is extracted; generating an input feature sequence of the predetermined time resolution from the output feature sequence, the input feature sequence having the feature values ​​varying over time at a resolution lower than the predetermined resolution; A training program that performs a step of constructing a trained model that has acquired an input / output relationship between the input feature sequence and a reference sound data sequence corresponding to the output feature sequence through machine learning.

Images