Signal generation method, display control method, and program

Abstract

A signal generation method according to an embodiment comprises: acquiring intermediate feature data corresponding to a predetermined time step by providing, to a first trained model, a first parameter in a predetermined time range including ranges before and after the predetermined time step of sound control data including the first parameter and a second parameter for controlling a generated sound in a plurality of time steps corresponding to the passage of time; updating the intermediate feature data when the value of the first parameter has been changed; and generating a sound signal in accordance with data obtained by providing the second parameter and the intermediate feature data to a second trained model when a reproduction start instruction has been received.

Description

Signal generation method, display control method and program

[0001] The present invention relates to a technique for generating an audio signal.

[0002] A technique for synthesizing sound using a deep neural network (DNN) is known. As shown in Patent Document 1, a technique for using information after the timing of the sound to be synthesized has also been developed in order to further improve the quality of the synthesized sound.

[0003] International Publication No. 2021 / 251364

[0004] A DAW (Digital Audio Workstation) used in music production functions as software that generates sound according to the playback position. In addition to generating sound, a DAW also provides a user interface that makes it easy to edit various data.

[0005] A typical DAW generates a sound signal using waveform data, MIDI (Musical Instruments Digital Interface) data, etc. When using data for controlling the generated sound (sound control data) such as MIDI data, the sound signal is generated using information corresponding to the playback position (current information) and information corresponding to a time before the playback position (past information). Therefore, a technology for generating a synthetic sound using information corresponding to a time after the playback position (future information) cannot be directly applied to a DAW. In the following description, a technology for generating a synthetic sound using future information may be simply referred to as a sound synthesis technology.

[0006] When using such sound synthesis technology in a DAW, waveform data corresponding to the synthesized sound is generally generated in advance by an external function such as a plug-in. By using the waveform data generated in advance by the external function, the DAW can generate a sound signal corresponding to the playback position in the same way as with normal waveform data.

[0007] On the other hand, because waveform data needs to be generated in advance, it is difficult to edit while listening to the sound, which results in a decrease in music production efficiency.In addition, not only sound synthesis technology that uses future information, but also the convenience of editing music in real time in a DAW greatly affects the efficiency of music production.

[0008] One of the objects of the present invention is to improve the efficiency of music production in a software environment that generates sounds according to playback positions.

[0009] In one embodiment, a signal generation method includes: obtaining intermediate feature data corresponding to a predetermined time step by providing a first trained model with sound control data including a first parameter and a second parameter for controlling generated sound at a plurality of time steps over time, the first parameter being within a predetermined time range including before and after the predetermined time step; updating the intermediate feature data when the value of the first parameter is changed; and generating a sound signal according to data obtained by providing the second parameter and the intermediate feature data to a second trained model when a playback start instruction is received.

[0010] In one embodiment, a signal generation method includes displaying a first area including an image corresponding to a first parameter, the first parameter including phoneme information or pitch information, for controlling a generated sound of a first track, a second area including an image corresponding to a second parameter for controlling the generated sound of the first track, and a third area including an image corresponding to a third parameter for controlling the generated sound of a second track, and upon receiving a playback start instruction, displaying images corresponding to a current playback position in the first area and the third area, generating a first sound signal of the first track and a second sound signal of the second track corresponding to the playback position, and outputting a mixed sound of the first sound signal and the second sound signal. Upon receiving an instruction to change the second parameter after the playback start instruction, the changed second parameter is reflected in the first sound signal at the playback position.

[0011] A display control method in one embodiment includes determining a control value for controlling a generated sound over time based on an instruction value from an operator and a setting value determined in advance in a time series; generating a sound signal based on sound control data for controlling the generated sound over time and the control value; and displaying a setting image including a first image area for indicating the instruction value and a second image area adjacent to the first image area for indicating the control value.

[0012] A display control method in one embodiment includes displaying in a predetermined display area a first image corresponding to a sound signal generated based on sound control data including a first parameter and a second parameter for controlling the generated sound over time, and a second image including an image related to the second parameter; changing the position of the second image in the display area in accordance with an input instruction; and, when changing the position of the second image, moving the first image to an area that satisfies predetermined conditions among a plurality of areas divided into the display area based on the second image.

[0013] According to the present invention, it is possible to improve the efficiency of music production in a software environment that generates sounds according to the playback position.

[0014] FIG. 1 is a block diagram showing a data processing device according to an embodiment; FIG. 2 is an example of a music editing screen according to an embodiment; FIG. 3 is a functional block diagram showing a sound output unit according to an embodiment; FIG. 4 is a functional block diagram showing a sound generation unit according to an embodiment; FIG. 5 is a flowchart showing a signal generation method according to an embodiment; FIG. 6 is a flowchart showing a signal generation method according to an embodiment; FIG. 7 is a flowchart showing sound playback processing according to an embodiment; FIG. 8 is a flowchart showing a signal generation method according to an embodiment; FIG. 9 is an example of a music editing screen according to an embodiment; FIG. 10 is an example of a setting image according to an embodiment; FIG. 11 is a flowchart showing a display control method according to an embodiment; FIG. 11 is an example of an editing window according to an embodiment; FIG. 12 is an example of an editing window according to an embodiment; FIG. 13 is a flowchart showing a display control method according to an embodiment; FIG. 14 is a diagram for explaining the positional relationship between a setting window image and a sound-related image according to an embodiment; FIG. 15 is a diagram for explaining the positional relationship between a setting window image and a sound-related image according to an embodiment.

[0015] An embodiment of the present invention will be described in detail below with reference to the drawings. The embodiments described below are merely examples, and the present invention should not be construed as being limited to these embodiments. In the drawings referred to in this embodiment, identical or similar reference numerals are used to designate identical parts or parts having similar functions, and repeated explanations thereof may be omitted. For clarity of explanation, the drawings may be illustrated schematically, with dimensional ratios different from actual ratios and some components omitted from the drawings.

[0016] [Overview] A data processing device according to one embodiment of the present invention is a device equipped with a computer, such as a desktop terminal, a laptop terminal, a smartphone, or a tablet terminal. The data processing device executes an installed application program to provide a user with a DAW for editing music, etc. The DAW generates sound signals based on data set corresponding to multiple tracks, such as sound control data, waveform data, and other information for generating sound.

[0017] The sound control data is data including information for controlling the generated sound at multiple time steps over time, such as MIDI data. The waveform data is data obtained by sampling a sound signal waveform, such as WAV data or MP3 data. The DAW reads information corresponding to the playback position from the data of each track and generates a sound signal corresponding to the playback position. By generating sound signals corresponding to multiple tracks in synchronization, the sounds corresponding to each track are output as a mixed sound (mixed sound).

[0018] At least information corresponding to the playback position, i.e., current information, is used to generate the sound signal. When a sound signal is generated based on sound control data, information corresponding to a time before the playback position, i.e., past information, also affects the sound signal generated corresponding to the playback position. In this example, sound control data for utilizing sound synthesis technology that further uses information corresponding to a time after the playback position, i.e., future information, among the sound control data, can also be assigned to at least one of the multiple tracks.

[0019] According to this sound synthesis technology, in this example, a sound that imitates a human voice can be generated at a set pitch. In the following description, the sound that imitates a human voice is referred to as a singing synthesized sound. This sound synthesis technology can also be used to generate synthetic musical instrument sounds, not just singing synthesized sounds. In the following description, the process of generating a singing synthesized sound is a process using sound synthesis technology that uses past information, present information, and future information, unless otherwise specified. A method for realizing this process can be understood by those skilled in the art based on the disclosure (particularly the disclosure of the second embodiment) in the above-mentioned Patent Document 1 (WO 2021 / 251364). A data processing device will be described below.

[0020] [Data Processing Device] FIG. 1 is a block diagram showing a data processing device according to one embodiment. The data processing device 1 includes a control unit 11, a storage unit 13, a display unit 15, an operation unit 17, an interface 21, and a communication unit 23. The control unit 11 is an example of a computer equipped with a processor such as a CPU and a storage unit such as RAM. The control unit 11 executes a program 13a stored in the storage unit 13 using the CPU, causing the data processing device 1 to realize functions for executing various processes. The various processes include processes for realizing DAW-related functions. These processes include processes related to a signal generation method and a display control method, which will be described later. Therefore, the control unit 11 also functions as a signal generation unit for executing the signal generation method and a display control unit for executing the display control method.

[0021] The display unit 15 is a display having a display area that displays various screens in accordance with the control of the control unit 11. The various screens include a music editing screen, which will be described later. The operation unit 17 is a device for inputting user instructions and includes operators that output operation signals to the control unit 11 in accordance with the input operations. The operators are, for example, slider-type operators such as faders, rotary-type operators such as knobs and rotary encoders, and button-type operators such as switches. The operators may be provided as operator images displayed on the display unit 15. The operation signals include information regarding the instruction values ​​of the operators.

[0022] The interface 21 includes a module for communicating with an external device via wired communication or wireless communication such as infrared communication or short-range wireless communication. In this example, the external device may be an electronic musical instrument or sound source device that generates a sound signal, or may be a speaker or headphones that output a sound signal generated by the control unit 11 to the outside. The interface 21 is used for communication without going through a network. The communication unit 23 is a communication module that connects to a network under the control of the control unit 11 and communicates with an external device connected to the network.

[0023] The storage unit 13 is a storage device such as a non-volatile memory, a hard disk drive, etc. The storage unit 13 stores a program 13a executed by the control unit 11 and various data required for the DAW realized by this program 13a. For example, the storage unit 13 stores music data 13b, a first trained model 120, and a second trained model 130.

[0024] The program 13a is downloaded from an external device via a network such as the Internet, and stored in the storage unit 13, whereby it is installed in the data processing device 1. The program 13a may be provided in a state recorded on a non-transitory computer-readable recording medium (e.g., a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, a semiconductor memory, etc.). In this case, the data processing device 1 only needs to be equipped with a device for reading this recording medium. The storage unit 13 can also be considered an example of a recording medium.

[0025] Similarly, the song data 13b, the first trained model 120, and the second trained model 130 may be downloaded from an external device via a network and stored in the storage unit 13, or may be provided in a state recorded on a non-transitory computer-readable recording medium. The song data 13b is data stored in the storage unit 13 for each song, and includes sound control data CDa, CDb, waveform data WD, intermediate feature data MV, etc. These data may be generated during the process of music production using a DAW, or may be provided to the data processing device 1 in advance in a manner similar to that of the program 13a.

[0026] The sound control data CDa and CDb are, for example, MIDI-format data and include multiple types of parameters for controlling the generated sound at multiple time steps over time. In this example, the sound control data CDa is data used when generating a synthesized singing sound. The sound control data CDb is data used when generating an instrument sound. The multiple parameters in the sound control data CDa are divided into at least parameters belonging to a first group and parameters belonging to a second group. The parameters in the first group include parameters corresponding to future information and may be referred to as first parameters P1. The parameters in the second group include parameters corresponding to present information and may be referred to as second parameters P2. The second parameters may include parameters related to past information. The information included in each of the first parameters P1 and the second parameters P2 described below is merely exemplary, and combinations other than these may also be used.

[0027] The first parameter P1 includes information determined for each note defined in time series. The first parameter P1 may be, for example, phoneme information, pitch information, period information, intensity information, etc. The first parameter P1 includes at least phoneme information or pitch information. The phoneme information includes information for generating sounds from characters, such as lyrics for singing, corresponding to each note. If the sound control data CDa is data for generating instrument sounds, phoneme information may not be used, or other information, such as information for reproducing the playing technique of an instrument, may be used instead of lyrics. The pitch information includes information for specifying the pitch corresponding to each note. The period information includes information for specifying the start position and duration (note value) of each note. The period information may be defined by multiple pieces of information, such as note-on and note-off. The intensity information includes information for specifying the velocity corresponding to each note.

[0028] The second parameter P2 includes information for controlling the generated sound at each time step. The second parameter P2 is, for example, power information, transpose information, formant information, etc. The power information includes information for specifying the dynamics of the sound; for example, a larger setting value reproduces a shouting voice sound, and a smaller setting value reproduces a whispering voice sound. The transpose information includes information for raising or lowering the pitch of the sound. The formant information includes information for specifying the voice quality; for example, a larger setting value reproduces a sound closer to a male voice, and a smaller setting value reproduces a sound closer to a female voice. The second parameter P2 may also be information for specifying the clarity, brightness, breathiness, mouth opening, etc. of the voice.

[0029] The setting value of the information included in the second parameter P2 may be used as is as a value (control value) for controlling the generated sound, or may be updated or corrected in accordance with the instruction information and used as the control value. The instruction information includes information corresponding to the instruction value of the operator. That is, the control value for the second parameter P2 may be determined based on the setting value and the instruction value. In this case, since the instruction value is related to the setting value of the information included in the second parameter P2, the instruction information can also be considered an example of the second parameter P2. For example, if the control value of power information is determined based on the instruction value, the instruction information can also be considered power information.

[0030] The plurality of parameters in the sound control data CDb include at least parameters belonging to a third group. The parameters in the third group include parameters corresponding to current information and may be referred to as third parameters P3. The third parameters P3 may also include parameters related to past information.

[0031] The third parameter P3 includes information determined for each sound defined in a time series and information for controlling the generated sound at each time step. The former information is similar to the first parameter P1 described above, but is used as present or past information rather than as future information. The latter information is similar to the second parameter P2 described above. The third parameter P3 is, for example, pitch information, period information, intensity information, etc. The third parameter P3 may also include transpose information, etc.

[0032] If the sound control data CDb is data for generating the sound of a percussion instrument such as a drum, it may include information specifying the type of percussion instrument, such as a snare, tom, or cymbal. Because percussion instruments may not require pitch information, type information may be included in the third parameter P3 instead of pitch information to specify the type of percussion instrument. While pitch information may be specified by a note number, type information is also specified by a note number. Therefore, even if the data is in the form of pitch information, it may not be information indicating pitch.

[0033] The waveform data WD is data obtained by sampling a sound signal waveform and indicates waveform data corresponding to each time step. The intermediate feature data MV is data generated by the first trained model 120 and is, for example, vector data corresponding to each time step.

[0034] The first trained model 120 and the second trained model 130 are statistical estimation models used when generating synthesized singing sounds, and in this example, a known machine learning model using DNN is used. Different types of models may be applied to the first trained model 120 and the second trained model 130. The machine learning model may be, for example, a machine learning model using a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or the like. Details of the first trained model 120 and the second trained model 130 will be described later.

[0035] [Music Editing Screen] The music editing screen will be described. The music editing screen is a screen provided by the DAW implemented in the data processing device 1.

[0036] 2 shows an example of a music editing screen in one embodiment. In the example shown in FIG. 2, the music editing screen WS includes a track area AT, an editing area AE, and an operation image area AS. In all cases, the horizontal direction indicates a temporal position, i.e., a time step. When playback processing is being performed, an image GT indicating the playback position is displayed in the track area AT and editing area AE.

[0037] The track area AT displays an image corresponding to the data corresponding to each track. In this example, it includes areas AT1, AT2, AT3, and AT4 corresponding to the first through fourth tracks. Sound control data CDa is assigned to the first track, sound control data CDb is assigned to the second and third tracks, and waveform data WD is assigned to the fourth track. Therefore, an image corresponding to the first parameter P1 in the sound control data CDa is displayed in area AT1 (first area). An image corresponding to the third parameter P3 in the sound control data CDb is displayed in areas AT2 and AT3 (third areas).

[0038] In this example, the editing area AE includes a piano roll area AP and a setting value area AC. The piano roll area AP is an area (first area) in which information on a first parameter P1 is displayed when sound control data CDa is assigned to a track selected in the track area AT, and a sound control image corresponding to the first parameter P1 is displayed. The piano roll area AP is an area (third area) in which information on a third parameter P3 is displayed when sound control data CDb is assigned to a track selected in the track area AT, and a sound control image corresponding to the third parameter P3 is displayed.

[0039] The sound control image is, for example, an image GL that shows phoneme information and an image GR that shows pitch information and period information. Other information such as intensity information may be further expressed by the color of the image GR, the width of the band, etc. The image GR includes information determined for each sound defined in time series, and in the case of sound control data CDa, it includes information on the first parameter P1, and in the case of sound control data CDb, it includes information on the third parameter P3. When waveform data WD is assigned to the track selected in the track area AT, the piano roll area AP is replaced with an editing area AE, which is an area for displaying waveforms.

[0040] The setting value area AC (second area) displays an image GP indicating the setting value of information for controlling the generated sound at each time step. That is, the image GP includes information on the second parameter P2 in the case of sound control data CDa, and information on the third parameter P3 in the case of sound control data CDb. The setting value area AC may also display an image related to information on the first parameter P1 that is not displayed in the editing area AE, in conjunction with the editing area AE.

[0041] When the user changes the position of an image, such as image GR or image GP, on the music editing screen WS according to the information of each data item by instructing a change operation on the image via the operation unit 17 or interface 21, the information of the corresponding data item is changed. Therefore, the image displayed in the piano roll area AP can be said to be an image related to a change in the first parameter P1 or the third parameter P3. The image displayed in the setting value area AC can be said to be an image related to a change in the second parameter P2 or the third parameter P3.

[0042] In this example, the operation image area AS includes operation images GC1, GC2, and GC3 for receiving user instructions. The operation image GC1 is an image modeled after a button for receiving instructions such as to start playback or stop playback. The operation image GC2 is an image modeled after a button for receiving settings for the playback position movement speed, i.e., tempo information. The operation image GC3 is an image modeled after a slider for receiving volume settings.

[0043] [Sound Output Unit] The function of the sound output unit that is realized by the control unit 11 executing the program 13a will be described.

[0044] 3 is a functional block diagram showing a sound output unit according to one embodiment. Sound output unit 1000 includes sound generation units 100a, 100b, and 100c, a data editing unit 200, a read position determination unit 300, and a sound mixing unit 500. The entire configuration of sound output unit 1000 may be realized as software by executing a program, and at least a portion of the configuration may be realized by hardware.

[0045] The data editing unit 200 edits the information of the sound control data CDa, CDb in response to a user instruction to update the data. The data editing unit 200 may also edit the waveform data WD in response to a user instruction. The read position determination unit 300 generates a read position TR in response to a playback instruction and the value of tempo information. The tempo information may be a value set by the operation image GC2 or may be a value set in the sound control data CDa, CDb. In this example, the read position TR is expressed by the position of a time step.

[0046] The sound generation unit 100a generates a sound signal Wa (first sound signal) corresponding to the track to which the sound control data CDa is assigned. The sound generation unit 100a generates intermediate feature data MV in the process of generating the sound signal Wa. The sound generation unit 100b generates a sound signal Wb (second sound signal) corresponding to the track to which the sound control data CDb is assigned. The sound generation unit 100c generates a sound signal Wc corresponding to the track to which the waveform data WD is assigned. The sound generation units 100a, 100b, and 100c read data according to the read position TR and generate the sound signals Wa, Wb, and Wc. In this way, as the playback position advances and the read position TR also advances, the sound signals Wa, Wb, and Wc corresponding to the playback position are output. As the playback position advances according to the read position TR, the position of the image GT displayed on the music editing screen WS described above also moves according to the read position TR.

[0047] The sound signal Wa includes information of the first parameter P1, and therefore includes information after the playback position, i.e., future information. The sound signal Wb includes information of the third parameter P3, and therefore includes information at or before the playback position, i.e., present information or past information.

[0048] The sound mixing unit 500 mixes the sound signals (sound signals Wa, Wb, and Wc in the example of FIG. 3) generated on each track, and outputs a mixed sound signal Wm.

[0049] 4 is a functional block diagram showing a sound generation unit in one embodiment. Fig. 4 shows a detailed configuration of the sound generation unit 100a. The sound generation unit 100a includes the first trained model 120 and the second trained model 130 described above, and further includes a waveform synthesis unit 140. The first trained model 120 includes an encoding model 121 and a first generation model 125.

[0050] The first learned model 120 is a model trained to generate intermediate feature data MV at a predetermined time step when provided with first parameters P1 and a tempo Z2 in a predetermined time range including before and after the predetermined time step. The tempo Z2 is a value of tempo information. The second learned model 130 includes an encoding model 131 and a second generation model 135. When provided with second parameters P2 and intermediate feature data MV at a predetermined time step, the first learned model 120 is a model trained to generate sound feature data F at the time step. At this time, sound feature data F from an earlier time step may also be provided.

[0051] As described above, the configuration of the sound generation unit 100a is similar to the configuration of the second embodiment shown in Fig. 10 in Patent Document 1 (WO 2021 / 251364). Therefore, although detailed description will be omitted, those skilled in the art can implement it by referring to the disclosure in Patent Document 1.

[0052] The encoding model 121 is a statistical estimation model for generating symbol data B at each time step based on a first parameter P1 of the sound control data CDa. Information about the first parameter P1 used in the encoding model 121 includes, for example, phoneme information. The first generation model 125 is a statistical estimation model for generating encoded data E at each time step based on the symbol data B, a tempo Z2, and the first parameter P1. Information about the first parameter P1 used in the first generation model 125 includes, for example, pitch information. The encoding model 121, the first generation model 125, the first parameter P1, the tempo Z2, the symbol data B, and the encoded data E correspond to the encoding model (21), the encoded data acquisition unit (22), the music data (D), the tempo (Z2), the symbol data (B), and the encoded data (E), respectively, in Patent Document 1.

[0053] The encoded data E at each time step is generated using information on the first parameter P1 for a predetermined time range including before and after the corresponding time step. That is, the encoded data E is generated using not only current information but also at least future information. The encoded data E corresponding to multiple time steps is acquired as intermediate feature data MV. The acquired intermediate feature data MV is stored in the storage unit 13. If the intermediate feature data MV has not yet been generated, the control unit 11 first provides information on the first parameter P1 to the first trained model 120. Processing in the first trained model 120 is performed using the provided information, and encoded data E is generated for each time step included in the period in which the first parameter P1 exists.

[0054] After the intermediate feature data MV is stored, when the data editing unit 200 updates the information of the first parameter P1 or the tempo information in the sound control data CDa, the control unit 11 provides the updated information to the first trained model 120. Processing in the first trained model 120 is performed using the provided information, and the intermediate feature data MV is updated using newly generated encoded data E. At this time, the entire intermediate feature data MV may be updated, or information may be provided to the first trained model 120 so that only the intermediate feature data MV in the time range affected by the updated information is updated.

[0055] The encoding model 131 is a statistical estimation model for generating encoded control data C at each time step based on the second parameter P2 of the sound control data CDa. The above-mentioned second generation model 135 reads out encoded data E at a time step corresponding to the read position TR from the intermediate feature data MV. The second generation model 135 is a statistical estimation model for generating sound feature data F at a time step corresponding to the read position TR based on the read encoded data E, control data C, and sound feature data F generated in a past time step (a time step before the time step corresponding to the read position TR).

[0056] The encoding model 131, the second generation model 135, the second parameter P2, the control data C, and the sound feature data F correspond to the generation model 32, the generation model 40, the instruction value Z1, the control data C, and the acoustic feature data F in Patent Document 1, respectively. Here, the second parameter P2 is information stored in the storage unit 13, while the instruction value Z1 in Patent Document 1 is determined by a user instruction. The second parameter P2 provided to the second trained model 130 is a control value and is information that can be updated by a user instruction. Therefore, the second parameter P2 can be treated in the same way whether it is information updated in real time by a user instruction or predetermined information.

[0057] The value (control value) of the second parameter P2 provided to the second trained model 130 may be a value obtained by correcting the value (set value) of the second parameter P2 of the sound control data CDa stored in the storage unit 13 by the instruction value of the instruction information. In this case, the updated sound control data CDa does not need to be stored in the storage unit 13. The value (control value) of the second parameter P2 provided to the second trained model 130 may be the value (set value) of the second parameter P2 of the sound control data CDa stored in the storage unit 13, or may be the instruction value of the instruction information.

[0058] When the user instructs to start playback, the control unit 11 provides the second parameter P2 at the time step corresponding to the read position TR and the encoded data E of the intermediate feature data MV corresponding to that time step to the second trained model 130. Data is provided to the first trained model 120 when the information of the first parameter P1 is updated, but this data is not provided to the second trained model 130 when the information of the second parameter P2 is updated. Processing in the second trained model 130 is performed using the provided information, and sound feature data F indicating the features of the sound signal at each time step is output.

[0059] The waveform synthesis unit 140 corresponds to the waveform synthesis unit (50) in Patent Document 1, and generates a sound signal Wa by synthesizing sound signals at each time step according to the sound feature data F. In this way, as the playback position advances and the read position TR also advances, the sound generation unit 100a outputs a sound signal Wa according to the playback position.

[0060] In the sound generation unit 100a, when the information of the first parameter P1 is updated, the intermediate feature data MV is updated, but when the information of the second parameter P2 is updated, the intermediate feature data MV is not updated. On the other hand, after an instruction to start playback is given, when the information of the second parameter P2 is updated, information reflecting the update is provided in real time to the second trained model 130. Therefore, the updated information of the second parameter P2 is also reflected in the generated sound signal Wa.

[0061] If the information of the first parameter P1 is updated after an instruction to start playback is given, the intermediate feature data MV may be updated after playback ends, or the intermediate feature data MV may be updated even during playback. In the latter case, if the position is a certain distance or more from the playback position, the updated information regarding the first parameter P1 is also reflected in the sound signal Wa that is generated when the playback position is reached.

[0062] [Signal Generation Method (DAW)] A signal generation method executed in the control unit 11 will be described. The signal generation method described here starts when the program 13a is executed. First, the overall flow of the DAW will be described.

[0063] 5 is a flowchart showing a signal generation method according to one embodiment. The control unit 11 displays the music editing screen WS (step S10). As described above, the music editing screen WS includes a track area AT and an editing area AE. For a track to which sound control data CDa is assigned, the track area AT displays an image corresponding to a first parameter P1 for controlling the generated sound. For a track to which sound control data CDb is assigned, the track area AT displays an image corresponding to a third parameter P3 for controlling the generated sound.

[0064] The editing area AE includes a piano roll area AP and a setting value area AC. For a track to which sound control data CDa is assigned, the piano roll area AP is used as an area for changing the information of the first parameter P1 by displaying an image related to a change in the first parameter P1. For a track to which sound control data CDa is assigned, the setting value area AC is used as an area for changing the information of the second parameter P2 by displaying an image related to a change in the second parameter P2. The editing window described below is also an example of displaying an image related to a change in the second parameter P2. For a track to which sound control data CDb is assigned, the piano roll area AP and the setting value area AC are used as areas for changing the information of the third parameter P3 by displaying an image related to a change in the third parameter P3.

[0065] The control unit 11 generates music data 13b by editing various parameter information in response to user instructions (step S20). The control unit 11 waits for a user instruction to start playback (step S30; No). Upon receiving the instruction to start playback (step S30; Yes), the control unit 11 determines a read position TR so as to advance the playback position by time steps (step S40). If the read position TR reaches the end position of music data 13b (step S50; Yes), the control unit 11 waits again for a playback start instruction (step S30; No).

[0066] If the read position TR has not reached the end position of the music data 13b (step S50; No), the control unit 11 generates a sound signal corresponding to each track (step S60), generates a mixed sound of the sound signals corresponding to each track (step S70), returns to the processing of step S40, and determines the read position TR by advancing the playback position.

[0067] [Signal Generation Method (Sound Generation Unit 100a)] A signal generation method executed by the sound generation unit 100a for generating sound for a track to which sound control data CDa is assigned will be described. When the sound control data CDa is assigned, the following process is started.

[0068] 6 is a flowchart showing a signal generation method according to one embodiment. The control unit 11 reads the sound control data CDa (step S101) and generates intermediate feature data MV (step S103). The control unit 11 waits for a user instruction to start playback or an instruction to edit parameters (steps S105; No, S107; No). This state is referred to as a standby process in the description of FIG. 6.

[0069] During the standby process, if a parameter editing instruction is received (step S107; Yes), the control unit 11 updates the sound control data CDa in accordance with the instruction (step S109). If the first parameter P1 has not been edited (step S111; No), the control unit 11 returns to the standby process. If the first parameter P1 has been edited (step S111; Yes), the control unit 11 updates the intermediate feature data MV (step S113) and returns to the standby process. If a playback start instruction is received during the standby process (step S105; Yes), the control unit 11 starts the sound playback process (step S200) and returns to the standby process again.

[0070] 7 is a flowchart showing the sound reproduction process according to an embodiment. The sound reproduction process may be executed in parallel with the processes from step S105 to step S113, or the process may be prevented from proceeding beyond the standby process while the sound reproduction process is being executed.

[0071] The control unit 11 determines a read position TR so as to advance the playback position by each time step (step S201). If the read position TR reaches the end position of the music data 13b (step S203; Yes), the control unit 11 terminates the playback process. If the read position TR has not reached the end position of the music data 13b (step S203; No), the control unit 11 reads out the sound control data CDa and intermediate feature data MV corresponding to the time step corresponding to the read position TR (step S209) unless the second parameter P2 has been edited (step S205; No). If the second parameter P2 has been edited (step S205; Yes), the control unit 11 updates the sound control data CDa (step S207) and executes the process of step S209.

[0072] The control unit 11 generates sound feature data F using the read data (step S211), generates a sound signal Wa (step S213), returns to the processing of step S201, and determines the read position TR by advancing the playback position.

[0073] In this way, when the information of the first parameter P1 is updated, the sound generation unit 100a performs processing using future information regardless of the playback position to generate intermediate feature data MV, and performs processing not using future information according to the playback position. This allows the sound generation unit 100a to generate a sound signal Wa that reflects updated information, even when using sound synthesis technology including future information. That is, the sound signal Wa reflects the information of the first parameter P1 and the second parameter P2 after editing. In particular, even after a playback start instruction is issued, the sound signal Wa may reflect the information of the second parameter P2 and the information of the first parameter P1. This improves music production efficiency in a software environment that generates sound according to the playback position, such as a DAW. When a user changes the second parameter P2 using the user interface provided on the music editing screen WS shown in FIG. 9 , the changed second parameter P2 is immediately reflected in the generation of the sound signal Wa, even after playback has started. This allows the user to experience checking the sound generated by sound synthesis technology including future information in real time.

[0074] [Signal Generation Method (Sound Generation Unit 100b)] The following describes a signal generation method executed by the sound generation unit 100b for generating sound for a track to which sound control data CDb is assigned. When the sound control data CDb is assigned and an instruction to start playback is received, the process described below begins.

[0075] FIG. 8 is a flowchart showing a signal generation method according to an embodiment. The control unit 11 determines a read position TR so as to advance the playback position by each time step (step S301). If the read position TR reaches the end of the music data 13b (step S303; Yes), the control unit 11 terminates the process. If the read position TR has not reached the end of the music data 13b (step S303; No), the control unit 11 reads sound control data CDb (corresponding to the information of the third parameter P3) corresponding to the time step corresponding to the read position TR (step S309) unless the parameter has been edited (step S305; No). If the parameter has been edited (step S305; Yes), the control unit 11 updates the sound control data CDb (step S307) and executes the process of step S309. The control unit 11 generates a sound signal Wb using the read data (step S313), returns to the process of step S301, and determines a read position TR by advancing the playback position. In this way, even after the reproduction start instruction is given, the information of the third parameter P3 after the change due to editing is reflected in the sound signal Wb.

[0076] [Edit Window] An edit window that can be displayed by inputting a predetermined operation on the music edit screen WS will be described.

[0077] Fig. 9 is an example of a music editing screen in one embodiment. The music editing screen WS shown in Fig. 9 displays an editing window WE under the control of the control unit 11. An editing window WE can be displayed for each track. Therefore, multiple editing windows WE may be displayed on the music editing screen WS. A sound-related image GV (first image) and a setting window image GS (second image) are displayed in the editing window WE. The setting window image GS is displayed as if a smaller window were opened inside the editing window WE.

[0078] The sound-related images GV are images corresponding to sound signals, and include an image GRs corresponding to the image GR in the music editing screen WS and an image GF indicating the pitch of the waveform. Although omitted from FIG. 9 , the sound-related images GV may be displayed with an image simulating the waveform itself, centered around the image GF. In this example, the editing window WE also displays an image GTs indicating the playback position. The position of the image GTs is fixed in the editing window WE. Therefore, as the playback position advances, the sound-related images GV relatively flow in the leftward direction AR. Therefore, the sound-related images GV are displayed in the editing window WE, expanding horizontally (left and right).

[0079] In this example, the setting window image GS includes operation images GC4, GC5, and GC6. The operation image GC4 is an image modeled after a button for transposing a key. The operation image GC5 is an image modeled after a slider for changing various setting values ​​from the reference values. The operation image GC6 is an image modeled after a slider for fine-tuning the sound generation timing. In this example, the setting window image GS further includes a setting image 800. The setting image 800 includes an image related to the second parameter P2.

[0080] 10 is an example of a setting image according to one embodiment. The setting image 800 includes an arc region 810 (first image region) for indicating an instruction value and an arc region 820 (second image region) for indicating a control value. The arc regions 810 and 820 are disposed adjacent to each other, with the arc region 810 positioned outside the arc region 820. In this example, both regions are defined as regions along concentric arcs. The start points SP of the arc regions 810 and 820 are positions indicating minimum values ​​and are adjacent to each other. The end points EP of the arc regions 810 and 820 are positions indicating maximum values ​​and are adjacent to each other.

[0081] In the arc region 810, an image CA indicating the instruction value is displayed extending from the start point SP. The image CA is displayed in the arc region 810 within a range according to the ratio of the instruction value to the maximum possible value of the instruction value. If the ratio of the instruction value is 100%, the image CA will be an image from the start point SP to the end point EP. The instruction value is indicated by the position of the end of the image CA, but may be displayed in other display forms as long as it is indicated by the position in the arc region 810.

[0082] In the arc region 820, an image LA indicating the control value is displayed extending from the start point SP. The image LA is displayed in the arc region 820 within a range according to the ratio of the control value to the maximum possible value of the control value. If the control value ratio is 100%, the image LA will be an image extending from the start point SP to the end point EP. The control value is indicated by the position of the end of the image LA, but may be displayed in other display forms as long as it is indicated by the position in the arc region 820.

[0083] As described above, the instruction value corresponds to the instruction value of the operator. The control value corresponds to a value that is actually reflected in the generation of the sound signal Wa, such as a value obtained by correcting the setting value of the second parameter P2 by the instruction value. That is, in a track that uses the sound generation unit 100a, this control value corresponds to the value of the second parameter P2 provided to the second trained model 130. In a track that uses the sound generation unit 100b, this control value may be the value of the third parameter P3.

[0084] In this example, the setting image 800 includes an indication value image NA and a type image PA, which are surrounded by arc regions 810 and 820. The indication value image NA is an image that indicates the percentage of the indication value described above. The type image PA is an image that indicates the type of information selected from the second parameters P2. Here, the type image PA indicates that power information has been selected from the second parameters P2. The indication value image NA indicates that the percentage of the indication value is 74%. Therefore, if the range from the start point SP to the end point EP is 100%, the image CA is displayed to indicate the range from the start point SP to 74%.

[0085] A description will now be given of a display control method for displaying the setting image 800. The display control method described here is executed by displaying the edit window WE.

[0086] FIG. 11 is a flowchart illustrating a display control method according to an embodiment. When the control unit 11 has not received an instruction to close the editing window WE (step S501; No), it acquires a setting value corresponding to the current playback position and a current instruction value (step S503). The control unit 11 calculates a control value based on the acquired setting value and instruction value (step S505). A formula for calculating the control value is preset using at least the setting value and instruction value as variables. The formula may differ depending on the type of second parameter P2. The control unit 11 reflects the instruction value in image CA and reflects the calculated control value in image LA (step S507), and then returns to the process of step S501. When the control unit 11 has received an instruction to close the editing window WE (step S501; Yes), the control unit 11 terminates the process.

[0087] In this way, in the setting image 800, the image CA corresponding to the instruction value and the image LA corresponding to the control value are arranged close to each other. Therefore, the user can easily check the instruction value when operating the controls and the control value that is actually reflected in the generated sound. Therefore, in a software environment that generates sound according to the playback position, music production efficiency can be improved.

[0088] The control unit 11 changes the position of the setting window image GS in the editing window WE in response to a user instruction. At this time, if the setting window image GS is moved to a position where it overlaps with the sound-related image GV, the setting window image GS is an image in a newly opened window in the editing window WE, and will obscure the sound-related image GV. In this example, the control unit 11 moves the position of the sound-related image GV in the editing window WE away from the setting window image GS so that the sound-related image GV is not obscured.

[0089] 12 and 13 show examples of an editing window in one embodiment. In the editing window WE shown in FIG. 12, when the setting window image GS is moved to the position shown in FIG. 13 in response to a user instruction, the position of the sound-related image GV also moves from the position shown in FIG. 12 to the position shown in FIG. 13. As shown in FIGS. 12 and 13, when the setting window image GS is moved, the sound-related image GV moves to a region of the editing window WE that is largely empty relative to the setting window image GS. Here, the intersection of the sound-related image GV (image GF in this example) and the image GTs is referred to as a reference point GVm. The reference point GVm may be a different position as long as it is linked to the sound-related image GV.

[0090] Next, a display control method for moving the setting window image GS will be described. The display control method described here is executed by displaying an edit window WE including the sound-related image GV and the setting window image GS.

[0091] 14 is a flowchart showing a display control method according to an embodiment. The control unit 11 waits for an instruction to close the editing window WE or an instruction to change the position of the setting window image GS (step S601; No, step S603; No). When an instruction to change the position of the setting window image GS is received (step S603; Yes), the control unit 11 determines a destination area of ​​the display area in the editing window WE that satisfies a predetermined condition (step S605). The control unit 11 moves the sound-related image GV to the destination area (step S607) and returns to the processing of step S601. When an instruction to close the editing window WE is received (step S601; Yes), the control unit 11 ends the processing.

[0092] The method for determining the destination area in step S605 will be described.

[0093] 15 and 16 are diagrams illustrating the positional relationship between the setting window image and the sound-related image in one embodiment. FIGS. 15 and 16 show a simplified configuration of the editing window WE. As shown in FIG. 15 , the display area of ​​the editing window WE is divided into multiple areas based on the setting window image GS. In this example, the multiple areas are three areas UA, MA, and BA, which are divided into the editing window WE in a vertical (up and down) arrangement. Area UA is the area above the top edge UL of the setting window image GS. Area BA is the area below the bottom edge BL of the setting window image GS. Area MA is the area sandwiched between the top edge UL and bottom edge BL of the setting window image GS. Depending on the position of the setting window image GS, area UA or area BA may not exist. For example, in the example shown in FIG. 16 , area BA does not exist.

[0094] The control unit 11 compares the area UA and the area BA and determines the larger area as the destination area. In this example, the larger area refers to the area with the larger vertical length. The larger area may also be an area with a larger surface area. In the example shown in FIG. 15 , the destination area is area BA. In the example shown in FIG. 16 , the destination area is area UA. The control unit 11 displays the sound-related image GV so that the reference point GVm is located at the vertical center of the destination area and on the image GTs. Therefore, as shown in FIGS. 15 and 16 , the length Y1 from the reference point GVm to the top edge of the destination area is the same as the length Y2 from the reference point GVm to the bottom edge of the destination area. The lengths Y1 and Y2 may be different. The relative relationship between the lengths Y1 and Y2 may differ depending on whether the destination area is area UA or area BA. For example, in the case of area UA, the length Y2 may be greater than the length Y1, and in the case of area BA, the length Y1 may be greater than the length Y2. In this way, the sound-related image GV is moved slightly away from the setting window image GS and closer to the outer periphery of the edit window WE.

[0095] When the setting window image GS moves from the position shown in Figure 15 toward the lower right and reaches the position shown in Figure 16, the reference point GVm of the sound-related image GV moves from the position shown in Figure 15 to the position shown in Figure 16. The movement of the reference point GVm may be performed after the movement of the setting window image GS has stopped. The movement of the reference point GVm may also be performed sequentially while the setting window image GS is moving. In the latter case, as the setting window image GS moves from the position shown in Figure 15 to the position shown in Figure 16, the reference point GVm first gradually moves downward. Thereafter, at the timing when the vertical length of the region UA ​​becomes larger than that of the region BA, the reference point GVm moves significantly from the region BA to the region UA.

[0096] In this way, when the setting window image GS is moved in the edit window WE, the sound-related image GV moves to a wider area, so that the sound-related image GV can be kept easily visible. Therefore, in a software environment that generates sounds according to the playback position, music production efficiency can be improved.

[0097] <Modifications> The present disclosure is not limited to the above-described embodiments, and includes various other modifications. For example, the above-described embodiments have been described in detail to clearly explain the present disclosure, and are not necessarily limited to those including all of the described configurations. Furthermore, other configurations may be added to the configuration of one embodiment, or some configurations may be deleted or replaced with other configurations. Some modifications will be described below.

[0098] (1) The processes executed by the above-described program 13a are not limited to being executed entirely in the data processing device 1. For example, some of the processes may be executed in an external device connected via a network. For example, processes that do not require strict real-time performance, such as a process of acquiring intermediate feature data MV using the first trained model, may be executed in the external device.

[0099] (2) The information displayed in the setting image 800 is not limited to information related to the second parameter P2 of the sound control data CDa. For example, information related to the third parameter P3 of the sound control data CDb may be displayed in the setting image 800.

[0100] (3) The areas in the setting image 800 where the images LA and CA are arranged are not limited to arc shapes such as the arc areas 810 and 820. For example, these areas may be linear, curved, or a combination of straight and curved lines. Regardless of the shape, it is preferable that the start points SP of the two areas are adjacent to each other, and it is preferable that the end points EP of the two areas are adjacent to each other.

[0101] (4) When dividing the display area in the editing window WE into multiple areas to determine the destination area of ​​the sound-related image GV, the division may be performed in other ways. For example, the multiple areas may be divided horizontally instead of vertically, or may be divided vertically or horizontally. In any case, it is sufficient that the area of ​​the editing window WE in which the setting window image GS does not exist is divided into multiple areas. It is sufficient that the largest area among the multiple areas satisfies the predetermined condition as the destination area. When the editing window WE is divided into three or more areas, the destination area may be the second largest area rather than the largest area.

[0102] (5) The intermediate feature data MV may be updated any time between when the first parameter P1 is changed and when playback is started in response to a playback start instruction. That is, the intermediate feature data MV may be updated immediately after the first parameter P1 is changed or a predetermined time after the change. The intermediate feature data MV may be updated at a time that is unrelated to when the first parameter P1 is changed, as exemplified below.

[0103] First example: The control unit 11 temporarily stores the first parameters P1 at predetermined time intervals. When storing the first parameters P1, the control unit 11 compares the information that has already been stored with the information that will be stored. If at least one piece of information among the temporarily stored first parameters P1 has changed, the control unit 11 updates the intermediate feature data MV.

[0104] Second example: When a playback start instruction is received, the control unit 11 determines whether at least one piece of information among the first parameters P1 has changed since the previous playback start instruction was received, and if so, updates the intermediate feature data MV. If no playback start instruction has been received since the program was executed (since the software was started), the control unit 11 may determine whether at least one piece of information among the first parameters P1 has changed when the software was started or after reading the music data 13b thereafter.

[0105] 1: Data processing device, 11: Control unit, 13: Memory unit, 13a: Program, 13b: Music data, 15: Display unit, 17: Operation unit, 21: Interface, 23: Communication unit, 100a to 100c: Sound generation unit, 120: First trained model, 121: Encoding model, 125: First generation model, 130: Second trained model, 131: Encoding model, 135: Second generation model, 140: Waveform synthesis unit, 200: Data editing unit, 300: Read position determination unit, 500: Sound mixing unit, 800: Setting image, 810: Arc region, 820: Arc region, 1000: Sound output unit

Claims

1. A signal generation method comprising: providing a first learned model with sound control data including a first parameter and a second parameter for controlling a generated sound at a plurality of time steps according to the passage of time, the first parameter for a predetermined time range including before and after the predetermined time step, thereby obtaining intermediate feature data corresponding to the predetermined time step; updating the intermediate feature data when a value of the first parameter is changed; and generating a sound signal according to data obtained by providing the second parameter and the intermediate feature data to a second learned model when a playback start instruction is received.

2. The signal generating method according to claim 1, wherein the second parameter includes instruction information corresponding to an instruction value of an operator.

3. The signal generation method according to claim 2, wherein the first parameter includes phoneme information or pitch information.

4. The signal generating method of claim 3, further comprising: displaying a sound control image corresponding to the value of the first parameter based on the sound control data; and changing the value of the first parameter when a change operation is received on the sound control image.

5. The signal generation method described in claim 1, wherein the second trained model generates sound feature data corresponding to each time step for generating the sound signal, and when the sound feature data corresponding to a predetermined time step is generated, sound feature data corresponding to a time step prior to the predetermined time step is further provided to the second trained model.

6. A signal generating method as described in claim 1, further comprising: determining a control value for controlling the generated sound in accordance with the passage of time based on an instruction value by an operator and a setting value determined in advance in a time series; and displaying a setting image including a first image area for indicating the instruction value and a second image area adjacent to the first image area for indicating the control value, wherein the second parameter includes the control value.

7. A signal generating method as described in claim 1, further comprising: displaying a first image corresponding to the sound signal and a second image including an image related to the second parameter in a specified display area; changing a position of the second image in the display area in response to an input instruction; and, when changing the position of the second image, moving the first image to an area that satisfies predetermined conditions among a plurality of areas divided into the display area based on the second image.

8. A signal generation method comprising: displaying a first area including an image including phoneme information or pitch information and corresponding to a first parameter for controlling a sound generated by a first track, a second area including an image corresponding to a second parameter for controlling the sound generated by the first track, and a third area including an image corresponding to a third parameter for controlling a sound generated by a second track; upon receiving a playback start instruction, displaying images corresponding to a current playback position in the first area and the third area; generating a first sound signal of the first track and a second sound signal of the second track corresponding to the playback position; and outputting a mixed sound of the first sound signal and the second sound signal; and upon receiving an instruction to change the second parameter after the playback start instruction, reflecting the changed second parameter in the first sound signal at the playback position.

9. The method of claim 8, wherein the first parameters include the phoneme information.

10. The signal generating method according to claim 8, wherein the second sound signal at the playback position is generated based on information of the third parameter at or before the playback position.

11. The signal generating method according to claim 8, further comprising the step of: when an instruction to change the third parameter is received after the instruction to start playback, reflecting the changed third parameter in the second sound signal at the playback position.

12. The signal generating method according to claim 8, wherein the first sound signal at the playback position includes information of the first parameter after the playback position.

13. The signal generating method according to claim 8, wherein an instruction to change the second parameter is input by operating an image displayed in the second area.

14. A program for causing a computer to execute the signal generating method according to any one of claims 1 to 13.

15. A display control method comprising: determining a control value for controlling a generated sound over time based on an instruction value by an operator and a setting value determined in advance in a time series; generating a sound signal based on sound control data for controlling the generated sound over time and the control value; and displaying a setting image including a first image area for indicating the instruction value and a second image area adjacent to the first image area for indicating the control value.

16. The display control method of claim 15, wherein the instruction value is indicated by a position in the first image area, the control value is indicated by a position in the second image area, a position indicating a minimum value in the first image area and a position indicating a minimum value in the second image area are adjacent to each other, and a position indicating a maximum value in the first image area and a position indicating a maximum value in the second image area are adjacent to each other.

17. A display control method comprising: displaying in a specified display area a first image corresponding to a sound signal generated based on sound control data including a first parameter and a second parameter for controlling sound generated over time, and a second image including an image related to the second parameter; changing a position of the second image in the display area in response to an input instruction; and, when changing the position of the second image, moving the first image to an area that satisfies predetermined conditions among a plurality of areas divided into the display area based on the second image.

18. A display control method as described in claim 17, wherein the first image is displayed extending left and right within the display area, the plurality of areas includes at least two areas divided so as to be aligned vertically with the second image as a reference, and the area satisfying the predetermined condition is the wider of the at least two areas.

19. A program for causing a computer to execute the display control method according to any one of claims 15 to 18.

Images