Signal generation method, display control method and program

Abstract

To improve music production efficiency in a software environment that generates sounds according to a playback position.SOLUTION: A signal generation method in one embodiment includes: obtaining intermediate feature data corresponding to a predetermined time step by providing a first parameter, which is in a predetermined time range including before and after the predetermined time step among sound control data including the first parameter for controlling a generated sound at a plurality of time step corresponding to elapsed time and a second parameter to a first learned model; updating the intermediate feature data when a first parameter value is changed; and generating a sound signal according to the obtained data by providing the second parameter and the intermediate feature data to a second learned model, when a playback start instruction is received.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a technique for generating an audio signal. [Background technology]

[0002] There is known a technology for synthesizing sound using a deep neural network (DNN). As shown in Patent Document 1, a technology has been developed that uses information after the timing of the sound to be synthesized in order to further improve the quality of the synthesized sound. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2021 / 251364 Summary of the Invention [Problem to be solved by the invention]

[0004] DAW (Digital Audio Workstation) used in music production functions as software that generates sounds according to the playback position. DAW not only generates sounds, but also provides a user interface that makes it easy to edit various data.

[0005] A typical DAW generates a sound signal using waveform data, MIDI (Musical Instruments Digital Interface) data, and the like. When using data for controlling generated sounds (sound control data) such as MIDI data, a sound signal is generated using information corresponding to the playback position (current information) and information corresponding to a time before the playback position (past information). Therefore, a technology for generating a synthetic sound using information corresponding to a time after the playback position (future information) cannot be directly applied to a DAW. In the following description, a technology for generating a synthetic sound using future information may be simply referred to as a sound synthesis technology.

[0006] When using such sound synthesis technology in a DAW, it is common for waveform data corresponding to the synthesized sound to be generated in advance by an external function such as a plug-in. By using the waveform data generated in advance by an external function, the DAW can generate a sound signal according to the playback position in the same way as with normal waveform data.

[0007] On the other hand, since it is necessary to generate waveform data in advance, it is difficult to edit while listening to the sound, which results in a decrease in the efficiency of music production. In addition, not only sound synthesis technology that uses future information, but also the convenience of editing music in real time in a DAW has a significant impact on the efficiency of music production.

[0008] One object of the present invention is to improve the efficiency of music production in a software environment that generates sounds according to the playback position. [Means for solving the problem]

[0009] A signal generation method in one embodiment includes: providing a first parameter of a predetermined time range including before and after a predetermined time step to a first learned model, the first parameter being included in sound control data including a first parameter and a second parameter for controlling a generated sound at a plurality of time steps according to the passage of time; acquiring intermediate feature data corresponding to the predetermined time step; updating the intermediate feature data when a value of the first parameter is changed; and generating a sound signal according to data obtained by providing the second parameter and the intermediate feature data to a second learned model when a playback start instruction is received.

[0010] A signal generating method in one embodiment includes displaying a first area including an image corresponding to a first parameter for controlling a generated sound of a first track, the first area including an image corresponding to a second parameter for controlling the generated sound of the first track, and a third area including an image corresponding to a third parameter for controlling the generated sound of a second track, and upon receiving a playback start instruction, displaying an image corresponding to a current playback position in the first area and the third area, generating a first sound signal of the first track and a second sound signal of the second track corresponding to the playback position, and outputting a mixed sound of the first sound signal and the second sound signal. Upon receiving an instruction to change the second parameter after the playback start instruction, the changed second parameter is reflected in the first sound signal at the playback position.

[0011] A display control method in one embodiment includes determining a control value for controlling a generated sound over time based on an instruction value by an operator and a setting value determined in advance in a time series, generating a sound signal based on sound control data for controlling the generated sound over time and the control value, and displaying a setting image including a first image area for indicating the instruction value and a second image area adjacent to the first image area for indicating the control value.

[0012] A display control method in one embodiment includes displaying in a predetermined display area a first image corresponding to a sound signal generated based on sound control data including a first parameter and a second parameter for controlling generated sound over time, and a second image including an image related to the second parameter, changing a position of the second image in the display area in response to an input instruction, and, when changing the position of the second image, moving the first image to an area that satisfies predetermined conditions among a plurality of areas divided into the display area based on the second image. Effect of the Invention

[0013] According to the present invention, it is possible to improve the efficiency of music production in a software environment in which sounds are generated according to a playback position. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating a data processing device according to an embodiment. [Diagram 2] 13 is an example of a music editing screen according to an embodiment. [Diagram 3] FIG. 2 is a functional block diagram illustrating a sound output unit according to an embodiment. [Figure 4] FIG. 2 is a functional block diagram illustrating a sound generating unit according to an embodiment. [Diagram 5] 1 is a flow chart illustrating a method for generating a signal in one embodiment. [Figure 6] 1 is a flow chart illustrating a method for generating a signal in one embodiment. [Figure 7] 10 is a flowchart illustrating a sound reproduction process according to an embodiment. [Figure 8] 1 is a flow chart illustrating a method for generating a signal in one embodiment. [Figure 9] 13 is an example of a music editing screen according to an embodiment. [Figure 10] 11 is an example of a setting image according to an embodiment. [Figure 11] 1 is a flowchart illustrating a display control method according to an embodiment. [Figure 12] 1 is an example of an editing window in one embodiment. [Figure 13] 1 is an example of an editing window in one embodiment. [Figure 14] 1 is a flowchart illustrating a display control method according to an embodiment. [Figure 15] 11 is a diagram for explaining the positional relationship between a setting window image and a sound-related image in one embodiment. FIG. [Figure 16] 11 is a diagram for explaining the positional relationship between a setting window image and a sound-related image in one embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The embodiments shown below are merely examples, and the present invention is not limited to these embodiments. In the drawings referred to in this embodiment, the same or similar symbols are used for the same parts or parts having similar functions, and repeated explanations may be omitted. In order to clarify the explanation, the drawings may be described in a schematic manner with dimensional ratios different from the actual ratios or parts of the configuration omitted from the drawings.

[0016] [overview] A data processing device according to an embodiment of the present invention is a device equipped with a computer, such as a desktop terminal, a notebook terminal, a smartphone, or a tablet terminal. The data processing device provides a user with a DAW for editing music by executing an installed application program. The DAW generates a sound signal based on data set corresponding to a plurality of tracks, such as information for generating sound, such as sound control data and waveform data.

[0017] The sound control data is data including information for controlling the generated sound at multiple time steps according to the passage of time, such as MIDI data. The waveform data is data obtained by sampling a sound signal waveform, such as WAV data or MP3 data. The DAW reads out information corresponding to the playback position from the data of each track, and generates a sound signal corresponding to the playback position. By synchronously generating sound signals corresponding to multiple tracks, the sounds corresponding to each track are output as a mixed sound (mixed sound).

[0018] At least information corresponding to the playback position, i.e., current information, is used to generate the sound signal. When a sound signal is generated according to sound control data, information corresponding to a time before the playback position, i.e., past information, also affects the sound signal generated corresponding to the playback position. In this example, sound control data for utilizing a sound synthesis technique that further uses information corresponding to a time after the playback position, i.e., future information, among the sound control data, can also be assigned to at least one of the multiple tracks.

[0019] According to this sound synthesis technology, in this example, a sound that imitates a human voice can be generated at a set pitch. In the following description, the sound that imitates a human voice is referred to as a singing synthetic sound. This sound synthesis technology can also be used to generate synthetic sounds of musical instruments, not limited to singing synthetic sounds. In the following description, the process of generating a singing synthetic sound is a process using a sound synthesis technology that uses past information, present information, and future information, unless otherwise specified. A method for realizing this process can be understood by a person skilled in the art according to the disclosure (particularly the disclosure of the second embodiment) in the above-mentioned Patent Document 1 (International Publication No. 2021 / 251364). The data processing device will be described below.

[0020] [Data processing device] FIG. 1 is a block diagram showing a data processing device in an embodiment. The data processing device 1 includes a control unit 11, a storage unit 13, a display unit 15, an operation unit 17, an interface 21, and a communication unit 23. The control unit 11 is an example of a computer equipped with a processor such as a CPU and a storage unit such as a RAM. The control unit 11 executes a program 13a stored in the storage unit 13 using the CPU, and realizes functions for executing various processes in the data processing device 1. The various processes include processes for realizing functions related to a DAW. The processes include processes related to a signal generation method and a display control method, which will be described later. Therefore, the control unit 11 also functions as a signal generation unit for executing the signal generation method and a display control unit for executing the display control method.

[0021] The display unit 15 is a display having a display area for displaying various screens in response to the control of the control unit 11. The various screens include a music editing screen, which will be described later. The operation unit 17 is a device for inputting user instructions, and is an operation device including operators that output operation signals to the control unit 11 in response to the input operation. The operators are, for example, slider-type operators such as faders, rotary-type operators such as knobs and rotary encoders, and button-type operators such as switches. The operators may be provided as operator images displayed on the display unit 15. The operation signals include information regarding the instruction values ​​of the operators.

[0022] The interface 21 includes a module for communicating with an external device by wired communication or wireless communication such as infrared communication or short-distance wireless communication. In this example, the external device may be an electronic musical instrument or sound source device that generates a sound signal, or may be a speaker or headphones that output a sound signal generated by the control unit 11 to the outside. The interface 21 is used for communication without going through a network. The communication unit 23 is a communication module for connecting to a network under the control of the control unit 11 and communicating with an external device connected to the network.

[0023] The storage unit 13 is a storage device such as a non-volatile memory, a hard disk drive, etc. The storage unit 13 stores a program 13a executed by the control unit 11 and various data required for the DAW realized by the program 13a. For example, the storage unit 13 stores music data 13b, a first trained model 120, and a second trained model 130.

[0024] The program 13a is downloaded from an external device via a network such as the Internet, and is stored in the storage unit 13, thereby being installed in the data processing device 1. The program 13a may be provided in a state in which it is recorded on a non-transient computer-readable recording medium (e.g., a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, a semiconductor memory, etc.). In this case, the data processing device 1 only needs to be equipped with a device for reading this recording medium. The storage unit 13 can also be considered as an example of a recording medium.

[0025] Similarly, the music data 13b, the first trained model 120, and the second trained model 130 may be downloaded from an external device via a network and stored in the storage unit 13, or may be provided in a state recorded on a non-transient computer-readable recording medium. The music data 13b is data stored in the storage unit 13 for each music piece, and includes sound control data CDa, CDb, waveform data WD, intermediate feature data MV, etc. These data may be generated in the process of music production using a DAW, or may be provided to the data processing device 1 in advance in a manner similar to that of the program 13a.

[0026] The sound control data CDa and CDb are, for example, data in a MIDI format, and include a plurality of types of parameters for controlling generated sounds at a plurality of time steps according to the passage of time. In this example, the sound control data CDa is data used when generating a singing synthesis sound. The sound control data CDb is data used when generating an instrument sound. The plurality of parameters in the sound control data CDa are divided into at least parameters belonging to a first group and parameters belonging to a second group. The parameters in the first group include parameters corresponding to future information, and may be referred to as first parameters P1. The parameters in the second group include parameters corresponding to current information, and may be referred to as second parameters P2. The second parameters may include parameters related to past information. The information included in each of the first parameters P1 and the second parameters P2 described below is merely an example, and may be other than these combinations.

[0027] The first parameter P1 includes information determined for each sound defined in a time series. The first parameter P1 is, for example, phoneme information, pitch information, period information, intensity information, etc. The first parameter P1 includes at least phoneme information or pitch information. The phoneme information includes information for generating characters such as lyrics for singing as sounds corresponding to each sound. When the sound control data CDa is data for generating musical instrument sounds, the phoneme information may not be used, or other information such as lyrics may be used instead of the phoneme information, for example, phoneme information may be used as information for reproducing the playing style of a musical instrument. The pitch information includes information for specifying a pitch corresponding to each sound. The period information includes information for specifying a start position and a duration (note value) of the sound corresponding to each sound. The period information may be defined by a plurality of pieces of information such as note-on and note-off. The intensity information includes information for specifying a velocity corresponding to each sound.

[0028] The second parameter P2 includes information for controlling the generated sound at each time step. The second parameter P2 is, for example, power information, transpose information, formant information, etc. The power information includes information for specifying the dynamics of the sound, and for example, a larger set value reproduces a shouting voice sound, and a smaller set value reproduces a whispering voice sound. The transpose information includes information for raising and lowering the pitch of the sound. The formant information includes information for specifying the voice quality, and for example, a larger set value reproduces a sound closer to a male voice, and a smaller set value reproduces a sound closer to a female voice. The second parameter P2 may be information for specifying the clarity, brightness, breath, mouth opening, etc. of the voice.

[0029] The set value of the information included in the second parameter P2 may be used as it is as a value (control value) for controlling the generated sound, or may be updated or corrected according to the instruction information and used as the control value. The instruction information includes information according to the instruction value of the operator. That is, the control value for the second parameter P2 may be determined based on the set value and the instruction value. In this case, since the instruction value is related to the set value of the information included in the second parameter P2, it can also be said that the instruction information is an example of the second parameter P2. For example, when the control value of the power information is determined based on the instruction value, it can also be said that the instruction information is power information.

[0030] The multiple parameters in the sound control data CDb include at least parameters belonging to a third group. The parameters in the third group include parameters corresponding to current information, and may be referred to as third parameters P3. The third parameters P3 may include parameters related to past information.

[0031] The third parameter P3 includes information determined for each sound defined in time series and information for controlling the generated sound at each time step. The former information is similar to the first parameter P1 described above, but is not used as future information, but is used as present information or past information. The latter information is similar to the second parameter P2 described above. The third parameter P3 is, for example, pitch information, period information, intensity information, etc. The third parameter P3 may include transpose information, etc.

[0032] If the sound control data CDb is data for generating the sound of a percussion instrument such as a drum, it may include information for identifying the type of percussion instrument, such as a snare, a tom, or a cymbal. Since a percussion instrument may not require pitch information, type information may be included in the third parameter P3 instead of pitch information as information for identifying the type of percussion instrument. Pitch information may be identified by, for example, a note number, but type information is also identified by a note number. Therefore, even if the data is in the form of pitch information, it may not be information indicating pitch.

[0033] The waveform data WD is data obtained by sampling a sound signal waveform, and indicates waveform data corresponding to each time step. The intermediate feature data MV is data generated by the first trained model 120, and is, for example, vector data corresponding to each time step.

[0034] The first trained model 120 and the second trained model 130 are statistical estimation models used when generating a singing synthesis sound, and in this example, a known machine learning model using DNN is used. The first trained model 120 and the second trained model 130 may be different types of models. The machine learning model may be, for example, a machine learning model using a CNN (Convolutional Neural Network) or an RNN (Recurrent Neural Network). Details of the first trained model 120 and the second trained model 130 will be described later.

[0035] [Song editing screen] The music editing screen is provided by the DAW realized in the data processing device 1.

[0036] Fig. 2 is an example of a music editing screen in one embodiment. In the example shown in Fig. 2, the music editing screen WS includes a track area AT, an editing area AE, and an operation image area AS. In each case, the horizontal direction indicates a time position, i.e., a time step. When a playback process is being performed, an image GT indicating the playback position is displayed in the track area AT and the editing area AE.

[0037] The track area AT is an area for displaying an image corresponding to the data corresponding to each track, and in this example includes areas AT1, AT2, AT3, and AT4 corresponding to the first to fourth tracks. Sound control data CDa is assigned to the first track, sound control data CDb is assigned to the second and third tracks, and waveform data WD is assigned to the fourth track. Therefore, an image corresponding to the first parameter P1 in the sound control data CDa is displayed in the area AT1 (first area). An image corresponding to the third parameter P3 in the sound control data CDb is displayed in the areas AT2 and AT3 (third area).

[0038] In this example, the editing area AE includes a piano roll area AP and a set value area AC. The piano roll area AP is an area (first area) where information of a first parameter P1 is displayed when sound control data CDa is assigned to a track selected in the track area AT, and a sound control image corresponding to the first parameter P1 is displayed. The piano roll area AP is an area (third area) where information of a third parameter P3 is displayed when sound control data CDb is assigned to a track selected in the track area AT, and a sound control image corresponding to the third parameter P3 is displayed.

[0039] The sound control image is, for example, an image GL showing phoneme information, and an image GR showing pitch information and period information. Other information such as intensity information may be further expressed by the color of the image GR, the width of the band, etc. The image GR includes information determined for each sound defined in time series, and in the case of sound control data CDa, it includes information of a first parameter P1, and in the case of sound control data CDb, it includes information of a third parameter P3. When waveform data WD is assigned to the track selected in the track area AT, the piano roll area AP is replaced by an editing area AE, which is an area for displaying a waveform.

[0040] The set value area AC (second area) displays an image GP showing the set value of information for controlling the generated sound at each time step. That is, the image GP includes information of the second parameter P2 in the case of sound control data CDa, and includes information of the third parameter P3 in the case of sound control data CDb. The set value area AC may display an image related to information of the first parameter P1 that is not displayed in the editing area AE in conjunction with the editing area AE.

[0041] When the user changes the position of the image GR, image GP, etc., on the music editing screen WS according to the information of each data by instructing a change operation on the image via the operation unit 17 or the interface 21, the information of the corresponding data is changed. Therefore, the image displayed in the piano roll area AP can be said to be an image related to the change of the first parameter P1 or the third parameter P3. The image displayed in the setting value area AC can be said to be an image related to the change of the second parameter P2 or the third parameter P3.

[0042] In this example, the operation image area AS includes operation images GC1, GC2, and GC3 for receiving user instructions. The operation image GC1 is an image simulating a button for receiving instructions such as to start playback and end playback. The operation image GC2 is an image simulating a button for receiving settings for the movement speed of the playback position, i.e., tempo information. The operation image GC3 is an image simulating a slider for receiving volume settings.

[0043] [Sound output section] The function of the sound output unit that is realized by the control unit 11 executing the program 13a will be described.

[0044] 3 is a functional block diagram showing a sound output unit in one embodiment. The sound output unit 1000 includes sound generating units 100a, 100b, and 100c, a data editing unit 200, a read position determining unit 300, and a sound mixing unit 500. The entire configuration of the sound output unit 1000 is not limited to being realized as software by executing a program, and at least a part of the configuration may be realized by hardware.

[0045] The data editing unit 200 edits the information of the sound control data CDa, CDb in response to a user instruction to update the data. The data editing unit 200 may edit the waveform data WD in response to a user instruction. The read position determination unit 300 generates a read position TR in response to a reproduction instruction and a value of tempo information. The tempo information may be a value set by the operation image GC2 or may be a value set in the sound control data CDa, CDb. In this example, the read position TR is expressed by the position of a time step.

[0046] The sound generating unit 100a generates a sound signal Wa (first sound signal) corresponding to the track to which the sound control data CDa is assigned. The sound generating unit 100a generates intermediate feature data MV in the generation process of the sound signal Wa. The sound generating unit 100b generates a sound signal Wb (second sound signal) corresponding to the track to which the sound control data CDb is assigned. The sound generating unit 100c generates a sound signal Wc corresponding to the track to which the waveform data WD is assigned. The sound generating units 100a, 100b, and 100c read data according to the read position TR and generate the sound signals Wa, Wb, and Wc. In this way, the playback position advances and the read position TR also advances, so that the sound signals Wa, Wb, and Wc according to the playback position are output. As the playback position advances according to the read position TR, the position of the image GT displayed on the above-mentioned music editing screen WS also moves according to the read position TR.

[0047] The sound signal Wa includes information of the first parameter P1, and therefore includes information after the playback position, i.e., future information. The sound signal Wb includes information of the third parameter P3, and therefore includes information at or before the playback position, i.e., present information or past information.

[0048] The sound mixing unit 500 mixes the sound signals (sound signals Wa, Wb, Wc in the example of FIG. 3) generated in each track, and outputs a mixed sound signal Wm.

[0049] Fig. 4 is a functional block diagram showing a sound generating unit in one embodiment. Fig. 4 shows a detailed configuration of the sound generating unit 100a. The sound generating unit 100a includes the above-mentioned first trained model 120 and second trained model 130, and further includes a waveform synthesis unit 140. The first trained model 120 includes an encoding model 121 and a first generation model 125.

[0050] The first learned model 120 is a model trained to generate intermediate feature data MV at a given time step when a first parameter P1 and a tempo Z2 in a given time range including before and after the given time step are provided. The tempo Z2 is a value of tempo information. The second learned model 130 includes an encoding model 131 and a second generation model 135. The second learned model 130 is a model trained to generate sound feature data F at a given time step when a second parameter P2 and intermediate feature data MV at the given time step are provided. At this time, sound feature data F prior to the given time step may be provided.

[0051] As described above, the configuration of the sound generation unit 100a is similar to the configuration of the second embodiment shown in Fig. 10 in Patent Document 1 (International Publication No. 2021 / 251364). Therefore, although detailed description is omitted, a person skilled in the art can implement it by referring to the disclosure in Patent Document 1.

[0052] The encoding model 121 is a statistical estimation model for generating symbol data B at each time step based on the first parameter P1 of the sound control data CDa. The information of the first parameter P1 used in the encoding model 121 includes, for example, phoneme information. The first generation model 125 is a statistical estimation model for generating encoded data E at each time step based on the symbol data B, the tempo Z2, and the first parameter P1. The information of the first parameter P1 used in the first generation model 125 includes, for example, pitch information. The encoding model 121, the first generation model 125, the first parameter P1, the tempo Z2, the symbol data B, and the encoded data E correspond to the encoding model (21), the encoded data acquisition unit (22), the music data (D), the tempo (Z2), the symbol data (B), and the encoded data (E) in Patent Document 1, respectively.

[0053] The encoded data E in each time step is generated using information on the first parameter P1 in a predetermined time range including before and after the corresponding time step. That is, the encoded data E is generated using not only current information but also at least future information. The encoded data E corresponding to a plurality of time steps is acquired as intermediate feature data MV. The acquired intermediate feature data MV is stored in the storage unit 13. If the intermediate feature data MV has not yet been generated, first, the control unit 11 provides information on the first parameter P1 to the first trained model 120. Processing in the first trained model 120 is executed based on the provided information, and encoded data E is generated for each time step included in the period in which the first parameter P1 exists.

[0054] When the data editing unit 200 updates the information of the first parameter P1 or the tempo information in the sound control data CDa after the intermediate feature data MV is stored, the control unit 11 provides the updated information to the first learned model 120. Processing in the first learned model 120 is executed based on the provided information, and the intermediate feature data MV is updated based on the newly generated encoded data E. At this time, the entire intermediate feature data MV may be updated, or information may be provided to the first learned model 120 so that only the intermediate feature data MV in the time range affected by the updated information is updated.

[0055] The encoding model 131 is a statistical estimation model for generating encoded control data C at each time step based on the second parameter P2 of the sound control data CDa. The above-mentioned second generation model 135 reads out encoded data E of a time step corresponding to the read position TR from the intermediate feature data MV. The second generation model 135 is a statistical estimation model for generating sound feature data F of a time step corresponding to the read position TR based on the read encoded data E, control data C, and sound feature data F generated in a past time step (a time step prior to the time step corresponding to the read position TR).

[0056] The encoding model 131, the second generation model 135, the second parameter P2, the control data C, and the sound feature data F correspond to the generation model 32, the generation model 40, the instruction value Z1, the control data C, and the sound feature data F in Patent Document 1, respectively. Here, the second parameter P2 is information stored in the storage unit 13, while the instruction value Z1 in Patent Document 1 is based on a user instruction. The second parameter P2 provided to the second trained model 130 is a control value and is information that can be updated by a user instruction. Therefore, the second parameter P2 can be treated in the same way whether it is information updated in real time by a user instruction or predetermined information.

[0057] The value (control value) of the second parameter P2 provided to the second trained model 130 may be a value obtained by correcting the value (set value) of the second parameter P2 of the sound control data CDa stored in the storage unit 13 by the instruction value of the instruction information. In this case, the updated sound control data CDa does not need to be stored in the storage unit 13. The value (control value) of the second parameter P2 provided to the second trained model 130 may be the value (set value) of the second parameter P2 of the sound control data CDa stored in the storage unit 13, or may be the instruction value of the instruction information.

[0058] When a user instructs to start playback, the control unit 11 provides the second parameter P2 at the time step corresponding to the read position TR and the encoded data E of the intermediate feature data MV corresponding to that time step to the second learned model 130. Data is provided to the first learned model 120 when the information of the first parameter P1 is updated, but these data are not provided to the second learned model 130 when the information of the second parameter P2 is updated. Processing in the second learned model 130 is executed based on the provided information, and sound feature data F indicating the features of the sound signal at each time step is output.

[0059] The waveform synthesis unit 140 corresponds to the waveform synthesis unit (50) in Patent Document 1, and generates a sound signal Wa by synthesizing sound signals at each time step according to the sound feature data F. In this way, as the playback position advances and the read position TR also advances, the sound generation unit 100a outputs a sound signal Wa according to the playback position.

[0060] In the sound generation unit 100a, when the information of the first parameter P1 is updated, the intermediate feature data MV is updated, but when the information of the second parameter P2 is updated, the intermediate feature data MV is not updated. On the other hand, after an instruction to start playback is given, when the information of the second parameter P2 is updated, information reflecting the update is provided in real time to the second trained model 130. Therefore, the update information of the second parameter P2 is also reflected in the generated sound signal Wa.

[0061] When the information of the first parameter P1 is updated after an instruction to start playback is given, the intermediate feature data MV may be updated after playback ends, or may be updated even during playback. In the latter case, if the position is a certain distance or more away from the playback position, the update information regarding the first parameter P1 is also reflected in the sound signal Wa that is generated when the playback position is reached.

[0062] [Signal generation method (DAW)] A description will now be given of a signal generation method executed in the control unit 11. The signal generation method described here starts when the program 13a is executed. First, the overall flow of the DAW will be described.

[0063] 5 is a flowchart showing a signal generation method in one embodiment. The control unit 11 displays a music editing screen WS (step S10). As described above, the music editing screen WS includes a track area AT and an editing area AE. The track area AT displays an image corresponding to a first parameter P1 for controlling the generated sound if the track is assigned sound control data CDa, and displays an image corresponding to a third parameter P3 for controlling the generated sound if the track is assigned sound control data CDb.

[0064] The editing area AE includes a piano roll area AP and a set value area AC. For a track to which sound control data CDa is assigned, the piano roll area AP is used as an area for changing information on the first parameter P1 by displaying an image related to the change of the first parameter P1. For a track to which sound control data CDa is assigned, the set value area AC is used as an area for changing information on the second parameter P2 by displaying an image related to the change of the second parameter P2. An editing window, which will be described later, is also an example of displaying an image related to the change of the second parameter P2. For a track to which sound control data CDb is assigned, the piano roll area AP and the set value area AC are used as areas for changing information on the third parameter P3 by displaying an image related to the change of the third parameter P3.

[0065] The control unit 11 generates music data 13b by editing various parameter information in response to a user instruction (step S20). The control unit 11 waits for a user instruction to start playback (step S30; No). When the control unit 11 receives the instruction to start playback (step S30; Yes), it determines a read position TR so as to advance the playback position by time steps (step S40). When the read position TR reaches the end position of music data 13b (step S50; Yes), the control unit 11 waits again for an instruction to start playback (step S30; No).

[0066] If the read position TR has not reached the end position of the music data 13b (step S50; No), the control unit 11 generates sound signals corresponding to each track (step S60), generates a mixed sound of the sound signals corresponding to each track (step S70), returns to the processing of step S40, and determines the read position TR by advancing the playback position.

[0067] [Signal generation method (sound generation unit 100a)] A signal generating method executed by the sound generating unit 100a for generating sound for a track to which sound control data CDa is assigned will be described below. When the sound control data CDa is assigned, the process described below is started.

[0068] 6 is a flowchart showing a signal generation method in one embodiment. The control unit 11 reads out the sound control data CDa (step S101) and generates intermediate feature data MV (step S103). The control unit 11 waits for a user's instruction to start playback or an instruction to edit parameters (steps S105; No, S107; No). This state is referred to as a standby process in the explanation of FIG. 6.

[0069] In the standby process, when an instruction to edit a parameter is received (step S107; Yes), the control unit 11 updates the sound control data CDa in response to the instruction (step S109). If the first parameter P1 has not been edited (step S111; No), the control unit 11 returns to the standby process. If the first parameter P1 has been edited (step S111; Yes), the control unit 11 updates the intermediate feature data MV (step S113) and returns to the standby process. In the standby process, when an instruction to start playback is received (step S105; Yes), the control unit 11 starts the sound playback process (step S200) and returns to the standby process again.

[0070] 7 is a flowchart showing a sound reproducing process according to an embodiment. The sound reproducing process may be executed in parallel with the processes from step S105 to step S113, or the process may not proceed beyond the standby process while the sound reproducing process is being executed.

[0071] The control unit 11 determines the read position TR so as to advance the playback position for each time step (step S201). If the read position TR reaches the end position of the music data 13b (step S203; Yes), the control unit 11 ends the playback process. If the read position TR does not reach the end position of the music data 13b (step S203; No), the control unit 11 reads out the sound control data CDa and intermediate feature data MV corresponding to the time step corresponding to the read position TR (step S209) unless the second parameter P2 has been edited (step S205; No). If the second parameter P2 has been edited (step S205; Yes), the control unit 11 updates the sound control data CDa (step S207) and executes the process of step S209.

[0072] The control unit 11 generates sound feature data F using the read data (step S211), generates a sound signal Wa (step S213), returns to the process of step S201, and determines the read position TR by advancing the playback position.

[0073] In this way, when the information of the first parameter P1 is updated, the sound generation unit 100a executes a process using future information regardless of the playback position to generate intermediate feature data MV, and executes a process not using future information according to the playback position. As a result, even if a sound synthesis technique including future information is used, the sound generation unit 100a can generate a sound signal Wa reflecting the update of information. That is, the sound signal Wa reflects the information of the first parameter P1 and the second parameter P2 after the change by editing. In particular, even after a playback start instruction, the sound signal Wa may reflect the information of the second parameter P2 and may also reflect the information of the first parameter P1. Therefore, in a software environment that generates a sound according to a playback position such as a DAW, music production efficiency can be improved. When a user changes the second parameter P2 through a user interface provided in the music editing screen WS shown in FIG. 9, the changed second parameter P2 is immediately reflected in the generation of the sound signal Wa even after playback starts. Therefore, the user can experience being able to check the sound generated by the sound synthesis technique including future information in real time.

[0074] [Signal generation method (sound generation unit 100b)] A signal generation method executed by the sound generation unit 100b for generating sound for a track to which sound control data CDb is assigned will be described below. When the sound control data CDb is assigned and an instruction to start playback is received, the process described below is started.

[0075] FIG. 8 is a flowchart showing a signal generating method in an embodiment. The control unit 11 determines the read position TR so as to advance the playback position for each time step (step S301). If the read position TR reaches the end position of the music data 13b (step S303; Yes), the control unit 11 ends the process. If the read position TR does not reach the end position of the music data 13b (step S303; No), the control unit 11 reads out the sound control data CDb (corresponding to the information of the third parameter P3) corresponding to the time step corresponding to the read position TR (step S309) if the parameter has not been edited (step S305; No). If the parameter has been edited (step S305; Yes), the control unit 11 updates the sound control data CDb (step S307) and executes the process of step S309. The control unit 11 generates a sound signal Wb using the read data (step S313), returns to the process of step S301, and determines the read position TR to which the playback position has been advanced. In this way, even after a playback start instruction is issued, the information of the third parameter P3 after the editing is changed is reflected in the sound signal Wb.

[0076] [Edit window] An edit window that can be displayed by inputting a predetermined operation on the music edit screen WS will now be described.

[0077] Fig. 9 is an example of a music editing screen in one embodiment. An editing window WE is displayed on the music editing screen WS shown in Fig. 9 under the control of the control unit 11. An editing window WE can be displayed for each track. Therefore, multiple editing windows WE may be displayed on the music editing screen WS. A sound-related image GV (first image) and a setting window image GS (second image) are displayed in the editing window WE. The setting window image GS is displayed as if a smaller window is opened inside the editing window WE.

[0078] The sound-related image GV is an image corresponding to a sound signal, and includes an image GRs corresponding to the image GR in the music editing screen WS, and an image GF showing the pitch of the waveform. Although omitted in FIG. 9, the sound-related image GV may be displayed with an image that imitates the waveform itself, centered on the image GF. In this example, the editing window WE further displays an image GTs showing the playback position. The position of the image GTs in the editing window WE is fixed. Therefore, as the playback position advances, the sound-related image GV relatively flows to the left AR. Therefore, the sound-related image GV is displayed expanding horizontally (left and right) in the editing window WE.

[0079] In this example, the setting window image GS includes operation images GC4, GC5, and GC6. The operation image GC4 is an image simulating a button for transposing the key. The operation image GC5 is an image simulating a slider for changing various setting values ​​from the reference values. The operation image GC6 is an image simulating a slider for fine-tuning the sound generation timing. In this example, the setting window image GS further includes a setting image 800. The setting image 800 includes an image related to the second parameter P2.

[0080] 10 is an example of a setting image in one embodiment. The setting image 800 includes an arc region 810 (first image region) for indicating an instruction value and an arc region 820 (second image region) for indicating a control value. The arc regions 810 and 820 are disposed adjacent to each other, and the arc region 810 is located outside the arc region 820. In this example, both regions are defined as regions along concentric arcs. The start points SP of the arc regions 810 and 820 are positions indicating minimum values ​​and are adjacent to each other. The end points EP of the arc regions 810 and 820 are positions indicating maximum values ​​and are adjacent to each other.

[0081] In the arc region 810, an image CA indicating the instruction value is displayed extending from the start point SP. The image CA is displayed in the arc region 810 within a range according to the ratio of the instruction value to the maximum possible value of the instruction value. If the ratio of the instruction value is 100%, the image CA is an image from the start point SP to the end point EP. The instruction value is indicated by the position of the end of the image CA, but may be indicated in other display forms as long as it is indicated by the position in the arc region 810.

[0082] In the arc region 820, an image LA indicating the control value is displayed extending from the start point SP. The image LA is displayed in the arc region 820 within a range according to the ratio of the control value to the maximum possible value of the control value. If the ratio of the control value is 100%, the image LA is an image from the start point SP to the end point EP. The control value is indicated by the position of the end of the image LA, but may be indicated in other display forms as long as it is indicated by the position in the arc region 820.

[0083] As described above, the instruction value corresponds to the instruction value of the operator. The control value corresponds to a value that is actually reflected in the generation of the sound signal Wa, such as a value obtained by correcting the setting value of the second parameter P2 by the instruction value. That is, in the case of a track using the sound generation unit 100a, this control value corresponds to the value of the second parameter P2 provided to the second trained model 130. In the case of a track using the sound generation unit 100b, this control value may be the value of the third parameter P3.

[0084] In this example, the setting image 800 includes an indication value image NA and a type image PA, which are surrounded by arc regions 810 and 820. The indication value image NA is an image indicating the ratio of the indication value described above. The type image PA is an image indicating the type of information selected from the second parameters P2. Here, the type image PA indicates that power information is selected from the second parameters P2. The indication value image NA indicates that the ratio of the indication value is 74%. Therefore, when the range from the start point SP to the end point EP is 100%, the image CA is displayed to indicate a range of 74% from the start point SP.

[0085] A description will now be given of a display control method for displaying the setting image 800. The display control method described here is executed by displaying the edit window WE.

[0086] FIG. 11 is a flowchart showing a display control method in an embodiment. When the control unit 11 has not received an instruction to close the editing window WE (step S501; No), it acquires a set value corresponding to the current playback position and a current instruction value (step S503). The control unit 11 calculates a control value according to the acquired set value and instruction value (step S505). A formula for calculating the control value is set in advance so that at least the set value and the instruction value are variables. The formula may differ depending on the type of the second parameter P2. The control unit 11 reflects the instruction value in the image CA, reflects the calculated control value in the image LA (step S507), and returns to the process of step S501. When the control unit 11 has received an instruction to close the editing window WE (step S501; Yes), the control unit 11 ends the process.

[0087] In this way, in the setting image 800, the image CA corresponding to the instruction value and the image LA corresponding to the control value are arranged close to each other. Therefore, the user can easily check the instruction value when operating the operator and the control value that is actually reflected in the generated sound. Therefore, in a software environment that generates a sound according to the playback position, the efficiency of music production can be improved.

[0088] The control unit 11 changes the position of the setting window image GS in the editing window WE in response to a user instruction. At this time, when the setting window image GS moves to a position where it overlaps with the sound-related image GV, the setting window image GS is an image on a newly opened window in the editing window WE, and covers up the sound-related image GV. In this example, the control unit 11 moves the position of the sound-related image GV in the editing window WE away from the setting window image GS so that the sound-related image GV is not covered up.

[0089] 12 and 13 are examples of an editing window in one embodiment. In the editing window WE shown in FIG. 12, when the setting window image GS is moved to the position shown in FIG. 13 by a user's instruction, the position of the sound-related image GV also moves from the position shown in FIG. 12 to the position shown in FIG. 13. As shown in FIG. 12 and 13, when the setting window image GS is moved, the sound-related image GV moves to a region of the editing window WE that is largely empty relative to the setting window image GS. Here, the intersection point between the sound-related image GV (image GF in this example) and the image GTs is referred to as a reference point GVm. The reference point GVm may be a different position as long as it is linked to the sound-related image GV.

[0090] Next, a display control method for moving the setting window image GS will be described. The display control method described here is executed by displaying an edit window WE including a sound-related image GV and the setting window image GS.

[0091] 14 is a flowchart showing a display control method in one embodiment. The control unit 11 waits for an instruction to close the edit window WE or an instruction to change the position of the setting window image GS (step S601; No, step S603; No). When an instruction to change the position of the setting window image GS is accepted (step S603; Yes), the control unit 11 determines a destination area that satisfies a predetermined condition among the display areas in the edit window WE (step S605). The control unit 11 moves the sound-related image GV to the destination area (step S607) and returns to the process of step S601. When an instruction to close the edit window WE is accepted (step S601; Yes), the control unit 11 ends the process.

[0092] The method for determining the destination region in step S605 will now be described.

[0093] 15 and 16 are diagrams for explaining the positional relationship between the setting window image and the sound-related image in an embodiment. FIG. 15 and FIG. 16 show a simplified configuration of the editing window WE. As shown in FIG. 15, the display area of ​​the editing window WE is divided into a plurality of areas based on the setting window image GS. In this example, the plurality of areas are three areas UA, MA, and BA obtained by dividing the editing window WE so as to be aligned vertically (up and down). The area UA is an area above the top end UL of the setting window image GS. The area BA is an area below the bottom end BL of the setting window image GS. The area MA is an area sandwiched between the top end UL and bottom end BL of the setting window image GS. Depending on the position of the setting window image GS, the area UA or the area BA may not exist. For example, in the example shown in FIG. 16, the area BA does not exist.

[0094] The control unit 11 compares the area UA and the area BA and determines the larger area as the destination area. In this example, the larger area refers to the area with the larger vertical length. The larger area may be an area with a larger area. In the example shown in FIG. 15, the destination area is the area BA. In the example shown in FIG. 16, the destination area is the area UA. The control unit 11 displays the sound-related image GV so that the reference point GVm is located at the vertical center of the destination area and on the image GTs. Therefore, as shown in FIG. 15 and FIG. 16, the length Y1 from the reference point GVm to the top end of the destination area is the same as the length Y2 from the reference point GVm to the bottom end of the destination area. The length Y1 and the length Y2 may be different. The relative relationship between the length Y1 and the length Y2 may be different depending on whether the destination area is the area UA or the area BA. For example, in the case of the area UA, the length Y2 may be greater than the length Y1, and in the case of the area BA, the length Y1 may be greater than the length Y2. By doing so, the sound-related image GV is moved slightly away from the setting window image GS and closer to the outer periphery of the edit window WE.

[0095] When the setting window image GS moves from the position shown in FIG. 15 toward the lower right and reaches the position shown in FIG. 16, the reference point GVm of the sound-related image GV moves from the position shown in FIG. 15 to the position shown in FIG. 16. The movement of the reference point GVm may be performed after the movement of the setting window image GS has stopped. The movement of the reference point GVm may be performed sequentially while the setting window image GS is moving. In the latter case, in the process of the setting window image GS moving from the position shown in FIG. 15 to the position shown in FIG. 16, the reference point GVm first moves gradually downward. Then, at the time when the vertical length of the area UA becomes larger than that of the area BA, the reference point GVm moves significantly from the area BA to the area UA.

[0096] In this way, when the setting window image GS is moved in the editing window WE, the sound-related image GV moves to a wider area, so that the sound-related image GV can be kept easily visible. Therefore, in a software environment that generates sounds according to the playback position, the efficiency of music production can be improved.

[0097] <Modification> The present disclosure is not limited to the above-described embodiment, and includes various other modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present disclosure, and is not necessarily limited to an embodiment having all of the configurations described. In addition, other configurations may be added to the configuration of one embodiment, and some configurations may be deleted or replaced with other configurations. Some modified examples will be described below.

[0098] (1) The processes executed by the above-described program 13a are not limited to being executed entirely in the data processing device 1. For example, some of the processes may be executed in an external device connected via a network. For example, processes that do not require strict real-time performance, such as a process of acquiring intermediate feature data MV using the first trained model, may be executed in an external device.

[0099] (2) The information displayed in the setting image 800 is not limited to the information related to the second parameter P2 of the sound control data CDa. For example, the information related to the third parameter P3 in the sound control data CDb may be displayed in the setting image 800.

[0100] (3) The area in which the images LA and CA are arranged in the setting image 800 is not limited to an arc shape such as the arc areas 810 and 820. For example, this area may be a straight line shape, a curved line shape, or a shape that combines a straight line and a curved line. Regardless of the shape, it is preferable that the start points SP of the two areas are adjacent to each other, and it is preferable that the end points EP of the two areas are adjacent to each other.

[0101] (4) When dividing the display area in the editing window WE into a plurality of areas to determine the destination area of ​​the sound-related image GV, the area may be divided in another manner. For example, the plurality of areas may be divided not so as to be arranged vertically but so as to be arranged horizontally, or so as to be arranged vertically and horizontally. In any case, it is sufficient that the area of ​​the editing window WE in which the setting window image GS does not exist is divided into a plurality of areas. It is sufficient that the widest area among the plurality of areas is determined as the destination area that satisfies a predetermined condition. When the editing window WE is divided into three or more areas, the destination area may be the second widest area instead of the widest area.

[0102] (5) The intermediate feature data MV may be updated any time between when the first parameter P1 is changed and when playback is started in response to a playback start instruction. That is, the intermediate feature data MV may be updated immediately after the first parameter P1 is changed or a predetermined time after the change. The intermediate feature data MV may be updated at a time that is unrelated to when the first parameter P1 is changed, and examples of such times are given below.

[0103] First example: The control unit 11 temporarily stores the first parameters P1 at predetermined time intervals. When storing, the control unit 11 compares information that has already been stored with information to be stored. If at least one piece of information among the temporarily stored first parameters P1 has been changed, the control unit 11 updates the intermediate feature data MV.

[0104] Second example: When a playback start instruction is received, the control unit 11 determines whether at least one piece of information among the first parameters P1 has been changed since the previous playback start instruction was received, and if so, updates the intermediate feature data MV. If no playback start instruction has been received since the program was executed (since the software was started), the control unit 11 may determine whether at least one piece of information among the first parameters P1 has been changed since the software was started or after the music data 13b was read thereafter. [Explanation of symbols]

[0105] 1: data processing device, 11: control unit, 13: storage unit, 13a: program, 13b: music data, 15: display unit, 17: operation unit, 21: interface, 23: communication unit, 100a to 100c: sound generation unit, 120: first trained model, 121: encoding model, 125: first generation model, 130: second trained model, 131: encoding model, 135: second generation model, 140: waveform synthesis unit, 200: data editing unit, 300: read position determination unit, 500: sound mixing unit, 800: setting image, 810: arc region, 820: arc region, 1000: sound output unit

Claims

1. providing a first learned model with the first parameters of a predetermined time range including before and after the predetermined time step out of sound control data including first parameters and second parameters for controlling a generated sound at a plurality of time steps according to the passage of time, the first parameters being included in the predetermined time range including before and after the predetermined time step, thereby acquiring intermediate feature data corresponding to the predetermined time step; updating the intermediate feature data when the value of the first parameter is changed; When a playback start instruction is received, generating a sound signal according to data obtained by providing the second parameters and the intermediate feature data to a second trained model; A signal generating method comprising:

2. the second parameter includes instruction information corresponding to an instruction value of an operator; The signal generating method according to claim 1 .

3. the first parameter includes phoneme information or pitch information; The signal generating method according to claim 2 .

4. displaying a sound control image corresponding to the value of the first parameter based on the sound control data; changing a value of the first parameter when a change operation on the sound control image is accepted; The method of claim 3 further comprising:

5. The second trained model generates sound feature data corresponding to each time step for generating the sound signal; When the sound feature data corresponding to a predetermined time step is generated, sound feature data corresponding to a time step prior to the predetermined time step is further provided to the second trained model. The signal generating method according to claim 1 .

6. determining a control value for controlling the generated sound in accordance with the passage of time based on a value indicated by an operator and a set value determined in advance in a time series; displaying a setting image including a first image area for indicating the instruction value and a second image area adjacent to the first image area for indicating the control value; Further comprising: the second parameter includes the control value; The signal generating method according to claim 1 .

7. displaying a first image corresponding to the sound signal and a second image including an image related to the second parameter in a predetermined display area; changing a position of the second image in the display area in response to an input instruction; When changing the position of the second image, moving the first image to an area that satisfies a predetermined condition among a plurality of areas obtained by dividing the display area based on the second image; Further comprising: The signal generating method according to claim 1 .

8. displaying a first area including an image corresponding to a first parameter including phoneme information or pitch information for controlling a generated sound of a first track, a second area including an image corresponding to a second parameter for controlling the generated sound of the first track, and a third area including an image corresponding to a third parameter for controlling the generated sound of the second track; When a playback start instruction is received, an image corresponding to a current playback position is displayed in the first area and the third area; generating a first sound signal of the first track and a second sound signal of the second track corresponding to the playback position; outputting a mixed sound of the first sound signal and the second sound signal; Including, when an instruction to change the second parameter is received after the instruction to start playback, the changed second parameter is reflected in the first sound signal at the playback position. Signal generation method.

9. The first parameter includes the phoneme information. The signal generating method according to claim 8.

10. the second sound signal at the playback position is generated based on information of the third parameter at or before the playback position; The signal generating method according to claim 8.

11. when an instruction to change the third parameter is received after the instruction to start playback, the changed third parameter is reflected in the second sound signal at the playback position. The signal generating method according to claim 8.

12. the first sound signal at the playback position includes information of the first parameter after the playback position; The signal generating method according to claim 8.

13. the instruction to change the second parameter is input by an operation on an image displayed in the second area; The signal generating method according to claim 8.

14. A program for causing a computer to execute the signal generating method according to any one of claims 1 to 13.

15. determining a control value for controlling the generated sound over time based on a value indicated by an operator and a set value determined in advance in a time series; generating a sound signal based on sound control data for controlling the generated sound over time and the control value; displaying a setting image including a first image area for indicating the instruction value and a second image area adjacent to the first image area for indicating the control value; A display control method comprising:

16. the indication value is indicated by a position in the first image region, the control value is represented by a position in the second image region; a position indicating a minimum value in the first image region and a position indicating a minimum value in the second image region are adjacent to each other, a position showing a maximum value in the first image region and a position showing a maximum value in the second image region are adjacent to each other; The display control method according to claim 15.

17. displaying, in a predetermined display area, a first image corresponding to a sound signal generated based on sound control data including a first parameter and a second parameter for controlling a generated sound according to the passage of time, and a second image including an image related to the second parameter; changing a position of the second image in the display area in response to an input instruction; When changing the position of the second image, moving the first image to an area that satisfies a predetermined condition among a plurality of areas obtained by dividing the display area based on the second image; A display control method comprising:

18. The first image is displayed so as to extend left and right in the display area, The plurality of regions include at least two regions that are vertically arranged on the second image, The region that satisfies the predetermined condition is the larger region of the at least two regions. The display control method according to claim 17.

19. A program for causing a computer to execute the display control method according to any one of claims 15 to 18.

Images