Data modification method, data playback method, and program

Abstract

In the data modification method according to one embodiment: a selection user interface is provided for selecting, from among a plurality of modification methods including a first modification method and a second modification method, a modification method to be applied to sound control data that defines sound timing information; when the application of the first modification is selected, the sound control data is modified by correcting the timing information and adding, to a specific data segment, correction information based on a correction amount of the timing information in said specific data segment; and when the application of the second modification method is applied, the sound control data is modified by correcting the timing information on the basis of the position of a beat based on a specific tempo.

Description

Technical Field

[0001] The present disclosure relates to a method for changing data for controlling pronunciation.

Background Art

[0002] There is a technique for generating a sound waveform signal by automatically performing music based on data for controlling pronunciation (hereinafter referred to as pronunciation control data) such as MIDI data. Generally, the pronunciation control data defines the speed (tempo) of the automatic performance. That is, when the pronunciation control data is reproduced, the performance sound recorded in the pronunciation control data is reproduced. As shown in Patent Documents 1, 2, and 3, techniques for controlling the tempo according to the user's movement have also been developed. The control of the tempo by the user is realized, for example, by the user waving a portable device held in the hand like a baton. Therefore, instead of reproducing the performance sound recorded in the pronunciation control data as it is, a performance sound with a reproduction speed changed according to the user's movement is reproduced.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] Either the automatic performance with tempo control based on the user's operation or the automatic performance with a predetermined tempo control may be selected and the selected automatic performance may be realized. In such a case, it is desirable that the pronunciation control data capable of tempo control by the user's operation and the pronunciation control data capable of a predetermined tempo control be generated as common data. Since pronunciation control data with a predetermined tempo generally exists, it is desirable to change the pronunciation control data with a predetermined tempo so that it can also be used as pronunciation control data capable of tempo control by the user.

[0005] One of the objects of the present disclosure is to easily change pronunciation control data in which performance sounds are recorded into data that enables tempo control by the user's operation.

Means for Solving the Problem

[0006] According to an embodiment of the present disclosure, a selection user interface for selecting a change method to be applied to pronunciation control data defining pronunciation timing information from a plurality of change methods including a first change method and a second change method is provided. When it is selected to apply the first change method, the timing information is corrected, and correction information corresponding to the correction amount of the timing information in the data section is added to a predetermined data section, thereby changing the pronunciation control data. When it is selected to apply the second change method, the pronunciation control data is changed by correcting the timing information based on the positions of beats corresponding to a predetermined tempo. A data change method is provided.

[0007] According to an embodiment of the present disclosure, a plurality of beat positions are detected based on pronunciation control data defining pronunciation timing information recorded at a predetermined tempo, the timing information is corrected based on the relationship between the plurality of beat positions and the tempo, and correction information corresponding to the correction amount of the timing information in the data section is added to a predetermined data section, thereby changing the pronunciation control data. A data change method is provided.

Advantages of the Invention

[0008] According to one embodiment of the present disclosure, pronunciation control data in which performance sounds are recorded can be easily changed to data that enables tempo control by the user's operation.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Mode for Carrying Out the Invention

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The following embodiments are examples, and the present invention is not construed as being limited to these embodiments. In the drawings referred to in this embodiment, the same parts or parts having the same function are given the same reference numerals or similar reference numerals (reference numerals with only A, B, etc. attached after the numbers), and the repeated description thereof may be omitted.

[0011] [1. Overall Configuration] FIG. 1 is a diagram showing a system configuration in one embodiment. The system shown in FIG. 1 includes a mobile device 10 and a data management server 90 connected via a network NW such as the Internet, and further includes an electronic musical instrument 80 connected to the mobile device 10. At least a part of the functions of the electronic musical instrument 80 may be included in the mobile device 10. The mobile device 10 is a portable communication terminal such as a smartphone in this example. The electronic musical instrument 80 is an electronic keyboard device such as an electronic piano in this example.

[0012] The portable device 10 can record a performance on the electronic musical instrument 80 and reproduce the recorded performance on the electronic musical instrument 80. In this reproduction, it is possible to select either reproducing the recorded performance as it is or reproducing it at a speed according to the user's operation (for example, an operation of waving the portable device 10 like a conducting baton). The portable device 10 can generate data corresponding to the recorded performance and change the data so that it can be used in any of the above reproductions. Specific processes for realizing such functions will be described later.

[0013] The data management server 90 includes a control unit 910, a storage unit 920, and a communication unit 980. The control unit 910 includes a CPU, a RAM, and a ROM. The control unit 910 performs processing according to the instructions described in the program by executing the program stored in the storage unit 920 using the CPU. The storage unit 920 includes storage devices such as a non-volatile memory and a hard disk drive. The communication unit 980 includes a communication module for connecting to the network NW and communicating with other devices. The data management server 90 stores a part of the data generated in the portable device 10 in the storage unit 920 and executes processing using the data. The content of the specific processing will be described later. In one embodiment, the data management server 90 may not exist.

[0014] [2-1. Configuration of Portable Device] FIG. 2 is a diagram showing the hardware configuration of the portable device and the electronic musical instrument in one embodiment. The portable device 10 includes a control unit 110, a storage unit 120, a display unit 130, an operation unit 140, a sensor unit 150, a speaker 170, a communication unit 180, and an interface 190. The portable device 10 is not limited to including all of these configurations, and may further include other configurations such as a camera and a position detection unit.

[0015] The control unit 110 is an example of a computer including a processor such as a CPU (Central Processing Unit) and a storage device such as a RAM. The control unit 110 executes the program 121 stored in the storage unit 120 using the CPU (processor), and realizes functions for executing various processes described later in the portable device 10.

[0016] The storage unit 120 is a storage device such as a non-volatile memory and a hard disk drive. The storage unit 120 stores the program 121 executed in the control unit 110 and various data required when executing this program 121. The program 121 is downloaded from the data management server 90 or another server via the network NW and installed in the portable device 10 by being stored in the storage unit 120. The program 121 may be provided in a state recorded on a non-transitory computer-readable recording medium (for example, a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, a semiconductor memory, etc.). In this case, the portable device 10 only needs to be equipped with a device for reading this recording medium. The storage unit 120 can also be said to be an example of a recording medium.

[0017] The data stored in the storage unit 120 includes, for example, performance recording data 123. The performance recording data 123 includes various data related to the performance, and includes, for example, metadata such as the name of the performance piece and data related to the performance sound. The data related to the performance sound includes pronunciation control data 125 that defines the timing information of the pronunciation by the performance. The pronunciation control data 125 is data described in a predetermined format, and is, for example, data described in the MIDI format. In this case, the pronunciation timing information is information indicating the timing when the sound is generated, and is indicated by information such as note on, note off, delta time, tempo, etc. Other information corresponding to the pronunciation, such as information on the pitch (note number), etc. is also included in the pronunciation control data 125.

[0018] The display unit 130 is a display having a display area for displaying various screens according to the control of the control unit 110. The displayed screens include a plurality of user interfaces described later. The operation unit 140 is an operation device that outputs a signal corresponding to a user operation to the control unit 110. In this example, the operation unit 140 is a touch sensor disposed on the surface of the display unit 130. Therefore, the display unit 130 and the operation unit 140 are used as a touch panel by their respective functions. The operation unit 140 may include a switch or the like disposed on the housing of the portable device 10.

[0019] The sensor unit 150 outputs a signal corresponding to the movement of the portable device 10 to the control unit 110. The sensor unit 150 includes, for example, an acceleration sensor, a gyro sensor, etc., and measures the movement of the portable device 10. The speaker 170 generates sound by amplifying and outputting the sound waveform signal supplied from the control unit 110.

[0020] The communication unit 180 is a wireless communication module that connects to the network NW under the control of the control unit 110 and communicates with other devices such as the data management server 90 connected to the network NW. The interface 190 includes a communication module for communicating with other devices such as the electronic musical instrument 80 by infrared communication or short-range wireless communication. The interface 190 is used for communication without going through the network NW. The interface 190 may have a module that performs wired communication instead of wireless communication.

[0021] [2-2. Configuration of Electronic Musical Instrument] As described above, the electronic musical instrument 80 is an electronic keyboard device such as an electronic piano, and includes a performance operator 810, a sound source unit 830, a speaker 870, and an interface 890. The performance operator 810 includes a plurality of keys and outputs a signal corresponding to an operation on each key to the sound source unit 830. The interface 890 includes a communication module for communicating with an external device wirelessly or by wire. In this example, the interface 890 is connected to the interface 190 of the portable device 10 by short-range wireless communication and transmits and receives data in a predetermined format (MIDI format in this example).

[0022] The sound source unit 830 includes a DSP (Digital Signal Processor) and generates a sound waveform signal in response to a sound production instruction signal. The sound production instruction signal corresponds to a signal output from the performance operator 810 and data transmitted from the portable device 10 via the interface 890. Further, the sound source unit 830 converts a sound production instruction signal corresponding to a signal output from the performance operator 810 into MIDI format data and outputs it to the interface 890. Thus, the electronic musical instrument 80 can also transmit data (hereinafter sometimes referred to as performance information) corresponding to an operation on the performance operator 810 to the portable device 10.

[0023] [3. Recording Mode] Next, the process when recording a performance on the electronic musical instrument 80 in the portable device 10 will be described. The processing mode for recording a performance in this way is called the recording mode. When an instruction to start the recording mode is input by the user, the portable device 10 starts the processing of the recording mode.

[0024] FIG. 3 is a flowchart showing the processing of the recording mode in an embodiment. FIG. 4 is a diagram showing an example of a recording user interface (before the recording start instruction). FIG. 5 is a diagram showing an example of a recording user interface (after the recording instruction). The control unit 110 provides a recording user interface (recording UI) to the user by displaying the screens described below on the display unit 130 (step S101). The recording user interface (hereinafter sometimes referred to as interface RD) shown in FIG. 4 is a display example before an instruction to start performance recording is given. The interface RD includes an area TS for setting the tempo during recording (hereinafter sometimes referred to as the recording tempo), an area RT for displaying the recording time, and a recording button RB for instructing the start and end of recording.

[0025] The control unit 110 waits until an instruction to start recording performance is input (step S103; No). When an instruction to start recording is input to the mobile device 10 by operating the recording button RB shown in FIG. 4 (step S103; Yes), the control unit 110 changes the display of the recording button RB as shown in FIG. 5 and starts providing beats to the user in order to start recording the performance (step S105). The provision of beats is realized, for example, by generating a metronome sound from the mobile device 10. The metronome sound is generated corresponding to each beat specified at intervals determined by the recording tempo. For example, if the recording tempo is 120, the metronome sound is generated at a length of 500 milliseconds per beat. The mobile device 10 may provide beats not only by sound to the user, but also by light, vibration, or the like.

[0026] The control unit 110 records the tone generation control data 125 in the storage unit 120 based on the performance information provided from the electronic musical instrument 80 until an instruction to end recording is input (step S109; No) (step S107). At this time, the provision of beats to the user continues. When an instruction to end recording is input to the mobile device 10 by operating the recording button RB shown in FIG. 5 (step S109; Yes), the control unit 110 ends the provision of beats (step S111). Subsequently, the control unit 110 provides a selection user interface (selection UI) by displaying the following-described screen on the display unit 130 (step S113).

[0027] By providing the selection user interface in this way, it is possible to provide the user with the processing to be performed after recording the tone generation control data 125 by performance. On the other hand, the control unit 110 may end once when the recording of the tone generation control data 125 is completed according to the user's instruction. In this case, the control unit 110 may provide a selection user interface according to the user's instruction and further provide a user interface for designating the tone generation control data 125 to be the target of processing. It may be possible to designate as the target of processing other than the tone generation control data 125 recorded in the recording mode.

[0028] FIG. 6 is a diagram showing an example of a selection user interface. As shown in FIG. 6, the selection user interface (hereinafter sometimes referred to as interface SD) includes an interface for selecting a method of changing the recorded pronunciation control data 125 from a plurality of different change methods. The plurality of change methods may be any change methods as long as they change the pronunciation control data 125 by different processes. The plurality of change methods may include at least two change methods that change the pronunciation control data 125 by correcting timing information in different ways. In this example, the plurality of change methods includes three change methods. The three change methods correspond to "AI quantization" (first change method), "MIDI quantization" (second change method), and "delay offset" (third change method). The interface SD includes change selection buttons AB, MB, and DB corresponding to the respective change methods. "AI quantization" is a process of detecting the beat position from the pronunciation control data 125 and changing the pronunciation control data 125 so as to correct the timing information according to the relationship between the beat position specified from the original pronunciation control data 125 and the detected beat position. When detecting the beat position, AI (Artificial Intelligence) technology is used. "MIDI quantization" is a process of changing the pronunciation control data 125 so as to correct the timing information according to the specified resolution. "Delay offset" is a process of changing the pronunciation control data 125 so as to offset the timing information corresponding to a plurality of sounds. These specific processing methods will be described later.

[0029] The interface SD includes a save button B1 for saving the pronunciation control data 125, a delete button B2 for deleting the pronunciation control data 125, and a playback button PB for playing back the pronunciation control data 125. When the playback button PB is operated, the pronunciation control data 125 is read out and the sound indicated by the pronunciation control data 125 is played back. This sound may be generated in the mobile device 10 or may be generated in the electronic musical instrument 80. When the sound is generated by the electronic musical instrument 80, a pronunciation instruction signal may be transmitted from the mobile device 10 to the electronic musical instrument 80. The playback position is indicated by the playback marker PM. By changing the position of the playback marker PM by the user, the playback position in the pronunciation control data 125 can also be changed.

[0030] FIG. 7 is a flowchart showing the processing of the recording mode (the part following FIG. 3) in an embodiment. The control unit 110 waits for an instruction input from the user to the interface SD (step S201; No, S203; No, S205; No, S207; No, S209; No). The state in which the control unit 110 waits for an instruction in this way is called an instruction waiting state.

[0031] When an instruction for AI quantization is input by operating the change selection button AB (step S201; Yes), the control unit 110 executes the AI quantization process (step S300) and returns to the instruction waiting state. The AI quantization process will be described later.

[0032] When an instruction for MIDI quantization is input by operating the change selection button MB (step S203; Yes), the control unit 110 executes the MIDI quantization process (step S400) and returns to the instruction waiting state. The MIDI quantization process will be described later.

[0033] When an instruction for delay offset is input by operating the change selection button DB (step S205; Yes), the control unit 110 executes delay offset processing (step S500) and returns to the instruction waiting state. The delay offset processing will be described later.

[0034] When a data save instruction is input by operating the save button B1 (step S207; Yes), the control unit 110 saves the pronunciation control data 125 (step S211) and ends the processing in the recording mode. When a data deletion instruction is input by operating the delete button B2 (step S209; Yes), the control unit 110 deletes the pronunciation control data 125 (step S213) and ends the processing in the recording mode.

[0035] [4. AI Quantization Processing] Subsequently, the AI quantization processing will be described.

[0036] FIG. 8 is a flowchart showing the AI quantization processing in an embodiment. The control unit 110 converts the pronunciation control data 125 into audio data (step S301). The audio data is data indicating pronunciation as a sound waveform signal according to the timing information defined in the pronunciation control data 125. This sound waveform signal may be generated in the mobile device 10 or may be generated by the electronic musical instrument 80 and received by the mobile device 10. The control unit 110 detects beats based on the audio data (step S303).

[0037] Beats can be detected based on changes in the amplitude of the sound waveform signal, etc. In this example, AI technology is used for beat detection. By inputting the audio data obtained by conversion into a learned model that has learned the relationship between the audio data and the positions of the beats, the positions of the beats can be obtained from the learned model. The specific method for detecting beats is not limited to the case of using AI technology, and known beat detection methods can be applied. Beat detection is not limited to being performed on the audio data, and may also be performed on the pronunciation control data 125.

[0038] The control unit 110 provides a beat correction interface (beat correction UI: first user interface) by displaying the screens described below on the display unit 130 (step S305). The control unit 110 waits until an instruction to correct the beat position or an instruction for data correction processing is input to the beat correction interface (step S311; No, S321; No). When an instruction to correct the beat position is input (step S311; Yes), the control unit 110 corrects the beat position (step S313) and waits again (step S311; No, S321; No). When data correction processing is instructed (step S321; Yes), the control unit 110 executes the data correction processing (step S330) and ends the AI quantization processing. There may be a case where data correction processing is instructed without an instruction to correct the beat position. In this case, the data correction processing is executed using the detected beat position as it is.

[0039] FIG. 9 is a diagram showing an example of a beat correction user interface (before correction instruction). FIG. 10 is a diagram showing an example of a beat correction user interface (before correction instruction). The beat correction interface (hereinafter sometimes referred to as interface AD) includes an area TS for setting the average tempo, line buttons LB and tap buttons TB for setting the editing method of the beat position, a play button PB, a re-detection button QB, and a save button B3. The average tempo is displayed with the tempo (recording tempo) at the time of recording the pronunciation control data 125 as the initial value. Usually, there is no need to change it from the initial value, but it may be changed. The interface AD described in FIGS. 9 and 10 is shown as the editing method when the line button LB is operated.

[0040] The interface AD further includes an overall area AA indicating the pronunciation defined in the pronunciation control data 125 over the entire range of the song, and an enlarged area AW indicating the enlarged range specified by the selection window SW. In these areas, the vertical axis indicates pitch, the horizontal axis indicates time, and each pronunciation is indicated by a note mark NM. The selection window SW can change its position in the overall area AA and can also change the magnification rate in the time axis direction when displayed in the enlarged area AW by changing the length in the horizontal axis direction.

[0041] The position of the beat detected from the pronunciation control data 125 is indicated by a beat position line BL in the enlarged area AW. Below the beat position line BL, circular beat markers BM are displayed. The beat markers BM are used when the user corrects the beat position. Among the beat markers BM, the beat markers BMs designated by the user as the objects to be changed are displayed in a form distinguishable from other beat markers BM. At this time, as shown in FIG. 9, the display form of the beat position line BL corresponding to the beat markers BMs may also change.

[0042] When the beat markers BMs designated for the user in FIG. 9 are shifted to the left as shown in FIG. 10, it corresponds to the input of a correction instruction for the beat position (FIG. 8, step S311; Yes). Accordingly, the control unit 110 corrects the position of the beat corresponding to the beat markers BMs to the position of the beat markers BMs after movement (FIG. 8, step S313). Such correction is performed when the detected beat position is different from the beat position assumed by the user.

[0043] When the play button PB is operated, the control unit 110 plays the sound corresponding to the pronunciation control data 125 in the range displayed in the enlarged area AW. At this time, a sound indicating the beat (metronome sound) may be generated at the detected beat position. In this way, the relationship between the detected beat position or the corrected beat position and the pronunciation timing may be recognized by the user through the sound.

[0044] When the re-detection button QB is operated, the control unit 110 returns to step S303 to execute beat detection. At this time, if there is a beat whose position has been corrected by the user, the control unit 110 may execute the beat detection process while fixing the position of the corrected beat.

[0045] FIG. 11 is a diagram for explaining an example of pronunciation control data. FIG. 11 shows the relationship between the timing information of the pronunciation control data 125 and the beat position at the time of "recording", "beat detection", and "beat correction". "Recording time" indicates the pronunciation control data 125 when recorded in the recording mode. "Beat detection time" indicates the pronunciation control data 125 when a beat is detected in the AI quantization process. "Beat correction time" indicates the pronunciation control data 125 when the detected beat position is corrected by the user. The horizontal axis represents time t. The sound NT indicates the position of the pronunciation corresponding to the timing information. Since the pronunciation control data 125 has not changed during the period from recording to beat correction, the position of the sound NT does not change.

[0046] The beat positions SBT1, SBT2, ··· indicate the positions of the beats at the time of recording. Therefore, the length of each beat is the same. The beat positions DBT1, DBT2, ··· indicate the positions of the beats at the time of beat detection. The detected beat positions are not necessarily equally spaced. Therefore, in the example of FIG. 11, there are beats whose positions (timings) are shifted between recording and beat detection. The beat correction time indicates the beats after correcting the beat position by moving the beat markers BMs at the interface AD. FIG. 11 shows an example in which the beat positions DBT4 and DBT6 are corrected. The correction of the beat position DBT4 corresponds to the example shown in FIG. 10. The correction of the beat position DBT6 is for adjusting the decorative sound NTs to the beat position. The purpose of such correction is related to the data reproduction process and will be described later.

[0047] When the save button B3 in the interface AD is operated, it corresponds to the input of an instruction for data correction processing (step S321; Yes). Therefore, when the save button B3 is operated, the control unit 110 executes data correction processing (step S330). At this time, the mobile device 10 may associate the pronunciation control data 125 or the audio data with the data indicating the corrected beat position and transmit it to the data management server 90. The data management server 90 registers the data received from the mobile device 10 in the database in the storage unit 920. The data management server 90 can execute processing using the data registered in the database. For example, such corrected data by the user can also be used to improve the accuracy of the beat detection technology or to improve the accuracy of the AI technology in beat detection (such as updating the learned model using the registered data as teacher data).

[0048] FIG. 12 is a flowchart showing data correction processing in an embodiment. The control unit 110 divides the pronunciation control data 125 at the positions of the above-described beat position DBT1, DBT2,... to divide it into data sections for each detected beat (corrected beat if corrected) (step S331). The control unit 110 expands and contracts the divided plurality of data sections to a length corresponding to the section length corresponding to the above-described average tempo (the length of one beat corresponding to the average tempo), and corrects the position of each pronunciation included in each data section, that is, the timing information, so as to correspond to the expansion and contraction amount (step S333). The control unit 110 adds correction information corresponding to the correction amount of the timing information to each data section (step S335), and combines the plurality of data sections to which the correction information with the corrected timing information is added (step S337). The data correction processing will be described in more detail with reference to FIGS. 13 and 14.

[0049] FIG. 13 is a diagram for explaining an example when correcting timing information. FIG. 13 shows an example of correcting timing information in a data section between beat positions DBT4 and DBT5 among a plurality of data sections. "After division" indicates pronunciation control data 125 in the data section before correction. "After correction" indicates pronunciation control data 125 after correcting the timing information. In this example, the data section length before correction (the length between beat positions DBT4 and DBT5) corresponds to the tempo "110".

[0050] In this example, the average tempo is set to "120". Therefore, the data section length is changed to a length corresponding to the tempo "120". Since the data section length has become "110 / 120" times, the timing information of the pronunciation included in this data section is corrected to a position "110 / 120" times based on the beginning of the beat (beat position DBT4). Thus, the correction amount is "110 / 120". The correction information is added as a value corresponding to the correction amount at the position corresponding to beat position DBT4 and the tempo value "110" corresponding to the original data section length. It may be added as a relative value to the average tempo as "110 / 120". In the case of MIDI format data, it is added as tempo change information. When the pronunciation control data 125 of the data section corrected in this way is reproduced at the tempo "110", the performance sound at the same timing as when the pronunciation control data 125 was recorded can be reproduced.

[0051] FIG. 14 is a diagram for explaining by comparing pronunciation control data before and after changes. FIG. 14 shows the relationship between the timing information of the pronunciation control data 125 and the position of the beats at the time of "recording" and "after combination". "At the time of recording" indicates the pronunciation control data 125 when recorded in the recording mode, that is, before the timing information is changed by the AI quantization process. "After combination" indicates the pronunciation control data 125 after combining the data sections in which the timing information is corrected by the AI quantization process and correction information is added, that is, after the timing information is changed. As shown in FIG. 14, in the AI quantization process, the detected beat (or further corrected beat) is aligned to a predetermined tempo (average tempo in this example), and correction information corresponding to the correction amount (tempo value corresponding to the length of the original beat in this example) is added so that the length of the original beat can be specified, thereby changing the pronunciation control data 125.

[0052] When the timing information is changed by the AI quantization process in this way, the pronunciation control data 125 can take a data format described as a constant tempo, and by reflecting the correction information as a tempo change and reproducing it, the performance sound when the pronunciation control data 125 was recorded can also be reproduced. This is particularly effective for songs for which a natural performance is desired. The above is the explanation of the AI quantization process.

[0053] [5. MIDI Quantization Process] Next, the MIDI quantization process will be described.

[0054] FIG. 15 is a flowchart showing the MIDI quantization process in an embodiment. The control unit 110 provides a quantization setting user interface (quantization process setting UI) by displaying the screens described below on the display unit 130 (step S401), and waits until an instruction for the quantization process is input (step S403; No). When an instruction for the quantization process is input (step S403; Yes), the control unit 110 executes the quantization process according to the settings (step S405), and ends the MIDI quantization process.

[0055] FIG. 16 is a diagram showing an example of a quantization setting user interface. The quantization setting user interface (hereinafter sometimes referred to as interface MD) includes a region QS for setting the resolution of quantization and an execution button B4 for inputting an instruction for quantization processing. In this example, interface MD includes a window RW for selecting the resolution to be set. If the resolution of quantization is "1 / 8", the timing information is corrected so that the timings are aligned with the eighth note as the unit. This processing is the same as general quantization processing. Interface MD may include a region for setting conditions for sounds that are not subject to quantization. Sounds that are not subject to quantization may move, for example, by the same amount as the nearest quantization target sound. Sounds that are not subject to quantization processing may be, for example, sounds with a length less than half of the resolution or sounds with a velocity (volume) smaller than a predetermined value. Conversely, interface MD may include a region where the target sound for quantization can be specified.

[0056] According to MIDI quantization processing, when a situation occurs where the rhythm fluctuates during performance when recording the pronunciation control data 125, the fluctuation of the rhythm of the pronunciation included in the pronunciation control data 125 can be removed and aligned to the beat position. This is particularly effective for songs where rhythm is important, such as dance music. The above is the explanation of MIDI quantization.

[0057] [6. Delay Offset Processing] Subsequently, the delay offset processing will be described.

[0058] FIG. 17 is a flowchart showing delay offset processing in one embodiment. The control unit 110 provides an offset setting user interface (offset setting UI: second user interface) by displaying the screens described below on the display unit 130 (step S501), and waits until an instruction for offset processing is input (step S503; No). When an instruction for offset processing is input (step S503; Yes), the control unit 110 executes offset processing according to the settings (step S505), and ends the delay offset processing.

[0059] FIG. 18 is a diagram showing an example of an offset setting user interface. The offset setting user interface (hereinafter sometimes referred to as interface DD) includes an area OS for setting the offset amount and an execution button B5 for inputting an instruction for offset processing. In this example, interface DD includes a window RW for selecting the offset amount to be set. If the offset amount is "30", the pronunciation control data 125 is changed by correcting the timing information so as to delay the whole by 30 ticks.

[0060] Interface DD may have an area for designating the sound to be processed. By designating the sound to be processed, the timing information may be corrected for the target sound and a plurality of sounds following the target sound. When the sound to be processed is not designated, the first sound of the pronunciation control data 125 may be designated as the sound to be processed. Two examples of the case where the correction is performed by designating the sound to be processed will be described with reference to FIGS. 19 and 20.

[0061] Figs. 19 and 20 are diagrams for explaining by comparing pronunciation control data before and after a delay offset. Both Figs. 19 and 20 show the relationship between the timing information of the pronunciation control data 125 and the position of the beat at "when the beat is detected" and "after the offset". "When the beat is detected" indicates the pronunciation control data 125 when the beat is detected in the AI quantization process. "After the offset" indicates the pronunciation control data 125 after the offset process is executed with the decorative sound NTs specified as the processing target sound. Since it is assumed that the decorative sound is adjusted to the beat position by the offset process, it is assumed that the offset amount is specified so as to move the decorative sound NTs to the beat position DBT6.

[0062] In the first example shown in Fig. 19, the timing information is corrected so that the decorative sound NTs move to the beat position DBT6 according to the offset amount. The timing information of any of the plurality of sounds following the decorative sound NTs is corrected by the same offset amount. At this time, the position of the correction information may be further changed. The change in the position of the correction information corresponds to the change in the data section corrected by the correction information. The amount by which the position of the correction information is changed may correspond to the offset amount.

[0063] In the second example shown in Fig. 20, the timing information is corrected so that the decorative sound NTs move to the beat position DBT6 in the same manner. On the other hand, for the plurality of sounds following the decorative sound NTs, the timing information is corrected so that they move with a smaller offset amount as they are farther from the decorative sound NTs. At this time, the timing information may be corrected only for the sounds within a predetermined number of beats (for example, within 4 beats) from the decorative sound NTs.

[0064] [7. Data playback processing] Next, a process of reading out the pronunciation control data 125 generated as described above (for example, the "after combination" pronunciation control data 125 shown in FIG. 14) and reproducing a sound waveform signal in the sound source unit 830 will be described. The mobile device 10 outputs a pronunciation instruction signal based on the pronunciation control data 125 to the electronic musical instrument 80 in order to reproduce the sound waveform signal in the sound source unit 830. When the mobile device 10 receives an instruction from the user to start the reproduction process of the pronunciation control data 125, it starts the data reproduction process. In the data reproduction process, the modes of reproducing the pronunciation control data 125 include an auto mode and a control mode.

[0065] The auto mode reads out the pronunciation control data 125 at a predetermined tempo, and corrects the tempo using the correction information set for each beat, thereby realizing the reproduction of the sound waveform signal at a timing where the position of the beat is substantially corrected. It is a mode for reproducing the performance sound when the data was recorded. If the correction information is not used when reading out the pronunciation control data 125, as shown in FIG. 14, a performance sound different from the performance sound when the data was recorded will be reproduced. However, the performance sound when the data was recorded can be reproduced by changing the reading speed using the correction information. When using the pronunciation control data 125 for which the AI quantization process has not been executed, since the correction information is not included, the pronunciation control data 125 will be read out at a predetermined tempo.

[0066] The control mode is a mode for reading out pronunciation control data 125 according to a tempo indication signal that advances the tempo, and reproducing the sound when played at the speed corresponding to the signal. By instructing the advancement of the tempo by the tempo indication signal, in the reading of the pronunciation control data 125, the timing at which each tempo starts is controlled, and the tempo between tempos is controlled based on the tempo between tempos indicated in the past. The tempo indication signal is generated according to the user's operation. For example, it is generated according to the timing of changing the swinging direction in the operation of swinging the mobile device 10 like a baton. The control unit 110 can specify the position of the tempo arranged at regular intervals in the pronunciation control data 125 from the tick value if it is in MIDI format. A data position with correction information added in the pronunciation control data 125 may be specified as the position of the tempo.

[0067] As described above, it may be desirable to correct the grace note to match the position of the tempo. The grace note often exists slightly before the position of the tempo. In such a case, it is desirable that the decorated note existing at the position of the tempo and the grace note existing before it are reproduced continuously. On the other hand, when operating in the control mode, since the pronunciation control data 125 is read out with the position of the tempo as a delimiter, a situation may occur where the grace note is reproduced at an early timing and the grace note and the decorated note are reproduced with a large separation. Assuming such a case, by correcting the timing information so that the grace note is located at the position of the tempo, the grace note and the decorated note can be reproduced as a single unit without being separated.

[0068] FIG. 21 is a flowchart showing data reproduction processing in an embodiment. The control unit 110 provides a reproduction user interface (reproduction UI) by displaying the screens described below on the display unit 130 (step S801), and waits until an instruction to start data reproduction is input (step S803; No).

[0069] When an instruction to start data playback is input (step S803; Yes), the control unit 110 executes pronunciation control processing by playing back pronunciation control data 125 until an instruction to stop data playback is input (step S821; No). When the auto mode is selected as the playback mode (step S811; Yes), the control unit 110 executes pronunciation control processing in the auto mode (step S813). On the other hand, when the auto mode is not selected, that is, when the control mode is selected as the playback mode (step S813; No), the control unit 110 executes pronunciation control processing in the control mode (step S815). When an instruction to stop data playback is input (step S821; Yes), the control unit 110 ends the data playback process.

[0070] FIG. 22 is a diagram showing an example of a playback user interface. The playback user interface (hereinafter sometimes referred to as interface PD) includes a switching button PMB for switching between the auto mode and the control mode. The switching button PMB includes an image indicating whether it is operating in the auto mode or the control mode. It may be selected so that the mode is switched each time the switching button PMB is operated, or either mode (for example, the control mode) may be selected only while the switching button PMB is being operated (while touching the switching button PMB).

[0071] When operating in the auto mode, a stop button B6 for stopping the progress of the song is displayed. When in the control mode, since the progress of the song stops if the user stops the operation of shaking the mobile device 10, the stop button B6 may not be displayed as shown in FIG. 22. Operating the stop button B6 corresponds to inputting an instruction to stop data playback (step S821; Yes), but it may be recognized as an instruction to temporarily stop data playback by returning to step S803.

[0072] Thus, according to the data change method in one embodiment, the pronunciation control data 125 recorded in response to performance can be changed into data that can be used in a plurality of different playback methods such as an auto mode and a control mode. Also, an appropriate change method can be selected according to the piece of music to change the pronunciation control data 125.

[0073] [8. Modification Example] The present disclosure is not limited to the above-described embodiments, and includes various other modification examples. For example, the above-described embodiments have been described in detail for the purpose of explaining the present disclosure clearly, and are not necessarily limited to those having all the configurations described. It is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations. Some modification examples will be described below.

[0074] (1) In the AI quantization process, although the correction information was added corresponding to each beat, it may be added corresponding to every two or more beats, or may be added for each sound.

[0075] (2) A process of analyzing an image of a musical score to extract bar lines, notes, performance symbols, etc., and correcting the timing information of the pronunciation control data 125 based on the extracted information may be selectable in the interface SD.

[0076] (3) When the portable device 10 executes the data playback method, in the control mode, although the progress instruction signal was generated by the operation of shaking the portable device 10, it may be generated by the operation of shaking the sensor terminal connected to the portable device 10 via the interface 190. The sensor terminal may include, for example, an acceleration sensor, a gyro sensor, etc. as a configuration corresponding to the sensor unit 150, and it only needs to have a function for measuring the movement of the sensor terminal and a function for connecting to the interface 190 of the portable device 10. When using the sensor terminal, the portable device 10 may not be a device that can be carried, but may be realized by a stationary desktop personal computer or the like. Some or all of the above functions realized in the portable device 10 may be realized by an information processing device such as one or more servers connected via the network NW. For example, when the pronunciation control data 125 is recorded in the recording mode, the portable device 10 transmits the pronunciation control data 125 to the server. The server may execute a process for changing the pronunciation control data 125 and transmit the changed pronunciation control data 125 to the portable device 10.

[0077] [Summary of the solution method] According to an embodiment of the present disclosure described above, it is possible to describe as the following configuration.

[0078] The data change method of the present disclosure provides a selection user interface for selecting a change method to be applied to pronunciation control data that defines pronunciation timing information from a plurality of change methods including a first change method and a second change method. When it is selected to apply the first change method, the timing information is corrected, and the pronunciation control data is changed by adding correction information corresponding to the correction amount of the timing information in the data section to the predetermined data section. When it is selected to apply the second change method, the pronunciation control data is changed by correcting the timing information based on the position of the beat according to a predetermined tempo.

[0079] The data change method of the present disclosure may provide a recording user interface for setting a tempo and instructing performance recording. When the start of the performance recording is instructed, beat information is provided at the set tempo, and the pronunciation control data may be recorded based on the performance information input while the beat information is being provided. When the stop of the performance recording is instructed, a selection user interface for selecting a change method to be applied to the recorded pronunciation control data may be provided.

[0080] The data change method of the present disclosure may further include a third change method among the plurality of change methods. When it is selected to apply the third change method, the pronunciation control data may be changed by changing the specified timing information of a predetermined pronunciation and changing the timing information of a plurality of pronunciations following the predetermined pronunciation.

[0081] The data change method of the present disclosure includes detecting a plurality of beat positions based on pronunciation control data that defines the timing information of pronunciations recorded at a predetermined tempo, correcting the timing information based on the relationship between the plurality of beat positions and the tempo, and changing the pronunciation control data by adding correction information corresponding to the correction amount of the timing information in the data section for a predetermined data section.

[0082] The data change method of the present disclosure may further include recording the pronunciation control data based on the beat information provided at a predetermined tempo and based on the performance information input while the beat information is being provided, and the correction information may be added corresponding to the timing of the provision of the beat information.

[0083] Detecting the plurality of beat positions may include converting the pronunciation control data into audio data and detecting a plurality of beat positions based on the audio data.

[0084] The data modification method of the present disclosure may further include providing a first user interface for correcting the detected beat positions, and modifying the pronunciation control data may include correcting the timing information based on the relationship between the corrected positions of the plurality of beats and the set tempo via the first user interface.

[0085] The data modification method of the present disclosure may register in a database by associating the recorded pronunciation control data or the audio data with the beat positions after being corrected via the first user interface.

[0086] The data modification method of the present disclosure may provide a second user interface for correcting the timing information of a predetermined pronunciation in the pronunciation control data, and may further modify the pronunciation control data by correcting the timing information of the predetermined pronunciation instructed via the second user interface and further correcting the timing information of a plurality of pronunciations following the predetermined pronunciation.

[0087] The data modification method of the present disclosure may correct the timing information of the predetermined pronunciation instructed via the second user interface, and when further correcting the timing information of a plurality of pronunciations following the predetermined pronunciation, may further modify the pronunciation control data by further changing the data section targeted by the correction information.

[0088] The data playback method of the present disclosure is a data playback method for reproducing a sound waveform signal by a sound source unit based on the pronunciation control data changed by the data change method described above. A third user interface for selecting a method for generating a pronunciation instruction signal for causing the sound source unit to reproduce a sound waveform signal from a plurality of playback methods including a first playback method and a second playback method is provided. When the first playback method is selected, the pronunciation control data is read out according to a predetermined tempo, and the pronunciation instruction signal is generated using the correction information. When the second playback method is selected, the pronunciation control data of the data section corresponding to a progress instruction signal for advancing the beat is read out to generate the pronunciation instruction signal.

[0089] The data change method of the present disclosure provides a recording user interface for instructing a performance recording. When the start of the performance recording is instructed, pronunciation control data defining pronunciation timing information is recorded based on the input performance information. When the stop of the performance recording is instructed, a selection user interface for selecting a change method to be applied to the recorded pronunciation control data from a plurality of change methods including a first change method and a second change method is provided. The pronunciation control data is changed based on the selected change method, and a performance sound is reproduced based on the changed pronunciation control data.

[0090] According to the present disclosure, it can also be used as a program for causing a computer to execute the above data change method or data playback method, and can also be used as a data change device for executing the data change method and a data playback device for executing the data playback method. That is, at least a part of the mobile device 10 can function as a data change device or a data playback device.

Explanation of Signs

[0091] 10: Mobile device, 110: Control unit, 120: Memory unit, 121: Program, 123: Performance recording data, 125: Sound generation control data, 130: Display unit, 140: Operation unit, 150: Sensor unit, 170: Speaker, 180: Communication unit, 190: Interface, 80: Electronic musical instrument, 810: Performance operator, 830: Sound source unit, 870: Speaker, 890: Interface, 90: Data management server, 910: Control unit, 920: Memory unit, 980: Communication unit

Claims

1. Provided is a selection user interface for selecting a modification method to be applied to pronunciation control data defining pronunciation timing information from a plurality of modification methods including a first modification method and a second modification method, when it is selected to apply the first modification method, the pronunciation control data is modified by correcting the timing information and adding correction information corresponding to the correction amount of the timing information in the data section to a predetermined data section, when it is selected to apply the second modification method, the pronunciation control data is modified by correcting the timing information based on the position of beats according to a predetermined tempo, A data modification method including the above.

2. Provided is a recording user interface for setting a tempo and instructing a performance recording, when the start of the performance recording is instructed, beat information is provided at the set tempo, the pronunciation control data is recorded based on the performance information input while the beat information is provided, when the stop of the performance recording is instructed, the selection user interface for selecting a modification method to be applied to the recorded pronunciation control data is provided. The data modification method according to claim 1.

3. The plurality of modification methods further includes a third modification method, when it is selected to apply the third modification method, the pronunciation control data is modified by correcting the timing information of a specified predetermined pronunciation and correcting the timing information of a plurality of pronunciations following the predetermined pronunciation. The data modification method according to claim 1 or claim 2.

4. Based on pronunciation control data defining the timing information of pronunciations recorded at a predetermined tempo, a plurality of beat positions are detected, the timing information is corrected based on the relationship between the plurality of beat positions and the tempo, and the pronunciation control data is modified by adding correction information corresponding to the correction amount of the timing information in the data section to a predetermined data section, A data modification method including the above.

5. Providing beat information at a predetermined tempo, recording the pronunciation control data based on the performance information input while the beat information is provided, further including, The data modification method according to claim 4, wherein the correction information is added corresponding to the timing of providing the beat information.

6. Detecting the plurality of beat positions is, Convert the pronunciation control data into audio data, detect the positions of a plurality of beats based on the audio data, including The data change method according to claim 4 or claim 5.

7. further comprising providing a first user interface for correcting the detected beat positions, changing the pronunciation control data includes correcting the timing information based on the relationship between the positions of the plurality of beats after being corrected via the first user interface and a set tempo, the data change method according to claim 6.

8. further comprising registering in a database by associating the recorded pronunciation control data or the audio data with the beat positions after being corrected via the first user interface, the data change method according to claim 7.

9. providing a second user interface for correcting the timing information of a predetermined pronunciation in the pronunciation control data, correcting the timing information of the predetermined pronunciation instructed via the second user interface, and further changing the pronunciation control data by further correcting the timing information of a plurality of pronunciations following the predetermined pronunciation, further comprising, the data change method according to any one of claims 6 to 8.

10. When correcting the timing information of the predetermined pronunciation instructed via the second user interface and further correcting the timing information of a plurality of pronunciations following the predetermined pronunciation, the pronunciation control data is further changed by further changing the data section targeted by the correction information, the data change method according to claim 9.

11. A data playback method for playing a sound waveform signal by a sound source unit based on the pronunciation control data changed by the data change method according to any one of claims 4 to 10, providing a third user interface for selecting a method for generating a pronunciation instruction signal for causing the sound source unit to play a sound waveform signal from a plurality of playback methods including a first playback method and a second playback method, when the first playback method is selected, reading the pronunciation control data according to a predetermined tempo, and generating the pronunciation instruction signal using the correction information, When the second playback method is selected, reading out the pronunciation control data of the data section corresponding to a progress instruction signal for advancing the beat and generating the pronunciation instruction signal A data playback method including this

12. Providing a recording user interface for instructing performance recording When the start of the performance recording is instructed, recording pronunciation control data defining pronunciation timing information based on the input performance information When the stop of the performance recording is instructed, providing a selection user interface for selecting a change method to be applied to the recorded pronunciation control data from a plurality of change methods including a first change method and a second change method Changing the pronunciation control data based on the selected change method Playing back performance sounds based on the changed pronunciation control data A data change method including this

13. A program for causing a computer to execute the data change method according to any one of Claims 1 to 10 and Claim 12

14. A program for causing a computer to execute the data playback method according to Claim 11

Images