Acoustic processing system, acoustic processing method, and program

Abstract

To enable a user to easily check and correct an analysis result of a reference signal in a process of generating reference data.SOLUTION: An acoustic processing system 10 includes an acoustic analysis unit 43 and a display control unit 36. The acoustic analysis unit 43 executes an analysis process for analyzing a reference signal Zr containing acoustics of a musical instrument 80. The analysis process includes a process of identifying a time series of a pronunciation index that is an index of probability that the reference signal Zr contains the acoustic component of the musical instrument 80, and a process of identifying a time series of pitch related to the acoustic component of the musical instrument 80. The display control unit 36 causes a display device 15 to display the time series of the pronunciation index and the time series of the pitch on a common time axis.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present disclosure relates to techniques for analyzing acoustic signals. [Background technology]

[0002] There have been proposed techniques for making the playback of a piece of music follow the performance of a performer. For example, Patent Document 1 discloses a technique for estimating a performance position in a piece of music by analyzing an audio signal representing a tone generated by the performance of the piece of music, and controlling the automatic performance of the piece of music according to the estimation result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-207615 A Summary of the Invention [Problem to be solved by the invention]

[0004] To estimate the playing position, it is necessary to prepare reference data in advance to be used for matching with the audio signal. To generate the reference data, it is necessary to analyze the reference signal that represents the musical tones that will be generated by the previous performance, and to modify the analysis results according to the user's instructions. In light of the above situation, there is a demand for a technology that allows users to easily check and modify the analysis results of the reference signal in the process of generating the reference data. [Means for solving the problem]

[0005] In order to solve the above problems, an acoustic processing system according to one embodiment of the present disclosure includes an acoustic analysis unit that executes an analysis process to analyze an acoustic signal that records the sound of a first sound source, the analysis process including a first process of identifying a time series of a pronunciation index, which is an index of the likelihood that the acoustic signal contains an acoustic component of the first sound source, and a process of identifying a time series of pitch related to the acoustic component of the first sound source, and a display control unit that causes a display device to display the time series of the pronunciation index and the time series of the pitch on a common time axis.

[0006] An acoustic processing method according to one embodiment of the present disclosure performs an analysis process for analyzing an acoustic signal recording the sound of a first sound source, the analysis process including a first process for identifying a time series of a pronunciation index, which is an index of the likelihood that the acoustic signal contains an acoustic component of the first sound source, and a process for identifying a time series of a pitch related to the acoustic component of the first sound source, and displays the time series of the pronunciation index and the time series of the pitch on a common time axis on a display device.

[0007] A program according to one embodiment of the present disclosure causes a computer system to function as an acoustic analysis unit that executes an analysis process to analyze an acoustic signal recording the sound of a first sound source, the analysis process including a first process of identifying a time series of a pronunciation index, which is an index of the likelihood that the acoustic signal contains an acoustic component of the first sound source, and a process of identifying a time series of pitch related to the acoustic component of the first sound source, and a display control unit that displays the time series of the pronunciation index and the time series of the pitch on a common time axis on a display device. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram illustrating the configuration of a playback system according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram of music data. [Diagram 3] 2 is an explanatory diagram of the configuration and state of an operating device. FIG. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of the sound processing system. [Diagram 5]10 is an explanatory diagram of the relationship between the reproduction of a reproduction part by a performance device and a first instruction and a second instruction. FIG. [Figure 6] 10 is a flowchart illustrating a specific procedure of a playback control process. [Figure 7] 13 is an explanatory diagram relating to a state of an operating device in a second embodiment. FIG. [Figure 8] 13 is an explanatory diagram relating to a state of an operating device in a third embodiment. FIG. [Figure 9] FIG. 13 is a block diagram illustrating an example of a functional configuration of a sound processing system according to a fourth embodiment. [Figure 10] FIG. 4 is an explanatory diagram of the operation of the edit processing unit. [Figure 11] 11 is a flowchart illustrating a specific procedure of an editing process. [Figure 12] FIG. 13 is a block diagram illustrating the configuration of a playback system in a fifth embodiment. [Figure 13] FIG. 13 is a block diagram illustrating an example of a functional configuration of a sound processing system according to a fifth embodiment. [Figure 14] 4 is a block diagram illustrating a specific configuration of an acoustic analysis unit. FIG. [Figure 15] FIG. 13 is a schematic diagram of a confirmation screen. [Figure 16] 13A and 13B are schematic diagrams showing how the confirmation screen changes. [Figure 17] 13A and 13B are schematic diagrams showing how the confirmation screen changes. [Figure 18] 13A and 13B are schematic diagrams showing how the confirmation screen changes. [Figure 19] 10 is a flowchart illustrating a specific procedure of an adjustment process. [Figure 20] FIG. 13 is a schematic diagram of a confirmation screen in a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A: First embodiment 1 is a block diagram illustrating the configuration of a playback system 100 according to the first embodiment. The playback system 100 is installed in an acoustic space in which a user U is present. The user U is a performer who plays a specific part of a piece of music (hereinafter referred to as a "performance part") using an instrument 80 such as a stringed instrument.

[0010] The playback system 100 is a computer system that plays back the music piece in parallel with the performance of the performance part by the user U. Specifically, the playback system 100 plays back parts other than the performance parts (hereinafter referred to as "playback parts") among the multiple parts that make up the music piece. The performance parts are, for example, one or more parts that make up the main melody of the music piece. The playback parts are, for example, one or more parts that make up the accompaniment of the music piece. As can be understood from the above explanation, the performance of the music piece is realized by performing the performance parts by the user U and the playback of the playback parts by the playback system 100 in parallel. Note that the performance parts and the playback parts may be common parts of the music piece. Also, the performance parts may make up the accompaniment of the music piece, and the playback parts may make up the main melody of the music piece.

[0011] The reproduction system 100 includes an audio processing system 10 and a performance device 20. The audio processing system 10 and the performance device 20 are configured separately and communicate with each other via wired or wireless communication. The audio processing system 10 and the performance device 20 may be configured as an integrated unit.

[0012] The performance device 20 is a playback device that plays the playback part of a musical piece under the control of the sound processing system 10. Specifically, the performance device 20 is an automatic performance instrument that executes automatic performance of the playback part. For example, an automatic performance instrument (e.g., an automatic performance piano) of a different type from the instrument 80 played by the user U is used as the performance device 20. As can be understood from the above explanation, automatic performance is one aspect of "playback".

[0013] The performance device 20 of the first embodiment includes a drive mechanism 21 and a sound generation mechanism 22. The sound generation mechanism 22 is a mechanism that generates musical tones. Specifically, the sound generation mechanism 22 includes a string-striking mechanism for each key that generates sound from a string (sound source) in response to the displacement of each key on the keyboard, similar to a natural musical instrument keyboard. The drive mechanism 21 drives the sound generation mechanism 22 to perform automatic performance of a musical piece. The drive mechanism 21 drives the sound generation mechanism 22 in response to an instruction from the sound processing system 10, thereby realizing automatic performance of the playback part.

[0014] The sound processing system 10 is a computer system that controls the playback of a playback part by a performance device 20, and includes a control device 11, a storage device 12, a sound collection device 13, and an operation device 14. The sound processing system 10 is realized by, for example, a portable terminal device such as a smartphone or a tablet terminal, or a portable or stationary terminal device such as a personal computer. The sound processing system 10 may be realized by a single device, or may be realized by multiple devices configured separately from each other.

[0015] The control device 11 is a single or multiple processors that control each element of the sound processing system 10. Specifically, the control device 11 is configured by one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0016] The storage device 12 is one or more memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the sound processing system 10, or a recording medium that can be written to or read from via a communication network (e.g., cloud storage) may be used as the storage device 12.

[0017] The storage device 12 stores, for each piece of music, music data D that specifies the time series of multiple notes that make up the piece of music. FIG. 2 is a schematic diagram of the music data D. The music data D includes reference data Da and performance data Db. The reference data Da specifies the time series of notes in the performance part that the user U plays. Specifically, the reference data Da specifies the pitch, sound duration, and sound intensity (velocity) for each of the multiple notes in the performance part. On the other hand, the performance data Db specifies the time series of notes in the playback part that the performance device 20 plays back. Specifically, the performance data Db specifies the pitch, sound duration, and sound intensity for each of the multiple notes in the playback part.

[0018] Each of the reference data Da and the performance data Db is time-series data in a Musical Instrument Digital Interface (MIDI) format in which, for example, instruction data instructing the generation or muting of musical tones and time data specifying the time point of the action instructed by the instruction data are arranged in a time series. The instruction data instructs an action such as generation or muting by specifying, for example, a pitch and intensity. The time data specifies, for example, the interval between adjacent instruction data. The sound generation period for a note of that pitch is the period from when instruction data instructs the generation of a specific pitch to when instruction data following the instruction data instructs the muting of that pitch.

[0019] The sound collection device 13 in FIG. 1 collects musical sounds produced by a musical instrument 80 during a performance by the user U, and generates an audio signal Z representing the waveform of the musical sounds. For example, a microphone is used as the sound collection device 13. For convenience, an A / D converter that converts the audio signal Z generated by the sound collection device 13 from analog to digital is omitted from the illustration. The user U may sing the performance part of a musical piece to the sound collection device 13. When the user U sings the performance part, the sound collection device 13 generates an audio signal Z representing the waveform of the singing sound produced by the user U. As can be understood from the above explanation, "performance" includes not only performance in the narrow sense using a musical instrument 80, but also singing of a musical piece by the user U.

[0020] In the first embodiment, a configuration in which the sound collection device 13 is mounted on the sound processing system 10 is exemplified, but the sound collection device 13 that is separate from the sound processing system 10 may be connected to the sound processing system 10 by wire or wirelessly. In addition, the sound processing system 10 may receive an output signal from an electric musical instrument such as an electric string instrument as the sound signal Z. As can be understood from the above description, the sound collection device 13 may be omitted from the sound processing system 10.

[0021] The operation device 14 is an input device that accepts instructions from the user U. As illustrated in FIG. 3, the operation device 14 of the first embodiment includes a movable unit 141 that moves when operated by the user U. The movable unit 141 is an operation pedal that the user U can operate with his / her foot. For example, a pedal-type MIDI controller is used as the operation device 14. The user U can operate the operation device 14 at a desired time in parallel with playing the musical instrument 80 while playing the musical instrument 80 with both hands. Note that a touch panel that detects contact by the user U may be used as the operation device 14.

[0022] The operating device 14 transitions from one of the released state and the operating state to the other in response to an operation by the user U. The released state is a state in which the operating device 14 is not operated by the user U. Specifically, the released state is a state in which the user U does not step on the movable part 141. The released state is also expressed as a state in which the movable part 141 is at position H1. On the other hand, the operating state is a state in which the operating device 14 is operated by the user U. Specifically, the operating state is a state in which the user U steps on the movable part 141. The operating state is also expressed as a state in which the movable part 141 is at position H2 different from position H1. The released state is an example of a "first state", and the operating state is an example of a "second state".

[0023] 4 is a block diagram illustrating an example of a functional configuration of the sound processing system 10. The control device 11 executes a program stored in the storage device 12 to realize a plurality of functions (a performance analysis unit 31, a playback control unit 32, and an instruction receiving unit 33) for controlling the playback of the playback part by the performance device 20.

[0024] The performance analysis unit 31 estimates a performance position X in the music piece by analyzing the audio signal Z supplied from the sound collection device 13. The performance position X is the point in time in the music piece where the user U is currently performing. The estimation of the performance position X is repeatedly performed in parallel with the performance of the performance part by the user U and the reproduction of the reproduction part by the performance device 20. That is, the performance position X is estimated at each of a plurality of points in time on the time axis. The performance position X moves backward in the music piece as time passes.

[0025] Specifically, the performance analysis unit 31 calculates the playing position X by comparing the reference data Da of the music data D with the audio signal Z. Any known analysis technique (score alignment technique) may be adopted for the estimation of the playing position X by the performance analysis unit 31. For example, the analysis technique disclosed in Japanese Patent Application Laid-Open No. 2016-099512 is used for estimating the playing position X. The performance analysis unit 31 may also estimate the playing position X by using a statistical estimation model such as a deep neural network or a hidden Markov model.

[0026] The playback control unit 32 causes the performance device 20 to play back each note specified by the performance data Db. That is, the playback control unit 32 causes the performance device 20 to automatically play the playback part. Specifically, the playback control unit 32 moves a position Y in the music piece to be played back (hereinafter referred to as the "playback position") backward over time, and sequentially supplies instruction data corresponding to the playback position Y in the performance data Db to the performance device 20. That is, the playback control unit 32 functions as a sequencer that sequentially supplies each instruction data included in the performance data Db to the performance device 20. The process by which the playback control unit 32 causes the performance device 20 to play back the playback part is executed in parallel with the performance of the performance part by the user U.

[0027] The playback control unit 32 makes the playing of the playback part by the playing device 20 follow the performance of the music piece by the user U according to the result of the estimation of the playing position X by the performance analysis unit 31. That is, the automatic performance of the playback part by the playing device 20 proceeds at the same tempo as the performance of the performance part by the user U. For example, when the progression of the playing position X (i.e., the performance speed by the user U) is fast, the playback control unit 32 increases the speed of progression of the playback position Y (the playback speed by the playing device 20), and when the progression of the playing position X is slow, the playback control unit 32 decreases the speed of progression of the playback position Y. That is, the automatic performance of the playback part is executed at the same performance speed as the performance by the user U so as to be synchronized with the progression of the playing position X. Therefore, the user U can play the performance part with the feeling that the playing device 20 is playing the playback part in accordance with his / her performance.

[0028] As described above, in the first embodiment, the playback of multiple notes in the playback part follows the performance of the instrument 80 by the user U, so that the intentions (e.g., performance expression) or preferences of the user U can be appropriately reflected in the playback of the playback part.

[0029] The instruction receiving unit 33 receives a first instruction Q1 and a second instruction Q2 from the user U. The first instruction Q1 and the second instruction Q2 are generated by the user U's operation of the operation device 14. The first instruction Q1 is an instruction to temporarily stop the playback of the playback part by the performance device 20. The second instruction Q2 is an instruction to resume the playback of the playback part stopped by the first instruction Q1.

[0030] Specifically, the instruction receiving unit 33 receives an operation by the user U to transition the operating device 14 from a released state to an operating state as the first instruction Q1. That is, the user U gives the first instruction Q1 to the sound processing system 10 by stepping on the movable part 141 of the operating device 14. For example, the instruction receiving unit 33 specifies the time when the movable part 141 starts to move from position H1 (released state) to position H2 (operated state) as the time of the first instruction Q1. Note that a configuration in which the instruction receiving unit 33 specifies the time when the movable part 141 reaches a position midway between position H1 and position H2 as the time of the first instruction Q1, or a configuration in which the instruction receiving unit 33 specifies the time when the movable part 141 reaches position H2 as the time of the first instruction Q1 is also assumed.

[0031] The instruction receiving unit 33 also receives, as the second instruction Q2, an operation by the user U to transition the operating device 14 from an operating state to a released state. That is, the user U gives the second instruction Q2 to the sound processing system 10 by releasing the movable part 141 of the operating device 14 from a state where the movable part 141 is depressed. For example, the instruction receiving unit 33 specifies the time when the movable part 141 starts to move from the position H2 (operated state) toward the position H1 (released state) as the time of the second instruction Q2. Note that a configuration in which the instruction receiving unit 33 specifies the time when the movable part 141 reaches a position midway between the position H2 and the position H1 as the time of the second instruction Q2, or a configuration in which the instruction receiving unit 33 specifies the time when the movable part 141 reaches the position H1 as the time of the second instruction Q2, is also assumed.

[0032] The user U can give the first instruction Q1 and the second instruction Q2 at any time during the performance of the performance part. Therefore, the interval between the first instruction Q1 and the second instruction Q2 is variable in length according to the intention of the user U. For example, the user U gives the first instruction Q1 before the start of a rest period in the music piece, and gives the second instruction Q2 when the rest period of the user U's desired length has elapsed.

[0033] 5 is an explanatory diagram of the relationship between the reproduction of the reproduction part by the performance device 20 and the first instruction Q1 and the second instruction Q2. The sounding period of each note specified by the performance data Db and the sounding period of each note actually reproduced by the performance device 20 are shown in FIG.

[0034] The note N1 in FIG. 5 is a note corresponding to the first instruction Q1 among the multiple notes specified by the performance data Db. Specifically, the note N1 is a note being reproduced by the performance device 20 at the time of the first instruction Q1 among the multiple notes in the reproduction part. After the first instruction Q1 is generated, the reproduction control unit 32 causes the performance device 20 to continue reproduction of the note N1 until the end point of the sounding period specified for the note N1 by the performance data Db. For example, the reproduction control unit 32 supplies the performance device 20 with instruction data for instructing the mute of the note N1 at the end point of the sounding period of the note N1. As can be understood from the above explanation, the reproduction of the note N1 is not immediately stopped at the time of the first instruction Q1, but is continued until the end point specified by the performance data Db even after the generation of the first instruction Q1. Note that the note N1 is an example of a "first note".

[0035] The note N2 in FIG. 5 is a note immediately after the note N1 among the multiple notes specified by the performance data Db. After the playback of the note N1 is stopped, the playback control unit 32 causes the performance device 20 to start playing the note N2 in response to the second instruction Q2 from the user U. That is, regardless of the position of the start point of the sounding period specified for the note N2 by the performance data Db and the time length of the interval between the notes N1 and N2 specified by the performance data Db, the playback of the note N2 is started on the condition that the second instruction Q2 is generated. Specifically, when the instruction receiving unit 33 receives the second instruction Q2, the playback control unit 32 supplies the instruction data of the note N2 in the performance data Db to the performance device 20. Therefore, the playback of the note N2 is started immediately after the second instruction Q2. The note N2 is an example of the "second note".

[0036] 6 is a flow chart showing an example of a specific procedure of the operation (hereinafter referred to as "playback control process") Sa in which the control device 11 controls the performance device 20. In response to an instruction from the user U, the playback control process Sa is started.

[0037] When the playback control process Sa is started, the control device 11 judges whether the standby data W is in a valid state (Sa1). The standby data W is data (e.g., a flag) that indicates that the playback of the playback part is temporarily stopped by the first instruction Q1, and is stored in the storage device 12. Specifically, the standby data W is set to a valid state (e.g., W=1) when the first instruction Q1 is generated, and is set to an invalid state (e.g., W=0) when the second instruction Q2 is generated. The standby data W can also be said as data that indicates a state of waiting for the playback of the playback part to be resumed.

[0038] If the standby data W is not valid (Sa1: NO), the control device 11 (performance analysis unit 31) estimates the performance position X by analyzing the audio signal Z supplied from the sound collection device 13 (Sa2). The control device 11 (playback control unit 32) causes the performance device 20 to proceed with the playback of the playback part according to the result of the estimation of the performance position X (Sa3). That is, the control device 11 controls the playback of the playback part by the performance device 20 so as to follow the performance of the performance part by the user U.

[0039] The control device 11 (instruction receiving unit 33) judges whether or not the first instruction Q1 has been received from the user U (Sa4). When the first instruction Q1 has been received (Sa4: YES), the control device 11 (playback control unit 32) causes the performance device 20 to continue playing the note N1 that is being played at the time of receiving the first instruction Q1 until the end point of the sounding period specified by the performance data Db (Sa5). Specifically, the control device 11 advances the playback position Y at the same speed (tempo) as when the first instruction Q1 was generated, and when the playback position Y reaches the end point of the sounding period of the note N1, supplies the performance device 20 with instruction data representing the muting of the note N1. After executing the above process, the control device 11 changes the standby data W from an invalid state to a valid state (W=1) (Sa6). Note that the standby data W may be updated (Sa6) before executing step Sa5.

[0040] When the standby data W is set to a valid state, the result of the judgment in step Sa1 becomes positive. When the standby data W is in a valid state (Sa1: YES), the estimation of the playing position X (Sa2), the playback control of the playing part (Sa3), and the processing related to the note N1 (Sa4-Sa6) are not executed. That is, the playback control of the playing part linked to the playing position X is stopped in response to the first instruction Q1 from the user U. Also, when the first instruction Q1 is not accepted (Sa4: NO), the processing related to the note N1 (Sa5, Sa6) is not executed.

[0041] The control device 11 (instruction receiving unit 33) judges whether or not the second instruction Q2 has been received from the user U (Sa7). When the second instruction Q2 has been received (Sa7: YES), the control device 11 (playback control unit 32) causes the performance device 20 to play the note N2 immediately following the note N1 (Sa8). Specifically, the control device 11 updates the playback position Y to the start point of the note N2. That is, the playback of the playback part stopped by the first instruction Q1 is resumed by the second instruction Q2. The control device 11 changes the standby data W from a valid state to an invalid state (W=0) (Sa9). As described above, when the standby data W is set to an invalid state, the result of the judgment in step Sa1 becomes negative. Therefore, the second instruction Q2 is used as a trigger to resume the estimation of the performance position X (Sa2) and the playback control of the playback part (Sa3). Note that the update of the standby data W (Sa8) may be executed before the execution of step Sa8.

[0042] The control device 11 judges whether or not to end the playback of the playback part by the performance device 20 (Sa10). For example, when playback of the playback part is completed to the end point, or when an end command is received from the user U, the control device 11 judges to end the playback of the playback part. If playback of the playback part is not to be ended (Sa10: NO), the control device 11 shifts the process to step Sa1 and repeats the above-exemplified processes (Sa1-Sa9). On the other hand, when the control device 11 judges that playback of the playback part is to be ended (Sa10: YES), the playback control process Sa is ended.

[0043] As described above, in the first embodiment, the note N1 corresponding to the first instruction Q1 is played, and after the playback of the note N1 is stopped, the playback of the note N2 immediately following the note N1 is started in response to the second instruction Q2 by the user U. Therefore, the interval between the playback of the note N1 and the playback of the note N2 (for example, the duration of the rest period in the music piece) can be changed according to each point in time of the first instruction Q1 and the second instruction Q2.

[0044] In the first embodiment, the note N1 being played at the time of the first instruction Q1 is continued to the end point of the note N1 specified by the performance data Db even after the first instruction Q1 is generated. Therefore, compared to a configuration in which the playback of the note N1 is stopped at the time of the first instruction Q1, it is possible to appropriately continue the playback of the note N1 according to the content of the performance data Db.

[0045] In the first embodiment, the user U can change the interval between the notes N1 and N2 to an appropriate time length according to the user U's intention or preference by operating the operation device 14. In particular, in the first embodiment, the first instruction Q1 is generated by transitioning the operation device 14 from a released state to an operating state, and after the operating state is maintained, the second instruction Q2 is generated by transitioning the operation device 14 from the operating state to the released state at a desired time point after the generation of the first instruction Q1. That is, the first instruction Q1 and the second instruction Q2 are generated by a series of operations that transition the operation state from the released state to the operating state and then transition it back to the released state. Therefore, the operation of the user U on the operation device 14 is simplified compared to a configuration in which an operation of transitioning the operation device 14 from the released state to the operating state is necessary for each of the first instruction Q1 and the second instruction Q2.

[0046] B: Second embodiment A second embodiment will be described. In each of the following exemplary embodiments, elements having the same functions as those in the first embodiment will be denoted by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions thereof will be omitted as appropriate.

[0047] In the first embodiment, when the first instruction Q1 is generated, the reproduction position Y is advanced at the same speed as the time point of the first instruction Q1, and when the reproduction position Y reaches the end point of the musical note N1, the reproduction of the musical note N1 is stopped. The reproduction control unit 32 of the second embodiment variably controls the advancing speed of the reproduction position Y (i.e., the reproduction speed of the reproduction part) according to the operation speed V1 of the movable part 141 after the generation of the first instruction Q1. The operation speed V1 is the speed at which the movable part 141 moves from the position H1 corresponding to the released state toward the position H2 corresponding to the operated state. For example, the average value of a plurality of speeds calculated during the period from the position H1 to the position H2 by the movable part 141 is the operation speed V1.

[0048] FIG. 7 is an explanatory diagram regarding the state of the operation device 14 in the second embodiment. As illustrated in FIG. 7, the instruction reception unit 33 receives the first instruction Q1 at the time point when the movable part 141 starts to move from the position H1 toward the position H2. The reproduction control unit 32 controls the advancing speed of the reproduction position Y according to the operation speed V1 of the movable part 141 after the generation of the first instruction Q1.

[0049] Specifically, the reproduction control unit 32 increases the advancing speed of the reproduction position Y as the operation speed V1 is higher. For example, as illustrated in FIG. 7, the advancing speed of the reproduction position Y when the operation speed V1 is the speed V1_H is higher than the advancing speed of the reproduction position Y when the operation speed V1 is the speed V1_L (V1_L < V1_H). Therefore, the longer the operation speed V1 is, the shorter the continuation length of the musical note N1 is. For example, the continuation length of the musical note N1 when the operation speed V1 is the speed V1_H is shorter than the continuation length of the musical note N1 when the operation speed V1 is the speed V1_L.

[0050] The second embodiment also achieves the same effect as the first embodiment. In the second embodiment, the duration of the note N1 is controlled according to the operation speed V1, so that the user U can adjust the duration of the note N1. In addition, in the second embodiment, the operation device 14 for giving the first instruction Q1 and the second instruction Q2 is also used for adjusting the duration of the note N1. Therefore, compared with a configuration in which the user U operates separate devices for giving the first instruction Q1 and the second instruction Q2 and for adjusting the duration of the note N1, the operation by the user U is simplified.

[0051] C: Third embodiment In the first embodiment, the playback of the note N2 is started immediately after the second instruction Q2. In the second embodiment, the time from the second instruction Q2 to the start of playback of the note N2 (hereinafter referred to as the "delay time") is variably controlled according to the operation speed V2. The operation speed V2 is the speed at which the movable part 141 moves from the position H2 corresponding to the operated state to the position H1 corresponding to the released state. For example, the average value of multiple velocities calculated within the period until the movable part 141 moves from the position H2 to the position H1 is the operation speed V2.

[0052] Fig. 8 is an explanatory diagram regarding the state of the operation device 14 in the third embodiment. As illustrated in Fig. 8, the instruction receiving unit 33 receives a second instruction Q2 at the time when the movable part 141 starts to move from position H2 to position H1. The playback control unit 32 variably controls the delay time according to the operation speed V2.

[0053] Specifically, the playback control unit 32 shortens the delay time as the operation speed V2 becomes faster. For example, as illustrated in Fig. 8, the delay time when the operation speed V2 is a speed V2_L is longer than the delay time when the operation speed V2 is a speed V2_H (V2_H>V2_L). Therefore, the slower the operation speed V2 is, the later the time point on the time axis at which the playback of the note N2 starts.

[0054] The third embodiment also achieves the same effect as the first embodiment. In the third embodiment, the time when the reproduction of the note N2 starts is controlled according to the operation speed V2, so that the user U can adjust the start point of the first note N2 after the reproduction part is resumed. In addition, in the third embodiment, the operation device 14 for giving the first instruction Q1 and the second instruction Q2 is also used for adjusting the start point of the note N2. Therefore, compared with a configuration in which the user U operates separate devices for giving the first instruction Q1 and the second instruction Q2 and adjusting the start point of the note N2, there is also an advantage that the operation by the user U is simplified. The configuration of the second embodiment may be applied to the third embodiment.

[0055] D: Fourth embodiment FIG. 9 is a block diagram illustrating a functional configuration of the sound processing system 10 in the fourth embodiment. The control device 11 in the fourth embodiment functions as an edit processing unit 34 in addition to the same elements as in the first embodiment (performance analysis unit 31, playback control unit 32, and instruction receiving unit 33). The edit processing unit 34 edits the performance data Db stored in the storage device 12 in response to an instruction from a user U. The operations of the elements other than the edit processing unit 34 are the same as in the first embodiment. Therefore, the fourth embodiment also achieves the same effects as the first embodiment. Note that the configuration of the second or third embodiment may be applied to the fourth embodiment.

[0056] Fig. 10 is an explanatory diagram of the operation of the edit processing unit 34. Fig. 10 illustrates notes N1 and N2 designated by the performance data Db of the playback part. As in each of the above-mentioned forms, the first instruction Q1 and the second instruction Q2 are generated at any time of the user U. Therefore, a time difference L occurs between the start point of the note N2 designated by the performance data Db and the time of the second instruction Q2. The edit processing unit 34 edits the performance data Db so as to reduce the time difference L.

[0057] 11 is a flow chart showing a specific procedure of a process (hereinafter referred to as "edit process") Sb in which the edit processing unit 34 edits the performance data Db. For example, the edit process Sb is executed every time the performance device 20 repeats the reproduction of the reproduction part (the above-mentioned reproduction control process Sa) a predetermined number of times. The edit process Sb may be started in response to an instruction from the user U.

[0058] When the edit process Sb is started, the edit processing unit 34 calculates the degree of dispersion Δ of the time differences L in a predetermined number of past playback control processes Sa (Sb1). The degree of dispersion Δ is a statistical quantity that represents the degree of dispersion regarding a plurality of time differences L. For example, the variance, standard deviation, or distribution range of the plurality of time differences L is used as the degree of dispersion Δ.

[0059] The edit processing unit 34 judges whether the dispersion Δ exceeds the threshold Δth (Sb2). If the dispersion Δ exceeds the threshold Δth, it is presumed that the user U is practicing playing the music while consciously varying the waiting time until the playback of the note N2 is resumed. Therefore, it is not appropriate to edit the performance data Db according to the multiple time differences L. On the other hand, if the dispersion Δ is below the threshold Δth, it is presumed that the multiple time differences L are values ​​that are in line with the intention or preference of the user U (i.e., preferred values ​​unique to the user U).

[0060] Taking the above tendency into consideration, when the dispersion degree Δ is below the threshold value Δth (Sb2: NO), the edit processing unit 34 edits (Sb3-Sb4) the performance data Db according to the multiple time differences L. On the other hand, when the dispersion degree Δ is above the threshold value Δth (Sb2: YES), the edit processing unit 34 ends the edit process Sb without executing the editing (Sb3, Sb4) of the performance data Db.

[0061] In editing the performance data Db, the edit processing unit 34 calculates an average time difference La by averaging a plurality of time differences L (Sb3). Then, the edit processing unit 34 changes the start point of the note N2 specified by the performance data Db by the average time difference La (Sb4). For example, when the average time difference La is a negative number, the edit processing unit 34 moves the start point of the note N2 specified by the performance data Db forward by the time equivalent to the average time difference La. Also, when the average time difference La is a positive number, the edit processing unit 34 moves the start point of the note N2 specified by the performance data Db backward by the time equivalent to the average time difference La. That is, when the user U tends to secure a sufficient waiting time immediately before the note N2, the start point of the note N2 specified by the performance data Db is changed backward, and when the waiting time tends to be short, the start point of the note N2 specified by the performance data Db is changed forward.

[0062] As can be understood from the above description, in the fourth embodiment, the performance data Db is edited according to the time difference L in the performance of the performance parts by the user U. Therefore, it is possible to reflect the tendencies unique to each user U in the performance data Db.

[0063] E: Fifth embodiment 12 is a block diagram illustrating a configuration of a reproduction system 100 according to a fifth embodiment. The sound processing system 10 of the fifth embodiment includes a display device 15 in addition to the same elements as those of the sound processing system 10 of the first embodiment (a control device 11, a storage device 12, a sound collection device 13, and an operation device 14). The display device 15 displays an image instructed by the control device 11. The display device 15 is, for example, a liquid crystal display panel or an organic EL display panel.

[0064] The user U1 plays the musical instrument 80 in the same manner as in the first embodiment. The musical instrument 80 is a natural musical instrument such as a stringed instrument that produces sound when played by the user U1. On the other hand, the performance device 20 in the fifth embodiment is an automatic musical instrument that functions not only as a playback device that executes automatic performance of the playback part of a musical piece, but also as an electronic musical instrument that can be manually played by the user U2. Specifically, the performance device 20 includes a drive mechanism 21 and a sound generation mechanism 22, similar to the performance devices 20 in the above-mentioned embodiments.

[0065] The manual performance by the user U2 is, for example, a performance by the user U2's physical movements such as pressing keys on a keyboard. The sound generation mechanism 22 operates in conjunction with the performance by the user U2, so that musical tones are generated from the performance device 20. In addition, in parallel with the performance by the user U2, the performance device 20 sequentially outputs instruction data d representing instructions for the performance to the sound processing system 10. The instruction data d specifies, for example, pitch and sound intensity, and specifies an operation such as sound generation or muting.

[0066] 13 is a block diagram illustrating a functional configuration of an acoustic processing system 10 in the fifth embodiment. The control device 11 in the fifth embodiment executes a program stored in the storage device 12 to function as a preparation processing unit 35 and a display control unit 36 ​​in addition to the same elements as those in the first embodiment (a performance analysis unit 31, a playback control unit 32, and an instruction receiving unit 33).

[0067] The preparation processing unit 35 generates music data D (reference data Da and performance data Db) to be used in the playback control process Sa. Specifically, the preparation processing unit 35 generates music data D according to the performance of the musical instrument 80 by the user U1 and the performance of the performance device 20 by the user U2. The display control unit 36 ​​causes the display device 15 to display various images.

[0068] The preparation processing unit 35 includes a first recording unit 41, a second recording unit 42, and an acoustic analysis unit 43. The acoustic analysis unit 43 generates reference data Da used in the playback control process Sa. Specifically, the acoustic analysis unit 43 generates the reference data Da by executing an adjustment process Sc (FIG. 15) before the start of the playback control process Sa. The reference data Da generated by the acoustic analysis unit 43 is stored in the storage device 12.

[0069] During the period before the execution of the adjustment process Sc (hereinafter referred to as the "preparation period"), the users U1 and U2 play a piece of music together. Specifically, during the preparation period, the user U1 plays the performance part of the piece of music on the musical instrument 80, and the user U2 plays the playback part of the piece of music on the performance device 20. The adjustment process Sc is a process for generating reference data Da using the results of the performance of the piece of music by the users U1 and U2 during the preparation period. Note that the user U1 may sing the performance part of the piece of music to the sound collection device 13.

[0070] The first recording unit 41 acquires the sound signal Z generated by the sound collection device 13 during the preparation period. For the sake of convenience, the sound signal Z acquired by the first recording unit 41 during the preparation period will be referred to as a “reference signal Zr” below. The first recording unit 41 stores the reference signal Zr in the storage device 12.

[0071] During the preparation period, in addition to the musical tones produced by the musical instrument 80 performed by the user U1, the musical tones produced by the performance device 20 performed by the user U2 reach the sound collection device 13. Therefore, the reference signal Zr is an audio signal including the acoustic components of the musical instrument 80 and the performance device 20. Note that the musical instrument 80 is an example of a "first sound source," and the performance device 20 is an example of a "second sound source." When the user U1 sings the performance part, the user U1 corresponds to the "first sound source."

[0072] The second recording unit 42 acquires performance data Db representing the performance of the performance device 20 during the preparation period. Specifically, the second recording unit 42 generates performance data Db in MIDI format in which instruction data d sequentially supplied from the performance device 20 according to the performance of the user U2 and time data specifying the interval between adjacent instruction data d are arranged in time series. The second recording unit 42 saves the performance data Db in the storage device 12. The performance data Db stored in the storage device 12 is used for the playback control process Sa, as exemplified in each of the above-mentioned embodiments. The performance data Db acquired by the second recording unit 42 may be edited by the editing processing unit 34 exemplified in the fourth embodiment.

[0073] As can be understood from the above description, during the preparation period, the reference signal Zr and the performance data Db are stored in the storage device 12. The acoustic analysis unit 43 generates the reference data Da by an adjustment process Sc using the reference signal Zr acquired by the first recording unit 41 and the performance data Db acquired by the second recording unit 42.

[0074] Fig. 14 is a block diagram illustrating a specific configuration of the acoustic analysis unit 43. As illustrated in Fig. 14, the acoustic analysis unit 43 includes an index calculation unit 51, a period estimation unit 52, a pitch estimation unit 53, and an information generation unit 54. The operation device 14 of the fifth embodiment receives an instruction regarding the numerical values ​​of each of a plurality of variables (α, β, γ) applied to the adjustment process Sc from a user U (U1 or U2). That is, the user U can set or change the numerical value of each variable by operating the operation device 14.

[0075] [Indicator calculation unit 51] The index calculation unit 51 calculates a pronunciation index C(t). The symbol t means one time point on the time axis. That is, the index calculation unit 51 specifies a time series of the pronunciation index C(t) corresponding to different time points t on the time axis. The pronunciation index C(t) is an index of the likelihood (likelihood or probability) that the reference signal Zr includes an acoustic component of the musical instrument 80. That is, the higher the likelihood that the acoustic component of the musical instrument 80 is included in the reference signal Zr at the time point t on the time axis, the larger the pronunciation index C(t) is set to. The index calculation unit 51 of the first embodiment includes a first analysis unit 511, a second analysis unit 512, a first calculation unit 513, and a second calculation unit 514.

[0076] The first analysis unit 511 calculates a first index B1(t,n) by analyzing the reference signal Zr. The symbol n means any one of N pitches P1 to PN (n=1 to N). Specifically, the first analysis unit 511 calculates N first indexes B1(t,1) to B1(t,N) corresponding to different pitches Pn for each time point t on the time axis. That is, the first analysis unit 511 calculates a time series of the first index B1(t,n).

[0077] The first index B1(t,n) corresponding to the pitch Pn is an index of the likelihood that the acoustic component of the pitch Pn in the performance device 20 or the musical instrument 80 is included in the reference signal Zr, and is set to a value within the range of 0 to 1. The greater the intensity of the acoustic component of the pitch Pn in one or both of the performance device 20 and the musical instrument 80, the greater the value of the first index B1(t,n). As can be understood from the above explanation, the first index B1(t,n) can also be said to be an index related to the intensity of the acoustic component of each pitch Pn in the reference signal Zr. The first analysis unit 511 may arbitrarily employ any known acoustic analysis technique (particularly a pitch estimation technique) to calculate the first index B1(t,n).

[0078] The second analysis unit 512 calculates the second indicator B2(t,n) by analyzing the performance data Db. Specifically, the second analysis unit 512 calculates N second indicators B2(t,1) to B2(t,N) corresponding to different pitches Pn for each time point t on the time axis. That is, the second analysis unit 512 calculates the time series of the second indicators B2(t,n).

[0079] The second indicator B2(t,n) corresponding to the pitch Pn is an indicator according to the sound intensity that the performance data Db specifies for the note of pitch Pn at time t, and is set to a value within the range of 0 to 1. The higher the sound intensity that the performance data Db specifies for the note of pitch Pn, the larger the value of the second indicator B2(t,n). If there is no note of pitch Pn at time t, the second indicator B2(t,n) is set to 0.

[0080] The second index B2(t, n) is calculated using a variable α set in response to an instruction from the user U. For example, the second analysis unit 512 calculates the second index B2(t, n) by the calculation of the following formula (1).

number

[0081] The symbol v(t,n) in formula (1) is a numerical value corresponding to the sound intensity that the performance data Db specifies for the pitch Pn at the point in time t on the time axis. If the point in time t on the time axis is within the sounding period of the note with the pitch Pn, the intensity v(t,n) is set to the sounding intensity that the performance data Db specifies for that note. On the other hand, if the point in time t on the time axis is outside the sounding period of the pitch Pn, the intensity v(t,n) is set to 0. The symbol c in formula (1) is a coefficient, which is set to a predetermined positive number.

[0082] As can be seen from formula (1), when the variable α is small, the second index B2(t,n) is set to a small value even if the intensity v(t,n) is a large value. On the other hand, when the variable α is large, the second index B2(t,n) is set to a large value even if the intensity v(t,n) is a small value. In other words, there is a tendency for the second index B2(t,n) to become a smaller value as the variable α becomes smaller.

[0083] 14 calculates the pronunciation index E(t,n) by subtracting the second index B2(t,n) from the first index B1(t,n). The pronunciation index E(t,n) is an index of the likelihood (likelihood or probability) that the acoustic component of the pitch Pn produced by the musical instrument 80 is included in the reference signal Zr at the time point t. Specifically, the first calculation unit 513 calculates the pronunciation index E(t,n) by the calculation of the following formula (2).

number

[0084] In formula (2), max{a,b} denotes an operation to select the larger of the numerical values ​​a and b. As can be seen from formula (2), the pronunciation index E(t,n) is a numerical value in the range of 0 to 1. The pronunciation index E(t,n) is set to a larger numerical value as the strength of the acoustic component of the pitch Pn sounded by the musical instrument 80 increases. In other words, the pronunciation index E(t,n) is also an index related to the strength of the acoustic component (pitch Pn) of the musical instrument 80 in the reference signal Zr.

[0085] As described above, the first index B1(t,n) is influenced by the acoustic components of both the performance device 20 and the musical instrument 80. On the other hand, the second index B2(t,n) is influenced by only the acoustic components of the performance device 20. Therefore, in the formula (2), the calculation of subtracting the second index B2(t,n) from the first index B1(t,n) corresponds to a process of suppressing the influence of the acoustic components of the performance device 20 from the first index B1(t,n). That is, the sound generation index E(t,n) corresponds to an index related to the strength of the acoustic components (pitch Pn) of the musical instrument 80 among the acoustic components of the reference signal Zr. As described above, there is a tendency that the larger the variable α, the larger the value of the second index B2(t,n) is set. Therefore, the variable α is a variable for controlling the degree of suppression of the influence of the acoustic components of the performance device 20 from the first index B1(t,n). That is, the larger the variable α (the larger the second index B2(t,n)), the more the influence of the acoustic components of the performance device 20 in the sound generation index E(t,n) is suppressed.

[0086] The second calculation unit 514 calculates the pronunciation index C(t) from the pronunciation index E(t,n) calculated by the first calculation unit 513. Specifically, the second calculation unit 514 calculates the pronunciation index C(t) by the following formula (3).

number

[0087] [Period Estimation Section 52] The period estimation unit 52 in FIG. 14 estimates a period during which the acoustic component of the musical instrument 80 exists on the time axis (hereinafter referred to as the "performance period"). Specifically, the period estimation unit 52 calculates a voice presence index G(t) using the sound presence index C(t) calculated by the index calculation unit 51. The voice presence index G(t) is an index indicating whether or not the acoustic component of the musical instrument 80 exists (voice / silence) at time point t. The voice presence index G(t) at time point t when the acoustic component of the musical instrument 80 exists is set to a numerical value g1 (e.g., g1=1), and the voice presence index G(t) at time point t when the acoustic component of the musical instrument 80 does not exist is set to a numerical value g0 (e.g., g0=0). A period consisting of one or more time points t where the voice presence index G(t) is the numerical value g1 corresponds to the performance period. The performance period is an example of the "sound presence period".

[0088] A first HMM (Hidden Markov Model) is used to estimate the performance period. The first HMM is a state transition model consisting of a voiced state corresponding to voiced (value g1) and a silent state corresponding to silent (value g0). Specifically, the period estimation unit 52 calculates a sequence of the most likely state generated by the first HMM as a voiced index G(t) by Viterbi search.

[0089] The probability that a voice-active state occurs in the first HMM (hereinafter referred to as "voice-active probability") Λ is expressed by the following mathematical formula (4). The symbol σ in mathematical formula (4) is a sigmoid function. The probability that a silent state occurs is (1-Λ). Moreover, the probability that a voice-active state or a silent state is maintained between two adjacent time points t on the time axis is set to a predetermined constant (e.g., 0.9).

number

[0090] As can be seen from formula (4), the calculation of the sound presence probability Λ applies a variable β set in response to an instruction from the user U. Specifically, the larger the variable β, the smaller the sound presence probability Λ is set to. Therefore, the larger the variable β, the more likely the sound presence index G(t) is to be set to the value g0, and as a result, the performance period tends to be shorter. On the other hand, the smaller the variable β, the more likely the sound presence probability Λ is set to a larger value. Therefore, the more likely the sound presence index G(t) is to be set to the value g1, and as a result, the performance period tends to be longer.

[0091] Also, as mentioned above, the larger the variable α, the smaller the value of the onset index C(t) during the period in which the acoustic components of the performance device 20 are dominant. As can be understood from the formula (4), the smaller the onset index C(t), the smaller the value of the voice presence probability Λ is set. Therefore, the larger the variable α, the more likely the voice presence index G(t) is to be set to the value g0, and as a result, the performance period tends to be shorter. As can be understood from the above explanation, the variable α affects not only the onset index C(t) but also the performance period. In other words, the variable α affects both the onset index C(t) and the voice presence index G(t), while the variable β affects only the voice presence index G(t).

[0092] [Pitch estimation section 53] The pitch estimation unit 53 identifies the pitch K(t) of the acoustic components of the musical instrument 80. That is, the pitch estimation unit 53 identifies a time series of pitches K(t) corresponding to different points in time t on the time axis. The pitch K(t) is set to one of N pitches P1 to PN.

[0093] The second HMM is used to estimate the pitch K(t). The second HMM is a state transition model consisting of N states corresponding to different pitches Pn. The probability density function ρ(x|μn,κn) of the observation probability x of pitch Pn is a Von Mises-Fisher distribution expressed by the following formula (5).

number

[0094] In the second HMM, the transition probability λ(n1, n2) from pitch Pn1 to pitch Pn2 is expressed by the following formula (6) (n1=1 to N, n2=1 to N, n1≠n2). For all combinations of two pitches Pn (Pn1, Pn2) selected from the N pitches P1 to PN, the transition probability λ(n1, n2) is set by formula (6).

number

[0095] As can be seen from formula (6), the estimation of pitch K(t) applies the variable γ set in response to an instruction from user U. Specifically, the smaller the variable γ, the closer the transition probability λ(n1,n2) is to the unit matrix I, making it more difficult for the pitch Pn1 to transition to Pn2 to occur. On the other hand, the larger the variable γ, the greater the influence of the transition probability τ(n1,n2) on the transition probability λ(n1,n2), making it easier for the pitch Pn1 to transition to Pn2 to occur.

[0096] [Information generation unit 54] The information generating unit 54 identifies M sounding points T1 to TM on the time axis and the pitch Fm (F1 to FM) at each sounding point Tm (m=1 to M). The number M of sounding points Tm in a piece of music is a variable value. Specifically, the information generating unit 54 identifies as the sounding point Tm a time point t at which the pitch K(t) changes during a performance period in which the sound presence index G(t) is the numerical value g1. The information generating unit 54 also identifies the pitch K(Tm) at each sounding point Tm as the pitch Fm.

[0097] As described above, the smaller the variable γ, the less likely a transition in pitch Pn occurs, so the fewer the number M of sounding points Tm and pitches Fm. On the other hand, the larger the variable γ, the more likely a transition in pitch Pn occurs, so the more the number M of sounding points Tm and pitches Fm. In other words, the variable γ is a parameter for controlling the number M of sounding points Tm and pitches Fm.

[0098] In addition, the information generation unit 54 stores in the storage device 12 reference data Da including the onset index E(t, n) calculated by the index calculation unit 51 (second calculation unit 514), each onset point Tm on the time axis, and the utterance index G(t) calculated by the period estimation unit 52.

[0099] The display control unit 36 ​​in Fig. 13 causes the display device 15 to display the results of the analysis by the acoustic analysis unit 43 described above. Specifically, the display control unit 36 ​​causes the display device 15 to display a confirmation screen 60 exemplified in Fig. 15. The confirmation screen 60 is an image for allowing the user U to confirm the results of the analysis by the acoustic analysis unit 43.

[0100] The confirmation screen 60 includes a first area 61 and a second area 62. A common time axis At is set in the first area 61 and the second area 62. The time axis At is an axis extending in the horizontal direction. The time axis At may be displayed as an image visible to the user U, or may not be displayed on the confirmation screen 60. The section of the song displayed on the confirmation screen 60 is changed in response to an instruction from the user U to the operation device 14 (e.g., an instruction to enlarge / reduce).

[0101] In the second area 62, a performance period 621 and a period other than the performance period 621 (hereinafter referred to as a "non-performance period") 622 on the time axis At are displayed in different display modes. For example, the performance period 621 and the non-performance period 622 are displayed in different hues. The performance period 621 is a range on the time axis At where the sound presence indicator G(t) is set to a numerical value g1. On the other hand, the non-performance period 622 is a range on the time axis At where the sound presence indicator G(t) is set to a numerical value g0. As can be understood from the above explanation, the performance period 621 estimated by the period estimation unit 52 is displayed on the confirmation screen 60.

[0102] A transition image 64 is displayed in the first area 61. The transition image 64 is an image that displays the time series of the pronunciation index C(t) calculated by the index calculation unit 51 on the time axis At. Specifically, the part of the transition image 64 that corresponds to the time point t on the time axis At is displayed in a display mode according to the pronunciation index C(t). The "display mode" means the property of an image that an observer can visually distinguish. For example, the concept of the "display mode" includes the three attributes of color, that is, hue (tone), saturation, and brightness (gradation), as well as patterns or shapes. For example, the gradation shade of the part of the transition image 64 that corresponds to the time point t is controlled according to the pronunciation index C(t). Specifically, the part of the transition image 64 that corresponds to the time point t where the pronunciation index C(t) is large is displayed in a dark gradation, and the part of the transition image 64 that corresponds to the time point t where the pronunciation index C(t) is small is displayed in a light gradation.

[0103] In the second area 62, a staff 65, a plurality of instruction images 67, and a plurality of note images 68 are displayed. The staff 65 is composed of five straight lines parallel to the time axis At. Each straight line constituting the staff 65 represents a different pitch. That is, in the second area 62, a pitch axis Ap representing the pitch is set. The pitch axis Ap is an axis extending in the vertical direction perpendicular to the time axis At. Note that the pitch axis Ap may be displayed as an image visible to the user U, or may not be displayed on the confirmation screen 60.

[0104] Each instruction image 67 is an image representing one sound generation point Tm generated by the information generating unit 54. That is, the time series of the sound generation points Tm is expressed by the multiple instruction images 67. Specifically, the instruction image 67 corresponding to the sound generation point Tm is a vertical line arranged at a position on the time axis At corresponding to the sound generation point Tm. Therefore, the multiple instruction images 67 corresponding to different sound generation points Tm are arranged on the time axis At.

[0105] Each note image 68 is an image representing one pitch Fm generated by the information generating unit 54. For example, an image representing the head of a note is exemplified as the note image 68. A time series of the pitch Fm is expressed by a plurality of note images 68. Since the pitch Fm is set for each sounding point Tm, the note image 68 is arranged for each sounding point Tm (i.e., for each instruction image 67). Specifically, the note image 68 representing the pitch Fm at the sounding point Tm is arranged on the line of the instruction image 67 representing the sounding point Tm in the direction of the time axis At. Moreover, the note image 68 representing the pitch Fm is arranged at a position corresponding to the pitch Fm in the direction of the pitch axis Ap. That is, each note image 68 is arranged at a position overlapping or adjacent to the staff 65.

[0106] As illustrated above, the confirmation screen 60 displays the time series of the pronunciation index C(t) (transition image 64), the time series of the sounding point Tm (indication image 67), and the time series of the pitch Fm (musical note image 68) on a common time axis At. Therefore, the user U can visually and intuitively confirm the temporal relationship between the pronunciation index C(t), the sounding point Tm, and the pitch Fm.

[0107] Furthermore, the confirmation screen 60 includes a plurality of operation images 71 (71a, 71b, 71c) and an operation image 72. Each operation image 71 is an operator that the user U can operate using the operation device 14. Specifically, the operation image 71a is an image for accepting an instruction from the user U to change the variable α. The operation image 71b is an image for accepting an instruction from the user U to change the variable β. The operation image 71c is an image for accepting an instruction from the user U to change the variable γ.

[0108] The index calculation unit 51 (the second analysis unit 512) changes the variable α in response to an instruction from the user U to the operation image 71a. The index calculation unit 51 calculates the pronunciation index C(t) by a calculation to which the changed variable α is applied. The display control unit 36 ​​updates the transition image 64 of the confirmation screen 60 every time the pronunciation index C(t) is calculated. As described above, the more the variable α increases, the smaller the pronunciation index C(t) becomes during the period in which the acoustic components of the performance device 20 are predominant, and as a result, the tendency is that the voice presence index G(t) is more likely to be set to the value g0. Therefore, as illustrated in FIG. 16, the more the variable α increases, the lighter the gradation becomes during the period in which the acoustic components of the performance device 20 are predominant in the transition image 64, and the longer the non-performance period 622 becomes. The user U operates the operation image 71a while checking the confirmation screen 60 so that the performance period 621 approaches the period in which the user U1 played the musical instrument 80 during the preparation period.

[0109] The period estimation unit 52 changes the variable β in response to an instruction from the user U to the operation image 71b. The period estimation unit 52 calculates the sound presence probability Λ by a calculation applying the changed variable β. The display control unit 36 ​​updates the performance period 621 on the confirmation screen 60 each time the sound presence probability Λ is calculated. As described above, the more the variable β increases, the more likely the sound presence index G(t) is to be set to the numerical value g0. Therefore, as illustrated in FIG. 17, the more the variable β increases, the more the non-performance period 622 expands. The user U operates the operation image 71b while checking the confirmation screen 60 so that the performance period 621 approaches the period during which the user U1 played the musical instrument 80 in the preparation period.

[0110] The pitch estimation unit 53 changes the variable γ in response to an instruction from the user U to the operation image 71c. The pitch estimation unit 53 calculates the transition probability λ(n1, n2) by a calculation that applies the changed variable γ. The display control unit 36 ​​updates the instruction image 67 and the note image 68 on the confirmation screen 60 each time the transition probability λ(n1, n2) is calculated. As described above, the transition probability λ(n1, n2) increases as the variable γ increases. Therefore, as illustrated in FIG. 18, the number of instruction images 67 (sound generation points Tm) and the number of note images 68 (pitch Fm) increase as the variable γ increases. The user U operates the operation image 71b while checking the confirmation screen 60 so as to approach the performance content of the user U1 in the preparation period.

[0111] The operation image 72 is an image for receiving an instruction to save the reference data Da from the user U. The information generating unit 54 saves the analysis contents (the pronunciation index E(t,n), each pronunciation point Tm, and the voiced index G(t)) at the time when the operation image 72 is operated in the storage device 12 as the reference data Da.

[0112] 19 is a flow chart illustrating a specific procedure of the adjustment process Sc. After the performance data Db and the reference signal Zr are acquired in the preparation period, the adjustment process Sc is started, for example, in response to an instruction from the user U via the operation device 14.

[0113] When the adjustment process Sc is started, the acoustic analysis unit 43 executes an analysis process Sc1 that analyzes the reference signal Zr. The analysis process Sc1 includes an index calculation process Sc11, a period estimation process Sc12, a pitch estimation process Sc13, and an information generation process Sc14. The index calculation process Sc11 is an example of a "first process", the period estimation process Sc12 is an example of a "second process", and the pitch estimation process Sc13 is an example of a "third process". Furthermore, the variable α is an example of a "first variable", the variable β is an example of a "second variable", and the variable γ is an example of a "third variable".

[0114] The index calculation unit 51 calculates the pronunciation index C(t) by using the performance data Db and the reference signal Zr (index calculation process Sc11). As described above, the index calculation process Sc11 includes the calculation of the first index B1(t,n) by the first analysis unit 511, the calculation of the second index B2(t,n) by the second analysis unit 512, the calculation of the pronunciation index E(t,n) by the first calculation unit 513, and the calculation of the pronunciation index C(t) by the second calculation unit 514. A variable α set according to an instruction from the user U is applied to the index calculation process Sc11.

[0115] The period estimation unit 52 estimates the performance period 621 by calculating a voice presence index G(t) using the onset index C(t) (period estimation process Sc12). A variable β set in response to an instruction from the user U is applied to the period estimation process Sc12. Furthermore, the pitch estimation unit 53 estimates the pitch K(t) of the acoustic component of the musical instrument 80 (pitch estimation process Sc13). A variable γ set in response to an instruction from the user U is applied to the pitch estimation process Sc13. Then, the information generation unit 54 identifies a plurality of onset points Tm on the time axis and the pitch Fm for each onset point Tm (information generation process Sc4).

[0116] The display control unit 36 ​​causes the display device 15 to display a confirmation screen 60 showing the result of the analysis process Sc1 exemplified above (Sc2). Specifically, a performance period 621 on the time axis At, a transition image 64 showing the sounding index C(t), an indication image 67 showing each sounding point Tm, and a note image 68 showing each pitch Fm are displayed on the confirmation screen 60.

[0117] The acoustic analysis unit 43 judges whether the operation image 71 (71a, 71b, 71c) has been operated (Sc3). That is, it is judged whether the user U has instructed to change the variable (α, β, or γ). If the operation image 71 has been operated (Sc3: YES), the acoustic analysis unit 43 executes an analysis process Sc1 to which each of the changed variables (α, β, or γ) has been applied, and updates (Sc2) the confirmation screen 60 according to the result of the analysis process Sc1. The calculation of the first index B1(t, n) by the first analysis unit 511 may be executed only once in an index calculation process Sc11 immediately after the start of the analysis process Sc1.

[0118] If the operation image 71 is not operated (Sc3: NO), the acoustic analysis unit 43 judges whether or not the operation image 72 is operated (Sc4). That is, it is judged whether or not the user U instructs to confirm the reference data Da. If the operation image 72 is not operated (Sc4: NO), the acoustic analysis unit 43 shifts the process to step Sc3. On the other hand, if the operation image 72 is operated (Sc4: YES), the information generation unit 54 saves the current result of the analysis process Sc1 (the pronunciation index E(t,n), each pronunciation point Tm, and the voiced index G(t)) in the storage device 12 as reference data Da (Sc5). The adjustment process Sc ends when the reference data Da is saved.

[0119] As described above, in the first embodiment, the time series of the pronunciation index C(t) (transition image 64) and the time series of the pitch Fm (musical note image 68) are displayed on a common time axis At, so that the user U can easily confirm and correct the analysis result of the reference signal Zr in the process of generating the reference data Da. Specifically, the user U can visually and intuitively confirm the temporal relationship between the pronunciation index C(t) and the pitch Fm.

[0120] In the first embodiment, the pronunciation index C(t) is calculated by subtracting the second index B2(t,n) calculated by analyzing the performance data Db from the first index B1(t,n) calculated by analyzing the reference signal Zr. Therefore, the pronunciation index C(t) in which the influence of the acoustic components of the performance device 20 is reduced can be calculated. In other words, the pronunciation index C(t) in which the acoustic components of the musical instrument 80 are emphasized can be calculated. In addition, the variable α set according to an instruction from the user U is applied to the calculation of the second index B2(t,n), so that the user U can adjust the pronunciation index C(t) to suit the performance of the musical instrument 80 in the preparation period.

[0121] In the first embodiment, in addition to the time series of the pronunciation index C(t) (transition image 64) and the time series of the pitch Fm (musical note image 68), the performance period 621 is displayed on a common time axis At. Therefore, the user U can visually and intuitively confirm the temporal relationship between the pronunciation index C(t), the pitch Fm, and the performance period 621. In addition, since the variable β set according to an instruction from the user U is applied to the period estimation process Sc12, the user U can adjust the performance period 621 to match the performance of the musical instrument 80 in the preparation period.

[0122] In the first embodiment, in addition to the time series of the onset index C(t) (transition image 64) and the time series of the pitch Fm (note image 68), the time series of the onset point Tm (instruction image 67) are displayed on a common time axis At. Therefore, the user U can visually and intuitively confirm the temporal relationship between the onset index C(t), the pitch Fm, and the onset point Tm. In addition, since the variable γ set according to an instruction from the user U is applied to the pitch estimation process Sc13, the user U can adjust the onset point Tm to match the performance of the musical instrument 80 during the preparation period.

[0123] F: Variation Specific modified embodiments added to each of the above-mentioned embodiments are exemplified below. Two or more embodiments selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0124] (1) In each of the above-mentioned embodiments, the instruction receiving unit 33 receives an operation for transitioning the operation device 14 from a released state to an operating state as the first instruction Q1, but the form of the first instruction Q1 is not limited to the above examples. For example, a specific motion by the user U is detected as the first instruction Q1. To detect the motion of the user U, various detection devices such as an imaging device or an acceleration sensor are used. For example, the instruction receiving unit 33 identifies various motions such as the motion of the user U lifting one hand, the motion of lifting the musical instrument 80, or the motion of breathing (for example, the motion of inhaling) as the first instruction Q1. The breathing by the user U is, for example, a breath (breathing) when playing a wind instrument as the musical instrument 80. The operation speed V1 in the second embodiment is comprehensively expressed as the speed of the motion of the user U identified as the first instruction Q1.

[0125] Specific data (hereinafter referred to as "first data") meaning the first instruction Q1 may be included in the performance data Db. The first data is, for example, data meaning a fermata in a piece of music. The instruction receiving unit 33 determines that the first instruction Q1 has occurred when the playback position Y reaches the point of the first data. As can be understood from the above explanation, the first instruction Q1 is not limited to an instruction from the user U. Note that, when the dispersion degree Δ exceeds the threshold value Δth in the editing process Sb, the editing processing unit 34 may add the first data to the note N1.

[0126] (2) In each of the above-described embodiments, the instruction receiving unit 33 receives an operation for transitioning the operating device 14 from an operating state to a released state as the second instruction Q2, but the aspect of the second instruction Q2 is not limited to the above examples. For example, similar to the first instruction Q1 in the first embodiment, the instruction receiving unit 33 may receive an operation for transitioning the operating device 14 from a released state to an operating state as the second instruction Q2. In other words, two operations including stepping on and releasing the movable part 141 may be detected as the first instruction Q1 and the second instruction Q2.

[0127] Also, a specific motion of the user U may be detected as the second instruction Q2. To detect the motion of the user U, various detection devices such as an imaging device or an acceleration sensor are used. For example, the instruction receiving unit 33 identifies various motions such as the motion of the user U lowering one hand, the motion of lowering the musical instrument 80, or the motion of breathing (for example, the motion of exhaling) as the second instruction Q2. The breathing of the user U is, for example, a breath when playing a wind instrument as the musical instrument 80. The operation speed V2 in the second embodiment is comprehensively expressed as the speed of the motion of the user U identified as the second instruction Q2.

[0128] Specific data (hereinafter referred to as "second data") representing the second instruction Q2 may be included in the performance data Db. The second data is, for example, data representing a fermata in a piece of music. The instruction receiving unit 33 determines that the second instruction Q2 has occurred when the playback position Y reaches the point of the second data. As can be understood from the above explanation, the second instruction Q2 is not limited to an instruction from the user U.

[0129] As shown in the above example, a configuration is assumed in which one of a series of mutually paired actions of the user U is accepted as the first instruction Q1, and the other is accepted as the second instruction Q2. For example, the action of the user U raising one hand is accepted as the first instruction Q1, and the action of lowering the one hand following that action is accepted as the second instruction Q2. Also, the action of the user U lifting the musical instrument 80 is accepted as the first instruction Q1, and the action of lowering the musical instrument 80 following that action is accepted as the second instruction Q2. Similarly, the action of the user U inhaling is accepted as the first instruction Q1, and the action of exhaling following that action is accepted as the second instruction Q2.

[0130] However, the first instruction Q1 and the second instruction Q2 do not have to be the same type of action by the user U. That is, separate actions that the user U can perform independently of each other may be detected as the first instruction Q1 and the second instruction Q2. For example, the instruction receiving unit 33 may detect an operation on the operating device 14 as the first instruction Q1, and detect another action, such as an action of lifting the musical instrument 80 or an action of breathing, as the second instruction Q2.

[0131] (3) In each of the above-mentioned embodiments, an automatic musical instrument is exemplified as the performance device 20, but the configuration of the performance device 20 is not limited to the above examples. For example, a sound source system including a sound source device that generates an audio signal of a musical tone in response to an instruction from the sound processing system 10 and a sound emitting device that reproduces the musical tone represented by the audio signal may be adopted as the performance device 20. The sound source device is realized as a hardware sound source or a software sound source. The same applies to the performance device 20 in the fifth embodiment.

[0132] (4) In the fifth embodiment, the entire analysis process Sc1 was executed every time the user U changed a variable (Sc3: YES), but the conditions for executing each process (Sc11-Sc14) included in the analysis process Sc1 are not limited to the above examples. In the following description, it is assumed that the user U changes a variable by operating the operation image 71 (71a, 71b, 71c). Specifically, the user U selects the operation image 71 by operating the operation device 14, and moves the operation image 71 while maintaining the selection. The numerical value of the variable is changed to a numerical value according to the position of the operation image 71 at the time the selection is released. In other words, the release of the selection of the operation image 71 means that the numerical value of the variable is confirmed.

[0133] Assume that the user U changes the variable α by operating the operation image 71a. In the process of moving the selected operation image 71a, the index calculation unit 51 updates the pronunciation index C(t) from time to time by repeating the index calculation process Sc11. The display control unit 36 ​​updates the transition image 64 according to the updated pronunciation index C(t) every time the index calculation process Sc11 is executed. That is, in parallel with the movement of the operation image 71a (change of the variable α), the index calculation process Sc11 and the update of the transition image 64 are executed. In the state where the operation image 71a is selected, the period estimation process Sc12, the pitch estimation process Sc13, and the information generation process Sc14 are not executed. When the selection of the operation image 71a is released, the period estimation process Sc12, the pitch estimation process Sc13, and the information generation process Sc14 are executed using the pronunciation index C(t) at that time, and the confirmation screen 60 is updated according to the processing result. In the above configuration, when the operation image 71a is selected, the period estimation process Sc12, the pitch estimation process Sc13, and the information generation process Sc14 are not executed, so that the processing load of the adjustment process Sc is reduced.

[0134] Assume that the user U changes the variable β by operating the operation image 71b. In the process of moving the selected operation image 71b, the period estimation unit 52 updates the sound presence indicator G(t) as needed by repeating the period estimation process Sc12. The display control unit 36 ​​updates the performance period 621 on the confirmation screen 60 each time the period estimation process Sc12 is executed. When the operation image 71a is selected, the indicator calculation process Sc11, the pitch estimation process Sc13, and the information generation process Sc14 are not executed. When the selection of the operation image 71b is deselected, the pitch estimation process Sc13 and the information generation process Sc14 are executed using the sound presence indicator G(t) at that time point, and the confirmation screen 60 is updated according to the processing results.

[0135] Assume that the user U changes the variable γ by operating the operation image 71c. While the selected operation image 71c is moving, the analysis process Sc1 and the update (Sc2) of the confirmation screen 60 are not executed. When the selection of the operation image 71c is deselected, a pitch estimation process Sc13 that applies the changed variable γ and an information generation process Sc14 that applies the pitch K(t) calculated in the pitch estimation process Sc13 are executed. With the above configuration, the number of times the pitch estimation process Sc13 and the information generation process Sc14 are executed is reduced, thereby reducing the processing load of the adjustment process Sc.

[0136] (5) The form of the transition image 64 representing the pronunciation index C(t) is not limited to the above examples. For example, as illustrated in FIG. 20, a waveform on the time axis may be displayed on the display device 15 as the transition image 64. The amplitude of the waveform of the transition image 64 at time point t on the time axis At is set according to the pronunciation index C(t). For example, a waveform with a large amplitude at time point t when the pronunciation index C(t) is large is displayed as the transition image 64. The transition image 64 may be displayed overlapping the musical staff 65.

[0137] (6) The process by which the index calculation unit 51 calculates the onset index C(t) is not limited to the process exemplified in the fifth embodiment. For example, the index calculation unit 51 may calculate the onset index E(t,n) by subtracting the amplitude spectrum of the musical sound of the performance device 20 from the amplitude spectrum of the reference signal Zr. The amplitude spectrum of the musical sound of the performance device 20 is generated, for example, by a known sound source process that generates a musical sound signal representing the musical sound specified by the performance data Db, and a frequency analysis such as a discrete Fourier transform of the musical sound signal. The amplitude spectrum after subtraction corresponds to a series of N onset indexes E(t,n) corresponding to different pitches Pn. The degree to which the amplitude spectrogram of the musical sound of the performance device 20 is subtracted is adjusted according to the variable α.

[0138] (7) The process by which the period estimation unit 52 estimates the performance period is not limited to the process exemplified in the fifth embodiment. For example, the period estimation unit 52 estimates, as the performance period, a period during which the signal strength of the reference signal Zr exceeds a threshold. The threshold is adjusted according to the variable β. In addition, the process by which the pitch estimation unit 53 estimates the pitch K(t) is not limited to the process exemplified above.

[0139] (8) The sound processing system 10 may be realized by a server device that communicates with a terminal device such as a smartphone or a tablet terminal. For example, the terminal device includes a sound collection device 13 that generates a sound signal Z according to a performance by a user U, and a performance device 20 that plays a piece of music according to an instruction from the sound processing system 10. The terminal device transmits the sound signal Z generated by the sound collection device 13 and a first instruction Q1 and a second instruction Q2 according to the movement of the user U to the sound processing system 10 via a communication network. The sound processing system 10 causes the performance device 20 of the terminal device to play a part of the music in accordance with the performance position X estimated from the sound signal Z and the first instruction Q1 and the second instruction Q2 received from the terminal device. The performance analysis unit 31 may be mounted on the terminal device. The terminal device transmits the performance position X estimated by the performance analysis unit 31 to the sound processing system 10. In the above configuration, the performance analysis unit 31 is omitted from the sound processing system 10. The sound processing system 10 of the fifth embodiment is also realized by a server device. For example, the sound processing system 10 generates reference data Da by performing an analysis process Sc1 using the reference signal Zr and performance data Db received from the terminal device, and transmits the reference data Da to the terminal device.

[0140] (9) As described above, the functions of the sound processing system 10 exemplified above are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12. The program according to the present disclosure can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, a storage device that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0141] G: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0142] The acoustic processing system according to one aspect (aspect 1) of the present disclosure includes an acoustic analysis unit that executes an analysis process for analyzing an acoustic signal that records the sound of a first sound source, the analysis process including a first process for identifying a time series of a pronunciation index that is an index of the likelihood that the acoustic signal contains an acoustic component of the first sound source, and a process for identifying a time series of a pitch related to the acoustic component of the first sound source, and a display control unit that causes a display device to display the time series of the pronunciation index and the time series of the pitch on a common time axis. In the above aspect, since the time series of the pronunciation index and the time series of the pitch are displayed on a common time axis, a user can easily confirm and correct the analysis result of the acoustic signal in the process of generating reference data. Specifically, a user can visually and intuitively confirm the temporal relationship between the pronunciation index and the pitch.

[0143] In a specific example (aspect 2) of aspect 1, the audio signal includes an audio component of the first sound source and an audio component of the second sound source, and the first process includes a process of calculating a first index according to the intensity of the audio signal, a process of calculating a second index according to the intensity of the audio component of the second sound source from performance data that specifies the audio intensity for each note for the second sound source, and a process of subtracting the second index from the first index, and the pronunciation index is an index according to the result of the subtraction. In the above aspect, the pronunciation index is calculated from the result of subtracting the second index calculated according to the performance data from the first index according to the intensity of the audio signal including the audio component of the first sound source and the audio component of the second sound source. Therefore, it is possible to calculate a pronunciation index in which the influence of the audio component of the second sound source is reduced (i.e., a pronunciation index in which the audio component of the first sound source is emphasized).

[0144] In a specific example (aspect 3) of aspect 2, a first variable set in response to an instruction from a user is applied to the process of calculating the second indicator, and the second indicator changes in response to the first variable. According to the above aspect, the user can adjust the pronunciation indicator so as to match the known pronunciation content (e.g., the performance content) of the first sound source.

[0145] In a specific example (Aspect 4) of any one of Aspects 1 to 3, the analysis process further includes a second process of identifying a pronunciation period in which an acoustic component of the first sound source exists, and the display control unit displays the pronunciation period on the time axis. According to the above aspect, a user can visually and intuitively confirm the temporal relationship between the pronunciation index and the pitch and the pronunciation period of the first sound source.

[0146] In a specific example (aspect 5) of aspect 4, a second variable set in response to an instruction from a user is applied to the second process, and the sound generation period changes in response to the second variable. According to the above aspect, the user can adjust the sound generation period to match the known sound generation content (e.g., the performance content) of the first sound source.

[0147] In a specific example (Aspect 6) of any one of Aspects 1 to 5, the analysis process further includes a third process of identifying a time series of the pitch of the acoustic component of the first sound source, and the display control unit displays on the time axis an onset point, which is a time point at which the pitch changes. According to the above aspects, a user can visually and intuitively confirm the temporal relationship between the onset index and the pitch and the onset point of the first sound source.

[0148] In a specific example (aspect 7) of aspect 6, a third variable set in response to an instruction from a user is applied to the third process, and the number of the sound generation points is changed in response to the third variable. According to the above aspect, the user can adjust the sound generation points to match the known sound generation content (e.g., the performance content) of the first sound source.

[0149] An acoustic processing method according to one aspect (aspect 8) of the present disclosure executes an analysis process for analyzing an acoustic signal recording the sound of a first sound source, the analysis process including a first process for identifying a time series of a pronunciation index, which is an index of the likelihood that the acoustic signal contains an acoustic component of the first sound source, and a process for identifying a time series of a pitch related to the acoustic component of the first sound source, and displays the time series of the pronunciation index and the time series of the pitch on a common time axis on a display device. Note that analysis process Sc1 in FIG. 19 is an example of an "analysis process" in aspect 8. Also, step Sc2 in FIG. 19 is an example of a "process for displaying the time series of the pronunciation index and the time series of the pitch on a common time axis on a display device" in aspect 8.

[0150] A program according to one aspect (aspect 9) of the present disclosure causes a computer system to function as an acoustic analysis unit that executes an analysis process to analyze an acoustic signal recording the sound of a first sound source, the analysis process including a first process of identifying a time series of a pronunciation index, which is an index of the likelihood that the acoustic signal contains an acoustic component of the first sound source, and a process of identifying a time series of pitch related to the acoustic component of the first sound source, and a display control unit that displays the time series of the pronunciation index and the time series of the pitch on a common time axis on a display device. [Explanation of symbols]

[0151] 100...playback system, 10...sound processing system, 11...control device, 12...storage device, 13...sound collection device, 14...operation device, 15...display device, 20...performance device, 21...drive mechanism, 22...sound generation mechanism, 31...performance analysis unit, 32...playback control unit, 33...instruction reception unit, 34...edit processing unit, 35...preparation processing unit, 36...display control unit, 41...first recording unit, 42...second recording unit, 43...sound analysis unit, 51...index calculation unit, 5 2...period estimation unit, 53...pitch estimation unit, 54...information generation unit, 60...confirmation screen, 61...first area, 62...second area, 64...transition image, 65...staff sheet, 67...instruction image, 68...note image, 71 (71a, 71b, 71c), 72...operation image, 80...musical instrument, 141...moving part, 511...first analysis unit, 512...second analysis unit, 513...first calculation unit, 514...second calculation unit, 621...performance period, 622...non-performance period.

Claims

1. an acoustic analysis unit that executes an analysis process for analyzing an acoustic signal that includes a first sound source and a second sound source, the analysis process including a first process for determining a time series of a pronunciation index that is an index of a probability that an acoustic component of the first sound source is included in the acoustic signal, and a process for determining a time series of a pitch related to the acoustic component of the first sound source; a display control unit that displays the time series of the pronunciation indexes and the time series of the pitches on a display device on a common time axis; Equipped with The first process includes: A process of calculating a first index according to the intensity of the acoustic signal; calculating a second indicator corresponding to the intensity of the acoustic component of the second sound source from performance data that specifies the sound intensity of each note of the second sound source; and subtracting the second index from the first index; The pronunciation index is an index according to the result of the subtraction. Sound processing system.

2. A first variable set in response to an instruction from a user is applied to the process of calculating the second index; The second index varies depending on the first variable. The sound processing system of claim 1 .

3. an acoustic analysis unit that executes an analysis process for analyzing an acoustic signal that records a sound of a first sound source, the analysis process including a first process for identifying a time series of a pronunciation index that is an index of a probability that the acoustic signal contains an acoustic component of the first sound source, a second process for identifying a pronunciation period in which the acoustic component of the first sound source exists, and a process for identifying a time series of a pitch related to the acoustic component of the first sound source; a display control unit that displays the time series of the pronunciation indexes and the time series of the pronunciation periods and the pitches on a display device on a common time axis; Equipped with A second variable set in response to an instruction from a user is applied to the second process, The sound generation period varies depending on the second variable. Sound processing system.

4. an acoustic analysis unit that executes an analysis process for analyzing an acoustic signal that records a sound of a first sound source, the analysis process including a first process for determining a time series of a pronunciation index that is an index of a probability that the acoustic signal contains an acoustic component of the first sound source, and a third process for determining a time series of a pitch of the acoustic component of the first sound source; a display control unit that displays, on a display device, a time series of the pronunciation indexes, a time series of the pitches, and the pronunciation points at which the pitches change, on a common time axis; Equipped with A third variable set in response to an instruction from a user is applied to the third process, The number of the sound generation points varies depending on the third variable. Sound processing system.

5. executing an analysis process for analyzing an audio signal that includes audio from a first sound source and audio from a second sound source, the analysis process including a first process for determining a time series of a pronunciation index that is an index of a probability that the audio signal includes an audio component of the first sound source, and a process for determining a time series of a pitch related to the audio component of the first sound source; displaying the time series of the pronunciation indicators and the time series of the pitches on a common time axis on a display device; The first process includes: A process of calculating a first index according to the intensity of the acoustic signal; calculating a second indicator corresponding to the intensity of the acoustic component of the second sound source from performance data that specifies the sound intensity of each note of the second sound source; and subtracting the second index from the first index; The pronunciation index is an index according to the result of the subtraction. An audio processing method implemented by a computer system.

6. Executing an analysis process for analyzing an audio signal that records audio from a first sound source, the analysis process including a first process for identifying a time series of pronunciation indices that are indices of a probability that the audio signal contains an audio component of the first sound source, a second process for identifying an audio period in which the audio component of the first sound source exists, and a process for identifying a time series of pitches related to the audio component of the first sound source, displaying the time series of the pronunciation indexes and the time series of the pronunciation periods and the pitches on a display device on a common time axis; A second variable set in response to an instruction from a user is applied to the second process, The sound generation period varies depending on the second variable. An audio processing method implemented by a computer system.

7. Executing an analysis process for analyzing an audio signal that records audio from a first sound source, the analysis process including a first process for specifying a time series of a pronunciation index that is an index of a probability that the audio signal contains an audio component of the first sound source, and a third process for specifying a time series of a pitch of the audio component of the first sound source, displaying, on a display device, a time series of the pronunciation indexes, a time series of the pitches, and the pronunciation points at which the pitches change, on a common time axis; A third variable set in response to an instruction from a user is applied to the third process, The number of the sound generation points varies depending on the third variable. An audio processing method implemented by a computer system.

8. an acoustic analysis unit that executes an analysis process for analyzing an acoustic signal that includes a sound from a first sound source and a sound from a second sound source, the analysis process including a first process for determining a time series of a pronunciation index that is an index of a probability that the acoustic signal includes an acoustic component of the first sound source, and a process for determining a time series of a pitch related to the acoustic component of the first sound source; and a display control unit that displays the time series of the pronunciation indexes and the time series of the pitches on a display device on a common time axis; A program for causing a computer system to function as The first process includes: A process of calculating a first index according to the intensity of the acoustic signal; calculating a second indicator corresponding to the intensity of the acoustic component of the second sound source from performance data that specifies the sound intensity of each note of the second sound source; and subtracting the second index from the first index; The pronunciation index is an index according to the result of the subtraction. program.

9. an acoustic analysis unit that executes an analysis process for analyzing an acoustic signal that records the sound of a first sound source, the analysis process including a first process for identifying a time series of a pronunciation index that is an index of a probability that the acoustic signal contains an acoustic component of the first sound source, a second process for identifying a pronunciation period in which the acoustic component of the first sound source exists, and a process for identifying a time series of a pitch related to the acoustic component of the first sound source; and a display control unit that displays, on a display device, a time series of the pronunciation indexes and a time series of the pronunciation periods and the pitches on a common time axis; A program for causing a computer system to function as A second variable set in response to an instruction from a user is applied to the second process, The sound generation period varies depending on the second variable. program.

10. an acoustic analysis unit that executes an analysis process for analyzing an acoustic signal that records a sound of a first sound source, the analysis process including a first process for determining a time series of a pronunciation index that is an index of a probability that the acoustic signal contains an acoustic component of the first sound source, and a third process for determining a time series of a pitch of the acoustic component of the first sound source; and a display control unit that displays, on a display device, a time series of the pronunciation indexes, a time series of the pitches, and the pronunciation points at which the pitches change, on a common time axis; A program for causing a computer system to function as A third variable set in response to an instruction from a user is applied to the third process, The number of the sound generation points varies depending on the third variable. program.

Images